97efc95efc
fix: better object store write error propagation
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-08-05 19:17:58 +07:00
83b8a390cd
fix: always do stream counting for object store upload
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-08-05 19:17:58 +07:00
398b106f0c
fix: validate written size for s3 multipart uploads
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-08-05 19:17:58 +07:00
5981b7eb51
chore: apply new CSFixer rules
...
Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
# Conflicts:
# apps/settings/lib/SetupChecks/PhpOpcacheSetup.php
2025-07-01 16:26:50 +07:00
2e1c316582
fix: remove double cache write in ObjectStoreStorage::touch
...
the file_put_contents already puts all of those values into the cache
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-06-18 22:22:51 +07:00
ea98e4b3f5
Merge pull request #53419 from nextcloud/leftybournes/fix/object_storage_large_uploads
...
fix(s3): retry failed multipart uploads with decreased concurrency
2025-06-18 21:07:06 +07:00
d7ae952dc8
fix(ObjectStore): Make S3 "connect_timeout" option configurable
...
The hardcoded connection timeout of 5 seconds may not be enough in some
cases, so now it is got from the ObjectStore arguments in Nextcloud
configuration, falling back to 5 if not set.
The connection timeout is set in seconds, but decimal precision can be
used for subsecond accuracy (for example, 4.2 for 4200 milliseconds).
Signed-off-by: Daniel Calviño Sánchez <danxuliu@gmail.com>
2025-06-16 20:17:50 +07:00
06b5ce5859
fix(s3): retry failed multipart uploads with decreased concurrency
...
Signed-off-by: Kent Delante <kent.delante@proton.me>
2025-06-10 17:39:59 +07:00
f4cb78b905
Merge pull request #52242 from nextcloud/artonge/fix/copy_subfolders_s3
2025-06-02 11:55:54 +07:00
5b3f4cd8fe
perf: set the folder size in the same query as we create it
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-05-28 15:52:00 +07:00
1f8b823992
fix(S3): Use original folder size during copy
...
This prevent having copied folders with a wrongly set size of 0KB.
- Fix https://github.com/nextcloud/server/issues/51916
Signed-off-by: Louis Chemineau <louis@chmn.me>
2025-05-28 15:52:00 +07:00
7599162c7b
feat: move primary object store configuration to a single place
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-05-28 15:32:31 +07:00
9592390070
Merge pull request #51779 from nextcloud/object-store-filename
2025-05-06 15:21:01 +07:00
49baa914cf
fix(objectstorage): add retry attempts to S3 connection
...
Signed-off-by: Anna Larch <anna@nextcloud.com>
2025-04-23 12:15:06 +07:00
8fdf2a7eae
fix: Transfer ownership with S3 as primary
...
When using S3 as primary storage, transferring ownership with the `--move` option fail with the following error:
`SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '8-45b963397aa40d4a0063e0d85e4fe7a1' for key 'fs_storage_path_hash'`
The `--move` option moves the entire home folder from one account to another.
The error means that the move failed because the destination folder already exist in `oc_filecache`.
- With S3 as primary storage, folders only exists as entries in `oc_filecache`.
- With S3 as primary storage, `moveFromStorage(...)` only moves the cache entry, as nothing needs to be moved on disk. This cache move does not delete potentially pre-existing destination folder.
- With Local storage, `moveFromStorage(...)` calls `rename(...)` which delete pre-existing folder.
- `transfer(...)`: 687a4d9ac7/apps/files/lib/Service/OwnershipTransferService.php (L112)687a4d9ac7/lib/private/Files/SetupManager.php (L261-L262)687a4d9ac7/lib/private/Files/ObjectStore/ObjectStoreStorage.php (L91-L135)687a4d9ac7/lib/private/Files/ObjectStore/ObjectStoreStorage.php (L635-L636)
2025-04-01 13:53:56 +07:00
8aae332622
feat: store original storage id and path in object store metadata
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-03-31 15:10:58 +07:00
94114b99f7
feat: more generic way of passing metadata to object storage backends for new objects
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-03-31 14:55:18 +07:00
fcde776683
feat: add command to list objects
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-03-31 14:30:40 +07:00
c3bc362f48
feat: add command to get object metadata
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-03-31 14:30:40 +07:00
12a92a2e8b
Merge branch 'master' into feat/support-aws-session-token
2025-03-04 18:27:54 +07:00
9bdee76535
perf(files): faster query to fetch incomplete directories
...
Signed-off-by: Benjamin Gaussorgues <benjamin.gaussorgues@nextcloud.com>
2025-02-13 10:11:02 +07:00
6b4c859a41
feat(object_store): Add support for session token in AWS credentials
...
- Pass session token, either null or with value, to the AWS Credentials constructor
Signed-off-by: Hector Valcarcel <hmvalcarcel@gmail.com>
2025-02-06 16:24:19 +07:00
f1c025dfd3
fix(s3): treat empty sse_c_key as not set
...
Co-authored-by: Christian Becker <christian@dabecka.de>
Signed-off-by: Richard Steinmetz <richard@steinmetz.cloud>
2025-01-28 22:59:59 +07:00
b48ee2e924
fix: Harden files scanner for invalid null access
...
Co-authored-by: Ferdinand Thiessen <opensource@fthiessen.de>
Co-authored-by: Kate <26026535+provokateurin@users.noreply.github.com>
Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
2025-01-28 20:08:46 +07:00
5896f8c37f
chore: cleanup leftover debug statement
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-12-09 15:40:01 +07:00
6cf66f95ce
fix: get object size from stream where possible
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-12-05 15:16:22 +07:00
ca547bb913
fix: don't perform the extra buffering in s3 stream write when the stream size is known
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-12-05 14:07:40 +07:00
7da8951c57
fix: use php://temp instead of php://memory for multi-part upload buffer
...
this should reduce potential memory issues if the limit is set very high
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-12-05 14:07:37 +07:00
df3a9e221d
fix: throw correct exception type when we can't verify if an s3 bucket exists
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-11-26 16:49:55 +07:00
0de4843b73
refactor(Storage): Align all Storage constructors
...
Signed-off-by: provokateurin <kate@provokateurin.de>
2024-10-23 13:24:18 +07:00
f28e74b7a8
refactor(Storage): Make all parameter types strong types
...
Signed-off-by: provokateurin <kate@provokateurin.de>
2024-10-07 15:00:05 +07:00
3e12e1e789
fix: rework move into object store to better preserve fileids
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-27 14:46:08 +07:00
888d06dff9
fix: preserve fileid when moving from objectstore to non-objectstore
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-27 14:45:42 +07:00
da21acfb3f
fix: ensure source folder is removed from cache when moving to objectstore
...
otherwise this causes confusion down the line as it's contents will be moved to the new cache
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-27 14:45:42 +07:00
7cdccd058f
fix(Storage): Fix IStorage return types
...
Signed-off-by: provokateurin <kate@provokateurin.de>
2024-09-26 18:13:03 +07:00
47d281865e
perf: reuse cache info we already have when moving to object store
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-19 19:06:06 +07:00
9836e9b164
chore(deps): Update nextcloud/coding-standard to v1.3.1
...
Signed-off-by: provokateurin <kate@provokateurin.de>
2024-09-19 14:21:20 +07:00
3c0c81bb94
fix: ensure parent folder exists when writing a file to object storage
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-18 14:11:27 +07:00
e0fb25f846
fix: fix object store id for test object store
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-18 13:54:55 +07:00
60f6347c97
fix: cleanup objectstore file_put_content
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-18 13:54:55 +07:00
3e94ee391f
fix: create intermediate directories for objectstore moveFromStorage
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-18 13:54:55 +07:00
879fcc4253
fix: get source file before moving the cache item in objectstore moveFromStorage
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-18 13:54:55 +07:00
f25acfb799
Merge pull request #47986 from cfiehe/fix_move_on_same_bucket
...
perf(ObjectStoreStorage): Improve (slow) move on same object bucket
2024-09-17 16:35:19 +07:00
d2d1e3284d
fix(Storage): Fix method docs inheritance
...
Signed-off-by: provokateurin <kate@provokateurin.de>
2024-09-17 10:20:38 +07:00
8ca6fcace7
fix(Storage): Document getOwner() can return false
...
Signed-off-by: provokateurin <kate@provokateurin.de>
2024-09-17 10:10:50 +07:00
492e6997d8
chore: Fix psalm issues, put back private versions of getter in private Storage interface
...
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2024-09-16 18:11:38 +07:00
9597072ada
perf(ObjectStoreStorage): Improve (slow) move on same object bucket
...
This commit fixes the issue #47856 . When you upload a file into a group folder and when you use a single S3 bucket as primary storage, the final move operation hangs for a long time. In the background, Nextcloud initiates a copy-delete sequence from the bucket into the bucket, with causes a lot unnecessary overhead. Nextcloud thinks that the file must be imported to another storage and does not recognize that everything is done on the same object bucket. In that case, the import step can be completely skipped, which saves time, network bandwidth and reduces the load on the object storage.
The behavior improves a lot with https://github.com/nextcloud/server/pull/46013 . However, there are still some put messages that are being sent to the object storage when you use an object storage as primary storage and upload files into a group folder.
Co-authored-by: Kate <26026535+provokateurin@users.noreply.github.com>
Signed-off-by: Christoph Fiehe <c.fiehe@eurodata.de>
2024-09-16 15:11:09 +07:00
7c1c02d6a5
fix(S3ConfigTrait): Allow proxy field to take false
...
Signed-off-by: provokateurin <kate@provokateurin.de>
2024-09-15 15:41:58 +07:00
508d8e7c01
fix: make swift connect exception message more informative
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-09 17:15:01 +07:00
6d2253e692
Merge pull request #46013 from nextcloud/obj-store-move-from-storage-preserve-fileid
...
fix: write object to the correct urn when moving from another storage to object store
2024-08-26 13:47:27 +07:00
Robin Appelman
Ferdinand Thiessen
Kent Delante
Daniel Calviño Sánchez
Louis
Kate
Anna Larch
Hector Manuel
Benjamin Gaussorgues
Hector Valcarcel
Richard Steinmetz
provokateurin
Stephan Orbaugh
Côme Chilliet
Christoph Fiehe
Julius Härtl