3e2582c4e6
feat(objectstore): add configurable S3 retry attempts
...
Add retriesMaxAttempts parameter to S3 objectstore configuration
to allow customization of AWS SDK retry behavior for handling
unreliable network conditions or proxy issues.
Defaults to 5 retries (AWS SDK default) if not specified.
Signed-off-by: nfebe <fenn25.fn@gmail.com>
Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>
2025-12-03 12:48:52 +07:00
57734fbd01
feat(ObjectStore): Allow overriding arguments per bucket
...
Signed-off-by: provokateurin <kate@provokateurin.de>
2025-12-01 16:44:55 +07:00
8c09cdd07d
Merge pull request #56766 from nextcloud/backport/52749/stable32
...
[stable32] feat(CertificateManager): Add option to specify the default certificates bundle path
2025-12-01 14:54:10 +07:00
6b2c19df98
feat(CertificateManager): Add option to specify the default certificates bundle path
...
Signed-off-by: provokateurin <kate@provokateurin.de>
2025-12-01 11:55:11 +07:00
e8802a57f4
feat: emit an event when an S3 bucket is created
...
Signed-off-by: Kent Delante <kent.delante@proton.me>
2025-11-26 08:18:09 +07:00
91434f1893
fix(s3): expose request_checksum_calculation and response_checksum_validation
...
Fix https://github.com/nextcloud/server/issues/56077
This commit makes the configuration settings 'request_checksum_calculation' and 'response_checksum_validation' of the S3Client from the AWS SDK for PHP configurable.
Signed-off-by: Fiehe Christoph <c.fiehe@eurodata.de>
Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>
2025-11-03 16:07:08 +07:00
2ad19b99f9
fix(S3): Only append streams if non-seekable
...
Later, when we rewind the stream in `writeMultiPart` during retry, both streams were rewinded, so the resulting stream was bigger than expected.
Inspired by https://github.com/aws/aws-sdk-php/blob/master/src/S3/ObjectUploader.php#L136-L146
Signed-off-by: Louis Chmn <louis@chmn.me>
2025-10-10 12:43:32 +07:00
8633dd58b1
fix: small typo fix
...
Signed-off-by: yemkareems <yemkareems@gmail.com>
2025-09-29 12:17:22 +07:00
a8ee28ba62
fix: ensure all object store configuration have distict bucket names
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-09-08 13:23:47 +07:00
02f4a82088
fix: make bucket mapper work with new multi-object-store config
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-08-28 17:51:25 +07:00
b3c53c7436
feat: allow object store configuration aliases for easier migrations
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-08-14 13:06:51 +07:00
2d4bba7b0c
feat: add command to get user objectstore config mappings
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-08-14 13:06:51 +07:00
385dd36ff8
feat: allow configuring multiple objectstore configurations
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-08-14 13:06:51 +07:00
97efc95efc
fix: better object store write error propagation
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-08-05 19:17:58 +07:00
83b8a390cd
fix: always do stream counting for object store upload
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-08-05 19:17:58 +07:00
398b106f0c
fix: validate written size for s3 multipart uploads
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-08-05 19:17:58 +07:00
5981b7eb51
chore: apply new CSFixer rules
...
Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
# Conflicts:
# apps/settings/lib/SetupChecks/PhpOpcacheSetup.php
2025-07-01 16:26:50 +07:00
2e1c316582
fix: remove double cache write in ObjectStoreStorage::touch
...
the file_put_contents already puts all of those values into the cache
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-06-18 22:22:51 +07:00
ea98e4b3f5
Merge pull request #53419 from nextcloud/leftybournes/fix/object_storage_large_uploads
...
fix(s3): retry failed multipart uploads with decreased concurrency
2025-06-18 21:07:06 +07:00
d7ae952dc8
fix(ObjectStore): Make S3 "connect_timeout" option configurable
...
The hardcoded connection timeout of 5 seconds may not be enough in some
cases, so now it is got from the ObjectStore arguments in Nextcloud
configuration, falling back to 5 if not set.
The connection timeout is set in seconds, but decimal precision can be
used for subsecond accuracy (for example, 4.2 for 4200 milliseconds).
Signed-off-by: Daniel Calviño Sánchez <danxuliu@gmail.com>
2025-06-16 20:17:50 +07:00
06b5ce5859
fix(s3): retry failed multipart uploads with decreased concurrency
...
Signed-off-by: Kent Delante <kent.delante@proton.me>
2025-06-10 17:39:59 +07:00
f4cb78b905
Merge pull request #52242 from nextcloud/artonge/fix/copy_subfolders_s3
2025-06-02 11:55:54 +07:00
5b3f4cd8fe
perf: set the folder size in the same query as we create it
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-05-28 15:52:00 +07:00
1f8b823992
fix(S3): Use original folder size during copy
...
This prevent having copied folders with a wrongly set size of 0KB.
- Fix https://github.com/nextcloud/server/issues/51916
Signed-off-by: Louis Chemineau <louis@chmn.me>
2025-05-28 15:52:00 +07:00
7599162c7b
feat: move primary object store configuration to a single place
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-05-28 15:32:31 +07:00
9592390070
Merge pull request #51779 from nextcloud/object-store-filename
2025-05-06 15:21:01 +07:00
49baa914cf
fix(objectstorage): add retry attempts to S3 connection
...
Signed-off-by: Anna Larch <anna@nextcloud.com>
2025-04-23 12:15:06 +07:00
8fdf2a7eae
fix: Transfer ownership with S3 as primary
...
When using S3 as primary storage, transferring ownership with the `--move` option fail with the following error:
`SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '8-45b963397aa40d4a0063e0d85e4fe7a1' for key 'fs_storage_path_hash'`
The `--move` option moves the entire home folder from one account to another.
The error means that the move failed because the destination folder already exist in `oc_filecache`.
- With S3 as primary storage, folders only exists as entries in `oc_filecache`.
- With S3 as primary storage, `moveFromStorage(...)` only moves the cache entry, as nothing needs to be moved on disk. This cache move does not delete potentially pre-existing destination folder.
- With Local storage, `moveFromStorage(...)` calls `rename(...)` which delete pre-existing folder.
- `transfer(...)`: 687a4d9ac7/apps/files/lib/Service/OwnershipTransferService.php (L112)687a4d9ac7/lib/private/Files/SetupManager.php (L261-L262)687a4d9ac7/lib/private/Files/ObjectStore/ObjectStoreStorage.php (L91-L135)687a4d9ac7/lib/private/Files/ObjectStore/ObjectStoreStorage.php (L635-L636)
2025-04-01 13:53:56 +07:00
8aae332622
feat: store original storage id and path in object store metadata
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-03-31 15:10:58 +07:00
94114b99f7
feat: more generic way of passing metadata to object storage backends for new objects
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-03-31 14:55:18 +07:00
fcde776683
feat: add command to list objects
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-03-31 14:30:40 +07:00
c3bc362f48
feat: add command to get object metadata
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-03-31 14:30:40 +07:00
12a92a2e8b
Merge branch 'master' into feat/support-aws-session-token
2025-03-04 18:27:54 +07:00
9bdee76535
perf(files): faster query to fetch incomplete directories
...
Signed-off-by: Benjamin Gaussorgues <benjamin.gaussorgues@nextcloud.com>
2025-02-13 10:11:02 +07:00
6b4c859a41
feat(object_store): Add support for session token in AWS credentials
...
- Pass session token, either null or with value, to the AWS Credentials constructor
Signed-off-by: Hector Valcarcel <hmvalcarcel@gmail.com>
2025-02-06 16:24:19 +07:00
f1c025dfd3
fix(s3): treat empty sse_c_key as not set
...
Co-authored-by: Christian Becker <christian@dabecka.de>
Signed-off-by: Richard Steinmetz <richard@steinmetz.cloud>
2025-01-28 22:59:59 +07:00
b48ee2e924
fix: Harden files scanner for invalid null access
...
Co-authored-by: Ferdinand Thiessen <opensource@fthiessen.de>
Co-authored-by: Kate <26026535+provokateurin@users.noreply.github.com>
Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
2025-01-28 20:08:46 +07:00
5896f8c37f
chore: cleanup leftover debug statement
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-12-09 15:40:01 +07:00
6cf66f95ce
fix: get object size from stream where possible
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-12-05 15:16:22 +07:00
ca547bb913
fix: don't perform the extra buffering in s3 stream write when the stream size is known
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-12-05 14:07:40 +07:00
7da8951c57
fix: use php://temp instead of php://memory for multi-part upload buffer
...
this should reduce potential memory issues if the limit is set very high
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-12-05 14:07:37 +07:00
df3a9e221d
fix: throw correct exception type when we can't verify if an s3 bucket exists
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-11-26 16:49:55 +07:00
0de4843b73
refactor(Storage): Align all Storage constructors
...
Signed-off-by: provokateurin <kate@provokateurin.de>
2024-10-23 13:24:18 +07:00
f28e74b7a8
refactor(Storage): Make all parameter types strong types
...
Signed-off-by: provokateurin <kate@provokateurin.de>
2024-10-07 15:00:05 +07:00
3e12e1e789
fix: rework move into object store to better preserve fileids
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-27 14:46:08 +07:00
888d06dff9
fix: preserve fileid when moving from objectstore to non-objectstore
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-27 14:45:42 +07:00
da21acfb3f
fix: ensure source folder is removed from cache when moving to objectstore
...
otherwise this causes confusion down the line as it's contents will be moved to the new cache
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-27 14:45:42 +07:00
7cdccd058f
fix(Storage): Fix IStorage return types
...
Signed-off-by: provokateurin <kate@provokateurin.de>
2024-09-26 18:13:03 +07:00
47d281865e
perf: reuse cache info we already have when moving to object store
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-19 19:06:06 +07:00
9836e9b164
chore(deps): Update nextcloud/coding-standard to v1.3.1
...
Signed-off-by: provokateurin <kate@provokateurin.de>
2024-09-19 14:21:20 +07:00
nfebe
provokateurin
Andy Scherzinger
Kent Delante
Fiehe Christoph
Louis Chmn
yemkareems
Robin Appelman
Ferdinand Thiessen
Daniel Calviño Sánchez
Kate
Anna Larch
Hector Manuel
Benjamin Gaussorgues
Hector Valcarcel
Richard Steinmetz