4b08384e95
feat: don't gate perBucket object store configuration behind multibucket
...
a setup can have multiple bucket without having `multibucket` enabled trough things like per-groupfolder buckets
Signed-off-by: Robin Appelman <robin@icewind.nl>
2026-01-05 15:50:10 +07:00
8d5dcbce79
feat(objectstore): add configurable S3 retry attempts
...
Add retriesMaxAttempts parameter to S3 objectstore configuration
to allow customization of AWS SDK retry behavior for handling
unreliable network conditions or proxy issues.
Defaults to 5 retries (AWS SDK default) if not specified.
Signed-off-by: nfebe <fenn25.fn@gmail.com>
Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>
2025-12-03 12:53:39 +07:00
e67c8e739e
feat(ObjectStore): Allow overriding arguments per bucket
...
Signed-off-by: provokateurin <kate@provokateurin.de>
2025-12-01 16:44:41 +07:00
3140daa1aa
feat(CertificateManager): Add option to specify the default certificates bundle path
...
Signed-off-by: provokateurin <kate@provokateurin.de>
2025-12-01 11:57:26 +07:00
8706dadcf0
Merge pull request #56616 from nextcloud/backport/56565/stable31
...
[stable31] feat: emit an event when an S3 bucket is created
2025-11-27 10:26:42 +07:00
bf4c094741
feat: emit an event when an S3 bucket is created
...
Signed-off-by: Kent Delante <kent.delante@proton.me>
2025-11-25 20:16:21 +07:00
d1a760d618
fix: ensure all object store configuration have distict bucket names
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-11-20 15:36:14 +07:00
6d487ac990
fix: make bucket mapper work with new multi-object-store config
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-11-20 15:36:13 +07:00
f21dcac9ef
feat: allow object store configuration aliases for easier migrations
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-11-20 15:36:11 +07:00
832c77ab52
feat: add command to get user objectstore config mappings
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-11-20 15:16:43 +07:00
ff74df3528
feat: allow configuring multiple objectstore configurations
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-11-20 15:16:39 +07:00
3cfc34ecfa
fix(S3): Only append streams if non-seekable
...
Later, when we rewind the stream in `writeMultiPart` during retry, both streams were rewinded, so the resulting stream was bigger than expected.
Inspired by https://github.com/aws/aws-sdk-php/blob/master/src/S3/ObjectUploader.php#L136-L146
Signed-off-by: Louis Chmn <louis@chmn.me>
2025-10-10 12:43:19 +07:00
6842d0746d
Merge pull request #55288 from nextcloud/backport/52816/stable31
...
[stable31] feat: move primary object store configuration to a single place
2025-09-30 10:35:42 +07:00
d615109126
fix: small typo fix
...
Signed-off-by: yemkareems <yemkareems@gmail.com>
2025-09-29 12:24:43 +07:00
7162c3e8ed
feat: move primary object store configuration to a single place
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-09-24 18:39:21 +07:00
05e64418a7
fix: better object store write error propagation
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-08-07 13:48:43 +07:00
d1af0f34a2
fix: always do stream counting for object store upload
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-08-07 13:48:42 +07:00
902cb3dbb9
fix: validate written size for s3 multipart uploads
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-08-07 13:48:42 +07:00
e483387189
feat: more generic way of passing metadata to object storage backends for new objects
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-08-07 13:48:37 +07:00
f40d14abac
fix(s3): retry failed multipart uploads with decreased concurrency
...
Signed-off-by: Kent Delante <kent.delante@proton.me>
2025-06-20 18:24:16 +07:00
019b535868
fix(ObjectStore): Make S3 "connect_timeout" option configurable
...
The hardcoded connection timeout of 5 seconds may not be enough in some
cases, so now it is got from the ObjectStore arguments in Nextcloud
configuration, falling back to 5 if not set.
The connection timeout is set in seconds, but decimal precision can be
used for subsecond accuracy (for example, 4.2 for 4200 milliseconds).
Signed-off-by: Daniel Calviño Sánchez <danxuliu@gmail.com>
2025-06-17 03:30:55 +07:00
edfb7b2393
perf: set the folder size in the same query as we create it
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-06-02 09:56:29 +07:00
9d26571eb2
fix(S3): Use original folder size during copy
...
This prevent having copied folders with a wrongly set size of 0KB.
- Fix https://github.com/nextcloud/server/issues/51916
Signed-off-by: Louis Chemineau <louis@chmn.me>
2025-06-02 09:56:29 +07:00
780766b63c
fix: keep IObjectStoreMetaData in private namespace for backports
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-05-16 10:41:27 +07:00
f0cab800ab
feat: add command to list objects
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-05-16 10:41:27 +07:00
a324dbec2e
feat: add command to get object metadata
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-05-16 10:41:27 +07:00
d393e98c9e
fix(objectstorage): add retry attempts to S3 connection
...
Signed-off-by: Anna Larch <anna@nextcloud.com>
2025-04-23 18:40:23 +07:00
27ed921ad5
fix: Transfer ownership with S3 as primary
...
When using S3 as primary storage, transferring ownership with the `--move` option fail with the following error:
`SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '8-45b963397aa40d4a0063e0d85e4fe7a1' for key 'fs_storage_path_hash'`
The `--move` option moves the entire home folder from one account to another.
The error means that the move failed because the destination folder already exist in `oc_filecache`.
- With S3 as primary storage, folders only exists as entries in `oc_filecache`.
- With S3 as primary storage, `moveFromStorage(...)` only moves the cache entry, as nothing needs to be moved on disk. This cache move does not delete potentially pre-existing destination folder.
- With Local storage, `moveFromStorage(...)` calls `rename(...)` which delete pre-existing folder.
- `transfer(...)`: 687a4d9ac7/apps/files/lib/Service/OwnershipTransferService.php (L112)687a4d9ac7/lib/private/Files/SetupManager.php (L261-L262)687a4d9ac7/lib/private/Files/ObjectStore/ObjectStoreStorage.php (L91-L135)687a4d9ac7/lib/private/Files/ObjectStore/ObjectStoreStorage.php (L635-L636)
2025-04-16 10:19:17 +07:00
d056e4dc74
perf(files): faster query to fetch incomplete directories
...
Signed-off-by: Benjamin Gaussorgues <benjamin.gaussorgues@nextcloud.com>
2025-02-13 14:24:44 +07:00
936ef804cd
fix(s3): treat empty sse_c_key as not set
...
Co-authored-by: Christian Becker <christian@dabecka.de>
Signed-off-by: Richard Steinmetz <richard@steinmetz.cloud>
2025-01-29 17:35:52 +07:00
26c93519f2
fix: Harden files scanner for invalid null access
...
Co-authored-by: Ferdinand Thiessen <opensource@fthiessen.de>
Co-authored-by: Kate <26026535+provokateurin@users.noreply.github.com>
Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
2025-01-28 19:49:03 +07:00
5896f8c37f
chore: cleanup leftover debug statement
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-12-09 15:40:01 +07:00
6cf66f95ce
fix: get object size from stream where possible
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-12-05 15:16:22 +07:00
ca547bb913
fix: don't perform the extra buffering in s3 stream write when the stream size is known
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-12-05 14:07:40 +07:00
7da8951c57
fix: use php://temp instead of php://memory for multi-part upload buffer
...
this should reduce potential memory issues if the limit is set very high
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-12-05 14:07:37 +07:00
df3a9e221d
fix: throw correct exception type when we can't verify if an s3 bucket exists
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-11-26 16:49:55 +07:00
0de4843b73
refactor(Storage): Align all Storage constructors
...
Signed-off-by: provokateurin <kate@provokateurin.de>
2024-10-23 13:24:18 +07:00
f28e74b7a8
refactor(Storage): Make all parameter types strong types
...
Signed-off-by: provokateurin <kate@provokateurin.de>
2024-10-07 15:00:05 +07:00
3e12e1e789
fix: rework move into object store to better preserve fileids
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-27 14:46:08 +07:00
888d06dff9
fix: preserve fileid when moving from objectstore to non-objectstore
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-27 14:45:42 +07:00
da21acfb3f
fix: ensure source folder is removed from cache when moving to objectstore
...
otherwise this causes confusion down the line as it's contents will be moved to the new cache
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-27 14:45:42 +07:00
7cdccd058f
fix(Storage): Fix IStorage return types
...
Signed-off-by: provokateurin <kate@provokateurin.de>
2024-09-26 18:13:03 +07:00
47d281865e
perf: reuse cache info we already have when moving to object store
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-19 19:06:06 +07:00
9836e9b164
chore(deps): Update nextcloud/coding-standard to v1.3.1
...
Signed-off-by: provokateurin <kate@provokateurin.de>
2024-09-19 14:21:20 +07:00
3c0c81bb94
fix: ensure parent folder exists when writing a file to object storage
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-18 14:11:27 +07:00
e0fb25f846
fix: fix object store id for test object store
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-18 13:54:55 +07:00
60f6347c97
fix: cleanup objectstore file_put_content
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-18 13:54:55 +07:00
3e94ee391f
fix: create intermediate directories for objectstore moveFromStorage
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-18 13:54:55 +07:00
879fcc4253
fix: get source file before moving the cache item in objectstore moveFromStorage
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-18 13:54:55 +07:00
f25acfb799
Merge pull request #47986 from cfiehe/fix_move_on_same_bucket
...
perf(ObjectStoreStorage): Improve (slow) move on same object bucket
2024-09-17 16:35:19 +07:00
Robin Appelman
nfebe
provokateurin
Kent Delante
Louis Chmn
yemkareems
Daniel Calviño Sánchez
Anna Larch
Benjamin Gaussorgues
Richard Steinmetz
Ferdinand Thiessen
Stephan Orbaugh