ce4d271386
perf(files): faster query to fetch incomplete directories
...
Signed-off-by: Benjamin Gaussorgues <benjamin.gaussorgues@nextcloud.com>
2025-02-14 11:00:07 +07:00
43d4864261
fix(s3): treat empty sse_c_key as not set
...
Co-authored-by: Christian Becker <christian@dabecka.de>
Signed-off-by: Richard Steinmetz <richard@steinmetz.cloud>
2025-01-29 17:33:42 +07:00
d2230f8479
fix: throw correct exception type when we can't verify if an s3 bucket exists
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-11-26 18:45:46 +07:00
8b42f46c3d
Merge pull request #48998 from nextcloud/backport/47986/stable29
...
[stable29] perf(ObjectStoreStorage): Improve (slow) move on same object bucket
2024-10-31 15:55:32 +07:00
68b9972889
perf(ObjectStoreStorage): Improve (slow) move on same object bucket
...
This commit fixes the issue #47856 . When you upload a file into a group folder and when you use a single S3 bucket as primary storage, the final move operation hangs for a long time. In the background, Nextcloud initiates a copy-delete sequence from the bucket into the bucket, with causes a lot unnecessary overhead. Nextcloud thinks that the file must be imported to another storage and does not recognize that everything is done on the same object bucket. In that case, the import step can be completely skipped, which saves time, network bandwidth and reduces the load on the object storage.
The behavior improves a lot with https://github.com/nextcloud/server/pull/46013 . However, there are still some put messages that are being sent to the object storage when you use an object storage as primary storage and upload files into a group folder.
Co-authored-by: Kate <26026535+provokateurin@users.noreply.github.com>
Signed-off-by: Christoph Fiehe <c.fiehe@eurodata.de>
2024-10-30 08:28:52 +07:00
73bff23dd7
fix: rework move into object store to better preserve fileids
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-10-09 14:50:02 +07:00
dd663a91d2
fix: preserve fileid when moving from objectstore to non-objectstore
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-10-09 14:50:02 +07:00
97862e7452
fix: ensure source folder is removed from cache when moving to objectstore
...
otherwise this causes confusion down the line as it's contents will be moved to the new cache
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-10-09 14:50:02 +07:00
1af2df8b4e
fix(s3): suppress php deprecation warning on php8
...
Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>
2024-09-26 12:38:05 +07:00
79e7c48144
fix: ensure parent folder exists when writing a file to object storage
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-19 17:44:53 +07:00
48cd657d26
fix: fix object store id for test object store
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-19 17:44:41 +07:00
6683711516
fix: cleanup objectstore file_put_content
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-19 17:44:34 +07:00
f71dce6c9e
fix: create intermediate directories for objectstore moveFromStorage
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-19 17:44:30 +07:00
68e82ba0e5
fix: get source file before moving the cache item in objectstore moveFromStorage
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-19 17:44:27 +07:00
21a37d79f1
Merge pull request #47873 from nextcloud/backport/47853/stable29
...
[stable29] fix: make swift connect exception message more informative
2024-09-15 22:14:49 +07:00
2cd8582467
fix(S3ConfigTrait): Allow proxy field to take false
...
Signed-off-by: provokateurin <kate@provokateurin.de>
2024-09-15 17:18:15 +07:00
ab283f7b1b
fix: make swift connect exception message more informative
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-10 14:46:21 +07:00
8a539df3a8
fix: write object to the correct urn when moving from another storage to object store
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-08-26 12:24:38 +07:00
aa24a4ad5d
fix(s3): Don't wait indefinitely for S3 to return
...
Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>
fix: lint
Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>
fix: use AwsException
Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>
fix: Throw on connection failure
Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>
fix: Wrap all in try catch block
Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>
fix: use RequestTimeout error message
Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>
log: use OCP Server class
Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>
fix: Handle connect timeout only
Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>
fix: Handle errors more generically
Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>
2024-07-09 07:51:43 +07:00
fcdf453b58
fix(ObjectStore): handle empty S3 hostname
...
Fixes #45637
The support for s3-accelerate added in #44496 introduced a regression in AWS S3 environments when `hostname` is blank (which is a valid configuration w/ AWS since the hostname gets auto-generated).
Signed-off-by: Josh <josh.t.richards@gmail.com>
2024-06-13 09:37:03 +07:00
cec057799d
feat: add option to enable s3 acceleration
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-04-26 13:44:59 +07:00
0b332ceac2
fix: Apply new coding standard to all files
...
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2024-04-04 11:45:22 +07:00
7880086de8
fix(S3): Adjust typing for 32bit compatibility
...
Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
2024-03-26 13:07:53 +07:00
32dee2f84a
feat(ObjectStore): Make S3 MultipartUpload concurrency configurable
...
Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
2024-03-21 16:25:49 +07:00
69c5e8ebde
fix(ObjectStore): Use common S3 trait so config and connection trait members are in sync
...
Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
2024-03-21 16:25:42 +07:00
951c330d39
Revert "don't perform an explicit setup when using the file cache"
2024-03-07 14:38:02 +07:00
b7088e7071
Merge pull request #32163 from nextcloud/file-cache-gc-no-full-setup
...
don't perform an explicit setup when using the file cache
2024-03-07 11:09:19 +07:00
4fba4cd14c
fix: fix user folder init
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-03-06 17:52:24 +07:00
c54f7bd761
fix: disable the internal s3sdk multi part copy logic
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-03-06 15:32:14 +07:00
c0ce272e9c
chore: Migrate away from OC::$server->getLogger
...
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2024-02-13 17:32:30 +07:00
30798eb6c2
refactor(objectstorage): cleanup types
...
Signed-off-by: Thomas Citharel <tcit@tcit.fr>
2024-02-02 14:59:21 +07:00
3be3dbdb3b
refactor(objectstorage): move $user property from ObjectStoreStorage class to HomeObjectStoreStorage
...
Signed-off-by: Thomas Citharel <tcit@tcit.fr>
2024-02-02 14:59:21 +07:00
4c618c4bf1
fix(objectstorage): cleanup HomeObjectStoreStorage
...
Fix invalid signature for getUser() method and change occurences of OC\User\User for OCP\IUser
Signed-off-by: Thomas Citharel <tcit@tcit.fr>
2024-02-02 14:59:20 +07:00
b777304a77
also updated parent etags when a changed etag is detected during scanning
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-01-31 18:06:18 +07:00
e4054370b1
fix(s3): Add config option to disable multipart copy for certain s3 providers
...
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2023-12-28 15:31:38 +07:00
1043c21b35
only do a multipart s3 copy when above the regular copy limit
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-12-28 15:31:37 +07:00
5172baaf8b
fix(ObjectStore): Make copying behavior consistent with local storage
...
Drop file permissions on copy like we do on local storage.
Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
2023-11-17 12:54:37 +07:00
cdf96fab92
fix(s3): support SSE-C headers for the MultipartCopy call
...
Signed-off-by: Tobias Zimmerer <3228193+ir0nhide@users.noreply.github.com>
2023-11-14 21:19:59 +07:00
ea06cf2f39
Convert isset ternary to null coalescing operator
...
Signed-off-by: Hamid Dehnavi <hamid.dev.pro@gmail.com>
2023-09-28 17:44:19 +07:00
dc347b39c1
Merge pull request #40501 from nextcloud/fix/fix-int-size-in-files-scanner
...
Store size in int|float for 32bit support
2023-09-25 16:32:32 +07:00
eab8dd3ca0
use multipart copy for s3
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-09-20 17:28:48 +07:00
5090848b25
Store size in int|float for 32bit support
...
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2023-09-19 13:43:01 +07:00
88a96e4db7
fix: Only store unencrypted_size if path should be encrypted
...
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2023-09-06 07:25:28 +07:00
64f62f7cfd
Merge pull request #39863 from nextcloud/sharing-mask-wrapper
...
move share permission logic to storage wrapper
2023-09-04 14:05:14 +07:00
3f28fc58aa
fix(s3): fix handling verify_bucket_exists parameter
...
If 'verify_bucket_exists' is set to false in the config.php s3 configuration, it's supposed to avoid
verifying that the bucket exists. However empty(falsy) will always return true, so this condition
would not work.
Signed-off-by: Thomas Citharel <tcit@tcit.fr>
2023-08-31 21:36:03 +07:00
a031bc4788
more share permission logic to storage wrapper
...
this way we only have to determine the share permissions once
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-08-14 18:17:27 +07:00
23f85b5ac2
fix(s3): add SSE-C parameters to headObject call
...
Signed-off-by: Tobias Zimmerer <3228193+ir0nhide@users.noreply.github.com>
2023-07-30 22:54:58 +07:00
77f249726d
Merge pull request #38934 from nextcloud/bugfix/noid/ssec-multipart
2023-07-20 11:00:08 +07:00
39a0554c09
Merge pull request #38858 from nextcloud/scanner-performance3
...
file scanner performance improvements
2023-06-26 17:01:18 +07:00
ea575e342f
fix(s3): Pass SSE-C parameters for multipart upload
...
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2023-06-21 18:32:24 +07:00
Benjamin Gaussorgues
Richard Steinmetz
Robin Appelman
Louis
Christoph Fiehe
Daniel Kesselberg
John Molakvoæ
provokateurin
Git'Fellow
Josh
Côme Chilliet
Ferdinand Thiessen
Joas Schilling
Thomas Citharel
Julius Härtl
Tobias Zimmerer
Hamid Dehnavi
Côme Chilliet