b48ee2e924
fix: Harden files scanner for invalid null access
...
Co-authored-by: Ferdinand Thiessen <opensource@fthiessen.de>
Co-authored-by: Kate <26026535+provokateurin@users.noreply.github.com>
Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
2025-01-28 20:08:46 +07:00
5896f8c37f
chore: cleanup leftover debug statement
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-12-09 15:40:01 +07:00
6cf66f95ce
fix: get object size from stream where possible
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-12-05 15:16:22 +07:00
ca547bb913
fix: don't perform the extra buffering in s3 stream write when the stream size is known
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-12-05 14:07:40 +07:00
7da8951c57
fix: use php://temp instead of php://memory for multi-part upload buffer
...
this should reduce potential memory issues if the limit is set very high
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-12-05 14:07:37 +07:00
df3a9e221d
fix: throw correct exception type when we can't verify if an s3 bucket exists
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-11-26 16:49:55 +07:00
0de4843b73
refactor(Storage): Align all Storage constructors
...
Signed-off-by: provokateurin <kate@provokateurin.de>
2024-10-23 13:24:18 +07:00
f28e74b7a8
refactor(Storage): Make all parameter types strong types
...
Signed-off-by: provokateurin <kate@provokateurin.de>
2024-10-07 15:00:05 +07:00
3e12e1e789
fix: rework move into object store to better preserve fileids
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-27 14:46:08 +07:00
888d06dff9
fix: preserve fileid when moving from objectstore to non-objectstore
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-27 14:45:42 +07:00
da21acfb3f
fix: ensure source folder is removed from cache when moving to objectstore
...
otherwise this causes confusion down the line as it's contents will be moved to the new cache
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-27 14:45:42 +07:00
7cdccd058f
fix(Storage): Fix IStorage return types
...
Signed-off-by: provokateurin <kate@provokateurin.de>
2024-09-26 18:13:03 +07:00
47d281865e
perf: reuse cache info we already have when moving to object store
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-19 19:06:06 +07:00
9836e9b164
chore(deps): Update nextcloud/coding-standard to v1.3.1
...
Signed-off-by: provokateurin <kate@provokateurin.de>
2024-09-19 14:21:20 +07:00
3c0c81bb94
fix: ensure parent folder exists when writing a file to object storage
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-18 14:11:27 +07:00
e0fb25f846
fix: fix object store id for test object store
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-18 13:54:55 +07:00
60f6347c97
fix: cleanup objectstore file_put_content
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-18 13:54:55 +07:00
3e94ee391f
fix: create intermediate directories for objectstore moveFromStorage
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-18 13:54:55 +07:00
879fcc4253
fix: get source file before moving the cache item in objectstore moveFromStorage
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-18 13:54:55 +07:00
f25acfb799
Merge pull request #47986 from cfiehe/fix_move_on_same_bucket
...
perf(ObjectStoreStorage): Improve (slow) move on same object bucket
2024-09-17 16:35:19 +07:00
d2d1e3284d
fix(Storage): Fix method docs inheritance
...
Signed-off-by: provokateurin <kate@provokateurin.de>
2024-09-17 10:20:38 +07:00
8ca6fcace7
fix(Storage): Document getOwner() can return false
...
Signed-off-by: provokateurin <kate@provokateurin.de>
2024-09-17 10:10:50 +07:00
492e6997d8
chore: Fix psalm issues, put back private versions of getter in private Storage interface
...
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2024-09-16 18:11:38 +07:00
9597072ada
perf(ObjectStoreStorage): Improve (slow) move on same object bucket
...
This commit fixes the issue #47856 . When you upload a file into a group folder and when you use a single S3 bucket as primary storage, the final move operation hangs for a long time. In the background, Nextcloud initiates a copy-delete sequence from the bucket into the bucket, with causes a lot unnecessary overhead. Nextcloud thinks that the file must be imported to another storage and does not recognize that everything is done on the same object bucket. In that case, the import step can be completely skipped, which saves time, network bandwidth and reduces the load on the object storage.
The behavior improves a lot with https://github.com/nextcloud/server/pull/46013 . However, there are still some put messages that are being sent to the object storage when you use an object storage as primary storage and upload files into a group folder.
Co-authored-by: Kate <26026535+provokateurin@users.noreply.github.com>
Signed-off-by: Christoph Fiehe <c.fiehe@eurodata.de>
2024-09-16 15:11:09 +07:00
7c1c02d6a5
fix(S3ConfigTrait): Allow proxy field to take false
...
Signed-off-by: provokateurin <kate@provokateurin.de>
2024-09-15 15:41:58 +07:00
508d8e7c01
fix: make swift connect exception message more informative
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-09 17:15:01 +07:00
6d2253e692
Merge pull request #46013 from nextcloud/obj-store-move-from-storage-preserve-fileid
...
fix: write object to the correct urn when moving from another storage to object store
2024-08-26 13:47:27 +07:00
af6de04e9e
style: update codestyle for coding-standard 1.2.3
...
Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>
2024-08-25 19:34:58 +07:00
bd740ac0b0
fix: write object to the correct urn when moving from another storage to object store
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-08-23 15:22:16 +07:00
c84e76a749
fix(s3): Don't wait indefinitely for S3 to return
...
Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>
fix: lint
Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>
fix: use AwsException
Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>
fix: Throw on connection failure
Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>
fix: Wrap all in try catch block
Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>
fix: use RequestTimeout error message
Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>
log: use OCP Server class
Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>
fix: Handle connect timeout only
Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>
fix: Handle errors more generically
Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>
2024-07-05 16:24:32 +07:00
ae1f5cc56d
fix(S3): Adjust usage of guzzle promise
...
`Promise\promise_for` was deprecated and is now removed and replaced with the static API (`Create::promiseFor`).
Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
2024-06-16 15:54:44 +07:00
63886147ba
fix(ObjectStore): handle empty S3 hostname
...
Fixes #45637
The support for s3-accelerate added in #44496 introduced a regression in AWS S3 environments when `hostname` is blank (which is a valid configuration w/ AWS since the hostname gets auto-generated).
Signed-off-by: Josh <josh.t.richards@gmail.com>
2024-06-12 15:43:20 +07:00
dae7c159f7
chore: Add SPDX header
...
Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
2024-05-24 13:11:22 +07:00
40d9a3fcab
feat: add option to enable s3 acceleration
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-04-26 14:40:24 +07:00
cdb40a3110
fix(openstack): Fix psalm issue with updated openstack version
...
Signed-off-by: Joas Schilling <coding@schilljs.com>
2024-04-24 07:01:57 +07:00
ec5133b739
fix: Apply new coding standard to all files
...
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2024-04-02 14:16:21 +07:00
7880086de8
fix(S3): Adjust typing for 32bit compatibility
...
Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
2024-03-26 13:07:53 +07:00
32dee2f84a
feat(ObjectStore): Make S3 MultipartUpload concurrency configurable
...
Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
2024-03-21 16:25:49 +07:00
69c5e8ebde
fix(ObjectStore): Use common S3 trait so config and connection trait members are in sync
...
Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
2024-03-21 16:25:42 +07:00
951c330d39
Revert "don't perform an explicit setup when using the file cache"
2024-03-07 14:38:02 +07:00
b7088e7071
Merge pull request #32163 from nextcloud/file-cache-gc-no-full-setup
...
don't perform an explicit setup when using the file cache
2024-03-07 11:09:19 +07:00
4fba4cd14c
fix: fix user folder init
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-03-06 17:52:24 +07:00
c54f7bd761
fix: disable the internal s3sdk multi part copy logic
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-03-06 15:32:14 +07:00
c0ce272e9c
chore: Migrate away from OC::$server->getLogger
...
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2024-02-13 17:32:30 +07:00
30798eb6c2
refactor(objectstorage): cleanup types
...
Signed-off-by: Thomas Citharel <tcit@tcit.fr>
2024-02-02 14:59:21 +07:00
3be3dbdb3b
refactor(objectstorage): move $user property from ObjectStoreStorage class to HomeObjectStoreStorage
...
Signed-off-by: Thomas Citharel <tcit@tcit.fr>
2024-02-02 14:59:21 +07:00
4c618c4bf1
fix(objectstorage): cleanup HomeObjectStoreStorage
...
Fix invalid signature for getUser() method and change occurences of OC\User\User for OCP\IUser
Signed-off-by: Thomas Citharel <tcit@tcit.fr>
2024-02-02 14:59:20 +07:00
b777304a77
also updated parent etags when a changed etag is detected during scanning
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-01-31 18:06:18 +07:00
e4054370b1
fix(s3): Add config option to disable multipart copy for certain s3 providers
...
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2023-12-28 15:31:38 +07:00
1043c21b35
only do a multipart s3 copy when above the regular copy limit
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-12-28 15:31:37 +07:00
5172baaf8b
fix(ObjectStore): Make copying behavior consistent with local storage
...
Drop file permissions on copy like we do on local storage.
Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
2023-11-17 12:54:37 +07:00
cdf96fab92
fix(s3): support SSE-C headers for the MultipartCopy call
...
Signed-off-by: Tobias Zimmerer <3228193+ir0nhide@users.noreply.github.com>
2023-11-14 21:19:59 +07:00
ea06cf2f39
Convert isset ternary to null coalescing operator
...
Signed-off-by: Hamid Dehnavi <hamid.dev.pro@gmail.com>
2023-09-28 17:44:19 +07:00
dc347b39c1
Merge pull request #40501 from nextcloud/fix/fix-int-size-in-files-scanner
...
Store size in int|float for 32bit support
2023-09-25 16:32:32 +07:00
eab8dd3ca0
use multipart copy for s3
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-09-20 17:28:48 +07:00
5090848b25
Store size in int|float for 32bit support
...
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2023-09-19 13:43:01 +07:00
88a96e4db7
fix: Only store unencrypted_size if path should be encrypted
...
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2023-09-06 07:25:28 +07:00
64f62f7cfd
Merge pull request #39863 from nextcloud/sharing-mask-wrapper
...
move share permission logic to storage wrapper
2023-09-04 14:05:14 +07:00
3f28fc58aa
fix(s3): fix handling verify_bucket_exists parameter
...
If 'verify_bucket_exists' is set to false in the config.php s3 configuration, it's supposed to avoid
verifying that the bucket exists. However empty(falsy) will always return true, so this condition
would not work.
Signed-off-by: Thomas Citharel <tcit@tcit.fr>
2023-08-31 21:36:03 +07:00
a031bc4788
more share permission logic to storage wrapper
...
this way we only have to determine the share permissions once
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-08-14 18:17:27 +07:00
23f85b5ac2
fix(s3): add SSE-C parameters to headObject call
...
Signed-off-by: Tobias Zimmerer <3228193+ir0nhide@users.noreply.github.com>
2023-07-30 22:54:58 +07:00
77f249726d
Merge pull request #38934 from nextcloud/bugfix/noid/ssec-multipart
2023-07-20 11:00:08 +07:00
39a0554c09
Merge pull request #38858 from nextcloud/scanner-performance3
...
file scanner performance improvements
2023-06-26 17:01:18 +07:00
ea575e342f
fix(s3): Pass SSE-C parameters for multipart upload
...
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2023-06-21 18:32:24 +07:00
de85610385
use source cache when listing folder during recursive copy
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-06-19 16:40:15 +07:00
561536fc58
file scanner performance improvements
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-06-16 16:31:48 +07:00
ea88ec1350
add utility command for object store objects
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-06-09 18:08:18 +07:00
9f1d497a0b
Merge pull request #38261 from fsamapoor/replace_strpos_calls_in_lib_private
...
Refactors "strpos" calls in lib/private to improve code readability.
2023-06-01 23:10:00 +07:00
cc0ad25c98
don't always check if we need to setup the object store root
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-05-23 17:44:40 +07:00
e7cc7653b8
Refactors "strpos" calls in lib/private to improve code readability.
...
Signed-off-by: Faraz Samapoor <fsamapoor@gmail.com>
2023-05-15 15:17:19 +07:00
2ea41dab93
repair -1 folder sizes for object store background scan
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-05-10 19:33:26 +07:00
45a0fe490c
Merge pull request #37534 from Rsplwe/master
...
Adjust the value of the "max-parts" parameter of the object storage 'ListPart' interface to 1000
2023-04-18 00:16:38 +07:00
85c9e75f33
Merge pull request #32314 from nextcloud/object-mkdir-logging
...
additional logging when mkdir fails for object storage
2023-04-17 23:42:08 +07:00
c6be96d04f
additional logging when mkdir fails for object storage
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-04-05 14:52:19 +07:00
d63fa2d991
improve objectstore rmdir handling
...
remove folder entries as they are cleared instead of in one go afterwards
otherwise they stick around if some of the child entries can't be removed
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-04-03 18:28:29 +07:00
5c6f35832c
Update lib/private/Files/ObjectStore/S3.php
...
Co-authored-by: Côme Chilliet <91878298+come-nc@users.noreply.github.com>
Signed-off-by: Rsplwe <i@rsplwe.com>
2023-04-03 21:59:59 +07:00
d85e623916
s3: get remaining truncation part
...
Signed-off-by: Rsplwe <i@rsplwe.com>
2023-04-01 22:56:15 +07:00
ccac296821
Adjust the value of 'maxparts' to 1000
...
Signed-off-by: Rsplwe <i@rsplwe.com>
2023-04-01 21:20:52 +07:00
e23aa8883e
feat(s3): Use multipart upload for chunked uploading
...
This allows to stream file chunks directly to S3 during upload.
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2023-03-08 14:00:04 +07:00
159a0c8411
feat(s3): Add option to specify an SSE-C customer provided key
...
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2023-01-24 10:36:03 +07:00
f5c361cf44
composer run cs:fix
...
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2023-01-20 11:45:08 +07:00
cf9bdc0274
Manage s3 storage class in objectstore
...
Signed-off-by: François Ménabé <francois.menabe@gmail.com>
2023-01-12 14:58:27 +07:00
cc639a8877
compare cached filesize on download
...
Signed-off-by: Maxence Lange <maxence@artificial-owl.com>
2022-12-01 15:42:15 +07:00
a5036a973c
allow disabling object store write check
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2022-10-26 18:45:59 +07:00
a5ea677370
Rename file1 and file2 to source and target in Storage abstraction
...
Signed-off-by: Carl Schwan <carl@carlschwan.eu>
2022-10-18 17:21:47 +07:00
214a50d071
Avoid allocating too much memory for the buffer
...
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2022-10-11 15:34:27 +07:00
3950deb42f
Merge pull request #32740 from jasperweyne/patch-2
...
Fix open_basedir issues by AWS SDK
2022-09-19 10:19:15 +07:00
457822c144
don't update the folder size if we know it hasn't changed
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2022-08-16 13:56:51 +07:00
44f6c931e7
Merge branch 'master' into patch-2
2022-08-11 08:54:08 +07:00
61fdd91dcf
don't set `null` as a bundle path
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2022-07-27 12:58:07 +07:00
f1486890d7
only use nextcloud bundle when explicitly enabled
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2022-07-05 16:06:55 +07:00
bffa67c48b
also use nextcloud certificate bundle when downloading from s3
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2022-07-05 15:57:54 +07:00
0633a1d9f5
Disable ~/.aws/config access by S3ConnectionTrait
...
This commit sets the 'use_aws_shared_config_files' option to false, in order to disable configuration loading from ~/.aws/config by the AWS SDK, specifically the S3Client. It is a continuation of #27040 , as that PR only changed the behaviour of the CredentialsProvider; this change affects the ConfigurationProvider as well.
Signed-off-by: Jasper Weyne <jasperweyne@gmail.com>
2022-06-07 13:39:14 +07:00
0f0e74a78c
Cleanup temporary files after finishing the write to object storage
...
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2022-06-02 12:57:18 +07:00
069477e608
Migrate more classes of lib/private to LoggerInterface
...
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2022-04-26 16:52:52 +07:00
7817845538
Add a metadata service to store file metadata
...
Signed-off-by: Carl Schwan <carl@carlschwan.eu>
2022-04-13 14:06:29 +07:00
ea23523c70
Adapt more code to migration to LoggerInterface
...
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2022-03-24 16:21:25 +07:00
6be7aa112f
Migrate from ILogger to LoggerInterface in lib/private
...
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2022-03-24 16:21:25 +07:00
1156214a26
don't try to get custom certs for s3 primary storage
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2022-03-15 15:50:08 +07:00
194a21f374
use the nextcloud certificate bundle for s3
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2022-03-14 18:09:48 +07:00
Ferdinand Thiessen
Robin Appelman
provokateurin
Stephan Orbaugh
Côme Chilliet
Christoph Fiehe
Julius Härtl
Daniel Kesselberg
Git'Fellow
Josh
Andy Scherzinger
Joas Schilling
Joas Schilling
John Molakvoæ
Thomas Citharel
Tobias Zimmerer
Hamid Dehnavi
Côme Chilliet
Faraz Samapoor
Simon L
Rsplwe
François Ménabé
Maxence Lange
Carl Schwan
Jasper Weyne