5172baaf8b
fix(ObjectStore): Make copying behavior consistent with local storage
...
Drop file permissions on copy like we do on local storage.
Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
2023-11-17 12:54:37 +07:00
cdf96fab92
fix(s3): support SSE-C headers for the MultipartCopy call
...
Signed-off-by: Tobias Zimmerer <3228193+ir0nhide@users.noreply.github.com>
2023-11-14 21:19:59 +07:00
ea06cf2f39
Convert isset ternary to null coalescing operator
...
Signed-off-by: Hamid Dehnavi <hamid.dev.pro@gmail.com>
2023-09-28 17:44:19 +07:00
dc347b39c1
Merge pull request #40501 from nextcloud/fix/fix-int-size-in-files-scanner
...
Store size in int|float for 32bit support
2023-09-25 16:32:32 +07:00
eab8dd3ca0
use multipart copy for s3
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-09-20 17:28:48 +07:00
5090848b25
Store size in int|float for 32bit support
...
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2023-09-19 13:43:01 +07:00
88a96e4db7
fix: Only store unencrypted_size if path should be encrypted
...
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2023-09-06 07:25:28 +07:00
64f62f7cfd
Merge pull request #39863 from nextcloud/sharing-mask-wrapper
...
move share permission logic to storage wrapper
2023-09-04 14:05:14 +07:00
3f28fc58aa
fix(s3): fix handling verify_bucket_exists parameter
...
If 'verify_bucket_exists' is set to false in the config.php s3 configuration, it's supposed to avoid
verifying that the bucket exists. However empty(falsy) will always return true, so this condition
would not work.
Signed-off-by: Thomas Citharel <tcit@tcit.fr>
2023-08-31 21:36:03 +07:00
a031bc4788
more share permission logic to storage wrapper
...
this way we only have to determine the share permissions once
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-08-14 18:17:27 +07:00
23f85b5ac2
fix(s3): add SSE-C parameters to headObject call
...
Signed-off-by: Tobias Zimmerer <3228193+ir0nhide@users.noreply.github.com>
2023-07-30 22:54:58 +07:00
77f249726d
Merge pull request #38934 from nextcloud/bugfix/noid/ssec-multipart
2023-07-20 11:00:08 +07:00
39a0554c09
Merge pull request #38858 from nextcloud/scanner-performance3
...
file scanner performance improvements
2023-06-26 17:01:18 +07:00
ea575e342f
fix(s3): Pass SSE-C parameters for multipart upload
...
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2023-06-21 18:32:24 +07:00
de85610385
use source cache when listing folder during recursive copy
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-06-19 16:40:15 +07:00
561536fc58
file scanner performance improvements
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-06-16 16:31:48 +07:00
ea88ec1350
add utility command for object store objects
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-06-09 18:08:18 +07:00
9f1d497a0b
Merge pull request #38261 from fsamapoor/replace_strpos_calls_in_lib_private
...
Refactors "strpos" calls in lib/private to improve code readability.
2023-06-01 23:10:00 +07:00
cc0ad25c98
don't always check if we need to setup the object store root
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-05-23 17:44:40 +07:00
e7cc7653b8
Refactors "strpos" calls in lib/private to improve code readability.
...
Signed-off-by: Faraz Samapoor <fsamapoor@gmail.com>
2023-05-15 15:17:19 +07:00
2ea41dab93
repair -1 folder sizes for object store background scan
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-05-10 19:33:26 +07:00
45a0fe490c
Merge pull request #37534 from Rsplwe/master
...
Adjust the value of the "max-parts" parameter of the object storage 'ListPart' interface to 1000
2023-04-18 00:16:38 +07:00
85c9e75f33
Merge pull request #32314 from nextcloud/object-mkdir-logging
...
additional logging when mkdir fails for object storage
2023-04-17 23:42:08 +07:00
c6be96d04f
additional logging when mkdir fails for object storage
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-04-05 14:52:19 +07:00
d63fa2d991
improve objectstore rmdir handling
...
remove folder entries as they are cleared instead of in one go afterwards
otherwise they stick around if some of the child entries can't be removed
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-04-03 18:28:29 +07:00
5c6f35832c
Update lib/private/Files/ObjectStore/S3.php
...
Co-authored-by: Côme Chilliet <91878298+come-nc@users.noreply.github.com>
Signed-off-by: Rsplwe <i@rsplwe.com>
2023-04-03 21:59:59 +07:00
d85e623916
s3: get remaining truncation part
...
Signed-off-by: Rsplwe <i@rsplwe.com>
2023-04-01 22:56:15 +07:00
ccac296821
Adjust the value of 'maxparts' to 1000
...
Signed-off-by: Rsplwe <i@rsplwe.com>
2023-04-01 21:20:52 +07:00
e23aa8883e
feat(s3): Use multipart upload for chunked uploading
...
This allows to stream file chunks directly to S3 during upload.
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2023-03-08 14:00:04 +07:00
159a0c8411
feat(s3): Add option to specify an SSE-C customer provided key
...
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2023-01-24 10:36:03 +07:00
f5c361cf44
composer run cs:fix
...
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2023-01-20 11:45:08 +07:00
cf9bdc0274
Manage s3 storage class in objectstore
...
Signed-off-by: François Ménabé <francois.menabe@gmail.com>
2023-01-12 14:58:27 +07:00
cc639a8877
compare cached filesize on download
...
Signed-off-by: Maxence Lange <maxence@artificial-owl.com>
2022-12-01 15:42:15 +07:00
a5036a973c
allow disabling object store write check
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2022-10-26 18:45:59 +07:00
a5ea677370
Rename file1 and file2 to source and target in Storage abstraction
...
Signed-off-by: Carl Schwan <carl@carlschwan.eu>
2022-10-18 17:21:47 +07:00
214a50d071
Avoid allocating too much memory for the buffer
...
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2022-10-11 15:34:27 +07:00
3950deb42f
Merge pull request #32740 from jasperweyne/patch-2
...
Fix open_basedir issues by AWS SDK
2022-09-19 10:19:15 +07:00
457822c144
don't update the folder size if we know it hasn't changed
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2022-08-16 13:56:51 +07:00
44f6c931e7
Merge branch 'master' into patch-2
2022-08-11 08:54:08 +07:00
61fdd91dcf
don't set `null` as a bundle path
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2022-07-27 12:58:07 +07:00
f1486890d7
only use nextcloud bundle when explicitly enabled
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2022-07-05 16:06:55 +07:00
bffa67c48b
also use nextcloud certificate bundle when downloading from s3
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2022-07-05 15:57:54 +07:00
0633a1d9f5
Disable ~/.aws/config access by S3ConnectionTrait
...
This commit sets the 'use_aws_shared_config_files' option to false, in order to disable configuration loading from ~/.aws/config by the AWS SDK, specifically the S3Client. It is a continuation of #27040 , as that PR only changed the behaviour of the CredentialsProvider; this change affects the ConfigurationProvider as well.
Signed-off-by: Jasper Weyne <jasperweyne@gmail.com>
2022-06-07 13:39:14 +07:00
0f0e74a78c
Cleanup temporary files after finishing the write to object storage
...
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2022-06-02 12:57:18 +07:00
069477e608
Migrate more classes of lib/private to LoggerInterface
...
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2022-04-26 16:52:52 +07:00
7817845538
Add a metadata service to store file metadata
...
Signed-off-by: Carl Schwan <carl@carlschwan.eu>
2022-04-13 14:06:29 +07:00
ea23523c70
Adapt more code to migration to LoggerInterface
...
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2022-03-24 16:21:25 +07:00
6be7aa112f
Migrate from ILogger to LoggerInterface in lib/private
...
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2022-03-24 16:21:25 +07:00
1156214a26
don't try to get custom certs for s3 primary storage
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2022-03-15 15:50:08 +07:00
194a21f374
use the nextcloud certificate bundle for s3
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2022-03-14 18:09:48 +07:00
bc2c02f451
Wrap S3 multipart upload exception
...
Signed-off-by: Louis Chemineau <louis@chmn.me>
2022-02-16 14:56:14 +07:00
0b768c17d2
Wrap S3 multipart upload exception
...
Signed-off-by: Louis Chemineau <louis@chmn.me>
2022-02-15 17:35:12 +07:00
2aac757805
Make max size for single put uploads configurable
...
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2021-12-29 13:30:22 +07:00
e475dfe16c
Fix php-cs for S3ConnectionTrait.php
...
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2021-12-21 21:35:46 +07:00
1acfbd0b06
Merge pull request #27493 from cuppett/cuppett/simplify-aws-credential-provider
2021-12-21 11:58:52 +07:00
66bdda568a
Fix Guzzle function calls
...
Since 1.7.0 and 2.0 these functions are turned into static methods.
See https://github.com/guzzle/psr7#upgrading-from-function-api
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2021-11-23 09:28:57 +07:00
4a6a2c2b48
Simplify S3ConnectionTrait to defaultProvider plus option
...
When we initially added the EC2 and ECS IAM role support in #24700 ,
we had to use a workaround by explicitly ordering the various providers
due to an inconsistency in the AWS SDK for PHP. We submitted a PR there
to get that squared away. Now, we've consumed that version upstream
for the SDK and can update our code here to be the most concise version
as well as position ourselves to pick up new methods as those become
available and prevalent in AWS (for acquiring credentials).
See also: https://github.com/nextcloud/server/pull/24700#issuecomment-747650892
See also: https://github.com/aws/aws-sdk-php/pull/2172
Signed-off-by: Stephen Cuppett <steve@cuppett.com>
2021-11-22 07:04:48 +07:00
6ab2feaa54
Implement multibucket shift for ObjectStore
...
Signed-off-by: John Molakvoæ <skjnldsv@protonmail.com>
2021-11-11 15:09:28 +07:00
df4e6bab69
Merge pull request #29115 from nextcloud/work/carl/correct-permissions-when-copying
...
Fix permissions when copying from ObjectStorage
2021-10-28 14:38:20 +07:00
bfa60aaf27
Fix permissions when copying from ObjectStorage
...
Make sure that when a user copy a file from a directory they don't have
all permissions to a directory where they have more permissions, the
permissions are correctly set to the one from the parent taget folder.
This was caused by the ObjectStoreStorage::copyFromStorage using
the jailed storage and cache entry instead of the unjailed one like other
storages (the local one).
Steps to reproduce
+ Use object storage
+ Create a groupfolder with one group having full permission and another one
who can just read files.
+ With an user who is in the second group, copy a file from the groupfolder to
the home folder of this user.
+ The file in the home folder of the user will be read only and can't be deleted
even though it is in their home folder and they are the owner. In oc_filecache,
the permissions stored for this file are 1 (READ)
Signed-off-by: Carl Schwan <carl@carlschwan.eu>
2021-10-28 13:29:50 +07:00
09ffac5e6d
s3 external storage listing rework
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2021-10-15 15:35:02 +07:00
3866f388b1
Refactor writeObject to only use MultipartUpload when required
...
Signed-off-by: Bernd Rederlechner <Bernd.Rederlechner@t-systems.com>
Co-authored-by: Julius Härtl <jus@bitgrid.net>
2021-08-20 17:02:25 +07:00
3e67637a4a
Merge pull request #26463 from Worteks/fix-s3proxy
2021-07-14 23:32:05 +07:00
62675eb5c2
Merge pull request #27586 from nextcloud/bugfi/noid/objectstore-checksum
...
Reset checksum when writing files to object store
2021-07-01 17:30:54 +07:00
388a458cd5
fix(proxy): use accessor
...
Signed-off-by: Maxime Besson <maxime.besson@worteks.com>
2021-06-30 09:25:04 +07:00
6d5cfe0c66
Move DateTime::RFC2822 to DateTimeInterface::2822
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2021-06-23 15:30:43 +07:00
21db463bc9
Reset checksum when writing files to object store
...
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2021-06-21 12:04:49 +07:00
215aef3cbd
Update php licenses
...
Signed-off-by: John Molakvoæ (skjnldsv) <skjnldsv@protonmail.com>
2021-06-04 22:02:41 +07:00
333665b43d
Merge pull request #27040 from nextcloud/FlorentCoppint-master
...
Avoid reading ~/.aws/config when using S3 provider
2021-05-25 10:19:22 +07:00
d4444f2472
Avoid reading ~/.aws/config when using S3 provider
...
When using S3 storage, we sometimes have open_basedir errors due to aws-sdk-php trying to read ~/.aws/config which is out of open_basedir restrictions.
Christoph Wurst already added csm=false config in #21406 but it wasn't enough, we also need to set use_arn_region=false, added in this commit.
Signed-off-by: Florent <florent@coppint.com>
2021-05-20 12:21:15 +07:00
ef6f2e68f0
explicitly close source stream on object store upload even if count wrapper isn't needed
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2021-04-29 17:01:19 +07:00
effb7dc8ba
set mimetype for objects uploaded to object storages
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2021-04-21 15:25:58 +07:00
547438527d
fix(proxy): reaching s3 storage behind some http proxy
...
Signed-off-by: Maxime Besson <maxime.besson@worteks.com>
2021-04-09 10:15:57 +07:00
03fe74b95e
fix(proxy): reaching s3 storage behind some http proxy
...
Signed-off-by: Maxime Besson <maxime.besson@worteks.com>
2021-04-09 10:15:57 +07:00
ad636ae12d
use returned file id for object store copy
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2021-03-12 17:22:40 +07:00
32551b9ff7
Merge pull request #25722 from nextcloud/objectstore-copy-cross
...
apply object store copy optimization when 'cross storage' copy is wit…
2021-03-12 16:10:18 +07:00
3eb764e65d
remove explicit fclose from S3->writeStream
...
streams get closed automatically when dropped, and in some cases the stream seems to be already closed by the S3 library, in which case trying to close it again will raise an error
Signed-off-by: Robin Appelman <robin@icewind.nl>
2021-03-11 18:05:04 +07:00
c87b1a50d9
apply object store copy optimization when 'cross storage' copy is within the same object store
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2021-03-09 20:31:27 +07:00
eb502c02ff
Bump nextcloud/coding-standard from 0.3.0 to 0.5.0
...
Bumps [nextcloud/coding-standard](https://github.com/nextcloud/coding-standard ) from 0.3.0 to 0.5.0.
- [Release notes](https://github.com/nextcloud/coding-standard/releases )
- [Changelog](https://github.com/nextcloud/coding-standard/blob/master/CHANGELOG.md )
- [Commits](https://github.com/nextcloud/coding-standard/compare/v0.3.0...v0.5.0 )
Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2021-02-18 13:31:24 +07:00
137636b651
Merge pull request #24700 from Imajie/s3-creds
...
Resolves #24699 , Support ES2 and ECS instance providers for S3 buckets
2021-02-02 21:48:22 +07:00
9ce3ea3368
Update license headers
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-12-30 14:07:05 +07:00
73c7d0dc81
Bump icewind/streams from 0.7.1 to 0.7.2
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-12-29 09:07:36 +07:00
5094e29ebd
Merge pull request #24594 from kofemann/dcache
2020-12-22 09:26:07 +07:00
4f2dc18f58
storage: update IStorage#file_put_contents docs to match usage
...
The current phpdoc of IStorage#file_put_contents doesnt corresponds to
it's actual usage in code, e.g.
Signed-off-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
2020-12-21 19:04:18 +07:00
d89a75be0b
Update all license headers for Nextcloud 21
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-12-16 18:48:22 +07:00
45a02ee30c
Fix failing tests
...
Signed-off-by: James Letendre <james.letendre@gmail.com>
2020-12-15 19:06:06 +07:00
ad95e51cbe
Add web identity provider to S3 connection chain
...
Signed-off-by: James Letendre <james.letendre@gmail.com>
2020-12-15 16:37:46 +07:00
5b756a9fb0
Replace defaultProvider with explicit calls to exclude user home directory lookup
...
Signed-off-by: James Letendre <james.letendre@gmail.com>
2020-12-14 16:22:41 +07:00
85aa77539b
Resolves #24699 , Support ES2 and ECS instance providers for S3 buckets
...
Signed-off-by: James Letendre <james.letendre@gmail.com>
2020-12-14 12:55:46 +07:00
9d4848e863
use in objectstore copy
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-11-24 15:16:58 +07:00
65141d4864
Allow config to specify the bucket exists
...
In the 99% case the bucket is just always there. And if it is not the
read/write will fail hard anyways. Esp on big instances the Objectstore
is not always fast and this can save a few hundered ms of each request
that acess the objectstore.
In short it is adding
'verify_bucket_exists' => false
To the S3 config part
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-11-06 11:08:38 +07:00
d9015a8c94
Format code to a single space around binary operators
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-10-05 20:25:24 +07:00
bb06b6cce4
Fix reading empty files from objectstorage
...
Since we try to do range requests this will fail hard.
However since empty files are not that interesting to read anyways we
just read from an emptry memory stream.
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-09-09 20:45:13 +07:00
9fdeed8cc0
Run stat less often for objectstorages
...
When we want to get the permissions we now do stat at least 5 times for
each entry. Which is a bit much. Especially since the permssions are all
just in the database already.
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-09-08 14:38:36 +07:00
789b33aba4
Only update the filecache entry once the file has been written to S3
...
If we already update before we have no way to revert if the upload
fails.
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-08-31 12:28:04 +07:00
6ffd7173f9
Don't lose filecache entry on s3 overwrite error
...
If the object store errors we should not always delete the filecache
entry. As this might lead to people losing access to their files.
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-08-31 12:25:20 +07:00
2a054e6c04
Update the license headers for Nextcloud 20
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-08-24 14:54:25 +07:00
5ef0f86ce7
Resolves #19790 , Provides Support for IAM Credentials
...
Includes support for either leveraging environment variables
passed to the PHP runtime or IAM instance profile present
on the host being used. The default and first choice is
still the parameter file as documented.
See also: https://docs.aws.amazon.com/sdk-for-php/v3/developer-guide/guide_credentials_provider.html#chaining-providers
Signed-off-by: Stephen Cuppett <steve@cuppett.com>
2020-08-20 15:54:33 +07:00
fedf9c69d9
Use matching parameter names form interfaces and implementations
...
Found by Psalm 3.14.1
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
2020-08-19 18:16:35 +07:00
438ac23e2a
Distribute preview folders in appdata in multibucket setup to multiple buckets
...
* introduces a new IRootMountProvider to register mount points inside the root storage
* adds a AppdataPreviewObjectStoreStorage to handle the split between preview folders and bucket number
Ref #22033
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
2020-08-06 22:19:20 +07:00
d3a4bc91f7
Fix static method call for s3 bucket compat check
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-07-03 09:02:58 +07:00
8aa99aa8ff
Update presign method to match with interface again.
...
Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>
2020-06-25 22:56:28 +07:00
9d392891be
Disable Client-Side Monitoring on AWS storage
...
The S3 client enables this by default and then tries to read
`.aws/config`. This causes `open_basedir` restriction related error for
some setups. So this patch disables the CSM because it's most likely
unused anyway.
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-06-15 09:13:08 +07:00
3594ba6971
Upload part size as S3 parameter instead of constant value
...
Some S3 providers need a custom upload part size (500 MB static value in Nextcloud).
Here is a commit to change this value via S3 configuration, instead of using S3_UPLOAD_PART_SIZE constant.
A new parameter is added for an S3 connection : uploadPartSize
Signed-off-by: Florent <florent@coppint.com>
2020-06-09 09:18:42 +07:00
d5850eb28f
Merge pull request #18955 from adrb/swift_upload_large_objects
...
Large Object support for OpenStack Swift
2020-05-06 13:03:24 +07:00
4e6d3d1880
Fix getDirectoryContent implementation for Jail wrapper
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-30 15:27:09 +07:00
f6b435d0d9
Run the cs fixer for green ci
...
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-04-30 10:28:44 +07:00
a1c1b354fc
Merge pull request #18883 from adrb/optimize_swift_14116
...
Optimize Openstack Swift files download
2020-04-30 09:06:19 +07:00
cb057829f7
Update license headers for 19
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-29 11:57:22 +07:00
e107519295
Large Object support for OpenStack Swift.
...
Until now, you wouldn't be able to create
objects larger that 5GB.
It's somewhat related with pull #18883
Signed-off-by: Adrian Brzezinski <adrian.brzezinski@eo.pl>
2020-04-27 14:23:59 +07:00
354c0996d8
It's a fix for issue #14116 ,
...
Improves efficiency when downloading files from Swift storage.
Before, files were downloaded and then pushed back to user.
That behaevior causes all kinds of performance problems.
Now, files are streamed directly to user.
Signed-off-by: Adrian Brzezinski <adrian.brzezinski@eo.pl>
2020-04-27 14:16:50 +07:00
9735b5d61f
Add method to storage backends to get directory content with metadata
...
Currently you need to use `opendir` and then call `getMetadata` for
every file, which adds overhead because most storage backends already
get the metadata when doing the `opendir`.
While storagebackends can (and do) use caching to relief this problem,
this adds cache invalidation dificulties and only a limited number of
items are generally cached (to prevent memory usage exploding when
scanning large storages)
With this new methods storage backends can use the child metadata they
got from listing the folder to return metadata without having to keep
seperate caches.
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-20 15:45:04 +07:00
3e849daf11
Fix issues with Keystone auth v3 in files_external app
...
Signed-off-by: Julien Lutran <julien.lutran@corp.ovh.com>
2020-04-15 10:33:08 +07:00
734c62bee0
Format code according to PSR2
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 16:56:50 +07:00
28f8eb5dba
Add visibility to all constants
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 16:54:27 +07:00
1584c9ae9c
Add visibility to all methods and position of static keyword
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 16:51:06 +07:00
caff1023ea
Format control structures, classes, methods and function
...
To continue this formatting madness, here's a tiny patch that adds
unified formatting for control structures like if and loops as well as
classes, their methods and anonymous functions. This basically forces
the constructs to start on the same line. This is not exactly what PSR2
wants, but I think we can have a few exceptions with "our" style. The
starting of braces on the same line is pracrically standard for our
code.
This also removes and empty lines from method/function bodies at the
beginning and end.
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 14:19:56 +07:00
14c996d982
Use elseif instead of else if
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 10:35:09 +07:00
44577e4345
Remove trailing and in between spaces
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-09 16:07:47 +07:00
afbd9c4e6e
Unify function spacing to PSR2 recommendation
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-09 13:54:22 +07:00
41b5e5923a
Use exactly one empty line after the namespace declaration
...
For PSR2
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-09 11:48:10 +07:00
7b07e7251c
make seekable s3 stream generic
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-01 15:21:05 +07:00
14401efb0f
Enable fseek for files in S3 storage
...
Signed-off-by: Lukas Stabe <lukas@stabe.de>
2020-04-01 15:21:01 +07:00
1a9330cd69
Update the license headers for Nextcloud 19
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-03-31 14:52:54 +07:00
b80ebc9674
Use the short array syntax, everywhere
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-03-26 16:34:56 +07:00
9a35e789b4
Get correct mimetype on objectstores
...
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-03-11 11:14:12 +07:00
5bf3d1bb38
Update license headers
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2019-12-05 15:38:45 +07:00
68748d4f85
Some php-cs fixes
...
* Order the imports
* No leading slash on imports
* Empty line before namespace
* One line per import
* Empty after imports
* Emmpty line at bottom of file
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-11-22 20:52:10 +07:00
733d4b6cca
dont delete cache entries if deleting an object from object store failed
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-09-25 18:09:45 +07:00
e45fb5fa3e
Fix typo in comment
...
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
2019-07-29 14:55:43 +07:00
d46744e2f1
Fix File#putContents(string) on ObjectStorage
...
Signed-off-by: Marcel Klehr <mklehr@gmx.net>
2019-07-17 14:58:56 +07:00
e953205908
Use HTTP1.1 to read S3 objects
...
Some of the READs otherwise use HTTP/1.0 which is not always supported
by all backends. HTTP/1.1 is there since 1999 way longer than S3 so safe
to assume it is always there IMO.
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-07-10 11:42:22 +07:00
96e892770d
Merge pull request #15715 from nextcloud/s3-bucket-name
...
Allow other code to use the s3 client from object store
2019-05-24 15:03:22 +07:00
834db3af2c
add method to get bucket name from s3 storage
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-05-24 14:02:03 +07:00
06a119242e
make ObjectStoreStorage::getURN public
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-05-22 12:17:33 +07:00
439b341e89
add method to get the used object store from the storage
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-05-21 17:18:00 +07:00
dedb84acb0
Always use a temp file for swift writes
...
Apparently the if statement doesn't work in all cases (even if I could
not reproduce it). So for the time being we will just not directly
stream to swift.
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-04-18 21:04:27 +07:00
1b27e9578d
Merge pull request #14424 from nextcloud/fix/13554/swift_to_tmp
...
Use a tmp file for swift writes
2019-03-06 15:33:15 +07:00
debd32b461
Merge pull request #14175 from skydiablo/master
...
Fix ObjectStorage expired token (v3)
2019-03-05 18:59:45 +07:00
1153123b90
Use a tmp file for swift writes
...
Else this leads to a seekable stream error with chunked uploads
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-03-04 13:11:59 +07:00
df55781814
Merge pull request #14254 from sparrowjack63/master
...
Dont check Bucket Name in Nextcloud
2019-03-01 13:46:04 +07:00
e3142a91b2
Bucket name - Alert in debug mode
...
Alert in debug mode only when trying to mount non compatible DNS bucket name (in order to not flood the logs)
Signed-off-by: Sébastien Cat <sebastien.cat@inra.fr>
2019-03-01 11:28:54 +07:00
381ae44d38
Correct code indent
...
Just correct code indent.
Signed-off-by: Sébastien Cat <sebastien.cat@inra.fr>
2019-03-01 11:28:54 +07:00
77473b0228
Warning on S3 not compatible DNS bucket name
...
The new created bucket should respect the DNS compatibility, nevertheless, Nextcloud should accept to mount "old created" buckets that does not respect DNS compatibility (Backward compatibility, or compatibility with CEPH).
Signed-off-by: Sébastien Cat <sebastien.cat@inra.fr>
2019-03-01 11:28:54 +07:00
59c3d8d85f
Dont check Bucket Name in Nextcloud
...
Signed-off-by: Sebastien Cat <sparrow.jack63@gmail.com>
Signed-off-by: Sébastien Cat <sebastien.cat@inra.fr>
2019-03-01 11:28:54 +07:00
f6f002e2ad
Merge pull request #14210 from nextcloud/fix/14192/fix_empty_uploads
...
Fix empty file uploads to S3 (and other streaming storages)
2019-02-18 15:39:17 +07:00
104616c1da
Use CountWrapper instead of almost identical CountReadStream
...
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-02-15 13:24:58 +07:00
61d66d7954
Fix empty file uploads to S3
...
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-02-15 08:41:39 +07:00
d06c25b9c8
Update SwiftFactory.php
...
Auth-Service (in explizit v3) will recheck the cached-token and will end in an "token expired exception".
2019-02-13 12:26:27 +07:00
c26bdd4731
Fix: Check if `$this->params['user']` is an array
...
Signed-off-by: William Pain <pain.william@gmail.com>
2019-02-05 09:56:42 +07:00
Ferdinand Thiessen
Tobias Zimmerer
Hamid Dehnavi
Côme Chilliet
Robin Appelman
Côme Chilliet
Julius Härtl
Thomas Citharel
Faraz Samapoor
Simon L
Rsplwe
François Ménabé
Maxence Lange
Carl Schwan
Jasper Weyne
Louis Chemineau
Stephen Cuppett
John Molakvoæ
Bernd Rederlechner
blizzz
Maxime Besson
Christoph Wurst
Morris Jobke
Florent
Samuel
dependabot-preview[bot]
Roeland Jago Douma
Tigran Mkrtchyan
James Letendre
Roeland Jago Douma
Daniel Kesselberg
Adrian Brzezinski
Julien Lutran
Lukas Stabe
Marcel Klehr
S. Cat
Sebastien Cat
Volker
William Pain