Commit Graph

181 Commits (8b7c8447a0e1eebc512cb58ea734542e03f29374)

Author SHA1 Message Date
Louis Chemineau bc2c02f451 Wrap S3 multipart upload exception
Signed-off-by: Louis Chemineau <louis@chmn.me>
2022-02-16 14:56:14 +07:00
Louis Chemineau 0b768c17d2 Wrap S3 multipart upload exception
Signed-off-by: Louis Chemineau <louis@chmn.me>
2022-02-15 17:35:12 +07:00
Julius Härtl 2aac757805
Make max size for single put uploads configurable
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2021-12-29 13:30:22 +07:00
Julius Härtl e475dfe16c
Fix php-cs for S3ConnectionTrait.php
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2021-12-21 21:35:46 +07:00
Julius Härtl 1acfbd0b06
Merge pull request #27493 from cuppett/cuppett/simplify-aws-credential-provider 2021-12-21 11:58:52 +07:00
Côme Chilliet 66bdda568a
Fix Guzzle function calls
Since 1.7.0 and 2.0 these functions are turned into static methods.
See https://github.com/guzzle/psr7#upgrading-from-function-api

Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2021-11-23 09:28:57 +07:00
Stephen Cuppett 4a6a2c2b48 Simplify S3ConnectionTrait to defaultProvider plus option
When we initially added the EC2 and ECS IAM role support in #24700,
we had to use a workaround by explicitly ordering the various providers
due to an inconsistency in the AWS SDK for PHP. We submitted a PR there
to get that squared away. Now, we've consumed that version upstream
for the SDK and can update our code here to be the most concise version
as well as position ourselves to pick up new methods as those become
available and prevalent in AWS (for acquiring credentials).

See also: https://github.com/nextcloud/server/pull/24700#issuecomment-747650892
See also: https://github.com/aws/aws-sdk-php/pull/2172

Signed-off-by: Stephen Cuppett <steve@cuppett.com>
2021-11-22 07:04:48 +07:00
John Molakvoæ 6ab2feaa54
Implement multibucket shift for ObjectStore
Signed-off-by: John Molakvoæ <skjnldsv@protonmail.com>
2021-11-11 15:09:28 +07:00
Carl Schwan df4e6bab69
Merge pull request #29115 from nextcloud/work/carl/correct-permissions-when-copying
Fix permissions when copying from ObjectStorage
2021-10-28 14:38:20 +07:00
Carl Schwan bfa60aaf27
Fix permissions when copying from ObjectStorage
Make sure that when a user copy a file from a directory they don't have
all permissions to a directory where they have more permissions, the
permissions are correctly set to the one from the parent taget folder.

This was caused by the ObjectStoreStorage::copyFromStorage using
the jailed storage and cache entry instead of the unjailed one like other
storages (the local one).
Steps to reproduce

+ Use object storage
+ Create a groupfolder with one group having full permission and another one
  who can just read files.
+ With an user who is in the second group, copy a file from the groupfolder to
  the home folder of this user.
+ The file in the home folder of the user will be read only and can't be deleted
  even though it is in their home folder and they are the owner. In oc_filecache,
  the permissions stored for this file are 1 (READ)

Signed-off-by: Carl Schwan <carl@carlschwan.eu>
2021-10-28 13:29:50 +07:00
Robin Appelman 09ffac5e6d
s3 external storage listing rework
Signed-off-by: Robin Appelman <robin@icewind.nl>
2021-10-15 15:35:02 +07:00
Bernd Rederlechner 3866f388b1
Refactor writeObject to only use MultipartUpload when required
Signed-off-by: Bernd Rederlechner <Bernd.Rederlechner@t-systems.com>

Co-authored-by: Julius Härtl <jus@bitgrid.net>
2021-08-20 17:02:25 +07:00
Julius Härtl 3e67637a4a
Merge pull request #26463 from Worteks/fix-s3proxy 2021-07-14 23:32:05 +07:00
blizzz 62675eb5c2
Merge pull request #27586 from nextcloud/bugfi/noid/objectstore-checksum
Reset checksum when writing files to object store
2021-07-01 17:30:54 +07:00
Maxime Besson 388a458cd5 fix(proxy): use accessor
Signed-off-by: Maxime Besson <maxime.besson@worteks.com>
2021-06-30 09:25:04 +07:00
Christoph Wurst 6d5cfe0c66
Move DateTime::RFC2822 to DateTimeInterface::2822
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2021-06-23 15:30:43 +07:00
Julius Härtl 21db463bc9
Reset checksum when writing files to object store
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2021-06-21 12:04:49 +07:00
John Molakvoæ (skjnldsv) 215aef3cbd
Update php licenses
Signed-off-by: John Molakvoæ (skjnldsv) <skjnldsv@protonmail.com>
2021-06-04 22:02:41 +07:00
Morris Jobke 333665b43d
Merge pull request #27040 from nextcloud/FlorentCoppint-master
Avoid reading ~/.aws/config when using S3 provider
2021-05-25 10:19:22 +07:00
Florent d4444f2472
Avoid reading ~/.aws/config when using S3 provider
When using S3 storage, we sometimes have open_basedir errors due to aws-sdk-php trying to read ~/.aws/config which is out of open_basedir restrictions.
Christoph Wurst already added csm=false config in #21406 but it wasn't enough, we also need to set use_arn_region=false, added in this commit.

Signed-off-by: Florent <florent@coppint.com>
2021-05-20 12:21:15 +07:00
Robin Appelman ef6f2e68f0
explicitly close source stream on object store upload even if count wrapper isn't needed
Signed-off-by: Robin Appelman <robin@icewind.nl>
2021-04-29 17:01:19 +07:00
Robin Appelman effb7dc8ba
set mimetype for objects uploaded to object storages
Signed-off-by: Robin Appelman <robin@icewind.nl>
2021-04-21 15:25:58 +07:00
Maxime Besson 547438527d fix(proxy): reaching s3 storage behind some http proxy
Signed-off-by: Maxime Besson <maxime.besson@worteks.com>
2021-04-09 10:15:57 +07:00
Samuel 03fe74b95e fix(proxy): reaching s3 storage behind some http proxy
Signed-off-by: Maxime Besson <maxime.besson@worteks.com>
2021-04-09 10:15:57 +07:00
Robin Appelman ad636ae12d
use returned file id for object store copy
Signed-off-by: Robin Appelman <robin@icewind.nl>
2021-03-12 17:22:40 +07:00
Robin Appelman 32551b9ff7
Merge pull request #25722 from nextcloud/objectstore-copy-cross
apply object store copy optimization when 'cross storage' copy is wit…
2021-03-12 16:10:18 +07:00
Robin Appelman 3eb764e65d
remove explicit fclose from S3->writeStream
streams get closed automatically when dropped, and in some cases the stream seems to be already closed by the S3 library, in which case trying to close it again will raise an error

Signed-off-by: Robin Appelman <robin@icewind.nl>
2021-03-11 18:05:04 +07:00
Robin Appelman c87b1a50d9 apply object store copy optimization when 'cross storage' copy is within the same object store
Signed-off-by: Robin Appelman <robin@icewind.nl>
2021-03-09 20:31:27 +07:00
dependabot-preview[bot] eb502c02ff
Bump nextcloud/coding-standard from 0.3.0 to 0.5.0
Bumps [nextcloud/coding-standard](https://github.com/nextcloud/coding-standard) from 0.3.0 to 0.5.0.
- [Release notes](https://github.com/nextcloud/coding-standard/releases)
- [Changelog](https://github.com/nextcloud/coding-standard/blob/master/CHANGELOG.md)
- [Commits](https://github.com/nextcloud/coding-standard/compare/v0.3.0...v0.5.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2021-02-18 13:31:24 +07:00
Roeland Jago Douma 137636b651
Merge pull request #24700 from Imajie/s3-creds
Resolves #24699, Support ES2 and ECS instance providers for S3 buckets
2021-02-02 21:48:22 +07:00
Christoph Wurst 9ce3ea3368
Update license headers
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-12-30 14:07:05 +07:00
Christoph Wurst 73c7d0dc81
Bump icewind/streams from 0.7.1 to 0.7.2
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-12-29 09:07:36 +07:00
Julius Härtl 5094e29ebd
Merge pull request #24594 from kofemann/dcache 2020-12-22 09:26:07 +07:00
Tigran Mkrtchyan 4f2dc18f58 storage: update IStorage#file_put_contents docs to match usage
The current phpdoc of IStorage#file_put_contents doesnt corresponds to
it's actual usage in code, e.g.

Signed-off-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
2020-12-21 19:04:18 +07:00
Christoph Wurst d89a75be0b
Update all license headers for Nextcloud 21
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-12-16 18:48:22 +07:00
James Letendre 45a02ee30c Fix failing tests
Signed-off-by: James Letendre <james.letendre@gmail.com>
2020-12-15 19:06:06 +07:00
James Letendre ad95e51cbe Add web identity provider to S3 connection chain
Signed-off-by: James Letendre <james.letendre@gmail.com>
2020-12-15 16:37:46 +07:00
James Letendre 5b756a9fb0 Replace defaultProvider with explicit calls to exclude user home directory lookup
Signed-off-by: James Letendre <james.letendre@gmail.com>
2020-12-14 16:22:41 +07:00
James Letendre 85aa77539b Resolves #24699, Support ES2 and ECS instance providers for S3 buckets
Signed-off-by: James Letendre <james.letendre@gmail.com>
2020-12-14 12:55:46 +07:00
Robin Appelman 9d4848e863
use in objectstore copy
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-11-24 15:16:58 +07:00
Roeland Jago Douma 65141d4864
Allow config to specify the bucket exists
In the 99% case the bucket is just always there. And if it is not the
read/write will fail hard anyways. Esp on big instances the Objectstore
is not always fast and this can save a few hundered ms of each request
that acess the objectstore.

In short it is adding

'verify_bucket_exists' => false

To the S3 config part

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-11-06 11:08:38 +07:00
Christoph Wurst d9015a8c94
Format code to a single space around binary operators
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-10-05 20:25:24 +07:00
Roeland Jago Douma bb06b6cce4
Fix reading empty files from objectstorage
Since we try to do range requests this will fail hard.
However since empty files are not that interesting to read anyways we
just read from an emptry memory stream.

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-09-09 20:45:13 +07:00
Roeland Jago Douma 9fdeed8cc0
Run stat less often for objectstorages
When we want to get the permissions we now do stat at least 5 times for
each entry. Which is a bit much. Especially since the permssions are all
just in the database already.

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-09-08 14:38:36 +07:00
Roeland Jago Douma 789b33aba4
Only update the filecache entry once the file has been written to S3
If we already update before we have no way to revert if the upload
fails.

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-08-31 12:28:04 +07:00
Roeland Jago Douma 6ffd7173f9
Don't lose filecache entry on s3 overwrite error
If the object store errors we should not always delete the filecache
entry. As this might lead to people losing access to their files.

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-08-31 12:25:20 +07:00
Christoph Wurst 2a054e6c04
Update the license headers for Nextcloud 20
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-08-24 14:54:25 +07:00
Stephen Cuppett 5ef0f86ce7
Resolves #19790, Provides Support for IAM Credentials
Includes support for either leveraging environment variables
passed to the PHP runtime or IAM instance profile present
on the host being used. The default and first choice is
still the parameter file as documented.

See also: https://docs.aws.amazon.com/sdk-for-php/v3/developer-guide/guide_credentials_provider.html#chaining-providers

Signed-off-by: Stephen Cuppett <steve@cuppett.com>
2020-08-20 15:54:33 +07:00
Morris Jobke fedf9c69d9
Use matching parameter names form interfaces and implementations
Found by Psalm 3.14.1

Signed-off-by: Morris Jobke <hey@morrisjobke.de>
2020-08-19 18:16:35 +07:00
Morris Jobke 438ac23e2a
Distribute preview folders in appdata in multibucket setup to multiple buckets
* introduces a new IRootMountProvider to register mount points inside the root storage
* adds a AppdataPreviewObjectStoreStorage to handle the split between preview folders and bucket number

Ref #22033

Signed-off-by: Morris Jobke <hey@morrisjobke.de>
2020-08-06 22:19:20 +07:00