Commit Graph

173 Commits (6d5f10eb577a80013c9db9996b156838e4b71908)

Author SHA1 Message Date
Carl Schwan df4e6bab69
Merge pull request #29115 from nextcloud/work/carl/correct-permissions-when-copying
Fix permissions when copying from ObjectStorage
2021-10-28 14:38:20 +07:00
Carl Schwan bfa60aaf27
Fix permissions when copying from ObjectStorage
Make sure that when a user copy a file from a directory they don't have
all permissions to a directory where they have more permissions, the
permissions are correctly set to the one from the parent taget folder.

This was caused by the ObjectStoreStorage::copyFromStorage using
the jailed storage and cache entry instead of the unjailed one like other
storages (the local one).
Steps to reproduce

+ Use object storage
+ Create a groupfolder with one group having full permission and another one
  who can just read files.
+ With an user who is in the second group, copy a file from the groupfolder to
  the home folder of this user.
+ The file in the home folder of the user will be read only and can't be deleted
  even though it is in their home folder and they are the owner. In oc_filecache,
  the permissions stored for this file are 1 (READ)

Signed-off-by: Carl Schwan <carl@carlschwan.eu>
2021-10-28 13:29:50 +07:00
Robin Appelman 09ffac5e6d
s3 external storage listing rework
Signed-off-by: Robin Appelman <robin@icewind.nl>
2021-10-15 15:35:02 +07:00
Bernd Rederlechner 3866f388b1
Refactor writeObject to only use MultipartUpload when required
Signed-off-by: Bernd Rederlechner <Bernd.Rederlechner@t-systems.com>

Co-authored-by: Julius Härtl <jus@bitgrid.net>
2021-08-20 17:02:25 +07:00
Julius Härtl 3e67637a4a
Merge pull request #26463 from Worteks/fix-s3proxy 2021-07-14 23:32:05 +07:00
blizzz 62675eb5c2
Merge pull request #27586 from nextcloud/bugfi/noid/objectstore-checksum
Reset checksum when writing files to object store
2021-07-01 17:30:54 +07:00
Maxime Besson 388a458cd5 fix(proxy): use accessor
Signed-off-by: Maxime Besson <maxime.besson@worteks.com>
2021-06-30 09:25:04 +07:00
Christoph Wurst 6d5cfe0c66
Move DateTime::RFC2822 to DateTimeInterface::2822
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2021-06-23 15:30:43 +07:00
Julius Härtl 21db463bc9
Reset checksum when writing files to object store
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2021-06-21 12:04:49 +07:00
John Molakvoæ (skjnldsv) 215aef3cbd
Update php licenses
Signed-off-by: John Molakvoæ (skjnldsv) <skjnldsv@protonmail.com>
2021-06-04 22:02:41 +07:00
Morris Jobke 333665b43d
Merge pull request #27040 from nextcloud/FlorentCoppint-master
Avoid reading ~/.aws/config when using S3 provider
2021-05-25 10:19:22 +07:00
Florent d4444f2472
Avoid reading ~/.aws/config when using S3 provider
When using S3 storage, we sometimes have open_basedir errors due to aws-sdk-php trying to read ~/.aws/config which is out of open_basedir restrictions.
Christoph Wurst already added csm=false config in #21406 but it wasn't enough, we also need to set use_arn_region=false, added in this commit.

Signed-off-by: Florent <florent@coppint.com>
2021-05-20 12:21:15 +07:00
Robin Appelman ef6f2e68f0
explicitly close source stream on object store upload even if count wrapper isn't needed
Signed-off-by: Robin Appelman <robin@icewind.nl>
2021-04-29 17:01:19 +07:00
Robin Appelman effb7dc8ba
set mimetype for objects uploaded to object storages
Signed-off-by: Robin Appelman <robin@icewind.nl>
2021-04-21 15:25:58 +07:00
Maxime Besson 547438527d fix(proxy): reaching s3 storage behind some http proxy
Signed-off-by: Maxime Besson <maxime.besson@worteks.com>
2021-04-09 10:15:57 +07:00
Samuel 03fe74b95e fix(proxy): reaching s3 storage behind some http proxy
Signed-off-by: Maxime Besson <maxime.besson@worteks.com>
2021-04-09 10:15:57 +07:00
Robin Appelman ad636ae12d
use returned file id for object store copy
Signed-off-by: Robin Appelman <robin@icewind.nl>
2021-03-12 17:22:40 +07:00
Robin Appelman 32551b9ff7
Merge pull request #25722 from nextcloud/objectstore-copy-cross
apply object store copy optimization when 'cross storage' copy is wit…
2021-03-12 16:10:18 +07:00
Robin Appelman 3eb764e65d
remove explicit fclose from S3->writeStream
streams get closed automatically when dropped, and in some cases the stream seems to be already closed by the S3 library, in which case trying to close it again will raise an error

Signed-off-by: Robin Appelman <robin@icewind.nl>
2021-03-11 18:05:04 +07:00
Robin Appelman c87b1a50d9 apply object store copy optimization when 'cross storage' copy is within the same object store
Signed-off-by: Robin Appelman <robin@icewind.nl>
2021-03-09 20:31:27 +07:00
dependabot-preview[bot] eb502c02ff
Bump nextcloud/coding-standard from 0.3.0 to 0.5.0
Bumps [nextcloud/coding-standard](https://github.com/nextcloud/coding-standard) from 0.3.0 to 0.5.0.
- [Release notes](https://github.com/nextcloud/coding-standard/releases)
- [Changelog](https://github.com/nextcloud/coding-standard/blob/master/CHANGELOG.md)
- [Commits](https://github.com/nextcloud/coding-standard/compare/v0.3.0...v0.5.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2021-02-18 13:31:24 +07:00
Roeland Jago Douma 137636b651
Merge pull request #24700 from Imajie/s3-creds
Resolves #24699, Support ES2 and ECS instance providers for S3 buckets
2021-02-02 21:48:22 +07:00
Christoph Wurst 9ce3ea3368
Update license headers
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-12-30 14:07:05 +07:00
Christoph Wurst 73c7d0dc81
Bump icewind/streams from 0.7.1 to 0.7.2
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-12-29 09:07:36 +07:00
Julius Härtl 5094e29ebd
Merge pull request #24594 from kofemann/dcache 2020-12-22 09:26:07 +07:00
Tigran Mkrtchyan 4f2dc18f58 storage: update IStorage#file_put_contents docs to match usage
The current phpdoc of IStorage#file_put_contents doesnt corresponds to
it's actual usage in code, e.g.

Signed-off-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
2020-12-21 19:04:18 +07:00
Christoph Wurst d89a75be0b
Update all license headers for Nextcloud 21
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-12-16 18:48:22 +07:00
James Letendre 45a02ee30c Fix failing tests
Signed-off-by: James Letendre <james.letendre@gmail.com>
2020-12-15 19:06:06 +07:00
James Letendre ad95e51cbe Add web identity provider to S3 connection chain
Signed-off-by: James Letendre <james.letendre@gmail.com>
2020-12-15 16:37:46 +07:00
James Letendre 5b756a9fb0 Replace defaultProvider with explicit calls to exclude user home directory lookup
Signed-off-by: James Letendre <james.letendre@gmail.com>
2020-12-14 16:22:41 +07:00
James Letendre 85aa77539b Resolves #24699, Support ES2 and ECS instance providers for S3 buckets
Signed-off-by: James Letendre <james.letendre@gmail.com>
2020-12-14 12:55:46 +07:00
Robin Appelman 9d4848e863
use in objectstore copy
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-11-24 15:16:58 +07:00
Roeland Jago Douma 65141d4864
Allow config to specify the bucket exists
In the 99% case the bucket is just always there. And if it is not the
read/write will fail hard anyways. Esp on big instances the Objectstore
is not always fast and this can save a few hundered ms of each request
that acess the objectstore.

In short it is adding

'verify_bucket_exists' => false

To the S3 config part

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-11-06 11:08:38 +07:00
Christoph Wurst d9015a8c94
Format code to a single space around binary operators
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-10-05 20:25:24 +07:00
Roeland Jago Douma bb06b6cce4
Fix reading empty files from objectstorage
Since we try to do range requests this will fail hard.
However since empty files are not that interesting to read anyways we
just read from an emptry memory stream.

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-09-09 20:45:13 +07:00
Roeland Jago Douma 9fdeed8cc0
Run stat less often for objectstorages
When we want to get the permissions we now do stat at least 5 times for
each entry. Which is a bit much. Especially since the permssions are all
just in the database already.

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-09-08 14:38:36 +07:00
Roeland Jago Douma 789b33aba4
Only update the filecache entry once the file has been written to S3
If we already update before we have no way to revert if the upload
fails.

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-08-31 12:28:04 +07:00
Roeland Jago Douma 6ffd7173f9
Don't lose filecache entry on s3 overwrite error
If the object store errors we should not always delete the filecache
entry. As this might lead to people losing access to their files.

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-08-31 12:25:20 +07:00
Christoph Wurst 2a054e6c04
Update the license headers for Nextcloud 20
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-08-24 14:54:25 +07:00
Stephen Cuppett 5ef0f86ce7
Resolves #19790, Provides Support for IAM Credentials
Includes support for either leveraging environment variables
passed to the PHP runtime or IAM instance profile present
on the host being used. The default and first choice is
still the parameter file as documented.

See also: https://docs.aws.amazon.com/sdk-for-php/v3/developer-guide/guide_credentials_provider.html#chaining-providers

Signed-off-by: Stephen Cuppett <steve@cuppett.com>
2020-08-20 15:54:33 +07:00
Morris Jobke fedf9c69d9
Use matching parameter names form interfaces and implementations
Found by Psalm 3.14.1

Signed-off-by: Morris Jobke <hey@morrisjobke.de>
2020-08-19 18:16:35 +07:00
Morris Jobke 438ac23e2a
Distribute preview folders in appdata in multibucket setup to multiple buckets
* introduces a new IRootMountProvider to register mount points inside the root storage
* adds a AppdataPreviewObjectStoreStorage to handle the split between preview folders and bucket number

Ref #22033

Signed-off-by: Morris Jobke <hey@morrisjobke.de>
2020-08-06 22:19:20 +07:00
Christoph Wurst d3a4bc91f7
Fix static method call for s3 bucket compat check
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-07-03 09:02:58 +07:00
Daniel Kesselberg 8aa99aa8ff
Update presign method to match with interface again.
Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>
2020-06-25 22:56:28 +07:00
Christoph Wurst 9d392891be
Disable Client-Side Monitoring on AWS storage
The S3 client enables this by default and then tries to read
`.aws/config`. This causes `open_basedir` restriction related error for
some setups. So this patch disables the CSM because it's most likely
unused anyway.

Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-06-15 09:13:08 +07:00
Florent 3594ba6971 Upload part size as S3 parameter instead of constant value
Some S3 providers need a custom upload part size (500 MB static value in Nextcloud).
Here is a commit to change this value via S3 configuration, instead of using S3_UPLOAD_PART_SIZE constant.
A new parameter is added for an S3 connection : uploadPartSize

Signed-off-by: Florent <florent@coppint.com>
2020-06-09 09:18:42 +07:00
Roeland Jago Douma d5850eb28f
Merge pull request #18955 from adrb/swift_upload_large_objects
Large Object support for OpenStack Swift
2020-05-06 13:03:24 +07:00
Robin Appelman 4e6d3d1880
Fix getDirectoryContent implementation for Jail wrapper
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-30 15:27:09 +07:00
Roeland Jago Douma f6b435d0d9
Run the cs fixer for green ci
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-04-30 10:28:44 +07:00
Roeland Jago Douma a1c1b354fc
Merge pull request #18883 from adrb/optimize_swift_14116
Optimize Openstack Swift files download
2020-04-30 09:06:19 +07:00