b6313f68d3
perf(s3): Expose pre-signed urls for S3
...
This is faster than going back to nextcloud to download the files.
This is an opt-in setting that can be enabled by setting
use_presigned_url in the object store config.
Additionally add support for the proxy config which is needed in a
docker setup. See https://github.com/juliusknorr/nextcloud-docker-dev/pull/431
Signed-off-by: Carl Schwan <carl.schwan@nextcloud.com>
2025-12-09 11:21:35 +07:00
8194be795d
fix: encode s3 metadata as base64 if needed
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-11-18 16:06:57 +07:00
00ec57efd6
fix(S3): Only append streams if non-seekable
...
Later, when we rewind the stream in `writeMultiPart` during retry, both streams were rewinded, so the resulting stream was bigger than expected.
Inspired by https://github.com/aws/aws-sdk-php/blob/master/src/S3/ObjectUploader.php#L136-L146
Signed-off-by: Louis Chmn <louis@chmn.me>
2025-10-10 09:54:05 +07:00
398b106f0c
fix: validate written size for s3 multipart uploads
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-08-05 19:17:58 +07:00
06b5ce5859
fix(s3): retry failed multipart uploads with decreased concurrency
...
Signed-off-by: Kent Delante <kent.delante@proton.me>
2025-06-10 17:39:59 +07:00
8aae332622
feat: store original storage id and path in object store metadata
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-03-31 15:10:58 +07:00
94114b99f7
feat: more generic way of passing metadata to object storage backends for new objects
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-03-31 14:55:18 +07:00
ca547bb913
fix: don't perform the extra buffering in s3 stream write when the stream size is known
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-12-05 14:07:40 +07:00
7da8951c57
fix: use php://temp instead of php://memory for multi-part upload buffer
...
this should reduce potential memory issues if the limit is set very high
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-12-05 14:07:37 +07:00
af6de04e9e
style: update codestyle for coding-standard 1.2.3
...
Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>
2024-08-25 19:34:58 +07:00
dae7c159f7
chore: Add SPDX header
...
Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
2024-05-24 13:11:22 +07:00
ec5133b739
fix: Apply new coding standard to all files
...
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2024-04-02 14:16:21 +07:00
32dee2f84a
feat(ObjectStore): Make S3 MultipartUpload concurrency configurable
...
Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
2024-03-21 16:25:49 +07:00
69c5e8ebde
fix(ObjectStore): Use common S3 trait so config and connection trait members are in sync
...
Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
2024-03-21 16:25:42 +07:00
c54f7bd761
fix: disable the internal s3sdk multi part copy logic
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-03-06 15:32:14 +07:00
e4054370b1
fix(s3): Add config option to disable multipart copy for certain s3 providers
...
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2023-12-28 15:31:38 +07:00
1043c21b35
only do a multipart s3 copy when above the regular copy limit
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-12-28 15:31:37 +07:00
cdf96fab92
fix(s3): support SSE-C headers for the MultipartCopy call
...
Signed-off-by: Tobias Zimmerer <3228193+ir0nhide@users.noreply.github.com>
2023-11-14 21:19:59 +07:00
eab8dd3ca0
use multipart copy for s3
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-09-20 17:28:48 +07:00
ea88ec1350
add utility command for object store objects
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-06-09 18:08:18 +07:00
159a0c8411
feat(s3): Add option to specify an SSE-C customer provided key
...
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2023-01-24 10:36:03 +07:00
cf9bdc0274
Manage s3 storage class in objectstore
...
Signed-off-by: François Ménabé <francois.menabe@gmail.com>
2023-01-12 14:58:27 +07:00
cc639a8877
compare cached filesize on download
...
Signed-off-by: Maxence Lange <maxence@artificial-owl.com>
2022-12-01 15:42:15 +07:00
214a50d071
Avoid allocating too much memory for the buffer
...
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2022-10-11 15:34:27 +07:00
61fdd91dcf
don't set `null` as a bundle path
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2022-07-27 12:58:07 +07:00
f1486890d7
only use nextcloud bundle when explicitly enabled
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2022-07-05 16:06:55 +07:00
bffa67c48b
also use nextcloud certificate bundle when downloading from s3
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2022-07-05 15:57:54 +07:00
bc2c02f451
Wrap S3 multipart upload exception
...
Signed-off-by: Louis Chemineau <louis@chmn.me>
2022-02-16 14:56:14 +07:00
0b768c17d2
Wrap S3 multipart upload exception
...
Signed-off-by: Louis Chemineau <louis@chmn.me>
2022-02-15 17:35:12 +07:00
2aac757805
Make max size for single put uploads configurable
...
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2021-12-29 13:30:22 +07:00
09ffac5e6d
s3 external storage listing rework
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2021-10-15 15:35:02 +07:00
3866f388b1
Refactor writeObject to only use MultipartUpload when required
...
Signed-off-by: Bernd Rederlechner <Bernd.Rederlechner@t-systems.com>
Co-authored-by: Julius Härtl <jus@bitgrid.net>
2021-08-20 17:02:25 +07:00
3e67637a4a
Merge pull request #26463 from Worteks/fix-s3proxy
2021-07-14 23:32:05 +07:00
215aef3cbd
Update php licenses
...
Signed-off-by: John Molakvoæ (skjnldsv) <skjnldsv@protonmail.com>
2021-06-04 22:02:41 +07:00
effb7dc8ba
set mimetype for objects uploaded to object storages
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2021-04-21 15:25:58 +07:00
547438527d
fix(proxy): reaching s3 storage behind some http proxy
...
Signed-off-by: Maxime Besson <maxime.besson@worteks.com>
2021-04-09 10:15:57 +07:00
03fe74b95e
fix(proxy): reaching s3 storage behind some http proxy
...
Signed-off-by: Maxime Besson <maxime.besson@worteks.com>
2021-04-09 10:15:57 +07:00
3eb764e65d
remove explicit fclose from S3->writeStream
...
streams get closed automatically when dropped, and in some cases the stream seems to be already closed by the S3 library, in which case trying to close it again will raise an error
Signed-off-by: Robin Appelman <robin@icewind.nl>
2021-03-11 18:05:04 +07:00
9d4848e863
use in objectstore copy
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-11-24 15:16:58 +07:00
2a054e6c04
Update the license headers for Nextcloud 20
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-08-24 14:54:25 +07:00
3594ba6971
Upload part size as S3 parameter instead of constant value
...
Some S3 providers need a custom upload part size (500 MB static value in Nextcloud).
Here is a commit to change this value via S3 configuration, instead of using S3_UPLOAD_PART_SIZE constant.
A new parameter is added for an S3 connection : uploadPartSize
Signed-off-by: Florent <florent@coppint.com>
2020-06-09 09:18:42 +07:00
cb057829f7
Update license headers for 19
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-29 11:57:22 +07:00
1584c9ae9c
Add visibility to all methods and position of static keyword
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 16:51:06 +07:00
7b07e7251c
make seekable s3 stream generic
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-01 15:21:05 +07:00
14401efb0f
Enable fseek for files in S3 storage
...
Signed-off-by: Lukas Stabe <lukas@stabe.de>
2020-04-01 15:21:01 +07:00
5bf3d1bb38
Update license headers
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2019-12-05 15:38:45 +07:00
e45fb5fa3e
Fix typo in comment
...
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
2019-07-29 14:55:43 +07:00
e953205908
Use HTTP1.1 to read S3 objects
...
Some of the READs otherwise use HTTP/1.0 which is not always supported
by all backends. HTTP/1.1 is there since 1999 way longer than S3 so safe
to assume it is always there IMO.
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-07-10 11:42:22 +07:00
61d66d7954
Fix empty file uploads to S3
...
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-02-15 08:41:39 +07:00
1d322d3b37
always use multipart uploader for s3 uploads
...
the multipart uploader handles non seekable streams while `upload` does not
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-01-28 14:51:19 +07:00
Carl Schwan
Robin Appelman
Louis Chmn
Kent Delante
Daniel Kesselberg
Andy Scherzinger
Côme Chilliet
Ferdinand Thiessen
Julius Härtl
Tobias Zimmerer
François Ménabé
Maxence Lange
Bernd Rederlechner
John Molakvoæ (skjnldsv)
Maxime Besson
Samuel
Christoph Wurst
Florent
Lukas Stabe
Morris Jobke
Roeland Jago Douma