492e6997d8
chore: Fix psalm issues, put back private versions of getter in private Storage interface
...
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2024-09-16 18:11:38 +07:00
ff0cab5956
Merge pull request #47852 from nextcloud/sharding-code-fixes
2024-09-16 11:39:58 +07:00
714671a8a6
fix(files): Adjust Cache::searchQuery() parameter name to match interface
...
Signed-off-by: provokateurin <kate@provokateurin.de>
2024-09-13 18:13:44 +07:00
da59fd4389
fix: misc code fixes around db sharding
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-09 16:59:59 +07:00
8af7ecb257
chore: adjust code to adhere to coding standard
...
Signed-off-by: Anna Larch <anna@nextcloud.com>
2024-09-05 21:23:38 +07:00
2574cbfa61
chore: Apply php:cs recommendations
...
Signed-off-by: Louis Chemineau <louis@chmn.me>
2024-08-28 10:44:18 +07:00
390f6a78b4
fix: hint storage id in more places
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-08-28 10:21:19 +07:00
62f8b6517f
feat: implement distributing partitioned queries over multiple shards
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-08-28 10:21:19 +07:00
af6de04e9e
style: update codestyle for coding-standard 1.2.3
...
Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>
2024-08-25 19:34:58 +07:00
5ca9d884d7
fix: don't recalculate folder size in Cache::delete if the entry didn't exist
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-08-20 12:50:42 +07:00
c5b687271b
fix: make batch propagator work with sharding restrictions
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-07-17 19:16:07 +07:00
ad88fd07e3
fix: make joining on tags in search queries work with sharding
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-07-17 15:46:39 +07:00
80f8c7949e
fix: always set storage id in Cache::get
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-07-17 15:46:31 +07:00
5ff7bde3fb
fix: add set storage id for more cache queries
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-07-17 15:46:21 +07:00
c82d382a20
feat: add base class for extending the query builder
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-07-12 17:01:39 +07:00
3a60e40a81
Merge pull request #45172 from nextcloud/test-scanner-no-transactions
...
feat: add option to disable scanner transactions
2024-06-13 14:01:34 +07:00
082c6c6e1d
fix: get child ids for folder in a separate query during move
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-06-13 10:38:57 +07:00
a93d3a5a10
fix(files): Use isRetryable to catch retryable exceptions
...
Signed-off-by: Louis Chemineau <louis@chmn.me>
2024-06-11 15:15:14 +07:00
dae7c159f7
chore: Add SPDX header
...
Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
2024-05-24 13:11:22 +07:00
e68544d9f4
Merge pull request #43794 from nextcloud/fix/files/scanner-metadata-diff
...
fix(files): handle multidimensional arrays in scanner
2024-05-13 11:05:40 +07:00
3c3e45f0ad
fix(Files): Change how scanner diffs for changed metadata
...
Fixes #43408
Signed-off-by: Josh <josh.t.richards@gmail.com>
2024-05-08 08:36:42 +07:00
6258823aba
Merge pull request #45058 from nextcloud/perf/db/sort-before-chunking
...
perf(db): Sort data for IN before chunking
2024-05-07 12:11:32 +07:00
f5219a19b2
feat: add option to disable scanner transactions
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-05-03 16:40:15 +07:00
8d870a32c2
fix: fix FileAccess::getByFileId(InStorage)
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-05-01 18:00:20 +07:00
f3d6309e36
perf(db): Sort data for IN before chunking
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2024-04-26 15:38:29 +07:00
b4004a2582
Merge pull request #43574 from nextcloud/enh/noid/lazyappconfig-migrate-metadata
...
enh(metadata): migrate metadata to lazy appconfig
2024-04-17 16:20:33 +07:00
d31ed8dbca
fix: use proper jailed patch in watcher
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-04-08 18:37:05 +07:00
ec5133b739
fix: Apply new coding standard to all files
...
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2024-04-02 14:16:21 +07:00
751795999a
feat(files)): Add retry logic to cover deadlock situations when moving many files
...
Signed-off-by: Louis Chemineau <louis@chmn.me>
2024-03-28 09:05:47 +07:00
1d9354a81e
feat: add interface for lower level filecache acess without having to do direct db queries
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-03-25 16:48:41 +07:00
45eb4a839d
Refactor `OC\Server::getLockingProvider`
...
Signed-off-by: Andrew Summers <18727110+summersab@users.noreply.github.com>
2024-03-15 13:11:33 +07:00
ffeb797ecc
refactor(mimeloader): modernize MimeTypeLoader
...
Signed-off-by: Thomas Citharel <tcit@tcit.fr>
2024-03-14 17:24:36 +07:00
575159b819
fix(filecache): Move from array_merge to avoid memory exhaustion
...
Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>
2024-03-04 18:13:45 +07:00
63ffaab95e
fix types + autoloader
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-02-15 17:55:43 +07:00
2e14a7a4a6
optimize query pattern used by storage filter
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-02-15 17:55:40 +07:00
4e1814acd1
migrate metadata to lazy appconfig
...
Signed-off-by: Maxence Lange <maxence@artificial-owl.com>
2024-02-14 13:30:27 +07:00
e9d97a568f
use OCP\Server
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-02-12 16:39:00 +07:00
e50c176428
also improe cache ci for shared cache
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-02-09 14:00:26 +07:00
7a91abb439
improve di performance for cache
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-02-09 13:28:30 +07:00
67a3ab48b5
Merge pull request #43131 from nextcloud/fix/files/cache-wrapper-no-partial-cache-entry-formatting
...
fix(files): Don't attempt to format a partial cache entry
2024-02-06 13:51:51 +07:00
b777304a77
also updated parent etags when a changed etag is detected during scanning
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-01-31 18:06:18 +07:00
e14bceb63f
dont reuse etag for folders marked explicitly unscanned
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-01-31 18:05:09 +07:00
7e5303bec7
fix(files): Don't attempt to format a partial cache entry
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2024-01-25 19:30:38 +07:00
cc75294eb6
perf: Avoid updating the folder size if we know the size difference
...
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2024-01-12 14:21:08 +07:00
394cd75322
Merge pull request #41272 from nextcloud/fixOccScans
...
fix: reduce memory consumption of scans
2023-11-28 11:36:59 +07:00
aa5f037af7
chore: apply changes from Nextcloud coding standards 1.1.1
...
Signed-off-by: Joas Schilling <coding@schilljs.com>
Signed-off-by: Benjamin Gaussorgues <benjamin.gaussorgues@nextcloud.com>
2023-11-23 10:36:13 +07:00
ee787cd1c6
Merge pull request #41459 from nextcloud/enh/noid/search-metadata-null
...
implements search on null/notnull metadata
2023-11-23 07:52:27 +07:00
7467422f57
implements search on null/notnull metadata
...
Signed-off-by: Maxence Lange <maxence@artificial-owl.com>
2023-11-22 12:43:27 +07:00
ea8f9a7e84
refactor: Repalce array_search with in_array in lib/
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2023-11-22 11:42:53 +07:00
761a688ec9
fix psalm
...
Signed-off-by: Maxence Lange <maxence@artificial-owl.com>
2023-11-14 10:11:24 +07:00
3af12b2a50
ignore metadata if table is empty
...
Signed-off-by: Maxence Lange <maxence@artificial-owl.com>
2023-11-14 09:28:58 +07:00
380b81315a
moving metadataquery
...
Signed-off-by: Maxence Lange <maxence@artificial-owl.com>
2023-11-13 22:25:37 +07:00
a75a93af8e
feat(files): Allow advanced search for files
...
Signed-off-by: Benjamin Gaussorgues <benjamin.gaussorgues@nextcloud.com>
2023-11-10 09:21:16 +07:00
d3a313f192
Support getting and setting metadata in DAV requests
...
Signed-off-by: Louis Chemineau <louis@chmn.me>
2023-11-08 16:23:53 +07:00
f497d8b6e5
IFilesMetadata
...
Signed-off-by: Maxence Lange <maxence@artificial-owl.com>
2023-11-07 12:43:08 +07:00
e62e9e3dbf
IFilesMetadata
...
Signed-off-by: Maxence Lange <maxence@artificial-owl.com>
2023-11-07 00:21:38 +07:00
b4a3773115
fix: reduce memory consumption of scans
...
Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>
Fix lint
Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>
2023-11-03 18:07:42 +07:00
460344336e
optimize cache jail creation
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-10-19 19:27:00 +07:00
ea06cf2f39
Convert isset ternary to null coalescing operator
...
Signed-off-by: Hamid Dehnavi <hamid.dev.pro@gmail.com>
2023-09-28 17:44:19 +07:00
5090848b25
Store size in int|float for 32bit support
...
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2023-09-19 13:43:01 +07:00
13b9dd2267
fix: Only read unencrypted_size when file is actually encrypted
...
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2023-09-06 07:25:28 +07:00
9ea03a9534
reuse dependencies from wrapped cache
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-08-18 11:14:52 +07:00
9c04c07622
fix: Prevent PHP warnings when optional CacheEntry attributes are unset
...
Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
2023-08-16 14:12:51 +07:00
658aed27ea
select the fileid first when looking for incomplete files
...
this seems to improve mariadbs index selection
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-08-14 18:08:01 +07:00
7b7d1e3c09
fix: simplify `sourceData` check
...
Co-authored-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>
Signed-off-by: John Molakvoæ <skjnldsv@users.noreply.github.com>
2023-08-05 12:05:36 +07:00
deecb48e54
Catch more invalid cache source storage paths
...
OC\Files\Cache\Cache::get can return string|false|null, not just string|false.
- nextcloud/server#26270 added handling of false, but null is needed too.
- Well, or we change the default $resullt to false, but I'm not sure if that has other ramifications and the real need here is to simply catch situations where the cache source storage path is not valid for whatever reason
Related: nextcloud/server#19009
Signed-off-by: Josh Richards <josh.t.richards@gmail.com>
2023-08-03 09:53:46 +07:00
c912a3f0f6
Merge pull request #38860 from nextcloud/scanner-change-summary
...
add summary of detected changes to files:scan output
2023-07-20 18:37:07 +07:00
14719110b9
chore: Replace \OC::$server->query with \OCP\Server::get in /lib
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2023-07-06 15:21:22 +07:00
eddb64f8c3
Merge pull request #37969 from nextcloud/poc/noid/systemtags-perf-tag-endpoint
2023-06-28 07:53:35 +07:00
39a0554c09
Merge pull request #38858 from nextcloud/scanner-performance3
...
file scanner performance improvements
2023-06-26 17:01:18 +07:00
49db546f78
fix: include invisible tags for admins
...
Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
2023-06-21 20:28:01 +07:00
783e32a64d
fix: favorites view and universal search against tags
...
Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
2023-06-21 18:19:08 +07:00
b0d1cf5730
fix: change if with conditionless else to switch; and a parameter value
...
Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
2023-06-21 18:19:08 +07:00
58f7fd2370
use efficient tag retrieval on DAV report request
...
- uses DAV search approach against valid files joined by systemtag selector
- reduced table join for tag/systemtag search
- supports pagination
- no changes to the output formats or similar
Example request body:
<?xml version="1.0"?>
<oc:filter-files xmlns:d="DAV:" xmlns:oc="http://owncloud.org/ns " xmlns:nc="http://nextcloud.org/ns " xmlns:ocs="http://open-collaboration-services.org/ns ">
<d:prop>
<d:getcontentlength/>
<d:getcontenttype/>
<d:getetag/>
<d:getlastmodified/>
<d:resourcetype/>
<nc:face-detections/>
<nc:file-metadata-size/>
<nc:has-preview/>
<nc:realpath/>
<oc:favorite/>
<oc:fileid/>
<oc:permissions/>
<nc:nbItems/>
</d:prop>
<oc:filter-rules>
<oc:systemtag>32</oc:systemtag>
</oc:filter-rules>
<d:limit>
<d:nresults>50</d:nresults>
<nc:firstresult>0</nc:firstresult>
</d:limit>
</oc:filter-files>
Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
2023-06-21 18:19:08 +07:00
71e41ffeeb
refactor: declare getMount() and getMountsIn() at IRootFolder
...
Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
2023-06-21 16:53:20 +07:00
f6ac874dbf
chore: ugly type juggling
...
Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
2023-06-21 16:53:20 +07:00
eb50b6c6aa
fix: expect interface, not a specific implementation
...
- fixes a regression when deleting folders while music app was enabled,
for a LazyRoot was passed to this method.
Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
2023-06-21 16:53:13 +07:00
292c0e53f8
add summary of detected changes to files:scan output
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-06-16 17:56:34 +07:00
561536fc58
file scanner performance improvements
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-06-16 16:31:48 +07:00
872c181c74
chore: Drop dead private methods in /lib
...
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2023-06-06 11:01:58 +07:00
702196dd86
Merge pull request #38478 from nextcloud/backport/38440/master
...
[master] fix: Catch Deadlock properly as execute throws Doctrine exceptions not our wrapped ones
2023-06-05 23:22:40 +07:00
9f1d497a0b
Merge pull request #38261 from fsamapoor/replace_strpos_calls_in_lib_private
...
Refactors "strpos" calls in lib/private to improve code readability.
2023-06-01 23:10:00 +07:00
9453c2d40d
fix: execute throws docrine exceptions not our wrapped ones
...
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2023-05-26 06:56:55 +07:00
8362eea14e
Merge pull request #38196 from nextcloud/fix/fix-32bits-freespace-and-sizes
...
Get rid of more int casts in file size manipulations
2023-05-15 17:05:46 +07:00
e7cc7653b8
Refactors "strpos" calls in lib/private to improve code readability.
...
Signed-off-by: Faraz Samapoor <fsamapoor@gmail.com>
2023-05-15 15:17:19 +07:00
ae525e1935
Fix return types in phpdoc
...
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2023-05-15 12:24:42 +07:00
e18f97fc95
Merge pull request #37709 from nextcloud/bugfix/deadlock
2023-05-13 13:13:04 +07:00
a90581b405
Get rid of more int casts in file size manipulations
...
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2023-05-11 12:46:16 +07:00
b6c034ac57
Merge pull request #37961 from nextcloud/poc/noid/systemtags-perf
...
SystemTags endpoint to return tags used by a user with meta data
2023-05-11 10:16:03 +07:00
2ea41dab93
repair -1 folder sizes for object store background scan
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-05-10 19:33:26 +07:00
df662f50bd
refactor: remove where specification from SELECT getter
...
- search constraints are now fully in control of
SystemTagsInFilesDetector::detectAssignedSystemTagsIn(), avoids
duplication of a WHERE statement
Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
2023-05-10 18:48:06 +07:00
dbfd2f936a
refactor: remove SystemTag logic from Folder into QuerySearchHelper
...
- adds OC\SystemTag\SystemTagsInFilesDetector where the search logic is
moved to
Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
2023-05-09 23:51:52 +07:00
2489a2d63a
feat: specify media type via url path: systemtags-current/$mediaType
...
- only the media part of the mime type can be search, but not the full
mime type. It can be added, should it become necessary.
- thus fixes previously hardcoded selector for image/ types
- also fixes a return type hint
- adds a return type hint
Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
2023-05-09 23:51:51 +07:00
6bff6a5667
PoC: SystemTags endpoint to return tags used by a user with meta data
...
Target case is photos app: when visiting the tags category, all systemtags
of the whole cloud are retrieved. In subequent steps the next tag is
requested until the browser view is filled with tag tiles (i.e. previews
are requested just as well).
With this approach, we incorpoate the dav search and look for user related
tags that are used by them, and already returns the statistics (number of
files tagged with the respective tag) as well as a file id for the purpose
to load the preview. This defaults to the file with the highest id.
Call:
curl -s -u 'user:password' \
'https://my.nc.srv/remote.php/dav/systemtags-current ' \
-X PROPFIND -H 'Accept: text/plain' \
-H 'Accept-Language: en-US,en;q=0.5' -H 'Depth: 1' \
-H 'Content-Type: text/plain;charset=UTF-8' \
--data @/home/doe/request-systemtag-props.xml
With request-systemtag-props.xml:
<?xml version="1.0" encoding="UTF-8"?>
<d:propfind xmlns:d="DAV:">
<d:prop xmlns:oc="http://owncloud.org/ns " xmlns:nc="http://nextcloud.org/ns ">
<oc:id/>
<oc:display-name/>
<oc:user-visible/>
<oc:user-assignable/>
<oc:can-assign/>
<nc:files-assigned/>
<nc:reference-fileid/>
</d:prop>
</d:propfind>
Example output:
…
<d:response>
<d:href>/master/remote.php/dav/systemtags/84</d:href>
<d:propstat>
<d:prop>
<oc:id>84</oc:id>
<oc:display-name>Computer</oc:display-name>
<oc:user-visible>true</oc:user-visible>
<oc:user-assignable>true</oc:user-assignable>
<oc:can-assign>true</oc:can-assign>
<nc:files-assigned>42</nc:files-assigned>
<nc:reference-fileid>924022</nc:reference-fileid>
</d:prop>
<d:status>HTTP/1.1 200 OK</d:status>
</d:propstat>
</d:response>
<d:response>
<d:href>/remote.php/dav/systemtags/97</d:href>
<d:propstat>
<d:prop>
<oc:id>97</oc:id>
<oc:display-name>Bear</oc:display-name>
<oc:user-visible>true</oc:user-visible>
<oc:user-assignable>true</oc:user-assignable>
<oc:can-assign>true</oc:can-assign>
<nc:files-assigned>1</nc:files-assigned>
<nc:reference-fileid>923422</nc:reference-fileid>
</d:prop>
<d:status>HTTP/1.1 200 OK</d:status>
</d:propstat>
</d:response>
…
Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
2023-05-09 23:51:48 +07:00
3bdd770129
fix: Check for wrapped retriable exceptions
...
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2023-05-02 19:42:11 +07:00
4953296b9e
Merge pull request #36857 from nextcloud/scan-repair-unencrypted-size
...
fix unencrypted_size for folders when scanning the filesystem with encryption enabled
2023-04-24 15:24:50 +07:00
0ac7230b7f
Merge pull request #37820 from nextcloud/parent-storage-mtime-ignore-error
...
ignore errors while trying to update parent storage_mtime
2023-04-21 19:11:27 +07:00
4b52dafbf9
ignore errors while trying to update parent storage_mtime
...
in the worst case this should only cause an extra rescan later
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-04-20 17:16:03 +07:00
b294edad80
Merge branch 'master' into enh/type-iconfig-getter-calls
...
Signed-off-by: Côme Chilliet <91878298+come-nc@users.noreply.github.com>
2023-04-20 16:52:38 +07:00
b1f352c614
improve handling of files we can't access in the scanner
...
instead of erroring, remove the items from the cache.
this situation can be triggered if a user has access to a file but looses it afterwards
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-04-05 14:44:55 +07:00
Côme Chilliet
John Molakvoæ
provokateurin
Robin Appelman
Anna Larch
Louis Chemineau
Daniel Kesselberg
Andy Scherzinger
Côme Chilliet
Josh
Christoph Wurst
Christoph Wurst
Maxence Lange
Andrew Summers
Thomas Citharel
Git'Fellow
Julius Härtl
Joas Schilling
Benjamin Gaussorgues
Hamid Dehnavi
Ferdinand Thiessen
Arthur Schiwon
Simon L
Faraz Samapoor