f-denkena/rclone
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
6,375 Commits 1 Branch 0 Tags
72227a01519f4cc9b0a1ecd390785c9d1d09d5d0
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick Craig-Wood 9f40cb114a Revert "jottacloud: always store username in config and use it to avoid initial api request" 
This reverts commit 9dbed02329.

See: #6309
 2022-07-25 18:23:09 +01:00
Nick Craig-Wood 7a24c173f6 build: disable revive linter pending a fix in golangci-lint 
The revive linter got extremely slow in golangci-lint 1.47.1 causing
the CI to time out.

Disable for the time being until it is fixed.

See: https://github.com/golangci/golangci-lint/issues/2997
 2022-07-20 23:07:20 +01:00
Nick Craig-Wood fb60aeddae Add Jordi Gonzalez Muñoz to contributors 2022-07-20 23:07:02 +01:00
Nick Craig-Wood 695736d1e4 Add Steve Kowalik to contributors 2022-07-20 23:07:02 +01:00
Nick Craig-Wood 440d0cd179 s3: fix --s3-no-head panic: reflect: Elem of invalid type s3.PutObjectInput 
In

22abd785eb s3: implement reading and writing of metadata #111

The reading information of objects was refactored to use the
s3.HeadObjectOutput structure.

Unfortunately the code branch with `--s3-no-head` was not tested
otherwise this panic would have been discovered.

This shows that this is path is not integration tested, so this adds a
new integration test.

Fixes #6322
 2022-07-18 23:38:50 +01:00
Nick Craig-Wood a047d30eca Add Yen Hu to contributors 2022-07-18 23:38:50 +01:00
Nick Craig-Wood 50f053cada dropbox: fix hang on quit with --dropbox-batch-mode off 
This problem was created by the fact that we are much more diligent
about calling Shutdown now, and the dropbox backend had a hang if the
batch mode was "off" in the Shutdown method.

See: https://forum.rclone.org/t/dropbox-lsjson-in-1-59-stuck-on-commiting-upload/31853
 2022-07-17 12:51:44 +01:00
Nick Craig-Wood 140af43c26 build: add 32 bit test runner to avoid problems like #6311 2022-07-14 20:13:03 +01:00
Nick Craig-Wood f467188876 Add Evan Spensley to contributors 2022-07-14 20:13:03 +01:00
Nick Craig-Wood b310490fa5 union: fix multiple files being uploaded when roots don't exist 
See: https://forum.rclone.org/t/union-backend-copying-to-all-remotes-while-it-shouldnt/31781
 2022-07-11 18:19:36 +01:00
Nick Craig-Wood 0ee0812a2b union: fix duplicated files when using directories with leading / 
See: https://forum.rclone.org/t/union-backend-copying-to-all-remotes-while-it-shouldnt/31781
 2022-07-11 18:19:36 +01:00
Nick Craig-Wood 55bbff6346 operations: add --server-side-across-configs global flag for any backend 2022-07-11 18:17:42 +01:00
Nick Craig-Wood 9c6cfc1ff0 combine: throw error if duplicate directory name is specified 
See: https://forum.rclone.org/t/v1-59-combine-qs/31814
 2022-07-10 15:40:30 +01:00
Nick Craig-Wood f753d7cd42 combine: fix docs showing remote= instead of upstreams= 
See: https://forum.rclone.org/t/v1-59-combine-qs/31814
 2022-07-10 15:34:48 +01:00
Nick Craig-Wood f5be1d6b65 Start v1.60.0-DEV development 2022-07-09 20:43:17 +01:00
Nick Craig-Wood 00a684d877 Version v1.59.0 2022-07-09 18:09:25 +01:00
Nick Craig-Wood 1c4ee2feee gcs: add --gcs-decompress flag to download gzip-encoded files 
By default these will be downloaded compressed.

This changes the default of the previous commit

2781f8e2f1 gcs: Fix download of "Content-Encoding: gzip" compressed objects

But will fit in better with the metadata framework when copying
gzip-encoded objects from backend to backend.
 2022-07-09 17:31:12 +01:00
Nick Craig-Wood 876f12f2c4 Add Ovidiu Victor Tatar to contributors 2022-07-09 12:35:59 +01:00
Nick Craig-Wood 6e9c1eebd9 Add Claudio Maradonna to contributors 2022-07-09 12:35:59 +01:00
Nick Craig-Wood 2e54b56a01 rcat: check checksums by default like copy does #6305 
Before this change we were calculating the checksum for an rcat
transfer but never checking it.

See: https://forum.rclone.org/t/optimize-rclone-on-raspberry-pi-4-8gb/31741
 2022-07-07 16:09:09 +01:00
Nick Craig-Wood 2515039e18 Add Lorenzo Maiorfi to contributors 2022-07-07 16:09:09 +01:00
Nick Craig-Wood a9c531b9eb Add Paul Norman to contributors 2022-07-07 16:09:09 +01:00
Nick Craig-Wood 0db50ecb2f Add zzr93 to contributors 2022-07-07 16:09:09 +01:00
Nick Craig-Wood 388da82762 Add Anthrazz to contributors 2022-07-07 16:09:09 +01:00
Nick Craig-Wood 06182a3443 s3: actually compress the payload for content-type gzip test 2022-07-04 09:42:49 +01:00
Nick Craig-Wood a58b482061 fstests: fix Metadata tests on remotes with additional config 2022-07-04 09:42:49 +01:00
Nick Craig-Wood accf91742c fs: add Type and FindFromFs to manage Fs and RegInfo 2022-07-04 09:42:49 +01:00
Nick Craig-Wood 7e7a8a95e9 hasher: support metadata 2022-06-29 17:30:37 +01:00
Nick Craig-Wood ed87ae51c0 union: support metadata 2022-06-29 17:30:37 +01:00
Nick Craig-Wood bf4a16ae30 crypt: support metadata 2022-06-29 17:30:37 +01:00
Nick Craig-Wood c198700812 compress: support metadata 2022-06-29 17:30:37 +01:00
Nick Craig-Wood 8c483daf85 combine: support metadata 2022-06-29 17:30:37 +01:00
Nick Craig-Wood ba5760ff38 chunker: mark as not supporting metadata 2022-06-29 17:30:37 +01:00
Nick Craig-Wood cd1735bb10 cache: mark as not supporting metadata 2022-06-29 17:30:37 +01:00
Nick Craig-Wood 866c873daa backend: allow wrapping backend tests to run in make quicktest 2022-06-29 17:30:37 +01:00
Nick Craig-Wood c556e98f49 local: add Metadata support #111 2022-06-29 14:29:36 +01:00
Nick Craig-Wood 22abd785eb s3: implement reading and writing of metadata #111 2022-06-29 14:29:36 +01:00
Nick Craig-Wood a692bd2cd4 s3: change metadata storage to normal map with lowercase keys 2022-06-29 14:29:36 +01:00
Nick Craig-Wood 776a083892 lsf: add metadata support with M flag 2022-06-29 14:29:36 +01:00
Nick Craig-Wood d823a38ce5 lsjson: add --metadata/-M flag 
Note that this removes the `-M` flag from `--encrypted` as it
conflicted with the global flag and adds it to `--metadata`.
 2022-06-29 14:29:36 +01:00
Nick Craig-Wood 78d52882ca fs: add --metadata/-M flag to control whether metadata is copied 2022-06-29 14:29:36 +01:00
Nick Craig-Wood c4451bc43a fs: add --metadata-set flag to specify metadata for uploads 2022-06-29 14:29:36 +01:00
Nick Craig-Wood 0652ec95db fs: implement MetadataInfo to show info about metadata in help and rc 
Info about this will appear in operations/fsinfo and in the backend
help (`rclone help backend s3`).
 2022-06-29 14:29:36 +01:00
Nick Craig-Wood 6a0e021dac fs: implement optional Metadata interface for Objects #111 
This implements integration tests for the feature also.
 2022-06-29 11:21:29 +01:00
Nick Craig-Wood 461d041c4d fstest: remove spurious contents return from PutTestContents and friends 2022-06-29 11:18:02 +01:00
Nick Craig-Wood 35f24d5b84 Add vyloy to contributors 2022-06-29 10:55:03 +01:00
Nick Craig-Wood 370c8fa220 Add mirekphd to contributors 2022-06-29 10:55:03 +01:00
Nick Craig-Wood 0fca4d2c86 Add buda to contributors 2022-06-29 10:55:03 +01:00
Nick Craig-Wood bc705e14d8 vfscache: fix fatal error: sync: unlock of unlocked mutex error 
This message is a double panic and was actually caused by an assertion
panic in:

vfs/vfscache/downloaders/downloaders.go

This is triggered by the code added relatively recently to fix a bug
with renaming files:

ec72432cec vfs: fix failed to _ensure cache internal error: downloaders is nil error

So it appears that item.o may be nil at this point.

This patch detects item.o being nil and fetches it again with NewObject.

Fixes #6190 Fixes #6235
 2022-06-21 14:28:53 +01:00
Nick Craig-Wood ea5bb79366 drive: document export for google apps scripts better 
This also adds some more mime types from the code.

See: https://forum.rclone.org/t/how-do-i-copy-google-apps-scripts-with-rclone/31373
 2022-06-21 14:28:53 +01:00
Nick Craig-Wood e95dff2fa1 drive: add backend commands exportformats and importformats for debugging 2022-06-21 14:28:53 +01:00
Nick Craig-Wood 99dfe1eeae Add Martin Czygan to contributors 2022-06-21 14:28:53 +01:00
Nick Craig-Wood ed92bf335d Add J-P Treen to contributors 2022-06-21 14:28:53 +01:00
Nick Craig-Wood 3d55b537c6 Add Caleb to contributors 2022-06-21 14:28:53 +01:00
Nick Craig-Wood d03fffdf8d Add eNV25 to contributors 2022-06-21 14:28:53 +01:00
Nick Craig-Wood 14e0396fcb test_all: allow internet archive backend more time 2022-06-18 15:13:18 +01:00
Nick Craig-Wood 100acc570a test_all: fix -clean so it works on remotes with paths 2022-06-18 15:10:09 +01:00
Nick Craig-Wood b9de37af80 test_all: Only run backend tests for Internet Archive as it is too slow 2022-06-17 16:52:30 +01:00
Nick Craig-Wood f7c36ce0f9 s3: unwrap SDK errors to reveal underlying errors on upload 
The SDK doesn't wrap errors in a Go standard way so they can't be
unwrapped and tested for - eg fatal error.

The code looks for a Serialization or RequestError and returns the
unwrapped underlying error if possible.

This fixes the fs/operations integration tests checking for fatal
errors being returned.
 2022-06-17 16:52:30 +01:00
Nick Craig-Wood f829ded456 Add Phil Shackleton to contributors 2022-06-17 16:52:30 +01:00
Nick Craig-Wood 2fac8fdde6 Add Scott Grimes to contributors 2022-06-17 16:52:30 +01:00
Nick Craig-Wood c85fbebce6 s3: simplify PutObject code to use the Request.SetStreamingBody method 
In this commit

e5974ac4b0 s3: use PutObject from the aws SDK to upload single part objects

rclone was made to upload objects to s3 using PUT requests rather than
using signed uploads.

However this change missed the fact that there is a supported way to
do this in the SDK using the SetStreamingBody method on the Request.

This therefore reverts a lot of the previous commit to do with making
an unsigned connection and other complication and uses the SDK
facility.
 2022-06-16 23:26:19 +01:00
Nick Craig-Wood e59801c69b Add Maciej Radzikowski to contributors 2022-06-16 23:26:19 +01:00
Nick Craig-Wood 5697dbc80f local: fix parsing of --local-nounc flag 2022-06-16 22:13:50 +01:00
Nick Craig-Wood 7d3648dc46 serve ftp: check --passive-port arguments are correct 
See: https://forum.rclone.org/t/serve-ftp-passive-port-validity-check/27458
 2022-06-16 22:13:50 +01:00
Nick Craig-Wood a6ca4b3817 test info: check file name lengths using 1,2,3,4 byte unicode characters 2022-06-16 22:13:50 +01:00
Nick Craig-Wood e57fe14b61 mount: log IO errors at ERROR level - fixes #6217 2022-06-16 22:13:50 +01:00
Nick Craig-Wood 115f1c2cc9 operations: speed up hash checking by aborting the other hash if first returns nothing 
This speeds up hash checks when a Hash() function returns "" - this
means that the hash can be canceled for the other side.

In the common case of local hash vs remote hash empty this saves a lot
of time.

See: https://forum.rclone.org/t/rclone-s3-backend-copy-is-2x-slower-than-aws-s3-cp/27321/9
 2022-06-16 22:13:50 +01:00
Nick Craig-Wood 5e4caa69ce local: make Hash function cancelable via context 2022-06-16 22:13:50 +01:00
Nick Craig-Wood e7483b40b3 fshttp: add --disable-http-keep-alives to disable HTTP Keep Alives 
See: https://forum.rclone.org/t/getting-rate-limited-before-advertised-limit-on-s3-compatible-object-storage/31010/
 2022-06-16 22:13:50 +01:00
Nick Craig-Wood fa48b880c2 s3: retry RequestTimeout errors 
See: https://forum.rclone.org/t/s3-failed-upload-large-files-bad-request-400/27695
 2022-06-16 22:13:50 +01:00
Nick Craig-Wood 4ac875a811 sync: fix --max-duration and --cutoff-mode soft 
Before this change using --max-duration and --cutoff-mode soft would
work like --cutoff-mode hard.

This bug was introduced in this commit which made transfers be
cancelable - before that transfers couldn't be canceled.

122a47fba6 accounting: Allow transfers to be canceled with context #3257

This change adds the timeout to the input context for reading files
rather than the transfer context so the files transfers themselves
aren't canceled if --cutoff-mode soft is in action.

This also adds a test for cutoff mode soft and max duration which was
missing.

See: https://forum.rclone.org/t/max-duration-and-retries-not-working-correctly/27738
 2022-06-16 22:13:50 +01:00
Nick Craig-Wood 3f61869179 cmount: add tracing for *xattr FUSE callbacks 
See: https://github.com/winfsp/cgofuse/issues/66
See: https://forum.rclone.org/t/cannot-copy-files-to-mounted-azure-storage-windows/30092
 2022-06-16 22:13:50 +01:00
Nick Craig-Wood 60d87185e1 sftp: add --sftp-set-env option to set environment variables 
Fixes #6094
 2022-06-16 22:13:50 +01:00
Nick Craig-Wood 78120d40d9 sftp: add --sftp-concurrency to improve high latency transfers 
See: https://forum.rclone.org/t/increasing-sftp-transfer-speed/29928
 2022-06-16 22:13:50 +01:00
Nick Craig-Wood 95e0934755 sftp: add --sftp-chunk-size to control packets sizes for high latency links 
See: https://forum.rclone.org/t/increasing-sftp-transfer-speed/29928
 2022-06-16 22:13:50 +01:00
Nick Craig-Wood 1651429041 union: add min_free_space option for lfs/eplfs policies - fixes #6071 2022-06-16 22:13:50 +01:00
Nick Craig-Wood 29e37749b3 union: fix eplus policy to select correct entry for existing files #6071 2022-06-16 22:13:50 +01:00
Nick Craig-Wood 1e1af46a12 union: fix get free space for remotes which don't support it #6071 
Before this fix GetFreeSpace returned math.MaxInt64 for remotes which
don't support reading free space, however this is used in various
comparison routines as a too large value, meaning that remotes of size
math.MaxInt64 were never being selected.

This fixes GetFreeSpace to return math.MaxInt64 - 1 so then can be selected.

It also fixes GetUsedSpace the same way however as the default for not
supported was 0 this was very unlikely to have ever caused a problem.
 2022-06-16 22:13:50 +01:00
Nick Craig-Wood 1d2fe0d856 union: enable passing of options to upstreams and policies #6071 
This factors out the options into a sub package so they can be passed
to upstreams and used in policies.
 2022-06-16 22:13:50 +01:00
Nick Craig-Wood 4f94b27800 check: implement --no-traverse and --no-unicode-normalization 
See: https://forum.rclone.org/t/rclone-check-head-or-list-object-from-source/30400
 2022-06-16 22:13:50 +01:00
Nick Craig-Wood 4d72abf389 dropbox: fix nil pointer exception on dropbox impersonate user not found 
Fixes #6139
 2022-06-16 22:13:50 +01:00
Nick Craig-Wood 411013dbdc drive: add --drive-resource-key for accessing link-shared files 2022-06-16 22:13:50 +01:00
Nick Craig-Wood e87e331f4c drive: make --drive-shared-with-me work with shared drives 
Fixes #6247
 2022-06-16 22:13:50 +01:00
Nick Craig-Wood a0cb3bbd02 mount: allow tests to run on CI 2022-06-16 16:48:09 +01:00
Nick Craig-Wood 4a382c09ec mount: run tests in a subprocess to fix deadlock - fixes #3259 
Before this change we ran the tests and the mount in the same process.
This could cause deadlocks and often did, and made the mount tests
very unreliable.

This fixes the problem by running the mount in a seperate process and
commanding it via a pipe over stdin/stdout.
 2022-06-16 16:48:09 +01:00
Nick Craig-Wood 626a416ff8 vfs: factor out the VFS option initialization for re-use #3259 2022-06-16 16:48:09 +01:00
Nick Craig-Wood 6c832a72ee Add CrossR to contributors 2022-06-16 16:48:09 +01:00
Nick Craig-Wood c390098262 Add Sven Gerber to contributors 2022-06-16 16:48:09 +01:00
Nick Craig-Wood 41f3ceb67d Add Art M. Gallagher to contributors 2022-06-16 16:48:09 +01:00
Nick Craig-Wood 592358148d Add m00594701 to contributors 2022-06-16 16:48:09 +01:00
Nick Craig-Wood cfe0911e0d sync: fix tests for overlapping with filter 
In commit

3ccf222acb sync: overlap check is now filter-sensitive

The tests were attempting to write invalid objects on some backends
due to a leading / on the object name.

This fix also adds a few more test cases and makes sure the tests can
be run individually.
 2022-06-09 14:26:43 +01:00
Nick Craig-Wood 7c1f2d7c84 dirtree: fix tests with -fast-list 
In commit

da404dc0f2 sync,copy: Fix --fast-list --create-empty-src-dirs and --exclude

The fix caused DirTree.AddDir to be called with the root directory.
This in turn caused a spurious directory entry in the DirTree which
caused tests with the -fast-list flag to fail with directory not found
errors.
 2022-06-09 14:26:43 +01:00
Nick Craig-Wood bb6edb3c39 build: update dependencies 
Also:

- azureblob: fix compile after API change in upstream library
 2022-06-08 18:29:42 +01:00
Nick Craig-Wood 6c2331ffd7 mount: skip tests on CI even if >= 2 processors 2022-06-08 18:29:42 +01:00
Nick Craig-Wood 08a897424b Add Noah Hsu to contributors 2022-06-08 18:29:42 +01:00
Nick Craig-Wood 4b358ff43b combine: backend to combine multiple remotes in one directory tree 
Fixes #5600
 2022-06-08 14:57:25 +01:00
Nick Craig-Wood e58d75e4d7 drive: make backend config -o config add a combined AllDrives remote 
This adjusts

    rclone backend drives -o config drive:

So that it also emits a config section called `AllDrives` which uses
the combine backend to make a backend which combines all the shared
drives into one.

It also makes sure that all the shared drive names are valid rclone
config names, deduplicating if necessary.

Fixes #4506
 2022-06-08 14:57:25 +01:00
Nick Craig-Wood fb58737142 fstests: check for wrapped errors in ListR test 2022-06-08 14:57:25 +01:00
Nick Craig-Wood 26db80c270 ftp: revert to upstream github.com/jlaffaye/ftp from our fork 
...now all of our patches have been merged #5810
 2022-06-08 11:58:32 +01:00