f-denkena/rclone
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
3,427 Commits 1 Branch 0 Tags
ae9c0e56c893869cacf739a12a68453d89046d6d
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick Craig-Wood ae9c0e56c8 operations: run hashing operations in parallel #3419 
Before this change for a post copy Hash check we would run the hashes sequentially.

Now we run the hashes in parallel for a useful speedup.

Note that this refactors the hash check in Copy to use the standard
hash checking routine.
 2019-08-14 15:07:38 +01:00
Nick Craig-Wood 402aaca7fe local: don't calculate any hashes by default #3419 
Before this change, if the caller didn't provide a hint, we would
calculate all hashes for reads and writes.

The new whirlpool hash is particularly expensive and that has become noticeable.

Now we don't calculate any hashes on upload or download unless hints are provided.

This means that some operations may run slower and these will need to be discovered!

It does not affect anything calling operations.Copy which already puts
the corrects hints in.
 2019-08-14 15:07:38 +01:00
Nick Craig-Wood 106cf1852d Add ginvine to contributors 2019-08-14 13:40:15 +01:00
Nick Craig-Wood 50b8f15b5d Add another email for Laura Hausmann to contributors 2019-08-14 13:40:07 +01:00
Nick Craig-Wood 23a0332185 config: don't offer hidden values for editing in the config - fixes #3416 2019-08-14 08:40:22 +01:00
Nick Craig-Wood 8247c8a6af rc: add anchor tags to the docs so links are consistent 2019-08-13 11:57:01 +01:00
Nick Craig-Wood 535f5f3c99 rc: fix --loopback with rc/list and others 
Before this change `rclone rc --loopback` would give the error "bad
JSON".

This was because the output of the `rc/list` command was not serialzed
through JSON.

This serializes it through JSON and fixes that (and probably other)
command.
 2019-08-13 11:51:16 +01:00
Nick Craig-Wood 7f7946564d error: make "bad record MAC" a retriable error - Fixes #3338 
The error "tls: bad record MAC" is very likely to be caused by
hardware issues.  It indicates that a packet got corrupted somewhere.

As a work around, this change treats it as retriable error which
allows the chunk to get retried and the transfer to continue.
 2019-08-12 20:37:10 +01:00
Nick Craig-Wood 5e0a30509c http: add --http-headers flag for setting arbitrary headers 2019-08-12 18:04:24 +01:00
Nick Craig-Wood cd7ca2a320 googlephotos: implement optional features UserInfo and Disconnect 
As part of rclone's UX review it was required that rclone had a means
of disconnecting from google photos and showing which user is
connected.
 2019-08-12 13:49:23 +01:00
Nick Craig-Wood a808e98fe1 config: add reconnect, userinfo and disconnect subcommands. 
- reconnect runs through the oauth flow again.
- userinfo shows the connected user info if available
- disconnect revokes the token
 2019-08-12 13:49:23 +01:00
Nick Craig-Wood 3ebcb555f4 fs: add optional features UserInfo and Disconnect 2019-08-12 13:49:23 +01:00
Nick Craig-Wood a1263e70cf premiumizeme: new backend for premiumize.me - Fixes #3063 2019-08-10 19:17:51 +01:00
Nick Craig-Wood f47e5220a2 Add Abhinav Sharma to contributors 2019-08-10 17:31:25 +01:00
Nick Craig-Wood 3ecbd603ab rc: move job expire flags to rc to fix initalization problem 
See: https://forum.rclone.org/t/rc-rc-job-expire-interval-bug/11188

rclone was ignoring the --rc-job-expire-duration and --rc-job-interval
flags.  This turned out to be an initialization order problem and was
fixed by moving those flags out of global config into rc config.
 2019-08-10 17:12:22 +01:00
Nick Craig-Wood 0693deea1c rc: fix unmarshalable http.AuthFn in options and put in test for marshalability 2019-08-10 16:22:17 +01:00
Nick Craig-Wood 99eaa76dc8 Add Macavirus to contributors 2019-08-10 14:13:24 +01:00
Nick Craig-Wood 7d85ccb11e fs/cache: test for fix cached values pointing to files #3424 2019-08-10 08:39:56 +01:00
Nick Craig-Wood 5195075677 Add Michał Matczuk to contributors 2019-08-08 23:42:03 +01:00
Nick Craig-Wood 6f87267b34 accounting: fix locking in Transfer to avoid deadlock with --progress 
Before this change, using -P occasionally deadlocked on the transfer
mutex and the stats mutex since they call each other via the progress
printing.

This is fixed by shortening the locking windows and converting the
mutex to a RW mutex.
 2019-08-08 15:46:46 +01:00
Nick Craig-Wood 9d1fb2f4e7 Revert "cmd: shorten the locking window when using --progress to avoid deadlock" 
This reverts commit fdef567da6.

The problem turned out to be elsewhere.
 2019-08-08 15:19:41 +01:00
Nick Craig-Wood 99b3154abd Revert "filter: Add BoundedRecursion method" 
This reverts commit 047f00a411.

It turns out that BoundedRecursion is the wrong thing to measure.
 2019-08-08 14:15:50 +01:00
Nick Craig-Wood 6c38bddf3e walk: fix listing with filters listing whole remote 
Prior to this fix, a request such as

    rclone lsf -R --include "/dir/**" remote:

Would use ListR which is very inefficient as it lists the whole remote
for one directory.

This changes it to use recursive walking if the filters imply any
directory filtering.  So `--include *.jpg` and `--exclude *.jpg` will
still use ListR wheras `--include "/dir/**` will not.
 2019-08-08 14:15:50 +01:00
Nick Craig-Wood a00a0471a8 filter: Add UsesDirectoryFilters method 2019-08-08 14:15:50 +01:00
Nick Craig-Wood 9e81fc343e swift: fix upload when using no_chunk to return the correct size 
When using the VFS with swift and --swift-no-chunk, PutStream was
returning objects with size -1 which was causing corrupted transfer
messages.

This was fixed by counting the bytes transferred in a streamed file
and updating the metadata with that.
 2019-08-08 12:41:46 +01:00
Nick Craig-Wood fdef567da6 cmd: shorten the locking window when using --progress to avoid deadlock 
Before this change, using -P occasionally deadlocked on the progress
mutex and the stats mutex since they call each other.

This is fixed by shortening the locking window in the progress routine
so as not to include the stats calculation.
 2019-08-08 12:37:50 +01:00
Nick Craig-Wood d377842395 vfs: make write without cache more efficient 
This updates the out of sequence write code to be more efficient using
a conditional lock with a timeout.
 2019-08-08 12:37:50 +01:00
Nick Craig-Wood c014b2e66b rcat: fix slowdown on systems with multiple hashes 
Before this fix rclone calculated all the hashes on transfer.  This
was particularly slow for the local backend.

After the fix we just calculate one hash which is enough for data
integrity.
 2019-08-08 12:37:50 +01:00
Nick Craig-Wood 62b769a0a7 serve sftp: fix spurious debugs on server close 2019-08-08 12:37:50 +01:00
Nick Craig-Wood 84b5da089e serve sftp: fix detection of whether server is authorized 2019-08-08 12:37:50 +01:00
Nick Craig-Wood d0c65b4c5e copyurl: fix copying files that return HTTP errors 2019-08-07 22:29:44 +01:00
Nick Craig-Wood e502be475a azureblob/b2/dropbox/gcs/koofr/qingstor/s3: fix 0 length files 
In 0386d22cc9 we introduced a test for 0 length files read the
way mount does.

This test failed on these backends which we fix up here.
 2019-08-06 15:18:08 +01:00
Nick Craig-Wood 5065c422b4 lib/random: unify random string generation into random.String 
This was factored from fstest as we were including the testing
enviroment into the main binary because of it.

This was causing opening the browser to fail because of 8243ff8bc8.
 2019-08-06 12:44:08 +01:00
Nick Craig-Wood 72d5b11d1b serve restic: rename test file to avoid it being linked into main binary 2019-08-06 12:42:52 +01:00
Nick Craig-Wood 526a3347ac rcd: Fix permissions problems on cache directory with web gui download 2019-08-06 12:06:57 +01:00
Nick Craig-Wood 23910ba53b servetest: add tests for --auth-proxy 2019-08-06 11:43:42 +01:00
Nick Craig-Wood ee7101e6af serve: factor out common testing parts for ftp, sftp and webdav tests 2019-08-06 11:43:42 +01:00
Nick Craig-Wood 36c1b37dd9 serve webdav: support --auth-proxy 2019-08-06 11:43:42 +01:00
Nick Craig-Wood 72782bdda6 serve ftp: implement --auth-proxy 2019-08-06 11:43:42 +01:00
Nick Craig-Wood b94eef16c1 serve ftp: refactor to bring into line with other serve commands 2019-08-06 11:43:42 +01:00
Nick Craig-Wood d75fbe4852 serve sftp: implement auth proxy 2019-08-06 11:43:42 +01:00
Nick Craig-Wood e6ab237fcd serve: add auth proxy infrastructure 2019-08-06 11:43:42 +01:00
Nick Craig-Wood a7eec91d69 vfs: add Fs() method to return underlying fs.Fs 2019-08-06 11:43:42 +01:00
Nick Craig-Wood b3e94b018c cache: factor fs cache into lib/cache 2019-08-06 11:43:42 +01:00
Nick Craig-Wood ca0e9ea55d build: add Azure Pipelines build status to README 2019-08-06 10:46:36 +01:00
Nick Craig-Wood 53e3c2e263 build: add azure pipelines build 2019-08-06 10:31:32 +01:00
Nick Craig-Wood 02eb747d71 serve http/webdav/restic: implement --prefix - fixes #3398 
--prefix enables the servers to serve from a non root prefix.  This
enables easier proxying.
 2019-08-06 10:30:48 +01:00
Nick Craig-Wood a9438cf364 build: add .gitattributes to mark generated files 
This makes sure that GitHub ignores the auto generated documentation
files for language detection and diffs.

See: https://github.com/github/linguist#overrides for more info
 2019-08-04 15:20:15 +01:00
Nick Craig-Wood 5ef3c988eb bin: add script to test all commits compile for git bisect 2019-08-04 13:29:59 +01:00
Nick Craig-Wood 78150e82a2 docs: update bugs and limitations document 2019-08-04 12:33:39 +01:00
Nick Craig-Wood 6f0cc51eeb Add Chaitanya Bankanhal to contributors 2019-08-04 12:33:39 +01:00
Nick Craig-Wood 0386d22cc9 vfs: add test for 0 length files read in the way mount does 2019-08-03 18:25:44 +01:00
Nick Craig-Wood 0be14120e4 swift: use FixRangeOption to fix 0 length files via the VFS 2019-08-03 18:25:44 +01:00
Nick Craig-Wood 95af1f9ccf fs: fix FixRangeOption so it works with 0 length files 2019-08-03 18:25:44 +01:00
Nick Craig-Wood 629b7eacd8 b2: fix integration tests after accounting changes 
In 53a1a0e3ef we started returning non nil from NewObject when
an object isn't found.  This breaks the integration tests and the API
expected of a backend.

This fixes it.
 2019-08-03 13:30:31 +01:00
Nick Craig-Wood 5be968c0ca drive: update API for teamdrive use - fixes #3348 2019-08-02 16:06:23 +01:00
Nick Craig-Wood f1a687c540 Add justina777 to contributors 2019-08-02 15:57:25 +01:00
Nick Craig-Wood c2635e39cc build: fix appveyor secure variables after project move 2019-07-28 22:46:26 +01:00
Nick Craig-Wood 8c511ec9cd docs: fix star count on website 2019-07-28 20:58:21 +01:00
Nick Craig-Wood ac0dce78d0 cmd: fix up stats printing on macOS after accounting change 2019-07-28 20:38:20 +01:00
Nick Craig-Wood f347514f62 build: fix up CI and CI badges after repo move 2019-07-28 20:07:04 +01:00
Nick Craig-Wood 57d5de6fba build: fix up package paths after repo move 
git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g'
goimports -w `find . -name \*.go`
 2019-07-28 18:47:38 +01:00
Nick Craig-Wood 2d561b51db Add EliEron to contributors 2019-07-28 12:33:21 +01:00
Nick Craig-Wood 9241a93c2d Add justinalin to contributors 2019-07-28 12:33:21 +01:00
Nick Craig-Wood 9af0a704af Add Paul Millar to contributors 2019-07-24 20:34:39 +01:00
Nick Craig-Wood 691e5ae5f0 Add Yi FU to contributors 2019-07-24 20:34:39 +01:00
Nick Craig-Wood 5a44bafa4e fstest: add fs.ErrorCantShareDirectories for backends which can only share files 2019-07-24 20:34:29 +01:00
Nick Craig-Wood 8fdce31700 config: Fix hiding of options from the configurator 2019-07-24 20:34:29 +01:00
Nick Craig-Wood 493dfb68fd opendrive: refactor to use existing lib/rest facilities for uploads 
This also checks the return of the call to make sure the number of
bytes written was as expected.
 2019-07-24 20:34:29 +01:00
Nick Craig-Wood 71587344c6 lib/rest: allow Form upload with no file to upload 2019-07-24 20:34:29 +01:00
Nick Craig-Wood 266600dba7 build: reduce parallelism in cross compile to reduce memory and fix Travis 
Before this change Travis builds were running out of memory when cross
compiling all the OSes.
 2019-07-22 17:10:26 +01:00
Nick Craig-Wood 1f1ab179a6 webdav: refresh token when it expires with --webdav-bearer-token-command 
Fixes #2380
 2019-07-22 16:01:55 +01:00
Nick Craig-Wood c642531a1e webdav: add --webdav-bearer-token-command - fixes #2380 
This can be used with oidc-agent to get a bearer token
 2019-07-22 15:59:54 +01:00
Nick Craig-Wood 5433021e8b drive: fix server side copy of big files 
Before this change rclone was sending a MimeType in the requests for
server side Move and Copy.

The conjecture is that if you attempt to set the MimeType to something
different in a Copy then Google Drive has to do an actual copy of the
file data.  This takes a very long time (since it is large) and fails
after a 90s timeout.

After the change we no longer set the MimeType in Move or Copy and the
copies happen instantly and correctly.

Many thanks to @darthShadow for discovering that this was causing the
problem.

Fixes #3070
Fixes #3033
Fixes #3300
Fixes #3155
 2019-07-05 10:49:19 +01:00
Nick Craig-Wood c9f77719e4 b2: enable server side copy to copy between buckets - fixes #3303 2019-07-05 10:07:05 +01:00
Nick Craig-Wood 3cd63a00be googlephotos: fix configuration walkthrough 2019-07-04 15:19:59 +01:00
Nick Craig-Wood d7016866e0 googlephotos: fix creation of duplicated albums 
Also make sure we don't list the albums twice
 2019-07-04 13:45:52 +01:00
Nick Craig-Wood 1510e12659 build: move race test to go modules build 
In an attempt to even out the build times.
 2019-07-03 12:07:29 +01:00
Nick Craig-Wood ede03258bc build: use go modules proxy when building modules 2019-07-03 12:07:29 +01:00
Nick Craig-Wood 7fcbb47b1c build: split other OS build into a separate builder 
This is in order to make longest build (the Linux build) quicker
 2019-07-03 12:07:29 +01:00
Nick Craig-Wood 9cafeeb4b6 dirtree: make tests more reliable 2019-07-02 16:29:40 +01:00
Nick Craig-Wood a1cfe61ffd googlephotos: Backend for accessing Google Photos #369 2019-07-02 15:26:55 +01:00
Nick Craig-Wood 5eebbaaac4 test_all: add tests parameter to limit which tests to run for a backend 2019-07-02 15:26:55 +01:00
Nick Craig-Wood bc70bff125 fs/dirtree: factor DirTree out of fs/walk and add tests 2019-07-02 15:26:55 +01:00
Nick Craig-Wood cf15b88efa build: make explicit which matrix items we will deploy 2019-07-02 14:13:46 +01:00
Nick Craig-Wood dcaee0016a build: add a builder for building with go modules 2019-07-02 12:33:03 +01:00
Nick Craig-Wood 387b496d1e operations: fix tests TestMoveFileBackupDir and TestCopyFileBackupDir again 
Commit 734f504d5f wasn't tested properly and had a typo which
caused it not to build :-(
 2019-07-02 12:22:29 +01:00
Nick Craig-Wood 734f504d5f operations: fix tests TestMoveFileBackupDir and TestCopyFileBackupDir 
..so they don't run on backends which can't move or copy.
 2019-07-02 10:46:49 +01:00
Nick Craig-Wood 5df5a3b78e vendor: tidy go.mod and go.sum - fixes #3317 2019-07-02 09:47:00 +01:00
Nick Craig-Wood 37c1144b46 Add Russell Davis to contributors 2019-07-02 07:57:18 +01:00
Nick Craig-Wood 8d116ba0c9 Add Matti Niemenmaa to contributors 2019-07-02 07:57:18 +01:00
Nick Craig-Wood 1be1fc073e Add AbelThar to contributors 2019-07-01 12:09:35 +01:00
Nick Craig-Wood af2596f98b Add yparitcher to contributors 2019-07-01 11:13:53 +01:00
Nick Craig-Wood 61fb326a80 Add full name for Laura Hausmann 2019-07-01 10:53:47 +01:00
Nick Craig-Wood 6bae3595a8 Add Laura to contributors 2019-06-30 20:07:35 +01:00
Nick Craig-Wood 9ed81ac451 vfs: fix tests for backends which can't upload 0 length files 2019-06-30 18:35:01 +01:00
Nick Craig-Wood 3245c0ae0d fstests: add integration test for uploading empty files 
This tests a remote can upload empty files.  If the remote can't
upload empty files it should return fs.ErrorCantUploadEmptyFiles.
 2019-06-30 18:35:01 +01:00
Nick Craig-Wood 07e2c3a50f b2: fix nil pointer error introduced by context propagation patch 
For some reason f78cd1e043 introduced an unrelated change -
perhaps a merge error.  Removing this change fixes the nil pointer
problem.
 2019-06-28 22:38:41 +01:00
Nick Craig-Wood d61ba7ef78 vendor: update all dependencies 2019-06-27 13:52:32 +01:00
Nick Craig-Wood b221d79273 Add nguyenhuuluan434 to contributors 2019-06-27 13:28:52 +01:00