f-denkena/rclone
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
3,387 Commits 1 Branch 0 Tags
c014b2e66bb75730891548fb701aa51e3334f6dd
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick Craig-Wood c014b2e66b rcat: fix slowdown on systems with multiple hashes 
Before this fix rclone calculated all the hashes on transfer.  This
was particularly slow for the local backend.

After the fix we just calculate one hash which is enough for data
integrity.
 2019-08-08 12:37:50 +01:00
Nick Craig-Wood 62b769a0a7 serve sftp: fix spurious debugs on server close 2019-08-08 12:37:50 +01:00
Nick Craig-Wood 84b5da089e serve sftp: fix detection of whether server is authorized 2019-08-08 12:37:50 +01:00
Nick Craig-Wood d0c65b4c5e copyurl: fix copying files that return HTTP errors 2019-08-07 22:29:44 +01:00
Nick Craig-Wood e502be475a azureblob/b2/dropbox/gcs/koofr/qingstor/s3: fix 0 length files 
In 0386d22cc9 we introduced a test for 0 length files read the
way mount does.

This test failed on these backends which we fix up here.
 2019-08-06 15:18:08 +01:00
Nick Craig-Wood 5065c422b4 lib/random: unify random string generation into random.String 
This was factored from fstest as we were including the testing
enviroment into the main binary because of it.

This was causing opening the browser to fail because of 8243ff8bc8.
 2019-08-06 12:44:08 +01:00
Nick Craig-Wood 72d5b11d1b serve restic: rename test file to avoid it being linked into main binary 2019-08-06 12:42:52 +01:00
Nick Craig-Wood 526a3347ac rcd: Fix permissions problems on cache directory with web gui download 2019-08-06 12:06:57 +01:00
Nick Craig-Wood 23910ba53b servetest: add tests for --auth-proxy 2019-08-06 11:43:42 +01:00
Nick Craig-Wood ee7101e6af serve: factor out common testing parts for ftp, sftp and webdav tests 2019-08-06 11:43:42 +01:00
Nick Craig-Wood 36c1b37dd9 serve webdav: support --auth-proxy 2019-08-06 11:43:42 +01:00
Nick Craig-Wood 72782bdda6 serve ftp: implement --auth-proxy 2019-08-06 11:43:42 +01:00
Nick Craig-Wood b94eef16c1 serve ftp: refactor to bring into line with other serve commands 2019-08-06 11:43:42 +01:00
Nick Craig-Wood d75fbe4852 serve sftp: implement auth proxy 2019-08-06 11:43:42 +01:00
Nick Craig-Wood e6ab237fcd serve: add auth proxy infrastructure 2019-08-06 11:43:42 +01:00
Nick Craig-Wood a7eec91d69 vfs: add Fs() method to return underlying fs.Fs 2019-08-06 11:43:42 +01:00
Nick Craig-Wood b3e94b018c cache: factor fs cache into lib/cache 2019-08-06 11:43:42 +01:00
Nick Craig-Wood ca0e9ea55d build: add Azure Pipelines build status to README 2019-08-06 10:46:36 +01:00
Nick Craig-Wood 53e3c2e263 build: add azure pipelines build 2019-08-06 10:31:32 +01:00
Nick Craig-Wood 02eb747d71 serve http/webdav/restic: implement --prefix - fixes #3398 
--prefix enables the servers to serve from a non root prefix.  This
enables easier proxying.
 2019-08-06 10:30:48 +01:00
Nick Craig-Wood a9438cf364 build: add .gitattributes to mark generated files 
This makes sure that GitHub ignores the auto generated documentation
files for language detection and diffs.

See: https://github.com/github/linguist#overrides for more info
 2019-08-04 15:20:15 +01:00
Nick Craig-Wood 5ef3c988eb bin: add script to test all commits compile for git bisect 2019-08-04 13:29:59 +01:00
Nick Craig-Wood 78150e82a2 docs: update bugs and limitations document 2019-08-04 12:33:39 +01:00
Nick Craig-Wood 6f0cc51eeb Add Chaitanya Bankanhal to contributors 2019-08-04 12:33:39 +01:00
Nick Craig-Wood 0386d22cc9 vfs: add test for 0 length files read in the way mount does 2019-08-03 18:25:44 +01:00
Nick Craig-Wood 0be14120e4 swift: use FixRangeOption to fix 0 length files via the VFS 2019-08-03 18:25:44 +01:00
Nick Craig-Wood 95af1f9ccf fs: fix FixRangeOption so it works with 0 length files 2019-08-03 18:25:44 +01:00
Nick Craig-Wood 629b7eacd8 b2: fix integration tests after accounting changes 
In 53a1a0e3ef we started returning non nil from NewObject when
an object isn't found.  This breaks the integration tests and the API
expected of a backend.

This fixes it.
 2019-08-03 13:30:31 +01:00
Nick Craig-Wood 5be968c0ca drive: update API for teamdrive use - fixes #3348 2019-08-02 16:06:23 +01:00
Nick Craig-Wood f1a687c540 Add justina777 to contributors 2019-08-02 15:57:25 +01:00
Nick Craig-Wood c2635e39cc build: fix appveyor secure variables after project move 2019-07-28 22:46:26 +01:00
Nick Craig-Wood 8c511ec9cd docs: fix star count on website 2019-07-28 20:58:21 +01:00
Nick Craig-Wood ac0dce78d0 cmd: fix up stats printing on macOS after accounting change 2019-07-28 20:38:20 +01:00
Nick Craig-Wood f347514f62 build: fix up CI and CI badges after repo move 2019-07-28 20:07:04 +01:00
Nick Craig-Wood 57d5de6fba build: fix up package paths after repo move 
git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g'
goimports -w `find . -name \*.go`
 2019-07-28 18:47:38 +01:00
Nick Craig-Wood 2d561b51db Add EliEron to contributors 2019-07-28 12:33:21 +01:00
Nick Craig-Wood 9241a93c2d Add justinalin to contributors 2019-07-28 12:33:21 +01:00
Nick Craig-Wood 9af0a704af Add Paul Millar to contributors 2019-07-24 20:34:39 +01:00
Nick Craig-Wood 691e5ae5f0 Add Yi FU to contributors 2019-07-24 20:34:39 +01:00
Nick Craig-Wood 5a44bafa4e fstest: add fs.ErrorCantShareDirectories for backends which can only share files 2019-07-24 20:34:29 +01:00
Nick Craig-Wood 8fdce31700 config: Fix hiding of options from the configurator 2019-07-24 20:34:29 +01:00
Nick Craig-Wood 493dfb68fd opendrive: refactor to use existing lib/rest facilities for uploads 
This also checks the return of the call to make sure the number of
bytes written was as expected.
 2019-07-24 20:34:29 +01:00
Nick Craig-Wood 71587344c6 lib/rest: allow Form upload with no file to upload 2019-07-24 20:34:29 +01:00
Nick Craig-Wood 266600dba7 build: reduce parallelism in cross compile to reduce memory and fix Travis 
Before this change Travis builds were running out of memory when cross
compiling all the OSes.
 2019-07-22 17:10:26 +01:00
Nick Craig-Wood 1f1ab179a6 webdav: refresh token when it expires with --webdav-bearer-token-command 
Fixes #2380
 2019-07-22 16:01:55 +01:00
Nick Craig-Wood c642531a1e webdav: add --webdav-bearer-token-command - fixes #2380 
This can be used with oidc-agent to get a bearer token
 2019-07-22 15:59:54 +01:00
Nick Craig-Wood 5433021e8b drive: fix server side copy of big files 
Before this change rclone was sending a MimeType in the requests for
server side Move and Copy.

The conjecture is that if you attempt to set the MimeType to something
different in a Copy then Google Drive has to do an actual copy of the
file data.  This takes a very long time (since it is large) and fails
after a 90s timeout.

After the change we no longer set the MimeType in Move or Copy and the
copies happen instantly and correctly.

Many thanks to @darthShadow for discovering that this was causing the
problem.

Fixes #3070
Fixes #3033
Fixes #3300
Fixes #3155
 2019-07-05 10:49:19 +01:00
Nick Craig-Wood c9f77719e4 b2: enable server side copy to copy between buckets - fixes #3303 2019-07-05 10:07:05 +01:00
Nick Craig-Wood 3cd63a00be googlephotos: fix configuration walkthrough 2019-07-04 15:19:59 +01:00
Nick Craig-Wood d7016866e0 googlephotos: fix creation of duplicated albums 
Also make sure we don't list the albums twice
 2019-07-04 13:45:52 +01:00
Nick Craig-Wood 1510e12659 build: move race test to go modules build 
In an attempt to even out the build times.
 2019-07-03 12:07:29 +01:00
Nick Craig-Wood ede03258bc build: use go modules proxy when building modules 2019-07-03 12:07:29 +01:00
Nick Craig-Wood 7fcbb47b1c build: split other OS build into a separate builder 
This is in order to make longest build (the Linux build) quicker
 2019-07-03 12:07:29 +01:00
Nick Craig-Wood 9cafeeb4b6 dirtree: make tests more reliable 2019-07-02 16:29:40 +01:00
Nick Craig-Wood a1cfe61ffd googlephotos: Backend for accessing Google Photos #369 2019-07-02 15:26:55 +01:00
Nick Craig-Wood 5eebbaaac4 test_all: add tests parameter to limit which tests to run for a backend 2019-07-02 15:26:55 +01:00
Nick Craig-Wood bc70bff125 fs/dirtree: factor DirTree out of fs/walk and add tests 2019-07-02 15:26:55 +01:00
Nick Craig-Wood cf15b88efa build: make explicit which matrix items we will deploy 2019-07-02 14:13:46 +01:00
Nick Craig-Wood dcaee0016a build: add a builder for building with go modules 2019-07-02 12:33:03 +01:00
Nick Craig-Wood 387b496d1e operations: fix tests TestMoveFileBackupDir and TestCopyFileBackupDir again 
Commit 734f504d5f wasn't tested properly and had a typo which
caused it not to build :-(
 2019-07-02 12:22:29 +01:00
Nick Craig-Wood 734f504d5f operations: fix tests TestMoveFileBackupDir and TestCopyFileBackupDir 
..so they don't run on backends which can't move or copy.
 2019-07-02 10:46:49 +01:00
Nick Craig-Wood 5df5a3b78e vendor: tidy go.mod and go.sum - fixes #3317 2019-07-02 09:47:00 +01:00
Nick Craig-Wood 37c1144b46 Add Russell Davis to contributors 2019-07-02 07:57:18 +01:00
Nick Craig-Wood 8d116ba0c9 Add Matti Niemenmaa to contributors 2019-07-02 07:57:18 +01:00
Nick Craig-Wood 1be1fc073e Add AbelThar to contributors 2019-07-01 12:09:35 +01:00
Nick Craig-Wood af2596f98b Add yparitcher to contributors 2019-07-01 11:13:53 +01:00
Nick Craig-Wood 61fb326a80 Add full name for Laura Hausmann 2019-07-01 10:53:47 +01:00
Nick Craig-Wood 6bae3595a8 Add Laura to contributors 2019-06-30 20:07:35 +01:00
Nick Craig-Wood 9ed81ac451 vfs: fix tests for backends which can't upload 0 length files 2019-06-30 18:35:01 +01:00
Nick Craig-Wood 3245c0ae0d fstests: add integration test for uploading empty files 
This tests a remote can upload empty files.  If the remote can't
upload empty files it should return fs.ErrorCantUploadEmptyFiles.
 2019-06-30 18:35:01 +01:00
Nick Craig-Wood 07e2c3a50f b2: fix nil pointer error introduced by context propagation patch 
For some reason f78cd1e043 introduced an unrelated change -
perhaps a merge error.  Removing this change fixes the nil pointer
problem.
 2019-06-28 22:38:41 +01:00
Nick Craig-Wood d61ba7ef78 vendor: update all dependencies 2019-06-27 13:52:32 +01:00
Nick Craig-Wood b221d79273 Add nguyenhuuluan434 to contributors 2019-06-27 13:28:52 +01:00
Nick Craig-Wood 9f4589a997 gcs: reduce oauth scope requested as suggested by Google 
As part of getting the rclone oauth consent screen approved by Google,
it came up that the scope in use by the gcs backend was too large.

This change reduces it to the minimum scope which still allows rclone
to work correctly.

Old scope: https://www.googleapis.com/auth/devstorage.full_control
New scope: https://www.googleapis.com/auth/devstorage.read_write
 2019-06-27 12:05:49 +01:00
Nick Craig-Wood a1840f6fc7 sftp: add missing interface check and fix About #3257 
This bug was introduced as part of adding context to the backends and
slipped through the net because the About call did not have an
interface assertion in the sftp backend.

I checked there were no other missing interface assertions on all the
optional methods on all the backends.
 2019-06-26 16:56:33 +01:00
Nick Craig-Wood 4e64ee38e2 mount: default --deamon-timout to 15 minutes on macOS and FreeBSD 
See: https://forum.rclone.org/t/macos-fuse-mount-contents-disappear-after-writes-while-using-vfs-cache/10566/
 2019-06-25 15:30:42 +01:00
Nick Craig-Wood 276f8cccf6 rc: return current settings if core/bwlimit called without parameters 2019-06-24 13:22:24 +01:00
Nick Craig-Wood 0ae844d1f8 config: reset environment variables in config file test to fix build 2019-06-22 17:49:23 +01:00
Nick Craig-Wood 4ee6de5c3e docs: add a new page with global flags and link to it from the command docs 
In f544234 we removed the global flags from each command as it was
making each page very big and causing 1000s of lines of duplication in
the man page.

This change adds a new flags page with all the global flags on and
links each command page to it.

Fixes #3273
 2019-06-20 16:45:44 +01:00
Nick Craig-Wood 71a19a1972 Add Maran to contributors 2019-06-19 19:36:20 +01:00
Nick Craig-Wood 6a2a075c14 fs/cache: fix locking 
This was causing `fatal error: sync: unlock of unlocked mutex` if a
panic ocurred in fsNewFs.
 2019-06-19 10:50:59 +01:00
Nick Craig-Wood 628530362a local: add --local-case-sensitive and --local-case-insensitive 
This is to force the remote to declare itself as case sensitive or
insensitive where the defaults for the operating system are wrong.

See: https://forum.rclone.org/t/duplicate-object-found-in-source-ignoring-dedupe-not-finding-anything/10465
 2019-06-17 17:09:48 +01:00
Nick Craig-Wood 4549305fec Start v1.48.0-DEV development 2019-06-15 18:32:17 +01:00
Nick Craig-Wood 245fed513a Version v1.48.0 2019-06-15 13:55:41 +01:00
Nick Craig-Wood 52332a4b24 moveto: fix detection of same file name to include the root 
Fixes problem introduced in d2be792d5e
 2019-06-15 13:55:41 +01:00
Nick Craig-Wood 3087c5d559 webdav: retry on 423 Locked errors #3263 2019-06-15 10:58:13 +01:00
Nick Craig-Wood 75606dcc27 sync: fix tests on union remote 2019-06-15 10:53:36 +01:00
Nick Craig-Wood f3719fe269 fs/cache: unlock mutex in cache.Get to allow recursive calls 
This fixes the test lockup in the union tests
 2019-06-15 10:42:53 +01:00
Nick Craig-Wood ce400a8fdc Add rsync.net as a provider #3254 2019-06-15 09:56:17 +01:00
Nick Craig-Wood 49c05cb89b Add notes on --b2-version not working with crypt #1627 
See also: https://forum.rclone.org/t/how-to-restore-a-previous-version-of-a-folde/10456/2
 2019-06-15 09:56:17 +01:00
Nick Craig-Wood d533de0f5c docs: clarify that s3 can't access Glacier Vault 2019-06-15 09:56:17 +01:00
Nick Craig-Wood 1a4fe4bc6c Add Aleksandar Jankovic to contributors 2019-06-15 09:56:17 +01:00
Nick Craig-Wood 22368b997c b2: implement SetModTime #3210 
SetModTime() is implemented by copying an object onto itself and
updating the metadata in the process.
 2019-06-13 17:31:33 +01:00
Nick Craig-Wood a5bed67016 b2: implement server side copy - fixes #3210 2019-06-13 17:31:33 +01:00
Nick Craig-Wood 12c2a750f5 build: fix build lockup by increasing GOMAXPROCS - Fixes #3154 
It was discovered after lots of experimentation that the cmd/mount
tests have a tendency to lock up if GOMAXPROCS=1 or 2.  Since the
Travis builders only have 2 vCPUs by default, this happens on the
build server very often.

This workaround increases GOMAXPROCS to make the mount test lockup
less likely.

Ideally this should be fixed in the mount tests at some point.
 2019-06-13 13:47:43 +01:00
Nick Craig-Wood 92bbae5cca Add Florian Apolloner to contributors 2019-06-13 13:47:43 +01:00
Nick Craig-Wood 64fb4effa7 docs: add FAQ entry about rclone using too much memory 
See: #2200 #1391 3196
 2019-06-12 11:08:19 +01:00
Nick Craig-Wood 4d195d5a52 gcs: Fix upload errors when uploading pre 1970 files 
Before this change rclone attempted to set the "updated" field in
uploaded objects to the modification time.

However when this modification time was before 1970, google drive
would return the rather cryptic error:

    googleapi: Error 400: Invalid value for UnsignedLong: -42000, invalid

However API docs: https://cloud.google.com/storage/docs/json_api/v1/objects#resource
state the "updated" field is read only and tests confirm that.  Even
though the field is read only, it looks like Google parses it.

This change therefore removes the attempt to set the "updated" field
(which was doing nothing anyway) and fixes the problem uploading pre
1970 files.

See #3196 and https://forum.rclone.org/t/invalid-value-for-unsignedlong-file-missing-date-modified/3466
 2019-06-12 10:51:49 +01:00
Nick Craig-Wood 550ab441c5 rc: Skip auth for OPTIONS request 
Before this change using --user and --pass was impossible on the rc
from a browser as the browser needed to make the OPTIONS request first
before sending Authorization: headers, but the OPTIONS request
required an Authorization: header.

After this change we allow OPTIONS requests to go through without
checking the Authorization: header.
 2019-06-10 19:33:45 +01:00
Nick Craig-Wood e24cadc7a1 box: Fix ineffectual assignment (ineffassign) 2019-06-10 19:33:10 +01:00