f-denkena/rclone
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
2,832 Commits 1 Branch 0 Tags
3de7ad5223a84e8621ddfb10363ad46b4df745b5
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick Craig-Wood 3de7ad5223 b2: for a bucket limited application key check the bucket name 
Before this fix rclone would just use the authorised bucket regardless
of what bucket you put on the command line.

This uses the new `bucketName` response in the API and checks that the
user is using the correct bucket name to avoid accidents.

Fixes #2839
 2018-12-20 12:07:35 +00:00
Nick Craig-Wood c1dd76788d httplib: make http serving with auth generate INFO messages on auth fail 
2018/12/13 12:13:44 INFO  : /: 127.0.0.1:39696: Basic auth challenge sent
2018/12/13 12:13:54 INFO  : /: 127.0.0.1:40050: Unauthorized request from ncw

Fixes #2834
 2018-12-14 13:38:49 +00:00
Nick Craig-Wood 5ee1816a71 filter: parallelise reading of --files-from - fixes #2835 
Before this change rclone would read the list of files from the
files-from parameter and check they existed one at a time.  This could
take a very long time for lots of files.

After this change, rclone will check up to --checkers in parallel.
 2018-12-13 13:22:30 +00:00
Nick Craig-Wood 63b51c6742 vendor: add golang.org/x/sync as a dependency 2018-12-13 10:45:52 +00:00
Nick Craig-Wood e7684b7ed5 Add William Cocker to contributors 2018-12-06 21:53:53 +00:00
Nick Craig-Wood feea0532cd azureblob: ignore directory markers - fixes #2806 
This ignores 0 length blobs if
- they end with /
- they have the metadata hdi_isfolder = true
 2018-12-06 21:47:03 +00:00
Nick Craig-Wood d3e8ae1820 Add Mark Otway to contributors 2018-12-06 15:13:03 +00:00
Nick Craig-Wood 91a9a959a2 Add Mathieu Carbou to contributors 2018-12-06 15:12:58 +00:00
Nick Craig-Wood fc654a4cec http: fix backend with --files-from and non-existent files 
Before this fix the http backend was returning the wrong error code
when files were not found.  This was causing --files-from to error on
missing files instead of skipping them like it should.
 2018-12-04 17:40:44 +00:00
Nick Craig-Wood 26b5f55cba Update after goimports change 2018-12-04 10:11:57 +00:00
Nick Craig-Wood 3f572e6bf2 webdav: fix infinite loop on failed directory creation - fixes #2714 2018-12-02 21:03:12 +00:00
Nick Craig-Wood 941ad6bc62 azureblob: use the s3 pacer for 0 delay - fixes #2799 2018-12-02 20:55:16 +00:00
Nick Craig-Wood 5d1d93e163 azureblob: use the rclone HTTP client - fixes #2654 
This enables --dump headers and --timeout to work properly.
 2018-12-02 20:55:16 +00:00
Nick Craig-Wood 35fba5bfdd Add Garry McNulty to contributors 2018-12-02 20:52:04 +00:00
Nick Craig-Wood 107293c80e copy,move: restore --no-traverse flag 
The --no-traverse flag was not implemented when the new sync routines
(using the march package) was implemented.

This re-implements --no-traverse in march by trying to find a match
for each object with NewObject rather than from a directory listing.
 2018-12-02 20:28:13 +00:00
Nick Craig-Wood e3c4ebd59a march: factor calling parameters into a structure 2018-12-02 18:07:26 +00:00
Nick Craig-Wood d99ffde7c0 s3: change --s3-upload-concurrency default to 4 to increase perfomance #2772 
Increasing the --s3-upload-concurrency to 4 (from 2) gives an
additional 45% throughput at the cost of 10MB extra memory per transfer.

After testing the upload perfoc
 2018-12-02 17:58:34 +00:00
Nick Craig-Wood 198c34ce21 s3: implement --s3-upload-cutoff for single part uploads below this - fixes #2772 
Before this change rclone would use multipart uploads for any size of
file.  However multipart uploads are less efficient for smaller files
and don't have MD5 checksums so it is advantageous to use single part
uploads if possible.

This implements single part uploads for all files smaller than the
upload_cutoff size.  Streamed files must be uploaded as multipart
files though.
 2018-12-02 17:58:34 +00:00
Nick Craig-Wood 0eba88bbfe sftp: check directory is empty before issuing rmdir 
Some SFTP servers allow rmdir on full directories which is allowed
according to the RFC so make sure we don't accidentally delete data
here.

See: https://forum.rclone.org/t/rmdir-and-delete-empty-src-dirs-file-does-not-exist/7737
 2018-12-02 11:16:30 +00:00
Nick Craig-Wood aeea4430d5 swift: efficiency: slim Object and reduce requests on upload 
- Slim down Object to only include necessary data
- Don't HEAD an object after PUT - read the hash from the response
 2018-12-02 10:23:55 +00:00
Nick Craig-Wood 4b15c4215c sftp: fix rmdir on Windows based servers (eg CrushFTP) 
Before this change we used Remove to remove directories.  This works
fine on Unix based systems but not so well on Windows based ones.
Swap to using RemoveDirectory instead.
 2018-11-29 21:34:37 +00:00
Nick Craig-Wood 50452207d9 swift: add --swift-no-chunk to disable segmented uploads in rcat/mount 
Fixes #2776
 2018-11-29 11:11:30 +00:00
Nick Craig-Wood 01fcad9b9c rc: fix docs for sync/{sync,copy,move} and operations/{copy,move}file 2018-11-29 11:11:30 +00:00
Nick Craig-Wood 89625e54cf vendor: update dependencies to latest 2018-11-26 14:10:33 +00:00
Nick Craig-Wood 58f7141c96 drive, googlecloudstorage: disallow on go1.8 due to dependent library changes 
golang.org/x/oauth2/google no longer builds on go1.8
 2018-11-26 14:10:33 +00:00
Nick Craig-Wood e56c6402a7 serve restic: disallow on go1.8 because of dependent library changes 
golang.org/x/net/http2 no longer builds on go1.8
 2018-11-26 14:10:33 +00:00
Nick Craig-Wood d0eb8ddc30 serve webdav: disallow on go1.8 due to dependent library changes 
golang.org/x/net/webdav no longer builds with go1.8
 2018-11-26 14:10:33 +00:00
Nick Craig-Wood a6c28a5faa Start v1.45-DEV development 2018-11-24 15:20:24 +00:00
Nick Craig-Wood d35bd15762 Version v1.45 2018-11-24 13:44:25 +00:00
Nick Craig-Wood 8b8220c4f7 azureblob: wait for up to 60s to create a just deleted container 
When a container is deleted, a container with the same name cannot be
created for at least 30 seconds; the container may not be available
for more than 30 seconds if the service is still processing the
request.

We sleep so that we wait at most 60 seconds.  This is mostly useful in
the integration tests where containers get deleted and remade
immediately.
 2018-11-24 10:57:37 +00:00
Nick Craig-Wood 5fe3b0ad71 Add Stephen Harris to contributors 2018-11-24 10:57:37 +00:00
Nick Craig-Wood bb10a51b39 test_all: limit to go1.11 so the template used is supported 2018-11-23 17:17:19 +00:00
Nick Craig-Wood df01f7a4eb test_all: fix regexp for retrying nested tests 2018-11-23 17:17:19 +00:00
Nick Craig-Wood e84790ef79 swift: add pacer for retries to make swift more reliable #2740 2018-11-22 22:15:52 +00:00
Nick Craig-Wood 369a8ee17b ncdu: fix deleting files 2018-11-22 21:41:17 +00:00
Nick Craig-Wood 84e21ade6b cmount: fix on Linux - only apply volname for Windows and macOS 2018-11-22 20:41:05 +00:00
Nick Craig-Wood 31e2ce03c3 fstests: re-arrange backend integration tests so they can be retried 
Before this change backend integration tests depended on each other,
so tests could not be retried.

After this change we nest tests to ensure that tests are provided with
the starting state they expect.

Tell the integration test runner that it can retry backend tests also.

This also includes bin/test_independence.go which runs each test
individually for a backend to prove that they are independent.
 2018-11-22 20:12:12 +00:00
Nick Craig-Wood e969505ae4 info: fix control character map output 2018-11-20 14:04:27 +00:00
Nick Craig-Wood 26e2f1a998 Add Alexander to contributors 2018-11-20 10:22:11 +00:00
Nick Craig-Wood 2191592e80 Add Henry Ptasinski to contributors 2018-11-19 13:33:59 +00:00
Nick Craig-Wood 18f758294e Add Peter Kaminski to contributors 2018-11-19 13:33:59 +00:00
Nick Craig-Wood 8c8dcdd521 webdav: fix config parsing so --webdav-user and --webdav-pass flags work 2018-11-17 13:14:54 +00:00
Nick Craig-Wood 141c133818 fstest: Wait for longer if neccessary in TestFsChangeNotify 2018-11-16 07:45:24 +00:00
Nick Craig-Wood 0f03e55cd1 fstests: ignore main directory creation in TestFsChangeNotify 2018-11-15 18:39:28 +00:00
Nick Craig-Wood 9e6ba92a11 fstests: attempt to fix TestFsChangeNotify flakiness 
This now uses testPut to upload the test files which will retry on
errors properly.
 2018-11-15 18:39:28 +00:00
Nick Craig-Wood 762561f88e fstest: factor out retry logic from put code and make testPut return the object too 2018-11-15 18:39:28 +00:00
Nick Craig-Wood 084fe38922 fstests: fixes the integration test errors running crypt over swift. 
Skip tests involving errors creating or removing dirs on non root
bucket based fs
 2018-11-15 18:39:28 +00:00
Nick Craig-Wood f92beb4e14 fstest: Fix TestPurge causing errors with subsequent tests on azure 
Before this change TestPurge would remove a container and subsequent
tests would fail because the container was still being deleted so
couldn't be created.

This was fixed by introducing an fstest.NewRunIndividual() test runner
for TestPurge which causes the test to be run on a new container.
 2018-11-14 17:14:02 +00:00
Nick Craig-Wood f7ce2e8d95 azureblob: fix erroneous Rmdir error "directory not empty" 
Before this change Rmdir would check the root rather than the
directory specified for being empty and return "directory not empty"
when it shouldn't have done.
 2018-11-14 17:13:39 +00:00
Nick Craig-Wood 3975d82b3b Add brused27 to contributors 2018-11-13 17:00:26 +00:00
Nick Craig-Wood b3704597f3 cmount: make --volname work for Windows - fixes #2679 2018-11-12 16:32:02 +00:00
Nick Craig-Wood 16f797a7d7 filter: add --ignore-case flag - fixes #502 
The --ignore-case flag causes the filtering of file names to be case
insensitive.  The flag name comes from GNU tar.
 2018-11-12 14:29:37 +00:00
Nick Craig-Wood ee700ec01a lib/readers: add mutex to RepeatableReader - fixes #2572 2018-11-12 12:02:05 +00:00
Nick Craig-Wood 9b3c951ab7 Add Jake Coggiano to contributors 2018-11-12 11:34:28 +00:00
Nick Craig-Wood 84202c7471 onedrive: note 50,000 files is limit for one directory #2707 2018-11-11 15:22:19 +00:00
Nick Craig-Wood 96a05516f9 acd,box,onedrive,pcloud: remote log.Fatal from NewFs 
And replace with error returns.
 2018-11-11 11:00:14 +00:00
Nick Craig-Wood 4f6a942595 cmd: Make --progress update the stats right at the end 
Before this when rclone exited the stats would just show the last
printed version, rather than the actual final state.
 2018-11-11 09:57:37 +00:00
Nick Craig-Wood c4b0a37b21 rc: improve docs on debugging 2018-11-10 10:18:13 +00:00
Nick Craig-Wood 9322f4baef Add Erik Swanson to contributors 2018-11-08 12:58:41 +00:00
Nick Craig-Wood 4ad08794c9 fserrors: add "server closed idle connection" to retriable errors 
This seems to be related to this go issue: https://github.com/golang/go/issues/19943

See: https://forum.rclone.org/t/copy-from-dropbox-to-google-drive-yields-failed-to-copy-failed-to-open-source-object-server-closed-idle-connection-error/7460
 2018-11-08 11:12:25 +00:00
Nick Craig-Wood c0f600764b Add Scott Edlund to contributors 2018-11-07 14:27:06 +00:00
Nick Craig-Wood c6786eeb2d move: don't create directories with --dry-run - fixes #2676 2018-11-06 13:34:15 +00:00
Nick Craig-Wood 57b85b8155 rc: fix job tests on Windows 2018-11-06 13:03:48 +00:00
Nick Craig-Wood 2b1194c57e rc: update docs with new methods 2018-11-05 15:44:40 +00:00
Nick Craig-Wood e6dd121f52 config: add rc operations for config 2018-11-05 15:44:40 +00:00
Nick Craig-Wood e600217666 config: create config directory on save if it is missing 2018-11-05 15:44:40 +00:00
Nick Craig-Wood bc17ca7ed9 rc: implement core/obscure 2018-11-05 15:44:40 +00:00
Nick Craig-Wood 1916410316 rc: add core/version and put definitions next to implementations 2018-11-05 15:44:40 +00:00
Nick Craig-Wood dddfbec92a cmd/version: factor version number parsing routines into fs/version 2018-11-05 15:44:40 +00:00
Nick Craig-Wood 75a88de55c rc/rcserver: with --rc-files if auth set, pass on to URL opened 
If `--rc-user` or `--rc-pass` is set then the URL that is opened with
`--rc-files` will have the authorization in the URL in the
`http://user:pass@localhost/` style.
 2018-11-05 15:44:40 +00:00
Nick Craig-Wood 2466f4d152 sync: add rc commands for sync/copy/move 2018-11-05 15:44:40 +00:00
Nick Craig-Wood 39283c8a35 operations: implement operations remote control commands 2018-11-05 15:44:40 +00:00
Nick Craig-Wood 46c2f55545 copyurl: factor code into operations and write test 2018-11-04 20:42:57 +00:00
Nick Craig-Wood fc2afcbcbd lsjson: factor internals of lsjson command into operations 2018-11-04 20:42:57 +00:00
Nick Craig-Wood fa0a9653d2 rc: methods marked as AuthRequired need auth unless --rc-no-auth 
Methods which can read or mutate external storage will require
authorisation - enforce this.  This can be overidden by `--rc-no-auth`.
 2018-11-04 20:42:57 +00:00
Nick Craig-Wood 181267e20e cmd/rc: add --user and --pass flags and interpret --rc-user, --rc-pass, --rc-addr 2018-11-04 20:42:57 +00:00
Nick Craig-Wood 75e8ea383c rc: implement rc.PutCachedFs for prefilling the remote cache 2018-11-04 20:42:57 +00:00
Nick Craig-Wood 8c8b58a7de rc: expire remote cache and fix tests under race detector 2018-11-04 20:42:57 +00:00
Nick Craig-Wood b961e07c57 rc: ensure rclone fails to start up if the --rc port is in use already 2018-11-04 15:11:51 +00:00
Nick Craig-Wood 0b80d1481a cache: make tests not start an rc but use the internal framework 2018-11-04 15:11:51 +00:00
Nick Craig-Wood 89550e7121 rcserver: serve directories as well as files 2018-11-04 15:11:51 +00:00
Nick Craig-Wood 370c218c63 cmd/http: factor directory serving routines into httplib/serve and write tests 2018-11-04 12:46:44 +00:00
Nick Craig-Wood b972dcb0ae rc: implement options/blocks,get,set and register options 2018-11-03 11:32:00 +00:00
Nick Craig-Wood 0bfa9811f7 rc: factor server code into rcserver and implement serving objects 
If a GET or HEAD request is receivied with a URL parameter of fs then
it will be served from that remote.
 2018-11-03 11:32:00 +00:00
Nick Craig-Wood aa9b2c31f4 serve/restic: factor object serving into cmd/httplib/serve 2018-11-03 11:32:00 +00:00
Nick Craig-Wood cff75db6a4 rcd: implement new command just to serve the remote control API 2018-11-03 11:32:00 +00:00
Nick Craig-Wood 75252e4a89 rc: add --rc-files flag to serve files on the rc http server 
This enables building a browser based UI for rclone
 2018-11-03 11:32:00 +00:00
Nick Craig-Wood 2089405e1b fs/rc: add more infrastructure to help writing rc functions 
- Fs cache for rc commands
- Helper functions for parsing the input
- Reshape command for manipulating JSON blobs
- Background Job starting, control, query and expiry
 2018-11-02 17:32:20 +00:00
Nick Craig-Wood a379eec9d9 fstest/mockfs: create mock fs.Fs for testing 2018-11-02 17:32:20 +00:00
Nick Craig-Wood 45d5339fcb cmd/rc: add --json flag for structured JSON input 2018-11-02 17:32:20 +00:00
Nick Craig-Wood bb5637d46a serve http, webdav, restic: ensure rclone exits if the port is in use 2018-11-02 17:32:20 +00:00
Nick Craig-Wood 1f05d5bf4a delete: clarify that it only deletes files not directories 2018-11-02 17:07:45 +00:00
Nick Craig-Wood baba6d67e6 s3: set ACL for server side copies to that provided by the user - fixes #2691 
Before this change the ACL for objects which were server side copied
was left at the default "private" settings. S3 doesn't copy the ACL
from the source when you copy an object, you have to set it afresh
which is what this does.
 2018-11-02 16:22:31 +00:00
Nick Craig-Wood 04c0564fe2 Add Ralf Hemberger to contributors 2018-11-02 09:53:23 +00:00
Nick Craig-Wood 948a5d25c2 operations: Fix Purge and Rmdirs when dir is not empty 
Before this change, Purge on the fallback path would try to delete
directories starting from the root rather than the dir passed in.
Rmdirs would also attempt to delete the root.
 2018-10-27 11:51:17 +01:00
Nick Craig-Wood f7c31cd210 Add Florian Gamboeck to contributors 2018-10-27 00:28:11 +01:00
Nick Craig-Wood 543e37f662 Require go1.8 for compilation 2018-10-25 17:06:33 +01:00
Nick Craig-Wood c514cb752d vendor: update to latest versions of everything 2018-10-25 17:06:33 +01:00
Nick Craig-Wood c0ca93ae6f opendrive: fix retries of upload chunks - fixes #2646 
Before this change, upload chunks were being emptied on retry.  This
change introduces a RepeatableReader to fix the problem.
 2018-10-25 11:50:38 +01:00
Nick Craig-Wood 38a89d49ae fstest/test_all: tidy HTML report 
- link test number to online copy
- style links
- attempt to make a nicer colour scheme
 2018-10-25 11:33:17 +01:00