f-denkena/rclone
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
2,774 Commits 1 Branch 0 Tags
762561f88ea32268389836d39f28ab94403bf9f0
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick Craig-Wood 762561f88e fstest: factor out retry logic from put code and make testPut return the object too 2018-11-15 18:39:28 +00:00
Nick Craig-Wood 084fe38922 fstests: fixes the integration test errors running crypt over swift. 
Skip tests involving errors creating or removing dirs on non root
bucket based fs
 2018-11-15 18:39:28 +00:00
Nick Craig-Wood f92beb4e14 fstest: Fix TestPurge causing errors with subsequent tests on azure 
Before this change TestPurge would remove a container and subsequent
tests would fail because the container was still being deleted so
couldn't be created.

This was fixed by introducing an fstest.NewRunIndividual() test runner
for TestPurge which causes the test to be run on a new container.
 2018-11-14 17:14:02 +00:00
Nick Craig-Wood f7ce2e8d95 azureblob: fix erroneous Rmdir error "directory not empty" 
Before this change Rmdir would check the root rather than the
directory specified for being empty and return "directory not empty"
when it shouldn't have done.
 2018-11-14 17:13:39 +00:00
Nick Craig-Wood 3975d82b3b Add brused27 to contributors 2018-11-13 17:00:26 +00:00
Nick Craig-Wood b3704597f3 cmount: make --volname work for Windows - fixes #2679 2018-11-12 16:32:02 +00:00
Nick Craig-Wood 16f797a7d7 filter: add --ignore-case flag - fixes #502 
The --ignore-case flag causes the filtering of file names to be case
insensitive.  The flag name comes from GNU tar.
 2018-11-12 14:29:37 +00:00
Nick Craig-Wood ee700ec01a lib/readers: add mutex to RepeatableReader - fixes #2572 2018-11-12 12:02:05 +00:00
Nick Craig-Wood 9b3c951ab7 Add Jake Coggiano to contributors 2018-11-12 11:34:28 +00:00
Nick Craig-Wood 84202c7471 onedrive: note 50,000 files is limit for one directory #2707 2018-11-11 15:22:19 +00:00
Nick Craig-Wood 96a05516f9 acd,box,onedrive,pcloud: remote log.Fatal from NewFs 
And replace with error returns.
 2018-11-11 11:00:14 +00:00
Nick Craig-Wood 4f6a942595 cmd: Make --progress update the stats right at the end 
Before this when rclone exited the stats would just show the last
printed version, rather than the actual final state.
 2018-11-11 09:57:37 +00:00
Nick Craig-Wood c4b0a37b21 rc: improve docs on debugging 2018-11-10 10:18:13 +00:00
Nick Craig-Wood 9322f4baef Add Erik Swanson to contributors 2018-11-08 12:58:41 +00:00
Nick Craig-Wood 4ad08794c9 fserrors: add "server closed idle connection" to retriable errors 
This seems to be related to this go issue: https://github.com/golang/go/issues/19943

See: https://forum.rclone.org/t/copy-from-dropbox-to-google-drive-yields-failed-to-copy-failed-to-open-source-object-server-closed-idle-connection-error/7460
 2018-11-08 11:12:25 +00:00
Nick Craig-Wood c0f600764b Add Scott Edlund to contributors 2018-11-07 14:27:06 +00:00
Nick Craig-Wood c6786eeb2d move: don't create directories with --dry-run - fixes #2676 2018-11-06 13:34:15 +00:00
Nick Craig-Wood 57b85b8155 rc: fix job tests on Windows 2018-11-06 13:03:48 +00:00
Nick Craig-Wood 2b1194c57e rc: update docs with new methods 2018-11-05 15:44:40 +00:00
Nick Craig-Wood e6dd121f52 config: add rc operations for config 2018-11-05 15:44:40 +00:00
Nick Craig-Wood e600217666 config: create config directory on save if it is missing 2018-11-05 15:44:40 +00:00
Nick Craig-Wood bc17ca7ed9 rc: implement core/obscure 2018-11-05 15:44:40 +00:00
Nick Craig-Wood 1916410316 rc: add core/version and put definitions next to implementations 2018-11-05 15:44:40 +00:00
Nick Craig-Wood dddfbec92a cmd/version: factor version number parsing routines into fs/version 2018-11-05 15:44:40 +00:00
Nick Craig-Wood 75a88de55c rc/rcserver: with --rc-files if auth set, pass on to URL opened 
If `--rc-user` or `--rc-pass` is set then the URL that is opened with
`--rc-files` will have the authorization in the URL in the
`http://user:pass@localhost/` style.
 2018-11-05 15:44:40 +00:00
Nick Craig-Wood 2466f4d152 sync: add rc commands for sync/copy/move 2018-11-05 15:44:40 +00:00
Nick Craig-Wood 39283c8a35 operations: implement operations remote control commands 2018-11-05 15:44:40 +00:00
Nick Craig-Wood 46c2f55545 copyurl: factor code into operations and write test 2018-11-04 20:42:57 +00:00
Nick Craig-Wood fc2afcbcbd lsjson: factor internals of lsjson command into operations 2018-11-04 20:42:57 +00:00
Nick Craig-Wood fa0a9653d2 rc: methods marked as AuthRequired need auth unless --rc-no-auth 
Methods which can read or mutate external storage will require
authorisation - enforce this.  This can be overidden by `--rc-no-auth`.
 2018-11-04 20:42:57 +00:00
Nick Craig-Wood 181267e20e cmd/rc: add --user and --pass flags and interpret --rc-user, --rc-pass, --rc-addr 2018-11-04 20:42:57 +00:00
Nick Craig-Wood 75e8ea383c rc: implement rc.PutCachedFs for prefilling the remote cache 2018-11-04 20:42:57 +00:00
Nick Craig-Wood 8c8b58a7de rc: expire remote cache and fix tests under race detector 2018-11-04 20:42:57 +00:00
Nick Craig-Wood b961e07c57 rc: ensure rclone fails to start up if the --rc port is in use already 2018-11-04 15:11:51 +00:00
Nick Craig-Wood 0b80d1481a cache: make tests not start an rc but use the internal framework 2018-11-04 15:11:51 +00:00
Nick Craig-Wood 89550e7121 rcserver: serve directories as well as files 2018-11-04 15:11:51 +00:00
Nick Craig-Wood 370c218c63 cmd/http: factor directory serving routines into httplib/serve and write tests 2018-11-04 12:46:44 +00:00
Nick Craig-Wood b972dcb0ae rc: implement options/blocks,get,set and register options 2018-11-03 11:32:00 +00:00
Nick Craig-Wood 0bfa9811f7 rc: factor server code into rcserver and implement serving objects 
If a GET or HEAD request is receivied with a URL parameter of fs then
it will be served from that remote.
 2018-11-03 11:32:00 +00:00
Nick Craig-Wood aa9b2c31f4 serve/restic: factor object serving into cmd/httplib/serve 2018-11-03 11:32:00 +00:00
Nick Craig-Wood cff75db6a4 rcd: implement new command just to serve the remote control API 2018-11-03 11:32:00 +00:00
Nick Craig-Wood 75252e4a89 rc: add --rc-files flag to serve files on the rc http server 
This enables building a browser based UI for rclone
 2018-11-03 11:32:00 +00:00
Nick Craig-Wood 2089405e1b fs/rc: add more infrastructure to help writing rc functions 
- Fs cache for rc commands
- Helper functions for parsing the input
- Reshape command for manipulating JSON blobs
- Background Job starting, control, query and expiry
 2018-11-02 17:32:20 +00:00
Nick Craig-Wood a379eec9d9 fstest/mockfs: create mock fs.Fs for testing 2018-11-02 17:32:20 +00:00
Nick Craig-Wood 45d5339fcb cmd/rc: add --json flag for structured JSON input 2018-11-02 17:32:20 +00:00
Nick Craig-Wood bb5637d46a serve http, webdav, restic: ensure rclone exits if the port is in use 2018-11-02 17:32:20 +00:00
Nick Craig-Wood 1f05d5bf4a delete: clarify that it only deletes files not directories 2018-11-02 17:07:45 +00:00
Nick Craig-Wood baba6d67e6 s3: set ACL for server side copies to that provided by the user - fixes #2691 
Before this change the ACL for objects which were server side copied
was left at the default "private" settings. S3 doesn't copy the ACL
from the source when you copy an object, you have to set it afresh
which is what this does.
 2018-11-02 16:22:31 +00:00
Nick Craig-Wood 04c0564fe2 Add Ralf Hemberger to contributors 2018-11-02 09:53:23 +00:00
Nick Craig-Wood 948a5d25c2 operations: Fix Purge and Rmdirs when dir is not empty 
Before this change, Purge on the fallback path would try to delete
directories starting from the root rather than the dir passed in.
Rmdirs would also attempt to delete the root.
 2018-10-27 11:51:17 +01:00
Nick Craig-Wood f7c31cd210 Add Florian Gamboeck to contributors 2018-10-27 00:28:11 +01:00
Nick Craig-Wood 543e37f662 Require go1.8 for compilation 2018-10-25 17:06:33 +01:00
Nick Craig-Wood c514cb752d vendor: update to latest versions of everything 2018-10-25 17:06:33 +01:00
Nick Craig-Wood c0ca93ae6f opendrive: fix retries of upload chunks - fixes #2646 
Before this change, upload chunks were being emptied on retry.  This
change introduces a RepeatableReader to fix the problem.
 2018-10-25 11:50:38 +01:00
Nick Craig-Wood 38a89d49ae fstest/test_all: tidy HTML report 
- link test number to online copy
- style links
- attempt to make a nicer colour scheme
 2018-10-25 11:33:17 +01:00
Nick Craig-Wood 25d0e59ef8 fstest/test_all: make sure Version is correct in build 2018-10-25 08:36:09 +01:00
Nick Craig-Wood b0db08fd2b fstest/test_all: constrain to go1.10 and above 2018-10-24 21:33:42 +01:00
Nick Craig-Wood 07addf74fd fstest/test_all: upload a copy of the report to "current" 2018-10-24 12:21:07 +01:00
Nick Craig-Wood 52c7c738ca fstest/test_all: limit concurrency and run tests in random order 2018-10-24 10:46:58 +01:00
Nick Craig-Wood 5c32b32011 fstest/test_all: fix directories that tests are run in 
- Don't build a binary for backend tests
- Run tests in their relevant directories
 2018-10-23 17:31:11 +01:00
Nick Craig-Wood fe61cff079 crypt: ensure integration tests run correctly when -remote is set 2018-10-23 17:12:38 +01:00
Nick Craig-Wood fbab1e55bb fstest/test_all: adapt to nested test definitions 2018-10-23 16:56:35 +01:00
Nick Craig-Wood 1bfd07567e fstest/test_all: add oneonly flag to only run one test per backend if required 2018-10-23 14:07:48 +01:00
Nick Craig-Wood f97c4c8d9d fstest/test_all: rework integration tests to improve output 
- Make integration tests use a config file
- Output individual logs for each test
- Make HTML report and open browser
- Optionally email and upload results
 2018-10-23 14:07:48 +01:00
Nick Craig-Wood c5ac96e9e7 Make --files-from only read the objects specified and don't scan directories 
Before this change using --files-from would scan all the directories
that the files could possibly be in causing rclone to do more work
that was necessary.

After this change, rclone constructs an in memory tree using the
--fast-list mechanism but from all of the files in the --files-from
list and without scanning any directories.

Any objects that are not found in the --files-from list are ignored
silently.

This mechanism is used for sync/copy/move (march) and all of the
listing commands ls/lsf/md5sum/etc (walk).
 2018-10-20 18:13:31 +01:00
Nick Craig-Wood 9959c5f17f webdav: add Content-Type to PUT requests - fixes #2664 2018-10-18 13:18:24 +01:00
Nick Craig-Wood e8d0a363fc opendrive: fix transfer of files with + and & in - fixes #2657 2018-10-17 14:22:04 +01:00
Nick Craig-Wood 67703a73de Start v1.44-DEV development 2018-10-15 12:33:27 +01:00
Nick Craig-Wood f96ce5674b Version v1.44 2018-10-15 11:03:08 +01:00
Nick Craig-Wood 7f0b204292 azureblob: work around SDK bug which causes errors for chunk-sized files (again) 
Until https://github.com/Azure/azure-storage-blob-go/pull/75 is merged
the SDK can't upload a single blob of exactly the chunk size, so
upload files of this size with a multpart upload as a work around.

The previous fix for this 6a773289e7 turned out to cause problems
uploading files with maximum chunk size so needed to be redone.

Fixes #2653
 2018-10-15 09:05:34 +01:00
Nick Craig-Wood 83b1ae4833 Add buergi to contributors 2018-10-14 17:20:34 +01:00
Nick Craig-Wood 5dac8e055f union: fix --backup-dir on union backend 
Before this fix --backup-dir would fail.

This is fixed by wrapping objects returned so that they belong to the
union Fs rather than the underlying Fs.
 2018-10-14 15:19:02 +01:00
Nick Craig-Wood c3a8eb1c10 fstests: make findObject() sleep a bit longer to fix b2 largePut tests 2018-10-14 14:45:23 +01:00
Nick Craig-Wood 0f2a5403db acd,box,onedrive,opendrive,ploud: fix Features() retaining the original receiver 
Before this change the Features() method would return a different Fs
to that the Features() method was called on if the remote was
instantiated on a file.

The practical effect of this is that optional features, eg `rclone
about` wouldn't work properly when called on a file, and likely this
has been causing low level problems for users of these backends for
ages.

Ideally there would be a test for this, but it turns out that this is
really hard, so instead of that all the backends have been converted
to not copy the Fs and a big warning comment inserted for future
readers.

Fixes #2182
 2018-10-14 14:41:26 +01:00
Nick Craig-Wood dcce84714e onedrive: fix link command for non root 2018-10-14 14:17:53 +01:00
Nick Craig-Wood eb8130f48a fstests: update TestPublicLink comment to show how to run solo 2018-10-14 14:17:05 +01:00
Nick Craig-Wood aa58f66806 build: add longer timeout to integration tests 2018-10-14 14:16:33 +01:00
Nick Craig-Wood a3dc591b8e Add teresy to contributors 2018-10-14 00:19:49 +01:00
Nick Craig-Wood dbedf33b9f s3: fix v2 signer on files with spaces - fixes #2438 
Before this fix the v2 signer was failing for files with spaces in.
 2018-10-14 00:10:29 +01:00
Nick Craig-Wood 0f02c9540c s3: make --s3-v2-auth flag 
This is an alternative to setting the region to "other-v2-signature"
which is inconvenient for multi-region providers.
 2018-10-14 00:10:29 +01:00
Nick Craig-Wood 06922674c8 drive, s3: review hidden config items 2018-10-13 23:30:13 +01:00
Nick Craig-Wood 8ad7da066c drive: when listing team drives, continue on failure 
This means that if the team drive listing returns a 500 error (which
seems reasonably common) rclone will continue to the point where it
asks for the team drive ID.

https://forum.rclone.org/t/many-team-drives-causes-rclone-to-fail/7159
 2018-10-13 23:30:13 +01:00
Nick Craig-Wood e1503add41 azureblob, b2, drive: implement set upload cutoff for chunked upload tests 2018-10-13 22:49:12 +01:00
Nick Craig-Wood 6fea75afde fstests: fix upload offsets not being set and redownload test files 
In chunked upload tests:

- Add ability to set upload offset
- Read back the uploaded file to check it is OK
 2018-10-13 22:49:12 +01:00
Nick Craig-Wood 6a773289e7 azureblob: work around SDK bug which causes errors for chunk-sized files 
See https://github.com/Azure/azure-storage-blob-go/pull/75 for details
 2018-10-13 22:49:12 +01:00
Nick Craig-Wood ade252f13b build: fixup code formatting after goimports change 2018-10-13 22:47:12 +01:00
Nick Craig-Wood bb2e361004 jottacloud: Fix socket leak on Object.Remove - fixes #2637 2018-10-13 22:47:12 +01:00
Nick Craig-Wood b24facb73d rest: Fix documentation so it is clearer when resp.Body is closed 2018-10-13 22:47:12 +01:00
Nick Craig-Wood 014d58a757 Add David Haguenauer to contributors 2018-10-13 12:56:21 +01:00
Nick Craig-Wood 249a523dd3 build: fix golint install with new path 2018-10-12 11:35:35 +01:00
Nick Craig-Wood 8d72ef8d1e cmd: Don't print non-ASCII characters with --progress on windows - fixes #2501 
This bug causes lots of strange behaviour with non-ASCII characters and --progress

https://github.com/Azure/go-ansiterm/issues/26
 2018-10-11 21:25:04 +01:00
Nick Craig-Wood bc8f0208aa rest: Remove auth headers on HTTP redirect 
Before this change the rest package would forward all the headers on
an HTTP redirect, including the Authorization: header.  This caused
problems when forwarded to a signed S3 URL ("Only one auth mechanism
allowed") as well as being a potential security risk.

After we use the go1.8+ mechanism for doing this instead of using our
own which does it correctly removing the Authorization: header when
redirecting to a different host.

This hasn't fixed the behaviour for rclone compiled with go1.7.

Fixes #2635
 2018-10-11 21:20:33 +01:00
Nick Craig-Wood ee25b6106a Add jackyzy823 to contributors 2018-10-11 14:50:33 +01:00
Nick Craig-Wood 5c1b135304 Add dcpu to contributors 2018-10-11 14:50:33 +01:00
Nick Craig-Wood 364fca5cea union: implement optional interfaces (Move, DirMove, Copy etc) - fixes #2619 
Implement optional interfaces
- Purge
- PutStream
- Copy
- Move
- DirMove
- DirCacheFlush
- ChangeNotify
- About

Make Hashes() return the intersection of all the hashes supported by the remotes
 2018-10-07 00:06:29 +01:00
Nick Craig-Wood 87e1efa997 mount, vfs: Remove EXPERIMENTAL tags 
rclone mount and the --vfs-cache-mode has been tested extensively by
users now so removing the EXPERIMENTAL tag is appropriate.
 2018-10-06 11:47:46 +01:00
Nick Craig-Wood 6709084e2f config: Show URL of backend help page when starting config 2018-10-06 11:47:46 +01:00
Nick Craig-Wood 6b1f915ebc fs: Implement RegInfo.FileName to return the on disk filename for a backend 
Use it in make_backend_docs.py
 2018-10-06 11:47:46 +01:00
Nick Craig-Wood 78b9bd77f5 docs: auto generate backend options documentation 
This inserts the output of "rclone help backend xxx" into the help
pages for each backend.
 2018-10-06 11:47:46 +01:00
Nick Craig-Wood a9273c5da5 docs: move documentation for options from docs/content into backends 
In the following commit, the documentation will be autogenerated.
 2018-10-06 11:47:46 +01:00