f-denkena/rclone
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
2,734 Commits 1 Branch 0 Tags
8c8b58a7de73b744392bb33bfe8ad264bcca3aa6
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick Craig-Wood 8c8b58a7de rc: expire remote cache and fix tests under race detector 2018-11-04 20:42:57 +00:00
Nick Craig-Wood b961e07c57 rc: ensure rclone fails to start up if the --rc port is in use already 2018-11-04 15:11:51 +00:00
Nick Craig-Wood 0b80d1481a cache: make tests not start an rc but use the internal framework 2018-11-04 15:11:51 +00:00
Nick Craig-Wood 89550e7121 rcserver: serve directories as well as files 2018-11-04 15:11:51 +00:00
Nick Craig-Wood 370c218c63 cmd/http: factor directory serving routines into httplib/serve and write tests 2018-11-04 12:46:44 +00:00
Nick Craig-Wood b972dcb0ae rc: implement options/blocks,get,set and register options 2018-11-03 11:32:00 +00:00
Nick Craig-Wood 0bfa9811f7 rc: factor server code into rcserver and implement serving objects 
If a GET or HEAD request is receivied with a URL parameter of fs then
it will be served from that remote.
 2018-11-03 11:32:00 +00:00
Nick Craig-Wood aa9b2c31f4 serve/restic: factor object serving into cmd/httplib/serve 2018-11-03 11:32:00 +00:00
Nick Craig-Wood cff75db6a4 rcd: implement new command just to serve the remote control API 2018-11-03 11:32:00 +00:00
Nick Craig-Wood 75252e4a89 rc: add --rc-files flag to serve files on the rc http server 
This enables building a browser based UI for rclone
 2018-11-03 11:32:00 +00:00
Nick Craig-Wood 2089405e1b fs/rc: add more infrastructure to help writing rc functions 
- Fs cache for rc commands
- Helper functions for parsing the input
- Reshape command for manipulating JSON blobs
- Background Job starting, control, query and expiry
 2018-11-02 17:32:20 +00:00
Nick Craig-Wood a379eec9d9 fstest/mockfs: create mock fs.Fs for testing 2018-11-02 17:32:20 +00:00
Nick Craig-Wood 45d5339fcb cmd/rc: add --json flag for structured JSON input 2018-11-02 17:32:20 +00:00
Nick Craig-Wood bb5637d46a serve http, webdav, restic: ensure rclone exits if the port is in use 2018-11-02 17:32:20 +00:00
Nick Craig-Wood 1f05d5bf4a delete: clarify that it only deletes files not directories 2018-11-02 17:07:45 +00:00
Nick Craig-Wood baba6d67e6 s3: set ACL for server side copies to that provided by the user - fixes #2691 
Before this change the ACL for objects which were server side copied
was left at the default "private" settings. S3 doesn't copy the ACL
from the source when you copy an object, you have to set it afresh
which is what this does.
 2018-11-02 16:22:31 +00:00
Nick Craig-Wood 04c0564fe2 Add Ralf Hemberger to contributors 2018-11-02 09:53:23 +00:00
Nick Craig-Wood 948a5d25c2 operations: Fix Purge and Rmdirs when dir is not empty 
Before this change, Purge on the fallback path would try to delete
directories starting from the root rather than the dir passed in.
Rmdirs would also attempt to delete the root.
 2018-10-27 11:51:17 +01:00
Nick Craig-Wood f7c31cd210 Add Florian Gamboeck to contributors 2018-10-27 00:28:11 +01:00
Nick Craig-Wood 543e37f662 Require go1.8 for compilation 2018-10-25 17:06:33 +01:00
Nick Craig-Wood c514cb752d vendor: update to latest versions of everything 2018-10-25 17:06:33 +01:00
Nick Craig-Wood c0ca93ae6f opendrive: fix retries of upload chunks - fixes #2646 
Before this change, upload chunks were being emptied on retry.  This
change introduces a RepeatableReader to fix the problem.
 2018-10-25 11:50:38 +01:00
Nick Craig-Wood 38a89d49ae fstest/test_all: tidy HTML report 
- link test number to online copy
- style links
- attempt to make a nicer colour scheme
 2018-10-25 11:33:17 +01:00
Nick Craig-Wood 25d0e59ef8 fstest/test_all: make sure Version is correct in build 2018-10-25 08:36:09 +01:00
Nick Craig-Wood b0db08fd2b fstest/test_all: constrain to go1.10 and above 2018-10-24 21:33:42 +01:00
Nick Craig-Wood 07addf74fd fstest/test_all: upload a copy of the report to "current" 2018-10-24 12:21:07 +01:00
Nick Craig-Wood 52c7c738ca fstest/test_all: limit concurrency and run tests in random order 2018-10-24 10:46:58 +01:00
Nick Craig-Wood 5c32b32011 fstest/test_all: fix directories that tests are run in 
- Don't build a binary for backend tests
- Run tests in their relevant directories
 2018-10-23 17:31:11 +01:00
Nick Craig-Wood fe61cff079 crypt: ensure integration tests run correctly when -remote is set 2018-10-23 17:12:38 +01:00
Nick Craig-Wood fbab1e55bb fstest/test_all: adapt to nested test definitions 2018-10-23 16:56:35 +01:00
Nick Craig-Wood 1bfd07567e fstest/test_all: add oneonly flag to only run one test per backend if required 2018-10-23 14:07:48 +01:00
Nick Craig-Wood f97c4c8d9d fstest/test_all: rework integration tests to improve output 
- Make integration tests use a config file
- Output individual logs for each test
- Make HTML report and open browser
- Optionally email and upload results
 2018-10-23 14:07:48 +01:00
Nick Craig-Wood c5ac96e9e7 Make --files-from only read the objects specified and don't scan directories 
Before this change using --files-from would scan all the directories
that the files could possibly be in causing rclone to do more work
that was necessary.

After this change, rclone constructs an in memory tree using the
--fast-list mechanism but from all of the files in the --files-from
list and without scanning any directories.

Any objects that are not found in the --files-from list are ignored
silently.

This mechanism is used for sync/copy/move (march) and all of the
listing commands ls/lsf/md5sum/etc (walk).
 2018-10-20 18:13:31 +01:00
Nick Craig-Wood 9959c5f17f webdav: add Content-Type to PUT requests - fixes #2664 2018-10-18 13:18:24 +01:00
Nick Craig-Wood e8d0a363fc opendrive: fix transfer of files with + and & in - fixes #2657 2018-10-17 14:22:04 +01:00
Nick Craig-Wood 67703a73de Start v1.44-DEV development 2018-10-15 12:33:27 +01:00
Nick Craig-Wood f96ce5674b Version v1.44 2018-10-15 11:03:08 +01:00
Nick Craig-Wood 7f0b204292 azureblob: work around SDK bug which causes errors for chunk-sized files (again) 
Until https://github.com/Azure/azure-storage-blob-go/pull/75 is merged
the SDK can't upload a single blob of exactly the chunk size, so
upload files of this size with a multpart upload as a work around.

The previous fix for this 6a773289e7 turned out to cause problems
uploading files with maximum chunk size so needed to be redone.

Fixes #2653
 2018-10-15 09:05:34 +01:00
Nick Craig-Wood 83b1ae4833 Add buergi to contributors 2018-10-14 17:20:34 +01:00
Nick Craig-Wood 5dac8e055f union: fix --backup-dir on union backend 
Before this fix --backup-dir would fail.

This is fixed by wrapping objects returned so that they belong to the
union Fs rather than the underlying Fs.
 2018-10-14 15:19:02 +01:00
Nick Craig-Wood c3a8eb1c10 fstests: make findObject() sleep a bit longer to fix b2 largePut tests 2018-10-14 14:45:23 +01:00
Nick Craig-Wood 0f2a5403db acd,box,onedrive,opendrive,ploud: fix Features() retaining the original receiver 
Before this change the Features() method would return a different Fs
to that the Features() method was called on if the remote was
instantiated on a file.

The practical effect of this is that optional features, eg `rclone
about` wouldn't work properly when called on a file, and likely this
has been causing low level problems for users of these backends for
ages.

Ideally there would be a test for this, but it turns out that this is
really hard, so instead of that all the backends have been converted
to not copy the Fs and a big warning comment inserted for future
readers.

Fixes #2182
 2018-10-14 14:41:26 +01:00
Nick Craig-Wood dcce84714e onedrive: fix link command for non root 2018-10-14 14:17:53 +01:00
Nick Craig-Wood eb8130f48a fstests: update TestPublicLink comment to show how to run solo 2018-10-14 14:17:05 +01:00
Nick Craig-Wood aa58f66806 build: add longer timeout to integration tests 2018-10-14 14:16:33 +01:00
Nick Craig-Wood a3dc591b8e Add teresy to contributors 2018-10-14 00:19:49 +01:00
Nick Craig-Wood dbedf33b9f s3: fix v2 signer on files with spaces - fixes #2438 
Before this fix the v2 signer was failing for files with spaces in.
 2018-10-14 00:10:29 +01:00
Nick Craig-Wood 0f02c9540c s3: make --s3-v2-auth flag 
This is an alternative to setting the region to "other-v2-signature"
which is inconvenient for multi-region providers.
 2018-10-14 00:10:29 +01:00
Nick Craig-Wood 06922674c8 drive, s3: review hidden config items 2018-10-13 23:30:13 +01:00
Nick Craig-Wood 8ad7da066c drive: when listing team drives, continue on failure 
This means that if the team drive listing returns a 500 error (which
seems reasonably common) rclone will continue to the point where it
asks for the team drive ID.

https://forum.rclone.org/t/many-team-drives-causes-rclone-to-fail/7159
 2018-10-13 23:30:13 +01:00
Nick Craig-Wood e1503add41 azureblob, b2, drive: implement set upload cutoff for chunked upload tests 2018-10-13 22:49:12 +01:00
Nick Craig-Wood 6fea75afde fstests: fix upload offsets not being set and redownload test files 
In chunked upload tests:

- Add ability to set upload offset
- Read back the uploaded file to check it is OK
 2018-10-13 22:49:12 +01:00
Nick Craig-Wood 6a773289e7 azureblob: work around SDK bug which causes errors for chunk-sized files 
See https://github.com/Azure/azure-storage-blob-go/pull/75 for details
 2018-10-13 22:49:12 +01:00
Nick Craig-Wood ade252f13b build: fixup code formatting after goimports change 2018-10-13 22:47:12 +01:00
Nick Craig-Wood bb2e361004 jottacloud: Fix socket leak on Object.Remove - fixes #2637 2018-10-13 22:47:12 +01:00
Nick Craig-Wood b24facb73d rest: Fix documentation so it is clearer when resp.Body is closed 2018-10-13 22:47:12 +01:00
Nick Craig-Wood 014d58a757 Add David Haguenauer to contributors 2018-10-13 12:56:21 +01:00
Nick Craig-Wood 249a523dd3 build: fix golint install with new path 2018-10-12 11:35:35 +01:00
Nick Craig-Wood 8d72ef8d1e cmd: Don't print non-ASCII characters with --progress on windows - fixes #2501 
This bug causes lots of strange behaviour with non-ASCII characters and --progress

https://github.com/Azure/go-ansiterm/issues/26
 2018-10-11 21:25:04 +01:00
Nick Craig-Wood bc8f0208aa rest: Remove auth headers on HTTP redirect 
Before this change the rest package would forward all the headers on
an HTTP redirect, including the Authorization: header.  This caused
problems when forwarded to a signed S3 URL ("Only one auth mechanism
allowed") as well as being a potential security risk.

After we use the go1.8+ mechanism for doing this instead of using our
own which does it correctly removing the Authorization: header when
redirecting to a different host.

This hasn't fixed the behaviour for rclone compiled with go1.7.

Fixes #2635
 2018-10-11 21:20:33 +01:00
Nick Craig-Wood ee25b6106a Add jackyzy823 to contributors 2018-10-11 14:50:33 +01:00
Nick Craig-Wood 5c1b135304 Add dcpu to contributors 2018-10-11 14:50:33 +01:00
Nick Craig-Wood 364fca5cea union: implement optional interfaces (Move, DirMove, Copy etc) - fixes #2619 
Implement optional interfaces
- Purge
- PutStream
- Copy
- Move
- DirMove
- DirCacheFlush
- ChangeNotify
- About

Make Hashes() return the intersection of all the hashes supported by the remotes
 2018-10-07 00:06:29 +01:00
Nick Craig-Wood 87e1efa997 mount, vfs: Remove EXPERIMENTAL tags 
rclone mount and the --vfs-cache-mode has been tested extensively by
users now so removing the EXPERIMENTAL tag is appropriate.
 2018-10-06 11:47:46 +01:00
Nick Craig-Wood 6709084e2f config: Show URL of backend help page when starting config 2018-10-06 11:47:46 +01:00
Nick Craig-Wood 6b1f915ebc fs: Implement RegInfo.FileName to return the on disk filename for a backend 
Use it in make_backend_docs.py
 2018-10-06 11:47:46 +01:00
Nick Craig-Wood 78b9bd77f5 docs: auto generate backend options documentation 
This inserts the output of "rclone help backend xxx" into the help
pages for each backend.
 2018-10-06 11:47:46 +01:00
Nick Craig-Wood a9273c5da5 docs: move documentation for options from docs/content into backends 
In the following commit, the documentation will be autogenerated.
 2018-10-06 11:47:46 +01:00
Nick Craig-Wood 14128656db cmd: Implement specialised help for flags and backends - fixes #2541 
Instead of showing all flags/backends all the time, you can type

    rclone help flags
    rclone help flags <regexp>
    rclone help backends
    rclone help backend <name>
 2018-10-06 11:47:45 +01:00
Nick Craig-Wood 1557287c64 fs: Make Option.GetValue() public #2541 2018-10-06 11:47:45 +01:00
Nick Craig-Wood e7e467fb3a cmd: factor FlagName into fs.Option #2541 2018-10-06 11:47:45 +01:00
Nick Craig-Wood 5fde7d8b12 cmd: split flags up into global and backend flags #2541 2018-10-06 11:47:45 +01:00
Nick Craig-Wood 3c086f5f7f cmd: Make default help less verbose #2541 
This stops the default help showing all the flags, backends, commands
 2018-10-06 11:47:45 +01:00
Nick Craig-Wood ddbd4fd881 Add Paul Kohout to contributors 2018-10-04 08:25:39 +01:00
Nick Craig-Wood 06ae4258be cmd: Fix -P not ending with a new line 
Before this fix rclone didn't wait for the stats to be finished before
exiting, so the final new line was never printed.

After this change rclone will wait for the stats routine to cease
before exiting.
 2018-10-03 21:46:18 +01:00
Nick Craig-Wood 59e14c25df vfs: enable rename for nearly all remotes using server side Move or Copy 
Before this change remotes without server side Move (eg swift, s3,
gcs) would not be able to rename files.

After it means nearly all remotes will be able to rename files on
rclone mount with the notable exceptions of b2 and yandex.

This changes checks to see if the remote can do Move or Copy then
calls `operations.Move` to do the actual move.  This will do a server
side Move or Copy but won't download and re-upload the file.

It also checks to see if the destination exists first which avoids
conflicts or duplicates.

Fixes #1965
Fixes #2569
 2018-09-29 14:56:20 +01:00
Nick Craig-Wood fc640d3a09 Add Frantisek Fuka to contributors 2018-09-29 14:55:11 +01:00
Nick Craig-Wood c7aa6b587b Add xnaas to contributors 2018-09-26 10:07:13 +01:00
Nick Craig-Wood 96ce49ec4e Add ssaqua to contributors 2018-09-24 17:08:47 +01:00
Nick Craig-Wood e2fb588eb9 Add frenos to contributors 2018-09-24 17:05:03 +01:00
Nick Craig-Wood 7b975bc1ff alias: Fix handling of Windows network paths 
Before this fix, the alias backend would mangle Windows paths like
//server/drive as it was treating them as unix paths.

See https://forum.rclone.org/t/smb-share-alias/6857
 2018-09-21 18:24:21 +01:00
Nick Craig-Wood 467fe30a5e vendor: update to latest versions of everything 2018-09-21 18:23:37 +01:00
Nick Craig-Wood 4415aa5c2e build: fix make update 2018-09-21 18:23:37 +01:00
Nick Craig-Wood 17ab38502d Revamp issue and PR templates and CONTRIBUTING guide 
Thanks to @fd0 of the restic project for a very useful blog post and
something to plagiarise :-)

https://restic.net/blog/2018-09-09/GitHub-issue-templates
 2018-09-21 18:17:32 +01:00
Nick Craig-Wood 9fa8c959ee local: preallocate files on linux with fallocate(2) 2018-09-19 16:04:57 +01:00
Nick Craig-Wood f29c6049fc local: preallocate files on Windows to reduce fragmentation #2469 
Before this change on Windows, files copied locally could become
heavily fragmented (300+ fragments for maybe 100 MB), no matter how
much contiguous free space there was (even if it's over 1TiB). This
can needlessly yet severely adversely affect performance on hard
disks.

This changes uses NtSetInformationFile to pre-allocate the space to
avoid this.

It does nothing on other OSes other than Windows.
 2018-09-19 16:04:57 +01:00
Nick Craig-Wood e44fa5db8e build: update git bisect scripts 2018-09-19 16:04:57 +01:00
Nick Craig-Wood b22ecbe174 Add Joanna Marek to contributors 2018-09-18 10:27:33 +01:00
Nick Craig-Wood c41be436c6 Add Antoine GIRARD to contributors 2018-09-18 10:27:33 +01:00
Nick Craig-Wood d27630626a webdav: add a small pause after failed upload before deleting file #2517 
This fixes the integration tests for `serve webdav` which uses the
webdav backend tests.
 2018-09-17 08:51:50 +01:00
Nick Craig-Wood c473c7cb53 ftp: add a small pause after failed upload before deleting file #2517 
This fixes the integration tests for `serve ftp` which uses the ftp
backend tests.
 2018-09-17 08:51:50 +01:00
Nick Craig-Wood ef3526b3b8 vfs: fix race condition detected by serve ftp tests 2018-09-17 08:50:34 +01:00
Nick Craig-Wood d4ee7277c0 serve ftp: disable on plan9 since it doesn't compile 2018-09-17 08:50:34 +01:00
Nick Craig-Wood a14f0d46d7 vendor: add github.com/goftp/server 2018-09-17 08:50:34 +01:00
Nick Craig-Wood a25875170b webdav: Add another time format to fix #2574 2018-09-15 21:46:56 +01:00
Nick Craig-Wood b3d8bc61ac build: add example scripts for bisecting rclone and go 2018-09-14 11:17:51 +01:00
Nick Craig-Wood b2a6a97443 Add Craig Miskell to contributors 2018-09-11 07:57:16 +01:00
Nick Craig-Wood 7cfd4e56f8 Add Santiago Rodríguez to contributors 
Another email address for Sandeep.
 2018-09-10 20:49:55 +01:00
Nick Craig-Wood a823d518ac docs: changelog from v1.43.1 branch 2018-09-07 17:16:11 +01:00
Nick Craig-Wood 87d4e32a6b build: instructions on how to make a point release 2018-09-07 17:10:29 +01:00