f-denkena/rclone
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,871 Commits 1 Branch 0 Tags
8e86a902e20ade146148a3932f41e4e080849f14
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick Craig-Wood 8e86a902e2 travis: switch to using the .x version notation for the go minor versions 2018-01-19 14:32:32 +00:00
Nick Craig-Wood a80d8a21dc vfs: add flags parameter to Dir.Create 2018-01-19 13:18:40 +00:00
Nick Craig-Wood 517bdc719b vfs: make specialized file Open functions private 2018-01-19 11:46:01 +00:00
Nick Craig-Wood 5ad226ab54 fs: Add dir option to fs.Purge #1891 
Purge optional interface needs fixing too.
 2018-01-19 11:45:50 +00:00
Nick Craig-Wood a375992186 fstest: Fix removal of test folders/buckets 2018-01-19 10:20:06 +00:00
Nick Craig-Wood b96c73bee6 test_all: fix -clean flag 2018-01-19 09:47:01 +00:00
Nick Craig-Wood 97c414f025 config/hash: rename more symbols after factoring into own package 2018-01-18 20:27:52 +00:00
Nick Craig-Wood 71722b5b95 config: factor Obscure and Reveal into its own package 2018-01-18 20:19:55 +00:00
Nick Craig-Wood 59a8108fc3 webdav: add a new time format #1952 2018-01-18 16:54:13 +00:00
Nick Craig-Wood 821be5ebed ncdu: add link to asciinema demo of it in action 2018-01-18 14:22:43 +00:00
Nick Craig-Wood 2030dc13b2 lib/oauthutil: fix Google drive oauth process 
The problem was introduced by the code refactoring in
11da2a6c9b
 2018-01-18 11:18:35 +00:00
Nick Craig-Wood ad76dd0adc Add Lucas Bremgartner to contributors 2018-01-16 19:53:59 +00:00
Nick Craig-Wood 4b0c5f79b5 qingstor: Only support on go1.7+ 2018-01-16 17:05:26 +00:00
Nick Craig-Wood 1848e26183 dropbox: Only support on go1.7+ 
See https://github.com/dropbox/dropbox-sdk-go-unofficial/pull/40
 2018-01-16 17:05:02 +00:00
Nick Craig-Wood 7d3a17725d vendor: update all dependencies to latest versions 2018-01-16 13:20:59 +00:00
Nick Craig-Wood 8e83fb6fb9 Makefile: Fix integration test runner 2018-01-16 13:14:41 +00:00
Nick Craig-Wood 11da2a6c9b Break the fs package up into smaller parts. 
The purpose of this is to make it easier to maintain and eventually to
allow the rclone backends to be re-used in other projects without
having to use the rclone configuration system.

The new code layout is documented in CONTRIBUTING.
 2018-01-15 17:51:14 +00:00
Nick Craig-Wood 92624bbbf1 Move all backends into backend directory 2018-01-12 20:27:08 +00:00
Nick Craig-Wood 60afda007b Move dircache, oauthutil, rest and pacer modules into lib 2018-01-12 17:07:38 +00:00
Nick Craig-Wood b8b620f5c2 Move all backends into backend directory 2018-01-12 17:07:38 +00:00
Nick Craig-Wood 712e6a8085 lsf: fix integration tests 2018-01-11 13:52:15 +00:00
Nick Craig-Wood 6d333da69f Add Will Gunn to contributors 2018-01-10 20:33:57 +00:00
Nick Craig-Wood 052c886317 sftp: read $USER in username fallback not $HOME 2018-01-08 21:39:16 +00:00
Nick Craig-Wood 28480c0570 sftp: use correct OS way of reading username - fixes running under crontab 2018-01-07 12:57:46 +00:00
Nick Craig-Wood 72349bdaae Add Jon Fautley to contributors 2018-01-07 11:19:14 +00:00
Nick Craig-Wood 0eba37d8f3 lsf: add --files-only and --dirs-only flags 2018-01-06 18:04:24 +00:00
Nick Craig-Wood c74c3b37da lsf: add option to print hashes 2018-01-06 17:53:37 +00:00
Nick Craig-Wood 7c71ee1a5b fs: fix TestListFormat on remotes which return 0 as dir size not -1 2018-01-06 17:47:42 +00:00
Nick Craig-Wood ed20fa5ee7 ls* commands: update docs and add defaults into options for lsf 2018-01-06 17:00:20 +00:00
Nick Craig-Wood 54a9fdf421 ls2: remove in favour of lsf 2018-01-06 14:41:36 +00:00
Nick Craig-Wood 8f47d7fc06 Add Chris Redekop to contributors 2018-01-06 14:30:27 +00:00
Nick Craig-Wood 65618afd8c serve/http: fix serving files with : in - fixes #1939 2018-01-05 17:25:05 +00:00
Nick Craig-Wood be4ed14525 rest: rename URLEscape to URLPathEscape for consistency with go1.8 2018-01-05 15:55:43 +00:00
Nick Craig-Wood ef89f1f1a7 webdav: parse time in alternate format for mydrive.ch - fixes #1952 2018-01-05 14:28:06 +00:00
Nick Craig-Wood b412c745a1 Start v1.39-DEV development 2017-12-23 13:40:28 +00:00
Nick Craig-Wood f34a9116d4 Version v1.39 2017-12-23 13:07:45 +00:00
Nick Craig-Wood 5683f74025 Add Yassine Imounachen to contributors 2017-12-21 10:33:43 +00:00
Nick Craig-Wood 255d3e925d s3: fix crash if a bad listing is received - fixes #1927 
Caringo Swarm is returning a listing with IsTruncated set but no
NextMarker and no Keys.  Rclone doesn't know how to continue the
listing at this point, so it returns an error rather than truncating
the listing or risking a loop.
 2017-12-20 16:51:07 +00:00
Nick Craig-Wood 4ba58884b1 webdav: decode multiple <s:propstat> more carefully - fixes nextcloud 12.0.4 
For some reason nextcloud sends multiple propstat responses now, one
with a 404 status.  rclone was interpreting the last status and
assuming the file was missing.
 2017-12-20 11:53:10 +00:00
Nick Craig-Wood 29d34426bc vfs: fix deletion of in use directories #1860 
This was causing errors if the cache cleaner was called between the
Open and the pendingOpen of a RW file.

The fix was to move the cache open to the Open from the openPending.
 2017-12-15 15:42:49 +00:00
Nick Craig-Wood 2a01fa9fa0 moveto,copyto: clarify error message if source doesn't exist - fixes #1022 2017-12-15 11:37:31 +00:00
Nick Craig-Wood 4c0e2f9b3b swift: fix crash on bad authentication - fixes #1919 
This also fixes Hubic not re-authenticating for long transfers.
 2017-12-14 14:23:55 +00:00
Nick Craig-Wood 240c97cd7a Update MAINTAINERS doc 2017-12-14 13:56:58 +00:00
Nick Craig-Wood 2fd0bec4e4 docs: note that script install checks the version 2017-12-14 11:00:22 +00:00
Nick Craig-Wood 7e585cda96 fs: fix TestRmdirsLeaveRoot test 2017-12-14 08:57:28 +00:00
Nick Craig-Wood 1b1593a894 Add lewapm to contributors 2017-12-13 10:24:16 +00:00
Nick Craig-Wood 0914ec316c b2: fix multipart upload retries #1733 
Prior to this fix we were uploading 0 length bodies if a retry was
needed on a multipart upload chunk.  This gave this error `http:
ContentLength=268435496 with Body length 0`.

Fix by remaking the hash appending reader in the Call loop.  This is
inefficient in the face of retries, but these are uncommon.
 2017-12-13 10:11:20 +00:00
Nick Craig-Wood 2cf808c825 ncdu: fix crashes on empty directories - fixes #1910 
Up arrow or right arrow in an empty directory would crash ncdu
 2017-12-12 13:54:15 +00:00
Nick Craig-Wood 66558213e0 b2: send correct fileName when using --hard-delete - fixes #1905 2017-12-12 07:48:06 +00:00
Nick Craig-Wood 845b22a628 Add Jon Fautley to contributors 2017-12-10 10:53:49 +00:00
Nick Craig-Wood 5061aaaf46 vendor: update github.com/dropbox/dropbox-sdk-go-unofficial to fix #1806 2017-12-07 22:14:36 +00:00
Nick Craig-Wood e00616b016 Write version.txt on building into root of downloads 2017-12-07 21:49:32 +00:00
Nick Craig-Wood 09f203f62b Add Filip Bartodziej to contributors 2017-12-07 21:37:09 +00:00
Nick Craig-Wood bb3ba7b314 Add Giovanni Pizzi to contributors 2017-12-07 21:31:15 +00:00
Nick Craig-Wood ebd7780188 fstest: don't error out if the target was not found at end of run 2017-12-04 15:58:29 +00:00
Nick Craig-Wood fa4a25a73b fs: only test one level of cache 
Can't test multiple caches at once as can only have 1 DB open at once
 2017-12-04 15:50:59 +00:00
Nick Craig-Wood 006b296c34 Tidy up Makefile to get rid of vendor directory avoidance workarounds 2017-12-03 13:03:20 +00:00
Nick Craig-Wood 38b85e94ea vfs: rename --cache-* options to --vfs-cache-* to save confusion 
..as the backend cache options are all called --cache-* too. Adjust
docs to point out what the vfs cache does vs the backend cache.
 2017-12-03 12:14:15 +00:00
Nick Craig-Wood 4b185355df fs: rcat - use in memory object and Copy for more reliable transfers 2017-12-03 12:14:15 +00:00
Nick Craig-Wood 7d15c33e42 fs: make Copy and Move return the destination object if possible 2017-12-03 12:14:15 +00:00
Nick Craig-Wood 11332a19a0 fs: make an in memory object for short transfers 2017-12-03 12:14:15 +00:00
Nick Craig-Wood a1f8318b29 Add Laurence to contributors 2017-12-03 10:24:53 +00:00
Nick Craig-Wood 56cfb810a8 Add Tim Cooijmans to contributors 2017-12-03 10:22:42 +00:00
Nick Craig-Wood 639e812789 fs: integration tests: add SUMMARY heading for log scraping 2017-11-29 15:55:37 +00:00
Nick Craig-Wood 1c6cad2252 fs: integration tests: add 30 minute timeout per test 2017-11-29 13:51:17 +00:00
Nick Craig-Wood 6d3df6f172 cmount: make tests more reliable on Windows 2017-11-28 20:39:24 +00:00
Nick Craig-Wood c16ac697a9 vfs: keep track of directories in the cache also #1860 
This makes managing empty directories more reliable.
 2017-11-28 20:39:23 +00:00
Nick Craig-Wood 0978957a2e vfs: make sure all 96 combinations of flags for Open work 2017-11-28 20:39:23 +00:00
Nick Craig-Wood d1b19f975d vfs: remove items from cache when deleted #1860 
Also fixes Error message when items have been deleted from the cache
(eg when Moved) when the cache reaper comes to delete them.
 2017-11-28 16:13:58 +00:00
Nick Craig-Wood 1248beb0b2 cachestats: Fix nil pointer if not a cache remote - fixes #1855 
Also don't retry or show stats
 2017-11-24 10:22:23 +00:00
Nick Craig-Wood 6448c445f5 acd: Fix download of large files failing - Fixes #1501 
Previously it was necessary to work around large files failing to
download with `--acd-templink-threshold`.  This change makes that flag
obsolete and all files should download.  Templinks may be useful under
some circumstances though the flag isn't being removed.

It does this by filtering `Authorization:` headers out in the
transport if the authorization is supplied in the URL.  This prevents
the "Only one auth mechanism allowed; only the X-Amz-Algorithm query
parameter, Signature query string parameter or the Authorization
header should be specified" error from AWS.
 2017-11-24 09:14:25 +00:00
Nick Craig-Wood fdb01437d8 fs: Allow the http Transport to have an optional filter request function 2017-11-24 09:07:56 +00:00
Nick Craig-Wood 729e1305b7 oauthutil: Allow the http.Client to be passed in 2017-11-24 09:07:03 +00:00
Nick Craig-Wood 02ffd43572 fs: Save the config before asking for a token - fixes #1220 
Before this if the client_id/client_secret was edited it would
disappear when asking for the new token.

This means the post config is done after the user has confirmed the
config is OK which can't be helped.
 2017-11-23 14:01:32 +00:00
Nick Craig-Wood e53892f53b fs,drive,dropbox: Make and use new RepeatableReader variants to lower memory use 
RepeatableReaderSized has a pre-allocated buffer which should help
with memory usage - before it grew the buffer.  Since we know the size
of the chunks, pre-allocating it should be much more efficient.

RepeatableReaderBuffer uses the buffer passed in.

RepeatableLimit* are convenience funcitions for wrapping a reader in
an io.LimitReader and then a RepeatableReader with the same buffer
size.
 2017-11-23 13:53:46 +00:00
Nick Craig-Wood c64ad851af Add David Minor to contributors 2017-11-23 08:57:34 +00:00
Nick Craig-Wood 8357a82eee dropbox: change default chunk size to 48MB now we are buffering them in memory 2017-11-22 17:15:37 +00:00
Nick Craig-Wood 483f4b8ad9 dropbox: multiparts uploads retry retry every error after the first chunk is done 2017-11-22 17:15:37 +00:00
Nick Craig-Wood 6f61da5c75 dropbox: buffer the chunks when uploading large files so they can be retried 
We use fs.RepeatableReader to buffer the chunks which plays nice with
the accounting.  The default chunk size is 128M which may be too
large.

Fixes #1806
 2017-11-22 17:15:37 +00:00
Nick Craig-Wood 159fce0106 fs: fix --cache-dir to have some effect 2017-11-22 17:05:02 +00:00
Nick Craig-Wood cbe5d7ce64 fs: Remove X-Auth-Token: from headers when dumping for swift 2017-11-21 17:32:07 +00:00
Nick Craig-Wood 1a65a4e769 fs: Add --dump flag, introduce --dump requests, responses and remove --dump-auth, --dump-filters 
Now --dump-flag is written as --dump flag. This is a comma separated list which can contain

  * headers - HTTP headers as before
  * bodies  - HTTP bodies as before
  * requests - HTTP request bodies
  * responses - HTTP response bodies
  * auth - HTTP auth
  * filters - Filter rexeps

Leave --dump-headers and --dump-bodies for the time being but remove
the other --dump-* flags as they aren't used very often.
 2017-11-21 17:32:07 +00:00
Nick Craig-Wood bcf1ece43b Update MAINTAINERS with our new maintainer Remus Bunduc 2017-11-21 17:32:07 +00:00
Nick Craig-Wood abbcb2f5e0 cache: disable another unreliable test #1844 2017-11-20 21:25:38 +00:00
Nick Craig-Wood cb6de4a2cf cache: disable unreliable test #1844 2017-11-20 19:55:00 +00:00
Nick Craig-Wood dc1c679c65 mount: support truncate properly 2017-11-20 19:42:35 +00:00
Nick Craig-Wood 3fb4fe31d2 vfs: make sure write only handles never truncate files they shouldn't 2017-11-20 19:42:25 +00:00
Nick Craig-Wood 76b151984c vfs: cache the size of the object in the read handle 2017-11-20 17:57:13 +00:00
Nick Craig-Wood f0ed384786 cache: fix default setting for warmup_age 2017-11-20 14:39:12 +00:00
Nick Craig-Wood f80f7a0509 cache: use fs.CacheDir to make the default directory for the cache 
NB this changes the default dir for the cache
 2017-11-20 14:38:28 +00:00
Nick Craig-Wood af50f31f7d mounttest: wait for Release after every Read to stop using in use files under Windows 2017-11-20 12:46:24 +00:00
Nick Craig-Wood 8e2213fbbd local: add error message for cross file system moves 2017-11-20 12:46:24 +00:00
Nick Craig-Wood 085c690798 build: add in 64bit path for WinFSP headers 2017-11-20 12:46:24 +00:00
Nick Craig-Wood 2b666187a6 cmount: disable tests on windows + race detector 
These either hang or produce incorrect results for reasons I haven't
worked out yet.
 2017-11-20 12:46:24 +00:00
Nick Craig-Wood 00b46a8b96 mounttest: wait for files to disappear from the directory listing 2017-11-20 12:46:24 +00:00
Nick Craig-Wood b21f227bd3 mounttest: fix crash when FUSE not present 2017-11-20 12:46:24 +00:00
Nick Craig-Wood e98e550021 mounttest: wait for all background Close/Release after writing a file 
The filesystem does a certain amount of things asynchronously waiting
for the file to be released after writing it means everything should
be in a consistent state.
 2017-11-20 12:46:23 +00:00
Nick Craig-Wood 60945d0a37 vfs: remove misleading comment 2017-11-20 12:46:23 +00:00
Nick Craig-Wood b4083b4371 vfs: rename Fsync to Sync and implement Sync on Node and Handle 2017-11-20 12:46:23 +00:00
Nick Craig-Wood eb3415db50 cmount: enable more tests for Windows 2017-11-20 12:46:23 +00:00