f-denkena/rclone
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,849 Commits 1 Branch 0 Tags
b8b620f5c2df8ae26ee3ef66de9cad71e423be6a
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick Craig-Wood b8b620f5c2 Move all backends into backend directory 2018-01-12 17:07:38 +00:00
Nick Craig-Wood 712e6a8085 lsf: fix integration tests 2018-01-11 13:52:15 +00:00
Nick Craig-Wood 6d333da69f Add Will Gunn to contributors 2018-01-10 20:33:57 +00:00
Nick Craig-Wood 052c886317 sftp: read $USER in username fallback not $HOME 2018-01-08 21:39:16 +00:00
Nick Craig-Wood 28480c0570 sftp: use correct OS way of reading username - fixes running under crontab 2018-01-07 12:57:46 +00:00
Nick Craig-Wood 72349bdaae Add Jon Fautley to contributors 2018-01-07 11:19:14 +00:00
Nick Craig-Wood 0eba37d8f3 lsf: add --files-only and --dirs-only flags 2018-01-06 18:04:24 +00:00
Nick Craig-Wood c74c3b37da lsf: add option to print hashes 2018-01-06 17:53:37 +00:00
Nick Craig-Wood 7c71ee1a5b fs: fix TestListFormat on remotes which return 0 as dir size not -1 2018-01-06 17:47:42 +00:00
Nick Craig-Wood ed20fa5ee7 ls* commands: update docs and add defaults into options for lsf 2018-01-06 17:00:20 +00:00
Nick Craig-Wood 54a9fdf421 ls2: remove in favour of lsf 2018-01-06 14:41:36 +00:00
Nick Craig-Wood 8f47d7fc06 Add Chris Redekop to contributors 2018-01-06 14:30:27 +00:00
Nick Craig-Wood 65618afd8c serve/http: fix serving files with : in - fixes #1939 2018-01-05 17:25:05 +00:00
Nick Craig-Wood be4ed14525 rest: rename URLEscape to URLPathEscape for consistency with go1.8 2018-01-05 15:55:43 +00:00
Nick Craig-Wood ef89f1f1a7 webdav: parse time in alternate format for mydrive.ch - fixes #1952 2018-01-05 14:28:06 +00:00
Nick Craig-Wood b412c745a1 Start v1.39-DEV development 2017-12-23 13:40:28 +00:00
Nick Craig-Wood f34a9116d4 Version v1.39 2017-12-23 13:07:45 +00:00
Nick Craig-Wood 5683f74025 Add Yassine Imounachen to contributors 2017-12-21 10:33:43 +00:00
Nick Craig-Wood 255d3e925d s3: fix crash if a bad listing is received - fixes #1927 
Caringo Swarm is returning a listing with IsTruncated set but no
NextMarker and no Keys.  Rclone doesn't know how to continue the
listing at this point, so it returns an error rather than truncating
the listing or risking a loop.
 2017-12-20 16:51:07 +00:00
Nick Craig-Wood 4ba58884b1 webdav: decode multiple <s:propstat> more carefully - fixes nextcloud 12.0.4 
For some reason nextcloud sends multiple propstat responses now, one
with a 404 status.  rclone was interpreting the last status and
assuming the file was missing.
 2017-12-20 11:53:10 +00:00
Nick Craig-Wood 29d34426bc vfs: fix deletion of in use directories #1860 
This was causing errors if the cache cleaner was called between the
Open and the pendingOpen of a RW file.

The fix was to move the cache open to the Open from the openPending.
 2017-12-15 15:42:49 +00:00
Nick Craig-Wood 2a01fa9fa0 moveto,copyto: clarify error message if source doesn't exist - fixes #1022 2017-12-15 11:37:31 +00:00
Nick Craig-Wood 4c0e2f9b3b swift: fix crash on bad authentication - fixes #1919 
This also fixes Hubic not re-authenticating for long transfers.
 2017-12-14 14:23:55 +00:00
Nick Craig-Wood 240c97cd7a Update MAINTAINERS doc 2017-12-14 13:56:58 +00:00
Nick Craig-Wood 2fd0bec4e4 docs: note that script install checks the version 2017-12-14 11:00:22 +00:00
Nick Craig-Wood 7e585cda96 fs: fix TestRmdirsLeaveRoot test 2017-12-14 08:57:28 +00:00
Nick Craig-Wood 1b1593a894 Add lewapm to contributors 2017-12-13 10:24:16 +00:00
Nick Craig-Wood 0914ec316c b2: fix multipart upload retries #1733 
Prior to this fix we were uploading 0 length bodies if a retry was
needed on a multipart upload chunk.  This gave this error `http:
ContentLength=268435496 with Body length 0`.

Fix by remaking the hash appending reader in the Call loop.  This is
inefficient in the face of retries, but these are uncommon.
 2017-12-13 10:11:20 +00:00
Nick Craig-Wood 2cf808c825 ncdu: fix crashes on empty directories - fixes #1910 
Up arrow or right arrow in an empty directory would crash ncdu
 2017-12-12 13:54:15 +00:00
Nick Craig-Wood 66558213e0 b2: send correct fileName when using --hard-delete - fixes #1905 2017-12-12 07:48:06 +00:00
Nick Craig-Wood 845b22a628 Add Jon Fautley to contributors 2017-12-10 10:53:49 +00:00
Nick Craig-Wood 5061aaaf46 vendor: update github.com/dropbox/dropbox-sdk-go-unofficial to fix #1806 2017-12-07 22:14:36 +00:00
Nick Craig-Wood e00616b016 Write version.txt on building into root of downloads 2017-12-07 21:49:32 +00:00
Nick Craig-Wood 09f203f62b Add Filip Bartodziej to contributors 2017-12-07 21:37:09 +00:00
Nick Craig-Wood bb3ba7b314 Add Giovanni Pizzi to contributors 2017-12-07 21:31:15 +00:00
Nick Craig-Wood ebd7780188 fstest: don't error out if the target was not found at end of run 2017-12-04 15:58:29 +00:00
Nick Craig-Wood fa4a25a73b fs: only test one level of cache 
Can't test multiple caches at once as can only have 1 DB open at once
 2017-12-04 15:50:59 +00:00
Nick Craig-Wood 006b296c34 Tidy up Makefile to get rid of vendor directory avoidance workarounds 2017-12-03 13:03:20 +00:00
Nick Craig-Wood 38b85e94ea vfs: rename --cache-* options to --vfs-cache-* to save confusion 
..as the backend cache options are all called --cache-* too. Adjust
docs to point out what the vfs cache does vs the backend cache.
 2017-12-03 12:14:15 +00:00
Nick Craig-Wood 4b185355df fs: rcat - use in memory object and Copy for more reliable transfers 2017-12-03 12:14:15 +00:00
Nick Craig-Wood 7d15c33e42 fs: make Copy and Move return the destination object if possible 2017-12-03 12:14:15 +00:00
Nick Craig-Wood 11332a19a0 fs: make an in memory object for short transfers 2017-12-03 12:14:15 +00:00
Nick Craig-Wood a1f8318b29 Add Laurence to contributors 2017-12-03 10:24:53 +00:00
Nick Craig-Wood 56cfb810a8 Add Tim Cooijmans to contributors 2017-12-03 10:22:42 +00:00
Nick Craig-Wood 639e812789 fs: integration tests: add SUMMARY heading for log scraping 2017-11-29 15:55:37 +00:00
Nick Craig-Wood 1c6cad2252 fs: integration tests: add 30 minute timeout per test 2017-11-29 13:51:17 +00:00
Nick Craig-Wood 6d3df6f172 cmount: make tests more reliable on Windows 2017-11-28 20:39:24 +00:00
Nick Craig-Wood c16ac697a9 vfs: keep track of directories in the cache also #1860 
This makes managing empty directories more reliable.
 2017-11-28 20:39:23 +00:00
Nick Craig-Wood 0978957a2e vfs: make sure all 96 combinations of flags for Open work 2017-11-28 20:39:23 +00:00
Nick Craig-Wood d1b19f975d vfs: remove items from cache when deleted #1860 
Also fixes Error message when items have been deleted from the cache
(eg when Moved) when the cache reaper comes to delete them.
 2017-11-28 16:13:58 +00:00
Nick Craig-Wood 1248beb0b2 cachestats: Fix nil pointer if not a cache remote - fixes #1855 
Also don't retry or show stats
 2017-11-24 10:22:23 +00:00
Nick Craig-Wood 6448c445f5 acd: Fix download of large files failing - Fixes #1501 
Previously it was necessary to work around large files failing to
download with `--acd-templink-threshold`.  This change makes that flag
obsolete and all files should download.  Templinks may be useful under
some circumstances though the flag isn't being removed.

It does this by filtering `Authorization:` headers out in the
transport if the authorization is supplied in the URL.  This prevents
the "Only one auth mechanism allowed; only the X-Amz-Algorithm query
parameter, Signature query string parameter or the Authorization
header should be specified" error from AWS.
 2017-11-24 09:14:25 +00:00
Nick Craig-Wood fdb01437d8 fs: Allow the http Transport to have an optional filter request function 2017-11-24 09:07:56 +00:00
Nick Craig-Wood 729e1305b7 oauthutil: Allow the http.Client to be passed in 2017-11-24 09:07:03 +00:00
Nick Craig-Wood 02ffd43572 fs: Save the config before asking for a token - fixes #1220 
Before this if the client_id/client_secret was edited it would
disappear when asking for the new token.

This means the post config is done after the user has confirmed the
config is OK which can't be helped.
 2017-11-23 14:01:32 +00:00
Nick Craig-Wood e53892f53b fs,drive,dropbox: Make and use new RepeatableReader variants to lower memory use 
RepeatableReaderSized has a pre-allocated buffer which should help
with memory usage - before it grew the buffer.  Since we know the size
of the chunks, pre-allocating it should be much more efficient.

RepeatableReaderBuffer uses the buffer passed in.

RepeatableLimit* are convenience funcitions for wrapping a reader in
an io.LimitReader and then a RepeatableReader with the same buffer
size.
 2017-11-23 13:53:46 +00:00
Nick Craig-Wood c64ad851af Add David Minor to contributors 2017-11-23 08:57:34 +00:00
Nick Craig-Wood 8357a82eee dropbox: change default chunk size to 48MB now we are buffering them in memory 2017-11-22 17:15:37 +00:00
Nick Craig-Wood 483f4b8ad9 dropbox: multiparts uploads retry retry every error after the first chunk is done 2017-11-22 17:15:37 +00:00
Nick Craig-Wood 6f61da5c75 dropbox: buffer the chunks when uploading large files so they can be retried 
We use fs.RepeatableReader to buffer the chunks which plays nice with
the accounting.  The default chunk size is 128M which may be too
large.

Fixes #1806
 2017-11-22 17:15:37 +00:00
Nick Craig-Wood 159fce0106 fs: fix --cache-dir to have some effect 2017-11-22 17:05:02 +00:00
Nick Craig-Wood cbe5d7ce64 fs: Remove X-Auth-Token: from headers when dumping for swift 2017-11-21 17:32:07 +00:00
Nick Craig-Wood 1a65a4e769 fs: Add --dump flag, introduce --dump requests, responses and remove --dump-auth, --dump-filters 
Now --dump-flag is written as --dump flag. This is a comma separated list which can contain

  * headers - HTTP headers as before
  * bodies  - HTTP bodies as before
  * requests - HTTP request bodies
  * responses - HTTP response bodies
  * auth - HTTP auth
  * filters - Filter rexeps

Leave --dump-headers and --dump-bodies for the time being but remove
the other --dump-* flags as they aren't used very often.
 2017-11-21 17:32:07 +00:00
Nick Craig-Wood bcf1ece43b Update MAINTAINERS with our new maintainer Remus Bunduc 2017-11-21 17:32:07 +00:00
Nick Craig-Wood abbcb2f5e0 cache: disable another unreliable test #1844 2017-11-20 21:25:38 +00:00
Nick Craig-Wood cb6de4a2cf cache: disable unreliable test #1844 2017-11-20 19:55:00 +00:00
Nick Craig-Wood dc1c679c65 mount: support truncate properly 2017-11-20 19:42:35 +00:00
Nick Craig-Wood 3fb4fe31d2 vfs: make sure write only handles never truncate files they shouldn't 2017-11-20 19:42:25 +00:00
Nick Craig-Wood 76b151984c vfs: cache the size of the object in the read handle 2017-11-20 17:57:13 +00:00
Nick Craig-Wood f0ed384786 cache: fix default setting for warmup_age 2017-11-20 14:39:12 +00:00
Nick Craig-Wood f80f7a0509 cache: use fs.CacheDir to make the default directory for the cache 
NB this changes the default dir for the cache
 2017-11-20 14:38:28 +00:00
Nick Craig-Wood af50f31f7d mounttest: wait for Release after every Read to stop using in use files under Windows 2017-11-20 12:46:24 +00:00
Nick Craig-Wood 8e2213fbbd local: add error message for cross file system moves 2017-11-20 12:46:24 +00:00
Nick Craig-Wood 085c690798 build: add in 64bit path for WinFSP headers 2017-11-20 12:46:24 +00:00
Nick Craig-Wood 2b666187a6 cmount: disable tests on windows + race detector 
These either hang or produce incorrect results for reasons I haven't
worked out yet.
 2017-11-20 12:46:24 +00:00
Nick Craig-Wood 00b46a8b96 mounttest: wait for files to disappear from the directory listing 2017-11-20 12:46:24 +00:00
Nick Craig-Wood b21f227bd3 mounttest: fix crash when FUSE not present 2017-11-20 12:46:24 +00:00
Nick Craig-Wood e98e550021 mounttest: wait for all background Close/Release after writing a file 
The filesystem does a certain amount of things asynchronously waiting
for the file to be released after writing it means everything should
be in a consistent state.
 2017-11-20 12:46:23 +00:00
Nick Craig-Wood 60945d0a37 vfs: remove misleading comment 2017-11-20 12:46:23 +00:00
Nick Craig-Wood b4083b4371 vfs: rename Fsync to Sync and implement Sync on Node and Handle 2017-11-20 12:46:23 +00:00
Nick Craig-Wood eb3415db50 cmount: enable more tests for Windows 2017-11-20 12:46:23 +00:00
Nick Craig-Wood 9fbd8a6419 mounttest: fixes for running under Windows 
* don't mount and unmount between cache runs - WinFSP doesn't suport it
  * use OS paths for opening things
 2017-11-20 12:46:23 +00:00
Nick Craig-Wood 9738f8532b vfs: Add FlushDirCache method 2017-11-20 12:46:23 +00:00
Nick Craig-Wood a5b034a992 vfs: add WaitForWriters to wait until all writers have finished 2017-11-20 12:46:23 +00:00
Nick Craig-Wood 321b6da7af vfs: don't remove file from writers until it is transferred 
This means that the list of active writers is up to date
 2017-11-20 12:46:23 +00:00
Nick Craig-Wood 1b22ee5b93 vfs: fix error handling in openPending so it returns the correct error 2017-11-20 12:46:23 +00:00
Nick Craig-Wood eab55ce882 vfs: add open files to directories 2017-11-20 12:46:23 +00:00
Nick Craig-Wood 61b6159a05 mount, cmount: add O_CREATE to Open calls since fuse doesn't seem to supply it 2017-11-20 12:46:22 +00:00
Nick Craig-Wood c560017934 vfs: add Path method to Node and use it to stop reading nil DirEntry 
All DirEntry calls now have been checked for nil or converted to use Path.
 2017-11-20 12:46:22 +00:00
Nick Craig-Wood 7c3584f4e6 mountlib: wait for mountpoint to disappear under Windows 2017-11-20 12:46:22 +00:00
Nick Craig-Wood 981cfb1bec mounttest: retry directory listings to account for slow updates on Windows 2017-11-20 12:46:22 +00:00
Nick Craig-Wood 992647b157 vfs: Don't error a r/w file open without cache; delay error until Read called 
If we open a file for r/w without the cache we now always return a
handle and return an error if the file is ever read from.  This fixes
incompatibility with cmount under windows.
 2017-11-20 12:46:22 +00:00
Nick Craig-Wood dec21ccf63 vfs, cmount: make truncate work properly in the presence or otherwise of open files 2017-11-20 12:46:22 +00:00
Nick Craig-Wood 94adf4f43b cmount: translate FUSE open flags into OS flags 
On Windows the fuse.O_* flags do not have the same values as the
os.O_* flags so translate between the two representations.  They are
mostly the same which is why this hasn't caused a problem before.
 2017-11-20 12:46:22 +00:00
Nick Craig-Wood e7f2935333 vfs: decode flags in Open/OpenFile for debug 2017-11-20 12:46:22 +00:00
Nick Craig-Wood f5f8c0c438 cmount: make Truncate call the correct Handle or Node method 2017-11-20 12:46:22 +00:00
Nick Craig-Wood 60cdcf784c cmount: use -o atomic_o_trunc to make sure O_TRUNC is supplied to Open() 2017-11-20 12:46:22 +00:00
Nick Craig-Wood 57a5c67729 mounttest: run the tests for all 4 VFS cache modes 2017-11-20 12:46:21 +00:00
Nick Craig-Wood d7908c06c9 mountlib: ensure we don't open files with read and write intent 2017-11-20 12:46:21 +00:00
Nick Craig-Wood 8951875c21 vfs,mount,cmount,mountlib: allow flags to be overriden by environment variables 2017-11-20 12:46:21 +00:00