f-denkena/rclone
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,057 Commits 1 Branch 0 Tags
b52c80e85c7d98b894ccbcfe5604150ad349d611
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick Craig-Wood b52c80e85c sync: don't update mod times if --dry-run set - fixes #1100 2017-02-15 23:09:44 +00:00
Nick Craig-Wood f15c6b68b6 Re-add the async buffer on seek - fixes #1137 2017-02-15 22:54:21 +00:00
Nick Craig-Wood 3f778d70f7 Add sync.Pool to async reader 2017-02-15 22:37:58 +00:00
Nick Craig-Wood 9a9d09845c mount: put read and write async buffers back - control with --buffer-size #1043 2017-02-14 22:59:52 +00:00
Nick Craig-Wood 7fa687b3e1 fs: Async buffer: use ReadFill to fill the chunks and increase to 1MB 2017-02-14 22:36:37 +00:00
Nick Craig-Wood 493da54113 Add --buffer-size parameter to control buffer size for copy 2017-02-14 22:36:37 +00:00
Nick Craig-Wood 541929258b check: Add --download flag to check all the data, not just hashes 2017-02-13 10:48:26 +00:00
Nick Craig-Wood 370f242fa2 local: Fix interaction between -x flag and --max-depth - fixes #1126 
This was causing the by directory sync to ignore the -x flag because
it was putting directories into the listing which should have been
excluded.
 2017-02-13 09:24:29 +00:00
Nick Craig-Wood 7047c67a5e sync: Fix log message containing <nil> 2017-02-13 09:23:21 +00:00
Nick Craig-Wood 18c75a81f9 Add notes on cryptcheck and backups to crypt docs 2017-02-12 16:49:31 +00:00
Nick Craig-Wood 01c747e7db Add cryptcheck command to check integrity of crypt remotes #1102 2017-02-12 16:30:18 +00:00
Nick Craig-Wood 186aedda98 Fix go vet on go 1.7 2017-02-12 12:43:13 +00:00
Nick Craig-Wood ca0e25b1a1 Remove spurious comment 2017-02-12 10:56:52 +00:00
Nick Craig-Wood f87a694d10 Make donation page easier to find and add bitcoin address 2017-02-11 23:03:05 +00:00
Nick Craig-Wood 006227baed Replace -v with -vv where necessary or change Debugf to Logf 2017-02-11 20:27:46 +00:00
Nick Craig-Wood 4d28b5ed22 Update list of commands in docs. 2017-02-11 20:27:46 +00:00
Nick Craig-Wood 499475bb41 Fix -vv by temporarily patching vendored cobra 
This is a temporary fix until this pull request gets merged

https://github.com/spf13/cobra/pull/391

See original ticket

https://github.com/spf13/pflag/issues/112
 2017-02-11 20:27:46 +00:00
Nick Craig-Wood 666dae4229 Add --syslog flag to optionally log to syslog on capable platforms 2017-02-11 20:27:46 +00:00
Nick Craig-Wood ac1c041377 Redo log level flags 
* -vv or --log-level DEBUG
  * -v or --log-level INFO
  * --log-level NOTICE (default)
  * -q --log-level ERROR

Replace Config.Verbose and Config.Quiet with Config.LogLevel

Fixes #739 Fixes #1108 Fixes #1000
 2017-02-11 20:22:42 +00:00
Nick Craig-Wood 0366ea39c5 Reassign some logging levels 2017-02-11 17:56:05 +00:00
Nick Craig-Wood 80f53176d9 Rename log functions and factor into own file 2017-02-11 17:54:50 +00:00
Nick Craig-Wood 40c02989f1 acd: Fix panic on token expiry - fixes #1117 2017-02-11 17:49:59 +00:00
Nick Craig-Wood 50e190ff54 cat: don't allocate buffers if not needed to reduce memory usage 2017-02-09 11:46:53 +00:00
Nick Craig-Wood dd20a297d6 cat: Fix go routine leak 2017-02-09 11:25:36 +00:00
Nick Craig-Wood c0ad29c06c Clarify logging and docs for --no-traverse incompatibilities - fixes #1059 2017-02-08 22:35:12 +00:00
Nick Craig-Wood d091d4a8bb rclone cat: add --head, --tail, --offset, --count and --discard 
Fixes #819
 2017-02-08 08:09:41 +00:00
Nick Craig-Wood 381b845307 acd: Fix nil pointer deref after Move #1098 
Don't attempt to read the info in moveNode as there are paths which
don't, read it again from the directory afterwards.
 2017-02-04 12:56:21 +00:00
Nick Craig-Wood 48cdedc97b Re-implement sync routine to work a directory at a time 
Multiple directories (up to --checkers worth) are scanned at once.

This uses much less memory than the previous scheme - only the amount
of memory needed to hold an entire directory listing of objects.

For directory based remotes the speed is unchanged.

For bucket based remotes, instead of doing one API call to list the
whole bucket, it does multiple calls, one for each pseudo directory.
However these are done in parallel so in practice this seems to speed
up directory listings.

This replaces the existing sync method as it performs faster and uses
less memory.

The old sync method is available with the temporary --old-sync-method
flag.

Fixes #517
Fixes #439
Fixes #236
Fixes #1067
 2017-02-04 10:30:25 +00:00
Nick Craig-Wood 7c6cd3a9e1 Make --delete-after the default and refactor --delete-{before,during,after} parsing 2017-02-04 10:30:25 +00:00
Nick Craig-Wood bcdd73369f Ignore --delete-before with --track-renames 2017-02-04 10:30:25 +00:00
Nick Craig-Wood 86bec20b56 sync: factor accumulating the rename checks 2017-02-04 10:30:25 +00:00
Nick Craig-Wood c3b2b89473 Add ListDirSorted function to list a directory 
* fix error return of readFilesFn also
 2017-02-04 10:30:25 +00:00
Nick Craig-Wood 85f05c57d1 Clean empty directories between test runs 2017-02-04 10:30:25 +00:00
Nick Craig-Wood 16d91246c4 sftp: Fix remote race on creating directories 
Because there is a period of time between checking a directory needs
creating and creating it, the leads to errors where directories are
attempting to be created twice.

Add locking on a per directory basis to fix while doing mkdir.
 2017-02-04 10:29:46 +00:00
Nick Craig-Wood 726cb43be9 Complete SFTP remote #521 
* Add unit tests
  * Fix up remote so it passes tests
  * Add docs
 2017-02-04 10:29:46 +00:00
Nick Craig-Wood 288302c2cf Make fallback purge delete empty directories too. 
This was implemented to make the SFTP unit tests pass.
 2017-02-04 10:29:46 +00:00
Nick Craig-Wood 609671aabc Add Jack Schmidt to contributors 2017-02-04 10:29:46 +00:00
Nick Craig-Wood 9d331ce04b Implement --ignore-checksum flag 
Fixes #793 Fixes #863 Fixes #981
 2017-02-03 08:11:10 +00:00
Nick Craig-Wood 916569102c b2: constrain memory usage when doing multipart uploads #439 
Each part of a multipart upload takes 96M of memory, so we make sure
that we don't user more than `--transfers` * 96M of memory buffering
the multipart uploads.

This has the consequence that some uploads may appear to be at 0% for
a while, however they will get going eventually so this won't
re-introduce #731.
 2017-02-03 08:03:04 +00:00
Nick Craig-Wood 28f9b9b611 drive: detect files using file size as well as md5 - fixes 980 2017-02-03 08:00:03 +00:00
Nick Craig-Wood 7679620f4b drive: Experimentally add --drive-list-chunk 2017-02-02 21:49:02 +00:00
Nick Craig-Wood 8a11da4e14 mount: Make fsync be a no-op for direectories too #1045 2017-02-02 21:31:41 +00:00
Nick Craig-Wood f11867d810 Add Jon Yergatian to contributors 2017-02-02 21:14:24 +00:00
Nick Craig-Wood 37fe6d56e5 mount: fix docs for umount flags - fixes #1036 2017-01-30 18:17:16 +00:00
Nick Craig-Wood ff8f11d79c Add Károly Oláh to contributors 2017-01-29 20:55:21 +00:00
Nick Craig-Wood 74702554da onedrive: use token renewer to stop auth errors on long uploads 
Fixes #820
 2017-01-29 20:45:45 +00:00
Nick Craig-Wood bd29015022 Factor token renewer from amazonclouddrive to oauthutil 2017-01-29 20:45:44 +00:00
Nick Craig-Wood 2192805360 rclone config: when choosing from a list, allow the value to be entered 2017-01-29 15:51:26 +00:00
Nick Craig-Wood db0b93c0ad rclone config: allow rename and copy of remotes - fixes #641 2017-01-29 15:37:44 +00:00
Nick Craig-Wood 94947f2523 Implement -L, --copy-links flag to allow rclone to follow symlinks 
Fixes #40
 2017-01-29 13:43:20 +00:00
Nick Craig-Wood 29c6e22024 mount: Make fsync be a no-op rather than returning an error - fixes #1045 2017-01-29 11:29:42 +00:00
Nick Craig-Wood 390f3cf35b crypt: Add --crypt-show-mapping to show encrypted file mapping 
Fixes #1004
 2017-01-29 10:14:17 +00:00
Nick Craig-Wood 20c033b484 Add a writing documentation section and update document 2017-01-29 09:47:47 +00:00
Nick Craig-Wood 8068ef96b6 Add Dario Giovannetti to contributors 2017-01-26 10:29:21 +00:00
Nick Craig-Wood 9fdeb82328 Fix tests under Windows 2017-01-20 17:12:05 +00:00
Nick Craig-Wood 2abfae283c crypt: fix crypt writer getting stuck in a loop #902 
This happened when the underlying reader returned io.ErrUnexpectedEOF.
The error handling for the call to io.ReadFull failed to take this
into account.

io.ErrUnexpectedEOF is reasonably common when SSL connections go wrong
when communicating with ACD, so it manifested itself as transfers from
non-encrypted ACD to encrypted ACD getting stuck.
 2017-01-20 16:00:55 +00:00
Nick Craig-Wood b6848a3edb Fix race in Lister.Finished which was causing the tests to be unreliable 2017-01-19 20:11:17 +00:00
Nick Craig-Wood e2bf9eb8e9 Implement --suffix for use with --backup-dir only #98 
This also makes sure we remove files we are about to override in the
--backup-dir properly.
 2017-01-19 20:11:17 +00:00
Nick Craig-Wood a77659e47d Make directory listing checks more reliable and easier to read 2017-01-19 20:11:17 +00:00
Nick Craig-Wood e9da14ac2e acd: After moving a file, wait for the file to no longer be in the directory 
This fixes a Move followed quickly by a Copy updating the wrong file.
 2017-01-19 20:11:17 +00:00
Nick Craig-Wood a4bf22e620 b2: fix upload url not being refreshed properly #825 2017-01-17 17:34:21 +00:00
Nick Craig-Wood a6b4065e13 mount: fix retry on network failure when reading off crypt - fixes #1042 2017-01-17 16:32:04 +00:00
Nick Craig-Wood 07ebf35987 Clarify what happens to files already in the --backup-dir DIR 2017-01-16 19:26:56 +00:00
Nick Craig-Wood 47ebd0789c Make "make quicktest" ignore a config file if present for local running 
This means "make quicktest" should give the same result as when run by
Travis.
 2017-01-16 17:54:18 +00:00
Nick Craig-Wood 166fd50451 oauthutil: copy the config before modifying it 
The stops simultaneous use of oauth configs with different client IDs
causing a problem.
 2017-01-16 17:33:25 +00:00
Nick Craig-Wood 0604d3dbf2 acd, onedrive: make sure we have found the root before purging 
If we don't, purge can try to trash the root node which fortunately
doesn't succeed.
 2017-01-16 17:33:25 +00:00
Nick Craig-Wood 1fa258c2b4 Define a new Features() method for Fs 
Optional interfaces are becoming more important in rclone,
--track-renames and --backup-dir both rely on them.

Up to this point rclone has used interface upgrades to define optional
behaviour on Fs objects.  However when one Fs object wraps another it
is very difficult for this scheme to work accurately.  rclone has
relied on specific error messages being returned when the interface
isn't supported - this is unsatisfactory because it means you have to
call the interface to see whether it is supported.

This change enables accurate detection of optional interfaces by use
of a Features struct as returned by an obligatory Fs.Features()
method.  The Features struct contains flags and function pointers
which can be tested against nil to see whether they can be used.

As a result crypt and hubic can accurately reflect the capabilities of
the underlying Fs they are wrapping.
 2017-01-16 17:33:25 +00:00
ncw 3745c526f1 Implement --backup-dir - fixes #98 
The parameter of backup-dir specifies a remote that all deleted or
overwritten files will be copied to.
 2017-01-16 17:33:25 +00:00
Nick Craig-Wood c123c702ab Fix fs.Overlapping and factor fs.SameConfig 2017-01-14 09:55:53 +00:00
ncw 4aae7bcca6 Factor server side move detection 2017-01-14 09:55:53 +00:00
Nick Craig-Wood aa62e93094 acd: fix panic when renaming files - fixes #973 
Fixed by no longer overwriting the parameters in a retry loop
 2017-01-14 09:50:45 +00:00
Nick Craig-Wood 45862f4c16 Add Brandur to contributors 2017-01-12 10:08:40 +00:00
Nick Craig-Wood a7d8ccd265 Add T.C. Ferguson to contributors 2017-01-10 13:22:51 +00:00
Nick Craig-Wood e426cb1d1a Add emyarod to contributors 2017-01-10 13:13:14 +00:00
Nick Craig-Wood 35a6436983 mount: implement proper directory handling (mkdir, rmdir) 
Before this change mount only simulated rmdir & mkdir, now it actually
runs mkdir & rmdir on the underlying remote, using the new parmaeters
to fs.Mkdir and fs.Rmdir.

Fixes #956
 2017-01-06 11:24:22 +00:00
Nick Craig-Wood 341745d4d5 Update docs on server side copy 2017-01-05 21:11:46 +00:00
Nick Craig-Wood 78c1f2839e Fix filters to add ** rules to directory rules 
This fixes `--exclude ".*{,/**}"` to exclude all . files and
. directories.
 2017-01-05 19:33:49 +00:00
Nick Craig-Wood de2d967abd Stop --track-renames hashing matching files - fixes #984 
Also only hash files of the correct size.

This speeds it up a lot.
 2017-01-05 17:58:01 +00:00
Nick Craig-Wood e1a49ca426 Document environment variable usage 2017-01-04 21:38:54 +00:00
Nick Craig-Wood f73ee5eade Make all config file variables be settable in the environment 
These are set in the form RCLONE_CONFIG_remote_option where remote is
the uppercased remote name and option is the uppercased config file
option name.  Note that RCLONE_CONFIG_remote_TYPE must be set if
defining a new remote.

Fixes #616
 2017-01-03 22:42:47 +00:00
Nick Craig-Wood 0d75d2585f Allow all options to be set from environment variables 
The option names are munged changing - to _ making upper case and
prepending RCLONE_.  The values are as parsed by pflag.
 2017-01-03 22:42:47 +00:00
Nick Craig-Wood aaeab58ce6 Add Lukas Loesche to contributors 2017-01-03 20:49:04 +00:00
Nick Craig-Wood f1221b510b Change --track-renames to use the length,hash pair stored in a map 
This makes it much faster in the case of many files and use less
memory.

This also detects use of --no-traverse and disables it.
 2017-01-03 20:37:06 +00:00
Nick Craig-Wood 274ab349f4 sync: Only allow --track-renames if have a common hash 2017-01-03 20:35:05 +00:00
Nick Craig-Wood 86392fb800 Add Bjørn Erik Pedersen to contributors 2017-01-03 20:35:05 +00:00
Nick Craig-Wood 5c89fd679d Fix incorrect vendoring for swift library 
(vendored a feature branch by accident)
 2017-01-03 17:39:56 +00:00
Nick Craig-Wood 1cad759306 Update vendor directory 2017-01-02 16:12:05 +00:00
Nick Craig-Wood 5b8b379feb Version v1.35 2017-01-02 15:33:06 +00:00
Nick Craig-Wood f538fd8eb4 Update RELEASE procedure 2017-01-02 14:38:14 +00:00
Nick Craig-Wood 4dd5428b13 Fix rmdirs test and integration tests which depend on each other 2017-01-02 14:15:07 +00:00
Nick Craig-Wood 64ec220d5d Fix --no-update-modtime test on remotes which don't support hashes 2016-12-31 15:19:26 +00:00
Nick Craig-Wood cbfec0d281 Fix tests for missing config file 2016-12-20 15:05:08 +00:00
Nick Craig-Wood 80c044f2d3 Stop overwriting global remote in tests 2016-12-20 14:15:11 +00:00
Nick Craig-Wood 1b2dda8c4c oauthutil: Reload config file off disk before updating token 
This fixes the config file being overwritten when two rclones are running.

Fixes #682
 2016-12-19 15:04:07 +00:00
Nick Craig-Wood 473bdad00b crypt: Prevent the user pointing crypt at itself - fixes #927 
This would hopefully have stopped the issues reported in #784 & #929
 2016-12-19 14:09:59 +00:00
Nick Craig-Wood 4482e75f38 Fix golint 2016-12-15 21:02:41 +00:00
Nick Craig-Wood 43c530922a Restore ability for any command to show stats by adding --stats flag 
Make default for `mount` command not to show stats - this can be
re-enabled by adding a `--stats` parameter.
 2016-12-15 17:40:17 +00:00
Nick Craig-Wood dd60f088ed mount: retry reads on error #873 2016-12-15 17:16:55 +00:00
Nick Craig-Wood 0117aeafbf mount: this removes the async buffering as it was killing seek performance 2016-12-15 17:08:52 +00:00
Nick Craig-Wood 442861581a Update release process - fixes #855 2016-12-14 17:49:26 +00:00