f-denkena/rclone
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,385 Commits 1 Branch 0 Tags
ec6c3f2686a8b053e18d8531b726465e762b6e35
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick Craig-Wood ec6c3f2686 vendor: remove github.com/tsenart/tb 2017-07-16 16:14:44 +01:00
Nick Craig-Wood 62e28d0a72 Replace token bucket limiter github.com/tsenart/tb with golang.org/x/time/rate 
In tests tsenart/tb has proved inaccurate at low rates.
 2017-07-16 16:14:44 +01:00
Nick Craig-Wood 470642f2b7 vendor: add vendor/golang.org/x/time/rate 2017-07-14 05:35:00 +01:00
Nick Craig-Wood b5002eb6a4 drive: document google docs sometimes fail to download 2017-07-10 23:15:30 +01:00
Nick Craig-Wood ee5698b3a9 drive: Add docs on duplicated files, and re-copying 2017-07-09 23:32:34 +01:00
Nick Craig-Wood 728ff231ab Link wiki from main website - fixes #1156 2017-07-09 22:48:52 +01:00
Nick Craig-Wood 542f938ce2 website: Decrease spacing between menu items 
...as they were overflowing the page before.  Thanks to Amy Craig-Wood
for CSS wrangling!
 2017-07-09 22:48:26 +01:00
Nick Craig-Wood e24d0ac94d Add slack invite to website menu - fixes #1145 2017-07-08 22:30:35 +01:00
Nick Craig-Wood da2e2544ee Fix tests on Windows 2017-07-08 16:26:41 +01:00
Nick Craig-Wood 72add5ab27 sync: state whether duplicates are objects are directories 2017-07-08 15:42:18 +01:00
Nick Craig-Wood 9ac72ee53f Make commit number in beta version tag be 3 digits always 2017-07-07 21:31:52 +01:00
Nick Craig-Wood c3dac2e385 dropbox: fix large directory listings 2017-07-07 21:20:07 +01:00
Nick Craig-Wood 92294a4a92 drive: Add --drive-trashed-only and remove obsolete --drive-full-list 
* Add --drive-trashed-only to show only the contents of the trash
  * Remove --drive-full-list as it is obsolete
  * Tidy the docs for the drive options
 2017-07-06 15:32:57 +01:00
Nick Craig-Wood 69ff009264 Use a stable sort for sorting directory entries 
This is useful if there are duplicates. Assuming the remote delivers
the entries in a consistent order, this will give the best user
experience in syncing as it will consistently use the first entry for
the sync comparison.
 2017-07-06 14:07:26 +01:00
Nick Craig-Wood 27b157580e Move make_test_files.go into bin 2017-07-06 11:54:57 +01:00
Nick Craig-Wood 3f288bc9ea Added decrypt_names.py to help decoding encrypted logs 2017-07-06 11:53:39 +01:00
Nick Craig-Wood ce1b9a7daf swift,hubic: fix paged directory listings 
This was caused by rclone adjusting the object names.  If the last
object in the listing page happened to be a directory, rclone would
remove the / which caused the next page to start in the wrong place.
 2017-07-06 11:31:37 +01:00
Nick Craig-Wood f0512d1a52 Fix missing fs.Dir -> fs.Directory 2017-07-06 11:31:36 +01:00
Nick Craig-Wood e86f62c3e8 Add rclone info internal command for testing out limits of remotes 2017-07-03 15:05:27 +01:00
Nick Craig-Wood 6c3bf629a1 yandex: fix fs.Name() 
Put in tests for fs.Root() and fs.Name() for all remotes
 2017-07-03 13:39:31 +01:00
Nick Craig-Wood 575e779b55 Warn about duplicate files when syncing - fixes #1506 
Error about unsorted directories and test thoroughly
 2017-06-30 21:24:13 +01:00
Nick Craig-Wood dc56ad9816 sftp, local: refactor to stop storing os.FileInfo in preparation for serialization 2017-06-30 14:27:27 +01:00
Nick Craig-Wood e7d04fc103 Create fs.Directory interface and use it everywhere 2017-06-30 14:26:59 +01:00
Nick Craig-Wood e2d7d413ef fs: rename BasicInfo to DirEntry 2017-06-30 14:26:58 +01:00
Nick Craig-Wood e7e9aa0dfa fs: Remove unused ListFser interface 2017-06-30 14:26:58 +01:00
Nick Craig-Wood f88300a153 Don't Mkdir at the start of sync - fixes #1131 
This is possible now that the bucket based remotes will create the
buckets on demand (9c1e703777).
 2017-06-29 12:31:53 +01:00
Nick Craig-Wood e54087ece1 Fix config tests to save configData which fixes subsequent tests 2017-06-29 12:31:53 +01:00
Nick Craig-Wood 54561fd2bc s3: work around eventual consistency in bucket creation 
Deleting a bucket then testing its existence can give the wrong
result.  Work around by keeping a flag as to whether we have deleted
the bucket.
 2017-06-29 12:31:52 +01:00
Nick Craig-Wood 479c5a514a swift, s3, gcs: create container if necessary on server side copy 2017-06-28 21:16:07 +01:00
Nick Craig-Wood f3c7e1a9dd Debug directory creation and removal - fixes #1192 2017-06-27 22:19:35 +01:00
Nick Craig-Wood 70b5b2f5c6 acd, onedrive: fix initialization order for token renewer - fixes #1442 2017-06-27 22:19:35 +01:00
Nick Craig-Wood aa20486485 Add --stats-log-level so can see --stats without -v - fixes #1180 
The most common use for this flag is likely to be showing the stats
without using -v by using `--stats-log-level NOTICE`.
 2017-06-26 22:50:37 +01:00
Nick Craig-Wood 33f302a06b Document workaround for files/dirs with : in - fixes #1331 2017-06-26 16:13:12 +01:00
Nick Craig-Wood 24cb739d1f b2: reduce minimum chunk size to 5MB - fixes #1289 2017-06-26 16:02:46 +01:00
Nick Craig-Wood f0abd6173d Add Harshavardhana and sainaen to contributors 2017-06-26 12:37:00 +01:00
Nick Craig-Wood b360527931 mount: fix hang on errored upload 
In certain circumstances if an upload failed then the mount could hang
indefinitely. This was fixed by closing the read pipe after the Put
completed.  This will cause the write side to return a pipe closed
error fixing the hang.

Fixes #1498
 2017-06-26 12:08:51 +01:00
Nick Craig-Wood d55f8f0492 sftp: add support for using ssh key files #1494 
Update docs about macOS and ssh-agent #1218
 2017-06-23 16:25:35 +01:00
Nick Craig-Wood b44d0ea088 drive: convert / in names to a unicode equivalent (／) - fixes #62 2017-06-20 21:27:14 +01:00
Nick Craig-Wood d981456ddc Add Vasiliy Tolstov to contributors 2017-06-20 21:27:14 +01:00
Nick Craig-Wood b22c4c4307 http: fix, tidy and rework ready for release 
* Fix remaining problems
  * Refactor to make testing easier and add a test suite
  * Make path parsing more robust.
  * Add single file operations
  * Add MimeType reading for objects
  * Add documentation
  * Note go1.7+ is required to build
 2017-06-20 21:27:14 +01:00
Nick Craig-Wood afc8cc550a http: Update interfaces for List/ListR/Put/Update 2017-06-20 21:27:14 +01:00
Nick Craig-Wood d5d635b7f3 http: Fix comments, remove optional methods which don't work 2017-06-20 21:27:14 +01:00
Nick Craig-Wood be0dd09801 vendor: golang.org/x/net/html for http 2017-06-20 21:27:14 +01:00
Nick Craig-Wood b76cd4abd2 Fix Range header option 2017-06-20 21:27:14 +01:00
Nick Craig-Wood 0dbf1230bc Update CONTRIBUTING with --fast-list 2017-06-20 21:27:14 +01:00
Nick Craig-Wood 4fd9570332 fs: Use an in place filter in ListDirSorted 2017-06-20 21:27:14 +01:00
Nick Craig-Wood dcce65b2b3 mount/cmount: factor duplicated code into mountlib 2017-06-19 14:36:51 +01:00
Nick Craig-Wood 4ce31555b2 vendor: update github.com/billziss-gh/cgofuse - fixes #1481 2017-06-19 09:53:34 +01:00
Nick Craig-Wood 5ed4bc97f3 travis: reduce number of parallel builds to avoid "Killed" error 2017-06-19 08:16:35 +01:00
Nick Craig-Wood 54e37be591 Only test with -race using go latest 2017-06-19 08:07:50 +01:00
Nick Craig-Wood eaa717b88a Fix crypt obfuscate tests with Windows 2017-06-18 22:53:19 +01:00
Nick Craig-Wood bbbc202ee6 Add ftp.md to docs builder and update docs 2017-06-15 20:12:26 +01:00
Nick Craig-Wood 97364fd0b6 ncdu: disable on plan9 and solaris as termbox isn't supported there 2017-06-15 20:10:54 +01:00
Nick Craig-Wood c34f11a92f rclone ncdu for exploring a remote with a text based user interface. 2017-06-15 17:44:17 +01:00
Nick Craig-Wood e31fc877e2 vendor: github.com/nsf/termbox-go and dependencies for rclone ncdu 2017-06-15 16:46:32 +01:00
Nick Craig-Wood e069fc439e crypt: use an in place filter for encrypting directory entries 2017-06-15 16:46:32 +01:00
Nick Craig-Wood 5250fcdf08 core: fix data race in walk 
This was detected by the race detector when the client of Walk() sorted entries.
 2017-06-15 16:46:32 +01:00
Nick Craig-Wood 64662bef8d Deprecate --old-sync-method it is replaced with --fast-list 
Remove old sync method code.
 2017-06-14 16:49:40 +01:00
Nick Craig-Wood 0b8d9084fc test_all: print command line so it can be cut and pasted into bash 2017-06-14 16:49:40 +01:00
Nick Craig-Wood 7be49249d3 Add lsjson command - fixes #1063 2017-06-14 16:49:40 +01:00
Nick Craig-Wood 8a6a8b9623 Change List interface and add ListR optional interface 
This simplifies the implementation of remotes.  The only required
interface is now `List` which is a simple one level directory list.

Optionally remotes may implement `ListR` if they have an efficient way
of doing a recursive list.
 2017-06-14 16:49:40 +01:00
Nick Craig-Wood 6fc88ff32e Use --fast-list flag for sync/copy/move - fixes #1277 
Redo test framework to take a -fast-list flag and test remotes with that flag.
 2017-06-14 16:49:40 +01:00
Nick Craig-Wood 50928a5027 Implement --fast-list flag. 
This is supported remotes which can do a recursive listing.  It will
use more memory.

This is related to #1277 but doesn't fix that issue yet.
 2017-06-14 16:49:40 +01:00
Nick Craig-Wood 3a431056e2 gcs, swift: increase directory listing chunk to 1000 to increase performance 2017-06-14 16:49:40 +01:00
Nick Craig-Wood 53c3e5f0ab Add placeholder support for ListR interface. 
The ListR interface will be implemented by remotes that can do a
recursive directory listing more efficiently than just recursing
through the directories.  These include the bucket based remotes.
 2017-06-14 16:49:40 +01:00
Nick Craig-Wood 0edb025257 Fixup tests with dirs vs bucket based fs 2017-06-14 16:49:40 +01:00
Nick Craig-Wood fded4dbea2 yandex: correct error return for listing empty directory 2017-06-14 16:49:40 +01:00
Nick Craig-Wood 7e20e16cff core: Implement Walk directory listing and use in place of Lister 
This is in preparation for removing the Lister code and replacing the
fundamental operation in the Fs with listing a single directory.
 2017-06-14 16:49:40 +01:00
Nick Craig-Wood 1e88f0702a dropbox: fix oauth configuration 
This was broken in c59a292719
 2017-06-14 16:46:46 +01:00
Nick Craig-Wood 68333d34a1 dropbox: make setting mod time on existing files work properly 
This is a fix left over from the v2 conversion.  Dropbox ignores the
client modification on an incoming file if it was identical to the
existing file.  This change deletes the existing file first before
re-uploading the new one.
 2017-06-13 13:58:39 +01:00
Nick Craig-Wood 740b3f6ae2 Fix problems found with ineffassign 2017-06-13 11:52:36 +01:00
Nick Craig-Wood 28fcc53e45 mount test: retry umount as it fails occasionally 
This is because of the background releasing of files which happens
after all the files are closed.
 2017-06-13 10:52:10 +01:00
Nick Craig-Wood 2ca477c57f swift: make sensible error if the user forgets the container - fixes #1470 2017-06-10 14:44:56 +01:00
Nick Craig-Wood 9a11d3efd9 Revert "Start Cat tests from 2 as onedrive doesn't support ranging from 1" 
Now that https://github.com/OneDrive/onedrive-api-docs/issues/543 is
fixed, this can be reverted.

This reverts commit 320c53eab0.
 2017-06-10 13:48:00 +01:00
Nick Craig-Wood 10d5377ed8 acd: remove revoked credentials, allow oauth proxy config and update docs 2017-06-10 12:02:34 +01:00
Nick Craig-Wood ee14efd3c2 config: fix menu selection when no remotes 2017-06-10 11:39:40 +01:00
Nick Craig-Wood b4be7d65a6 Update build to go1.8.3 2017-06-09 12:06:28 +01:00
Nick Craig-Wood 52e1bfae2a oauth: Allow auth_url and token_url to be set in the config file 
If set in the config file, these override the ones configured into the
remote.  This enables alternative oauth servers to be used for all
oauth remotes.  This can only be altered by editing the config file
for the moment.
 2017-06-08 20:35:32 +01:00
Nick Craig-Wood 9c1e703777 swift, b2, gcs, s3: Fix moveto and copyto 
We now make sure the container/bucket is created before creating any objects.
 2017-06-07 14:34:59 +01:00
Nick Craig-Wood b49821956a Fix copyto/moveto test error (see #1261) 2017-06-07 14:08:46 +01:00
Nick Craig-Wood a61ba1e7c4 moveto, copyto: report transfers and checks as per move and copy 2017-06-07 13:02:21 +01:00
Nick Craig-Wood d30cc1e119 Factor RemoteSplit into fs 2017-06-07 12:27:33 +01:00
Nick Craig-Wood 74a3dfc4e1 Fix TestHashSums 2017-06-06 23:21:47 +01:00
Nick Craig-Wood 3fe9448229 drive, acd, onedrive: Cache the directory IDs when reading the parent directory 
This makes directory listings much more efficient (one less
transaction needed) and also fixes #1439 (which was caused by having
to look up a directory name with quotes in which isn't dealt with well
by the list routine) by not doing a directory lookup at all.
 2017-06-05 12:26:30 +01:00
Nick Craig-Wood a5cfdfd233 drive: add team drive support - fixes #885 2017-06-04 22:38:29 +01:00
Nick Craig-Wood bdc19b7c8a fstests: fix -remote flag to override test target 2017-06-04 22:38:29 +01:00
Nick Craig-Wood e92cc8fe2b Add Edward Q. Bridges to contributors 2017-06-04 22:38:10 +01:00
Nick Craig-Wood b047402294 config: Fix save of temp file under Windows - fixes #1458 2017-06-01 16:38:19 +01:00
Nick Craig-Wood 7693cecd17 Add Fabian Möller to contributors 2017-06-01 16:23:48 +01:00
Nick Craig-Wood 48508cb5b7 Add Ruwbin to contributors 2017-06-01 09:03:56 +01:00
Nick Craig-Wood 71028e0f06 dropbox/dbhash: fix errcheck warning 2017-05-30 22:08:49 +01:00
Nick Craig-Wood 52e96bc0e2 dropbox: add missing dbhashsum command 
This was missed from 6381959850
 2017-05-30 19:26:06 +01:00
Nick Craig-Wood 178ff62d6a vendor: add github.com/ncw/dropbox-sdk-go-unofficial and remove github.com/stacktic/dropbox 
In due course this will become github.com/dropbox/dropbox-sdk-go-unofficial
when the fate of https://github.com/dropbox/dropbox-sdk-go-unofficial/pull/14
has been decided.
 2017-05-30 15:49:29 +01:00
Nick Craig-Wood 9d335eb5cb dropbox: add low level retries 2017-05-30 14:49:09 +01:00
Nick Craig-Wood 20da3e6352 Add options to Put, PutUnchecked and Update, add HashOption and speed up local 
* Add options to Put, PutUnchecked and Update for all Fses
  * Use these to create HashOption
  * Implement this in local
  * Pass the option in fs.Copy

This has the effect that we only calculate hashes we need to in the
local Fs which speeds up transfers significantly.
 2017-05-29 12:04:52 +01:00
Nick Craig-Wood 6381959850 dropbox: support Dropbox content hashing scheme - fixes #1302 
* add support to hashing module
  * add dbhashsum to list the hashes
  * add support to dropbox module

This means objects up and downloaded to/from Dropbox will have their
hashes checked.

Note after this change local objects are calculating MD5, SHA1 and
DBHASH which is excessive and needs to be fixed.
 2017-05-29 12:04:44 +01:00
Nick Craig-Wood 8916455e4f dropbox: implement dropbox hasher #1302 2017-05-29 12:04:34 +01:00
Nick Craig-Wood 8e214e838e dropbox: Update dropbox to use the v2 API #349 
This is feature complete with the old version but now supports modification times.
 2017-05-29 12:04:33 +01:00
Nick Craig-Wood 23acd3ce01 oauthutil: Don't expect tokens to have refresh URL 2017-05-29 12:04:33 +01:00
Nick Craig-Wood 5455d34f8c Fix ssh agent on Windows - fixes #1279 2017-05-26 10:21:07 +01:00