f-denkena/rclone
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
675 Commits 1 Branch 0 Tags
f6a053df6e0ec83a8a4fce9db436285d99cafba5
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick Craig-Wood f6a053df6e Automatically set --no-traverse when copying a single file 2016-06-29 17:38:56 +01:00
Nick Craig-Wood af4ef8ad8d Implement --no-traverse flag to stop copy traversing the destination remote. 
Refactor sync/copy/move
  * Don't load the src listing unless doing a sync and --delete-before
  * Don't load the dst listing if doing copy/move and --no-traverse is set

`rclone --no-traverse copy src dst` now won't load either of the
listings into memory so will use the minimum amount of memory.

This change will reduce the amount of memory rclone uses dramatically
too as in normal operations (copy without --notraverse or sync) as it
no longer loads the source file listing into memory at all.

Fixes #8
Fixes #544
Fixes #546
 2016-06-29 17:38:50 +01:00
Nick Craig-Wood 13797a1fb8 Make retry logs be debug in main copy routine 2016-06-28 08:51:57 +01:00
Nick Craig-Wood 3ad8fb8634 Make DeleteFile and DeleteFiles return errors 2016-06-28 08:51:57 +01:00
Nick Craig-Wood ab43005422 Make NewObject return an error 
* make it return an error
  * make a canonical error fs.ErrorNotFound
  * make a test for it
  * remove logs/debugs of error
 2016-06-28 08:51:57 +01:00
Nick Craig-Wood b1f131964e Rename NewFsObject to NewObject 2016-06-28 08:51:57 +01:00
Nick Craig-Wood 1a87b69376 Get rid of LimitedFs - FIXME needs docs on copying single files 
If remote:path points to a file make NewFs return a sentinel error
fs.ErrorIsFile and an Fs which points to the parent.

Use this to remove the LimitedFs and just add this file to the
--files-from list.

This means that server side operations can be used also.

Fixes #518
Fixes #545
 2016-06-28 08:51:43 +01:00
Nick Craig-Wood 5a3b109e25 Fix issues identified by go vet -shadow - fixes #530 2016-06-21 21:17:52 +01:00
Nick Craig-Wood a67c7461ee s3: skip SetModTime for objects > 5GB - fixes #534 2016-06-19 17:26:44 +01:00
Nick Craig-Wood ab0947ee37 Fix typo in changelog 2016-06-18 16:58:37 +01:00
Nick Craig-Wood bd0227450e Version v1.30 2016-06-18 16:41:46 +01:00
Nick Craig-Wood f438f1e9ef Fix stats print 2016-06-18 16:41:46 +01:00
Nick Craig-Wood 3f7b2c1ade Add Justin R. Wilson to contributors 2016-06-18 14:31:17 +01:00
Nick Craig-Wood d3dd672640 Document recursion requirements for Fses 2016-06-18 14:12:47 +01:00
Nick Craig-Wood 2a46be8cf3 b2: implement large file uploading - fixes #456 2016-06-18 13:38:05 +01:00
Nick Craig-Wood 1b4370bde1 Rework retry logic when copying objects 
* Fix off by one retry logic - fixes #406
  * Retry any retriable errors
  * Restructure code
 2016-06-18 10:55:58 +01:00
Nick Craig-Wood cc6a776034 drive, acd: Tweak logging after changing Fs.Put so that it must cope with existing files 2016-06-18 10:54:42 +01:00
Nick Craig-Wood 2cfb3834f2 Log errors with %v 2016-06-18 09:36:47 +01:00
Nick Craig-Wood 46135d830e Add --ignore-size flag - fixes #399 2016-06-17 17:20:08 +01:00
Nick Craig-Wood 318e42e35b Add a section on quoting in the shell to the docs - fixes #473 2016-06-17 16:28:50 +01:00
Nick Craig-Wood c7f04e24d3 Document that you can't repeat filter flags - fixes #506 2016-06-17 16:06:21 +01:00
Nick Craig-Wood e4650eff58 drive: fix retry of multipart uploads - fixes #520 
Reset the reader on retry otherwise it is empty when read again.
 2016-06-15 21:48:30 +01:00
Nick Craig-Wood 869d91269d Debug cause of low level retries 2016-06-15 21:48:14 +01:00
Nick Craig-Wood df1092ef33 Change Fs.Put so that it must cope with existing files 
This should fix duplicate files on drive and 409 errors on
amazonclouddrive however it will slow down the upload slightly as
another roundtrip will be needed.

None of the other Fses needed adjusting.

Fixes #483
 2016-06-13 19:29:10 +01:00
Nick Craig-Wood 4c5b2833b3 Convert to using github.com/pkg/errors everywhere 2016-06-13 17:43:03 +01:00
Nick Craig-Wood 7fe653c350 Unwrap errors properly for patform specific connection retry code. 
Include more possible errors for Windows.

For #442
 2016-06-10 13:48:41 +01:00
Nick Craig-Wood 661715733a Make sure we don't use conflicting content types on upload - fixes #513 2016-06-09 17:52:58 +01:00
Nick Craig-Wood f17cb1bf50 Fix retry of Windows wsaend errors #442 
Make the test for wsaend error less specific
 2016-06-09 15:34:13 +01:00
Nick Craig-Wood 9ec06df79f Be explicit about which arch we support which fixes failure to build with new gox 2016-06-09 15:33:26 +01:00
Nick Craig-Wood 67d0375b98 Audit use of log.Print and change to Debug, Log, or ErrorLog as appropriate 2016-06-06 21:23:54 +01:00
Nick Craig-Wood 4882b8ba67 Tweak website footer 2016-06-06 21:23:22 +01:00
Nick Craig-Wood 108760e17b Log -v output to stdout by default - fixes #228 2016-06-04 18:49:27 +01:00
Nick Craig-Wood f15e7e89d2 Add version string to debug startup message 2016-06-03 23:08:14 +01:00
Nick Craig-Wood e2788aa729 Display the transfer stats in more human readable form - fixes #428 2016-06-03 22:49:50 +01:00
Nick Craig-Wood 772f99fd74 Make SizeSuffix output without b suffix for more useful printouts 2016-06-03 22:49:14 +01:00
Nick Craig-Wood 9bbcdeefd0 Start the logger earlier so all messages go there - fixes #486 2016-06-03 22:08:27 +01:00
Nick Craig-Wood a21cc161de Make 0 size files specifiable with --max-size 0b - fixes #450 2016-06-03 21:54:27 +01:00
Nick Craig-Wood e818b7c206 Represent -1 as "off" for SIZE values 2016-06-03 21:51:39 +01:00
Nick Craig-Wood 5723d788a4 Add b suffix so we can specify bytes in --bwlimit, --min-size etc 
Fixes #449
 2016-06-03 21:16:48 +01:00
Nick Craig-Wood 1d6698a754 Build tweaks - fixes #484 
* disable CGO for static builds everywhere
  * override Version in release build script
  * don't output symbol table in release binaries
 2016-06-03 20:34:19 +01:00
Nick Craig-Wood 1fce83b936 swift: add auth version parameter - fixes #407 2016-06-03 17:52:24 +01:00
Nick Craig-Wood ccdd1ea6c4 Add --max-depth parameter 
This will apply to ls/lsd/sync/copy etc

Fixes #412
Fixes #213
 2016-06-03 17:05:39 +01:00
Nick Craig-Wood 348734584b Try OS X 10.11 to fix travis build 2016-05-30 20:32:35 +01:00
Nick Craig-Wood c6a79ff72d Fix remaining places in listing where we were logging errors not returning them 2016-05-30 19:51:15 +01:00
Nick Craig-Wood b6f1391da3 Fix new style directory listing on windows 2016-05-30 19:44:15 +01:00
Nick Craig-Wood ce94c0e729 Update go versions in travis 2016-05-28 20:45:25 +01:00
Nick Craig-Wood 58befe280c Fix directory name normalisation on OS X 2016-05-28 20:23:37 +01:00
Nick Craig-Wood 4c0f4ccb65 Fix destination of Facebook share link - fixes #499 2016-05-28 17:27:25 +01:00
Nick Craig-Wood 085677d511 acd: Work around spurious 403 errors 
Sometimes ACD gives this error on reauthentication

HTTP code 403: "403 Forbidden", reponse body: {"message":"Authorization header requires 'Credential' parameter. Authorization header requires 'Signature' parameter. Authorization header requires 'SignedHeaders' parameter. Authorization header requires existence of either a 'X-Amz-Date' or a 'Date' header. Authorization=Bearer"}

This code retries this error if it is received.
 2016-05-28 16:49:26 +01:00
Nick Craig-Wood 0a922ad1dc acd: Reauth on 401 errors 
Fixes #493
Fixes #501
 2016-05-28 16:49:26 +01:00
Nick Craig-Wood 83c3bb2f1a Add Romain Lapray to contributors 2016-05-28 16:39:17 +01:00
Nick Craig-Wood cadf202107 Clarify filtering docs #489 2016-05-19 12:39:16 +01:00
Nick Craig-Wood 36700d36a7 Fix dropbox root directory listings 2016-05-16 17:54:59 +01:00
Nick Craig-Wood ad85f6e413 Implement directory include filtering for efficiency 
Fixes #395
 2016-05-16 17:14:04 +01:00
Nick Craig-Wood 536526cc92 amazonclouddrive: Restart directory listings on error - fixes #475 
Before this change rclone would retry only the page that was missing
from the directory listing.  However it turns out that on 429 errors
at least, that page is gone from the directory listing which results
in missing files in the list.  The workaround for this is to restart
the directory listing on any retryable errors.
 2016-05-14 17:15:42 +01:00
Nick Craig-Wood ac9c20b048 Make IsRetryError function 2016-05-14 17:11:19 +01:00
Nick Craig-Wood 2db35f0ce7 Dump out unexpected state in integration test 2016-05-07 21:19:26 +01:00
Nick Craig-Wood dbfa7031d2 Factor Lister into own file, write tests and fix 2016-05-07 17:17:43 +01:00
Nick Craig-Wood c2d0e86431 Add more tests for List() and fix resulting problems 2016-05-07 14:50:35 +01:00
Nick Craig-Wood 68ec6a9f5b Add a directory parameter to Fs.List() 2016-05-06 16:52:34 +01:00
Nick Craig-Wood 753b0717be Refactor the List and ListDir interface 
Gives more accurate error propagation, control of depth of recursion
and short circuit recursion where possible.

Most of the the heavy lifting is done in the "fs" package, making file
system implementations a bit simpler.

This commit contains some code originally by Klaus Post.

Fixes #316
 2016-05-06 16:52:34 +01:00
Nick Craig-Wood 3bdad260b0 Fix typo (thanks Saverio Proto) 2016-05-06 14:09:12 +01:00
Nick Craig-Wood d205dc23e9 Fix oddities using a file in the root - fixes #471 
* Check return from NewFsObject which caused nil ptr deref
  * Correct root directory from "" to string(os.PathSeparator) in getDirFile
 2016-05-06 13:52:50 +01:00
Nick Craig-Wood bdd26d71b2 Clarify swift errors - fixes #460 2016-05-02 12:34:15 +01:00
Nick Craig-Wood 8b2f6faf18 Re-enable OS X in travis tests 2016-05-01 13:13:20 +01:00
Nick Craig-Wood 1752ee3c8b Retry errors which indicate the connection closed prematurely. 
See discussion in #442
 2016-04-29 17:29:34 +01:00
Nick Craig-Wood 5c2d8ffe33 Retry only the failing tests in the integration tests 2016-04-26 10:20:07 +01:00
Nick Craig-Wood 7fecd5c8c6 Add Leigh Klotz to contributors 2016-04-22 21:12:45 +01:00
Nick Craig-Wood b053234eb1 Add Fabian Ruff to contributors 2016-04-22 21:02:54 +01:00
Nick Craig-Wood 8af68e779f Add Michal Witkowski to contributors 2016-04-22 20:09:16 +01:00
Nick Craig-Wood 3a1198cac5 gcs: Don't configure the oauth token if service_account_file is supplied 2016-04-22 20:07:10 +01:00
Nick Craig-Wood 17aac9b15f Note certificates FAQ works on Solaris too 2016-04-22 11:53:56 +01:00
Nick Craig-Wood 70496c15e1 Add Jim Tittsler to contributors 2016-04-21 19:37:41 +01:00
Nick Craig-Wood bb75d80d33 Fix frontmatter 2016-04-18 18:55:07 +01:00
Nick Craig-Wood 157d7d45f5 Version v1.29 2016-04-18 18:30:29 +01:00
Nick Craig-Wood b5cba73cc3 Make test more reliable 2016-04-18 17:48:52 +01:00
Nick Craig-Wood dd36264aad Add FAQ All my uploaded docx/xlsx/pptx files appear as archive/zip 
Fixes #417
 2016-04-12 21:41:24 +01:00
Nick Craig-Wood ddb47758f3 drive: increase default chunk size to 8 MB and document - fixes #397 2016-04-12 21:33:55 +01:00
Nick Craig-Wood 9539bbf78a Fix appveyor build after vet removal from tools repo 2016-04-07 20:07:00 +01:00
Nick Craig-Wood 0f8e7c3843 Make rclone check obey the --size-only flag - fixes #419 2016-04-07 15:01:45 +01:00
Nick Craig-Wood b835330714 Use "application/octet-stream" if mime.TypeByExtension returns invalid type 
Fixes #424
 2016-04-07 14:32:01 +01:00
Nick Craig-Wood 310db14ed6 Notes on --transfers and B2 2016-04-04 17:58:36 +01:00
Nick Craig-Wood 6cc9c09610 drive: preserve mime type on file update - fixes #417 2016-04-04 16:58:42 +01:00
Nick Craig-Wood 93c60c34e1 b2: Fix incorrect value of Precision - should be 1ms not 1s 2016-03-24 15:23:27 +00:00
Nick Craig-Wood ced3a4bc19 Implement -I, --ignore-times for unconditional upload - fixes #311 2016-03-22 17:02:27 +00:00
Nick Craig-Wood cb22583212 b2: Enable mod time syncing - fixes #348 2016-03-22 15:56:44 +00:00
Nick Craig-Wood 414b35ea56 Change the interface of SetModTime to return an error - #348 2016-03-22 15:56:44 +00:00
Nick Craig-Wood f469905d07 dropbox: Note 10,000 files limitation on purge - fixes #374 2016-03-22 14:46:43 +00:00
Nick Craig-Wood 20f4b2c91d b2: update API to new version - fixes #393 
* Make reading mod time and SHA1 much more efficient
    * removes an HTTP transaction to increase speed
  * Reduce memory usage of the objects
 2016-03-22 14:39:56 +00:00
Nick Craig-Wood 37543bd1d9 b2: Fix parsing of mod time when not in metadata 
This files this error `Failed to parse mod time string "":
"src_last_modified_millis" not found in metadata`.
 2016-03-22 10:26:37 +00:00
Nick Craig-Wood 0dc0052e93 Note that filters must use / not \ - #394 2016-03-19 17:40:54 +00:00
Nick Craig-Wood bd27473762 swift: Don't return an MD5SUM for static large objects - #392 
* rename isManifest to isDynamicLargeObject for clarity
 2016-03-17 17:36:20 +00:00
Nick Craig-Wood 9dccf91da7 swift/hubic: document segmented object MD5SUM limitations - fixes #392 2016-03-16 17:39:44 +00:00
Nick Craig-Wood a1323eb204 s3: Fix uploading files bigger than 50GB - fixes #386 2016-03-10 16:48:55 +00:00
Nick Craig-Wood fdd4b4ee22 drive: Add missing retries for Move and DirMove 2016-03-06 18:15:01 +00:00
Nick Craig-Wood 8ef551bf9c Make dedupe remove identical copies without asking and add non interactive mode - fixes #338 
* Now removes identical copies without asking
  * Now obeys `--dry-run`
  * Implement `--dedupe-mode` for non interactive running
    * `--dedupe-mode interactive` - interactive the default.
    * `--dedupe-mode skip` - removes identical files then skips anything left.
    * `--dedupe-mode first` - removes identical files then keeps the first one.
    * `--dedupe-mode newest` - removes identical files then keeps the newest one.
    * `--dedupe-mode oldest` - removes identical files then keeps the oldest one.
    * `--dedupe-mode rename` - removes identical files then renames the rest to be different.
  * Add tests which will only run on Google Drive.
 2016-03-06 18:15:01 +00:00
Nick Craig-Wood 2119fb4314 drive: tweak pacer to speed up directory listings and make more reliable 2016-03-06 18:15:01 +00:00
Nick Craig-Wood 0166544319 Add Attack constant to pacer 2016-03-05 20:29:05 +00:00
Nick Craig-Wood 874a64e5f6 A script to make a directory heirarchy for testing 2016-03-05 20:26:15 +00:00
Nick Craig-Wood e0c03a11ab Commit missing docs changes and adjust RELEASE.md to make sure it doesn't happen again 2016-03-01 17:42:27 +00:00