f-denkena/rclone
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
2,266 Commits 1 Branch 0 Tags
aa5c5ec5d3d6e0b49fa30f38800fe6788817d1d6
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick Craig-Wood aa5c5ec5d3 build: mask linter errors we can't fix 2018-05-05 17:32:41 +01:00
Nick Craig-Wood e80ae4e09c build: remove unused struct fields spotted by structcheck 2018-05-05 17:32:41 +01:00
Nick Craig-Wood 1320e84bc2 build: remove unused code spotted by the deadcode linter 2018-05-05 17:32:41 +01:00
Nick Craig-Wood cb5bd47e61 build: fix errors spotted by ineffassign linter 
These were mostly caused by shadowing err and a good fraction of them
will have caused errors not to be propagated properly.
 2018-05-05 17:32:41 +01:00
Nick Craig-Wood 790a8a9aed build: add gometalinter and gometalinter_install Makefile targets 2018-05-05 17:32:41 +01:00
Nick Craig-Wood f1a43eca4d mount: make --daemon work for macOS without CGO 2018-05-05 16:23:47 +01:00
Nick Craig-Wood 7ea68f1fc6 sftp: require go1.9+ after golang.org/x/crypto/ssh update 2018-05-05 16:23:47 +01:00
Nick Craig-Wood 6427029c4e vendor: update all dependencies 
* Update all dependencies
  * Remove all `[[constraint]]` from Gopkg.toml
  * Add in the minimum number of `[[override]]` to build
  * Remove go get of github.com/inconshreveable/mousetrap as it is vendored
  * Update docs with new policy on constraints
 2018-05-05 15:52:24 +01:00
Nick Craig-Wood 21383877df cmd: make exit code 8 for --max-transfer exceeded 2018-05-05 12:58:28 +01:00
Nick Craig-Wood f95835d613 fserrors: Look deeper into errors for Fatal/Retry/NoRetry errors. 
Before this change fatal errors which were wrapped in a system error (eg a
URLError) were not recognised as fatal errors.
 2018-05-05 12:58:28 +01:00
Nick Craig-Wood be79b47a7a sync: log when we abandon the sync due to a fatal error 2018-05-05 12:58:28 +01:00
Nick Craig-Wood be22735609 fs/accounting: fix deadlock on GetBytes 
A deadlock could occur since we have now put a mutex on GetBytes from
StatsInfo.String (s.mu) - progress (acc.statmu) and read (acc.statmu)
- GetBytes (s.mu).

Fix this by giving stringSet its own locking and excluding the call
which caused the deadlock from the mutex in StatsInfo.String.
 2018-05-05 12:58:28 +01:00
Nick Craig-Wood 1b1b3c13cd sync: add a test for aborting on max upload 2018-05-05 12:58:28 +01:00
Nick Craig-Wood 5c128272fd Implement --max-transfer flag to quit transferring at a limit #1655 2018-05-05 12:58:28 +01:00
Nick Craig-Wood d178233e74 sync,march: check the cancel context on every channel send and receive 
This fixes a deadlock on sync when all the copying channels receive a
Fatal Error.
 2018-05-05 12:58:28 +01:00
Nick Craig-Wood fe25cb9c54 drive: fix about (and df on a mount) for team drives - fixes #2288 
Before this fix team drives would return the drive quota which is
incorrect and mis-leading.

Team drives don't appear to have an API for reading the bytes used or
the quota so we now return that the quota and usage are unknown.
 2018-05-03 08:59:14 +01:00
Nick Craig-Wood f2608e2a64 Add NoLooseEnds to contributors 2018-05-01 09:43:18 +01:00
Nick Craig-Wood 50dc5fe92e Add Rodrigo to contributors 2018-04-30 17:37:43 +01:00
Nick Craig-Wood 3116249692 make sign_upload: only sign the v1.xx releases not the current ones 2018-04-30 17:29:50 +01:00
Nick Craig-Wood d049e5c680 make build_dep: make sure we update the whole command for nfpm 2018-04-30 17:29:50 +01:00
Nick Craig-Wood 1c9572aba1 Add Piotr Oleszczyk to contributors 2018-04-30 17:29:50 +01:00
Nick Craig-Wood 0479c7dcf5 add github-release to make release_dep 2018-04-28 12:38:30 +01:00
Nick Craig-Wood 55674c0bfc Start v1.41-DEV development 2018-04-28 12:37:55 +01:00
Nick Craig-Wood e4c380b2a8 Version v1.41 2018-04-28 11:46:27 +01:00
Nick Craig-Wood 74cbdea0ef Revert "copy: create (pseudo copy) empty source directories to destination" 
Unfortunately this commit attempts to create every directory rather
than just the empty ones, so will need re-working.

Removing this feature for the 1.41 release

This reverts commit 0daced29db.
 2018-04-28 10:02:32 +01:00
Nick Craig-Wood a3bf6b9c2c drive, gcs: fix service account authentication - fixes #2279 
This fixes a problem introduced in b78af517de where it would
attempt to read a non-existent service account file.
 2018-04-28 09:33:43 +01:00
Nick Craig-Wood d8e88f10cd rc: take note of the --rc-addr flag too as per the docs - fixes #2184 2018-04-26 17:00:44 +01:00
Nick Craig-Wood 849db6699d Add Richard Yang to contributors 2018-04-26 16:23:52 +01:00
Nick Craig-Wood da4a5e1fb3 docs: note that copytruncate is needed for --log-file with logrotate #2259 2018-04-26 15:30:46 +01:00
Nick Craig-Wood ae562b5a4f ftp: more workarounds for FTP servers to fix mkParentDir - fixes #2181 2018-04-26 14:58:04 +01:00
Nick Craig-Wood c01177bc28 ftp: work around strange response from box FTP server 
The Box FTP server seems to send 450 instead of 550 - work around that.

See: https://forum.rclone.org/t/using-box-com-over-ftp-problems/5313
 2018-04-26 14:58:04 +01:00
Nick Craig-Wood 9f04ce282e rc: fix setting bwlimit to unlimited 2018-04-26 12:21:29 +01:00
Nick Craig-Wood 764440068e filter: fix --min-age and --max-age together check 
Somehow in the code reorganisation of
11da2a6c9b the check for --min-age and
--max-age got switched around.  This commit fixes that and means you
can use --min-age and --max-age together.
 2018-04-26 09:17:22 +01:00
Nick Craig-Wood a703216286 filter: take double negatives out of filter flag help 2018-04-26 09:17:13 +01:00
Nick Craig-Wood 96a62d55a2 lsd: Add -R flag and fix and update docs for all ls commands 2018-04-26 08:55:03 +01:00
Nick Craig-Wood d0f32b62fd Revert "build: Temporary workaround for golint being missing." 
This reverts commit be8bd89674.
 2018-04-25 16:17:54 +01:00
Nick Craig-Wood cc8799e0d6 Add new email address for Oliver Heyme to contributors 2018-04-25 15:52:41 +01:00
Nick Craig-Wood be8bd89674 build: Temporary workaround for golint being missing. 
See https://github.com/golang/lint/issues/397
 2018-04-24 11:22:38 +01:00
Nick Craig-Wood 9ab2521ef2 rc: autogenerate and tidy the docs and commands 
* Rename rc/pid -> core/pid
  * Sort the output of `rc list`
  * Make a script to autogenerate the docs
  * Tidy docs
 2018-04-23 20:57:17 +01:00
Nick Craig-Wood 21a10e58c9 rc: implement core/memstats to print internal memory usage info 2018-04-23 20:49:36 +01:00
Nick Craig-Wood d36b80f587 vendor: update bazil.org/fuse - corrects df -i - fixes #2089 2018-04-21 22:57:08 +01:00
Nick Craig-Wood 24980d7123 config: fix typo in error message #2268 2018-04-21 22:49:30 +01:00
Nick Craig-Wood 870c58f7f8 sftp: fail soft with a debug on hash failure #1474 
If md5sum/sha1sum fails we debug what it outputed on stderr and return
an empty hash indicating we didn't have a hash, rather than
hash.ErrUnsupported indicating that we don't support this hash type.

This fixes lots of ERROR messages for sftp and synology NAS which,
while it supports md5sum the SFTP paths and the SSH paths are
different so md5sum doesn't work.

We also stop disabling md5sum/sha1sum on errors since typically Hashes
is only checked at the start of a sync run and isn't expected to
change dynamically.
 2018-04-21 09:02:53 +01:00
Nick Craig-Wood b3c6f5f4b8 sftp: Update docs with Synology quirks 2018-04-21 09:02:53 +01:00
Nick Craig-Wood 311a962011 s3: Look in S3 named profile files for credentials - fixes #2243 2018-04-21 09:00:20 +01:00
Nick Craig-Wood da7a77ef2e ftp: Fix no error on listing non-existent directory 2018-04-20 23:22:46 +01:00
Nick Craig-Wood 9fbc40c5b9 fstests: List missing dir must return ErrorDirNotFound for non bucket based remotes 
List or ListR of an non existent directory must return
ErrorDirNotFound for non bucket based remotes.  For bucket based
remotes it may return ErrorDirNotFound or it may return no error and
no entries.
 2018-04-20 23:22:46 +01:00
Nick Craig-Wood 56ce784301 Add hensur to contributors 2018-04-20 21:44:12 +01:00
Nick Craig-Wood dc59836021 webdav: strip leading and trailing / off root - fixes #2257 2018-04-20 21:43:54 +01:00
Nick Craig-Wood 1a3fb21a77 onedrive: add QuickXorHash support for OneDrive for business - fixes #2262 2018-04-20 21:03:03 +01:00
Nick Craig-Wood bcdb7719c6 fs/hash: install QuickXorHash as a supported rclone hash type #2262 2018-04-20 21:02:57 +01:00
Nick Craig-Wood c51d97c752 hashsum: make generic tool for any hash to produce md5sum like output 2018-04-20 21:02:37 +01:00
Nick Craig-Wood 57a5b72d60 onedrive: implement quickXorHash algorithm #2262 2018-04-20 21:02:37 +01:00
Nick Craig-Wood 34ba17deec Add Chris Redekop second email to contributors 2018-04-20 20:53:15 +01:00
Nick Craig-Wood e3a1bc9cd3 Add Michael G. Noll to contributors 2018-04-20 20:51:31 +01:00
Nick Craig-Wood a0c65deca8 box: Parse file/directory size as a floating point number 
Very large directories can have their sizes returned as floating point
numbers, eg `1.0034576985781e+14` from the box API.

Before this change this would fail to parse as an int64.

This change parses the size as a float64 instead which will be
perfectly accurate for sizes up to 2**56 which is about 9 PB.

It is unknown whether box themselves use a float64 as an intermediate
representation in the API or not - it seems likely.

Fixes #2261
 2018-04-19 21:04:52 +01:00
Nick Craig-Wood 1f255a8567 Add a mega.nz remote #163 
Not supported yet:
  * Hash
  * ModTime
  * Server Side Copy

Otherwise fully functional and passing all the tests.
 2018-04-18 21:09:54 +01:00
Nick Craig-Wood f50b85278a vendor: github.com/t3rm1n4l for backend/mega 2018-04-18 21:09:54 +01:00
Nick Craig-Wood 9948b39dba about: don't attempt retries 2018-04-18 21:09:54 +01:00
Nick Craig-Wood 2b855751fc vfs,mount,cmount: use About to return the correct disk total/used/free 
Disks total, used, free now shows correctly for mount and cmount (eg
`df` for Unix or in the Windows explorer).
 2018-04-18 18:27:34 +01:00
Nick Craig-Wood ef3bcec76c fs: Extend SizeSuffix to include TB and PB for rclone about 2018-04-17 21:53:42 +01:00
Nick Craig-Wood 1ac6dacf0f about: complete other providers and re-work internals 
* Implement about for:
    * local, crypt, cache, drive, swift, hubic, onedrive, pcloud, dropbox
  * Implement `--json` and `---full` flag for `rclone about`
  * change About interface to return a Usage structure
  * Remove operations.About as it is too thin an interface
  * Implement Integration test

Relates to #1138 and #1564
 2018-04-17 21:53:27 +01:00
Nick Craig-Wood b83814082b backend/http: if HEAD didn't return Content-Length use -1 as size 
This means that the files will be treated as an unknown length and
will download properly.

Fixes #2247
 2018-04-16 19:40:02 +01:00
Nick Craig-Wood 2b7957cc74 vfs: Only make the VFS cache if --vfs-cache-mode > Off 
This stops the cache cleaner running unnecessarily and saves
resources.

This also helps with issue #2227 which was caused by a second mount
deleting objects in the first mounts cache.
 2018-04-16 17:06:41 +01:00
Nick Craig-Wood 3d5106e52b drive: fix DirMove leaving a hardlinked directory behind #2245 
This bug was introduced by the v3 API conversion in 07f20dd1fd.

The problem was that dircache.FindPath doesn't work for the root directory.

This adds an internal error for dircache.FindPath being called with
the root directory.  This makes a failing test, which the fix to the
drive backend fixes.

This also improves the DirCache integration test.
 2018-04-15 10:12:21 +01:00
Nick Craig-Wood 29ce1c2747 fstest: fix CheckListingWithPrecision with non Windows safe chars 
* Factor WinPath from fstest to fstests
  * Use it to normalize the directory names while checking them
 2018-04-15 10:12:20 +01:00
Nick Craig-Wood dc247d21ff s3: add in config for all the supported S3 providers #2140 
These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other.

This configures endpoints where known and makes sure config doesn't
appear where it isn't valid where possible.
 2018-04-13 16:33:26 +01:00
Nick Craig-Wood 8c3740c2c5 config: Improve the Provider matching to have a negated match #2140 
This makes it easier to make classes of provider in the config.
 2018-04-13 16:06:37 +01:00
Nick Craig-Wood 2015f98f0c Add Craig Rachel to contributors 2018-04-13 13:36:46 +01:00
Nick Craig-Wood 905e40b3e6 Add Peter Baumgartner to contributors 2018-04-13 13:33:22 +01:00
Nick Craig-Wood 6b67489133 Add Animosity022 to contributors 2018-04-13 13:26:41 +01:00
Nick Craig-Wood e6d9720d7b Add Mateusz Piotrowski to contributors 2018-04-13 13:25:16 +01:00
Nick Craig-Wood 18317a2747 vendor: update github.com/pkg/sftp because dep insisted 2018-04-13 13:23:55 +01:00
Nick Craig-Wood ef412c1985 drive: fix misplaced log in dedupe MergeDirs 2018-04-13 13:23:55 +01:00
Nick Craig-Wood d97fe3b824 fs/operations: make dedupe work with mega 
* factor into its own files
  * remove assumptions about having a given hash type
  * make tests work if the remote has no hash
 2018-04-13 13:23:55 +01:00
Nick Craig-Wood 792c9e185e Add Antoine GIRARD to contributors 2018-04-13 13:23:55 +01:00
Nick Craig-Wood e82452ce9a drive: check Open calls for google error messages 
This should also enable Open calls to retry properly
 2018-04-11 20:55:58 +01:00
Nick Craig-Wood dcf8334673 fs: add --dump goroutines and --dump openfiles 
These are developer flags useful for tracking down resource leaks.
 2018-04-11 20:55:58 +01:00
Nick Craig-Wood 37be78705d fs/fshttp: limit MaxIdleConns and MaxIdleConnsPerHost 
Before this change mega (which uses a different host per download)
would open too many sockets.
 2018-04-11 20:51:28 +01:00
Nick Craig-Wood 4b5ff33125 fstest: retry cleaning the integration test directory if necessary 2018-04-11 20:51:13 +01:00
Nick Craig-Wood d5b2ec32f1 local: add --local-no-check-updated to disable update checks #2206 
This disables the `can't copy - source file is being updated` checks.
 2018-04-09 15:27:58 +01:00
Nick Craig-Wood aeedacfb50 Add Michael P. Dubner to contributors 2018-04-09 13:33:27 +01:00
Nick Craig-Wood 05e32cfcf9 dropbox: Fix crypt+obfuscate on dropbox - fixes #2191 
Before this change we lowercased the dropbox root directory.  This was
likely a leftover from when we used to build a dictionary to translate
the cases of dropbox files.  Now with the v2 API we can rely on
dropbox to do that for us, so we no longer need to lowercase the root.

This fixes issues using crypt with name obfuscation on dropbox.
 2018-04-09 11:53:41 +01:00
Nick Craig-Wood cbec59146a lsf: make sure we use localtime in tests - fixes Box integration tests 
This problem was introduced with eca99b33c0.  It seems Box is the only
remote which converts time zones, so if you give it a GMT time zone,
it returns a PST time zone which represents the same instant.
 2018-04-09 11:46:49 +01:00
Nick Craig-Wood 06e3fa3aba mounttest: reduce duplicated code and improve test output #2154 
The written out list of tests was replaced with a nested test for
mount and cmount. The tests for each VFS cache mode were also replaced
with nested tests which makes the output and the code much cleaner.
 2018-04-08 15:04:14 +01:00
Nick Craig-Wood 0fa700b3cf Make integration tests use go1.7+ nested tests #2154 
* Removed generated code and code generator
  * Updated docs on how to write integration tests
  * Tidied up the actual integration tests
 2018-04-08 15:04:14 +01:00
Nick Craig-Wood 42f0963bf9 local: retry remove on Windows sharing violation error #2202 
Before this change asynchronous closes in cmount could cause sharing
violations under Windows on Remove which manifest themselves
frequently as test failures.

This change lets the Remove be retried on a sharing violation under
Windows.
 2018-04-07 17:36:26 +01:00
Nick Craig-Wood be54fd8f70 Remove builds conditional on go1.7 since that is now guaranteed #2154 
Old fallback code was deleted and the go1.7 style code inlined where
appropriate.
 2018-04-07 11:42:55 +01:00
Nick Craig-Wood e5be471ce0 Use io.SeekStart/End/Current constants now for go1.7+ #2154 2018-04-07 11:42:36 +01:00
Nick Craig-Wood 80588a5a6b Replace "golang.org/x/net/context" with "context" for go1.7+ #2154 2018-04-07 11:42:08 +01:00
Nick Craig-Wood 67023f0040 Require go1.7 for compilation #2154 
* Update the travis tests to exclude go1.6
  * Update the compile check to require go1.7+
  * Update misc go1.6 workarounds marked in the source
 2018-04-06 20:18:14 +01:00
Nick Craig-Wood 32e02bd367 fstests: Fix TestObjectRemove failures 
This was failing because TestPublicLink was causing the file to be
modified with Google drive.
 2018-04-06 16:27:19 +01:00
Nick Craig-Wood c749cf8d99 dropbox: fix repeatedly uploading the same files - fixes #2218 
In #2134 and dfd0f4c5a4 some testing
changes got committed by accident which caused this regression.

This patch reverts it to how it was before.
 2018-04-06 15:34:56 +01:00
Nick Craig-Wood 92cfb57fbd fstest/test_all: make -clean work better with google cloud storage 2018-04-06 14:54:33 +01:00
Nick Craig-Wood 0cb5c4aa73 gcs: detect bucket presence by listing it - fixes #2193 
Doing it like this enables the use of a service account that only has
the "Storage Object Admin" role.
 2018-04-06 12:45:15 +01:00
Nick Craig-Wood 0358e9e724 Add Eri Bastos to contributors 2018-04-05 20:20:53 +01:00
Nick Craig-Wood 92c5aa3786 s3: add --s3-chunk-size option - fixes #2203 2018-04-05 15:40:08 +01:00
Nick Craig-Wood fbe1c7f1ea dropbox: remove unused code 2018-04-05 15:23:23 +01:00
Nick Craig-Wood c4531daa43 local: work on spurious "can't copy - source file is being updated" errors #2206 
Update all the time comparisons to use time.Time.Equal instead of ==

Improve the logging for that error so we can see exactly what has changed
 2018-04-05 14:57:30 +01:00
Nick Craig-Wood 0865e38917 Add Matt Holt to contributors 2018-04-04 14:56:50 +01:00