Commit Graph
100 Commits
Author SHA1 Message Date
Nick Craig-Wood 8c0335a176 build: fix for goimports format change
See https://github.com/golang/go/issues/23709
2018-07-03 22:33:15 +01:00
Nick Craig-Wood 794e55de27 mega: wait for events instead of arbitrary sleeping 2018-07-02 14:50:09 +01:00
Nick Craig-Wood 038ed1aaf0 vendor: update github.com/t3rm1n4l/go-mega - fixes #2366
This update fixes files being missing from mega directory listings.
2018-07-02 14:50:09 +01:00
Nick Craig-Wood 97beff5370 build: keep track of compile failures better in cross-compile 2018-07-02 10:09:18 +01:00
Nick Craig-Wood b9b9bce0db ftp: fix Put mkParentDir failed: 521 for BunnyCDN - fixes #2363
According to RFC 959, error 521 is the correct error return to mean
"dir already exists", so add support for this.
2018-06-30 14:29:47 +01:00
Nick Craig-Wood 947e10eb2b config: fix error reading password from piped input - fixes #1308 2018-06-28 11:54:15 +01:00
Nick Craig-Wood 6b42421374 build: build macOS beta releases with native compiler on travis #2309 2018-06-26 09:39:44 +01:00
Nick Craig-Wood fa051ff970 webdav: add bearer token (Macaroon) support for dCache - fixes #2360 2018-06-25 17:54:36 +01:00
Nick Craig-Wood 69164b3dda build: move non master beta builds into branch subdirectory 2018-06-25 16:49:04 +01:00
Nick Craig-Wood 935533e57f filter: raise --include and --exclude warning to ERROR so it appears without -v 2018-06-22 22:18:55 +01:00
Nick Craig-Wood 1550f70865 webdav: Don't accept redirects when reading metadata #2350
Go can't redirect PROPFIND requests properly, it changes the method to
GET, so we disable redirects when reading the metadata and assume the
object does not exist if we receive a redirect.

This is to work-around the qnap redirecting requests for directories
without /.
2018-06-18 12:22:13 +01:00
Nick Craig-Wood 1a65c3a740 rest: add NoRedirect flag to Options 2018-06-18 12:21:50 +01:00
Nick Craig-Wood a29a1de43d webdav: if root ends with / then don't check if it is a file 2018-06-18 12:13:47 +01:00
Nick Craig-Wood e7ae5e8ee0 webdav: ensure we call MKCOL with a URL with a trailing / #2350
This is an attempt to fix rclone and qnap interop.
2018-06-18 11:16:58 +01:00
Nick Craig-Wood 08021c4636 vendor: update all dependencies 2018-06-17 17:59:12 +01:00
Nick Craig-Wood 3f0789e2db deletefile: fix typo in docs 2018-06-17 16:58:37 +01:00
Nick Craig-Wood 7110349547 Start v1.42-DEV development 2018-06-16 21:25:58 +01:00
Nick Craig-Wood a9adb43896 Version v1.42 2018-06-16 18:21:09 +01:00
Nick Craig-Wood c47a4c9703 opendrive: re-read hash when updating objects
Previously this was reading a stale hash from the object leading to
broken integration tests.

This fixes these integration tests TestSyncDoesntUpdateModtime,
TestSyncAfterChangingFilesSizeOnly, TestSyncAfterChangingContentsOnly,
TestSyncWithUpdateOlder, TestSyncUTFNorm.
2018-06-15 14:50:17 +01:00
Nick Craig-Wood d9d00a7dd7 rcat: remove --checksum flag from the docs as it is not usually effective 2018-06-14 16:15:54 +01:00
Nick Craig-Wood b82e66daaa Add themylogin to contributors 2018-06-14 16:15:53 +01:00
Nick Craig-Wood d18928962c dropbox: make dropbox for business folders accessible #2003
Paths prefixed with / on a dropbox for business plan will now start at
the root instead of the users home directory.
2018-06-13 11:03:34 +01:00
Nick Craig-Wood 1c75581959 sync: fix TestCopyRedownload after ModifyWindow changes #2310 2018-06-10 17:34:00 +01:00
Nick Craig-Wood 4d793b8ee8 drive: remove part of workaround for #1675
Now that https://issuetracker.google.com/issues/64468406 has been
fixed, we can remove part of the workaround which fixed #1675 -
019adc35609c2136

This will make queries marginally more efficient.  We still need the
other part of the workaround since the `=` operator is case
insensitive.
2018-06-10 15:28:32 +01:00
Nick Craig-Wood 9289aead9b drive: Add --drive-acknowledge-abuse to download flagged files - fixes #2317
Also if rclone gets the cannotDownloadAbusiveFile suggest using the
--drive-acknowledge-abuse flag.
2018-06-10 15:25:21 +01:00
Nick Craig-Wood 1053d7e123 local: fix symlink/junction point directory handling under Windows
Before this commit rclone's handling of symlinks and junction points
under Windows was broken.  rclone treated them as files and attempted
to transfer them which gave the error "The handle is invalid".

Ultimately the cause of this was 3e43ff7414 which was a
workaround so files with reparse points (which are a kind of symlink)
would transfer correctly.

The solution implemented is to revert the above commit which will mean
that #614 will break again.  However there is now a work-around (which
will be signaled by rclone) to use the -L flag which wasn't available
when the original commit was made.

Fixes #2336
2018-06-10 12:25:03 +01:00
Nick Craig-Wood 017297af70 s3: Fix --s3-chunk-size which was always using the minimum - fixes #2345 2018-06-10 12:22:30 +01:00
Nick Craig-Wood 334ef28012 Add Benjamin Joseph Dag to contributors 2018-06-08 16:12:57 +01:00
Nick Craig-Wood 05edb5f501 drive: Fix change list polling with team drives - fixes #2330
In the drive v3 conversion we forgot the IncludeTeamDriveItems
parameter when calling the changes API.  Adding it fixes the changes
polling with team drives.
2018-06-07 11:35:55 +01:00
Nick Craig-Wood ba43acb6aa sync: fix TestCopyEmptyDirectories after ModifyWindow changes #2310 2018-06-04 21:41:25 +01:00
Nick Craig-Wood 3ef938ebde lsf: add --absolute flag to add a leading / onto path names 2018-06-03 10:42:34 +01:00
Nick Craig-Wood 5302e5f9b1 docs: add a note about SIGINFO for macOS 2018-06-02 17:38:05 +01:00
Nick Craig-Wood 2b332bced2 Add Kasper Byrdal Nielsen to contributors 2018-05-31 09:42:36 +01:00
Nick Craig-Wood 500085d244 vendor: update github.com/dropbox/dropbox-sdk-go-unofficial #2158 2018-05-29 15:56:40 +01:00
Nick Craig-Wood 3d8e529441 rc: return error from remote on failure 2018-05-29 10:48:01 +01:00
Nick Craig-Wood d4213c0ac5 sftp: Fix slow downloads for long latency connections - fixes #1158
This was caused by using the sftp.File.Read method which resets the
streaming window after each call.  Replacing it with sftp.File.WriteTo
and an io.Pipe fixes the problem bringing the speed to the same as the
sftp binary.
2018-05-24 15:10:28 +01:00
Nick Craig-Wood 3a2248aa5f rc: add core/gc to run a garbage collection on demand 2018-05-24 15:10:28 +01:00
Nick Craig-Wood 573ef4c8ee rc: enable go profiling by default on the --rc port
This means you can use the pprof tool on a running rclone, eg

    go tool pprof http://localhost:5572/debug/pprof/heap
2018-05-24 15:10:28 +01:00
Nick Craig-Wood 7bf2d389a8 Add John Clayton to contributors 2018-05-22 11:48:20 +01:00
Nick Craig-Wood e5ff375948 Use config.FileGet instead of fs.ConfigFileGet 2018-05-22 09:43:24 +01:00
Nick Craig-Wood 512f4b4487 Update error checking on fmt.Fprint* after errcheck update
Now we need to check or ignore errors on fmt.Fprint* explicitly -
previously errcheck just ignored them for us.
2018-05-22 09:41:13 +01:00
Nick Craig-Wood a38f8b87ce docs: fix Nextcloud typo spotted by Eugene Mlodik 2018-05-16 16:43:52 +01:00
Nick Craig-Wood 9697754707 drive: Don't attempt to choose Team Drives when using rclone config create 2018-05-16 09:10:09 +01:00
Nick Craig-Wood 8e625e0bc3 config: add ConfirmWithDefault to change the default on AutoConfig 2018-05-16 09:09:41 +01:00
Nick Craig-Wood e52ecba295 crypt: check the crypted hash of files when uploading #2303
This checks the checksum of the streamed encrypted data against the
checksum of the encrypted object returned from the remote and returns
an error if it is different.
2018-05-15 14:50:36 +01:00
Nick Craig-Wood e62d2fd309 oauthutil: Fix custom redirect URL message - fixes #2306 2018-05-13 17:28:09 +01:00
Nick Craig-Wood e56be0dfd8 lsf: Add --csv flag for compliant CSV output 2018-05-13 12:18:21 +01:00
Nick Craig-Wood 2a32e2d838 operations: turn ListFormatted into a Format method on ListFormat 2018-05-13 12:17:55 +01:00
Nick Craig-Wood db4c206e0e lsjson: add MimeType to the output 2018-05-13 12:17:55 +01:00
Nick Craig-Wood f77efc7649 lsf: Add 'm' format specifier to show the MimeType 2018-05-13 12:17:55 +01:00
Nick Craig-Wood aadbcce486 fs: Add MimeTypeDirEntry to return the MimeType of a DirEntry 2018-05-13 12:17:55 +01:00
Nick Craig-Wood f162116132 lsjson: add ID field to output to show Object ID - fixes #1901 2018-05-13 12:17:55 +01:00
Nick Craig-Wood 909c3a92d6 lsf: implement 'i' format for showing object ID - fixes #1476 2018-05-13 12:17:55 +01:00
Nick Craig-Wood 826975c341 fs: add Optional ID() method to Object and implement it in backends
ID() shows the internal ID of the Object if available.
2018-05-13 12:17:55 +01:00
Nick Craig-Wood cdde8fa75a opendrive: finish off #1026
* Fix errcheck and golint warnings
  * Remove unused constants and fix comments
  * Parse error responses properly
  * Fix Open with RangeOption
  * Fix Move, Copy and DirMove
  * Implement DirCacheFlush
  * Check interfaces are correct
  * Remove debugs and update overview
  * Correct feature flags
  * Pare replacement characters down to the minimum set
  * Add to the integration tests
2018-05-12 10:10:46 +01:00
Nick Craig-Wood 5ede6f6d09 Add Jakub Karlicek to contributors 2018-05-12 10:10:45 +01:00
Nick Craig-Wood ad02d1be3f fstest: update comments on how to run individual tests 2018-05-11 14:04:36 +01:00
Nick Craig-Wood 63f413f477 webdav: show all available information when printing errors 2018-05-11 08:43:53 +01:00
Nick Craig-Wood f1ffe8e309 fstests: fix test crash if NewFs fails 2018-05-11 08:43:53 +01:00
Nick Craig-Wood d85b9bc9d6 webdav: workarounds for biz.mail.ru
* Add "Depth: 1" on read metadata PROPFIND call
  * Accept 406 to mean directory already exists
2018-05-11 08:43:53 +01:00
Nick Craig-Wood b07e51cf73 webdav: read the body of messages into the error if XML parse fails 2018-05-11 08:43:53 +01:00
Nick Craig-Wood f073db81b1 drive: add --drive-alternate-export to fix large doc export - fixes #2243
The official drive APIs seem to have trouble downloading large
documents sometimes.

This commit adds a --drive-alternate-export flag to use an different,
unofficial set of export URLS which seem to download large files OK.
2018-05-10 10:04:39 +01:00
Nick Craig-Wood 9698a2babb gcs: low level retry all operations if necessary
Google cloud storage doesn't normally need retries, however certain
things (eg bucket creation and removal) are rate limited and do
generate 429 errors.

Before this change the integration tests would regularly blow up with
errors from GCS rate limiting bucket creation and removal.

After this change we low level retry all operations using the same
exponential backoff strategy as used in the google drive backend.
2018-05-10 09:24:09 +01:00
Nick Craig-Wood 5eecbd83ee bin: make make_test_files.go work properly on Windows 2018-05-09 16:59:29 +01:00
Nick Craig-Wood e42edc8e8c copy, move: Copy single files directly, don't use --files-from work-around
Before this change rclone would inefficiently and confusingly read all
the files in the source directory when copy or moving a single file.
This caused confusion for the users to see log messages about files
which weren't part of the sync.

After the change the copy and move commands use the new infrastructure
made for the copyto and moveto command for single file copy and move.
2018-05-07 20:39:52 +01:00
Nick Craig-Wood 291954baba cmd: make names of argument parsing functions more consistent 2018-05-07 20:39:52 +01:00
Nick Craig-Wood 9d8d7ae1f0 mount,cmount: make --noappledouble --noapplexattr and change defaults #2287
Before this change we would unconditionally set the OSXFUSE options
noappledouble and noapplexattr.

However the noapplexattr options caused problems with copies in the
Finder.

Now the default for noapplexattr is false so we don't add the option
by default and the user can override the defaults using the
--noappledouble and --noapplexattr flags.
2018-05-07 20:37:09 +01:00
Nick Craig-Wood 6ce32e4661 mount,cmount: Add --volname flag and remove special chars from it #2287
Before this change rclone would set the volume name from the
remote:path normally.  However this has `:` and `/` in which make it
difficult to use in macOS.

Now rclone will remove the special characters and replace them with
spaces.  It also allows the volume name to be set with the --volname
flag.
2018-05-07 20:37:09 +01:00
Nick Craig-Wood 1755ffd1f3 mount: make Get/List/Set/Remove xattr return ENOSYS #2287
By default bazil fuse will return ENOTSUPP for these.  However if we
return ENOSYS then OSXFUSE (at least) will never call them again
saving round trips though fuse.
2018-05-07 20:37:09 +01:00
Nick Craig-Wood aa5c5ec5d3 build: mask linter errors we can't fix 2018-05-05 17:32:41 +01:00
Nick Craig-Wood e80ae4e09c build: remove unused struct fields spotted by structcheck 2018-05-05 17:32:41 +01:00
Nick Craig-Wood 1320e84bc2 build: remove unused code spotted by the deadcode linter 2018-05-05 17:32:41 +01:00
Nick Craig-Wood cb5bd47e61 build: fix errors spotted by ineffassign linter
These were mostly caused by shadowing err and a good fraction of them
will have caused errors not to be propagated properly.
2018-05-05 17:32:41 +01:00
Nick Craig-Wood 790a8a9aed build: add gometalinter and gometalinter_install Makefile targets 2018-05-05 17:32:41 +01:00
Nick Craig-Wood f1a43eca4d mount: make --daemon work for macOS without CGO 2018-05-05 16:23:47 +01:00
Nick Craig-Wood 7ea68f1fc6 sftp: require go1.9+ after golang.org/x/crypto/ssh update 2018-05-05 16:23:47 +01:00
Nick Craig-Wood 6427029c4e vendor: update all dependencies
* Update all dependencies
  * Remove all `[[constraint]]` from Gopkg.toml
  * Add in the minimum number of `[[override]]` to build
  * Remove go get of github.com/inconshreveable/mousetrap as it is vendored
  * Update docs with new policy on constraints
2018-05-05 15:52:24 +01:00
Nick Craig-Wood 21383877df cmd: make exit code 8 for --max-transfer exceeded 2018-05-05 12:58:28 +01:00
Nick Craig-Wood f95835d613 fserrors: Look deeper into errors for Fatal/Retry/NoRetry errors.
Before this change fatal errors which were wrapped in a system error (eg a
URLError) were not recognised as fatal errors.
2018-05-05 12:58:28 +01:00
Nick Craig-Wood be79b47a7a sync: log when we abandon the sync due to a fatal error 2018-05-05 12:58:28 +01:00
Nick Craig-Wood be22735609 fs/accounting: fix deadlock on GetBytes
A deadlock could occur since we have now put a mutex on GetBytes from
StatsInfo.String (s.mu) - progress (acc.statmu) and read (acc.statmu)
- GetBytes (s.mu).

Fix this by giving stringSet its own locking and excluding the call
which caused the deadlock from the mutex in StatsInfo.String.
2018-05-05 12:58:28 +01:00
Nick Craig-Wood 1b1b3c13cd sync: add a test for aborting on max upload 2018-05-05 12:58:28 +01:00
Nick Craig-Wood 5c128272fd Implement --max-transfer flag to quit transferring at a limit #1655 2018-05-05 12:58:28 +01:00
Nick Craig-Wood d178233e74 sync,march: check the cancel context on every channel send and receive
This fixes a deadlock on sync when all the copying channels receive a
Fatal Error.
2018-05-05 12:58:28 +01:00
Nick Craig-Wood fe25cb9c54 drive: fix about (and df on a mount) for team drives - fixes #2288
Before this fix team drives would return the drive quota which is
incorrect and mis-leading.

Team drives don't appear to have an API for reading the bytes used or
the quota so we now return that the quota and usage are unknown.
2018-05-03 08:59:14 +01:00
Nick Craig-Wood f2608e2a64 Add NoLooseEnds to contributors 2018-05-01 09:43:18 +01:00
Nick Craig-Wood 50dc5fe92e Add Rodrigo to contributors 2018-04-30 17:37:43 +01:00
Nick Craig-Wood 3116249692 make sign_upload: only sign the v1.xx releases not the current ones 2018-04-30 17:29:50 +01:00
Nick Craig-Wood d049e5c680 make build_dep: make sure we update the whole command for nfpm 2018-04-30 17:29:50 +01:00
Nick Craig-Wood 1c9572aba1 Add Piotr Oleszczyk to contributors 2018-04-30 17:29:50 +01:00
Nick Craig-Wood 0479c7dcf5 add github-release to make release_dep 2018-04-28 12:38:30 +01:00
Nick Craig-Wood 55674c0bfc Start v1.41-DEV development 2018-04-28 12:37:55 +01:00
Nick Craig-Wood e4c380b2a8 Version v1.41 2018-04-28 11:46:27 +01:00
Nick Craig-Wood 74cbdea0ef Revert "copy: create (pseudo copy) empty source directories to destination"
Unfortunately this commit attempts to create every directory rather
than just the empty ones, so will need re-working.

Removing this feature for the 1.41 release

This reverts commit 0daced29db.
2018-04-28 10:02:32 +01:00
Nick Craig-Wood a3bf6b9c2c drive, gcs: fix service account authentication - fixes #2279
This fixes a problem introduced in b78af517de where it would
attempt to read a non-existent service account file.
2018-04-28 09:33:43 +01:00
Nick Craig-Wood d8e88f10cd rc: take note of the --rc-addr flag too as per the docs - fixes #2184 2018-04-26 17:00:44 +01:00
Nick Craig-Wood 849db6699d Add Richard Yang to contributors 2018-04-26 16:23:52 +01:00
Nick Craig-Wood da4a5e1fb3 docs: note that copytruncate is needed for --log-file with logrotate #2259 2018-04-26 15:30:46 +01:00
Nick Craig-Wood ae562b5a4f ftp: more workarounds for FTP servers to fix mkParentDir - fixes #2181 2018-04-26 14:58:04 +01:00
Nick Craig-Wood c01177bc28 ftp: work around strange response from box FTP server
The Box FTP server seems to send 450 instead of 550 - work around that.

See: https://forum.rclone.org/t/using-box-com-over-ftp-problems/5313
2018-04-26 14:58:04 +01:00