f-denkena/rclone
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
2,169 Commits 1 Branch 0 Tags
d97fe3b82474bd1f3c612b02d01d46514e0ece0b
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick Craig-Wood d97fe3b824 fs/operations: make dedupe work with mega 
* factor into its own files
  * remove assumptions about having a given hash type
  * make tests work if the remote has no hash
 2018-04-13 13:23:55 +01:00
Nick Craig-Wood 792c9e185e Add Antoine GIRARD to contributors 2018-04-13 13:23:55 +01:00
Nick Craig-Wood e82452ce9a drive: check Open calls for google error messages 
This should also enable Open calls to retry properly
 2018-04-11 20:55:58 +01:00
Nick Craig-Wood dcf8334673 fs: add --dump goroutines and --dump openfiles 
These are developer flags useful for tracking down resource leaks.
 2018-04-11 20:55:58 +01:00
Nick Craig-Wood 37be78705d fs/fshttp: limit MaxIdleConns and MaxIdleConnsPerHost 
Before this change mega (which uses a different host per download)
would open too many sockets.
 2018-04-11 20:51:28 +01:00
Nick Craig-Wood 4b5ff33125 fstest: retry cleaning the integration test directory if necessary 2018-04-11 20:51:13 +01:00
Nick Craig-Wood d5b2ec32f1 local: add --local-no-check-updated to disable update checks #2206 
This disables the `can't copy - source file is being updated` checks.
 2018-04-09 15:27:58 +01:00
Nick Craig-Wood aeedacfb50 Add Michael P. Dubner to contributors 2018-04-09 13:33:27 +01:00
Nick Craig-Wood 05e32cfcf9 dropbox: Fix crypt+obfuscate on dropbox - fixes #2191 
Before this change we lowercased the dropbox root directory.  This was
likely a leftover from when we used to build a dictionary to translate
the cases of dropbox files.  Now with the v2 API we can rely on
dropbox to do that for us, so we no longer need to lowercase the root.

This fixes issues using crypt with name obfuscation on dropbox.
 2018-04-09 11:53:41 +01:00
Nick Craig-Wood cbec59146a lsf: make sure we use localtime in tests - fixes Box integration tests 
This problem was introduced with eca99b33c0.  It seems Box is the only
remote which converts time zones, so if you give it a GMT time zone,
it returns a PST time zone which represents the same instant.
 2018-04-09 11:46:49 +01:00
Nick Craig-Wood 06e3fa3aba mounttest: reduce duplicated code and improve test output #2154 
The written out list of tests was replaced with a nested test for
mount and cmount. The tests for each VFS cache mode were also replaced
with nested tests which makes the output and the code much cleaner.
 2018-04-08 15:04:14 +01:00
Nick Craig-Wood 0fa700b3cf Make integration tests use go1.7+ nested tests #2154 
* Removed generated code and code generator
  * Updated docs on how to write integration tests
  * Tidied up the actual integration tests
 2018-04-08 15:04:14 +01:00
Nick Craig-Wood 42f0963bf9 local: retry remove on Windows sharing violation error #2202 
Before this change asynchronous closes in cmount could cause sharing
violations under Windows on Remove which manifest themselves
frequently as test failures.

This change lets the Remove be retried on a sharing violation under
Windows.
 2018-04-07 17:36:26 +01:00
Nick Craig-Wood be54fd8f70 Remove builds conditional on go1.7 since that is now guaranteed #2154 
Old fallback code was deleted and the go1.7 style code inlined where
appropriate.
 2018-04-07 11:42:55 +01:00
Nick Craig-Wood e5be471ce0 Use io.SeekStart/End/Current constants now for go1.7+ #2154 2018-04-07 11:42:36 +01:00
Nick Craig-Wood 80588a5a6b Replace "golang.org/x/net/context" with "context" for go1.7+ #2154 2018-04-07 11:42:08 +01:00
Nick Craig-Wood 67023f0040 Require go1.7 for compilation #2154 
* Update the travis tests to exclude go1.6
  * Update the compile check to require go1.7+
  * Update misc go1.6 workarounds marked in the source
 2018-04-06 20:18:14 +01:00
Nick Craig-Wood 32e02bd367 fstests: Fix TestObjectRemove failures 
This was failing because TestPublicLink was causing the file to be
modified with Google drive.
 2018-04-06 16:27:19 +01:00
Nick Craig-Wood c749cf8d99 dropbox: fix repeatedly uploading the same files - fixes #2218 
In #2134 and dfd0f4c5a4 some testing
changes got committed by accident which caused this regression.

This patch reverts it to how it was before.
 2018-04-06 15:34:56 +01:00
Nick Craig-Wood 92cfb57fbd fstest/test_all: make -clean work better with google cloud storage 2018-04-06 14:54:33 +01:00
Nick Craig-Wood 0cb5c4aa73 gcs: detect bucket presence by listing it - fixes #2193 
Doing it like this enables the use of a service account that only has
the "Storage Object Admin" role.
 2018-04-06 12:45:15 +01:00
Nick Craig-Wood 0358e9e724 Add Eri Bastos to contributors 2018-04-05 20:20:53 +01:00
Nick Craig-Wood 92c5aa3786 s3: add --s3-chunk-size option - fixes #2203 2018-04-05 15:40:08 +01:00
Nick Craig-Wood fbe1c7f1ea dropbox: remove unused code 2018-04-05 15:23:23 +01:00
Nick Craig-Wood c4531daa43 local: work on spurious "can't copy - source file is being updated" errors #2206 
Update all the time comparisons to use time.Time.Equal instead of ==

Improve the logging for that error so we can see exactly what has changed
 2018-04-05 14:57:30 +01:00
Nick Craig-Wood 0865e38917 Add Matt Holt to contributors 2018-04-04 14:56:50 +01:00
Nick Craig-Wood ab2fa59fc4 Add Alexander Neumann to contributors 2018-04-04 14:56:50 +01:00
Nick Craig-Wood 06a8d3011d Add Chih-Hsuan Yen to contributors 2018-04-02 11:43:22 +01:00
Nick Craig-Wood eca99b33c0 lsd,lsf: make sure all times we output are in local time - fixes #2183 
Previous to this change times from lsd/lsf were output in whatever
timezone they were in whereas times from lsl were converted to
localtime.
 2018-04-01 15:40:04 +01:00
Nick Craig-Wood d45c750f76 Add Steve Kriss to contributors 2018-03-30 19:55:49 +01:00
Nick Craig-Wood 9df266a6b4 onedrive: Fix socket leak in multipart session upload 
This had gone unnoticed until recently when we changed to uploading
all files with a multipart session.
 2018-03-28 21:03:19 +01:00
Nick Craig-Wood 1ba3ffdc59 Add Keith Goldfarb to contributors 2018-03-26 21:03:18 +01:00
Nick Craig-Wood 72f1b097a7 Add gbadanahatti to contributors 2018-03-26 21:03:18 +01:00
Nick Craig-Wood 885044d0a5 Add seuffert to contributors 2018-03-26 21:03:18 +01:00
Nick Craig-Wood 9b140b42c9 docs: fix current download link 2018-03-26 17:45:45 +01:00
Nick Craig-Wood 0bfbde8856 fstest: make ChangeNotify test clean up after itself and be more reliable 
Previous to this fix old notifications could creep in and cause the
test to fail.  It also left files around which upset the TestObjectRemove test.

Fixes #2177
 2018-03-24 19:57:44 +00:00
Nick Craig-Wood 98a924602f mount, cmount: set --attr-timeout default to 1s - fixes #2157 
This  works around these 3 problems:

  * rclone using too much memory #2157
  * rclone not serving files to samba
    * https://forum.rclone.org/t/rclone-1-39-vs-1-40-mount-issue/5112
  * excessive time listing directories #2095
 2018-03-23 22:42:51 +00:00
Nick Craig-Wood 7e80e609e8 docs: install.sh add macOS fallback for mktemp - fixes #2173 2018-03-23 22:24:28 +00:00
Nick Craig-Wood 32e6eee341 release: add another step to update the release dependencies #2172 2018-03-23 12:43:18 +00:00
Nick Craig-Wood c5f1d501ed docs: fix download links for .deb and .rpm 2018-03-23 12:43:18 +00:00
Nick Craig-Wood 5ba5be9b37 gcs: ignore zero length directory markers at the root too 2018-03-21 20:10:00 +00:00
Nick Craig-Wood e9a2cbec37 s3: ignore zero length directory markers at the root too 2018-03-21 20:09:37 +00:00
Nick Craig-Wood 4f6f07c074 cmount: fix error handling for Open/OpenDir 2018-03-21 19:44:30 +00:00
Nick Craig-Wood f6020f1308 gcs: ignore zero length directory markers 2018-03-19 17:42:27 +00:00
Nick Craig-Wood a46f2a9eb7 s3: ignore zero length directory markers - fixes #1621 2018-03-19 17:41:46 +00:00
Nick Craig-Wood 911a78ce6d sftp: require go1.8+ after github.com/pkg/sftp update 2018-03-19 16:37:40 +00:00
Nick Craig-Wood d64789528d vendor: update all dependencies 2018-03-19 15:51:38 +00:00
Nick Craig-Wood 940df88eb2 Start v1.40-DEV development 2018-03-19 14:20:48 +00:00
Nick Craig-Wood 19ca9fb939 release: Put the releases into a v1.XX subdirectory 2018-03-19 14:20:09 +00:00
Nick Craig-Wood 26f1c55987 Version v1.40 2018-03-19 10:06:13 +00:00
Nick Craig-Wood 1afac32d80 serve restic: script for running integration test against all remotes 2018-03-18 19:15:39 +00:00
Nick Craig-Wood 26fbd00b4f serve restic: don't buffer the JSON output in memory for the list command 2018-03-18 16:26:58 +00:00
Nick Craig-Wood 1313b529ff serve restic: use ListR (--fast-list) if available 
For Restic's use case, --fast-list will use less transactions and
calling ListR directly means we can avoid the usual memory overhead.
 2018-03-18 16:22:05 +00:00
Nick Craig-Wood 82e835d6fc serve restic: make it easy to run integration tests against any remote 
Just `cd cmd/serve/restic` then `go test -v -remote TestRemote:`
 2018-03-18 14:23:56 +00:00
Nick Craig-Wood fa867a9a4c serve restic: implement accounting for uploads and downloads 
This means the bandwidth stats will be correct and the bandwidth
throttling will work correctly.  This was forgotten as a previous
iteration of the code was using the higher level operations.Rcat which
took care of this.
 2018-03-18 14:19:43 +00:00
Nick Craig-Wood 38d9475a34 release: include a source tarball and sign everything #1449 2018-03-17 15:06:04 +00:00
Nick Craig-Wood c21c7e75b0 Add Stefan Lindblom to contributors 2018-03-17 12:12:23 +00:00
Nick Craig-Wood 012d4a1235 docs: fix download icon 2018-03-17 12:00:14 +00:00
Nick Craig-Wood 854d3c3025 Add Dave Pedu to contributors 2018-03-17 12:00:14 +00:00
Nick Craig-Wood e62fe06763 s3: document --ignore-checksum workaround for KMS #1824 2018-03-17 10:51:45 +00:00
Nick Craig-Wood 4295428a0f fs/accounting: add remote control of bwlimit 2018-03-17 10:34:02 +00:00
Nick Craig-Wood 2db0c4dd95 vfs: add remote control for directory cache flushing 2018-03-17 10:34:02 +00:00
Nick Craig-Wood 5bf639048f sync: log an error that --track-renames doesn't work with sync or move 
Fixes #2008
 2018-03-17 10:34:02 +00:00
Nick Craig-Wood d4cca8d9f9 onedrive: fix upload of zero length files #1716 
Unfortunately multi part upload can't upload zero length files so
bring back the single part upload for zero length files only.

This was broken when we made all uploads multipart uploads.
 2018-03-17 09:48:28 +00:00
Nick Craig-Wood a9e386b153 Add wolfv to contributors 2018-03-17 09:06:51 +00:00
Nick Craig-Wood d1bb8efb88 sftp: follow symlinks correctly - fixes #2145 
The sftp library delivers the attributes of the symlink rather than
the object pointed to in directory listings, however when we use Stat
from the library it points to the objects.

Previous to this fix this caused items pointed to by symlinks to be
unusable.

After the fix both symlinked files and directories work as expected.
 2018-03-16 15:36:47 +00:00
Nick Craig-Wood c19e675ca6 vfs: unify locking for RWFileHandle.openPending,.close and File.Delete #2141 
Without this fix the cached file can be removed as the file is being
uploaded or downloaded.  This can cause the directory listings to
become inconsistent (this issue) or data loss (if a retry was needed
in the Copy).

Remove file needs to be excluded from running at the same time as both
openPending and close so it makes sense to unify the locking between
all 3.
 2018-03-15 20:49:07 +00:00
Nick Craig-Wood 34c45a7c04 mount, cmount: remove addition of O_CREATE to flags on file open #2141 
Previously this was adding it in to all file opens which was causing
inefficiencies under Windows where it stats the file using
open/fstat/close.

This change will make stat operations run much quicker under Windows
as they won't have to open the underlying file

This problem was introduced in61b6159a05336bd7ba105766de2d2ff171f7fb81
where we added O_CREATE to all file opens and creates.
 2018-03-15 20:48:56 +00:00
Nick Craig-Wood 0a0318df20 Add Leo R. Lundgren to contributors 2018-03-15 20:24:42 +00:00
Nick Craig-Wood d551137635 Add Giri Badanahatti to contributors 2018-03-15 20:21:12 +00:00
Nick Craig-Wood 5981f9fab5 acd: disable integration tests 
We no longer have any working keys for Amazon Cloud Drive so disable
the integration tests.
 2018-03-14 22:44:46 +00:00
Nick Craig-Wood c1a3e363a6 mount: return ENOSYS rather than EIO on attempted link 
This fixes FileZilla accessing an rclone mount served over sftp.

See: https://forum.rclone.org/t/moving-files-on-rclone-mount-with-filezilla/5029
 2018-03-14 21:10:20 +00:00
Nick Craig-Wood 7ccc6080b0 serve restic: add more info to GET request error 2018-03-14 21:09:47 +00:00
Nick Craig-Wood 86e5a35491 Implement Remote Control for rclone #2111 
This implements a remote control protocol activated with the --rc flag
and a new command `rclone rc` to use that interface.

Still to do
  * docs - need finishing
  * tests
 2018-03-14 22:58:20 +02:00
Nick Craig-Wood 8bb2854fe4 httplib: allow the flags to be prefixed when instantiating a server 2018-03-14 22:58:20 +02:00
Nick Craig-Wood 89748feaa5 s3: update docs to discourage use of v2 auth - fixes #2120 
From testing it appears that CEPH no longer works properly with v2
auth and neither does Dreamhost, so update the docs anc configuration
to recommend v4 auth.
 2018-03-13 20:47:29 +00:00
Nick Craig-Wood dfd0f4c5a4 sync: when using --backup-dir don't delete files if we can't set their modtime 
This is a problem when syncing a file which just needed its modtime
set with dropbox which can't set the mod time of a file without
re-uploading it.

Before this change we would delete the file, then the server side move
would fail moving the file to the backup-dir because it no longer
existed.

After this change the destination file is moved to the backup-dir
instead of being deleted and the new file is uploaded.

Fixes #2134
 2018-03-13 16:05:06 +00:00
Nick Craig-Wood 0c9dc006c5 fs: make display of default values of --min-age/--max-age be off - Fixes #2121 2018-03-13 09:06:07 +00:00
Nick Craig-Wood 4e90ad04d5 serve restic: only accept v2 API requests for list 2018-03-11 17:35:01 +00:00
Nick Craig-Wood 43c7ea81df Add Alexander Neumann to contributors 2018-03-11 17:35:01 +00:00
Nick Craig-Wood d073efdc6c serve restic: serves a remote in restic REST API format 2018-03-11 14:43:03 +00:00
Nick Craig-Wood 9e48748182 httplib: Note that authentication is a good idea for non localhost 2018-03-11 14:38:54 +00:00
Nick Craig-Wood b6058e0106 docs/install.sh: don't create root owned .config/rclone directory #2127 2018-03-10 11:09:13 +00:00
Nick Craig-Wood 66c69fe620 mount: wait longer for consistency after rm in tests 2018-03-09 23:15:38 +00:00
Nick Craig-Wood a2336ad774 vfs: fix deadlock in mount tests 
This was caused by this sequence of calls

1> file.Release
1> file.close  -> takes the file lock
2> vfs.waitforWriters
2> dir.walk -> takes the dir lock
1> file.setObject
1> dir.addObject -> attempts to take the dir lock - BLOCKS
2> file.activeWriters -> tries to take file lock - BLOCKS - DEADLOCK

The fix is to make activeWriters not take the file lock and use atomic
operations to read the number of writers instead.
 2018-03-09 23:15:38 +00:00
Nick Craig-Wood 7713acf23d mount: skip failing test TestFileModTimeWithOpenWriters on Windows 2018-03-09 23:15:38 +00:00
Nick Craig-Wood 473a388f6d mount: disable failing test TestWriteFileDoubleClose on OSX 2018-03-09 23:15:37 +00:00
Nick Craig-Wood c8a4d437a0 Make travis test mount and cmount - fixes #2100 
Previously FUSE wasn't found in the container so these tests weren't
run.  Move to VM based testing and install FUSE dependencies.
 2018-03-09 23:15:37 +00:00
Nick Craig-Wood 09c14af6d1 cmd: Fix go routines at exit message to make it less confusing 2018-03-09 17:15:48 +00:00
Nick Craig-Wood 0861207ace fstest/test_all: set cache backend wait time to 30m to fix integration tests 2018-03-08 21:14:09 +00:00
Nick Craig-Wood a7dbf32c53 cache: Implement --cache-db-wait-time flag 
This can be used to make the cache wait for other running cache
backends to finish rather than erroring after 1 second.
 2018-03-08 21:14:09 +00:00
Nick Craig-Wood 6025bb6ad1 local: fix race conditions updating the hashes 
This was causing occasional test failures for the -race test of mount
and cmount.
 2018-03-08 21:08:41 +00:00
Nick Craig-Wood b3f55d6bda vendor: Update github.com/Unknwon/goconfig to fix section listing 
This fixes listing sections just after creation which means the rclone
config list will have all the keys in now.
 2018-03-08 13:18:27 +00:00
Nick Craig-Wood d9094f1a45 vendor: Gopkg.lock file format changes only after go dep update 2018-03-08 13:16:59 +00:00
Nick Craig-Wood 572ee5ec96 Sign the tags as part of the release process #1449 2018-03-07 15:18:13 +00:00
Nick Craig-Wood 316dac25c2 travis: add encrypted GITHUB_USER and GITHUB_TOKEN for using the API 2018-03-07 10:18:10 +00:00
Nick Craig-Wood ee3c45676f bin/get-github-release.go: use GITHUB_USER/GITHUB_TOKEN when available 
This should help with rate limiting problems when running under
travis.
 2018-03-07 10:18:09 +00:00
Nick Craig-Wood 2e7e15461b bin/get-github-release.go: report body of HTTP responses with errors 2018-03-07 10:18:06 +00:00
Nick Craig-Wood 0175332987 vfs: fix applying modtime for an open Write Handle 
The symptom of this was that the time set when the file was open was
lost.  This was causing one of the mount tests to fail too.
 2018-03-06 21:58:11 +00:00
Nick Craig-Wood 85e0b87c99 build: add .deb and .rpm output for the build 
This uses https://github.com/goreleaser/nfpm to create the .deb and
.rpm packages from the standard build output.
 2018-03-06 12:37:44 +00:00