Commit Graph
100 Commits
Author SHA1 Message Date
Nick Craig-Wood 57d5de6fba build: fix up package paths after repo move
git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g'
goimports -w `find . -name \*.go`
2019-07-28 18:47:38 +01:00
Nick Craig-Wood 2d561b51db Add EliEron to contributors 2019-07-28 12:33:21 +01:00
Nick Craig-Wood 9241a93c2d Add justinalin to contributors 2019-07-28 12:33:21 +01:00
Nick Craig-Wood 9af0a704af Add Paul Millar to contributors 2019-07-24 20:34:39 +01:00
Nick Craig-Wood 691e5ae5f0 Add Yi FU to contributors 2019-07-24 20:34:39 +01:00
Nick Craig-Wood 5a44bafa4e fstest: add fs.ErrorCantShareDirectories for backends which can only share files 2019-07-24 20:34:29 +01:00
Nick Craig-Wood 8fdce31700 config: Fix hiding of options from the configurator 2019-07-24 20:34:29 +01:00
Nick Craig-Wood 493dfb68fd opendrive: refactor to use existing lib/rest facilities for uploads
This also checks the return of the call to make sure the number of
bytes written was as expected.
2019-07-24 20:34:29 +01:00
Nick Craig-Wood 71587344c6 lib/rest: allow Form upload with no file to upload 2019-07-24 20:34:29 +01:00
Nick Craig-Wood 266600dba7 build: reduce parallelism in cross compile to reduce memory and fix Travis
Before this change Travis builds were running out of memory when cross
compiling all the OSes.
2019-07-22 17:10:26 +01:00
Nick Craig-Wood 1f1ab179a6 webdav: refresh token when it expires with --webdav-bearer-token-command
Fixes #2380
2019-07-22 16:01:55 +01:00
Nick Craig-Wood c642531a1e webdav: add --webdav-bearer-token-command - fixes #2380
This can be used with oidc-agent to get a bearer token
2019-07-22 15:59:54 +01:00
Nick Craig-Wood 5433021e8b drive: fix server side copy of big files
Before this change rclone was sending a MimeType in the requests for
server side Move and Copy.

The conjecture is that if you attempt to set the MimeType to something
different in a Copy then Google Drive has to do an actual copy of the
file data.  This takes a very long time (since it is large) and fails
after a 90s timeout.

After the change we no longer set the MimeType in Move or Copy and the
copies happen instantly and correctly.

Many thanks to @darthShadow for discovering that this was causing the
problem.

Fixes #3070
Fixes #3033
Fixes #3300
Fixes #3155
2019-07-05 10:49:19 +01:00
Nick Craig-Wood c9f77719e4 b2: enable server side copy to copy between buckets - fixes #3303 2019-07-05 10:07:05 +01:00
Nick Craig-Wood 3cd63a00be googlephotos: fix configuration walkthrough 2019-07-04 15:19:59 +01:00
Nick Craig-Wood d7016866e0 googlephotos: fix creation of duplicated albums
Also make sure we don't list the albums twice
2019-07-04 13:45:52 +01:00
Nick Craig-Wood 1510e12659 build: move race test to go modules build
In an attempt to even out the build times.
2019-07-03 12:07:29 +01:00
Nick Craig-Wood ede03258bc build: use go modules proxy when building modules 2019-07-03 12:07:29 +01:00
Nick Craig-Wood 7fcbb47b1c build: split other OS build into a separate builder
This is in order to make longest build (the Linux build) quicker
2019-07-03 12:07:29 +01:00
Nick Craig-Wood 9cafeeb4b6 dirtree: make tests more reliable 2019-07-02 16:29:40 +01:00
Nick Craig-Wood a1cfe61ffd googlephotos: Backend for accessing Google Photos #369 2019-07-02 15:26:55 +01:00
Nick Craig-Wood 5eebbaaac4 test_all: add tests parameter to limit which tests to run for a backend 2019-07-02 15:26:55 +01:00
Nick Craig-Wood bc70bff125 fs/dirtree: factor DirTree out of fs/walk and add tests 2019-07-02 15:26:55 +01:00
Nick Craig-Wood cf15b88efa build: make explicit which matrix items we will deploy 2019-07-02 14:13:46 +01:00
Nick Craig-Wood dcaee0016a build: add a builder for building with go modules 2019-07-02 12:33:03 +01:00
Nick Craig-Wood 387b496d1e operations: fix tests TestMoveFileBackupDir and TestCopyFileBackupDir again
Commit 734f504d5f wasn't tested properly and had a typo which
caused it not to build :-(
2019-07-02 12:22:29 +01:00
Nick Craig-Wood 734f504d5f operations: fix tests TestMoveFileBackupDir and TestCopyFileBackupDir
..so they don't run on backends which can't move or copy.
2019-07-02 10:46:49 +01:00
Nick Craig-Wood 5df5a3b78e vendor: tidy go.mod and go.sum - fixes #3317 2019-07-02 09:47:00 +01:00
Nick Craig-Wood 37c1144b46 Add Russell Davis to contributors 2019-07-02 07:57:18 +01:00
Nick Craig-Wood 8d116ba0c9 Add Matti Niemenmaa to contributors 2019-07-02 07:57:18 +01:00
Nick Craig-Wood 1be1fc073e Add AbelThar to contributors 2019-07-01 12:09:35 +01:00
Nick Craig-Wood af2596f98b Add yparitcher to contributors 2019-07-01 11:13:53 +01:00
Nick Craig-Wood 61fb326a80 Add full name for Laura Hausmann 2019-07-01 10:53:47 +01:00
Nick Craig-Wood 6bae3595a8 Add Laura to contributors 2019-06-30 20:07:35 +01:00
Nick Craig-Wood 9ed81ac451 vfs: fix tests for backends which can't upload 0 length files 2019-06-30 18:35:01 +01:00
Nick Craig-Wood 3245c0ae0d fstests: add integration test for uploading empty files
This tests a remote can upload empty files.  If the remote can't
upload empty files it should return fs.ErrorCantUploadEmptyFiles.
2019-06-30 18:35:01 +01:00
Nick Craig-Wood 07e2c3a50f b2: fix nil pointer error introduced by context propagation patch
For some reason f78cd1e043 introduced an unrelated change -
perhaps a merge error.  Removing this change fixes the nil pointer
problem.
2019-06-28 22:38:41 +01:00
Nick Craig-Wood d61ba7ef78 vendor: update all dependencies 2019-06-27 13:52:32 +01:00
Nick Craig-Wood b221d79273 Add nguyenhuuluan434 to contributors 2019-06-27 13:28:52 +01:00
Nick Craig-Wood 9f4589a997 gcs: reduce oauth scope requested as suggested by Google
As part of getting the rclone oauth consent screen approved by Google,
it came up that the scope in use by the gcs backend was too large.

This change reduces it to the minimum scope which still allows rclone
to work correctly.

Old scope: https://www.googleapis.com/auth/devstorage.full_control
New scope: https://www.googleapis.com/auth/devstorage.read_write
2019-06-27 12:05:49 +01:00
Nick Craig-Wood a1840f6fc7 sftp: add missing interface check and fix About #3257
This bug was introduced as part of adding context to the backends and
slipped through the net because the About call did not have an
interface assertion in the sftp backend.

I checked there were no other missing interface assertions on all the
optional methods on all the backends.
2019-06-26 16:56:33 +01:00
Nick Craig-Wood 4e64ee38e2 mount: default --deamon-timout to 15 minutes on macOS and FreeBSD
See: https://forum.rclone.org/t/macos-fuse-mount-contents-disappear-after-writes-while-using-vfs-cache/10566/
2019-06-25 15:30:42 +01:00
Nick Craig-Wood 276f8cccf6 rc: return current settings if core/bwlimit called without parameters 2019-06-24 13:22:24 +01:00
Nick Craig-Wood 0ae844d1f8 config: reset environment variables in config file test to fix build 2019-06-22 17:49:23 +01:00
Nick Craig-Wood 4ee6de5c3e docs: add a new page with global flags and link to it from the command docs
In f544234 we removed the global flags from each command as it was
making each page very big and causing 1000s of lines of duplication in
the man page.

This change adds a new flags page with all the global flags on and
links each command page to it.

Fixes #3273
2019-06-20 16:45:44 +01:00
Nick Craig-Wood 71a19a1972 Add Maran to contributors 2019-06-19 19:36:20 +01:00
Nick Craig-Wood 6a2a075c14 fs/cache: fix locking
This was causing `fatal error: sync: unlock of unlocked mutex` if a
panic ocurred in fsNewFs.
2019-06-19 10:50:59 +01:00
Nick Craig-Wood 628530362a local: add --local-case-sensitive and --local-case-insensitive
This is to force the remote to declare itself as case sensitive or
insensitive where the defaults for the operating system are wrong.

See: https://forum.rclone.org/t/duplicate-object-found-in-source-ignoring-dedupe-not-finding-anything/10465
2019-06-17 17:09:48 +01:00
Nick Craig-Wood 4549305fec Start v1.48.0-DEV development 2019-06-15 18:32:17 +01:00
Nick Craig-Wood 245fed513a Version v1.48.0 2019-06-15 13:55:41 +01:00
Nick Craig-Wood 52332a4b24 moveto: fix detection of same file name to include the root
Fixes problem introduced in d2be792d5e
2019-06-15 13:55:41 +01:00
Nick Craig-Wood 3087c5d559 webdav: retry on 423 Locked errors #3263 2019-06-15 10:58:13 +01:00
Nick Craig-Wood 75606dcc27 sync: fix tests on union remote 2019-06-15 10:53:36 +01:00
Nick Craig-Wood f3719fe269 fs/cache: unlock mutex in cache.Get to allow recursive calls
This fixes the test lockup in the union tests
2019-06-15 10:42:53 +01:00
Nick Craig-Wood ce400a8fdc Add rsync.net as a provider #3254 2019-06-15 09:56:17 +01:00
Nick Craig-Wood 49c05cb89b Add notes on --b2-version not working with crypt #1627
See also: https://forum.rclone.org/t/how-to-restore-a-previous-version-of-a-folde/10456/2
2019-06-15 09:56:17 +01:00
Nick Craig-Wood d533de0f5c docs: clarify that s3 can't access Glacier Vault 2019-06-15 09:56:17 +01:00
Nick Craig-Wood 1a4fe4bc6c Add Aleksandar Jankovic to contributors 2019-06-15 09:56:17 +01:00
Nick Craig-Wood 22368b997c b2: implement SetModTime #3210
SetModTime() is implemented by copying an object onto itself and
updating the metadata in the process.
2019-06-13 17:31:33 +01:00
Nick Craig-Wood a5bed67016 b2: implement server side copy - fixes #3210 2019-06-13 17:31:33 +01:00
Nick Craig-Wood 12c2a750f5 build: fix build lockup by increasing GOMAXPROCS - Fixes #3154
It was discovered after lots of experimentation that the cmd/mount
tests have a tendency to lock up if GOMAXPROCS=1 or 2.  Since the
Travis builders only have 2 vCPUs by default, this happens on the
build server very often.

This workaround increases GOMAXPROCS to make the mount test lockup
less likely.

Ideally this should be fixed in the mount tests at some point.
2019-06-13 13:47:43 +01:00
Nick Craig-Wood 92bbae5cca Add Florian Apolloner to contributors 2019-06-13 13:47:43 +01:00
Nick Craig-Wood 64fb4effa7 docs: add FAQ entry about rclone using too much memory
See: #2200 #1391 3196
2019-06-12 11:08:19 +01:00
Nick Craig-Wood 4d195d5a52 gcs: Fix upload errors when uploading pre 1970 files
Before this change rclone attempted to set the "updated" field in
uploaded objects to the modification time.

However when this modification time was before 1970, google drive
would return the rather cryptic error:

    googleapi: Error 400: Invalid value for UnsignedLong: -42000, invalid

However API docs: https://cloud.google.com/storage/docs/json_api/v1/objects#resource
state the "updated" field is read only and tests confirm that.  Even
though the field is read only, it looks like Google parses it.

This change therefore removes the attempt to set the "updated" field
(which was doing nothing anyway) and fixes the problem uploading pre
1970 files.

See #3196 and https://forum.rclone.org/t/invalid-value-for-unsignedlong-file-missing-date-modified/3466
2019-06-12 10:51:49 +01:00
Nick Craig-Wood 550ab441c5 rc: Skip auth for OPTIONS request
Before this change using --user and --pass was impossible on the rc
from a browser as the browser needed to make the OPTIONS request first
before sending Authorization: headers, but the OPTIONS request
required an Authorization: header.

After this change we allow OPTIONS requests to go through without
checking the Authorization: header.
2019-06-10 19:33:45 +01:00
Nick Craig-Wood e24cadc7a1 box: Fix ineffectual assignment (ineffassign) 2019-06-10 19:33:10 +01:00
Nick Craig-Wood 903ede52cd config: make config create/update encrypt passwords where necessary
Before this change when using "rclone config create" it wasn't
possible to add passwords in one go, it was necessary to call "rclone
config password" to add the passwords afterwards as "rclone config
create" didn't obscure passwords.

After this change "rclone config create" and "rclone config update"
will obscure passwords as necessary as will the corresponding API
calls config/create and config/update.

This makes "rclone config password" and its API config/password
obsolete, however they will be left for backwards compatibility.
2019-06-10 18:08:55 +01:00
Nick Craig-Wood f681d32996 rc: Fix serving bucket based objects with --rc-serve
Before this change serving bucket based objects
`[remote:bucket]/path/to/object` would fail with 404 not found.

This was because the leading `/` in `/path/to/object` was being passed
to NewObject.
2019-06-10 11:59:06 +01:00
Nick Craig-Wood 45f41c2c4a Add forgems to contributors 2019-06-09 16:02:20 +01:00
Nick Craig-Wood fb6966b5fe docs: Fix warnings after hugo upgrade to v0.55 2019-06-08 11:55:51 +01:00
Nick Craig-Wood 454dfd3c9e rc: Add operations/fsinfo: Return information about the remote
This returns a information about the remote including Name, Root,
Hashes and optional Features.
2019-06-08 09:19:07 +01:00
Nick Craig-Wood e1cf551ded fs: add Features.Enabled to return map of enabled features by name 2019-06-08 08:46:53 +01:00
Nick Craig-Wood bd10344d65 rc: add --loopback flag to run commands directly without a server 2019-06-08 08:45:55 +01:00
Nick Craig-Wood 1aa65d60e1 lsjson: add IsBucket field for bucket based remote listing of the root 2019-06-07 17:28:15 +01:00
Nick Craig-Wood aa81957586 drive: add --drive-server-side-across-configs
In #2728 and 55b9a4e we decided to allow server side operations
between google drives with different configurations.

This works in some cases (eg between teamdrives) but does not work in
the general case, and this caused breakage in quite a number of
people's workflows.

This change makes the feature conditional on the
--drive-server-side-across-configs flag which defaults to off.

See: https://forum.rclone.org/t/gdrive-to-gdrive-error-404-file-not-found/9621/10

Fixes #3119
2019-06-07 12:12:49 +01:00
Nick Craig-Wood b7800e96d7 vendor: update golang.org/x/net/webdav - fixes #3002
This fixes duplicacy working with rclone serve webdav
2019-06-07 11:54:57 +01:00
Nick Craig-Wood ac4c8d8dfc cmd/providers: add DefaultStr, ValueStr and Type fields
These fields are auto generated:
- DefaultStr - a string rendering of Default
- ValueStr - a string rendering of Value
- Type - the type of the option
2019-06-05 16:23:42 +01:00
Nick Craig-Wood e2b6172f7d build: stop using a GITHUB_USER as secure variable to fix build log
Before this change any occurences of `ncw` were escaped as `[secure]`
in the build log which is needless obfuscation and quite confusing.
2019-06-05 16:23:16 +01:00
Nick Craig-Wood 32f2895472 Add garry415 to contributors 2019-06-03 21:12:55 +01:00
Nick Craig-Wood cd5a2d80ca Add JorisE to contributors 2019-06-03 21:08:15 +01:00
Nick Craig-Wood 5a941cdcdc local: fix preallocate warning on Linux with ZFS
Under Linux, rclone attempts to preallocate files for efficiency.

Before this change, pre-allocation would fail on ZFS with the error

    Failed to pre-allocate: operation not supported

After this change rclone tries a different flag combination for ZFS
then disables pre-allocate if that doesn't work.

Fixes #3066
2019-06-03 18:02:26 +01:00
Nick Craig-Wood 62681e45fb Add Philip Harvey to contributors 2019-06-03 17:34:44 +01:00
Nick Craig-Wood ec4e7316f2 docs: update advice for avoiding mega logins 2019-05-29 17:37:53 +01:00
Nick Craig-Wood e7c20e0bce operations: make move and copy individual files obey --backup-dir
Before this change, when using rclone copy or move with --backup-dir
and the source was a single file, rclone would fail to use the backup
directory.

This change looks up the backup directory in the Fs cache and uses it
as appropriate.

This affects any commands which call operations.MoveFile or
operations.CopyFile which includes rclone move/moveto/copy/copyto
where the source is a single file.

Fixes #3219
2019-05-27 16:14:55 +01:00
Nick Craig-Wood 8ee6034b23 Look for Fs in the cache rather than calling NewFs directly
This will save operations when rclone is used in remote control mode
or with the same remote multiple times in the command line.
2019-05-27 16:14:55 +01:00
Nick Craig-Wood 206e1caa99 fs/cache: factor Fs caching from fs/rc into its own package 2019-05-27 16:14:55 +01:00
Nick Craig-Wood da3b685cd8 vendor: update github.com/pkg/sftp to fix sftp client issues
See: https://forum.rclone.org/t/failed-to-copy-sftp-folder-not-found-c-ftpsites-ssh-fx-failure/9778
See: https://github.com/pkg/sftp/issues/288
2019-05-24 15:03:23 +01:00
Nick Craig-Wood 9aac2d6965 drive: add notes that cleanup works in the background on drive 2019-05-22 20:48:59 +01:00
Nick Craig-Wood cff85f0b95 docs: add note on failure to login to mega docs 2019-05-16 10:10:58 +01:00
Nick Craig-Wood 9c0dac4ccd Add Robert Marko to contributors 2019-05-15 13:36:15 +01:00
Nick Craig-Wood 2f3d794ec6 Add id01 to contributors 2019-05-14 07:55:38 +01:00
Nick Craig-Wood 9c6f3ae82c local: log errors when listing instead of returning an error
Before this change, rclone would return an error from the listing if
there was an unreadable directory, or if there was a problem stat-ing
a directory entry.  This was frustrating because the command
completely aborts at that point when there is work it could do.

After this change rclone lists the directories and reports ERRORs for
unreadable directories or problems stat-ing files, but does return an
error from the listing.  It does set the error flag which means the
command will fail (and objects won't be deleted with `rclone sync`).

This brings rclone's behaviour exactly in to line with rsync's
behaviour.  It does as much as possible, but doesn't let the errors
pass silently.

Fixes #3179
2019-05-13 18:30:33 +01:00
Nick Craig-Wood 870b15313e cmd: log an ERROR for all commands which exit with non-zero status
Before this change, rclone didn't report errors for commands which
didn't return an error directly.  For example `rclone ls` could
encounter an error and rclone would log nothing, even though the exit
code was non zero.

After this change we always log a message if we are exiting with a
non-zero exit code.
2019-05-13 18:28:21 +01:00
Nick Craig-Wood 62a7e44e86 Add didil to contributors 2019-05-13 17:37:46 +01:00
Nick Craig-Wood a0b9d4a239 docs: fix doc generators home directory in backend docs
Thanks @ctlaltdefeat for spotting this
2019-05-13 17:28:36 +01:00
Nick Craig-Wood 99bc013c0a crypt: remove stray debug in ChangeNotify 2019-05-12 16:50:03 +01:00
Nick Craig-Wood d9cad9d10b mega: cleanup: add logs for -v and -vv 2019-05-12 10:46:21 +01:00
Nick Craig-Wood 0e23c4542f sync: fix integrations tests
2eb31a4f1d broke the integration tests for remotes which use
Copy+Delete as server side Move.
2019-05-12 09:50:20 +01:00
Nick Craig-Wood f544234e26 gendocs: remove global flags from command help pages 2019-05-11 23:39:50 +01:00
Nick Craig-Wood dbf9800cbc docs: add serve to README, main page and menu 2019-05-11 23:39:50 +01:00