Commit Graph
100 Commits
Author SHA1 Message Date
Nick Craig-Wood 1b2ffbeca0 cmd: fix environment variables not setting command line flags
Before this fix quite a lot of the commands were ignoring environment
variables intended to set flags.
2019-10-14 17:02:09 +01:00
Nick Craig-Wood 19429083ad cmd: fix spelling of Definition 2019-10-14 17:02:09 +01:00
Nick Craig-Wood 6e378d7d32 config: fix setting of non top level flags from environment variables
Before this fix, attempting to set a non top level environment
variable would fail with "Couldn't find flag".

This fixes it by passing in the flags that the env var is being set
from.

Fixes #3615
2019-10-14 17:02:09 +01:00
Nick Craig-Wood 1fe1a19339 vfs: stop empty dirs disappearing when renamed on bucket based remotes
Before this change when we renamed a directory this cleared the
directory cache for the parent directory too.

If the directory was remaining in the same parent this wasn't
necessary and caused the empty directory to fall out of the cache.

Fixes #3597
2019-10-14 14:38:30 +01:00
Nick Craig-Wood b4b59c53f1 mount: fix "mount_fusefs: -o timeout=: option not supported" on FreeBSD
Before this change `rclone mount` would give this error on FreeBSD

    mount helper error: mount_fusefs: -o timeout=: option not supported

Because the default value for FreeBSD was set to 15m for
--daemon-timeout and that FreeBSD does not support the timeout option.

This change sets the default for --daemon-timeout to 0 on FreeBSD
which fixes the problem.

Fixes #3610
2019-10-13 11:36:51 +01:00
Nick Craig-Wood f3b0f8a9f0 sync: --update/-u not transfer files that haven't changed - fixes #3232
Before this change --update would transfer any file which was newer
than the destination regardless of whether it had changed or not.
This is needlessly wasteful of bandwidth.

After this change --update will only transfer files if they are newer
**and** they are different (checked with checksum and size).
2019-10-12 11:54:56 +01:00
Nick Craig-Wood 65a82fe77d dropbox: fix nil pointer exception on restricted files
See: https://forum.rclone.org/t/issues-syncing-dropbox/12233
2019-10-11 16:21:24 +01:00
Nick Craig-Wood c892a6f8ef Add Michele Caci to contributors 2019-10-11 16:17:24 +01:00
Nick Craig-Wood bc45f6f952 Add Arijit Biswas to contributors 2019-10-11 15:25:20 +01:00
Nick Craig-Wood a4d572d004 Add Vighnesh SK to contributors 2019-10-10 16:05:29 +01:00
Nick Craig-Wood 6abaa9e22c fstests: allow skipping of the broken UTF-8 test for the cache backend 2019-10-10 10:36:18 +01:00
Nick Craig-Wood e8b92f4853 sftp: fix test failures
This was introduced by 50a3a96e27
2019-10-09 17:43:03 +01:00
Nick Craig-Wood 50a3a96e27 serve sftp: fix crash on unsupported operations (eg Readlink)
Before this change the sftp handler returned a nil error for unknown
operations which meant the server crashed when one was encountered.

In particular the "Readlink" operations was causing problems.

After this change the handler returns ErrSshFxOpUnsupported which
signals to the remote end that we don't support that operation.

See: https://forum.rclone.org/t/rclone-serve-sftp-not-working-in-windows/12209
2019-10-09 16:12:21 +01:00
Nick Craig-Wood 3f40849343 Add Brett Dutro to contributors 2019-10-09 10:07:50 +01:00
Nick Craig-Wood 7271a404db Add Tyler to contributors 2019-10-09 10:07:50 +01:00
Nick Craig-Wood 8c1edf410c dropbox: make disallowed filenames return no retry error - fixes #3569
Before this change we silently skipped uploads to dropbox of
disallowed file names.  However this then caused "corrupted on
transfer" errors because the sizes were wrong.

After this change we return an no retry error which will mean that the
sync fails (as it should - not all files were uploaded) but no
unecessary retries happened.
2019-10-08 19:59:47 +01:00
Nick Craig-Wood 1833167d10 vendor: run go mod tidy / go mod vendor with go1.13 2019-10-08 19:59:47 +01:00
Nick Craig-Wood 455b9280ba config: use alternating Red/Green in config to make more obvious 2019-10-08 19:59:47 +01:00
Nick Craig-Wood 45e440d356 vendor: remove github.com/Azure/go-ansiterm 2019-10-08 19:59:47 +01:00
Nick Craig-Wood 593de059be lib/terminal: factor from cmd/progress, swap Azure/go-ansiterm for mattn/go-colorable 2019-10-08 19:59:47 +01:00
Nick Craig-Wood c78d1dd18b vendor: add github.com/mattn/go-colorable 2019-10-08 19:59:47 +01:00
Nick Craig-Wood 2a82aca225 Add Sezal Agrawal to contributors 2019-10-08 19:59:47 +01:00
Nick Craig-Wood 15b1feea9d mount: fix panic on File.Open - Fixes #3595
This problem was introduced in "mount: allow files of unkown size to
be read properly" 0baafb158f by failure to check that the
DirEntry was nil or not.
2019-10-06 19:26:58 +01:00
Nick Craig-Wood d210fecf3b Add Raphael to contributors 2019-10-05 17:07:02 +01:00
Nick Craig-Wood f962fb9499 Add SwitchJS to contributors 2019-10-05 17:07:02 +01:00
Nick Craig-Wood d15425e8c8 Start v1.49.5-DEV development 2019-10-05 12:42:28 +01:00
Nick Craig-Wood b3faee9471 build: fix macOS build after brew changes 2019-10-05 11:51:28 +01:00
Nick Craig-Wood 5271fe3b3f yandex: use lib/encoder 2019-10-05 10:22:43 +01:00
Nick Craig-Wood 7da1c84a7f build: don't deploy xgo build on pull requests 2019-10-04 16:53:51 +01:00
Nick Craig-Wood cbdab14057 Add 庄天翼 to contributors 2019-10-04 16:53:51 +01:00
Nick Craig-Wood d21ddf280c mailru: comment out some debugging statements 2019-10-02 20:10:01 +01:00
Nick Craig-Wood 135717e12b mailru: use lib/encoder 2019-10-02 20:10:01 +01:00
Nick Craig-Wood b94b2a3723 mega: fix after lib/encoder change 2019-10-02 12:41:52 +01:00
Nick Craig-Wood e2914c0097 test_all: ignore some encoding tests with nextcloud integration test 2019-10-02 11:34:08 +01:00
Nick Craig-Wood fd51f24906 putio: use lib/encoder
And in the process
- fix a bug with + and & in file name
- fix NewObject returning directories as files
2019-10-02 11:34:08 +01:00
Nick Craig-Wood 4615343b73 premiumizeme: use lib/encoder 2019-10-02 11:34:08 +01:00
Nick Craig-Wood def411da62 build: use the release builds not master of nfpm and github-release
Fixes #3580
2019-10-01 16:23:36 +01:00
Nick Craig-Wood f73dae1e77 bin/get-github-release: support tar.bz2 files 2019-10-01 16:23:36 +01:00
Nick Craig-Wood 77a520c97c fichier: fix accessing files > 2GB on 32 bit systems - fixes #3581 2019-10-01 16:03:49 +01:00
Nick Craig-Wood 23bf6bb4d8 test_all: mark expected failures for minio, wasabi and FTP 2019-10-01 15:40:32 +01:00
Nick Craig-Wood 04eb96b50b fichier: fix NewObject after lib/encoder changes
This bug was introduced as part of the lib/encoder changes in
8d8fad724b.  It caused NewObject not to work for a file with
escaped characters in it.
2019-10-01 15:30:51 +01:00
Nick Craig-Wood b581f2de26 sharefile: use lib/encoder 2019-09-30 22:00:25 +01:00
Nick Craig-Wood 5cef5f8b49 lib/encoder: add LeftPeriod encoding 2019-09-30 22:00:25 +01:00
Nick Craig-Wood 8d8fad724b ficher: use lib/encoder 2019-09-30 22:00:25 +01:00
Nick Craig-Wood 4098907511 lib/encoder: add more encode symbols and split existing 2019-09-30 22:00:25 +01:00
Nick Craig-Wood 5b8a339baf docs: Add notes on how to find out the encodings used in a backend 2019-09-30 22:00:25 +01:00
Nick Craig-Wood 3e53376a49 build_csv: fix output of control characters 2019-09-30 22:00:25 +01:00
Nick Craig-Wood d122d1d191 qingstor: use lib/encoder 2019-09-30 22:00:25 +01:00
Nick Craig-Wood 35d6ff89bf azureblob: use lib/encoder 2019-09-30 22:00:24 +01:00
Nick Craig-Wood 53bec33027 swift: use lib/encoder 2019-09-30 22:00:24 +01:00
Nick Craig-Wood f55a99218c lib/encoder: add CrLf encoding 2019-09-30 22:00:24 +01:00
Nick Craig-Wood 6e053ecbd0 s3: only ask for URL encoded directory listings if we need them on Ceph
This works around a bug in Ceph which doesn't encode CommonPrefixes
when using URL encoded directory listings.

See: https://tracker.ceph.com/issues/41870
2019-09-30 22:00:24 +01:00
Nick Craig-Wood 7e738c9d71 fstest: remove WinPath as it is no longer needed 2019-09-30 22:00:24 +01:00
Nick Craig-Wood a8adce9c59 s3: fix encoding for control characters - Fixes #3345 2019-09-30 22:00:24 +01:00
Nick Craig-Wood 6ae7bd7914 local: encode invalid UTF-8 on macOS 2019-09-30 22:00:24 +01:00
Nick Craig-Wood 32af4cd6f3 ftp: use lib/encoder 2019-09-30 22:00:24 +01:00
Nick Craig-Wood ced2616da5 fstests: allow Purge to fail with ErrorDirNotFound 2019-09-30 22:00:24 +01:00
Nick Craig-Wood b90e4a8769 sftp: fix hashes of files with backslashes 2019-09-30 22:00:24 +01:00
Nick Craig-Wood 57c1284df7 fstests: make integration tests to check all backends can store any file name
This tests the encoder is working properly
2019-09-30 22:00:24 +01:00
Nick Craig-Wood 4627ac5709 New backend for Citrix Sharefile - Fixes #1543
Many thanks to Bob Droog for organizing a test account and extensive
testing.
2019-09-30 12:28:33 +01:00
Nick Craig-Wood 1e7144eb63 docs: Add more notes on making backend docs 2019-09-30 12:27:03 +01:00
Nick Craig-Wood f29e5b6e7d lib/oauthutil: refactor web server and allow an auth callback 2019-09-30 11:34:30 +01:00
Nick Craig-Wood 25a0e7e8aa lib/oauthutil: add a new redirect URL for oauth.rclone.org
This is for use with oauth providers which won't accept http: links.
2019-09-30 11:23:21 +01:00
Nick Craig-Wood 262ba28dec config: add ReadNonEmptyLine utility function 2019-09-30 11:23:21 +01:00
Nick Craig-Wood 74f6300875 Start v1.49.4-DEV development 2019-09-29 19:47:59 +01:00
Nick Craig-Wood 86dcb54c38 fstests: make tests pass when using -remote :backend: 2019-09-29 17:25:54 +01:00
Nick Craig-Wood 25a0703b45 Add Richard Patel to contributors 2019-09-29 11:14:33 +01:00
Nick Craig-Wood 349112df6b oauthutil: fix security problem when running with two users on the same machine
Before this change two users could run `rclone config` for the same
backend on the same machine at the same time.

User A would get as far as starting the web server.  User B would then
fail to start the webserver, but it would open the browser on the
/auth URL which would redirect the user to the login.  This would then
cause user B to authenticate to user A's rclone.

This changes fixes the problem in two ways.

Firstly it passes the state to the /auth call before redirecting and
checks it there, erroring with a 403 error if it doesn't match.  This
would have fixed the problem on its own.

Secondly it delays the opening of the web browser until after the auth
webserver has started which prevents the user entering the credentials
if another auth server is running.

Fixes #3573
2019-09-29 10:42:02 +01:00
Nick Craig-Wood fef8b98be2 ftp: fix listing of an empty root returning: error dir not found
Before this change if rclone listed an empty root directory then it
would return an error dir not found.

After this change we assume the root directory exists and don't
attempt to check it which was failing before.

See: https://forum.rclone.org/t/ftp-empty-directory-yields-directory-not-found-error/12069/
2019-09-28 18:01:12 +01:00
Nick Craig-Wood 6750af6167 build: make VERSION file be master of the last release - fixes #3570
Prior to this beta releases would appear to be older than the point
release, eg v1.49.0-096-gc41812fc which was released after v1.49.3 and
contains all the patches from v1.49.3.
2019-09-26 16:51:44 +01:00
Nick Craig-Wood 8681ef36d6 build: replace Circle CI build and make GitHub actions the default CI 2019-09-25 16:38:10 +01:00
Nick Craig-Wood ec9914205f build: remove Appveyor, Circle CI, Travis and Pkgr builds 2019-09-25 16:38:10 +01:00
Nick Craig-Wood 28cc2009d4 Add Anthony Rusdi to contributors 2019-09-21 14:39:03 +01:00
Nick Craig-Wood dd4fe9ff60 Add David to contributors 2019-09-21 14:39:03 +01:00
Nick Craig-Wood 1031bcfc5a build: remove azure pipelines build 2019-09-20 16:08:18 +01:00
Nick Craig-Wood ce00c0a0d9 build: build rclone with github actions 2019-09-20 16:08:18 +01:00
Nick Craig-Wood 1164eed2af lib/pacer: make tests more reliable 2019-09-20 16:07:55 +01:00
Nick Craig-Wood 557edecd40 log: add Stack() function for debugging who calls what 2019-09-20 11:53:08 +01:00
Nick Craig-Wood b242b0a078 lib/cache,rc/jobs: make tests more reliable 2019-09-20 11:53:08 +01:00
Nick Craig-Wood 08b86cc94b mount: skip tests on <= 2 CPUs to avoid lockup in #3154 2019-09-20 11:53:08 +01:00
Nick Craig-Wood 56544bb2fd accounting: fix file handle leak on errors - fixes #3547
In 53a1a0e3ef we introduced a problem where if there was an
error on the file being transferred then the file was re-opened and
the old one wasn't closed.

This was partially fixed in bfbddab46b however this didn't
address the case of the old file being closed.

This is now fixed by
- marking the file as open again in UpdateReader
- moving the stopping the accounting machinery to a new method Done
2019-09-19 16:20:07 +01:00
Nick Craig-Wood 3cff258577 sftp: fix --sftp-ask-password trying to contact the ssh agent
See: https://forum.rclone.org/t/rclone-command-line/11766
2019-09-16 11:16:27 +01:00
Nick Craig-Wood 66347aff2a fstest: calculate hashes for uploaded test files to fix minio integration tests
Before this change we didn't calculate any hashes for test files
created in the Run framework.

This means that files were uploaded to S3 without a `Content-MD5`
header.  This in turn caused minio to disengage `--compat` mode which
in turn caused the `TestSyncAfterChangingModtimeOnlyWithNoUpdateModTime`
test to fail in `fs/sync`.

After this change we supply all hashes supported by the destination Fs
on the upload object.

This means that the `Content-MD5` is set and minio engages `--compat`
mode to fix the problem.  Using `--compat` on the command line also
fixes the problem.

This much better replicates how objects are actually uploaded with
operations.Copy so should improve the integration tests.
2019-09-16 10:59:01 +01:00
Nick Craig-Wood b8b12a4000 Start v1.49.3-DEV development 2019-09-15 18:10:08 +01:00
Nick Craig-Wood b6065561cf test_all: add ignores for tests which will never pass
- s3 backends which don't support SetTier
- mega which makes a duplicate for TestDirRename
2019-09-15 13:16:15 +01:00
Nick Craig-Wood ef7bfd3f03 fs: Make prefix free backend config read prefix free env var also
Before this change you could only configure the local backend flags
which don't have the local prefix (eg `--copy-links`) with
`RCLONE_LOCAL_COPY_LINKS`.

This change makes `RCLONE_COPY_LINKS` valid too which is much more
logical for the users.

Fixes #3534
2019-09-14 18:26:07 +01:00
Nick Craig-Wood ae2edc3b5b help: add short options to backend documentation also 2019-09-14 18:24:05 +01:00
Nick Craig-Wood 0baafb158f mount: allow files of unkown size to be read properly
Before this change, files of unknown size (eg Google Docs) would
appear in file listings with 0 size and would only allow 0 bytes to be
read.

This change sets the direct_io flag in the FUSE return which bypasses
the cache for these files.  This means that they can be read properly.

This is compatible with some, but not all applications.
2019-09-14 13:22:33 +01:00
Nick Craig-Wood ba121eddf0 vfs: make objects of unknown size readable through the VFS
These objects (eg Google Docs) appear with 0 length in the VFS.

Before this change, these only read 0 bytes.

After this change, even though the size appears to be 0, the objects
can be read to the end.  If the objects are read to the end then the
size on the handle will be updated.
2019-09-14 13:09:07 +01:00
Nick Craig-Wood 2e80e035c9 fstest/mockobject: add UnknownSize() method to make Size() return -1 2019-09-14 13:07:01 +01:00
Nick Craig-Wood ea9b6087cf fstest/mockfs: allow fs.Objects to be added to the root 2019-09-14 13:05:36 +01:00
Nick Craig-Wood 6959c997e2 config: remove error: can't use --size-only and --ignore-size together.
As sometimes they are useful together, for example when the remote
changes the sizes of the uploaded files.

See: https://forum.rclone.org/t/files-upload-will-be-auto-compress-how-do-i-sync-a-file-to-remote/10578
2019-09-14 10:13:44 +01:00
Nick Craig-Wood 25786cafd3 s3: fix SetModTime on GLACIER/ARCHIVE objects and implement set/get tier
- Read the storage class for each object
- Implement SetTier/GetTier
- Check the storage class on the **object** before using SetModTime

This updates the fix in 1a2fb52 so that SetModTime works when you are
using objects which have been migrated to GLACIER but you aren't using
GLACIER as a storage class.

Fixes #3522
2019-09-14 09:18:55 +01:00
Nick Craig-Wood 23dc313fa5 azureblob: add missing type assertions for GetTier/SetTier 2019-09-14 09:18:55 +01:00
Nick Craig-Wood 1a16849df0 http: fix race introduced in 7982aaf151 2019-09-14 08:48:13 +01:00
Nick Craig-Wood 3b68340eac http: add --http-no-head to stop rclone doing HEAD in listings #3523 2019-09-14 00:17:39 +01:00
Nick Craig-Wood 7982aaf151 http: HEAD directory entries in parallel to speedup #3523 2019-09-14 00:16:44 +01:00
Nick Craig-Wood 7b29ed8ec1 Add Lars Lehtonen to contributors 2019-09-13 23:50:54 +01:00
Nick Craig-Wood 3b91fb6a2f Add David Baumgold to contributors 2019-09-13 18:39:36 +01:00
Nick Craig-Wood bfbddab46b fs/accounting: Fix "file already closed" on transfer retries
This was caused by the recent reworking of the accounting interface.
The Transfer object was recycling the Accounting object without
resetting the stream.

See: https://forum.rclone.org/t/error-file-already-closed/11469/
See: https://forum.rclone.org/t/rclone-b2-sync-post-error-method-not-supported/11718/
2019-09-13 18:35:02 +01:00