f-denkena/rclone
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
5,146 Commits 1 Branch 0 Tags
bcac8fdc8377af57e2f29bc30b5626052ceb3981
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick Craig-Wood bcac8fdc83 Use http.NewRequestWithContext where possible after go1.13 minimum version 2021-02-03 17:41:27 +00:00
Nick Craig-Wood 15e1a6bee7 build: raise minimum go version to go1.13 2021-02-03 17:41:06 +00:00
Nick Craig-Wood 9710ded60f b2: automatically raise upload cutoff to avoid spurious error 
Before this change, if --b2-chunk-size was raised above 200M then this
error would be produced:

    b2: upload cutoff: 200M is less than chunk size 1G

This change automatically reaises --b2-upload-cutoff to be the value
of --b2-chunk-size if it is below it, which stops this error being
generated.

Fixes #4475
 2021-02-03 16:29:32 +00:00
Nick Craig-Wood 5f3672102c s3: add --s3-no-head to reducing costs docs - Fixes #2163 2021-02-03 16:18:29 +00:00
Nick Craig-Wood 644cc69108 build: update GitHub release tool to use gh and put a link to changelog 
Fixes #4994
 2021-02-03 14:44:40 +00:00
Nick Craig-Wood 1415666074 lsjson: fix unterminated JSON in the presence of errors 
See: https://forum.rclone.org/t/rclone-lsjson-invalid-json-produced-no-at-the-end/22046
 2021-02-02 17:46:56 +00:00
Nick Craig-Wood beff081abb Start v1.55.0-DEV development 2021-02-02 16:30:58 +00:00
Nick Craig-Wood 7f5ee5d81f Version v1.54.0 2021-02-02 14:17:09 +00:00
Nick Craig-Wood 8b41dfa50a s3: add --s3-no-head parameter to minimise transactions on upload 
See: https://forum.rclone.org/t/prevent-head-on-amazon-s3-family/21935
 2021-02-02 10:07:48 +00:00
Nick Craig-Wood 0d8bcc08da Revert "fs/accounting: make edge bandwidth limiters have smaller bursts to make smoother" #4987 
This reverts commit 463a18aa07.

In practice this caused Windows to have a max bandwidth limit of 8
MiB/s. This is because of the limited resolution (1ms) of the Windows
timer.

This needs fixing in a different way.

See: https://forum.rclone.org/t/bug-report-for-v1-54-0-beta-5133/22015
 2021-02-01 22:23:50 +00:00
Nick Craig-Wood f66928a846 drive: fix copyid command with a bare filename: can't use empty string as a path 
Before this change, running

    rclone backend copyid drive: ID file.txt

Failed with the error

    command "copyid" failed: failed copying "ID" "file.txt": can't use empty string as a path

This fixes the problem.
 2021-02-01 15:17:25 +00:00
Nick Craig-Wood 3b1122c888 azureblob: require go1.14+ to compile due to SDK changes 2021-01-30 18:01:12 +00:00
Nick Craig-Wood 463a18aa07 fs/accounting: make edge bandwidth limiters have smaller bursts to make smoother 
This change decreases the edge limiter burst size which dramatically
increases the smoothness of the bandwidth limiting.

The core bandwidth limiter remains with a large burst so it isn't
affected by double rate limiting on the edge limiters.

See: #4395
See: https://forum.rclone.org/t/bwlimit-is-not-really-smooth/20947
 2021-01-30 17:39:30 +00:00
Nick Craig-Wood 0a932dc1f2 Add --bwlimit for upload and download #1873 2021-01-30 17:39:30 +00:00
Nick Craig-Wood 8856e0e559 fshttp: Obey bwlimit in http Transport - fixes #4395 
This change uses the bwlimit code to apply limits to the receive and
transmit data functions in the HTTP Transport.

This means that all HTTP transactions will have limiting applied -
this includes listings for example.

For HTTP based transorts this makes the limiting in Accounting
redundant and possibly counter productive
 2021-01-30 17:39:30 +00:00
Nick Craig-Wood 3b6df71838 accounting: refactor bwlimit code to allow for multiple slots 2021-01-30 17:39:30 +00:00
Nick Craig-Wood 31de631b22 vendor: run go mod tidy 2021-01-30 17:28:27 +00:00
Nick Craig-Wood 189ef5f257 azureblob: fix memory usage by upgrading the SDK and implementing a TransferManager 
In the Azure SDK there was a bug which caused excessive memory use
when doing repeated transfers:

https://github.com/Azure/azure-storage-blob-go/issues/233

This patch updates the SDK to v0.13.0 which allowed us to implement a
custom TransferManager which integrates with rclone's memory pool.

This fixes the excessive memory consumption.

See: https://forum.rclone.org/t/ask-for-settings-recommendation-for-azureblob/21505/
 2021-01-30 17:26:59 +00:00
Nick Craig-Wood 2f67681e3b cmount: don't attempt to unmount if fs has been destroyed already #4957 2021-01-30 17:19:28 +00:00
Nick Craig-Wood 41127965b0 fstest: add Onedrive Business and Onedrive China to the integration tests 2021-01-30 16:32:32 +00:00
Nick Craig-Wood 8171671d82 Add Bob Pusateri to contributors 2021-01-30 16:24:54 +00:00
Nick Craig-Wood 75617c0c3b Add Pau Rodriguez-Estivill to contributors 2021-01-30 16:24:54 +00:00
Nick Craig-Wood 8b9d23916b Add Nicolas Rueff to contributors 2021-01-30 16:24:54 +00:00
Nick Craig-Wood 06f1c0c61c move: fix data loss when moving the same object 
This checks to see if IDs are the same of the source and destination
object before deleting one of them and potentially causing data loss.

See: https://forum.rclone.org/t/files-deleted-and-not-moved/21830
 2021-01-28 16:14:16 +00:00
Nick Craig-Wood e6a9f005d6 sftp: implement --sftp-use-fstat 
See: https://forum.rclone.org/t/sftp-fails-to-sync-to-local-failed-to-copy-file-does-not-exist/21759
 2021-01-28 16:07:26 +00:00
Nick Craig-Wood 8f6f4b053c obscure: make rclone osbcure - ignore newline at end of line 
See: https://forum.rclone.org/t/authentification-issues-with-webdav-server/21891
 2021-01-28 15:54:41 +00:00
Nick Craig-Wood fe15a2eeeb Add Riccardo Iaconelli to contributors 2021-01-28 15:54:41 +00:00
Nick Craig-Wood 019667170f Add Zach Kipp to contributors 2021-01-28 15:54:41 +00:00
Nick Craig-Wood 78afe01d15 filefabric: fix finding directories in a case insensitive way #4830 2021-01-27 14:28:17 +00:00
Nick Craig-Wood 4eac88babf premiumizeme: fix finding directories in a case insensitive way #4830 2021-01-27 14:28:17 +00:00
Nick Craig-Wood b4217fabd3 opendrive: fix finding directories in a case insensitive way #4830 2021-01-27 14:28:17 +00:00
Nick Craig-Wood 92b9dabf3c fstests: only test with ASCII uppercase for case insensitive tests 
Upper/lower case for multilingual is provider dependent so we don't go
there in the tests.
 2021-01-27 14:28:17 +00:00
Nick Craig-Wood 3e188495f5 sugarsync: fix finding directories in a case insentive way #4830 2021-01-26 14:48:33 +00:00
Nick Craig-Wood acb9e17eb3 box: fix finding directories in a case insentive way #4830 2021-01-26 14:48:33 +00:00
Nick Craig-Wood c8ab4f1d02 fstests: add a test for finding a directory in a case insensitive way #4830 2021-01-26 14:48:06 +00:00
Nick Craig-Wood e776a1b122 fstests: don't run encoding tests on -short 2021-01-26 14:46:23 +00:00
Nick Craig-Wood c57af26de9 docs: add note about --check-first being useful with --order-by 2021-01-26 12:12:26 +00:00
Nick Craig-Wood 7d89912666 Add lluuaapp to contributors 2021-01-26 12:12:26 +00:00
Nick Craig-Wood 3e986cdf54 dedup: add warning if used on a remote which can't have duplicate names 2021-01-22 15:25:35 +00:00
Nick Craig-Wood b80d498304 vfs: fix file leaks with --vfs-cache-mode full and --buffer-size 0 
Before this change using --vfs-cache-mode full and --buffer-size 0
together caused the vfs downloader to open more and more downloaders.

This is fixed by introducing a minimum size of 1M for the window to
look for an existing downloader.

Fixes #4892
 2021-01-21 18:35:04 +00:00
Nick Craig-Wood 757e696a6b Add Sơn Trần-Nguyễn to contributors 2021-01-21 18:35:04 +00:00
Nick Craig-Wood e3979131f2 Add CokeMine to contributors 2021-01-21 18:35:04 +00:00
Nick Craig-Wood a774f6bfdb qingstor: fix rclone cleanup 
This patch changes to using the default page limit for listing
unfinished multpart uploads rather than 1000. 1000 is the maximum
specified in the docs, but setting anything larger than 200 gives an
error.
 2021-01-21 17:35:31 +00:00
Nick Craig-Wood d7cd35e2ca qingstor: fix error propagation in CleanUp 
Before this change errors cleaning multiple buckets were passing silently
 2021-01-21 17:35:31 +00:00
Nick Craig-Wood 14c0d8a93e cryptdecode: fix formatting 2021-01-21 10:39:51 +00:00
Nick Craig-Wood 4daf8b7083 Changelog updates from Version v1.53.4 2021-01-20 22:34:04 +00:00
Nick Craig-Wood 0be69018b8 drive: log that emptying the trash can take some time - fixes #4915 2021-01-19 18:09:36 +00:00
Nick Craig-Wood 9b9ab5f3e8 gcs: Fix Entry doesn't belong in directory "" (same as directory) - ignoring 
This change allows directory markers to be non-zero in size.

See: https://forum.rclone.org/t/public-gcs-bucket-and-entry-doesnt-belong-in-directory-same-as-directory/21753/
 2021-01-19 16:50:37 +00:00
Nick Craig-Wood 072464cbdb gcs: fix anonymous client to use rclone's HTTP client 2021-01-19 16:50:37 +00:00
Nick Craig-Wood b0491dec88 Add new email address for buengese to contributors 2021-01-19 16:50:28 +00:00
Nick Craig-Wood ccfefedb47 Add Patrik Nordlén to contributors 2021-01-19 16:48:37 +00:00
Nick Craig-Wood 2fffcf9e7f Add Janne Johansson to contributors 2021-01-19 16:48:37 +00:00
Nick Craig-Wood cef51d58ac jottacloud: fix token refresh failed: is not a regular file error 
Before this change the jottacloud token renewer would run and give the
error:

    Token refresh failed: is not a regular file

This is because the refresh runs on the root and it isn't a file.

This was fixed by ignoring that specific error.

See: https://forum.rclone.org/t/jottacloud-crypt-3-gb-copy-runs-for-a-week-without-completing/21173
 2021-01-12 17:09:44 +00:00
Nick Craig-Wood e0b5a13a13 jottacloud: fix token renewer to fix long uploads 
See: https://forum.rclone.org/t/jottacloud-crypt-3-gb-copy-runs-for-a-week-without-completing/21173
 2021-01-11 16:44:11 +00:00
Nick Craig-Wood de21356154 Add Denis Neuling to contributors 2021-01-11 16:44:11 +00:00
Nick Craig-Wood 3877df4e62 s3: update help for --s3-no-check-bucket #4913 2021-01-10 17:54:19 +00:00
Nick Craig-Wood a7689d7023 rcserver: fix 500 error when marshalling errors from core/command 
Before this change attempting to return an error from core/command
failed with a 500 error and a message about unmarshable types.

This is because it was attempting to marshal the input parameters
which get _response added to them which contains an unmarshalable
field.

This was fixed by using the original parameters in the error response
rather than the one modified during the error handling.

This also adds end to end tests for the streaming facilities as used
in core/command.
 2021-01-10 16:34:46 +00:00
Nick Craig-Wood 847a44e7ad fs/rc: add Copy method to rc.Params 2021-01-10 16:34:46 +00:00
Nick Craig-Wood b3710c962e rc: fix core/command giving 500 internal error - fixes #4914 
Before this change calling core/command gave the error

    error: response object is required expecting *http.ResponseWriter value for key "_response" (was *http.response)

This was because the http.ResponseWriter is an interface not an object.

Removing the `*` fixes the problem.

This also indicates that this bit of code wasn't properly tested.
 2021-01-10 16:34:46 +00:00
Nick Craig-Wood 35ccfe1721 Add kice to contributors 2021-01-10 16:34:46 +00:00
Nick Craig-Wood a97effa27c build: add go1.16beta1 to the build matrix 2021-01-08 12:22:37 +00:00
Nick Craig-Wood 01adee7554 build: raise minimum go version to go1.12 2021-01-08 12:17:09 +00:00
Nick Craig-Wood e775328523 ftp,sftp: Make --tpslimit apply - fixes #4906 2021-01-08 10:29:57 +00:00
Nick Craig-Wood 50344e7792 accounting: factor --tpslimit code into accounting from fshttp 2021-01-08 10:29:57 +00:00
Nick Craig-Wood d58fdb10db onedrive: enhance link creation with expiry, scope, type and password 
This change makes the --expire flag in `rclone link` work.

It also adds the new flags

    --onedrive-link-type
    --onedrive-link-scope
    --onedrive-link-password

See: https://forum.rclone.org/t/create-share-link-within-the-organization-only/21498
 2021-01-08 09:22:50 +00:00
Nick Craig-Wood feaacfd226 hdfs: correct username parameter in integration tests 2021-01-08 09:05:25 +00:00
Nick Craig-Wood e3c238ac95 build: add -buildmode to cross-compile.go 
This builds on

768e4c4735 build: Temporary fix for Windows build errors

But passes the -buildmode flag down to the cross-compile.go command
too.
 2021-01-08 08:58:50 +00:00
Nick Craig-Wood 752997c5e8 Add Yury Stankevich to contributors 2021-01-08 08:58:50 +00:00
Nick Craig-Wood c553ad5158 serve sftp: fix authentication on one connection blocking others - fixes #4882 
Before this change, if one connection was authenticating this would
block any others from authenticating.

This was due to ssh.NewServerConn not being called in a go routine
after the Accept call.

This is fixed by running the ssh authentication in a go routine.

Thanks to @FiloSottile for advice on how to fix this.

See: https://github.com/golang/go/issues/43521
 2021-01-06 15:34:07 +00:00
Nick Craig-Wood dd67a3d5f5 operations: add size if known to skipped items and JSON log - fixes #4624 2021-01-05 19:44:25 +00:00
Nick Craig-Wood e972f2c98a log: make it easier to add parameters to JSON logging 2021-01-05 19:44:25 +00:00
Nick Craig-Wood acbcb1ea9d Add Ilyess Bachiri to contributors 2021-01-05 19:44:25 +00:00
Nick Craig-Wood d4444375ac Add Kerry Su to contributors 2021-01-05 19:44:25 +00:00
Nick Craig-Wood 2af7b61fc3 cmd: fix wording for no retries message 
See: https://forum.rclone.org/t/immutable-should-set-retries-1-when-source-and-dest-are-different/21326/
 2021-01-02 11:20:02 +00:00
Nick Craig-Wood cb97c2b0d3 azureblob: fix crash on startup 
This was introduced by accidental code deletion in

08b9ede217 azureblob: add support for managed identities
 2020-12-31 18:39:09 +00:00
Nick Craig-Wood 35da38e93f operations: fix --immutable error message 
This was accidentally changed in

53a1a0e3ef accounting: add reference to completed transfers
 2020-12-31 18:16:51 +00:00
Nick Craig-Wood 963c0f28b9 sync: Only print "There was nothing to transfer" if no errors 
See: https://forum.rclone.org/t/immutable-should-set-retries-1-when-source-and-dest-are-different/21326
 2020-12-31 18:16:51 +00:00
Nick Craig-Wood b3815dc0c2 sync: fix --immutable errors retrying many times 
See: https://forum.rclone.org/t/immutable-should-set-retries-1-when-source-and-dest-are-different/21326
 2020-12-31 18:16:51 +00:00
Nick Craig-Wood 8053fc4e16 fs: correct default implementation of fs.CountError 2020-12-31 18:16:51 +00:00
Nick Craig-Wood 62c9074132 compress: fix icon in docs 2020-12-30 17:56:08 +00:00
Nick Craig-Wood a854cb9617 webdav: add "Depth: 0" to GET requests to fix bitrix 
See: https://forum.rclone.org/t/bitrix24-de-remote-support/21112/
 2020-12-30 10:14:50 +00:00
Nick Craig-Wood fbf9942fe7 Add Ingo Weiss to contributors 2020-12-30 10:14:50 +00:00
Nick Craig-Wood f425950a52 fs: Always show stats when using --dry-run or --interactive #4624 2020-12-29 21:11:12 +00:00
Nick Craig-Wood ba51409c3c sftp: implement keyboard interactive authentication - fixes #4177 
Some ssh servers are set up with keyboard interactive authentication
which previously the sftp backkend was ignoring.
 2020-12-29 19:48:09 +00:00
Nick Craig-Wood a64fc05385 Add Benjamin Gustin to contributors 2020-12-29 19:48:09 +00:00
Nick Craig-Wood 5601652d65 ncdu: add ! (errror) and . (unreadable) file flags to go with e (empty) 2020-12-28 17:25:46 +00:00
Nick Craig-Wood 65eee674b9 webdav: fix Open Range requests to fix 4shared mount 
Before this change the webdav backend didn't truncate Range requests
to the size of the object. Most webdav providers are OK with this (it
is RFC compliant), but it causes 4shared to return 500 internal error.

Because Range requests are used in mounting, this meant that mounting
didn't work for 4shared.

This change truncates the Range request to the size of the object.

See: https://forum.rclone.org/t/cant-copy-use-files-on-webdav-mount-4shared-that-have-foreign-characters/21334/
 2020-12-28 15:45:40 +00:00
Nick Craig-Wood 72eb74e94a Add Claudio Bantaloukas to contributors 2020-12-28 15:45:40 +00:00
Nick Craig-Wood 6bfec25165 Add Mitsuo Heijo to contributors 2020-12-28 15:45:39 +00:00
Nick Craig-Wood 1c61d51448 Add Brad Ackerman to contributors 2020-12-28 15:45:39 +00:00
Nick Craig-Wood f347a198f7 azureblob: delete archive tier blobs before update if --azureblob-archive-tier-delete 
Before this change, attempting to update an archive tier blob failed
with a 409 error message:

    409 This operation is not permitted on an archived blob.

This change detects if we are overwriting a blob and either generates
the error (if `--azureblob-archive-tier-delete` is not set):

    can't update archive tier blob without --azureblob-archive-tier-delete

Or deletes the blob first before uploading it again (if
`--azureblob-archive-tier-delete` is set).

Fixes #4819
 2020-12-28 12:31:24 +00:00
Nick Craig-Wood 060642ad14 flags: improve error message when reading environment vars #4888 
The message now includes the flag name to help the user work out what
is happening.

    Invalid value for environment variable "RCLONE_VERSION" when setting default
    for --version: strconv.ParseBool: parsing "yes": invalid syntax
 2020-12-28 12:26:23 +00:00
Nick Craig-Wood 629c0d0f65 serve http: fix serving files of unknown length 
Before this change serving files of unknown length were always
returned as 0 length files.

This change serves them correctly, but does not support Range:
requests on them.

See: https://forum.rclone.org/t/serve-http-behavior-when-the-size-is-unknown/21319
 2020-12-27 22:01:41 +00:00
Nick Craig-Wood f7404f52e7 azureblob: fix crash when listing outside a SAS URL's root - fixes #4851 
Before this change if you attempted to list a remote set up with a SAS
URL outside its container then it would crash the Azure SDK.

A check is done to make sure the root is inside the container when
starting the backend which is usually enough, but when two SAS URL
based remotes are mounted in a union, the union backend attempts to
read paths outside the named container. This was causing a mysterious
crash in the Azure SDK.

This fixes the problem by checking to see if the container in the
listing is the one in the SAS URL before listing the directory and
returning directory not found if it isn't.
 2020-12-27 15:55:00 +00:00
Nick Craig-Wood 74a321e156 Add gtorelly to contributors 2020-12-27 15:55:00 +00:00
Nick Craig-Wood fce885c0cd Add Milly to contributors 2020-12-27 15:55:00 +00:00
Nick Craig-Wood 4028a245b0 Add kelv to contributors 2020-12-27 15:55:00 +00:00
Nick Craig-Wood c5b07a6714 Add lostheli to contributors 2020-12-27 15:55:00 +00:00
Nick Craig-Wood b0965bf34f Add Nathan Collins to contributors 2020-12-27 15:54:33 +00:00
Nick Craig-Wood 1eaca9fb45 Add Bob Bagwill to contributors 2020-12-27 15:54:33 +00:00