f-denkena/rclone
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
8,275 Commits 1 Branch 0 Tags
88b0757288c092cebddf2f12761b009ebc09c9da
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick Craig-Wood 2d1c2b1f76 config encryption: set, remove and check to manage config file encryption #7859 2024-09-06 10:34:29 +01:00
Nick Craig-Wood ffb2e2a6de config: use --password-command to set config file password if supplied 
Before this change, rclone ignored the --password-command on the
rclone config setting except when decrypting an existing config file.

This change allows for offloading the password storage/generation into
external hardware key or other protected password storage.

Fixes #7859
 2024-09-06 10:34:29 +01:00
Nick Craig-Wood c9c283533c config: factor --password-command code into its own function #7859 2024-09-06 10:34:29 +01:00
Nick Craig-Wood 71799d7efd Add yuval-cloudinary to contributors 2024-09-06 10:34:29 +01:00
Nick Craig-Wood 8f4fdf6cc8 Add nipil to contributors 2024-09-06 10:34:29 +01:00
Nick Craig-Wood 1a8b7662e7 serve nfs: unify the nfs library logging with rclone's logging better 
Before this we ignored the logging levels and logged everything as
debug. This will obey the rclone logging flags and log at the correct
level.
 2024-09-04 10:50:21 +01:00
Nick Craig-Wood 6ba3e24853 serve nfs: fix incorrect user id and group id exported to NFS #7973 
Before this change all exports were exported as root and the --uid and
--gid flags of the VFS were ignored.

This fixes the issue by exporting the UID and GID correctly which
default to the current user and group unless set explicitly.
 2024-09-04 10:50:21 +01:00
Nick Craig-Wood 802a938bd1 zoho: fix inefficiencies uploading with new API to avoid throttling 
Before this fix, rclone queried the uploaded object to find its size
and modtime after upload as the API did not return these items.

Zoho have subsequently modified the API to return these items so
rclone uses them to avoid an API call.

This should help with rclone being throttled by Zoho.

See: https://forum.rclone.org/t/second-followup-on-the-older-topic-rclone-invokes-more-number-of-workdrive-s-files-listing-api-calls-which-exceeds-the-throttling-limit/45697/20
 2024-09-04 10:45:47 +01:00
Nick Craig-Wood 9deb3e8adf Add crystalstall to contributors 2024-09-04 10:45:12 +01:00
Nick Craig-Wood 6b58cd0870 s3: fix accounting for mulpart transfers after migration to SDKv2 #4989 2024-08-31 20:11:53 +01:00
Nick Craig-Wood 1bb89bc818 docs: add missing sftp providers to README and main docs page - fixes #8038 2024-08-28 07:25:49 +01:00
Nick Craig-Wood a365503750 nfsmount: fix stale handle problem after converting options to new style 
This problem was caused by the defaults not being set for the options
after the conversion to the new config system in

28ba4b832d serve nfs: convert options to new style

This makes the nfs serve options globally available so nfsmount can
use them directly.

Fixes #8029
 2024-08-28 07:08:23 +01:00
Nick Craig-Wood 3bb6d0a42b docs: mark flags.md as auto generated so contributors don't edit it 2024-08-28 07:08:23 +01:00
Nick Craig-Wood f65755b3a3 Add Pawel Palucha to contributors 2024-08-28 07:08:21 +01:00
Nick Craig-Wood 33c5f35935 Add John Oxley to contributors 2024-08-28 07:07:34 +01:00
Nick Craig-Wood 4367b999c9 Add Georg Welzel to contributors 2024-08-28 07:04:02 +01:00
Nick Craig-Wood b57e6213aa Add Péter Bozsó to contributors 2024-08-28 07:04:02 +01:00
Nick Craig-Wood cd90ba4337 Add Sam Harrison to contributors 2024-08-28 07:04:02 +01:00
Nick Craig-Wood 089c168fb9 fstests: attempt to fix flaky serve s3 test 
Sometimes (particularly on macOS amd64) the serve s3 test fails with
TestIntegration/FsMkdir/FsPutError where it wasn't expecting to get an
object but it did.

This is likely caused by a race between the serve s3 goroutine
deleting the half uploaded file and the fstests code looking for it to
not exist.

This fix treats it like any other eventual consistency problem and
retries the check using the test framework.
 2024-08-15 16:30:29 +01:00
Nick Craig-Wood 70e8ad456f serve nfs: implement on disk cache for file handles 2024-08-14 21:55:26 +01:00
Nick Craig-Wood 55b9b3e33a serve nfs: factor caching to its own file 2024-08-14 21:55:26 +01:00
Nick Craig-Wood ce7dfa075c serve nfs: update github.com/willscott/go-nfs to latest 
This fixes various cache invalidation bugs
 2024-08-14 21:55:26 +01:00
Nick Craig-Wood a697d27455 serve nfs: store billy FS in the Handler 2024-08-14 21:55:26 +01:00
Nick Craig-Wood cae22a7562 serve nfs: mask unimplemented error from chmod 2024-08-14 21:55:26 +01:00
Nick Craig-Wood 877321c2fb serve nfs: add tracing to filesystem calls 2024-08-14 21:55:26 +01:00
Nick Craig-Wood 574378e871 serve nfs: rename types and methods which should be internal 2024-08-14 21:55:26 +01:00
Nick Craig-Wood 50d42babd8 nfsmount: require --vfs-cache-mode writes or above in tests 
These tests fail for --vfs-cache-mode minimal on Linux for the same
reason they don't work properly with --vfs-cache-mode off
 2024-08-14 21:55:26 +01:00
Nick Craig-Wood 13ea77dd71 nfsmount: allow tests to run on any unix where sudo mount/umount works 2024-08-14 21:55:26 +01:00
Nick Craig-Wood 62b76b631c nfsmount: make the --sudo flag work for umount as well as mount 2024-08-14 21:55:26 +01:00
Nick Craig-Wood 96f92b7364 nfsmount: add tcp option to NFS mount options to fix mounting under Linux 2024-08-14 21:55:26 +01:00
Nick Craig-Wood 7c02a63884 build: install NFS client libraries to allow nfsmount tests to run 2024-08-14 21:55:26 +01:00
Nick Craig-Wood 67d4394a37 vfstest: fix crash if open failed 2024-08-14 21:55:26 +01:00
Nick Craig-Wood c1a98768bc Implement Gofile backend - fixes #4632 2024-08-14 21:15:37 +01:00
Nick Craig-Wood bac9abebfb lib/encoder: add Exclamation mark encoding 2024-08-14 21:15:37 +01:00
Nick Craig-Wood 27b281ef69 chunkedreader: add --vfs-read-chunk-streams to parallel read chunks 
This converts the ChunkedReader into an interface and provides two
implementations one sequential and one parallel.

This can be used to improve the performance of the VFS on high
bandwidth or high latency links.

Fixes #4760
 2024-08-14 21:13:09 +01:00
Nick Craig-Wood 10270a4354 accounting: fix race detected by the race detector 2024-08-14 21:13:09 +01:00
Nick Craig-Wood d08b49d723 pool: Add ability to wait for a write to RW 2024-08-14 21:13:09 +01:00
Nick Craig-Wood cb2d2d72a0 pool: Make RW thread safe so can read and write at the same time 2024-08-14 21:13:09 +01:00
Nick Craig-Wood e686e34f89 multipart: make pool buffer size public 2024-08-14 21:13:09 +01:00
Nick Craig-Wood 5f66350331 Add Fornax to contributors 2024-08-14 21:12:56 +01:00
Nick Craig-Wood e1d935b854 build: use go1.23 for the linter 
This reverts commit 485aa90d13.

As the upstream problem is now fixed by golangci-lint v1.60.1
 2024-08-14 18:27:13 +01:00
Nick Craig-Wood 61b27cda80 build: fix govet lint errors with golangci-lint v1.60.1 
There were a lot of instances of this lint error

    printf: non-constant format string in call to github.com/rclone/rclone/fs.Logf (govet)

Which were fixed by re-arranging the arguments and adding "%s".

There were quite a few genuine bugs which were found too.
 2024-08-14 18:25:40 +01:00
Nick Craig-Wood 83613634f9 build: bisync: fix govet lint errors with golangci-lint v1.60.1 
There were a lot of instances of this lint error

    printf: non-constant format string in call to github.com/rclone/rclone/fs.Logf (govet)

Most of these could not easily be fixed so had nolint lines added.

This should probably be done in a neater way perhaps by making
LogColorf/ErrorColorf functions.
 2024-08-14 18:21:31 +01:00
Nick Craig-Wood 1c80cbd13a build: fix staticcheck lint errors with golangci-lint v1.60.1 2024-08-14 17:48:24 +01:00
Nick Craig-Wood 9d5315a944 build: fix gosimple lint errors with golangci-lint v1.60.1 2024-08-14 17:46:12 +01:00
Nick Craig-Wood 8d1d096c11 drive: fix copying Google Docs to a backend which only supports SHA1 
When copying Google Docs to Backblaze B2 errors like this would happen

    ERROR : test.docx: Failed to calculate src hash: hash type not supported
    ERROR : test.docx: corrupted on transfer: sha1 hashes differ src

This was due to an oversight in

8fd66daab6 drive: add support of SHA-1 and SHA-256 checksum

Which omitted to change the base object (which includes Google Docs) so
that it supported SHA-1 and SHA-256.
 2024-08-12 20:27:12 +01:00
Nick Craig-Wood 4b922d86d7 drive: update docs on creating admin service accounts 2024-08-12 20:27:12 +01:00
Nick Craig-Wood 642d4082ac test_backend_sizes.py calculates space in the binary each backend uses #7998 2024-08-09 12:13:24 +01:00
Nick Craig-Wood c8555d1b16 serve s3: update to AWS SDKv2 by updating github.com/rclone/gofakes3 
This is the last dependency for the SDKv1 and this commit removes it
from go.mod also.
 2024-08-07 16:35:39 +01:00
Nick Craig-Wood 3ec0ff5d8f s3: fix SSE-C after SDKv2 change 
The new SDK apparently keeds the customer key to be base64 encoded
where the old one did that for you automatically.

See: https://github.com/aws/aws-sdk-go-v2/issues/2736
See: https://forum.rclone.org/t/new-s3-backend-help-testing-needed/47139/3
 2024-08-07 12:13:13 +01:00
Nick Craig-Wood 8aef1de695 s3: fix Cloudflare R2 integration tests after SDKv2 update #4989 
Cloudflare will normally automatically decompress files with
`Content-Encoding: gzip` when downloaded. This is not what AWS S3 does
and it breaks the integration tests.

This fudges the integration tests to upload the test file with
`Cache-Control: no-transform` on Cloudflare R2 and puts a note in the
docs about this problem.
 2024-08-07 12:13:13 +01:00
Nick Craig-Wood cb611b8330 s3: add --s3-sdk-log-mode to control SDK debugging 2024-08-07 12:13:13 +01:00
Nick Craig-Wood 66ae050a8b s3: fix GCS provider after SDKv2 update #4989 
This also adds GCS via S3 to the integration tester.
 2024-08-07 12:13:13 +01:00
Nick Craig-Wood fd9049c83d s3: update to using AWS SDK v2 - fixes #4989 
SDK v2 conversion

Changes

  - `--s3-sts-endpoint` is no longer supported
  - `--s3-use-unsigned-payload` to control use of trailer checksums (needed for non AWS)
 2024-08-07 12:13:13 +01:00
Nick Craig-Wood a1f52bcf50 fstest: implement method to skip ChunkedCopy tests 2024-08-06 12:45:07 +01:00
Nick Craig-Wood 0470450583 build: disable wasm/js build due to go bug 
Rclone is too big for js/wasm until
https://github.com/golang/go/issues/64856 is fixed
 2024-08-04 12:18:34 +01:00
Nick Craig-Wood 1901bae4eb Add @dmcardle as gitannex maintainer 2024-08-01 17:48:39 +01:00
Nick Craig-Wood 9866d1c636 docs: s3: add section on using too much memory #7974 2024-08-01 16:33:09 +01:00
Nick Craig-Wood c5c7bcdd45 docs: link the workaround for big directory syncs in the FAQ #7974 2024-08-01 16:33:09 +01:00
Nick Craig-Wood d5c7b55ba5 Add David Seifert to contributors 2024-08-01 16:33:09 +01:00
Nick Craig-Wood feafbfca52 Add Will Miles to contributors 2024-08-01 16:33:09 +01:00
Nick Craig-Wood abe01179ae Add Ernie Hershey to contributors 2024-08-01 16:33:09 +01:00
Nick Craig-Wood ba381f8721 b2: update versions documentation - fixes #7878 2024-07-24 10:52:05 +01:00
Nick Craig-Wood 8f0ddcca4e s3: document need to set force_path_style for buckets with invalid DNS names 
Fixes #6110
 2024-07-23 11:34:08 +01:00
Nick Craig-Wood 404ef80025 ncdu: document that excludes are not shown - fixes #6087 2024-07-23 11:29:07 +01:00
Nick Craig-Wood 13fa583368 sftp: clarify the docs for key_pem - fixes #7921 2024-07-23 10:07:44 +01:00
Nick Craig-Wood e111ffba9e serve ftp: fix failed startup due to config changes 
See: https://forum.rclone.org/t/failed-to-ftp-failed-to-parse-host-port/46959
 2024-07-22 14:54:32 +01:00
Nick Craig-Wood 30ba7542ff docs: add Route4Me as a sponsor 2024-07-22 14:48:41 +01:00
Nick Craig-Wood b3edc9d360 fs: fix --use-json-log and -vv after config reorganization 2024-07-20 12:49:08 +01:00
Nick Craig-Wood 04f35fc3ac Add Tobias Markus to contributors 2024-07-20 12:49:08 +01:00
Nick Craig-Wood b809e71d6f lib/mmap: fix lint error on deprecated reflect.SliceHeader 
reflect.SliceHeader is deprecated, however the replacement gives a go
vet warning so this disables the lint warning in one use of
reflect.SliceHeader and replaces it in the other.
 2024-07-20 10:54:47 +01:00
Nick Craig-Wood d149d1ec3e lib/http: fix tests after go1.23 update 
go1.22 output the Content-Length on a bad Range request on a file but
go1.23 doesn't - adapt the tests accordingly.
 2024-07-20 10:54:47 +01:00
Nick Craig-Wood 3b51ad24b2 rc: fix tests after go1.23 upgrade 
go1.23 adds a doctype to the HTML output when serving file listings.
This adapts the tests for that.
 2024-07-20 10:54:47 +01:00
Nick Craig-Wood 485aa90d13 build: use go1.22 for the linter to fix excess memory usage 
golangci-lint seems to have a bug which uses excess memory under go1.23

See: https://github.com/golangci/golangci-lint/issues/4874
 2024-07-20 10:54:47 +01:00
Nick Craig-Wood 8958d06456 build: update all dependencies 2024-07-20 10:54:47 +01:00
Nick Craig-Wood ca24447090 build: update to go1.23rc1 and make go1.21 the minimum required version 2024-07-20 10:54:47 +01:00
Nick Craig-Wood d008381e59 Add AThePeanut4 to contributors 2024-07-20 10:54:47 +01:00
Nick Craig-Wood 4824837eed azureblob: allow anonymous access for public resources 
See: https://forum.rclone.org/t/azure-blob-public-resources/46882
 2024-07-18 11:13:29 +01:00
Nick Craig-Wood 5287a9b5fa Add Ke Wang to contributors 2024-07-18 11:13:29 +01:00
Nick Craig-Wood f2ce1767f0 Add itsHenry to contributors 2024-07-18 11:13:29 +01:00
Nick Craig-Wood 7f048ac901 Add Tomasz Melcer to contributors 2024-07-18 11:13:29 +01:00
Nick Craig-Wood b0d0e0b267 Add Paul Collins to contributors 2024-07-18 11:13:29 +01:00
Nick Craig-Wood f5eef420a4 Add Russ Bubley to contributors 2024-07-18 11:13:29 +01:00
Nick Craig-Wood afd2663057 rc: add option blocks parameter to options/get and options/info 2024-07-16 15:02:50 +01:00
Nick Craig-Wood 5ddedae431 fstest: fix compile after merge 
After merging this commit

56caab2033 b2: Include custom upload headers in large file info

The compile failed as a change had been missed. Should have rebased
before merging!
 2024-07-15 12:18:14 +01:00
Nick Craig-Wood d9bd6f35f2 fs/test: fix erratic test 2024-07-15 11:09:54 +01:00
Nick Craig-Wood 532a0818f7 fs: make sure we load the options defaults to start with 2024-07-15 11:09:54 +01:00
Nick Craig-Wood 91558ce6aa fs: fix the defaults overriding the actual config 
After re-organising the config it became apparent that there was a bug
in the config system which hadn't manifested until now.

This was the default config overriding the main config and was fixed
by noting when the defaults had actually changed.
 2024-07-15 11:09:54 +01:00
Nick Craig-Wood 8fbb259091 rc: add options/info call to enumerate options 
This also makes some fields in the Options block optional - these are
documented in rc.md
 2024-07-15 11:09:54 +01:00
Nick Craig-Wood 4d2bc190cc fs: convert main options to new config system 
There are some flags which haven't been converted which could be
converted in the future.
 2024-07-15 11:09:54 +01:00
Nick Craig-Wood c2bf300dd8 accounting: fix creating of global stats ignoring the config 
Before this change the global stats were created before the global
config which meant they ignored the global config completely.
 2024-07-15 11:09:54 +01:00
Nick Craig-Wood c954c397d9 filter: convert options to new style 2024-07-15 11:09:54 +01:00
Nick Craig-Wood 25c6379688 filter: rename Opt to Options for consistency 2024-07-15 11:09:54 +01:00
Nick Craig-Wood ce1859cd82 rc: convert options to new style 2024-07-15 11:09:54 +01:00
Nick Craig-Wood cf25ae69ad lib/http: convert options to new style 
There are still users of the old style options which haven't been
converted yet.
 2024-07-15 11:09:54 +01:00
Nick Craig-Wood dce8317042 log: convert options to new style 2024-07-15 11:09:54 +01:00
Nick Craig-Wood eff2497633 serve sftp: convert options to new style 2024-07-15 11:09:54 +01:00
Nick Craig-Wood 28ba4b832d serve nfs: convert options to new style 2024-07-15 11:09:54 +01:00
Nick Craig-Wood 58da1a165c serve ftp: convert options to new style 2024-07-15 11:09:54 +01:00
Nick Craig-Wood eec95a164d serve dlna: convert options to new style 2024-07-15 11:09:54 +01:00