The cmd/gitannex end to end tests routinely take 230-530s and on slow
CI runners the package exceeds the go test default timeout of 10m.
Raise the per-package timeout to 20m in quicktest and racequicktest to
give slow runners headroom.
Previously cross-compile.go ran NumCPU builds in parallel, each of
which ran an unrestricted go build using -p NumCPU internally, giving
up to NumCPU^2 concurrent compile processes and enormous load averages.
Pass -p to each go build, sized so the total parallelism is about
NumCPU, sharing the CPUs between however many builds are actually
selected. This can be overridden with the new -build-p flag.
The local backend built every OS path by joining the root with the source
name converted through the configured encoding, so the encoding was the only
thing keeping a name from turning into path syntax.
With an encoding which omits Dot (Slash, None, Raw) rclone's standard "．．"
decodes back to a real "..", and with an encoding which omits BackSlash a name
like "..\file" becomes a native path on Windows. filepath.Join then resolved
those out of the destination the user chose, so a source object called
"../marker.txt" - an s3 key of "tenant/../marker.txt" listed with the remote
rooted at "tenant", say - created or overwrote a file outside it.
localPath now joins the name to the root and checks with filepath.Rel that the
result is still inside it. localPath is the only place the root is joined to a
name, so threading the error through newObject and newDirectory covers every
operation.
Default configurations were not affected, as encoder.OS includes Dot on all
platforms and BackSlash on Windows.
Fixes GHSA-7p4m-qxvv-g567
A request path beginning with "../" escaped the path the server was
started on, letting a client list, read, create, overwrite and delete
objects outside it.
The check added for CVE-2026-59733 rejected non-canonical paths by
comparing them with path.Clean, but path.Clean cannot resolve leading
".." elements in a relative path so it leaves them in place and the
comparison comes out equal. Only interior traversal such as "a/../../x"
was rejected. Whether a path then escaped depended on the backend:
those which join the root with the remote before encoding it - webdav,
ftp, sftp, http and memory - resolved the ".." away, while local and s3
encode the dot elements first and were unaffected.
A bare "." was accepted for the same reason, which on bucket backends
addresses the served directory's own key.
Validate with io/fs.ValidPath instead, which rejects ".", ".." and empty
elements wherever they appear. The empty path stays valid as the root of
the API, and "." is excluded explicitly because ValidPath accepts it as
the root of an FS.
The VFS calls backends from goroutines of its own. A panic on any of these
cannot be recovered. So a backend panicking on a single file killed the whole
process, taking down a mount or every user of a serve command, even for servers
such as serve http whose library recovers panics raised on its own request
goroutines.
Recover panics at those goroutines and log them with a stack trace. Where the
surrounding code already handles a failure, recover around the backend call
itself rather than the whole goroutine, so a panicking upload is retried like
any other failed upload and a panicking download is reported to the waiters,
instead of abandoning the work part way through and leaving the bookkeeping
inconsistent.
Addresses GHSA-6jcg-q3wp-x2f4
Three ways a single client could deny service to everyone else connected to
the same serve sftp process:
A session "subsystem" request payload is a length-prefixed string, but it was
sliced at a fixed offset without checking its length, so a client sending a
truncated payload panicked the out-of-band request goroutine and killed the
process. Decode the payload instead, the way the neighbouring "exec" request
already does.
Rejecting a request then left the goroutine handling that channel waiting
forever to learn what kind of channel it was, because nothing was ever sent
on the channel it waits on. A client could open channels in a loop making
unsupported requests and grow the server's goroutines and memory without
bound. Signal the waiter when the requests run out so the channel is torn
down.
Separately, the request handlers - and reads, writes and closes on the file
handles they return - run on pkg/sftp packet worker goroutines which have no
panic recovery. A panic raised by a backend while serving one request took the
process down with it. Recover panics at that boundary, log them with a stack
trace, and return them to the requesting client as an error instead.
Addresses GHSA-6jcg-q3wp-x2f4.
The archive backend passes remote .sqfs bytes straight to the go-diskfs squashfs
parser, which does not validate several attacker-controlled superblock and
metadata fields. A crafted image can make the parser panic.
Recover panics at the go-diskfs boundary and return an "invalid or corrupt
squashfs image" error instead. As well as the parse entry points (Read, ReadDir,
OpenFile) this wraps the reader returned by Open, since the parser reads file
data lazily and can panic long after the image opened successfully.
Addresses GHSA-6jcg-q3wp-x2f4.
The builtin authentication compared the configured username and password
with ==, whose run time depends on how much of the value matches, giving a
timing side-channel that could in principle help guess the password.
serve sftp, serve s3 and the auth proxy already use subtle.ConstantTimeCompare
so bring serve ftp in line with them. An empty configured password
still accepts any password.
Addresses GHSA-mfvx-7rcj-9m5g finding 4.
When --cert and --key were supplied TLS was only applied to the listener if
exactly one --addr was given. With two or more --addr flags every listener
without an explicit tls:// prefix silently served cleartext HTTP, so adding a
second --addr to an HTTPS server quietly disabled TLS on both.
Now when TLS is configured every listener serves TLS. An individual listener
can be prefixed with http:// to serve unencrypted HTTP on that address, and
tls:// still marks a listener as TLS explicitly. Using a tls:// address
without --cert and --key is now an error instead of silently serving
cleartext with an https:// URL.
Addresses GHSA-mfvx-7rcj-9m5g finding 3.
With --rc-serve set the root listing enumerated the names of all configured
remotes without any authentication.
Make the root listing obey the same fail-closed rule as the rest of
the rc endpoints: it now requires authentication to be configured or
an explicit opt out with --rc-no-auth.
Addresses GHSA-mfvx-7rcj-9m5g finding 2.
The pprof debug handlers were accessible without authentication disclosing the
process command line (which can carry backend credentials passed on the command
line) and runtime profiles.
Mount the pprof handlers only when when auth is configured or --rc-no-auth was
passed - so they obey the same rule as the rc endpoints.
Addresses GHSA-mfvx-7rcj-9m5g finding 1.
Before this change rclone read a proxy response with http.ReadResponse
over an unrestricted buffered reader. A malicious or compromised
configured proxy, or an active on-path actor controlling a plaintext
HTTP-proxy hop, can grow memory until the process fails.
This fixes the problem by restrincting the read to 1MB maximum.
Before this change, rclone sent stack traces to the client on panic
capture in the rc. Stack traces can leak information which could be
useful to an attacker.
The FTP control channel is line oriented and the ftp library writes
command arguments (paths) straight onto it without escaping, so a
filename containing CR/LF can inject an independent FTP command.
This fix makes sure CR/LF are therefore always encoded to safe symbols
regardless of the configured encoding, which is what the default
encoding already does.
An audit of the shell escaping alongside the PowerShell fix found the
Windows Command Prompt path only rejected the double quote delimiter. A
remote filename containing % or ! can trigger environment variable expansion
even inside double quotes, and a newline or carriage return ends the
command, so a crafted filename could alter the command run during
server-side hashing. Reject paths containing these characters, matching
the existing treatment of the double quote.
PowerShell treats several Unicode smart-quote characters (U+2018, U+2019,
U+201A, U+201B) as single-quote delimiters in addition to the ASCII
apostrophe. The quoting helper only doubled the ASCII apostrophe, so a
remote filename containing one of these could close the quoted path and
inject statements that ran as the SSH account during server-side hashing.
Double all five delimiters when wrapping a PowerShell path so a filename
is always treated as data.
Fixes GHSA-2m8m-jhrm-w6j2
The AWS SDK signs S3 Express (directory bucket) requests with a session
token in the x-amz-s3session-token header. Go's HTTP client treats it as
an ordinary custom header and copies it when following a redirect to
another host, and it was missing from the list of secret headers the
redirect policy strips. Add it to the list.
The redirect tests derived their inputs from the production header list,
so a header accidentally dropped from that list would silently lose test
coverage rather than fail. The test list is now a deliberately literal
copy, kept in sync with the production list by a new test, so removing a
header from either list is a test failure. There is also a new
regression test verifying the Referer header that net/http generates
automatically - which for a presigned request carries the signed query
string - is not forwarded across hosts.
See GHSA-8mxv-9xhp-86h4
The CheckRedirect policy stripped only the X-Amz-Security-Token header when a
redirect crossed a host, leaving other origin-bound secrets exposed:
- Go removes Authorization on a hostname change but not on a same-host scheme
downgrade, so an IBM IAM bearer token was re-sent over plaintext HTTP.
- The SSE-C and copy-source SSE-C customer-key headers (which Go has no reason
to treat as sensitive) were forwarded across a redirect to another host.
- On a cross-host redirect net/http copies the previous request URL into the
Referer header; for a presigned request that URL carries the signature and
session token in its query string, disclosing them to the new origin.
This now refuses outright to follow any HTTPS->HTTP redirect - an S3 endpoint
has no legitimate reason to downgrade the transport, and this closes the
plaintext-exposure class regardless of which header or query string carries the
secret. On a cross-host redirect also strip the known credential headers and the
Referer.
The IDrive e2 region-lookup call posts the access key ID to a fixed HTTPS
endpoint but used a bare http.Client that would follow a redirect downgrading
to plaintext HTTP. Apply the same CheckRedirect policy as the main S3 client so
the request can't be replayed over an unencrypted connection.
Fixes GHSA-8mxv-9xhp-86h4
A server that redirects an HTTPS request to a plaintext HTTP URL on the
same host would cause Go's http.Client to replay the configured
credentials (Basic Authorization, cookies, secret headers) over the
network in cleartext.
Refuse to follow such downgrade redirects by default in lib/rest and wire
the webdav backend's client to use it. The `auth_redirect` option remains
the opt-in escape hatch for servers that legitimately need auth preserved
across redirects.
Fixes GHSA-h4mf-4v27-hggj
When making a release from master, the range since the last minor
release includes all the commits which were cherry-picked to the
previous stable branch and already published in its point releases,
so their changelog lines had to be deleted by hand.
make_changelog.py now finds the previous stable branch (eg
v1.74-stable) from the version being released and skips any commits
released there, detected via cherry-pick -x trailers and git cherry
patch equivalence. Skipped commits are listed on stderr for review.
Releases made from a stable branch are unaffected.
In this commit we attempted to wait for the success report of an
upload to fix the 500 error:
fe78b559d1 yandex: fix 500 errors by waiting for uploads to complete before setting modtime
However Yandex Disk finalizes an upload asynchronously on its servers.
Waiting for the upload operation to report success is not enough -
under load the server reports the operation as successful slightly
before the file is fully finalized, so setting the modification time
straight after an upload can still fail with 500 Internal Server
Error.
Yandex support recommend waiting 1.5s - 3s after the upload before
modifying the file's metadata, so add an --yandex-upload-wait option
(default off) to insert a delay between the upload completing and the
modification time being set.
Runner image ubuntu24/20260726.254 updated podman from 4.9.3 to 5.8.4
which is installed from the podman-static bundle. This bundle ships a
non-setuid fusermount3 in /usr/local/bin which shadows the setuid
/usr/bin/fusermount3 from the fuse3 package (as /usr/local/bin comes
first in PATH). A non-setuid fusermount3 cannot mount as an
unprivileged user, so the cmd/mount and cmd/mount2 tests failed with:
fusermount3: mount failed: Operation not permitted
Fix by removing the podman bundled fusermount binaries so the distro
provided setuid one is used.
The session close flag was only computed after each append, so a
known-size upload which fits in a single chunk sent all its data and
then issued a zero-payload append purely to close the session - one
wasted round trip per small file on the default batched upload path.
Set the close flag before the first append when the size is known to
fit in one chunk.
Since batch mode became the default all uploads go through
uploadChunked, which allocated a full chunk-size retry buffer (48 MiB
by default) regardless of the file size. With the `--transfers 32`
recommended for small file uploads that is ~1.5 GiB of buffer to
upload tiny files.
Size the buffer to the file size when it is known and smaller than a
chunk.
Before this change, if an object compressed with "Content-Encoding:
gzip" was downloaded, a length and hash mismatch would occur since the
go runtime automatically decompressed the object on download, giving
errors like
corrupted on transfer: sizes differ
This change sets "Accept-Encoding: gzip" on all requests which stops
the go runtime decompressing objects on download, so compressed
objects are downloaded as-is with intact size and hash information.
If --oos-decompress is set then rclone will decompress objects with
"Content-Encoding: gzip" as they are received, at the cost of not
being able to check the length or the hash of the downloaded object.
Fixes#9694
DirMove returned fs.ErrorDirExists for any error from the destination
existence check not just when the destination actually exists. That
made sync silently fall back to file-by-file moves and masked the real
failure.
Return the underlying error instead when the check fails for any other
reason.
DirMove checked whether the destination exists using the raw path but
performed the rename with the encoded path, so for directory names
needing encoding (trailing spaces or periods, characters like
\* ? : < > | " or a literal backslash) the existence check looked at
a different server path than the rename used.
If revealing the password, creating the Kerberos client or the SMB
handshake failed after the TCP connection was established, the
connection was never closed.
A new KerberosFactory was constructed for every dial, so the client,
error and ccache modification time caches it holds were discarded
after a single use. Every new SMB connection re-read the Kerberos
config, re-parsed the ccache and did a fresh KDC exchange.
Share a single factory so clients are cached across connections as
intended, and refreshed when the ccache file changes.
Object.Open dereferenced the response's ContentLength pointer without checking
it. The OCI SDK leaves ContentLength nil when the server replies without a
Content-Length header or ContentRange which caused a nil pointer panic.
Now the size is only updated when the response actually provides one, leaving
the size from the object metadata in place otherwise.
This also fixes the same potential problem in the newObject code.
The multipart upload complete endpoint returns 200 with an empty body,
but rclone tried to decode that body as JSON, failed with EOF and
retried until the retries ran out, so every upload failed even though
the server had actually completed it.
Fixed by not attempting to decode the response body.
This was a regression introduced in
a4972de505 shade: retry server errors instead of failing the transfer
which started treating the JSON decode error as fatal where previously
it was accidentally ignored.
Add a new "list_parallelism" advanced option (requires use_arrow_list) which
splits a directory's blob-name keyspace into ranges and lists them
concurrently using the Arrow startFrom/endBefore range parameters. Both
recursive (ListR) and single-directory listings benefit.
The keyspace is sharded with a single-case (digit+lowercase) character
ladder. Single case is deliberate: the service validates startFrom /
endBefore case-insensitively (while listing case-sensitively), so a
mixed-case boundary pair is rejected with a 400; its endBefore filtering is
byte-ordered, so the listing is exact.
The service only honours endBefore on the Arrow listing path of accounts with
Blob Listing with Apache Arrow enabled: elsewhere it either rejects the
request with 400 OperationNotSupportedWithFeatureMissing or, if it fell back
to XML, would ignore the shards' upper bounds. Both cases are detected and
the parallel listing falls back to a sequential listing.
Listing 100,000 objects from a well-connected server:
config time vs XML
xml 23.9s 1.0x
use_arrow_list 6.9s 3.5x
list_parallelism=2 4.4s 5.5x
list_parallelism=4 2.8s 8.4x
list_parallelism=6 2.2s 10.9x
list_parallelism=10 1.5s 15.4x
list_parallelism=20 1.4s 17.6x
list_parallelism=30 1.1s 21.7x
So Apache Arrow listing alone is ~3.5x faster than XML, and parallel listing
takes that to ~20x. The ladder caps at 37 shards, so the benefit plateaus
beyond list_parallelism=30.
The option is hidden from the configurator, command line help and docs
until the feature is officially released.
Add a new "use_arrow_list" advanced option which fetches directory listings
using the ListBlobs Apache Arrow response format instead of XML. This can be
substantially faster for very large containers.
This is EXPERIMENTAL and requires the "Blob Listing with Apache Arrow" preview
feature to be enabled on the storage account. It is not supported on accounts
with a hierarchical namespace (ADLS Gen2). If the feature is not enabled the
service returns XML and the listing transparently falls back to it. Connection
string auth is not supported - it also falls back to the normal XML listing.
The option is hidden from the configurator, command line help and docs until the
feature is officially released.
This adds a temporary subpackage implementing the experimental "Blob
Listing with Apache Arrow" feature on top of the released Azure azblob
SDK.
The Azure SDK for Go supports Arrow listing on its unreleased
feature/storage/bifrost branch (commit c6fa341ca22b) but that can't be
used in a release. This package exposes the same options and pager
interface as the experimental SDK, built on the released SDK's public
API plus verbatim copies of its Arrow decoder and of the two internal
auth policies (shared key signing and storage bearer challenge) which
the released SDK does not export. Copied files carry MIT attribution
headers pinned to their upstream source.
When Arrow listing ships in a released azblob the whole package will be
deleted and callers pointed back at the SDK - the interface is
deliberately source compatible to make that a mechanical change.
Also exports auth.Transporter so the new package's pipeline can use
rclone's fshttp transport, and adds github.com/apache/arrow-go/v18 as a
direct dependency for decoding the Arrow IPC stream.
Backend options with Hide: fs.OptionHideBoth are hidden from the config
wizard and from the command line flag listing, but "rclone help backend"
(and therefore the autogenerated backend documentation) still showed
them. Skip them there too so fully hidden options no longer appear in
the docs.
This commit introduced aux.go which unfortunately is illegal on windows.
e006d7c13f vfs: fix crash when multiple mounts or servers share the same VFS
Rename aux_test.go too to keep the pair together.
Add two new options, pin_host_key and host_keys, that
together provide a TOFU host-key validation mode for users who don't
maintain a known_hosts file. When --sftp-pin-host-key is used, rclone
records the server's host key into host_keys on the first successful
connection and verifies it on every subsequent connection.
host_keys is always validated when non-empty, so it can also be used
by hand to pin a known fingerprint without enabling TOFU writing.
known_hosts_file takes precedence if both are set. SSH host
certificates are rejected with a clear message pointing at
known_hosts_file. On-the-fly remotes log a warning since the captured
key cannot be persisted.
The connection pacer in getSftpConnection used to retry every error,
so permanent failures (host key mismatch, certificate rejection, auth
failure, etc.) were looped 10 times before reporting to the user.
Switch to using fserrors.ShouldRetry which matches the pattern other
backends use so only genuinely retriable errors (timeouts, EOF,
network blips) are retried and permanent errors are surfaced
immediately.
The VFS is shared between users with the same remote and options, for
example two mounts created over the rc, or a mount and an NFS server.
Each node has a single Sys() slot which mount, mount2 and serve nfs
all used to attach their per-node data. With a shared VFS the users
overwrote each other's data: at best churning the cached FUSE nodes,
and since the slot was an atomic.Value, panicking with "store of
inconsistently typed value" as soon as two users stored different
types on the same node.
This change gives each node auxiliary values keyed by owner, set with
SetAux and read with Aux, so each user of the VFS has an independent
slot. The mounts now cache their FUSE nodes under their own key,
leaving Sys - which is read through the os.FileInfo interface -
reserved for users like serve nfs which need to control what that
returns.
Nodes with nothing attached use less memory than before (one pointer
instead of an atomic.Value) and reads remain lock free.
Bug discovered while thinking about #9617
Streamed multipart uploads had two problems when the client uploaded parts
concurrently:
Parts arriving ahead of the next part needed by the backend stream were buffered
in memory without limit, acknowledging each part as soon as it was received. A
client uploading faster than the backend could drain would therefore balloon the
server's memory to the size of the upload. Buffering is now bounded a new
--multipart-streaming-buffer-limit flag (default 256Mi, 0 for unlimited): a part
that would take the buffer over the limit is not read until the stream drains,
applying backpressure to the client instead of using unbounded memory.
A part uploaded again with the same number - typically a client retrying after
its request timed out - left a stale copy in the reorder buffer which made
CompleteMultipartUpload fail with InvalidPart, aborting the whole upload.
Re-uploaded parts are now handled properly: a copy still in the buffer is
replaced, an identical copy of an already streamed part is accepted as a no-op,
and only replacing an already streamed part with different content (which the
in-order stream cannot honour) is rejected.
Every write operation waits for the server's processing tasks which
currently take minutes each, so the full test suite takes much longer
than the timeout. Running with -short skips the FsEncoding subtests
(which are already on the ignore list) and the chunked upload tests,
removing a large number of these waits.
Multiple security vulnerabilities have been identified and addressed
in grpc-go affecting the xDS RBAC authorization engine
(internal/xds/rbac) and the HTTP/2 transport server implementation
(internal/transport).
This updates to master to add the fix which allows it to compile on
plan9.
CloudKit is inconsistent about how it encodes the isDeleted field on
album records, returning a JSON boolean (true/false) for some accounts
and a number (0/1) for others. The numeric form caused listing of a
photo library to fail with:
json: cannot unmarshal number into Go struct field
ckBoolField.records.fields.isDeleted.value of type bool
The encoding also varies over time, not just per account: a full HTTP
dump from the reporting user showed the server sending
"isDeleted" : { "value" : 0, "type" : "INT64" }
but the same account later reverted to the boolean encoding with no
client change. Asset records already deliver isDeleted as a number, so
both encodings are in active use server side and either may appear.
Accept both encodings when parsing CloudKit boolean fields.
See: https://forum.rclone.org/t/error-when-trying-to-list-contents-of-primarysync-directory-in-icloud-photos/54028
The old rclone-integration-test item has accumulated years of catalog
task churn and IA now deprioritises and periodically holds its tasks,
which makes the tests wait on the server's processing tasks for a
very long time. Start again with a fresh item (created with noindex
set so the test files stay out of IA's search index).
The session cache was only populated on a fresh username and password
login, so once a session ID was stored in the config every Fs
instance created its own Mega session with its own copy of the
account's node tree. The server side move code relies on all Fs
instances of a user sharing one session, and with separate sessions a
move between two rclone remotes grafted a node from one tree into
another, where the asynchronous event replays of the two sessions
raced and could detach the moved file from the destination directory
so it disappeared from listings.
Cache the session however the login was done. This also stops every
extra Fs instance re-downloading the whole account node tree.
Sync refuses to delete files when the global error stats are non-zero
so a single backend error in one transform test made every following
transform test in the same test binary fail with "not deleting files
as there were IO errors". Reset the stats at the start of each test.
The pacer callbacks only retried 429 responses, so transient server
errors like 502 Bad Gateway failed transfers immediately. Retry the
standard retryable status codes and transport errors everywhere, make
chunk upload retries resend the whole chunk, and fix a potential nil
pointer dereference when logging a bad token response.
The go-diskfs library now requires io/fs.ValidPath style paths (no
leading slash, "." for the root) so convert paths at the library
boundary. This fixes listing failures and the resulting test hangs
after the update to go-diskfs v1.9.3.
This fixes the code to compile with the updated dependencies:
- squashfs: implement the new Path method required by the go-diskfs
backend.Storage interface, returning an empty string as there is no
underlying path.
- dropbox: convert to and from the SDK's new DBXTime type (an alias
for time.Time) for ClientModified, TimeInvited and Expires.
- dropbox: remove a stray debug fmt.Printf from the shared folder
listing.
- pin go-systemd to v22.6.0 to fix netbsd builds
go-systemd v22.7.0 uses unix.ClockGettime and unix.CLOCK_MONOTONIC
which golang.org/x/sys does not define for netbsd, so the build fails.
See: https://github.com/coreos/go-systemd/issues/512
- pin google.golang.org/grpc to v1.80.0 to fix plan9 builds
grpc v1.81.0 and later use syscall.Errno, syscall.ECONNRESET and
syscall.ECONNABORTED which do not exist on plan9, so the build fails
See: https://github.com/grpc/grpc-go/issues/9253
The /file-entries/duplicate endpoint returns a 500 Server Error for
every request (reported to Drime 2026-06) which made all server side
copies fail after 10 retries. Remove the Copy method so rclone falls
back to downloading and re-uploading instead. It can be restored if
Drime fix the endpoint.
This updates go-mega to fix the handling of the event stream when a
file is moved or renamed. The server reports a move as a delete of
the old node followed by an add of the same node, and go-mega was
losing track of the node's identity in the process, which could make
the old name reappear in listings and leave the backend unable to
remove it.
Microsoft has been progressively disabling sharing invitations on
both Business and Personal accounts - the driveItem invite API
returns 400 sharingFailed for any recipient on affected accounts -
which makes the permission writing tests impossible. Probe the API
once and skip the tests which need it when it is refused.
ownCloud does not carry the checksum over to the destination of a
server side copy and refuses attempts to set it afterwards, so the
destination legitimately has no hash. The logger vs lsf check
compared the predicted hash against the empty hash and failed.
Treat an empty hash in the listing as unknown rather than wrong,
matching how sync itself compares hashes.
ImageKit can't store zero length files. The test only skips itself
when the failure is reported synchronously, but with --vfs-cache-mode
writes the writeback happens asynchronously and deliberately swallows
the error, so the test fails on the missing files. Other backends
which can't store empty files already ignore this test.
Cloudinary's search API which NewObject uses is eventually
consistent, so the --compare-dest lookup of a just uploaded file
misses it and the test fails. TestCopyFileCompareDest is already
ignored for the same reason.
Open decoded range options against an unknown size, producing a
negative offset for suffix ranges (eg the last N bytes) and sending
syntactically invalid Range headers which the server ignored. A
suffix range then returned the whole file instead of the requested
tail.
Decode the options against the known object size so the offset is
always absolute, only send a Range header when one was requested, and
honour the requested count when the server ignores the Range header.
The Linkbox API is fronted by bot protection which under load
intermittently returns an HTML challenge page with a 200 status
instead of JSON. This surfaced as a fatal JSON decode error, and the
web API path misread the empty decoded result as an expired token,
hammering the login endpoint with requests which also failed to
decode and keeping the bot protection triggered.
Treat responses which fail to decode as retryable errors so the pacer
backs off until the block lifts, and only refresh the token when the
response actually parsed as JSON and reported an error.
Yandex Disk stores the modtime in a custom property set just after
the upload completes. The server sometimes silently drops this
property (the request returns success but the property is gone when
read back), leaving the object showing the upload time instead.
Check the modtime read back after upload and set it again if it
didn't stick.
put.io serializes trash operations per account so concurrent deletes
can fail with a 400 TRASH_LOCK_TIMEOUT error saying "There is an
ongoing blocking trash operation". This transient error was treated
as fatal - retry it instead.
Objects looked up by name used the /folder/itembyname.json endpoint
which does not return the file's MD5 hash, so NewObject returned
objects with no hash. Read the metadata with /file/info.json (which
does return the hash) instead, using itembyname only to find the file
id when it isn't known.
Copy and Move returned objects with only the id and size filled in,
leaving the modtime and hash empty, and SetModTime kept nanosecond
precision in memory while the server stores seconds. These made the
returned objects differ from a fresh listing of the same objects.
The Move method compared the source and destination paths relative to
their own Fs roots, so a server side move between two differently
rooted Fs instances on the same remote (as used by --backup-dir and
sync moves between remotes) was skipped as "already there" whenever
the relative paths coincided. With --backup-dir this destroyed the
file which was supposed to be backed up.
Compare the full paths including the Fs roots instead.
Previously running rclone config (or driving it via the rc API or web
GUI) with -vv would write secrets to the debug log.
This was dangerous as users debugging a failing config flow often
paste their -vv logs into the forum or GitHub issues.
These values are now redacted from the log as "XXX". Values whose
option is known are only redacted if the option is marked IsPassword
or Sensitive, so normal answers remain visible.
Use --dump auth to see the unredacted values when debugging a config
flow - rclone prints a warning that secrets will appear in the log
when this is in effect.
This was discovered by CodeQL: https://github.com/rclone/rclone/security/code-scanning/182
The logging functions take an object which is rendered into the log
line. Rendering it with %+v dumps all its fields, which for an object
holding backend config would include decrypted credentials. Every
object currently logged is a string or has a String method, so render
anything else as just its type to keep credentials out of the logs.
See: https://github.com/rclone/rclone/security/code-scanning/183
When processItems filled the last free transfer slot it checked the
next queued item's expiry before the transfer limit, so if that
expiry was still fractionally in the future the timer was reset
instead of stopped. This caused intermittent failures in
TestWriteBackMaxQueue which asserts the timer is stopped once
--transfers uploads are in progress.
Check the transfer limit first so the timer is always stopped when
the transfer limit is reached. The timer is restarted when an upload
finishes so nothing stalls.
Also fix a typo in TestWriteBackMaxQueue which named every queued
item "number1".
The server stores the SMB last write time with 100 ns (FILETIME)
precision and returns it in full in listings, so advertise that as
the precision instead of one second. This makes syncs preserve
sub-second modtimes when comparing and copying files.
Note that this means rclone will consider files with modtimes
differing by less than a second as needing their modtime set where
before it did not.
The server computes the MD5 checksum asynchronously after upload, so
the object returned from Put and Update often had no MD5 while a
fresh listing shortly afterwards would report one. This broke
wrappers which compare exact hashes, such as the hasher backend's
fingerprint and the VFS cache. Retry reading the metadata for a short
time after upload until the MD5 appears.
The object returned from Put and Update had an empty MD5 checksum
while a fresh listing would report the MD5 the server computed for
the upload. This broke wrappers which compare exact hashes, such as
the hasher backend's fingerprint and the VFS cache. Stat the file
after upload to pick up the server computed MD5.
Update (which Put also uses) cleared the object's MD5 checksum after
upload, so until the object was re-read it had no hash while a fresh
listing would report the MD5 the server computed. This broke wrappers
which compare exact hashes, such as the hasher backend's fingerprint
and the VFS cache. Re-read the object's metadata after upload to pick
up the server computed MD5 (and authoritative size).
Nick Craig-Wood