This commit introduced aux.go which unfortunately is illegal on windows.
e006d7c13f vfs: fix crash when multiple mounts or servers share the same VFS
Rename aux_test.go too to keep the pair together.
Add two new options, pin_host_key and host_keys, that
together provide a TOFU host-key validation mode for users who don't
maintain a known_hosts file. When --sftp-pin-host-key is used, rclone
records the server's host key into host_keys on the first successful
connection and verifies it on every subsequent connection.
host_keys is always validated when non-empty, so it can also be used
by hand to pin a known fingerprint without enabling TOFU writing.
known_hosts_file takes precedence if both are set. SSH host
certificates are rejected with a clear message pointing at
known_hosts_file. On-the-fly remotes log a warning since the captured
key cannot be persisted.
The connection pacer in getSftpConnection used to retry every error,
so permanent failures (host key mismatch, certificate rejection, auth
failure, etc.) were looped 10 times before reporting to the user.
Switch to using fserrors.ShouldRetry which matches the pattern other
backends use so only genuinely retriable errors (timeouts, EOF,
network blips) are retried and permanent errors are surfaced
immediately.
The VFS is shared between users with the same remote and options, for
example two mounts created over the rc, or a mount and an NFS server.
Each node has a single Sys() slot which mount, mount2 and serve nfs
all used to attach their per-node data. With a shared VFS the users
overwrote each other's data: at best churning the cached FUSE nodes,
and since the slot was an atomic.Value, panicking with "store of
inconsistently typed value" as soon as two users stored different
types on the same node.
This change gives each node auxiliary values keyed by owner, set with
SetAux and read with Aux, so each user of the VFS has an independent
slot. The mounts now cache their FUSE nodes under their own key,
leaving Sys - which is read through the os.FileInfo interface -
reserved for users like serve nfs which need to control what that
returns.
Nodes with nothing attached use less memory than before (one pointer
instead of an atomic.Value) and reads remain lock free.
Bug discovered while thinking about #9617
Streamed multipart uploads had two problems when the client uploaded parts
concurrently:
Parts arriving ahead of the next part needed by the backend stream were buffered
in memory without limit, acknowledging each part as soon as it was received. A
client uploading faster than the backend could drain would therefore balloon the
server's memory to the size of the upload. Buffering is now bounded a new
--multipart-streaming-buffer-limit flag (default 256Mi, 0 for unlimited): a part
that would take the buffer over the limit is not read until the stream drains,
applying backpressure to the client instead of using unbounded memory.
A part uploaded again with the same number - typically a client retrying after
its request timed out - left a stale copy in the reorder buffer which made
CompleteMultipartUpload fail with InvalidPart, aborting the whole upload.
Re-uploaded parts are now handled properly: a copy still in the buffer is
replaced, an identical copy of an already streamed part is accepted as a no-op,
and only replacing an already streamed part with different content (which the
in-order stream cannot honour) is rejected.
Every write operation waits for the server's processing tasks which
currently take minutes each, so the full test suite takes much longer
than the timeout. Running with -short skips the FsEncoding subtests
(which are already on the ignore list) and the chunked upload tests,
removing a large number of these waits.
Multiple security vulnerabilities have been identified and addressed
in grpc-go affecting the xDS RBAC authorization engine
(internal/xds/rbac) and the HTTP/2 transport server implementation
(internal/transport).
This updates to master to add the fix which allows it to compile on
plan9.
CloudKit is inconsistent about how it encodes the isDeleted field on
album records, returning a JSON boolean (true/false) for some accounts
and a number (0/1) for others. The numeric form caused listing of a
photo library to fail with:
json: cannot unmarshal number into Go struct field
ckBoolField.records.fields.isDeleted.value of type bool
The encoding also varies over time, not just per account: a full HTTP
dump from the reporting user showed the server sending
"isDeleted" : { "value" : 0, "type" : "INT64" }
but the same account later reverted to the boolean encoding with no
client change. Asset records already deliver isDeleted as a number, so
both encodings are in active use server side and either may appear.
Accept both encodings when parsing CloudKit boolean fields.
See: https://forum.rclone.org/t/error-when-trying-to-list-contents-of-primarysync-directory-in-icloud-photos/54028
The old rclone-integration-test item has accumulated years of catalog
task churn and IA now deprioritises and periodically holds its tasks,
which makes the tests wait on the server's processing tasks for a
very long time. Start again with a fresh item (created with noindex
set so the test files stay out of IA's search index).
The session cache was only populated on a fresh username and password
login, so once a session ID was stored in the config every Fs
instance created its own Mega session with its own copy of the
account's node tree. The server side move code relies on all Fs
instances of a user sharing one session, and with separate sessions a
move between two rclone remotes grafted a node from one tree into
another, where the asynchronous event replays of the two sessions
raced and could detach the moved file from the destination directory
so it disappeared from listings.
Cache the session however the login was done. This also stops every
extra Fs instance re-downloading the whole account node tree.
Sync refuses to delete files when the global error stats are non-zero
so a single backend error in one transform test made every following
transform test in the same test binary fail with "not deleting files
as there were IO errors". Reset the stats at the start of each test.
The pacer callbacks only retried 429 responses, so transient server
errors like 502 Bad Gateway failed transfers immediately. Retry the
standard retryable status codes and transport errors everywhere, make
chunk upload retries resend the whole chunk, and fix a potential nil
pointer dereference when logging a bad token response.
The go-diskfs library now requires io/fs.ValidPath style paths (no
leading slash, "." for the root) so convert paths at the library
boundary. This fixes listing failures and the resulting test hangs
after the update to go-diskfs v1.9.3.
This fixes the code to compile with the updated dependencies:
- squashfs: implement the new Path method required by the go-diskfs
backend.Storage interface, returning an empty string as there is no
underlying path.
- dropbox: convert to and from the SDK's new DBXTime type (an alias
for time.Time) for ClientModified, TimeInvited and Expires.
- dropbox: remove a stray debug fmt.Printf from the shared folder
listing.
- pin go-systemd to v22.6.0 to fix netbsd builds
go-systemd v22.7.0 uses unix.ClockGettime and unix.CLOCK_MONOTONIC
which golang.org/x/sys does not define for netbsd, so the build fails.
See: https://github.com/coreos/go-systemd/issues/512
- pin google.golang.org/grpc to v1.80.0 to fix plan9 builds
grpc v1.81.0 and later use syscall.Errno, syscall.ECONNRESET and
syscall.ECONNABORTED which do not exist on plan9, so the build fails
See: https://github.com/grpc/grpc-go/issues/9253
The /file-entries/duplicate endpoint returns a 500 Server Error for
every request (reported to Drime 2026-06) which made all server side
copies fail after 10 retries. Remove the Copy method so rclone falls
back to downloading and re-uploading instead. It can be restored if
Drime fix the endpoint.
This updates go-mega to fix the handling of the event stream when a
file is moved or renamed. The server reports a move as a delete of
the old node followed by an add of the same node, and go-mega was
losing track of the node's identity in the process, which could make
the old name reappear in listings and leave the backend unable to
remove it.
Microsoft has been progressively disabling sharing invitations on
both Business and Personal accounts - the driveItem invite API
returns 400 sharingFailed for any recipient on affected accounts -
which makes the permission writing tests impossible. Probe the API
once and skip the tests which need it when it is refused.
ownCloud does not carry the checksum over to the destination of a
server side copy and refuses attempts to set it afterwards, so the
destination legitimately has no hash. The logger vs lsf check
compared the predicted hash against the empty hash and failed.
Treat an empty hash in the listing as unknown rather than wrong,
matching how sync itself compares hashes.
ImageKit can't store zero length files. The test only skips itself
when the failure is reported synchronously, but with --vfs-cache-mode
writes the writeback happens asynchronously and deliberately swallows
the error, so the test fails on the missing files. Other backends
which can't store empty files already ignore this test.
Cloudinary's search API which NewObject uses is eventually
consistent, so the --compare-dest lookup of a just uploaded file
misses it and the test fails. TestCopyFileCompareDest is already
ignored for the same reason.
Open decoded range options against an unknown size, producing a
negative offset for suffix ranges (eg the last N bytes) and sending
syntactically invalid Range headers which the server ignored. A
suffix range then returned the whole file instead of the requested
tail.
Decode the options against the known object size so the offset is
always absolute, only send a Range header when one was requested, and
honour the requested count when the server ignores the Range header.
The Linkbox API is fronted by bot protection which under load
intermittently returns an HTML challenge page with a 200 status
instead of JSON. This surfaced as a fatal JSON decode error, and the
web API path misread the empty decoded result as an expired token,
hammering the login endpoint with requests which also failed to
decode and keeping the bot protection triggered.
Treat responses which fail to decode as retryable errors so the pacer
backs off until the block lifts, and only refresh the token when the
response actually parsed as JSON and reported an error.
Yandex Disk stores the modtime in a custom property set just after
the upload completes. The server sometimes silently drops this
property (the request returns success but the property is gone when
read back), leaving the object showing the upload time instead.
Check the modtime read back after upload and set it again if it
didn't stick.
put.io serializes trash operations per account so concurrent deletes
can fail with a 400 TRASH_LOCK_TIMEOUT error saying "There is an
ongoing blocking trash operation". This transient error was treated
as fatal - retry it instead.
Objects looked up by name used the /folder/itembyname.json endpoint
which does not return the file's MD5 hash, so NewObject returned
objects with no hash. Read the metadata with /file/info.json (which
does return the hash) instead, using itembyname only to find the file
id when it isn't known.
Copy and Move returned objects with only the id and size filled in,
leaving the modtime and hash empty, and SetModTime kept nanosecond
precision in memory while the server stores seconds. These made the
returned objects differ from a fresh listing of the same objects.
The Move method compared the source and destination paths relative to
their own Fs roots, so a server side move between two differently
rooted Fs instances on the same remote (as used by --backup-dir and
sync moves between remotes) was skipped as "already there" whenever
the relative paths coincided. With --backup-dir this destroyed the
file which was supposed to be backed up.
Compare the full paths including the Fs roots instead.
Previously running rclone config (or driving it via the rc API or web
GUI) with -vv would write secrets to the debug log.
This was dangerous as users debugging a failing config flow often
paste their -vv logs into the forum or GitHub issues.
These values are now redacted from the log as "XXX". Values whose
option is known are only redacted if the option is marked IsPassword
or Sensitive, so normal answers remain visible.
Use --dump auth to see the unredacted values when debugging a config
flow - rclone prints a warning that secrets will appear in the log
when this is in effect.
This was discovered by CodeQL: https://github.com/rclone/rclone/security/code-scanning/182
The logging functions take an object which is rendered into the log
line. Rendering it with %+v dumps all its fields, which for an object
holding backend config would include decrypted credentials. Every
object currently logged is a string or has a String method, so render
anything else as just its type to keep credentials out of the logs.
See: https://github.com/rclone/rclone/security/code-scanning/183
When processItems filled the last free transfer slot it checked the
next queued item's expiry before the transfer limit, so if that
expiry was still fractionally in the future the timer was reset
instead of stopped. This caused intermittent failures in
TestWriteBackMaxQueue which asserts the timer is stopped once
--transfers uploads are in progress.
Check the transfer limit first so the timer is always stopped when
the transfer limit is reached. The timer is restarted when an upload
finishes so nothing stalls.
Also fix a typo in TestWriteBackMaxQueue which named every queued
item "number1".
The server stores the SMB last write time with 100 ns (FILETIME)
precision and returns it in full in listings, so advertise that as
the precision instead of one second. This makes syncs preserve
sub-second modtimes when comparing and copying files.
Note that this means rclone will consider files with modtimes
differing by less than a second as needing their modtime set where
before it did not.
The server computes the MD5 checksum asynchronously after upload, so
the object returned from Put and Update often had no MD5 while a
fresh listing shortly afterwards would report one. This broke
wrappers which compare exact hashes, such as the hasher backend's
fingerprint and the VFS cache. Retry reading the metadata for a short
time after upload until the MD5 appears.
The object returned from Put and Update had an empty MD5 checksum
while a fresh listing would report the MD5 the server computed for
the upload. This broke wrappers which compare exact hashes, such as
the hasher backend's fingerprint and the VFS cache. Stat the file
after upload to pick up the server computed MD5.
Update (which Put also uses) cleared the object's MD5 checksum after
upload, so until the object was re-read it had no hash while a fresh
listing would report the MD5 the server computed. This broke wrappers
which compare exact hashes, such as the hasher backend's fingerprint
and the VFS cache. Re-read the object's metadata after upload to pick
up the server computed MD5 (and authoritative size).
Update (which Put also uses) kept the caller's full precision modtime
in memory while the upload finalize request sends it rounded to
microseconds, so until the object was re-read the modtime did not
match what a fresh listing would report. This broke wrappers which
compare exact modtimes, such as the hasher backend's fingerprint and
the VFS cache. Round the modtime to microseconds to match what the
server stores.
Update (which Put also uses) kept the caller's full precision modtime
in memory while the server stores second precision, so until the
object was re-read the modtime did not match what a fresh listing
would report. This broke wrappers which compare exact modtimes, such
as the hasher backend's fingerprint and the VFS cache. Truncate the
modtime to second precision to match what the server stores.
The objects returned from Put, Move and Update kept the caller's full
nanosecond precision modtime (which write responses echo back) while
the server stores millisecond precision, so until the object was
re-read the modtime did not match what a fresh listing would report.
Update also discarded the upload response so the object had no SHA256
checksum until re-read. This broke wrappers which compare exact
modtimes and hashes, such as the hasher backend's fingerprint and the
VFS cache. Truncate the modtime to millisecond precision to match
what the server stores and populate the object from the upload
response.
Update and SetModTime kept the caller's full nanosecond precision
modtime in memory while the server stores SMB LastWriteTime with
100 ns (FILETIME) precision, so until the object was re-read the
modtime did not match what a fresh listing would report. This broke
wrappers which compare exact modtimes, such as the hasher backend's
fingerprint and the VFS cache. Truncate the modtime to 100 ns
precision to match what the server stores.
SetModTime kept the caller's full precision modtime in memory while
the PROPPATCH sets it on the server with second precision, so until
the object was re-read the modtime did not match what a fresh listing
would report. This broke wrappers which compare exact modtimes, such
as the hasher backend's fingerprint and the VFS cache. Truncate the
modtime to second precision to match what the server stores.
SetModTime kept the caller's full precision modtime in memory while
the server stores second precision, so until the object was re-read
the modtime did not match what a fresh listing would report. This
broke wrappers which compare exact modtimes, such as the hasher
backend's fingerprint and the VFS cache. Truncate the modtime to
second precision to match what the server stores.
SetModTime kept the caller's full precision modtime in memory while
the server stores second precision, so until the object was re-read
the modtime did not match what a fresh listing would report. This
broke wrappers which compare exact modtimes, such as the hasher
backend's fingerprint and the VFS cache. Truncate the modtime to
second precision to match what the server stores.
SetModTime kept the caller's full precision modtime in memory while
the server stores second precision, so until the object was re-read
the modtime did not match what a fresh listing would report. This
broke wrappers which compare exact modtimes, such as the hasher
backend's fingerprint and the VFS cache. Truncate the modtime to
second precision to match what the server stores.
SetModTime (and the no_check_upload Update path) kept the caller's
full precision modtime in memory while the server stores second
precision, so until the object was re-read the modtime did not match
what a fresh listing would report. This broke wrappers which compare
exact modtimes, such as the hasher backend's fingerprint and the VFS
cache. Truncate the modtime to second precision to match what the
server stores.
Add an AGENTS.md at the repository root describing the project's build,
test and code conventions for AI coding agents (Claude Code, Codex,
Cursor and similar). CLAUDE.md imports it so Claude Code reads the same
guidance.
Add an "AI-assisted contributions" section to CONTRIBUTING.md asking
contributors to understand and test AI-generated code themselves and to
trim the verbose comments these tools tend to produce, and add a
matching checkbox to the pull request template.
Previously if a cached file had grown larger than the remote object
and the cached range metadata was out of sync with the cache file
(e.g. after an unclean shutdown) - reloading the file failed with an
"invalid seek position" error. This aborted the writeback and left the
file inaccessible.
Rclone now recovers the bytes that are still available from the remote
and logs an ERROR that the local file is likely corrupted after an
interrupted upload.
See #9231.
Docker may re-send Create requests for volumes that already exist,
especially after a plugin restart. Previously this returned
ErrVolumeExists which Docker surfaced as "volume name must be unique".
Now if a volume with the same name already exists, Create returns
success (no-op), matching the Docker volume plugin protocol's
expectation of idempotent operations.
When restoring multiple volumes from saved state, each volume's
filesystem setup (including connecting to the remote) was done
sequentially. Now volumes are restored concurrently, so a single
slow or unreachable remote doesn't delay the restoration of other
volumes.
When restoring volumes from saved state during plugin startup, a slow
or unreachable remote could block indefinitely in fs.NewFs. Add a
30-second per-volume timeout context so that individual volume failures
are logged and skipped rather than blocking the entire plugin.
Previously, restoreState in NewDriver would restore volumes AND perform
FUSE mounts synchronously before the Docker plugin socket was created.
This caused two problems:
1. The monChan was created after restoreState, but mount() sends on
monChan, causing a deadlock (send on nil channel blocks forever).
2. Even with the channel fix, slow or hanging mounts during state
restore would block the server socket from being created, causing
Docker to time out after ~13 seconds with "no such file or
directory" for the plugin socket.
Fix by:
- Moving monChan creation and monitor goroutine start before
restoreState
- Splitting state restore into two phases: restoreState (metadata +
filesystem setup only) and RestoreMounts (actual FUSE mounts)
- Calling RestoreMounts asynchronously after the server starts
listening
- Performing mount restoration concurrently across volumes
Fixes#9231
shouldRetry treated every Code=200501 API error as a transient storage
block error and retried it. Proton also returns Code=200501 with an
HTTP 422 for permanent validation failures (e.g. a content key packet
that cannot be verified, or an upload format the account is not
enabled for), so these were retried until the operation timed out
causing a multi-minute hang.
This fixes it by only retrying Code=200501 when it is not a permanent
client (4xx) error.
Update to github.com/ProtonMail/gopenpgp/v3 from v2 by updating to
- github.com/rclone/go-proton-api@v1.0.3
- github.com/rclone/Proton-API-Bridge@v1.0.4
Fixes#9434
SetModTime (which Put and Update also use) kept the caller's full
precision modtime in memory while Chtimes sets it with second
precision, so until the object was re-read the modtime did not match
what a fresh listing would report. This broke wrappers which compare
exact modtimes, such as the hasher backend's fingerprint and the VFS
cache. Truncate the modtime to second precision to match what the
server stores.
Update and SetModTime kept the caller's full precision modtime in
memory while the server stores second precision, so until the object
was re-read the modtime did not match what a fresh listing would
report. This broke wrappers which compare exact modtimes, such as the
hasher backend's fingerprint and the VFS cache. Truncate the modtime
to second precision to match what the server stores.
Layers which wrap backends, such as the hasher backend and the VFS
cache, use fs.Fingerprint to detect whether an object has changed.
This only works if the object a backend hands back from a write
operation is identical to the object a fresh listing returns. If a
backend keeps the source's nanosecond precision modtime in memory
while the server stores milliseconds, or omits a hash the server
knows, every fingerprint comparison fails and cached hashes or files
are silently invalidated.
The existing tests compared modtimes within f.Precision() so they
could not detect these sub-precision divergences.
Add a checkFingerprint helper which asserts the fast and slow
fs.Fingerprint of the object returned from a write operation exactly
match those of the object read back from the remote, and wire it into
all the write paths which return or mutate an object: Put (all
variants), PutStream, large and streamed uploads, Update, SetModTime,
server side Copy and Move, and the metadata mutating Copy/Move
subtests.
Rework the ObjectOpenFingerprint test to use the same helper against
a fully read reference object, with subtests reporting which kind of
read is being checked.
Object.Update and SetModTime left the in-memory modtime at the
source's nanosecond precision, while the server stores milliseconds.
Afterwards the in-memory state did not match what a fresh List would
report, which broke wrappers that compare modtimes (e.g. the hasher
backend's fingerprint, which made it lose track of hashes whenever a
file was replaced).
The SDK serializes modtimes as UnixMilli and NewIncompleteFile
already rounds to milliseconds, so Put and the chunked upload path
are unaffected. Round the modtime to milliseconds in Update and
SetModTime too, so every path holds the same value the server
returns, without needing to re-read the metadata from the server
after upload.
Fixes#9308
When rclone sync replaced an existing file, it called Update which
pruned the old hash but never computed or stored the new one. This
left the file with no hash entry in the bolt DB.
This applies the same hashing logic in Put to Update: compute hashes
during the transfer via a hashingReader and store them afterwards. The
common hash-wrapping and hash-storing logic was extracted into a
function.
Fixes#9308
When doing a copy (no delete mode) without a logger, the destination
listing can be cancelled as soon as the source listing finishes, since
dst-only entries won't be processed.
This is particularly beneficial with --fast-list where the dst listing
may fetch the entire directory tree upfront via ListR. Cancelling it
early avoids waiting for a potentially large listing that won't be used.
Adds NoProcessDstOnly flag to March which, when set, cancels the dst
listing context once the source channel is exhausted in matchListings.
Fixes#9226
Add a context parameter to listDirFn so that each call site can pass
its own context. The closures in makeListDir previously captured
m.Ctx at creation time; they now use the context passed at call time
instead. This is needed so that processJob can pass a cancellable
context for the destination listing independently of the source.
Note: callers must pass m.Ctx (or a child of it) to preserve the
existing cancellation behaviour where listings stop when the march
context is cancelled.
Interactively-entered passwords were run through NFKC Unicode
normalization before being obscured, which silently rewrote characters
such as ª (U+00AA) to a. The obscured password then revealed to
something different from what the user typed confusing everyone.
Normalization is only needed for the config encryption master
password, so apply it there (in SetConfigPassword) rather than in the
shared checkPassword used for backend password options.
When `operations/stat` / StatJSON is called on a directory path it
lists the parent directory to find the target entry. If the parent has
millions of entries this is very expensive.
This fixes the problem for bucket-based backends with ListP by listing
the target directory itself first. It will stop the listing
immediately if any files are found meaning it is safe to run on
directories with millions of files.
After PUTting a file to the upload URL, Yandex keeps the file locked
for writing until the upload operation finishes committing on the
server. The PUT returned before this happened, so the following
SetModTime raced the still-in-progress write and got spurious 500
Internal Server Error responses.
Capture the operation_id returned with the upload URL and poll the
operation status until it reports success before returning, so the file
is fully committed before we access it.
The background chunk cleaner slept for the whole ChunkCleanInterval
(default 1 minute) before checking its stop channel, and only ran
CleanUpCache via the select default branch. This meant a cache that
had been stopped by StopBackgroundRunners could keep running
CleanUpCache for up to an interval afterwards.
The cache backend tests all share a single on-disk chunk store (the
TestInternalCache remote), so a lingering cleaner from a finished test
could call CleanChunksBySize and os.RemoveAll chunks that a later,
unrelated test had just written. The later test would then read a
chunk back and get an unexpected EOF - eg
TestInternalMaxChunkSizeRespected failing intermittently on CI.
Wait on a timer and the stop channel together so a stop is honoured
immediately and the cleaner can never run again once stopped.
The TestLogger/TestRepoCompare and TestLogger/TestBeforeVsAfter
testscript scenarios filled src and dst by downloading two old rclone
source archives from GitHub with `rclone copyurl`. Whenever GitHub or
the network hiccuped (eg a 502 Bad Gateway) the downloads failed and
the tests failed with it, making them flaky on CI.
Generate two overlapping trees of files in the test Setup instead.
They cover the same comparison categories the scripts exercise
(matching, differing, src-only and dst-only files) so the tests are
just as meaningful but no longer depend on the network.
The volume plugin parses the remote option as a trusted connection
string, which can run local commands via backend options. Spell out
that access to the unix or TCP socket is equivalent to command
execution as the serving user, document the unix socket permissions,
and warn that the TCP socket is unauthenticated.
A folder shortcut pointing at one of its own ancestor folders (for
example a shortcut to FolderX placed inside FolderX) made rclone recurse
forever when dereferencing shortcuts, duplicating the folder contents
until the disk was full.
Rclone now detects when a folder shortcut targets an ancestor directory
using the directory cache, leaves that shortcut out of the listing and
logs an ERROR, so the rest of the drive can still be copied.
Fixes#7118Fixes#9565Closes#9051
cleanRootPath used filepath.Abs which prepends the current directory,
but the resulting absolute path does not always refer to the same
directory as the original relative path - for example when the current
directory is shadowed by a mount or has been removed. This made
"rclone copy --links . ../dst" fail where "cp -ra . ../dst" succeeds.
rclone now cleans the path lexically on non-Windows platforms instead,
leaving relative roots relative so the OS resolves them against the live
working directory. Windows still makes the path absolute as required for
UNC long-path conversion.
Previously the only way to remove an option from a remote was to set it
to an empty string, which is not the same as deleting it - a present but
empty value overrides the option's default whereas a deleted key
restores it. Editing the file by hand isn't an option for an encrypted
config either.
This adds a "config unset" command and a "config/unset" rc endpoint to
remove one or more keys from an existing remote.
Before this change a streamed multipart upload wrote its parts
straight to the object's final path on the underlying remote. That
meant an in-progress upload overwrote any object already stored under
that name, and aborting or failing the upload destroyed it. The
opposite of the S3 guarantee that an object only changes on a
successful CompleteMultipartUpload.
Remotes that upload atomically already (PartialUploads is false, e.g.
object stores) are safe to stream straight to the destination, so they
still do. Remotes where a partial upload is visible (PartialUploads is
true, e.g. local) now stream the parts to a temporary object instead
and move it, server-side, into its final place only when the upload
completes. A failed or aborted upload then just removes the temporary
object and leaves any pre-existing object untouched.
The temporary-object path needs the remote to support a server-side
move or copy in addition to PutStream uploads fall back to being
buffered in memory as before. The temporary objects are named with a
leading ".rclone_multipart_upload_" and hidden from listings.
Before this change when an aborted upload had overwritten a
pre-existing object of the same name, this left a ghost of that object
in every listing.
This invalidate the VFS cache on all the multipart upload abort paths,
so listings reflect what is actually on the underlying Fs.
The folder list limiter granted the caller-supplied time immediately in
the burst phase, so concurrent callers observing time.Now() out of order
could record grant times that moved backwards. The sliding safety log
indexes grants as an ordered history, so out-of-order grants could also
breach the rolling-window cap. Clamp each grant to be at or after the
previous one so grant times are always monotonic.
Closing a stream in the parallel chunked reader cancels the stream's
context, so the in-flight read returns context.Canceled. This was wrapped
and returned as "failed to read stream", which the VFS cache downloader
treats as a real download error - it only recognises
asyncreader.ErrorStreamAbandoned as a benign teardown, as returned by the
sequential reader.
Return asyncreader.ErrorStreamAbandoned for a cancellation so tearing down
the parallel reader (on close, seek or reposition) is recognised as benign,
matching the sequential reader, instead of logging download errors and
retrying when --vfs-read-chunk-streams is used with --vfs-cache-mode full.
_createFile opened the cache file with O_RDWR but not O_CREATE, relying
on the file already existing. When the file had been removed underneath
rclone - either by _checkObject dropping a stale entry during open, or by
external deletion - the open failed and surfaced a hard "IO error" to the
application instead of recreating the file. Add O_CREATE so the cache
self-heals in that case.
Add TestObjectOpenFingerprint which reads an object's fast and slow
fingerprints, then opens it in several ways - a full read, a seek read
and ranged reads - and after a refresh, checking the fingerprint never
changes.
On a range request Azure returns the whole-blob MD5 in the
x-ms-blob-content-md5 header (BlobContentMD5) and leaves Content-MD5
empty. The download metadata decoder only read Content-MD5, so every
ranged read overwrote the object's MD5 with an empty string.
With --vfs-cache-mode full this changed the object's fingerprint between
opens, so the VFS cache judged every reopened file as stale and
re-downloaded it, defeating the cache and inflating egress. Prefer
BlobContentMD5 and never overwrite a known hash with an empty one.
With -l/--links rclone recreates a .rclonelink object as a symlink. A
malicious or compromised source could serve a symlink whose target points
outside the destination, plus a sibling object whose path traverses it, so
that rclone followed the planted symlink and wrote outside the destination
causing arbitrary file write.
When translating symlinks, rclone now performs all destination writes
(directory creation, file writes and symlink creation) through an os.Root
anchored at the destination. os.Root resolves every path component relative
to the destination's file descriptor and refuses any that escapes the root,
even under concurrent modification, so a planted symlink can never be
traversed out of the destination.
Symlinks are still reproduced verbatim - including ones whose target points
outside the destination - so backups remain faithful. Only writing
*through* such a link is refused. In-tree symlinks are unaffected.
Fixes CVE-2026-54572
Fixes GHSA-cf44-9pgv-m4xc
When applying the "mode" from --metadata the local backend cast the
source value straight to an os.FileMode, so a source that supplied a
mode with Go's setuid, setgid or sticky bits set would have those bits
applied to the freshly written file. As both the file content and its
metadata come from the source remote, a malicious source could plant a
setuid binary, and a victim running "rclone copy -M" as root against
an untrusted remote could end up with a root-owned setuid binary with
attacker-controlled content.
Rclone records "mode" in the unix st_mode layout where the special
bits live in different positions to Go's os.FileMode, so honest
sources never actually round-tripped these bits in the first place.
Apply only the permission bits by default, which closes this off and
is backwards compatible, and add the --local-metadata-restore-special-bits
lag to restore the previous behaviour for trusted sources such as
restoring a system backup made by rclone.
See: GHSA-945v-v9p3-v5xw
The CheckRedirect policy strips the X-Amz-Security-Token header when a
redirect chain crosses a host, but it only compared the host and ignored
the scheme. A redirect that kept the same host:port but downgraded
https:// to http:// was treated as the same host, so the STS session
token was re-sent over a plaintext connection where it could be observed.
Fixes GHSA-cf44-9pgv-m4xc
A user could reach another user's private repository by sending a path
such as /<me>/../<victim>/config. The authorization check compares the
first path segment against the authenticated user, while the backend
object key was built from the raw, un-cleaned URL path.
Reject any non-canonical request path so the authorization segment and
the backend object key can no longer disagree.
Fixes GHSA-fqj9-69pf-6pjg
Archive entry names are attacker controlled. `rclone archive extract` stripped
only a leading `./` and then joined the entry name onto the destination
directory with `path.Join`, which collapses `..` segments. An entry such as
`../escaped.txt` extracted into `:s3:bucket/safe/prefix` therefore resolved to
`bucket/safe/escaped.txt`, outside the selected `prefix` directory - a path
traversal ("Zip Slip") attack that could create or overwrite sibling objects on
any destination remote.
Entry names are now validated before use: a leading `./` is still stripped (tar
archives created with `tar -czf archive.tar.gz .` rely on this), but any entry
with a `..` path component is rejected. Both `/` and `\` are treated as
separators when looking for `..`, as the local backend treats `\` as a path
separator on Windows.
Fixes: GHSA-4vr5-p2gc-h23p
S3 object keys are opaque names that may legally contain `..` segments. `serve
s3` built backend paths with `path.Join(bucket, key)`, which normalised the key
so a request such as `GET /bucket/../root-secret.txt` resolved to a file outside
the selected bucket elsewhere under the serve root. Listing prefixes and
multipart uploads were affected also.
This did not allow reading of files outside the root, but did allow reading of
files in the root which normally aren't visible; only directories are visible as
buckets normally.
Because `serve s3` maps keys to file paths it cannot represent every opaque S3
key, so rather than normalising keys (which would alias distinct keys onto one
file as well as allow traversal) it now rejects any key that is not already in
canonical path form - containing `..`, `.`, `//` or a leading or trailing slash
- with a 400 Bad Request, as MinIO does. Directory listing prefixes are
validated the same way but allow the empty bucket-root prefix and an optional
trailing slash.
Fixes: GHSA-8v25-v8p6-qf7v
When mounting or otherwise opening an S3 prefix without a trailing
slash, rclone probes the path with a HEAD request to see whether it is
actually a file. Since v1.72.0 (#8975) any error other than "not
found" from that probe was fatal, so credentials scoped to a prefix -
which return 403 rather than 404 for the prefix key - could no longer
open the prefix at all.
6440052fbd s3: fix single file copying behavior with low permission
A 403 on the probe is ambiguous: it can mean either "this is the file
you named but you may not HEAD it" or "this is a prefix you may list
but not HEAD". When the HEAD is not permitted we now fall back to a
listing to disambiguate: if the path has children it is treated as a
directory, otherwise it is treated as a file.
Fixes#9582
rclone's shared Google Drive and Google Photos client_id is being
retired and will stop working during 2026. When creating a new remote
that would use it, the config wizard now warns the user and asks the
user to enter their own client_id and secret instead. Service account
and environment auth are unaffected as they don't use the shared
client_id.
See: https://forum.rclone.org/t/google-drive-and-google-photos-users-action-required/54005
The shared Google Drive and Google Photos client_id is being retired and
will stop working during 2026. Warn users who rely on it (ie who have not
configured their own client_id) so they can create their own in advance.
The warning is only shown for auth flows that actually use the shared
client_id, not for service account, environment or anonymous auth.
See: https://forum.rclone.org/t/google-drive-and-google-photos-users-action-required/54005
The --disable-zip flag was registered manually and was missing from
OptionsInfo, so it could not be set over the rc interface. Move it
into OptionsInfo like serve webdav does, which keeps the command line
flag and also makes it settable via rc.
Nick Craig-Wood
