The rclone web GUI uses the Prefer header for sync/copy operations,
but the CORS middleware's preflight response did not include it in
Access-Control-Allow-Headers. This caused the browser to block
remote GUI copy requests with a CORS policy error:
Request header field prefer is not allowed by
Access-Control-Allow-Headers in preflight response.
Add 'Prefer' to the allowed headers list so remote GUI
access works correctly for file copy operations.
Fixes#9614
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Zoho config system ended every interactive create, update and
reconnect by calling m.Set("root_folder_id", workspaceID) in the
workspace_end state, with fs.ConfigChoose defaulting to the first
workspace. Any existing root_folder_id was therefore overwritten and
the remote silently repointed to the first workspace root.
This matters because reconnect is the documented fix for the 401
INVALID_OAUTHSCOPE download error - tokens issued before the
ZohoFiles.files.ALL scope was added lack download access - so users are
told to reconnect and then find all subsequent list/copy/sync/delete
operations pointed at a different, often shared, workspace.
Gate the workspace selection the way the drive backend does for team
drives (#5454): if a root_folder_id is already set, ask "Change current
root folder id ...?" defaulting to No and keep it; only run workspace
selection when it is empty or the user opts in. The token type rewrite
still runs on every reconnect so the scope refresh is unaffected. Also
expose root_folder_id as a standard advanced option (Sensitive, so
config redacted masks it) so it can be set and discovered like on
drive/box/onedrive.
Fixes#9575
Previously running rclone config (or driving it via the rc API or web
GUI) with -vv would write secrets to the debug log.
This was dangerous as users debugging a failing config flow often
paste their -vv logs into the forum or GitHub issues.
These values are now redacted from the log as "XXX". Values whose
option is known are only redacted if the option is marked IsPassword
or Sensitive, so normal answers remain visible.
Use --dump auth to see the unredacted values when debugging a config
flow - rclone prints a warning that secrets will appear in the log
when this is in effect.
This was discovered by CodeQL: https://github.com/rclone/rclone/security/code-scanning/182
The logging functions take an object which is rendered into the log
line. Rendering it with %+v dumps all its fields, which for an object
holding backend config would include decrypted credentials. Every
object currently logged is a string or has a String method, so render
anything else as just its type to keep credentials out of the logs.
See: https://github.com/rclone/rclone/security/code-scanning/183
Go 1.27 changes compress/flate output, which also changes archive/zip
byte output. The HTTP zip download tests currently compare raw zip bytes
against golden files, so they fail even though the generated zip archives
contain the expected files.
Compare zip entries and decompressed contents instead of the exact
compressed byte stream.
Signed-off-by: Mikel Olasagasti Uranga <mikel@olasagasti.info>
When processItems filled the last free transfer slot it checked the
next queued item's expiry before the transfer limit, so if that
expiry was still fractionally in the future the timer was reset
instead of stopped. This caused intermittent failures in
TestWriteBackMaxQueue which asserts the timer is stopped once
--transfers uploads are in progress.
Check the transfer limit first so the timer is always stopped when
the transfer limit is reached. The timer is restarted when an upload
finishes so nothing stalls.
Also fix a typo in TestWriteBackMaxQueue which named every queued
item "number1".
The server stores the SMB last write time with 100 ns (FILETIME)
precision and returns it in full in listings, so advertise that as
the precision instead of one second. This makes syncs preserve
sub-second modtimes when comparing and copying files.
Note that this means rclone will consider files with modtimes
differing by less than a second as needing their modtime set where
before it did not.
The server computes the MD5 checksum asynchronously after upload, so
the object returned from Put and Update often had no MD5 while a
fresh listing shortly afterwards would report one. This broke
wrappers which compare exact hashes, such as the hasher backend's
fingerprint and the VFS cache. Retry reading the metadata for a short
time after upload until the MD5 appears.
The object returned from Put and Update had an empty MD5 checksum
while a fresh listing would report the MD5 the server computed for
the upload. This broke wrappers which compare exact hashes, such as
the hasher backend's fingerprint and the VFS cache. Stat the file
after upload to pick up the server computed MD5.
Update (which Put also uses) cleared the object's MD5 checksum after
upload, so until the object was re-read it had no hash while a fresh
listing would report the MD5 the server computed. This broke wrappers
which compare exact hashes, such as the hasher backend's fingerprint
and the VFS cache. Re-read the object's metadata after upload to pick
up the server computed MD5 (and authoritative size).
Update (which Put also uses) kept the caller's full precision modtime
in memory while the upload finalize request sends it rounded to
microseconds, so until the object was re-read the modtime did not
match what a fresh listing would report. This broke wrappers which
compare exact modtimes, such as the hasher backend's fingerprint and
the VFS cache. Round the modtime to microseconds to match what the
server stores.
Update (which Put also uses) kept the caller's full precision modtime
in memory while the server stores second precision, so until the
object was re-read the modtime did not match what a fresh listing
would report. This broke wrappers which compare exact modtimes, such
as the hasher backend's fingerprint and the VFS cache. Truncate the
modtime to second precision to match what the server stores.
The objects returned from Put, Move and Update kept the caller's full
nanosecond precision modtime (which write responses echo back) while
the server stores millisecond precision, so until the object was
re-read the modtime did not match what a fresh listing would report.
Update also discarded the upload response so the object had no SHA256
checksum until re-read. This broke wrappers which compare exact
modtimes and hashes, such as the hasher backend's fingerprint and the
VFS cache. Truncate the modtime to millisecond precision to match
what the server stores and populate the object from the upload
response.
Update and SetModTime kept the caller's full nanosecond precision
modtime in memory while the server stores SMB LastWriteTime with
100 ns (FILETIME) precision, so until the object was re-read the
modtime did not match what a fresh listing would report. This broke
wrappers which compare exact modtimes, such as the hasher backend's
fingerprint and the VFS cache. Truncate the modtime to 100 ns
precision to match what the server stores.
SetModTime kept the caller's full precision modtime in memory while
the PROPPATCH sets it on the server with second precision, so until
the object was re-read the modtime did not match what a fresh listing
would report. This broke wrappers which compare exact modtimes, such
as the hasher backend's fingerprint and the VFS cache. Truncate the
modtime to second precision to match what the server stores.
SetModTime kept the caller's full precision modtime in memory while
the server stores second precision, so until the object was re-read
the modtime did not match what a fresh listing would report. This
broke wrappers which compare exact modtimes, such as the hasher
backend's fingerprint and the VFS cache. Truncate the modtime to
second precision to match what the server stores.
SetModTime kept the caller's full precision modtime in memory while
the server stores second precision, so until the object was re-read
the modtime did not match what a fresh listing would report. This
broke wrappers which compare exact modtimes, such as the hasher
backend's fingerprint and the VFS cache. Truncate the modtime to
second precision to match what the server stores.
SetModTime kept the caller's full precision modtime in memory while
the server stores second precision, so until the object was re-read
the modtime did not match what a fresh listing would report. This
broke wrappers which compare exact modtimes, such as the hasher
backend's fingerprint and the VFS cache. Truncate the modtime to
second precision to match what the server stores.
SetModTime (and the no_check_upload Update path) kept the caller's
full precision modtime in memory while the server stores second
precision, so until the object was re-read the modtime did not match
what a fresh listing would report. This broke wrappers which compare
exact modtimes, such as the hasher backend's fingerprint and the VFS
cache. Truncate the modtime to second precision to match what the
server stores.
TestRcDu relied on the default cache directory already existing. In clean
or container environments diskusage.New returned ENOENT, which the test
ignored before type asserting a nil result.
Use a temporary directory and require a successful response before
checking the disk usage values.
Add an AGENTS.md at the repository root describing the project's build,
test and code conventions for AI coding agents (Claude Code, Codex,
Cursor and similar). CLAUDE.md imports it so Claude Code reads the same
guidance.
Add an "AI-assisted contributions" section to CONTRIBUTING.md asking
contributors to understand and test AI-generated code themselves and to
trim the verbose comments these tools tend to produce, and add a
matching checkbox to the pull request template.
Previously if a cached file had grown larger than the remote object
and the cached range metadata was out of sync with the cache file
(e.g. after an unclean shutdown) - reloading the file failed with an
"invalid seek position" error. This aborted the writeback and left the
file inaccessible.
Rclone now recovers the bytes that are still available from the remote
and logs an ERROR that the local file is likely corrupted after an
interrupted upload.
See #9231.
Docker may re-send Create requests for volumes that already exist,
especially after a plugin restart. Previously this returned
ErrVolumeExists which Docker surfaced as "volume name must be unique".
Now if a volume with the same name already exists, Create returns
success (no-op), matching the Docker volume plugin protocol's
expectation of idempotent operations.
When restoring multiple volumes from saved state, each volume's
filesystem setup (including connecting to the remote) was done
sequentially. Now volumes are restored concurrently, so a single
slow or unreachable remote doesn't delay the restoration of other
volumes.
When restoring volumes from saved state during plugin startup, a slow
or unreachable remote could block indefinitely in fs.NewFs. Add a
30-second per-volume timeout context so that individual volume failures
are logged and skipped rather than blocking the entire plugin.
Previously, restoreState in NewDriver would restore volumes AND perform
FUSE mounts synchronously before the Docker plugin socket was created.
This caused two problems:
1. The monChan was created after restoreState, but mount() sends on
monChan, causing a deadlock (send on nil channel blocks forever).
2. Even with the channel fix, slow or hanging mounts during state
restore would block the server socket from being created, causing
Docker to time out after ~13 seconds with "no such file or
directory" for the plugin socket.
Fix by:
- Moving monChan creation and monitor goroutine start before
restoreState
- Splitting state restore into two phases: restoreState (metadata +
filesystem setup only) and RestoreMounts (actual FUSE mounts)
- Calling RestoreMounts asynchronously after the server starts
listening
- Performing mount restoration concurrently across volumes
Fixes#9231
shouldRetry treated every Code=200501 API error as a transient storage
block error and retried it. Proton also returns Code=200501 with an
HTTP 422 for permanent validation failures (e.g. a content key packet
that cannot be verified, or an upload format the account is not
enabled for), so these were retried until the operation timed out
causing a multi-minute hang.
This fixes it by only retrying Code=200501 when it is not a permanent
client (4xx) error.
Update to github.com/ProtonMail/gopenpgp/v3 from v2 by updating to
- github.com/rclone/go-proton-api@v1.0.3
- github.com/rclone/Proton-API-Bridge@v1.0.4
Fixes#9434
SetModTime (which Put and Update also use) kept the caller's full
precision modtime in memory while Chtimes sets it with second
precision, so until the object was re-read the modtime did not match
what a fresh listing would report. This broke wrappers which compare
exact modtimes, such as the hasher backend's fingerprint and the VFS
cache. Truncate the modtime to second precision to match what the
server stores.
Update and SetModTime kept the caller's full precision modtime in
memory while the server stores second precision, so until the object
was re-read the modtime did not match what a fresh listing would
report. This broke wrappers which compare exact modtimes, such as the
hasher backend's fingerprint and the VFS cache. Truncate the modtime
to second precision to match what the server stores.
Layers which wrap backends, such as the hasher backend and the VFS
cache, use fs.Fingerprint to detect whether an object has changed.
This only works if the object a backend hands back from a write
operation is identical to the object a fresh listing returns. If a
backend keeps the source's nanosecond precision modtime in memory
while the server stores milliseconds, or omits a hash the server
knows, every fingerprint comparison fails and cached hashes or files
are silently invalidated.
The existing tests compared modtimes within f.Precision() so they
could not detect these sub-precision divergences.
Add a checkFingerprint helper which asserts the fast and slow
fs.Fingerprint of the object returned from a write operation exactly
match those of the object read back from the remote, and wire it into
all the write paths which return or mutate an object: Put (all
variants), PutStream, large and streamed uploads, Update, SetModTime,
server side Copy and Move, and the metadata mutating Copy/Move
subtests.
Rework the ObjectOpenFingerprint test to use the same helper against
a fully read reference object, with subtests reporting which kind of
read is being checked.
Object.Update and SetModTime left the in-memory modtime at the
source's nanosecond precision, while the server stores milliseconds.
Afterwards the in-memory state did not match what a fresh List would
report, which broke wrappers that compare modtimes (e.g. the hasher
backend's fingerprint, which made it lose track of hashes whenever a
file was replaced).
The SDK serializes modtimes as UnixMilli and NewIncompleteFile
already rounds to milliseconds, so Put and the chunked upload path
are unaffected. Round the modtime to milliseconds in Update and
SetModTime too, so every path holds the same value the server
returns, without needing to re-read the metadata from the server
after upload.
Fixes#9308
When rclone sync replaced an existing file, it called Update which
pruned the old hash but never computed or stored the new one. This
left the file with no hash entry in the bolt DB.
This applies the same hashing logic in Put to Update: compute hashes
during the transfer via a hashingReader and store them afterwards. The
common hash-wrapping and hash-storing logic was extracted into a
function.
Fixes#9308
When doing a copy (no delete mode) without a logger, the destination
listing can be cancelled as soon as the source listing finishes, since
dst-only entries won't be processed.
This is particularly beneficial with --fast-list where the dst listing
may fetch the entire directory tree upfront via ListR. Cancelling it
early avoids waiting for a potentially large listing that won't be used.
Adds NoProcessDstOnly flag to March which, when set, cancels the dst
listing context once the source channel is exhausted in matchListings.
Fixes#9226
Add a context parameter to listDirFn so that each call site can pass
its own context. The closures in makeListDir previously captured
m.Ctx at creation time; they now use the context passed at call time
instead. This is needed so that processJob can pass a cancellable
context for the destination listing independently of the source.
Note: callers must pass m.Ctx (or a child of it) to preserve the
existing cancellation behaviour where listings stop when the march
context is cancelled.
Interactively-entered passwords were run through NFKC Unicode
normalization before being obscured, which silently rewrote characters
such as ª (U+00AA) to a. The obscured password then revealed to
something different from what the user typed confusing everyone.
Normalization is only needed for the config encryption master
password, so apply it there (in SetConfigPassword) rather than in the
shared checkPassword used for backend password options.
When `operations/stat` / StatJSON is called on a directory path it
lists the parent directory to find the target entry. If the parent has
millions of entries this is very expensive.
This fixes the problem for bucket-based backends with ListP by listing
the target directory itself first. It will stop the listing
immediately if any files are found meaning it is safe to run on
directories with millions of files.
After PUTting a file to the upload URL, Yandex keeps the file locked
for writing until the upload operation finishes committing on the
server. The PUT returned before this happened, so the following
SetModTime raced the still-in-progress write and got spurious 500
Internal Server Error responses.
Capture the operation_id returned with the upload URL and poll the
operation status until it reports success before returning, so the file
is fully committed before we access it.
The background chunk cleaner slept for the whole ChunkCleanInterval
(default 1 minute) before checking its stop channel, and only ran
CleanUpCache via the select default branch. This meant a cache that
had been stopped by StopBackgroundRunners could keep running
CleanUpCache for up to an interval afterwards.
The cache backend tests all share a single on-disk chunk store (the
TestInternalCache remote), so a lingering cleaner from a finished test
could call CleanChunksBySize and os.RemoveAll chunks that a later,
unrelated test had just written. The later test would then read a
chunk back and get an unexpected EOF - eg
TestInternalMaxChunkSizeRespected failing intermittently on CI.
Wait on a timer and the stop channel together so a stop is honoured
immediately and the cleaner can never run again once stopped.
The TestLogger/TestRepoCompare and TestLogger/TestBeforeVsAfter
testscript scenarios filled src and dst by downloading two old rclone
source archives from GitHub with `rclone copyurl`. Whenever GitHub or
the network hiccuped (eg a 502 Bad Gateway) the downloads failed and
the tests failed with it, making them flaky on CI.
Generate two overlapping trees of files in the test Setup instead.
They cover the same comparison categories the scripts exercise
(matching, differing, src-only and dst-only files) so the tests are
just as meaningful but no longer depend on the network.
The volume plugin parses the remote option as a trusted connection
string, which can run local commands via backend options. Spell out
that access to the unix or TCP socket is equivalent to command
execution as the serving user, document the unix socket permissions,
and warn that the TCP socket is unauthenticated.
A folder shortcut pointing at one of its own ancestor folders (for
example a shortcut to FolderX placed inside FolderX) made rclone recurse
forever when dereferencing shortcuts, duplicating the folder contents
until the disk was full.
Rclone now detects when a folder shortcut targets an ancestor directory
using the directory cache, leaves that shortcut out of the listing and
logs an ERROR, so the rest of the drive can still be copied.
Fixes#7118Fixes#9565Closes#9051
cleanRootPath used filepath.Abs which prepends the current directory,
but the resulting absolute path does not always refer to the same
directory as the original relative path - for example when the current
directory is shadowed by a mount or has been removed. This made
"rclone copy --links . ../dst" fail where "cp -ra . ../dst" succeeds.
rclone now cleans the path lexically on non-Windows platforms instead,
leaving relative roots relative so the OS resolves them against the live
working directory. Windows still makes the path absolute as required for
UNC long-path conversion.
Previously the only way to remove an option from a remote was to set it
to an empty string, which is not the same as deleting it - a present but
empty value overrides the option's default whereas a deleted key
restores it. Editing the file by hand isn't an option for an encrypted
config either.
This adds a "config unset" command and a "config/unset" rc endpoint to
remove one or more keys from an existing remote.
Nick Craig-Wood
Erol OzcanandNick Craig-Wood
Mikel Olasagasti UrangaandNick Craig-Wood
dougalandNick Craig-Wood
Cao YuhangandNick Craig-Wood
