When making a release from master, the range since the last minor
release includes all the commits which were cherry-picked to the
previous stable branch and already published in its point releases,
so their changelog lines had to be deleted by hand.
make_changelog.py now finds the previous stable branch (eg
v1.74-stable) from the version being released and skips any commits
released there, detected via cherry-pick -x trailers and git cherry
patch equivalence. Skipped commits are listed on stderr for review.
Releases made from a stable branch are unaffected.
In this commit we attempted to wait for the success report of an
upload to fix the 500 error:
fe78b559d1 yandex: fix 500 errors by waiting for uploads to complete before setting modtime
However Yandex Disk finalizes an upload asynchronously on its servers.
Waiting for the upload operation to report success is not enough -
under load the server reports the operation as successful slightly
before the file is fully finalized, so setting the modification time
straight after an upload can still fail with 500 Internal Server
Error.
Yandex support recommend waiting 1.5s - 3s after the upload before
modifying the file's metadata, so add an --yandex-upload-wait option
(default off) to insert a delay between the upload completing and the
modification time being set.
Runner image ubuntu24/20260726.254 updated podman from 4.9.3 to 5.8.4
which is installed from the podman-static bundle. This bundle ships a
non-setuid fusermount3 in /usr/local/bin which shadows the setuid
/usr/bin/fusermount3 from the fuse3 package (as /usr/local/bin comes
first in PATH). A non-setuid fusermount3 cannot mount as an
unprivileged user, so the cmd/mount and cmd/mount2 tests failed with:
fusermount3: mount failed: Operation not permitted
Fix by removing the podman bundled fusermount binaries so the distro
provided setuid one is used.
The session close flag was only computed after each append, so a
known-size upload which fits in a single chunk sent all its data and
then issued a zero-payload append purely to close the session - one
wasted round trip per small file on the default batched upload path.
Set the close flag before the first append when the size is known to
fit in one chunk.
Since batch mode became the default all uploads go through
uploadChunked, which allocated a full chunk-size retry buffer (48 MiB
by default) regardless of the file size. With the `--transfers 32`
recommended for small file uploads that is ~1.5 GiB of buffer to
upload tiny files.
Size the buffer to the file size when it is known and smaller than a
chunk.
Before this change, if an object compressed with "Content-Encoding:
gzip" was downloaded, a length and hash mismatch would occur since the
go runtime automatically decompressed the object on download, giving
errors like
corrupted on transfer: sizes differ
This change sets "Accept-Encoding: gzip" on all requests which stops
the go runtime decompressing objects on download, so compressed
objects are downloaded as-is with intact size and hash information.
If --oos-decompress is set then rclone will decompress objects with
"Content-Encoding: gzip" as they are received, at the cost of not
being able to check the length or the hash of the downloaded object.
Fixes#9694
DirMove returned fs.ErrorDirExists for any error from the destination
existence check not just when the destination actually exists. That
made sync silently fall back to file-by-file moves and masked the real
failure.
Return the underlying error instead when the check fails for any other
reason.
DirMove checked whether the destination exists using the raw path but
performed the rename with the encoded path, so for directory names
needing encoding (trailing spaces or periods, characters like
\* ? : < > | " or a literal backslash) the existence check looked at
a different server path than the rename used.
If revealing the password, creating the Kerberos client or the SMB
handshake failed after the TCP connection was established, the
connection was never closed.
A new KerberosFactory was constructed for every dial, so the client,
error and ccache modification time caches it holds were discarded
after a single use. Every new SMB connection re-read the Kerberos
config, re-parsed the ccache and did a fresh KDC exchange.
Share a single factory so clients are cached across connections as
intended, and refreshed when the ccache file changes.
Object.Open dereferenced the response's ContentLength pointer without checking
it. The OCI SDK leaves ContentLength nil when the server replies without a
Content-Length header or ContentRange which caused a nil pointer panic.
Now the size is only updated when the response actually provides one, leaving
the size from the object metadata in place otherwise.
This also fixes the same potential problem in the newObject code.
The multipart upload complete endpoint returns 200 with an empty body,
but rclone tried to decode that body as JSON, failed with EOF and
retried until the retries ran out, so every upload failed even though
the server had actually completed it.
Fixed by not attempting to decode the response body.
This was a regression introduced in
a4972de505 shade: retry server errors instead of failing the transfer
which started treating the JSON decode error as fatal where previously
it was accidentally ignored.
Add a new "list_parallelism" advanced option (requires use_arrow_list) which
splits a directory's blob-name keyspace into ranges and lists them
concurrently using the Arrow startFrom/endBefore range parameters. Both
recursive (ListR) and single-directory listings benefit.
The keyspace is sharded with a single-case (digit+lowercase) character
ladder. Single case is deliberate: the service validates startFrom /
endBefore case-insensitively (while listing case-sensitively), so a
mixed-case boundary pair is rejected with a 400; its endBefore filtering is
byte-ordered, so the listing is exact.
The service only honours endBefore on the Arrow listing path of accounts with
Blob Listing with Apache Arrow enabled: elsewhere it either rejects the
request with 400 OperationNotSupportedWithFeatureMissing or, if it fell back
to XML, would ignore the shards' upper bounds. Both cases are detected and
the parallel listing falls back to a sequential listing.
Listing 100,000 objects from a well-connected server:
config time vs XML
xml 23.9s 1.0x
use_arrow_list 6.9s 3.5x
list_parallelism=2 4.4s 5.5x
list_parallelism=4 2.8s 8.4x
list_parallelism=6 2.2s 10.9x
list_parallelism=10 1.5s 15.4x
list_parallelism=20 1.4s 17.6x
list_parallelism=30 1.1s 21.7x
So Apache Arrow listing alone is ~3.5x faster than XML, and parallel listing
takes that to ~20x. The ladder caps at 37 shards, so the benefit plateaus
beyond list_parallelism=30.
The option is hidden from the configurator, command line help and docs
until the feature is officially released.
Add a new "use_arrow_list" advanced option which fetches directory listings
using the ListBlobs Apache Arrow response format instead of XML. This can be
substantially faster for very large containers.
This is EXPERIMENTAL and requires the "Blob Listing with Apache Arrow" preview
feature to be enabled on the storage account. It is not supported on accounts
with a hierarchical namespace (ADLS Gen2). If the feature is not enabled the
service returns XML and the listing transparently falls back to it. Connection
string auth is not supported - it also falls back to the normal XML listing.
The option is hidden from the configurator, command line help and docs until the
feature is officially released.
This adds a temporary subpackage implementing the experimental "Blob
Listing with Apache Arrow" feature on top of the released Azure azblob
SDK.
The Azure SDK for Go supports Arrow listing on its unreleased
feature/storage/bifrost branch (commit c6fa341ca22b) but that can't be
used in a release. This package exposes the same options and pager
interface as the experimental SDK, built on the released SDK's public
API plus verbatim copies of its Arrow decoder and of the two internal
auth policies (shared key signing and storage bearer challenge) which
the released SDK does not export. Copied files carry MIT attribution
headers pinned to their upstream source.
When Arrow listing ships in a released azblob the whole package will be
deleted and callers pointed back at the SDK - the interface is
deliberately source compatible to make that a mechanical change.
Also exports auth.Transporter so the new package's pipeline can use
rclone's fshttp transport, and adds github.com/apache/arrow-go/v18 as a
direct dependency for decoding the Arrow IPC stream.
Backend options with Hide: fs.OptionHideBoth are hidden from the config
wizard and from the command line flag listing, but "rclone help backend"
(and therefore the autogenerated backend documentation) still showed
them. Skip them there too so fully hidden options no longer appear in
the docs.
This commit introduced aux.go which unfortunately is illegal on windows.
e006d7c13f vfs: fix crash when multiple mounts or servers share the same VFS
Rename aux_test.go too to keep the pair together.
Add two new options, pin_host_key and host_keys, that
together provide a TOFU host-key validation mode for users who don't
maintain a known_hosts file. When --sftp-pin-host-key is used, rclone
records the server's host key into host_keys on the first successful
connection and verifies it on every subsequent connection.
host_keys is always validated when non-empty, so it can also be used
by hand to pin a known fingerprint without enabling TOFU writing.
known_hosts_file takes precedence if both are set. SSH host
certificates are rejected with a clear message pointing at
known_hosts_file. On-the-fly remotes log a warning since the captured
key cannot be persisted.
The connection pacer in getSftpConnection used to retry every error,
so permanent failures (host key mismatch, certificate rejection, auth
failure, etc.) were looped 10 times before reporting to the user.
Switch to using fserrors.ShouldRetry which matches the pattern other
backends use so only genuinely retriable errors (timeouts, EOF,
network blips) are retried and permanent errors are surfaced
immediately.
The VFS is shared between users with the same remote and options, for
example two mounts created over the rc, or a mount and an NFS server.
Each node has a single Sys() slot which mount, mount2 and serve nfs
all used to attach their per-node data. With a shared VFS the users
overwrote each other's data: at best churning the cached FUSE nodes,
and since the slot was an atomic.Value, panicking with "store of
inconsistently typed value" as soon as two users stored different
types on the same node.
This change gives each node auxiliary values keyed by owner, set with
SetAux and read with Aux, so each user of the VFS has an independent
slot. The mounts now cache their FUSE nodes under their own key,
leaving Sys - which is read through the os.FileInfo interface -
reserved for users like serve nfs which need to control what that
returns.
Nodes with nothing attached use less memory than before (one pointer
instead of an atomic.Value) and reads remain lock free.
Bug discovered while thinking about #9617
Streamed multipart uploads had two problems when the client uploaded parts
concurrently:
Parts arriving ahead of the next part needed by the backend stream were buffered
in memory without limit, acknowledging each part as soon as it was received. A
client uploading faster than the backend could drain would therefore balloon the
server's memory to the size of the upload. Buffering is now bounded a new
--multipart-streaming-buffer-limit flag (default 256Mi, 0 for unlimited): a part
that would take the buffer over the limit is not read until the stream drains,
applying backpressure to the client instead of using unbounded memory.
A part uploaded again with the same number - typically a client retrying after
its request timed out - left a stale copy in the reorder buffer which made
CompleteMultipartUpload fail with InvalidPart, aborting the whole upload.
Re-uploaded parts are now handled properly: a copy still in the buffer is
replaced, an identical copy of an already streamed part is accepted as a no-op,
and only replacing an already streamed part with different content (which the
in-order stream cannot honour) is rejected.
Every write operation waits for the server's processing tasks which
currently take minutes each, so the full test suite takes much longer
than the timeout. Running with -short skips the FsEncoding subtests
(which are already on the ignore list) and the chunked upload tests,
removing a large number of these waits.
Multiple security vulnerabilities have been identified and addressed
in grpc-go affecting the xDS RBAC authorization engine
(internal/xds/rbac) and the HTTP/2 transport server implementation
(internal/transport).
This updates to master to add the fix which allows it to compile on
plan9.
CloudKit is inconsistent about how it encodes the isDeleted field on
album records, returning a JSON boolean (true/false) for some accounts
and a number (0/1) for others. The numeric form caused listing of a
photo library to fail with:
json: cannot unmarshal number into Go struct field
ckBoolField.records.fields.isDeleted.value of type bool
The encoding also varies over time, not just per account: a full HTTP
dump from the reporting user showed the server sending
"isDeleted" : { "value" : 0, "type" : "INT64" }
but the same account later reverted to the boolean encoding with no
client change. Asset records already deliver isDeleted as a number, so
both encodings are in active use server side and either may appear.
Accept both encodings when parsing CloudKit boolean fields.
See: https://forum.rclone.org/t/error-when-trying-to-list-contents-of-primarysync-directory-in-icloud-photos/54028
The old rclone-integration-test item has accumulated years of catalog
task churn and IA now deprioritises and periodically holds its tasks,
which makes the tests wait on the server's processing tasks for a
very long time. Start again with a fresh item (created with noindex
set so the test files stay out of IA's search index).
The session cache was only populated on a fresh username and password
login, so once a session ID was stored in the config every Fs
instance created its own Mega session with its own copy of the
account's node tree. The server side move code relies on all Fs
instances of a user sharing one session, and with separate sessions a
move between two rclone remotes grafted a node from one tree into
another, where the asynchronous event replays of the two sessions
raced and could detach the moved file from the destination directory
so it disappeared from listings.
Cache the session however the login was done. This also stops every
extra Fs instance re-downloading the whole account node tree.
Sync refuses to delete files when the global error stats are non-zero
so a single backend error in one transform test made every following
transform test in the same test binary fail with "not deleting files
as there were IO errors". Reset the stats at the start of each test.
The pacer callbacks only retried 429 responses, so transient server
errors like 502 Bad Gateway failed transfers immediately. Retry the
standard retryable status codes and transport errors everywhere, make
chunk upload retries resend the whole chunk, and fix a potential nil
pointer dereference when logging a bad token response.
The go-diskfs library now requires io/fs.ValidPath style paths (no
leading slash, "." for the root) so convert paths at the library
boundary. This fixes listing failures and the resulting test hangs
after the update to go-diskfs v1.9.3.
This fixes the code to compile with the updated dependencies:
- squashfs: implement the new Path method required by the go-diskfs
backend.Storage interface, returning an empty string as there is no
underlying path.
- dropbox: convert to and from the SDK's new DBXTime type (an alias
for time.Time) for ClientModified, TimeInvited and Expires.
- dropbox: remove a stray debug fmt.Printf from the shared folder
listing.
- pin go-systemd to v22.6.0 to fix netbsd builds
go-systemd v22.7.0 uses unix.ClockGettime and unix.CLOCK_MONOTONIC
which golang.org/x/sys does not define for netbsd, so the build fails.
See: https://github.com/coreos/go-systemd/issues/512
- pin google.golang.org/grpc to v1.80.0 to fix plan9 builds
grpc v1.81.0 and later use syscall.Errno, syscall.ECONNRESET and
syscall.ECONNABORTED which do not exist on plan9, so the build fails
See: https://github.com/grpc/grpc-go/issues/9253
The /file-entries/duplicate endpoint returns a 500 Server Error for
every request (reported to Drime 2026-06) which made all server side
copies fail after 10 retries. Remove the Copy method so rclone falls
back to downloading and re-uploading instead. It can be restored if
Drime fix the endpoint.
This updates go-mega to fix the handling of the event stream when a
file is moved or renamed. The server reports a move as a delete of
the old node followed by an add of the same node, and go-mega was
losing track of the node's identity in the process, which could make
the old name reappear in listings and leave the backend unable to
remove it.
Microsoft has been progressively disabling sharing invitations on
both Business and Personal accounts - the driveItem invite API
returns 400 sharingFailed for any recipient on affected accounts -
which makes the permission writing tests impossible. Probe the API
once and skip the tests which need it when it is refused.
ownCloud does not carry the checksum over to the destination of a
server side copy and refuses attempts to set it afterwards, so the
destination legitimately has no hash. The logger vs lsf check
compared the predicted hash against the empty hash and failed.
Treat an empty hash in the listing as unknown rather than wrong,
matching how sync itself compares hashes.
ImageKit can't store zero length files. The test only skips itself
when the failure is reported synchronously, but with --vfs-cache-mode
writes the writeback happens asynchronously and deliberately swallows
the error, so the test fails on the missing files. Other backends
which can't store empty files already ignore this test.
Cloudinary's search API which NewObject uses is eventually
consistent, so the --compare-dest lookup of a just uploaded file
misses it and the test fails. TestCopyFileCompareDest is already
ignored for the same reason.
Open decoded range options against an unknown size, producing a
negative offset for suffix ranges (eg the last N bytes) and sending
syntactically invalid Range headers which the server ignored. A
suffix range then returned the whole file instead of the requested
tail.
Decode the options against the known object size so the offset is
always absolute, only send a Range header when one was requested, and
honour the requested count when the server ignores the Range header.
The Linkbox API is fronted by bot protection which under load
intermittently returns an HTML challenge page with a 200 status
instead of JSON. This surfaced as a fatal JSON decode error, and the
web API path misread the empty decoded result as an expired token,
hammering the login endpoint with requests which also failed to
decode and keeping the bot protection triggered.
Treat responses which fail to decode as retryable errors so the pacer
backs off until the block lifts, and only refresh the token when the
response actually parsed as JSON and reported an error.
Yandex Disk stores the modtime in a custom property set just after
the upload completes. The server sometimes silently drops this
property (the request returns success but the property is gone when
read back), leaving the object showing the upload time instead.
Check the modtime read back after upload and set it again if it
didn't stick.
put.io serializes trash operations per account so concurrent deletes
can fail with a 400 TRASH_LOCK_TIMEOUT error saying "There is an
ongoing blocking trash operation". This transient error was treated
as fatal - retry it instead.
Objects looked up by name used the /folder/itembyname.json endpoint
which does not return the file's MD5 hash, so NewObject returned
objects with no hash. Read the metadata with /file/info.json (which
does return the hash) instead, using itembyname only to find the file
id when it isn't known.
Copy and Move returned objects with only the id and size filled in,
leaving the modtime and hash empty, and SetModTime kept nanosecond
precision in memory while the server stores seconds. These made the
returned objects differ from a fresh listing of the same objects.
The Move method compared the source and destination paths relative to
their own Fs roots, so a server side move between two differently
rooted Fs instances on the same remote (as used by --backup-dir and
sync moves between remotes) was skipped as "already there" whenever
the relative paths coincided. With --backup-dir this destroyed the
file which was supposed to be backed up.
Compare the full paths including the Fs roots instead.
Previously running rclone config (or driving it via the rc API or web
GUI) with -vv would write secrets to the debug log.
This was dangerous as users debugging a failing config flow often
paste their -vv logs into the forum or GitHub issues.
These values are now redacted from the log as "XXX". Values whose
option is known are only redacted if the option is marked IsPassword
or Sensitive, so normal answers remain visible.
Use --dump auth to see the unredacted values when debugging a config
flow - rclone prints a warning that secrets will appear in the log
when this is in effect.
This was discovered by CodeQL: https://github.com/rclone/rclone/security/code-scanning/182
The logging functions take an object which is rendered into the log
line. Rendering it with %+v dumps all its fields, which for an object
holding backend config would include decrypted credentials. Every
object currently logged is a string or has a String method, so render
anything else as just its type to keep credentials out of the logs.
See: https://github.com/rclone/rclone/security/code-scanning/183
When processItems filled the last free transfer slot it checked the
next queued item's expiry before the transfer limit, so if that
expiry was still fractionally in the future the timer was reset
instead of stopped. This caused intermittent failures in
TestWriteBackMaxQueue which asserts the timer is stopped once
--transfers uploads are in progress.
Check the transfer limit first so the timer is always stopped when
the transfer limit is reached. The timer is restarted when an upload
finishes so nothing stalls.
Also fix a typo in TestWriteBackMaxQueue which named every queued
item "number1".
The server stores the SMB last write time with 100 ns (FILETIME)
precision and returns it in full in listings, so advertise that as
the precision instead of one second. This makes syncs preserve
sub-second modtimes when comparing and copying files.
Note that this means rclone will consider files with modtimes
differing by less than a second as needing their modtime set where
before it did not.
The server computes the MD5 checksum asynchronously after upload, so
the object returned from Put and Update often had no MD5 while a
fresh listing shortly afterwards would report one. This broke
wrappers which compare exact hashes, such as the hasher backend's
fingerprint and the VFS cache. Retry reading the metadata for a short
time after upload until the MD5 appears.
The object returned from Put and Update had an empty MD5 checksum
while a fresh listing would report the MD5 the server computed for
the upload. This broke wrappers which compare exact hashes, such as
the hasher backend's fingerprint and the VFS cache. Stat the file
after upload to pick up the server computed MD5.
Update (which Put also uses) cleared the object's MD5 checksum after
upload, so until the object was re-read it had no hash while a fresh
listing would report the MD5 the server computed. This broke wrappers
which compare exact hashes, such as the hasher backend's fingerprint
and the VFS cache. Re-read the object's metadata after upload to pick
up the server computed MD5 (and authoritative size).
Update (which Put also uses) kept the caller's full precision modtime
in memory while the upload finalize request sends it rounded to
microseconds, so until the object was re-read the modtime did not
match what a fresh listing would report. This broke wrappers which
compare exact modtimes, such as the hasher backend's fingerprint and
the VFS cache. Round the modtime to microseconds to match what the
server stores.
Update (which Put also uses) kept the caller's full precision modtime
in memory while the server stores second precision, so until the
object was re-read the modtime did not match what a fresh listing
would report. This broke wrappers which compare exact modtimes, such
as the hasher backend's fingerprint and the VFS cache. Truncate the
modtime to second precision to match what the server stores.
The objects returned from Put, Move and Update kept the caller's full
nanosecond precision modtime (which write responses echo back) while
the server stores millisecond precision, so until the object was
re-read the modtime did not match what a fresh listing would report.
Update also discarded the upload response so the object had no SHA256
checksum until re-read. This broke wrappers which compare exact
modtimes and hashes, such as the hasher backend's fingerprint and the
VFS cache. Truncate the modtime to millisecond precision to match
what the server stores and populate the object from the upload
response.
Update and SetModTime kept the caller's full nanosecond precision
modtime in memory while the server stores SMB LastWriteTime with
100 ns (FILETIME) precision, so until the object was re-read the
modtime did not match what a fresh listing would report. This broke
wrappers which compare exact modtimes, such as the hasher backend's
fingerprint and the VFS cache. Truncate the modtime to 100 ns
precision to match what the server stores.
SetModTime kept the caller's full precision modtime in memory while
the PROPPATCH sets it on the server with second precision, so until
the object was re-read the modtime did not match what a fresh listing
would report. This broke wrappers which compare exact modtimes, such
as the hasher backend's fingerprint and the VFS cache. Truncate the
modtime to second precision to match what the server stores.
SetModTime kept the caller's full precision modtime in memory while
the server stores second precision, so until the object was re-read
the modtime did not match what a fresh listing would report. This
broke wrappers which compare exact modtimes, such as the hasher
backend's fingerprint and the VFS cache. Truncate the modtime to
second precision to match what the server stores.
SetModTime kept the caller's full precision modtime in memory while
the server stores second precision, so until the object was re-read
the modtime did not match what a fresh listing would report. This
broke wrappers which compare exact modtimes, such as the hasher
backend's fingerprint and the VFS cache. Truncate the modtime to
second precision to match what the server stores.
SetModTime kept the caller's full precision modtime in memory while
the server stores second precision, so until the object was re-read
the modtime did not match what a fresh listing would report. This
broke wrappers which compare exact modtimes, such as the hasher
backend's fingerprint and the VFS cache. Truncate the modtime to
second precision to match what the server stores.
SetModTime (and the no_check_upload Update path) kept the caller's
full precision modtime in memory while the server stores second
precision, so until the object was re-read the modtime did not match
what a fresh listing would report. This broke wrappers which compare
exact modtimes, such as the hasher backend's fingerprint and the VFS
cache. Truncate the modtime to second precision to match what the
server stores.
Add an AGENTS.md at the repository root describing the project's build,
test and code conventions for AI coding agents (Claude Code, Codex,
Cursor and similar). CLAUDE.md imports it so Claude Code reads the same
guidance.
Add an "AI-assisted contributions" section to CONTRIBUTING.md asking
contributors to understand and test AI-generated code themselves and to
trim the verbose comments these tools tend to produce, and add a
matching checkbox to the pull request template.
Previously if a cached file had grown larger than the remote object
and the cached range metadata was out of sync with the cache file
(e.g. after an unclean shutdown) - reloading the file failed with an
"invalid seek position" error. This aborted the writeback and left the
file inaccessible.
Rclone now recovers the bytes that are still available from the remote
and logs an ERROR that the local file is likely corrupted after an
interrupted upload.
See #9231.
Docker may re-send Create requests for volumes that already exist,
especially after a plugin restart. Previously this returned
ErrVolumeExists which Docker surfaced as "volume name must be unique".
Now if a volume with the same name already exists, Create returns
success (no-op), matching the Docker volume plugin protocol's
expectation of idempotent operations.
When restoring multiple volumes from saved state, each volume's
filesystem setup (including connecting to the remote) was done
sequentially. Now volumes are restored concurrently, so a single
slow or unreachable remote doesn't delay the restoration of other
volumes.
When restoring volumes from saved state during plugin startup, a slow
or unreachable remote could block indefinitely in fs.NewFs. Add a
30-second per-volume timeout context so that individual volume failures
are logged and skipped rather than blocking the entire plugin.
Previously, restoreState in NewDriver would restore volumes AND perform
FUSE mounts synchronously before the Docker plugin socket was created.
This caused two problems:
1. The monChan was created after restoreState, but mount() sends on
monChan, causing a deadlock (send on nil channel blocks forever).
2. Even with the channel fix, slow or hanging mounts during state
restore would block the server socket from being created, causing
Docker to time out after ~13 seconds with "no such file or
directory" for the plugin socket.
Fix by:
- Moving monChan creation and monitor goroutine start before
restoreState
- Splitting state restore into two phases: restoreState (metadata +
filesystem setup only) and RestoreMounts (actual FUSE mounts)
- Calling RestoreMounts asynchronously after the server starts
listening
- Performing mount restoration concurrently across volumes
Fixes#9231
shouldRetry treated every Code=200501 API error as a transient storage
block error and retried it. Proton also returns Code=200501 with an
HTTP 422 for permanent validation failures (e.g. a content key packet
that cannot be verified, or an upload format the account is not
enabled for), so these were retried until the operation timed out
causing a multi-minute hang.
This fixes it by only retrying Code=200501 when it is not a permanent
client (4xx) error.
Update to github.com/ProtonMail/gopenpgp/v3 from v2 by updating to
- github.com/rclone/go-proton-api@v1.0.3
- github.com/rclone/Proton-API-Bridge@v1.0.4
Fixes#9434
SetModTime (which Put and Update also use) kept the caller's full
precision modtime in memory while Chtimes sets it with second
precision, so until the object was re-read the modtime did not match
what a fresh listing would report. This broke wrappers which compare
exact modtimes, such as the hasher backend's fingerprint and the VFS
cache. Truncate the modtime to second precision to match what the
server stores.
Update and SetModTime kept the caller's full precision modtime in
memory while the server stores second precision, so until the object
was re-read the modtime did not match what a fresh listing would
report. This broke wrappers which compare exact modtimes, such as the
hasher backend's fingerprint and the VFS cache. Truncate the modtime
to second precision to match what the server stores.
Layers which wrap backends, such as the hasher backend and the VFS
cache, use fs.Fingerprint to detect whether an object has changed.
This only works if the object a backend hands back from a write
operation is identical to the object a fresh listing returns. If a
backend keeps the source's nanosecond precision modtime in memory
while the server stores milliseconds, or omits a hash the server
knows, every fingerprint comparison fails and cached hashes or files
are silently invalidated.
The existing tests compared modtimes within f.Precision() so they
could not detect these sub-precision divergences.
Add a checkFingerprint helper which asserts the fast and slow
fs.Fingerprint of the object returned from a write operation exactly
match those of the object read back from the remote, and wire it into
all the write paths which return or mutate an object: Put (all
variants), PutStream, large and streamed uploads, Update, SetModTime,
server side Copy and Move, and the metadata mutating Copy/Move
subtests.
Rework the ObjectOpenFingerprint test to use the same helper against
a fully read reference object, with subtests reporting which kind of
read is being checked.
Object.Update and SetModTime left the in-memory modtime at the
source's nanosecond precision, while the server stores milliseconds.
Afterwards the in-memory state did not match what a fresh List would
report, which broke wrappers that compare modtimes (e.g. the hasher
backend's fingerprint, which made it lose track of hashes whenever a
file was replaced).
The SDK serializes modtimes as UnixMilli and NewIncompleteFile
already rounds to milliseconds, so Put and the chunked upload path
are unaffected. Round the modtime to milliseconds in Update and
SetModTime too, so every path holds the same value the server
returns, without needing to re-read the metadata from the server
after upload.
Fixes#9308
When rclone sync replaced an existing file, it called Update which
pruned the old hash but never computed or stored the new one. This
left the file with no hash entry in the bolt DB.
This applies the same hashing logic in Put to Update: compute hashes
during the transfer via a hashingReader and store them afterwards. The
common hash-wrapping and hash-storing logic was extracted into a
function.
Fixes#9308
When doing a copy (no delete mode) without a logger, the destination
listing can be cancelled as soon as the source listing finishes, since
dst-only entries won't be processed.
This is particularly beneficial with --fast-list where the dst listing
may fetch the entire directory tree upfront via ListR. Cancelling it
early avoids waiting for a potentially large listing that won't be used.
Adds NoProcessDstOnly flag to March which, when set, cancels the dst
listing context once the source channel is exhausted in matchListings.
Fixes#9226
Add a context parameter to listDirFn so that each call site can pass
its own context. The closures in makeListDir previously captured
m.Ctx at creation time; they now use the context passed at call time
instead. This is needed so that processJob can pass a cancellable
context for the destination listing independently of the source.
Note: callers must pass m.Ctx (or a child of it) to preserve the
existing cancellation behaviour where listings stop when the march
context is cancelled.
Interactively-entered passwords were run through NFKC Unicode
normalization before being obscured, which silently rewrote characters
such as ª (U+00AA) to a. The obscured password then revealed to
something different from what the user typed confusing everyone.
Normalization is only needed for the config encryption master
password, so apply it there (in SetConfigPassword) rather than in the
shared checkPassword used for backend password options.
When `operations/stat` / StatJSON is called on a directory path it
lists the parent directory to find the target entry. If the parent has
millions of entries this is very expensive.
This fixes the problem for bucket-based backends with ListP by listing
the target directory itself first. It will stop the listing
immediately if any files are found meaning it is safe to run on
directories with millions of files.
Nick Craig-Wood
