When a multi-thread source ignores ranged reads, abort the partial
destination and retry the copy as a single stream. Reset accounting
before the fallback so transfer progress remains accurate.
Validate ranged GET responses before returning their bodies. Close invalid
responses, retry malformed partial responses through the pacer, and return
without retrying when a server deterministically ignores Range.
Fixes#6980
Before this change, when no_data_encryption was set, uploads from
local disk advertised the hash of the encrypted data even though the
data was uploaded unencrypted.
On backends which check upload hashes (eg b2) this made uploads of
small files fail with errors like "Checksum did not match data
received", and made chunked uploads store an incorrect hash so the
files failed their checksum on download with "corrupted on transfer:
SHA1 hashes differ".
See: https://forum.rclone.org/t/sha1-mismatches-on-b2-with-no-data-encryption-true/54121
Writing any file into a third-party app container - the Obsidian, Pages or
Shortcuts folders that iCloud Drive shows alongside your own - failed with
HTTP error 412 (412 Precondition Failed) returned body:
"{ ... \"error_code\" : \"VALIDATING_REFERENCE_ERROR\", \"reason\" :
\"Request has out of order children to be chained but the parents were
missing\" }"
Reading from those paths worked, and so did creating directories in them, so
the failure looked like a missing parent when the parent was plainly there.
Items in an app container live in a different zone from ordinary iCloud Drive
folders: a folder under Documents has a drive ID like
FOLDER::com.apple.CloudDocs::<uuid>, while the Obsidian container has
FOLDER::iCloud.md.obsidian::documents#o2v. DownloadFile already accounts for
this - it deconstructs the item's own ID and addresses the zone it finds - but
CreateUpload and UpdateFile hardcoded defaultZone, and UpdateFile built the
resulting Drivewsid with a hardcoded com.apple.CloudDocs as well.
So rclone asked Apple to chain the new document to a parent in
com.apple.CloudDocs while the parent lived in iCloud.md.obsidian. The parent
really was missing from the zone being addressed, which is what the error said.
Take the zone from the parent's drive ID instead, the same way the download
path does, and build the new item's ID with ConstructDriveID. Uploads outside
an app container are unaffected: their parents are in com.apple.CloudDocs, so
the derived zone is the value that was previously hardcoded.
Verified against a real remote: files now upload into an Obsidian vault inside
the container and read back correctly with an unpatched binary afterwards.
If the source supplied fewer bytes than its declared size, the upload
request failed but a retry could report success even though the stored
file was truncated, because the retry re-sent an already exhausted
reader.
Count the bytes actually read from the source and if they do not match
the declared size return an error.
This was found by the new FsPutShortEOF integration test.
If the source supplied fewer bytes than its declared size, the upload
request failed but a retry could report success even though the stored
file was truncated, because the retry re-sent an already exhausted
reader.
Count the bytes actually read from the source and if they do not match
the declared size, remove the partially uploaded file and return an
error.
This was found by the new FsPutShortEOF integration test.
If the source supplied fewer bytes than its declared size, the
multipart upload was completed anyway, storing a truncated file and
reporting a successful upload.
Check the number of bytes read from the source against the declared
size before finalising and abort the upload with an error if they do
not match.
This was found by the new FsPutShortEOF integration test.
If the source supplied fewer bytes than its declared size, the
truncated file was stored and the upload reported success with the
object claiming the declared size.
Count the bytes actually read from the source and if they do not match
the declared size, remove the truncated file and return an error.
This was found by the new TestRcatSizeShortEOF integration test.
The file is created at the declared size and the data then written
with ranged writes, so if the source supplied fewer bytes than
declared, the remainder of the file was left as zeroes and the upload
reported success.
Count the bytes actually read from the source and if they do not match
the declared size, delete the partially uploaded file (if newly
created) and return an error.
This was found by the new FsPutShortEOF and TestRcatSizeShortEOF
integration tests.
When using Microsoft Entra ID credentials, Azure Blob server-side copy
uses a user delegation SAS URL for the private copy source.
The SAS start time was set to the current local time. Azure Storage
validates the copy source from the service side, and small clock
differences can make that SAS appear not yet valid. The service then
returns 403 CannotVerifyCopySource with AuthenticationFailed.
Start the copy-source SAS 15 minutes in the past, matching Microsoft SAS
guidance for clock skew.
The precision field in the backend overview YAML files can hold
fs.ModTimeNotSupported (100 years in nanoseconds) which overflows int
on 32 bit platforms, making the YAML for those backends fail to parse
and causing rclone to log 18 internal errors on every invocation.
Use int64 for the precision field and add a test that parses every
embedded backend YAML file so this is caught on 32 bit test runs.
When an upload failed part way through with a retryable error (eg a
502 from the block storage servers) the pacer retried the whole upload
call with the same input stream. The stream had already been partially
consumed, so the retry re-created the upload draft and committed just
the remainder of the stream as a complete file, silently truncating
it. With restic over serve restic this corrupted the repository as the
truncated pack was reported as successfully uploaded.
This fixes it by using CallNoRetry for the upload, as the other
backends do, so retryable errors are returned wrapped in a RetryError
for the caller to retry the upload with a fresh stream.
When the source supplied fewer bytes than its declared size, the
compressed data file was stored under a name containing the declared
size while the metadata recorded the actual number of bytes read.
NewObject looks the data file up by the size in the metadata, so the
resulting object could never be read again, and the upload reported
success.
Check that the number of bytes read matches the declared size after
uploading the data and before writing the metadata, and remove the
data file and return an error if it does not.
This was found by the new FsPutShortEOF integration test.
The append loop retries everything once the upload session has
started, so a cancelled context error was retried through all the low
level retries with exponential backoff before the upload gave up.
A source which returned EOF before supplying as many bytes as it
declared would either commit a truncated file (if the shortfall was
within the final chunk) or loop forever appending empty chunks to the
upload session. Return an error wrapping io.ErrUnexpectedEOF instead.
Note that all dropbox uploads use the chunked upload path with the
default batch_mode of sync, so this affected uploads of every size.
Object.Update held its connection until the deferred putConnection ran at
function exit, so the SetModTime it does at the end of every upload had to take
a second connection from the pool, dialling a whole new SMB session when the
pool was empty. With N transfers in flight the pool grew to roughly 2N sessions
for no reason.
Return the connection as soon as the file is closed. At that point the upload
has succeeded and remove() can no longer be reached, so nothing else needs it,
and SetModTime picks the same connection straight back out of the pool.
putConnection nils the pointer, so the deferred putConnection becomes a no-op
and the connection is not returned twice.
purgeCheck listed a directory's children with includeAll set, so the Drive API
returned trashed children as well as live ones. The only thing that came of
that was the trashedFiles flag, which is used solely to decide whether to trash
the directory or delete it permanently, and when use_trash is on the directory
gets trashed either way.
That made the common case pay for nothing: after trashing N files in a
directory, removing the now empty directory paged through all N trashed entries,
about one API call and pacer delay per 1000, instead of running a single query
that returns nothing.
Ask the server to filter out trashed children when the directory is going to be
trashed anyway. Hard deletes still enumerate them, since there the distinction
decides whether the directory can be removed permanently (#1040), and
--drive-trashed-only is left alone because it deliberately lists trashed items.
Direct lookups of exported Dropbox Paper files retained the
caller-visible extension before export metadata processing appended it
again. Track when metadata was resolved through an export path so the
object keeps the requested remote name while listing behavior remains
unchanged.
The local backend built every OS path by joining the root with the source
name converted through the configured encoding, so the encoding was the only
thing keeping a name from turning into path syntax.
With an encoding which omits Dot (Slash, None, Raw) rclone's standard "．．"
decodes back to a real "..", and with an encoding which omits BackSlash a name
like "..\file" becomes a native path on Windows. filepath.Join then resolved
those out of the destination the user chose, so a source object called
"../marker.txt" - an s3 key of "tenant/../marker.txt" listed with the remote
rooted at "tenant", say - created or overwrote a file outside it.
localPath now joins the name to the root and checks with filepath.Rel that the
result is still inside it. localPath is the only place the root is joined to a
name, so threading the error through newObject and newDirectory covers every
operation.
Default configurations were not affected, as encoder.OS includes Dot on all
platforms and BackSlash on Windows.
Fixes GHSA-7p4m-qxvv-g567
The archive backend passes remote .sqfs bytes straight to the go-diskfs squashfs
parser, which does not validate several attacker-controlled superblock and
metadata fields. A crafted image can make the parser panic.
Recover panics at the go-diskfs boundary and return an "invalid or corrupt
squashfs image" error instead. As well as the parse entry points (Read, ReadDir,
OpenFile) this wraps the reader returned by Open, since the parser reads file
data lazily and can panic long after the image opened successfully.
Addresses GHSA-6jcg-q3wp-x2f4.
The FTP control channel is line oriented and the ftp library writes
command arguments (paths) straight onto it without escaping, so a
filename containing CR/LF can inject an independent FTP command.
This fix makes sure CR/LF are therefore always encoded to safe symbols
regardless of the configured encoding, which is what the default
encoding already does.
An audit of the shell escaping alongside the PowerShell fix found the
Windows Command Prompt path only rejected the double quote delimiter. A
remote filename containing % or ! can trigger environment variable expansion
even inside double quotes, and a newline or carriage return ends the
command, so a crafted filename could alter the command run during
server-side hashing. Reject paths containing these characters, matching
the existing treatment of the double quote.
PowerShell treats several Unicode smart-quote characters (U+2018, U+2019,
U+201A, U+201B) as single-quote delimiters in addition to the ASCII
apostrophe. The quoting helper only doubled the ASCII apostrophe, so a
remote filename containing one of these could close the quoted path and
inject statements that ran as the SSH account during server-side hashing.
Double all five delimiters when wrapping a PowerShell path so a filename
is always treated as data.
Fixes GHSA-2m8m-jhrm-w6j2
The AWS SDK signs S3 Express (directory bucket) requests with a session
token in the x-amz-s3session-token header. Go's HTTP client treats it as
an ordinary custom header and copies it when following a redirect to
another host, and it was missing from the list of secret headers the
redirect policy strips. Add it to the list.
The redirect tests derived their inputs from the production header list,
so a header accidentally dropped from that list would silently lose test
coverage rather than fail. The test list is now a deliberately literal
copy, kept in sync with the production list by a new test, so removing a
header from either list is a test failure. There is also a new
regression test verifying the Referer header that net/http generates
automatically - which for a presigned request carries the signed query
string - is not forwarded across hosts.
See GHSA-8mxv-9xhp-86h4
The CheckRedirect policy stripped only the X-Amz-Security-Token header when a
redirect crossed a host, leaving other origin-bound secrets exposed:
- Go removes Authorization on a hostname change but not on a same-host scheme
downgrade, so an IBM IAM bearer token was re-sent over plaintext HTTP.
- The SSE-C and copy-source SSE-C customer-key headers (which Go has no reason
to treat as sensitive) were forwarded across a redirect to another host.
- On a cross-host redirect net/http copies the previous request URL into the
Referer header; for a presigned request that URL carries the signature and
session token in its query string, disclosing them to the new origin.
This now refuses outright to follow any HTTPS->HTTP redirect - an S3 endpoint
has no legitimate reason to downgrade the transport, and this closes the
plaintext-exposure class regardless of which header or query string carries the
secret. On a cross-host redirect also strip the known credential headers and the
Referer.
The IDrive e2 region-lookup call posts the access key ID to a fixed HTTPS
endpoint but used a bare http.Client that would follow a redirect downgrading
to plaintext HTTP. Apply the same CheckRedirect policy as the main S3 client so
the request can't be replayed over an unencrypted connection.
Fixes GHSA-8mxv-9xhp-86h4
A server that redirects an HTTPS request to a plaintext HTTP URL on the
same host would cause Go's http.Client to replay the configured
credentials (Basic Authorization, cookies, secret headers) over the
network in cleartext.
Refuse to follow such downgrade redirects by default in lib/rest and wire
the webdav backend's client to use it. The `auth_redirect` option remains
the opt-in escape hatch for servers that legitimately need auth preserved
across redirects.
Fixes GHSA-h4mf-4v27-hggj
In this commit we attempted to wait for the success report of an
upload to fix the 500 error:
fe78b559d1 yandex: fix 500 errors by waiting for uploads to complete before setting modtime
However Yandex Disk finalizes an upload asynchronously on its servers.
Waiting for the upload operation to report success is not enough -
under load the server reports the operation as successful slightly
before the file is fully finalized, so setting the modification time
straight after an upload can still fail with 500 Internal Server
Error.
Yandex support recommend waiting 1.5s - 3s after the upload before
modifying the file's metadata, so add an --yandex-upload-wait option
(default off) to insert a delay between the upload completing and the
modification time being set.
The session close flag was only computed after each append, so a
known-size upload which fits in a single chunk sent all its data and
then issued a zero-payload append purely to close the session - one
wasted round trip per small file on the default batched upload path.
Set the close flag before the first append when the size is known to
fit in one chunk.
Since batch mode became the default all uploads go through
uploadChunked, which allocated a full chunk-size retry buffer (48 MiB
by default) regardless of the file size. With the `--transfers 32`
recommended for small file uploads that is ~1.5 GiB of buffer to
upload tiny files.
Size the buffer to the file size when it is known and smaller than a
chunk.
Before this change, if an object compressed with "Content-Encoding:
gzip" was downloaded, a length and hash mismatch would occur since the
go runtime automatically decompressed the object on download, giving
errors like
corrupted on transfer: sizes differ
This change sets "Accept-Encoding: gzip" on all requests which stops
the go runtime decompressing objects on download, so compressed
objects are downloaded as-is with intact size and hash information.
If --oos-decompress is set then rclone will decompress objects with
"Content-Encoding: gzip" as they are received, at the cost of not
being able to check the length or the hash of the downloaded object.
Fixes#9694
DirMove returned fs.ErrorDirExists for any error from the destination
existence check not just when the destination actually exists. That
made sync silently fall back to file-by-file moves and masked the real
failure.
Return the underlying error instead when the check fails for any other
reason.
DirMove checked whether the destination exists using the raw path but
performed the rename with the encoded path, so for directory names
needing encoding (trailing spaces or periods, characters like
\* ? : < > | " or a literal backslash) the existence check looked at
a different server path than the rename used.
If revealing the password, creating the Kerberos client or the SMB
handshake failed after the TCP connection was established, the
connection was never closed.
A new KerberosFactory was constructed for every dial, so the client,
error and ccache modification time caches it holds were discarded
after a single use. Every new SMB connection re-read the Kerberos
config, re-parsed the ccache and did a fresh KDC exchange.
Share a single factory so clients are cached across connections as
intended, and refreshed when the ccache file changes.
Object.Open dereferenced the response's ContentLength pointer without checking
it. The OCI SDK leaves ContentLength nil when the server replies without a
Content-Length header or ContentRange which caused a nil pointer panic.
Now the size is only updated when the response actually provides one, leaving
the size from the object metadata in place otherwise.
This also fixes the same potential problem in the newObject code.
The multipart upload complete endpoint returns 200 with an empty body,
but rclone tried to decode that body as JSON, failed with EOF and
retried until the retries ran out, so every upload failed even though
the server had actually completed it.
Fixed by not attempting to decode the response body.
This was a regression introduced in
a4972de505 shade: retry server errors instead of failing the transfer
which started treating the JSON decode error as fatal where previously
it was accidentally ignored.
Since around mid-2026 Apple's idmsa endpoints `POST
/verify/trusteddevice/securitycode` and `POST /verify/phone/securitycode`
return HTTP 409 (instead of 2xx) even when the submitted code is accepted:
the response body carries `"securityCode": {..., "valid": true}` and the
response headers include a fresh X-Apple-Session-Token, scnt and
X-Apple-Auth-Attributes, which are only issued on successful validation.
rclone treated any 409 as failure and aborted before TrustSession, so
configuring an iclouddrive remote always failed after the 2FA step with:
validate2FACode failed: HTTP error 409 (409 ) returned body:
"{... \"securityCode\": {\"code\": \"...\", \"valid\": true} ...}"
Treat a 409 response that carries X-Apple-Session-Token as success: absorb
the session headers and continue to TrustSession. Applies to both the
trusted-device and SMS validation paths.
Fixes#9488Closes#9534
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Add a new "list_parallelism" advanced option (requires use_arrow_list) which
splits a directory's blob-name keyspace into ranges and lists them
concurrently using the Arrow startFrom/endBefore range parameters. Both
recursive (ListR) and single-directory listings benefit.
The keyspace is sharded with a single-case (digit+lowercase) character
ladder. Single case is deliberate: the service validates startFrom /
endBefore case-insensitively (while listing case-sensitively), so a
mixed-case boundary pair is rejected with a 400; its endBefore filtering is
byte-ordered, so the listing is exact.
The service only honours endBefore on the Arrow listing path of accounts with
Blob Listing with Apache Arrow enabled: elsewhere it either rejects the
request with 400 OperationNotSupportedWithFeatureMissing or, if it fell back
to XML, would ignore the shards' upper bounds. Both cases are detected and
the parallel listing falls back to a sequential listing.
Listing 100,000 objects from a well-connected server:
config time vs XML
xml 23.9s 1.0x
use_arrow_list 6.9s 3.5x
list_parallelism=2 4.4s 5.5x
list_parallelism=4 2.8s 8.4x
list_parallelism=6 2.2s 10.9x
list_parallelism=10 1.5s 15.4x
list_parallelism=20 1.4s 17.6x
list_parallelism=30 1.1s 21.7x
So Apache Arrow listing alone is ~3.5x faster than XML, and parallel listing
takes that to ~20x. The ladder caps at 37 shards, so the benefit plateaus
beyond list_parallelism=30.
The option is hidden from the configurator, command line help and docs
until the feature is officially released.
Add a new "use_arrow_list" advanced option which fetches directory listings
using the ListBlobs Apache Arrow response format instead of XML. This can be
substantially faster for very large containers.
This is EXPERIMENTAL and requires the "Blob Listing with Apache Arrow" preview
feature to be enabled on the storage account. It is not supported on accounts
with a hierarchical namespace (ADLS Gen2). If the feature is not enabled the
service returns XML and the listing transparently falls back to it. Connection
string auth is not supported - it also falls back to the normal XML listing.
The option is hidden from the configurator, command line help and docs until the
feature is officially released.
This adds a temporary subpackage implementing the experimental "Blob
Listing with Apache Arrow" feature on top of the released Azure azblob
SDK.
The Azure SDK for Go supports Arrow listing on its unreleased
feature/storage/bifrost branch (commit c6fa341ca22b) but that can't be
used in a release. This package exposes the same options and pager
interface as the experimental SDK, built on the released SDK's public
API plus verbatim copies of its Arrow decoder and of the two internal
auth policies (shared key signing and storage bearer challenge) which
the released SDK does not export. Copied files carry MIT attribution
headers pinned to their upstream source.
When Arrow listing ships in a released azblob the whole package will be
deleted and callers pointed back at the SDK - the interface is
deliberately source compatible to make that a mechanical change.
Also exports auth.Transporter so the new package's pipeline can use
rclone's fshttp transport, and adds github.com/apache/arrow-go/v18 as a
direct dependency for decoding the Arrow IPC stream.
Rmdir checked that the directory existed with GetMetadata and then
checked it was empty with ListFolder, but ListFolder already reports a
missing path and a path that is a file, so the first request was
redundant.
Drop the GetMetadata call and map the ListFolder lookup errors onto the
same sentinel errors as before, so a missing directory still returns
fs.ErrorDirNotFound and a file still returns fs.ErrorIsFile. Limit is
set to 1 as only the presence of an entry matters, and HasMore is
checked as well so a full first page is not read as an empty directory.
Add Scality as an S3 provider covering both Scality RING (S3 Connector)
and ARTESCA, which share the same CloudServer + Vault S3 implementation.
The only quirk required is force_path_style: both products support
path-style addressing, and virtual-hosted style needs wildcard DNS that
on-prem deployments usually lack.
Nick Craig-Wood
MoraxandNick Craig-Wood
Christian De SantisandGitHub
Edward KleselandNick Craig-Wood
alliasgherandNick Craig-Wood
Loi NguyenandGitHub
debadityaandGitHub
acoeurandGitHub
SocialprankerandNick Craig-Wood
Dzmitry NianakhauandNick Craig-Wood