The code which opens the --combined, --differ etc report files (or
stdout for "-") and closes them afterwards was duplicated between the
check command and the sync logger flags. This moves it into
operations.OpenReportFiles which both now use. It also closes any
files already opened if a later one fails to open.
The sync report flags (--combined, --missing-on-src, --missing-on-dst,
--match, --differ, --error and --dest-after) were only wired up in the
CLI commands so there was no way to get these reports over the rc or
from librclone.
This adds boolean parameters of the same names as operations/check
(combined, missingOnSrc, missingOnDst, match, differ, error and
destAfter) to sync/sync, sync/copy and sync/move. Each requested
report is returned as an array of strings in the output, just as
operations/check does. All default to off so existing callers see no
change in the output.
To share the code between the CLI and the rc the report writer helper
from operations/check is exported as operations.RcReportWriter, the
lsf defaults for --dest-after are moved into
operations.NewSyncLoggerOpt and the listing setup and --no-traverse
warnings from operationsflags.ConfigureLoggers into LoggerOpt.Init.
Before this change `rclone dedupe --dedupe-mode rename` probed the
backend for each candidate `name-N.ext` in turn and gave up when it
had tried 100 names for a given object. With daily runs against the
same duplicated filename this ceiling was eventually reached and
rclone logged "Could not find an available new name". Each probe was
also a backend lookup, so a run against 99 existing names took
minutes on Google Drive.
The rename now uses the listing dedupe has already made to skip names
known to be taken without asking the backend, and only confirms the
final candidate with NewObject (the listing may be incomplete because
of filters). The suffix counter is shared between the objects being
renamed so no name is checked twice. The safety limit is raised to
10000 which, thanks to the listing, no longer costs a lookup per name.
The deferred cleanup in the constructor checked a local error variable
rather than the error being returned, so failures after the VFS was
created (such as an invalid --passive-port) never shut it down. Name
the error return so the cleanup sees the returned error.
When the HTTP server failed to initialise, for example because the
listen address was already in use, rclone panicked with a nil pointer
dereference instead of reporting the error.
The deferred cleanup in the constructor read the provider from the
named return value, but `return nil, err` sets that to nil before the
deferred function runs. Use a local variable for the server instead.
Writing to the mount with os.WriteFile made the Go runtime register
the file with its poller so the kernel then polled the file from
epoll_ctl and epoll_wait, sending POLL requests to the FUSE server
running in this same process. A thread waiting inside epoll cannot be
preempted by the runtime, so a garbage collection starting while such
a POLL was outstanding stopped the world for good - the test binary
could not be killed even with SIGKILL and the mount was left behind,
wedging anything that touched it.
Now we write through the mount with a descriptor straight from
open(2), which os.NewFile keeps out of the poller, check that it
really is out of the poller with SetDeadline, and check at the end of
the test that the mountpoint is unmounted.
In this commit we fixed the same problem for mount by running in a
subprocess however changing one write file routine here was much
easier than re-arranging the tests.
4a382c09ec mount: run tests in a subprocess to fix deadlock - #3259
Note that go-fuse (and hence mount2) works around this problem it by
forcing an early POLL it can answer with ENOSYS.
See: https://github.com/golang/go/issues/21014
The check that an entry returned by an archiver is a direct child of
the directory being listed normalised a parent of "/" to the root, so
an entry named "/x" passed as a child of the root while "x/" and
"dir//x" were rejected.
Decide by stripping the directory prefix and checking what is left
with sanitize.Leaf, which rejects an empty name, ".", ".." and any
name containing a "/". This also covers the leading slash case.
A file entry in a zip whose name refers to a directory, such as ".",
"/", "" or "sub/.", was only skipped when it named the root of an
archive which was itself the root of the remote. When the archive was
found by listing its parent directory the entry appeared as a file
with the same name as the archive alongside the directory for it, and
copying the archive tried to write both. When the entry named a
subdirectory it appeared as a file alongside that directory, and with
that subdirectory mounted as the archive root the entry was taken to
be the single file the root points at, hiding every real entry.
Skip any file entry whose last path component is "", "." or "..",
checked on the raw name before it is cleaned or joined on the prefix.
The zip archiver handed out its cached directory tree directly. Any
caller which filters a listing in place (as the core listing code
does) altered the cache, so later listings of the same directory could
be corrupted.
Return a copy of the cached listing instead.
When an upload failed with a 500 error the upload was retried with a
new upload link but the same input stream. The stream had already been
consumed by the first attempt so the retry uploaded an empty file.
This fixes it by returning a RetryError instead so the caller retries
the upload with a fresh stream, which will fetch a new upload link.
When an upload failed with a retryable error the pacer retried the
whole upload with the same input stream. The stream had already been
consumed by the first attempt so the retry uploaded an empty file.
This fixes it by using CallNoRetry for the upload, as the other
backends do, so retryable errors are returned wrapped in a RetryError
for the caller to retry the upload with a fresh stream.
It also makes 5xx errors from the upload storage servers retryable.
These come back from the SDK as a different error type to API errors
so were not being retried at all.
When an upload failed with a retryable error the pacer retried the
whole PUT with the same input stream. The stream had already been
consumed by the first attempt so the retry uploaded an empty file.
This fixes it by using CallNoRetry for the upload, as the other
backends do, so retryable errors are returned wrapped in a RetryError
for the caller to retry the upload with a fresh stream.
MkdirModTime checked --interactive/--dry-run itself and then called
MkdirMetadata or Mkdir which check again, so --interactive asked twice
about making the same directory and --dry-run skipped before the
operation could be shown in the progress display or counted as a
check.
Now MkdirModTime decides how to make the directory first and delegates
entirely to MkdirMetadata, or Mkdir followed by SetDirModTime, each of
which does its own --interactive/--dry-run check exactly once. This
also means the modtime setting fallback shows in the progress display,
respects --no-update-dir-modtime and has its errors counted.
When syncing to a backend which supports directory modtimes, a
directory which needed its modtime updated and which had files
transferred into it would get its modtime set twice - once when the
directory was checked and once in the pass at the end of the sync.
Now, when the end of sync pass is in use (which it is for all default
syncs), directories which need their modtime (or metadata) updating
are marked for that pass instead of being updated immediately. This
halves the number of directory modtime updates in a typical sync and
makes the "Updated dirs" stat count each directory once.
Syncs to backends which preserve directory modification times (eg
sftp, local) can update the modtime or metadata on many directories.
This count makes that work visible in the stats output, the core/stats
rc and the prometheus metrics (as dirs_updated_total).
Syncs which update lots of directories (eg to sftp) could spend a long
time setting directory modification times, making directories or
removing directories with no feedback in the --progress display or
stats, making rclone appear to have hung.
This shows directory operations (setting modtime, updating metadata,
making and removing directories) in the Checking section of the stats
and counts them as checks, in the same way file deletes are shown.
This creates a checking transfer which is shown in the progress
display while it is running but is not kept in the completed
transfers history, so it never appears in core/transferred and is not
retained in memory after it finishes.
This is for repeated bookkeeping operations (eg directory modtime
updates) which would otherwise crowd file transfers out of the
history.
The headers set with --header and --header-download are added to
every request by the rclone transport, including redirect hops which
net/http makes to other hosts, so a credential passed with --header
for one host could be sent to any host that server chose to redirect
to.
The transport now walks the redirect chain net/http records on each
redirected request and, once the chain has visited a host other than
the one originally requested, removes the headers rather than adding
them.
Also restore the global --client-cert and --client-key config after
TestCertificates so its temporary files are not used by later tests.
The headers set with --http-headers are documented for passing
credentials such as Authorization or Cookie. The backend used the
default net/http redirect policy which copies all but a handful of
well known headers to any redirect target, so a redirect from the
configured server to another host would send those credentials to
that host, and a redirect from https to http would send them in
plaintext.
When headers are configured this installs a CheckRedirect function
which:
- removes the configured headers from every hop once the redirect
chain has left the originally requested host
- refuses a redirect from https to http with an error
SameHost compares two URLs by host name (case insensitively) and port
(treating the scheme's default port as no port) so redirect policies
can tell a real change of host from a server spelling its own host
differently, e.g. redirecting "https://example.com/" to
"https://EXAMPLE.com:443/".
The HTTPS downgrade check now compares the redirect target against
the original request rather than the previous hop, so a chain which
started on plaintext http, passed through an https server and came
back to http is no longer refused - nothing is being downgraded
relative to what the user asked for. A chain which started on https
and reaches http via any number of hops is still refused.
Whether an archive entry name can escape the archive's namespace was
left entirely to each archiver. Enforce it in the archive backend too.
List only passes on direct children of the directory listed and
NewObject only returns the object asked for, so a future archiver
which forgets to validate names cannot expose a traversal to fs/sync
and fs/operations.
The path inside the archive was compared against the cleaned entry
names without being cleaned itself, so `archive.zip/sub/./dir` or
`archive.zip/sub//dir` failed to list even though `archive.zip/sub/dir`
worked.
A zip containing a file entry whose name refers to the archive's own
root (".", "/" or "") was presented as a single file called "." and
all its other entries disappeared. A file at the root can only be the
archive member the backend was pointed at, so with no root such an
entry is skipped like any other unsafe name.
Entry names read from a squashfs directory are not sanitized by
go-diskfs. The squashfs backend joined each leaf name onto its
directory to form the object's remote, so a crafted image could escape
its directory.
Use sanitize.Leaf to skip unsafe entries in List. A "\" is an
ordinary character in a file name on the systems squashfs images are
made on and in an rclone remote path, so it is deliberately not
rejected; making it safe for the destination is the destination
backend's job.
Skipped entries are logged at DEBUG with a single NOTICE count per
listing so a crafted image under a mount cannot flood the log.
When a zip archive was mounted at a subdirectory root, readZip used a bare
strings.HasPrefix to decide which entries fell inside the root. This
matched on a raw string prefix rather than a path boundary, so mounting
root "foo" also exposed sibling entries such as "foobar/..." with their
names left uncorrected.
Require a path boundary when filtering by root.
The zip backend mounts a zip file as a browsable Fs. Go's archive/zip
does not sanitize entry names, and readZip applied path.Clean but did
not reject a cleaned name that still pointed outside the archive. A
crafted zip could make rclone copy/sync attempt writes outside the
intended destination.
Sanitize entry names with sanitize.Path - the same check used by
rclone archive extract - skipping any entry with a ".." path
component, whether separated by "/" or "\". A backslash is otherwise
kept as an ordinary character in the name, as archive extract does. It
is up to the destination backend to make names safe for its storage.
Skipped entries are logged as a single count per archive so a crafted
archive with many escaping entries cannot flood the log.
Move the archive entry name validation added for CVE-2026-59732 from
cmd/archive/extract into a new lib/sanitize package as sanitize.Path,
so the same check can be shared with the archive backend which mounts
archives as a filesystem.
sanitize.Path keeps the extract semantics - reject any name with a
".." path component, treating both "/" and "\" as separators - and
additionally cleans the name with path.Clean. This corrects two edge
cases in extract: a repeated "./" prefix ("././file.txt") is now fully
stripped rather than only the first, and a bare "." entry is now
treated as the archive root and skipped.
Add sanitize.Leaf, which rejects a name that is empty, ".", ".." or
contains a "/", for checking a single directory entry name read from
an archive.
The names handled are rclone remote paths, in which "/" is the only
separator and "\" an ordinary character, so Leaf does not reject a
backslash: making a name safe for its storage is the destination
backend's job (the local backend encodes "\" on Windows and refuses
paths which escape its root). Path's rejection of ".." between
backslashes is kept as defence in depth for extract.
With --links/-l, a symlink is served as a .rclonelink object whose
content is the target path. A Range request with a start offset beyond
the target length (e.g. "Range: bytes=99999999999-") reached
openTranslatedLink and sliced the target string at that offset, panicking
with "slice bounds out of range".
Clamp the offset to the target length so an out-of-range start reads
empty, matching how a real file read past EOF behaves.
The birth-time (btime) write in writeMetadataToFile followed symlinks for
any object that was not a translated link, so under -l/--links a symlink
planted by an untrusted source at the destination path could redirect the
btime write to a target outside the backup destination on OSes where
birth time is settable (Windows).
Use the NOFOLLOW birth-time write whenever translating symlinks, not only
for translated links. It is a no-op on a real file or directory and stops
a planted symlink from being followed out of the destination.
With -l/--links the local backend faithfully recreates a source ".rclonelink" as
a real symlink at the destination. Directory metadata (chmod/chown/chtimes),
however, was applied with the raw following syscalls
os.Chmod/os.Chown/os.Chtimes rather than through the os.Root sandbox used for
content writes. A Directory is never a translatedLink, so when the destination
path already existed as a symlink planted by an untrusted source, the metadata
was applied through it to a target outside the backup destination.
Route directory metadata through os.Root when translating symlinks, so a planted
symlink can no longer redirect chmod/chown/chtimes out of the destination, while
legitimate in-tree directories are unaffected.
An empty or "." volume name joined onto the base directory resolves to the
base directory itself. newVolume does not call validate, so such a name
would mount a remote over the base directory and shadow every other
volume's mountpoint.
Require the resolved mountpoint to be a strict descendant of the base
directory so these degenerate names are refused.
When the plugin restarts it reads its persisted state file and used the
stored mountpoint verbatim. A state file written by an older rclone that
allowed escaping volume names, or one that was tampered with, could point
the mountpoint outside the base directory, so upgrading did not remediate
an already-escaped volume.
Re-derive the mountpoint from the base directory and the volume name on
restore, confined to the base directory, rather than trusting the stored
path.
A Docker VolumeDriver.Create request carries a raw volume name that was
joined onto the base directory with filepath.Join and used verbatim as the
mountpoint. filepath.Join collapses ".." components, so a crafted name such
as "../../../etc/foo" resolved to a host path outside the base directory,
where the plugin then created a directory and mounted the remote.
Confine the mountpoint to the base directory and refuse any name that
resolves outside it.
When serving FTP with --auth-proxy, the obscured password was cached in a
driver-global map keyed only by the username. Two sessions that logged in
with the same username but different credentials shared one map entry, so a
later login overwrote it and every subsequent operation on the earlier,
still-authenticated session was re-authorized with the later session's
credential and executed against the later session's backend.
Bind the credential to the FTP session by storing the obscured password in
the per-session goftp Session.Data map instead, so each session always
resolves the backend it authenticated for.
With --auth-proxy set and --auth-key unset, serve s3 registered every client
supplied access key ID with an empty secret and verified the SigV4 signature
against that, so anyone could sign a request for an arbitrary access key ID with
an empty secret and be let in. The proxy program was only ever given the access
key ID (as both user and pass) so it had nothing with which to authenticate the
client either.
An S3 client never sends its secret, only a signature made with it, so the
server has to know the secret to check the request. The auth proxy protocol as
been changed to handle this. For serve s3 the proxy program is given just the
access key ID as the user (no pass or public_key) and must return the matching
secret as _secret_access_key in its output. rclone verifies the request's
signature against that secret, refusing the request if the proxy rejects the
access key ID, doesn't return a secret or returns an empty one, or the signature
doesn't match. The secret is only used for this server's own verification and is
never registered with gofakes3, so other serve s3 instances in the same process
don't honour it.
The proxy's answers are cached. If a signature fails against a cached secret the
proxy is consulted again so a rotated secret takes effect immediately - but only
for a signature mismatch, and at most once every 10 seconds per access key ID
and client IP, so a stream of bad signatures can't make the proxy program run
for every request. A rotation never shuts down the cached backend under requests
still using it. A cached answer is checked with the proxy again once it is 5
minutes old even if in constant use, so revoking an access key ID takes effect
within 5 minutes.
This means --auth-key is no longer needed with --auth-proxy: it is ignored and a
warning is given at startup if both are set. The proxy is the source of truth
for both the credentials and the backend they map to. Presigned URLs (credential
in the query string) are now recognised by the proxy middleware too. The auth
proxy docs are added to serve s3.
Note that the serve s3 auth proxy protocol has changed. The proxy program is now
given the access key ID as "user" (it was previously given an MD5 hash of it,
with the access key ID as "pass") and must return the matching secret as
"_secret_access_key".
This needs gofakes3 v0.0.9 for signature.V4SignVerifyWithSecret.
gofakes3 kept the keys given with --auth-key in a store global to the process,
so when more than one serve s3 was running in one rclone (eg started via the rc)
each accepted the others' credentials and a client with the key for one server
could read and write the backend of another.
This updates gofakes3 to v0.0.9 which keeps auth keys per instance and adds a
test that two servers only accept their own keys.
From v1.70.0, an SFTP server started through the rc serve/start API with
a per-server proxyOpt.AuthProxy decided whether to enable proxy
authentication by checking the process-global proxy.Opt.AuthProxy
instead of the supplied proxyOpt.AuthProxy. In the normal rc case the
global is empty, so the auth proxy was silently ignored: the server
either failed to start with "no authorization found" or authenticated
against the local authorized_keys file instead of routing each login
through the proxy the operator configured.
The serve Provider refactor (f425f8d46) fixed the constructor by building the
provider from the supplied proxyOpt, but the authorized-keys handling in
configure() still consulted the global option. Make it depend on whether
proxy mode is actually active, and add a regression test for the
per-server option.
From v1.70.0 until the serve Provider refactor (f425f8d46), an S3 server started
through the rc serve/start API with a per-server proxyOpt.AuthProxy
decided whether to enable proxy authentication by checking the
process-global proxy.Opt.AuthProxy instead of the supplied
proxyOpt.AuthProxy. In the normal rc case the global is empty, so the
auth proxy was silently ignored and the server served the fixed
filesystem supplied to serve/start rather than routing each access key
to the backend chosen by the proxy, bypassing the operator's intended
per-key authorization.
The Provider refactor fixed this incidentally by building the provider
from the proxyOpt passed to the constructor. This adds a regression test
so the per-server option cannot silently stop working again, and only
logs "allowing anonymous access" when neither an auth key nor an auth
proxy is configured so the log reflects the effective mode.
From v1.70.0 until the serve Provider refactor (f425f8d46), an FTP server started
through the rc serve/start API with a per-server proxyOpt.AuthProxy
decided whether to enable proxy authentication by checking the
process-global proxy.Opt.AuthProxy instead of the supplied
proxyOpt.AuthProxy. In the normal rc case the global is empty, so the
auth proxy was silently ignored and the server fell back to its
fixed-backend mode, whose default account accepts user "anonymous" with
any password - a complete authentication bypass.
The Provider refactor fixed this incidentally by building the provider
from the proxyOpt passed to the constructor. This adds a regression test
so the per-server option cannot silently stop working again.
The multipart reorder-buffer admission trusted the client-declared part length.
A negative length was accepted, and `buffered + size` could overflow int64 for
a huge declared length, wrapping the running total negative and admitting
further parts past --multipart-streaming-buffer-limit.
Reject a negative length and use the overflow-safe comparison `size <=
bufferLimit - buffered` so an untrusted Content-Length can neither poison nor
overflow the budget.
Streamed multipart UploadPart called Reserve(contentLength) before reading any
body bytes, so the pool immediately allocated one 1 MiB page per MiB of the
client-declared Content-Length (or X-Amz-Decoded-Content-Length). An client
could declare a huge part size, send no body, and force an arbitrarily large
allocation without paying the bandwidth cost of the declared body.
Drop the Reserve so the pool-backed buffer grows a page at a time as the body
is actually read: memory now tracks the bytes received, not the unverified
header.
The rclone core does not sanitise ".." in an object's Remote(). Such a name can
arrive from a malicious or buggy backend - an object store permits keys
containing ".." or a leading "/" - and, if acted on, lets a listing or transfer
escape the configured root. A source object named "../../other/x" is copied to
"other/x" outside the destination root, and a crafted listing name surfaces
outside the directory being listed.
Add list.RemoteEscapesRoot, which reports whether a Remote climbs above the
root when joined onto it, and list.RemoveEscaping, which drops and logs such
entries.
Apply RemoveEscaping unconditionally - independent of the include/exclude
filters - at the three per-entry filtering points every listing passes through:
filterDir, walk.listR and walk.walkRDirTree (recursive ListR).
operations.StatJSON calls List and NewObject directly, bypassing those, so it
rejects an escaping remote up front.
This confines every backend at once, so no per-backend change is needed.
CVE-2026-78662: a malicious peer could flood an undecided channel's
incoming requests, deadlocking the whole connection in
golang.org/x/crypto/ssh (GO-2026-6354)
CVE-2026-56855: a malicious peer could send crafted messages on an
established channel, deadlocking the whole connection in
golang.org/x/crypto/ssh (GO-2026-6355)
When bisync is interrupted with a graceful shutdown it keeps the files
which transferred successfully in its listings and rolls the rest back.
An operator precedence mistake in that check meant a transfer of an
empty file (or one of unknown size) was kept even when it had failed,
so bisync recorded it as synced when it had not been.
Single part uploads with Object Lock parameters need a Content-MD5
header, which the SDK can't compute from a stream, so the whole body
was read into memory with io.ReadAll to hash it - up to
--s3-upload-cutoff per file. prepareUpload already sets Content-MD5
from the source object's hash when it has one, so skip the buffering
entirely in that case and only buffer when the hash is unavailable.
When buffering is needed, read the body into a multipart.NewRW buffer
from the global pool, hashing in transit, so the memory is reused
across uploads and released after the request. The presigned request
path hands the body straight to http.NewRequest, so wrap it in
readers.NoCloser there to stop the transport closing the pooled buffer.
With speedup enabled, files up to --mailru-speedup-max-memory are read
into memory so their hash can be tried against the server before
uploading. This used io.ReadAll, which allocates a fresh heap slice per
file and grows it by doubling, so with the default 32 MiB limit and
several transfers this churned a lot of garbage outside rclone's memory
accounting.
Buffer the file with multipart.NewRW instead, hashing it in transit,
so the memory comes from the global pool and is reused.
When the hash isn't known to the server the buffered file is uploaded
from the same buffer. Previously a low level retry of that upload
resent an already drained reader, so the retry always failed. Rewind
seekable bodies at the start of each attempt so retries resend the
whole file. Add a test which drops the connection on the first attempt
and checks the retried body is complete.
The body is sent through lib/rest, which wraps it in readers.NoCloser,
so the transport can't close the pooled buffer early; Update closes it
when it returns.
The Linkbox API needs the MD5 of the first 10 MiB of each uploaded
file, so Update reads that prefix into memory before the upload. This
used io.ReadAll, which allocates a fresh heap slice per file and grows
it by doubling, churning well over 10 MiB of garbage per upload.
Read the prefix into a multipart.NewRW buffer instead so the memory
comes from rclone's global pool and is reused across uploads, and hash
it in transit rather than computing the same MD5 twice.
The PUT body goes through lib/rest, which stops the http transport
closing it, so Update owns the buffer and closes it on every exit path.
When the source has no MD5, Update reads the whole file into memory to
hash it before uploading if it is under --jottacloud-md5-memory-limit.
This used io.ReadAll, which grows a fresh heap slice per file (up to
10 MiB by default, roughly doubled by the growth strategy), so syncs
of many files churned allocations and GC.
Buffer the data with multipart.NewRW instead so the memory comes from
rclone's global pool, is reused across uploads and is released by the
existing cleanup function.
Unknown sized streams previously took the in-memory branch regardless
of the limit, so an rcat of an arbitrarily large stream could read it
all into memory. Spool those to the temporary file instead, as is
already done for files over the limit.
The buffered body is sent through lib/rest, which wraps request bodies
in readers.NoCloser, so the transport can't close the pooled buffer
early.
The multipart upload allocated a fresh chunk-sized buffer (64 MiB by
default) plus a 1 MiB scratch buffer per large file, copying every byte
twice, and never returned them to rclone's memory pool.
Buffer each part with multipart.NewRW instead so the memory is reused
across uploads and part of rclone's central memory management.
The pooled buffer is seekable, so a part can now be re-sent.
uploadPart previously had no retry at all and any transient error
failed the whole upload. It is now wrapped in the pacer with the
backend's usual shouldRetry rules, seeking to the start before each
attempt. The body is wrapped in readers.NoCloser so the http transport
can't close the pooled buffer between attempts, and Content-Length is
set explicitly since net/http can't infer it from a pool.RW.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
Each part of a multipart upload allocated a fresh part-sized buffer
(the size is chosen by Box, typically 8-32 MiB) with up to --transfers
parts in flight, so large uploads churned allocations and GC.
Buffer parts with multipart.NewRW instead so the memory comes from
rclone's global pool, is reused across parts and files, and is part of
rclone's central memory management.
The pool.RW is seekable so the retry closure seeks back to the start
before each attempt instead of rebuilding a bytes.Reader, and the
per-part SHA1 digest is computed by reading the buffer and seeking
back. The body goes through lib/rest which already stops the transport
from closing it; the uploading goroutine owns and closes the buffer.
The whole-file SHA1 used for the commit is unchanged.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
Each upload allocated a fresh chunk-sized buffer (10 MiB by default)
regardless of the file size, so bulk transfers of many files churned
allocations and GC.
Buffer chunks with multipart.NewRW instead so chunk memory is reused
across uploads and is part of rclone's central memory management. The
pool.RW is seekable, so the existing rewind on retry carries over.
The body goes through lib/rest which already wraps it so the transport
can't close the pool buffer. The upload loop closes it after every
chunk, on error paths included. A source which ends before the
declared size is now reported as a short read before the chunk is sent
rather than as an incomplete write afterwards.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
Each chunked upload allocated a fresh chunk-sized buffer (48 MiB by
default), so bulk transfers of many files churned allocations and GC.
Buffer chunks with multipart.NewRW instead so chunk memory is reused
across uploads and is part of rclone's central memory management. The
pool.RW is seekable, so the existing IncorrectOffset recovery which
skips already-received bytes on retry carries over unchanged, and the
"chunk received OK" check now compares against the bytes actually
buffered so a short final chunk is recognised too.
The Dropbox SDK wraps the request body in io.NopCloser, so the transport
never closes the pool buffer. The upload loop closes it after every
chunk, on error paths included.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
Each upload allocated a fresh 48 MiB chunk buffer, so bulk transfers of
many files churned allocations and GC, and small files paid for the full
buffer.
Buffer chunks with multipart.NewRW instead so chunk memory is reused
across uploads and is part of rclone's central memory management.
The pool.RW implements io.Closer, so the PATCH request body is wrapped in
readers.NoCloser to stop the http transport closing it after a failed
attempt and freeing its pages before the retry. The retry closure now
seeks the chunk back to the start explicitly before resending, a short
read of the source is reported as an error rather than sent as an
under-length chunk, and the request carries an explicit ContentLength.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
The compressibility heuristic compressed a 1 MiB sample of each upload
into a bytes.Buffer only to read its length, growing up to ~1 MiB of
garbage per file. Write the sample through a counting io.Discard-style
writer instead so no output buffer is allocated at all.
Every compressed upload allocated a fresh buffer the size of
--compress-ram-cache-limit (20 MiB by default) regardless of how big
the file actually was, so uploading many small files churned large
allocations and the memory sat outside rclone's pool accounting.
Read the head of the stream into a multipart.NewRW instead, which takes
pages from the global pool only for the bytes actually read and returns
them when the upload finishes. The pool.RW is seekable, so a wrapped
backend which needs to retry a small upload can rewind the body, which
the previous bytes.Buffer did not allow. A read error while filling the
cache is returned rather than falling through to the streaming path.
Add a unit test covering the buffered, streamed and spooled paths which
checks the body handed to the wrapped remote, that it can be re-read
for a retry, and that the pool pages are returned.
Each resumable upload allocated a fresh chunk-sized buffer (8 MiB by
default, up to 64 MiB), so bulk transfers of large files churned
allocations and GC.
Buffer each chunk in a pool.RW from the global page pool instead so
the memory is reused across uploads and bounded by rclone's central
memory management.
The pool.RW is seekable, so the chunk is rewound at the start of each
retry rather than re-wrapped. lib/rest wraps request bodies in
readers.NoCloser so the transport cannot return the pages to the pool
between attempts - the Content-Length is set through rest.Opts because
the wrapped body is not a *bytes.Reader net/http can measure. A source
that runs dry before its declared size is reported as an unexpected EOF
rather than sending the short chunk.
Files below upload_cutoff (up to 20 MiB) were assembled into a bytes.Buffer
which grows by doubling, so each upload allocated roughly twice its size
and threw it away afterwards, churning the GC on bulk transfers. Write the
multipart/related body into a pool.RW from the global page pool instead so
the memory is reused across uploads and bounded by rclone's memory
management.
The pool.RW is seekable, so the same body is rewound at the start of each
retry rather than being re-wrapped. lib/rest already wraps request bodies
in readers.NoCloser so the transport cannot free the pages between
attempts; the Content-Length is passed explicitly because the wrapped body
is no longer a *bytes.Reader net/http can measure.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
Updating a file below --hidrive-upload-cutoff copied the whole file into
an append-grown slice so the request could be retried, costing roughly
twice the file size in transient allocations for every such update.
Buffer the file in a multipart.NewRW from rclone's global page pool
instead and return it to the pool once the request has finished. The
pool.RW is seekable so retries re-send the same buffer. Accounting is
applied as the buffer is sent so bandwidth limits and progress still
track the upload. The request now carries an explicit Content-Length
rather than being sent chunked.
The upload is bounded by the size the source declares - a source that
delivers more bytes than its declared size has the excess ignored, where
previously the stream was sent to EOF.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
Each chunk of a chunked upload allocated a fresh buffer of
--hidrive-chunk-size bytes (48 MiB by default), with up to
--hidrive-upload-concurrency of them in flight, so large uploads churned
allocations and GC and a short final chunk still cost a whole chunk.
Buffer chunks with multipart.NewRW from rclone's global page pool
instead, sized to the data actually read, and return each buffer to the
pool once its PATCH request has finished. The pool.RW is seekable, so a
chunk which fails with a retryable error is re-sent from the same
buffer. Accounting is applied as a chunk is sent so bandwidth limits
and progress still track the upload. Chunk requests now carry an
explicit Content-Length rather than being sent chunked.
The prefix sent with the creating request in PutUnchecked is buffered
through the same helper.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
Every upload allocated a fresh buffer of --hidrive-upload-cutoff bytes
(96 MiB by default) to hold the part of the file sent with the creating
request, however small the file was, so copying many small files churned
large allocations and GC.
Buffer that prefix in a multipart.NewRW from rclone's global page pool
instead, sized to the smaller of the declared file size and the cutoff,
and return it to the pool once the file has been created. Accounting is
applied as the buffer is sent so bandwidth limits and progress still
track the upload. The request now carries an explicit Content-Length
rather than being sent chunked.
The upload is bounded by the size the source declares - a source that
delivers more bytes than its declared size has the excess ignored,
where previously they were read up to the cutoff.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
Upload chunks are buffered into a bytes.Reader and then, when transfer
accounting is active (always for a real copy), re-wrapped in accounting
before being handed to cachedReader. cachedReader only recognised a bare
*bytes.Reader, so the accounted chunk fell through to
readers.NewRepeatableReader, which copied the whole chunk again into an
append-grown slice. Every chunk (and the upload-cutoff prefix of every
file) therefore cost roughly twice its size in memory.
Look through the accounting wrapper when deciding whether the reader is
already a seekable buffer and, if so, seek the buffer underneath while
still reading through the accounting, so retries rewind without a copy.
Each chunk of a resumable upload was buffered in a fresh RepeatableReader
which grew by appending, so bulk transfers churned up to a chunk size (10
MiB by default) of heap per chunk and GC pressure.
Buffer chunks instead with multipart.NewRW from the global page pool
instead so memory is reused across uploads and bounded by rclone's
central memory management.
A source which delivers fewer bytes than its declared size now fails
before the chunk is sent with an unexpected EOF error rather than being
rejected by the transport.
Note that accounting now happens as the chunk is read into the buffer
rather than as it is sent, as in the drive backend.
When the source can't be re-opened, the whole input is read once to
compute its gcid before upload and held back for the upload proper. For
inputs at or below --pikpak-hash-memory-limit this used a bytes.Buffer,
a fresh heap allocation of up to the limit (and more while growing) per
file.
Hold the data in a buffer from the global memory pool instead so the
pages are reused and released on cleanup.
Inputs of unknown size were also always held in memory regardless of
their length, as only sizes above the limit chose the temp file.
Spool unknown sizes to the temp file so a large stream can't exhaust
memory.
The multipart uploader kept its own private buffer pool, a copy of the
one in lib/pool with identical settings, so its chunk memory was never
shared with the rest of rclone. Pages cached here were invisible to
other backends and vice versa, costing up to 64 MiB of extra idle cache.
Allocate chunks with multipart.NewRW, the global pool used by the
other backends instead.
WriteChunk copied the whole chunk, which lib/multipart already hands over
in a buffer from the global memory pool, into a bytes.Buffer so that
retries could re-send it. That doubled the per-part memory and made a
fresh chunk-sized heap allocation (64 MiB by default) for every part,
times the upload concurrency.
The chunk reader is seekable, so find its size with Seek and rewind it
inside the pacer closure instead, sending the pooled buffer directly.
Also fix the error for a part that fails to upload, which formatted the
buffer instead of the part number.
Each upload chunk is buffered in a pool.RW from the global memory pool
but was never closed, so its pages were never returned to the pool.
Close the buffer after each chunk is uploaded and on the read error
path.
A chunk that failed with a retryable error was also retried without
rewinding the buffer, so the retry sent an empty body with the original
Content-Length and Content-Range and failed.
Seek the chunk back to the start inside the pacer closure so each
attempt re-sends it in full.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
Before this change the transport could still be reading the request
body when Call returned, in two ways:
- the goroutine writing a multipart upload form was never stopped or
waited for, so after a failed request it kept reading the file body
while the caller retried
- the transport itself is documented to close the request body,
possibly in a different goroutine after the request has finished,
and hiding the body's Close method removed any way to wait for that
Both of these raced with callers which seek to the start of a pooled
buffer and retry the request, corrupting the retried body, and the
abandoned multipart goroutine could read pooled pages after they had
been returned to the pool.
Call now waits for the transport to finish with the request body
before returning, and the multipart form writer is stopped and waited
for once the request has finished.
Add an FsPutRetry integration test which uses the fshttp fault injector
to fail the first request carrying upload data, before it is sent, with
either a synthetic 500 or a connection reset, and then checks that Put
either succeeds with the correct contents or fails cleanly without
leaving a corrupt object behind.
It runs a single part upload and, for backends implementing
SetUploadChunkSizer, a chunked one. This catches backends which retry
an upload from a buffer which has already been consumed or returned to
the memory pool. Backends which don't use HTTP skip the test.
Also check in FsPutRetry and the large upload tests that the number of
buffers in use in the global memory pool is unchanged after an upload,
to catch backends which leak pooled buffers.
Add fshttp.SetFaultInjector, which installs a function consulted by
every Transport before a request is sent. The injector can synthesise
an error status code or a transport error for chosen requests. The
request body is drained and closed as a real round trip would, but
nothing reaches the server.
This lets the integration tests check that backends cope with a
transient failure part way through an upload - in particular that a
retry re-sends the same data rather than an already consumed or
freed buffer - without needing a fake server for each backend.
NoCloserNotify hides the Close method of the reader passed in like
NoCloser, but calls a notify function (once only) when the returned
body is closed. This lets callers of http.NewRequest find out when the
transport has finished with a request body, as it is documented to
possibly close it in a different goroutine after the request has
finished.
Account.WriteTo wrote each buffer to the destination in full before
trimming the byte count for --max-transfer, so up to one buffer past
the limit could reach the wire and go unaccounted. This matters now
that NoCloser forwards WriteTo.
Truncate the write to the remaining allowance before writing.
NoCloser hides the Close method of an io.Reader but in doing so it
also hid io.WriterTo if the underlying reader implemented it, forcing
io.Copy to fall back to a buffered Read loop.
Following io.NopCloser, return a variant which forwards WriteTo when
the wrapped reader supports it.
*pool.RW implements io.Closer, so when one is used directly as an
http.Request body the transport closes it after each attempt. A retry
which then seeks and re-reads the RW would index the freed pages and
panic with "index out of range" in readPage.
Read, Write, ReadFrom, WriteTo and Seek now return ErrClosed (which
wraps io/fs.ErrClosed) after Close, and Close is safe to call more
than once.
Directory names which look like they have a --b2-versions version
string are now encrypted in full, so directories created by older
rclone (which left the version string in plain text) no longer
decrypt and vanished silently from listings.
DecryptDirName now falls back to the old form for such names so the
directory is listed, and logs the name it needs to be renamed to on
the underlying remote to make it accessible again. Document this in
the crypt docs.
Pikpak never returns MD5 for uploads which causes this test to fail.
Perhaps Pikpak should not declare MD5 but that is a bigger decision
being discussed in #9826
Nextcloud only stores a checksum which is supplied in the OC-Checksum
header of an upload, and discards it again when the modification time
is set with PROPPATCH. Re-sending the checksum in the PROPPATCH (as is
done for ownCloud) is rejected by Nextcloud with 403 Forbidden which
made the whole PROPPATCH fail, so SetModTime returned an error on any
object which had a hash. Uploads from sources without hashes, eg
streamed uploads with `rclone rcat`, were stored with no hash at all.
Use the Nextcloud PATCH extension with the X-Recalculate-Hash header
to have the server calculate and store the SHA1 of an object after a
streamed upload and after setting the modification time. This gives
a server side hash of the stored data which also lets rclone verify
streamed uploads.
If the source supplied fewer bytes than its declared size, the single
part upload path accepted the short body and stored a truncated file
recorded with the declared size, reporting a successful upload. The
multipart path already checks for this.
Count the bytes actually read and fail the upload if they do not match
the declared size, which cancels the partially created file.
This was found by the FsPutShortEOF integration test.
Nick Craig-Wood