Refuse --delta-list without --check-access. Abort if a listing goes empty
or shrinks past --max-delete versus the prior snapshot (full relist and
local walk included). Persist cursor before listing so a crash cannot
pair a new listing with a stale cursor. Reconstruct delta remotes from
path_lower plus leaf casing; ListR probes a non-recursive list when the
recursive result is empty.
Skip the per-directory Dropbox walk on successive bisync runs by applying
list_folder/continue deltas to the stored listing. Cursor reset, --max-delete,
and atomic listing+cursor writes abort without a half-applied snapshot. The
local side still walks; there is no NTFS USN dependency.
The test asks InstallUpdate to install the latest beta into an
unwritable file and expects an error. When the binary under test
reports exactly the latest beta version (as make quicktest does right
after a beta is published from the same commit), InstallUpdate
correctly decides there is nothing to do and returns nil, and the test
then dereferences the nil error and panics.
Pin fs.Version to a fixed old value for the duration of the test so an
update is always attempted, and use require.Error so a missing error
fails the test instead of crashing it.
applyOptions consumes the "path" option into vol.Path rather than leaving
it in vol.Options, but restoreState rebuilt the options with only fs and
type. The explicit path was therefore dropped when the plugin restarted,
and since fsString is rebuilt from those options the volume was remounted
at the root of the remote instead of at its subpath.
Before this change a volume created with type + path lost its path
completely, and one created with remote + path silently fell back to the
path of the connection string. With a backend whose credentials are
scoped to the subpath the restored mount then failed every operation
rather than serving the wrong directory.
Feed the persisted path back like fs and type, so applyOptions applies
the same precedence on restore that it applies when the volume is
first created.
Fixes#9853
The code which opens the --combined, --differ etc report files (or
stdout for "-") and closes them afterwards was duplicated between the
check command and the sync logger flags. This moves it into
operations.OpenReportFiles which both now use. It also closes any
files already opened if a later one fails to open.
The deferred cleanup in the constructor checked a local error variable
rather than the error being returned, so failures after the VFS was
created (such as an invalid --passive-port) never shut it down. Name
the error return so the cleanup sees the returned error.
When the HTTP server failed to initialise, for example because the
listen address was already in use, rclone panicked with a nil pointer
dereference instead of reporting the error.
The deferred cleanup in the constructor read the provider from the
named return value, but `return nil, err` sets that to nil before the
deferred function runs. Use a local variable for the server instead.
Writing to the mount with os.WriteFile made the Go runtime register
the file with its poller so the kernel then polled the file from
epoll_ctl and epoll_wait, sending POLL requests to the FUSE server
running in this same process. A thread waiting inside epoll cannot be
preempted by the runtime, so a garbage collection starting while such
a POLL was outstanding stopped the world for good - the test binary
could not be killed even with SIGKILL and the mount was left behind,
wedging anything that touched it.
Now we write through the mount with a descriptor straight from
open(2), which os.NewFile keeps out of the poller, check that it
really is out of the poller with SetDeadline, and check at the end of
the test that the mountpoint is unmounted.
In this commit we fixed the same problem for mount by running in a
subprocess however changing one write file routine here was much
easier than re-arranging the tests.
4a382c09ec mount: run tests in a subprocess to fix deadlock - #3259
Note that go-fuse (and hence mount2) works around this problem it by
forcing an early POLL it can answer with ENOSYS.
See: https://github.com/golang/go/issues/21014
Move the archive entry name validation added for CVE-2026-59732 from
cmd/archive/extract into a new lib/sanitize package as sanitize.Path,
so the same check can be shared with the archive backend which mounts
archives as a filesystem.
sanitize.Path keeps the extract semantics - reject any name with a
".." path component, treating both "/" and "\" as separators - and
additionally cleans the name with path.Clean. This corrects two edge
cases in extract: a repeated "./" prefix ("././file.txt") is now fully
stripped rather than only the first, and a bare "." entry is now
treated as the archive root and skipped.
Add sanitize.Leaf, which rejects a name that is empty, ".", ".." or
contains a "/", for checking a single directory entry name read from
an archive.
The names handled are rclone remote paths, in which "/" is the only
separator and "\" an ordinary character, so Leaf does not reject a
backslash: making a name safe for its storage is the destination
backend's job (the local backend encodes "\" on Windows and refuses
paths which escape its root). Path's rejection of ".." between
backslashes is kept as defence in depth for extract.
An empty or "." volume name joined onto the base directory resolves to the
base directory itself. newVolume does not call validate, so such a name
would mount a remote over the base directory and shadow every other
volume's mountpoint.
Require the resolved mountpoint to be a strict descendant of the base
directory so these degenerate names are refused.
When the plugin restarts it reads its persisted state file and used the
stored mountpoint verbatim. A state file written by an older rclone that
allowed escaping volume names, or one that was tampered with, could point
the mountpoint outside the base directory, so upgrading did not remediate
an already-escaped volume.
Re-derive the mountpoint from the base directory and the volume name on
restore, confined to the base directory, rather than trusting the stored
path.
A Docker VolumeDriver.Create request carries a raw volume name that was
joined onto the base directory with filepath.Join and used verbatim as the
mountpoint. filepath.Join collapses ".." components, so a crafted name such
as "../../../etc/foo" resolved to a host path outside the base directory,
where the plugin then created a directory and mounted the remote.
Confine the mountpoint to the base directory and refuse any name that
resolves outside it.
When serving FTP with --auth-proxy, the obscured password was cached in a
driver-global map keyed only by the username. Two sessions that logged in
with the same username but different credentials shared one map entry, so a
later login overwrote it and every subsequent operation on the earlier,
still-authenticated session was re-authorized with the later session's
credential and executed against the later session's backend.
Bind the credential to the FTP session by storing the obscured password in
the per-session goftp Session.Data map instead, so each session always
resolves the backend it authenticated for.
With --auth-proxy set and --auth-key unset, serve s3 registered every client
supplied access key ID with an empty secret and verified the SigV4 signature
against that, so anyone could sign a request for an arbitrary access key ID with
an empty secret and be let in. The proxy program was only ever given the access
key ID (as both user and pass) so it had nothing with which to authenticate the
client either.
An S3 client never sends its secret, only a signature made with it, so the
server has to know the secret to check the request. The auth proxy protocol as
been changed to handle this. For serve s3 the proxy program is given just the
access key ID as the user (no pass or public_key) and must return the matching
secret as _secret_access_key in its output. rclone verifies the request's
signature against that secret, refusing the request if the proxy rejects the
access key ID, doesn't return a secret or returns an empty one, or the signature
doesn't match. The secret is only used for this server's own verification and is
never registered with gofakes3, so other serve s3 instances in the same process
don't honour it.
The proxy's answers are cached. If a signature fails against a cached secret the
proxy is consulted again so a rotated secret takes effect immediately - but only
for a signature mismatch, and at most once every 10 seconds per access key ID
and client IP, so a stream of bad signatures can't make the proxy program run
for every request. A rotation never shuts down the cached backend under requests
still using it. A cached answer is checked with the proxy again once it is 5
minutes old even if in constant use, so revoking an access key ID takes effect
within 5 minutes.
This means --auth-key is no longer needed with --auth-proxy: it is ignored and a
warning is given at startup if both are set. The proxy is the source of truth
for both the credentials and the backend they map to. Presigned URLs (credential
in the query string) are now recognised by the proxy middleware too. The auth
proxy docs are added to serve s3.
Note that the serve s3 auth proxy protocol has changed. The proxy program is now
given the access key ID as "user" (it was previously given an MD5 hash of it,
with the access key ID as "pass") and must return the matching secret as
"_secret_access_key".
This needs gofakes3 v0.0.9 for signature.V4SignVerifyWithSecret.
gofakes3 kept the keys given with --auth-key in a store global to the process,
so when more than one serve s3 was running in one rclone (eg started via the rc)
each accepted the others' credentials and a client with the key for one server
could read and write the backend of another.
This updates gofakes3 to v0.0.9 which keeps auth keys per instance and adds a
test that two servers only accept their own keys.
From v1.70.0, an SFTP server started through the rc serve/start API with
a per-server proxyOpt.AuthProxy decided whether to enable proxy
authentication by checking the process-global proxy.Opt.AuthProxy
instead of the supplied proxyOpt.AuthProxy. In the normal rc case the
global is empty, so the auth proxy was silently ignored: the server
either failed to start with "no authorization found" or authenticated
against the local authorized_keys file instead of routing each login
through the proxy the operator configured.
The serve Provider refactor (f425f8d46) fixed the constructor by building the
provider from the supplied proxyOpt, but the authorized-keys handling in
configure() still consulted the global option. Make it depend on whether
proxy mode is actually active, and add a regression test for the
per-server option.
From v1.70.0 until the serve Provider refactor (f425f8d46), an S3 server started
through the rc serve/start API with a per-server proxyOpt.AuthProxy
decided whether to enable proxy authentication by checking the
process-global proxy.Opt.AuthProxy instead of the supplied
proxyOpt.AuthProxy. In the normal rc case the global is empty, so the
auth proxy was silently ignored and the server served the fixed
filesystem supplied to serve/start rather than routing each access key
to the backend chosen by the proxy, bypassing the operator's intended
per-key authorization.
The Provider refactor fixed this incidentally by building the provider
from the proxyOpt passed to the constructor. This adds a regression test
so the per-server option cannot silently stop working again, and only
logs "allowing anonymous access" when neither an auth key nor an auth
proxy is configured so the log reflects the effective mode.
From v1.70.0 until the serve Provider refactor (f425f8d46), an FTP server started
through the rc serve/start API with a per-server proxyOpt.AuthProxy
decided whether to enable proxy authentication by checking the
process-global proxy.Opt.AuthProxy instead of the supplied
proxyOpt.AuthProxy. In the normal rc case the global is empty, so the
auth proxy was silently ignored and the server fell back to its
fixed-backend mode, whose default account accepts user "anonymous" with
any password - a complete authentication bypass.
The Provider refactor fixed this incidentally by building the provider
from the proxyOpt passed to the constructor. This adds a regression test
so the per-server option cannot silently stop working again.
The multipart reorder-buffer admission trusted the client-declared part length.
A negative length was accepted, and `buffered + size` could overflow int64 for
a huge declared length, wrapping the running total negative and admitting
further parts past --multipart-streaming-buffer-limit.
Reject a negative length and use the overflow-safe comparison `size <=
bufferLimit - buffered` so an untrusted Content-Length can neither poison nor
overflow the budget.
Streamed multipart UploadPart called Reserve(contentLength) before reading any
body bytes, so the pool immediately allocated one 1 MiB page per MiB of the
client-declared Content-Length (or X-Amz-Decoded-Content-Length). An client
could declare a huge part size, send no body, and force an arbitrarily large
allocation without paying the bandwidth cost of the declared body.
Drop the Reserve so the pool-backed buffer grows a page at a time as the body
is actually read: memory now tracks the bytes received, not the unverified
header.
When bisync is interrupted with a graceful shutdown it keeps the files
which transferred successfully in its listings and rolls the rest back.
An operator precedence mistake in that check meant a transfer of an
empty file (or one of unknown size) was kept even when it had failed,
so bisync recorded it as synced when it had not been.
The NFS section under Mounting on macOS talked about serve nfs without
pointing at rclone nfsmount, which is the command that actually does the
NFS-based mount on macOS.
Fixes#7869
Signed-off-by: Dean Chen <862469039@qq.com>
A client which started a multipart upload and vanished without either
completing or aborting it used to hold on to its resources forever.
Incomplete multipart uploads which have had no activity for
--multipart-expiry (default 24h) are now aborted and cleaned up
exactly as if the client had called AbortMultipartUpload, with a
NOTICE logged.
An upload with a part still being received is never expired, and each
completed part restarts the clock. Late operations on an expired
upload fail with NoSuchUpload, as they do on real S3 when a lifecycle
rule has aborted the upload.
Set --multipart-expiry 0 to keep incomplete uploads forever.
Multipart uploads used to be streamed directly to the remote with their
own PutStream machinery, bypassing the VFS, a design left over from
before the VFS could abandon a streaming write.
They are now written through the VFS exactly like plain object PUTs in
every cache mode. The parts are written, in part-number order, to a
temporary object which is renamed into place server-side on
completion.
With the default --vfs-cache-mode off the parts stream through the VFS
to the remote as they arrive. With --vfs-cache-mode writes or above
they are buffered in the VFS cache and uploaded by its write-back.
User visible changes:
- Multipart uploads now show in rclone's transfer stats and obey
--bwlimit (previously they bypassed both).
- Remotes without streaming upload support now spool the upload to a
temporary file on local disk instead of buffering it in memory.
- Multipart uploads are never buffered in memory because of missing
remote capabilities - only --disable-multipart-streaming does that.
- Remotes that upload atomically now also write to a temporary object
renamed into place, so an in-progress multipart upload is no longer
briefly visible under its final key.
- On the few remotes with no server-side move or copy the parts are
written straight to the final object in all cache modes.
- With --vfs-cache-mode writes, plain PUTs and multipart uploads to the
same key go through the same cache entry, so an earlier PUT still in
the write-back window can no longer be written back over a newer
multipart upload.
- Failed write-backs are retried by the VFS without the client having
to restart the upload, and completed objects are served from the
cache for read-after-write.
The temporary objects that uploads are written to before being renamed
into place are now named .rclone_temp_put_* and .rclone_temp_multipart_*,
and the whole .rclone_temp_ prefix is reserved: any object whose name
starts with it is hidden from S3 listings. This gives a single pattern
for cleaning up leftovers from killed servers:
rclone delete --min-age 24h --include ".rclone_temp_*" remote:path
The .rclone_multipart_upload_* objects rclone v1.75 used are still
hidden from listings so leftovers from an older server stay invisible
to S3 clients.
The mtime metadata fallback was nested inside the X-Amz-Meta-Mtime
branch, so it only ran when X-Amz-Meta-Mtime was present but invalid -
and then set the modtime from the invalid value's failed parse rather
than parsing mtime. An object PUT with only mtime metadata kept the
upload time as its modtime.
Now the two keys are checked independently, as TouchObject already
does.
streamPart did not check whether the upload had been torn down, but
AbortMultipartUpload sets the reorder buffer map to nil, so an abort
arriving while a part body was still being received panicked with an
assignment to a nil map once the part was buffered.
Now a part whose upload has been aborted or completed under it is
rejected with NoSuchUpload.
A PUT which failed part way through removed the object at the
destination key. As well as differing from real S3 (where a failed PUT
never affects the stored object), this raced with the client's
automatic retry of the same PUT: the retry stored the object and
returned 200 OK, then the failed first attempt's cleanup deleted it,
silently losing an acknowledged upload. The interrupted upload could
also be committed as a truncated object, since closing the write handle
gave the streaming upload a clean end of stream.
Now a failed or interrupted PUT never disturbs the object at the key:
- The object at the key is never removed on error.
- On backends where a partial upload is visible at its final name
(PartialUploads), and when the VFS cache mode is writes or above, the
upload is written to a temporary object which is renamed into place
on success and removed on failure, as streamed multipart uploads
already do. Backends which upload atomically are still streamed
straight to the destination.
- An interrupted or short body fails the upload via
WriteFileHandle.CloseWithError instead of committing truncated data,
and a body which ends cleanly short of its declared size is rejected
with IncompleteBody.
Before this change, `fastCopy` created a cancellable context for the sync and
stored its cancel func on the `bisyncRun`, but only ever called it when
gracefully shutting down. On a normal run, it was never called, and until it is
cancelled, a context from `context.WithCancel` stays registered with its nearest
cancellable ancestor.
For an rc job, that ancestor is the job's own context, which the job registry
retains for `--rc-job-expire-duration`. The sync context carries bisync's
`LoggerOpt`, whose `LoggerFn` is a method value on `*bisyncRun`, so a finished
run was kept alive -- including the Path1 and Path2 listings -- for as long as the
job was.
This change fixes the issue by cancelling the sync context when `fastCopy`
returns. The cancel func is still stored on the `bisyncRun`, so a graceful
shutdown can still interrupt a sync that is in progress.
Write the git configuration for each test's fake home directory as a
file instead of running three "git config --global" commands, and drop
the "git annex version" invocation from repository setup. This removes
four subprocess launches from each of the fifteen test cases.
The subtests within each end to end test function already run in
parallel, but the three test functions themselves ran one after
another.
Marking the functions parallel lets all their subtests overlap.
The end to end tests exercise rclone via a separate subprocess spawned
by git-annex, and that subprocess is not built with race
instrumentation.
Running them in the race test therefore adds several minutes to CI
without providing any race coverage. The unit tests in this package
still run under the race detector.
Run the full "git annex testremote" suite for a single layout mode and
use "testremote --fast" for the rest.
The full suite repeats the same protocol operations across a matrix of
key sizes and chunk configurations, which exercises client side
git-annex behaviour rather than rclone.
The migration test ran the full "git annex testremote" suite for every
layout mode after verifying the migration with "git annex fsck".
The fsck calls already prove the migrated data is accessible via the
builtin special remote, and TestEndToEnd covers the special remote
protocol with testremote, so the extra five full testremote runs
duplicated coverage at a cost of tens of seconds each on CI.
The auth proxy was only given the user and their password or public
key, so a proxy program had no way to restrict logins to particular
networks, or to record where an authentication attempt came from.
The JSON sent to the program now has a client_ip key holding the bare
IP the client connected from, with the port stripped so IPv6 arrives
as 2001:db8::1 rather than [2001:db8::1]:52344. An IPv4-mapped IPv6
address is reported as plain IPv4 so that a client arriving over a
dual-stack listener still matches IPv4 networks. The key is omitted
when the client has no IP address.
The IP is also mixed into the backend cache key. That is needed as the
program is only run on a cache miss, so a client from a
non-allowlisted address presenting valid credentials within the 5
minute cache lifetime would get a cache hit and be let in without the
program being consulted at all.
The macOS NFS client sends SETATTR after SYMLINK, which arrives as
Chmod/Chown on the link path. These opened the target with vfs.Open,
which follows symlinks - a freshly created symlink usually dangles, so
the open failed with ENOENT, surfaced to the client as NFS3ERR_IO even
though the link was created.
Add path-based VFS.Chmod and VFS.Chown mirroring VFS.Chtimes. They do
not follow symlinks (lstat semantics, matching VFS.Stat) and return
ENOSYS when the node exists, since the VFS stores neither permissions
nor ownership; serve nfs calls them and masks ENOSYS as before.
Fixes#9627
A request path beginning with "../" escaped the path the server was
started on, letting a client list, read, create, overwrite and delete
objects outside it.
The check added for CVE-2026-59733 rejected non-canonical paths by
comparing them with path.Clean, but path.Clean cannot resolve leading
".." elements in a relative path so it leaves them in place and the
comparison comes out equal. Only interior traversal such as "a/../../x"
was rejected. Whether a path then escaped depended on the backend:
those which join the root with the remote before encoding it - webdav,
ftp, sftp, http and memory - resolved the ".." away, while local and s3
encode the dot elements first and were unaffected.
A bare "." was accepted for the same reason, which on bucket backends
addresses the served directory's own key.
Validate with io/fs.ValidPath instead, which rejects ".", ".." and empty
elements wherever they appear. The empty path stays valid as the root of
the API, and "." is excluded explicitly because ValidPath accepts it as
the root of an FS.
Three ways a single client could deny service to everyone else connected to
the same serve sftp process:
A session "subsystem" request payload is a length-prefixed string, but it was
sliced at a fixed offset without checking its length, so a client sending a
truncated payload panicked the out-of-band request goroutine and killed the
process. Decode the payload instead, the way the neighbouring "exec" request
already does.
Rejecting a request then left the goroutine handling that channel waiting
forever to learn what kind of channel it was, because nothing was ever sent
on the channel it waits on. A client could open channels in a loop making
unsupported requests and grow the server's goroutines and memory without
bound. Signal the waiter when the requests run out so the channel is torn
down.
Separately, the request handlers - and reads, writes and closes on the file
handles they return - run on pkg/sftp packet worker goroutines which have no
panic recovery. A panic raised by a backend while serving one request took the
process down with it. Recover panics at that boundary, log them with a stack
trace, and return them to the requesting client as an error instead.
Addresses GHSA-6jcg-q3wp-x2f4.
f-denkena
Nicholas VeltenandNick Craig-Wood
Murat TopcuandNick Craig-Wood
phatlcandNick Craig-Wood
Hakan İSMAİLandNick Craig-Wood
Dean ChenandNick Craig-Wood
Jiacheng XuandNick Craig-Wood
nielash
am-at-enrollvbandGitHub
SillyZirandNick Craig-Wood