The NFS section under Mounting on macOS talked about serve nfs without
pointing at rclone nfsmount, which is the command that actually does the
NFS-based mount on macOS.
Fixes#7869
Signed-off-by: Dean Chen <862469039@qq.com>
A client which started a multipart upload and vanished without either
completing or aborting it used to hold on to its resources forever.
Incomplete multipart uploads which have had no activity for
--multipart-expiry (default 24h) are now aborted and cleaned up
exactly as if the client had called AbortMultipartUpload, with a
NOTICE logged.
An upload with a part still being received is never expired, and each
completed part restarts the clock. Late operations on an expired
upload fail with NoSuchUpload, as they do on real S3 when a lifecycle
rule has aborted the upload.
Set --multipart-expiry 0 to keep incomplete uploads forever.
Multipart uploads used to be streamed directly to the remote with their
own PutStream machinery, bypassing the VFS, a design left over from
before the VFS could abandon a streaming write.
They are now written through the VFS exactly like plain object PUTs in
every cache mode. The parts are written, in part-number order, to a
temporary object which is renamed into place server-side on
completion.
With the default --vfs-cache-mode off the parts stream through the VFS
to the remote as they arrive. With --vfs-cache-mode writes or above
they are buffered in the VFS cache and uploaded by its write-back.
User visible changes:
- Multipart uploads now show in rclone's transfer stats and obey
--bwlimit (previously they bypassed both).
- Remotes without streaming upload support now spool the upload to a
temporary file on local disk instead of buffering it in memory.
- Multipart uploads are never buffered in memory because of missing
remote capabilities - only --disable-multipart-streaming does that.
- Remotes that upload atomically now also write to a temporary object
renamed into place, so an in-progress multipart upload is no longer
briefly visible under its final key.
- On the few remotes with no server-side move or copy the parts are
written straight to the final object in all cache modes.
- With --vfs-cache-mode writes, plain PUTs and multipart uploads to the
same key go through the same cache entry, so an earlier PUT still in
the write-back window can no longer be written back over a newer
multipart upload.
- Failed write-backs are retried by the VFS without the client having
to restart the upload, and completed objects are served from the
cache for read-after-write.
The temporary objects that uploads are written to before being renamed
into place are now named .rclone_temp_put_* and .rclone_temp_multipart_*,
and the whole .rclone_temp_ prefix is reserved: any object whose name
starts with it is hidden from S3 listings. This gives a single pattern
for cleaning up leftovers from killed servers:
rclone delete --min-age 24h --include ".rclone_temp_*" remote:path
The .rclone_multipart_upload_* objects rclone v1.75 used are still
hidden from listings so leftovers from an older server stay invisible
to S3 clients.
The mtime metadata fallback was nested inside the X-Amz-Meta-Mtime
branch, so it only ran when X-Amz-Meta-Mtime was present but invalid -
and then set the modtime from the invalid value's failed parse rather
than parsing mtime. An object PUT with only mtime metadata kept the
upload time as its modtime.
Now the two keys are checked independently, as TouchObject already
does.
streamPart did not check whether the upload had been torn down, but
AbortMultipartUpload sets the reorder buffer map to nil, so an abort
arriving while a part body was still being received panicked with an
assignment to a nil map once the part was buffered.
Now a part whose upload has been aborted or completed under it is
rejected with NoSuchUpload.
A PUT which failed part way through removed the object at the
destination key. As well as differing from real S3 (where a failed PUT
never affects the stored object), this raced with the client's
automatic retry of the same PUT: the retry stored the object and
returned 200 OK, then the failed first attempt's cleanup deleted it,
silently losing an acknowledged upload. The interrupted upload could
also be committed as a truncated object, since closing the write handle
gave the streaming upload a clean end of stream.
Now a failed or interrupted PUT never disturbs the object at the key:
- The object at the key is never removed on error.
- On backends where a partial upload is visible at its final name
(PartialUploads), and when the VFS cache mode is writes or above, the
upload is written to a temporary object which is renamed into place
on success and removed on failure, as streamed multipart uploads
already do. Backends which upload atomically are still streamed
straight to the destination.
- An interrupted or short body fails the upload via
WriteFileHandle.CloseWithError instead of committing truncated data,
and a body which ends cleanly short of its declared size is rejected
with IncompleteBody.
Before this change, `fastCopy` created a cancellable context for the sync and
stored its cancel func on the `bisyncRun`, but only ever called it when
gracefully shutting down. On a normal run, it was never called, and until it is
cancelled, a context from `context.WithCancel` stays registered with its nearest
cancellable ancestor.
For an rc job, that ancestor is the job's own context, which the job registry
retains for `--rc-job-expire-duration`. The sync context carries bisync's
`LoggerOpt`, whose `LoggerFn` is a method value on `*bisyncRun`, so a finished
run was kept alive -- including the Path1 and Path2 listings -- for as long as the
job was.
This change fixes the issue by cancelling the sync context when `fastCopy`
returns. The cancel func is still stored on the `bisyncRun`, so a graceful
shutdown can still interrupt a sync that is in progress.
Write the git configuration for each test's fake home directory as a
file instead of running three "git config --global" commands, and drop
the "git annex version" invocation from repository setup. This removes
four subprocess launches from each of the fifteen test cases.
The subtests within each end to end test function already run in
parallel, but the three test functions themselves ran one after
another.
Marking the functions parallel lets all their subtests overlap.
The end to end tests exercise rclone via a separate subprocess spawned
by git-annex, and that subprocess is not built with race
instrumentation.
Running them in the race test therefore adds several minutes to CI
without providing any race coverage. The unit tests in this package
still run under the race detector.
Run the full "git annex testremote" suite for a single layout mode and
use "testremote --fast" for the rest.
The full suite repeats the same protocol operations across a matrix of
key sizes and chunk configurations, which exercises client side
git-annex behaviour rather than rclone.
The migration test ran the full "git annex testremote" suite for every
layout mode after verifying the migration with "git annex fsck".
The fsck calls already prove the migrated data is accessible via the
builtin special remote, and TestEndToEnd covers the special remote
protocol with testremote, so the extra five full testremote runs
duplicated coverage at a cost of tens of seconds each on CI.
The auth proxy was only given the user and their password or public
key, so a proxy program had no way to restrict logins to particular
networks, or to record where an authentication attempt came from.
The JSON sent to the program now has a client_ip key holding the bare
IP the client connected from, with the port stripped so IPv6 arrives
as 2001:db8::1 rather than [2001:db8::1]:52344. An IPv4-mapped IPv6
address is reported as plain IPv4 so that a client arriving over a
dual-stack listener still matches IPv4 networks. The key is omitted
when the client has no IP address.
The IP is also mixed into the backend cache key. That is needed as the
program is only run on a cache miss, so a client from a
non-allowlisted address presenting valid credentials within the 5
minute cache lifetime would get a cache hit and be let in without the
program being consulted at all.
The macOS NFS client sends SETATTR after SYMLINK, which arrives as
Chmod/Chown on the link path. These opened the target with vfs.Open,
which follows symlinks - a freshly created symlink usually dangles, so
the open failed with ENOENT, surfaced to the client as NFS3ERR_IO even
though the link was created.
Add path-based VFS.Chmod and VFS.Chown mirroring VFS.Chtimes. They do
not follow symlinks (lstat semantics, matching VFS.Stat) and return
ENOSYS when the node exists, since the VFS stores neither permissions
nor ownership; serve nfs calls them and masks ENOSYS as before.
Fixes#9627
A request path beginning with "../" escaped the path the server was
started on, letting a client list, read, create, overwrite and delete
objects outside it.
The check added for CVE-2026-59733 rejected non-canonical paths by
comparing them with path.Clean, but path.Clean cannot resolve leading
".." elements in a relative path so it leaves them in place and the
comparison comes out equal. Only interior traversal such as "a/../../x"
was rejected. Whether a path then escaped depended on the backend:
those which join the root with the remote before encoding it - webdav,
ftp, sftp, http and memory - resolved the ".." away, while local and s3
encode the dot elements first and were unaffected.
A bare "." was accepted for the same reason, which on bucket backends
addresses the served directory's own key.
Validate with io/fs.ValidPath instead, which rejects ".", ".." and empty
elements wherever they appear. The empty path stays valid as the root of
the API, and "." is excluded explicitly because ValidPath accepts it as
the root of an FS.
Three ways a single client could deny service to everyone else connected to
the same serve sftp process:
A session "subsystem" request payload is a length-prefixed string, but it was
sliced at a fixed offset without checking its length, so a client sending a
truncated payload panicked the out-of-band request goroutine and killed the
process. Decode the payload instead, the way the neighbouring "exec" request
already does.
Rejecting a request then left the goroutine handling that channel waiting
forever to learn what kind of channel it was, because nothing was ever sent
on the channel it waits on. A client could open channels in a loop making
unsupported requests and grow the server's goroutines and memory without
bound. Signal the waiter when the requests run out so the channel is torn
down.
Separately, the request handlers - and reads, writes and closes on the file
handles they return - run on pkg/sftp packet worker goroutines which have no
panic recovery. A panic raised by a backend while serving one request took the
process down with it. Recover panics at that boundary, log them with a stack
trace, and return them to the requesting client as an error instead.
Addresses GHSA-6jcg-q3wp-x2f4.
The builtin authentication compared the configured username and password
with ==, whose run time depends on how much of the value matches, giving a
timing side-channel that could in principle help guess the password.
serve sftp, serve s3 and the auth proxy already use subtle.ConstantTimeCompare
so bring serve ftp in line with them. An empty configured password
still accepts any password.
Addresses GHSA-mfvx-7rcj-9m5g finding 4.
Cover the portmap registration OpenBSD's kernel NFS client needs (recipe
from hjicks in #8578), the mount_nfs -T requirement, and why the server
advertises AUTH_UNIX, so an OpenBSD user doesn't need to read through the
issue thread to get a working mount.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The mount served fine on Linux and macOS but *BSD kernel NFS clients
refused it with "Authentication error". The MOUNT reply only ever
offered AUTH_NULL, and the OpenBSD/FreeBSD/NetBSD clients won't mount a
server unless AUTH_UNIX is among the offered flavors.
Add AUTH_UNIX to the advertised list. The server still doesn't inspect
the credential (there's no per-user access control here) so the AUTH_UNIX
cred the client then sends is read as an opaque blob and ignored, exactly
as the AUTH_NULL one was. No behaviour change for existing Linux/macOS
clients, and OpenBSD now mounts and reads files over the share.
Verified on a real OpenBSD 7.9 arm64 VM: registered rclone's server in
portmap and ran "mount_nfs -T localhost:/ /mnt"; the mount now succeeds
(MOUNT and GetAttr RPCs go through) and files read back correctly through
the mount, where before it stopped at the auth stage.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The previous OpenBSD path built the right options ("-o port=N -T") but
still handed them to mount(8). On OpenBSD "-T" is a mount_nfs(8) flag,
not a mount(8) one, so mount rejected it with "mount: unknown option --
T" and the mount never ran. Call mount_nfs(8) directly on OpenBSD; the
options are already in its native syntax.
Verified on a real OpenBSD 7.9 arm64 VM: with this change the command
becomes "mount_nfs -o port=N -T localhost:/ <mnt>" and mount_nfs accepts
the flags. The mount then fails later with "Stale NFS file handle"
because the OpenBSD client and rclone's in-process NFS server disagree
on the root filehandle - that is a separate issue in the NFS server,
not in the mount options this PR is about.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
nfsmount built its mount options for GNU/Linux syntax unconditionally:
"-o port=N", "-o mountport=N" and "-o tcp". OpenBSD's mount_nfs(8)
rejects "-o mountport" outright ("option not supported", per the
reporter's log) and has no "tcp" suboption either, since it selects
TCP with the separate "-T" flag instead of an -o suboption.
Add a runtime.GOOS == "openbsd" branch that builds the option list
OpenBSD's mount_nfs actually accepts: "-o port=N" plus "-T" for TCP,
with no mountport option since OpenBSD's mountd is located via
portmap rather than a fixed, settable port. This follows the same
GOOS-branching pattern already used in this file's unmount function
(darwin) and in cmd/cmount/mount.go for openbsd/freebsd differences.
FreeBSD's mount_nfs(8) documents "port=", "mountport=" and "tcp" as
-o suboptions identical to Linux, so the existing option set is left
unchanged for freebsd and all other platforms.
Verified by cross-compiling (go build and go vet) for GOOS=openbsd,
freebsd, linux and darwin, all of which succeed. Actually mounting
via mount_nfs on OpenBSD needs a BSD machine to confirm at runtime,
which wasn't available here.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Previously, GET requests for a directory URL always returned an HTML
directory listing. There was no way to suppress this, unlike
`serve webdav` which has supported --disable-dir-list since #4191.
This adds the same flag to `serve http`. When set, GET requests for
directory URLs return 404 instead of a listing, while file downloads
continue to work normally.
Based on the approach suggested in #6306.
Fixes#4000
This resolves an issue where mount filters supplied to the rc API
(such as `_filter` in remote control requests) were ignored during
FUSE mounts.
By passing the request context containing the parsed filter config to
`vfs.New`, the VFS layer now correctly respects the active filter
rules.
Fixes#8838
Backend options with Hide: fs.OptionHideBoth are hidden from the config
wizard and from the command line flag listing, but "rclone help backend"
(and therefore the autogenerated backend documentation) still showed
them. Skip them there too so fully hidden options no longer appear in
the docs.
This updates `mountRc` in `cmd/mountlib/rc.go` to parse options using the
unified `rc.ParseOptions` helper. It also enforces parameter validation by
calling `rc.CheckParamsUsed`.
- Replace custom options parsing with rc.ParseOptions for vfsOpt and mountOpt.
- Delete consumed params (mountPoint, mountType, fs) and call CheckParamsUsed
before initiating FUSE mount to reject unknown parameters.
- Clean up duplication tests (TestRcFlatOptions, TestRcFlatOptionsNull) in rc_test.go.
- Update TestRc in rc_test.go to pass a clean params map to unmount.Fn.
This refactors all 8 serve protocols to use `rc.ParseOptions` for VFS
and protocol options decoding. It also implements parameter validation in the
main runner using `rc.CheckParamsUsed`.
- Update dlna, ftp, http, nfs, restic, s3, sftp, and webdav to call
rc.ParseOptions, enabling nested option block support.
- Remove unused configstruct imports.
- Update startRc in cmd/serve/rc.go to copy input parameters and
call rc.CheckParamsUsed to reject unknown parameters.
- Add TestRcStartFlatNestedAndUnknownRejection to cmd/serve/rc_test.go.
- Update inline documentation in serve/start command help to include
nested blocks information (vfsOpt, proxyOpt, opt) and a new WebDAV example.
The unauthenticated DLNA control endpoint decoded arbitrary SOAP bodies
into an in-memory XML field. A LAN client could send a large request
and exhaust the server's memory.
Limit SOAP request bodies to 1 MiB and return 413 when the limit is
exceeded.
The VFS is shared between users with the same remote and options, for
example two mounts created over the rc, or a mount and an NFS server.
Each node has a single Sys() slot which mount, mount2 and serve nfs
all used to attach their per-node data. With a shared VFS the users
overwrote each other's data: at best churning the cached FUSE nodes,
and since the slot was an atomic.Value, panicking with "store of
inconsistently typed value" as soon as two users stored different
types on the same node.
This change gives each node auxiliary values keyed by owner, set with
SetAux and read with Aux, so each user of the VFS has an independent
slot. The mounts now cache their FUSE nodes under their own key,
leaving Sys - which is read through the os.FileInfo interface -
reserved for users like serve nfs which need to control what that
returns.
Nodes with nothing attached use less memory than before (one pointer
instead of an atomic.Value) and reads remain lock free.
Bug discovered while thinking about #9617
Streamed multipart uploads had two problems when the client uploaded parts
concurrently:
Parts arriving ahead of the next part needed by the backend stream were buffered
in memory without limit, acknowledging each part as soon as it was received. A
client uploading faster than the backend could drain would therefore balloon the
server's memory to the size of the upload. Buffering is now bounded a new
--multipart-streaming-buffer-limit flag (default 256Mi, 0 for unlimited): a part
that would take the buffer over the limit is not read until the stream drains,
applying backpressure to the client instead of using unbounded memory.
A part uploaded again with the same number - typically a client retrying after
its request timed out - left a stale copy in the reorder buffer which made
CompleteMultipartUpload fail with InvalidPart, aborting the whole upload.
Re-uploaded parts are now handled properly: a copy still in the buffer is
replaced, an identical copy of an already streamed part is accepted as a no-op,
and only replacing an already streamed part with different content (which the
in-order stream cannot honour) is rejected.
Fixes#9640
The auto-derived CORS origin logic (added in #9603 for the wildcard-bind
case) ran unconditionally, so it clobbered an explicitly-passed
--rc-allow-origin too - the flag was silently ignored. That breaks the
documented Docker/remote-browser setup where the bind address and the
address the browser actually reaches the GUI at are different.
Now it only kicks in when the user hasn't set --rc-allow-origin
themselves.
Go 1.27 changes compress/flate output, which also changes archive/zip
byte output. The HTTP zip download tests currently compare raw zip bytes
against golden files, so they fail even though the generated zip archives
contain the expected files.
Compare zip entries and decompressed contents instead of the exact
compressed byte stream.
Signed-off-by: Mikel Olasagasti Uranga <mikel@olasagasti.info>
Docker may re-send Create requests for volumes that already exist,
especially after a plugin restart. Previously this returned
ErrVolumeExists which Docker surfaced as "volume name must be unique".
Now if a volume with the same name already exists, Create returns
success (no-op), matching the Docker volume plugin protocol's
expectation of idempotent operations.
When restoring multiple volumes from saved state, each volume's
filesystem setup (including connecting to the remote) was done
sequentially. Now volumes are restored concurrently, so a single
slow or unreachable remote doesn't delay the restoration of other
volumes.
When restoring volumes from saved state during plugin startup, a slow
or unreachable remote could block indefinitely in fs.NewFs. Add a
30-second per-volume timeout context so that individual volume failures
are logged and skipped rather than blocking the entire plugin.
Nick Craig-Wood
Hakan İSMAİLandNick Craig-Wood
Dean ChenandNick Craig-Wood
Jiacheng XuandNick Craig-Wood
nielash
am-at-enrollvbandGitHub
SillyZirandNick Craig-Wood
Leon BrocardandNick Craig-Wood
Hakan İSMAİLandGitHub
Acts1631andGitHub
GiridharandNick Craig-Wood
KyueandNick Craig-Wood
FTCHDandNick Craig-Wood
Mikel Olasagasti UrangaandNick Craig-Wood