The previous OpenBSD path built the right options ("-o port=N -T") but
still handed them to mount(8). On OpenBSD "-T" is a mount_nfs(8) flag,
not a mount(8) one, so mount rejected it with "mount: unknown option --
T" and the mount never ran. Call mount_nfs(8) directly on OpenBSD; the
options are already in its native syntax.
Verified on a real OpenBSD 7.9 arm64 VM: with this change the command
becomes "mount_nfs -o port=N -T localhost:/ <mnt>" and mount_nfs accepts
the flags. The mount then fails later with "Stale NFS file handle"
because the OpenBSD client and rclone's in-process NFS server disagree
on the root filehandle - that is a separate issue in the NFS server,
not in the mount options this PR is about.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
nfsmount built its mount options for GNU/Linux syntax unconditionally:
"-o port=N", "-o mountport=N" and "-o tcp". OpenBSD's mount_nfs(8)
rejects "-o mountport" outright ("option not supported", per the
reporter's log) and has no "tcp" suboption either, since it selects
TCP with the separate "-T" flag instead of an -o suboption.
Add a runtime.GOOS == "openbsd" branch that builds the option list
OpenBSD's mount_nfs actually accepts: "-o port=N" plus "-T" for TCP,
with no mountport option since OpenBSD's mountd is located via
portmap rather than a fixed, settable port. This follows the same
GOOS-branching pattern already used in this file's unmount function
(darwin) and in cmd/cmount/mount.go for openbsd/freebsd differences.
FreeBSD's mount_nfs(8) documents "port=", "mountport=" and "tcp" as
-o suboptions identical to Linux, so the existing option set is left
unchanged for freebsd and all other platforms.
Verified by cross-compiling (go build and go vet) for GOOS=openbsd,
freebsd, linux and darwin, all of which succeed. Actually mounting
via mount_nfs on OpenBSD needs a BSD machine to confirm at runtime,
which wasn't available here.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Previously, GET requests for a directory URL always returned an HTML
directory listing. There was no way to suppress this, unlike
`serve webdav` which has supported --disable-dir-list since #4191.
This adds the same flag to `serve http`. When set, GET requests for
directory URLs return 404 instead of a listing, while file downloads
continue to work normally.
Based on the approach suggested in #6306.
Fixes#4000
This resolves an issue where mount filters supplied to the rc API
(such as `_filter` in remote control requests) were ignored during
FUSE mounts.
By passing the request context containing the parsed filter config to
`vfs.New`, the VFS layer now correctly respects the active filter
rules.
Fixes#8838
Backend options with Hide: fs.OptionHideBoth are hidden from the config
wizard and from the command line flag listing, but "rclone help backend"
(and therefore the autogenerated backend documentation) still showed
them. Skip them there too so fully hidden options no longer appear in
the docs.
This updates `mountRc` in `cmd/mountlib/rc.go` to parse options using the
unified `rc.ParseOptions` helper. It also enforces parameter validation by
calling `rc.CheckParamsUsed`.
- Replace custom options parsing with rc.ParseOptions for vfsOpt and mountOpt.
- Delete consumed params (mountPoint, mountType, fs) and call CheckParamsUsed
before initiating FUSE mount to reject unknown parameters.
- Clean up duplication tests (TestRcFlatOptions, TestRcFlatOptionsNull) in rc_test.go.
- Update TestRc in rc_test.go to pass a clean params map to unmount.Fn.
This refactors all 8 serve protocols to use `rc.ParseOptions` for VFS
and protocol options decoding. It also implements parameter validation in the
main runner using `rc.CheckParamsUsed`.
- Update dlna, ftp, http, nfs, restic, s3, sftp, and webdav to call
rc.ParseOptions, enabling nested option block support.
- Remove unused configstruct imports.
- Update startRc in cmd/serve/rc.go to copy input parameters and
call rc.CheckParamsUsed to reject unknown parameters.
- Add TestRcStartFlatNestedAndUnknownRejection to cmd/serve/rc_test.go.
- Update inline documentation in serve/start command help to include
nested blocks information (vfsOpt, proxyOpt, opt) and a new WebDAV example.
The unauthenticated DLNA control endpoint decoded arbitrary SOAP bodies
into an in-memory XML field. A LAN client could send a large request
and exhaust the server's memory.
Limit SOAP request bodies to 1 MiB and return 413 when the limit is
exceeded.
The VFS is shared between users with the same remote and options, for
example two mounts created over the rc, or a mount and an NFS server.
Each node has a single Sys() slot which mount, mount2 and serve nfs
all used to attach their per-node data. With a shared VFS the users
overwrote each other's data: at best churning the cached FUSE nodes,
and since the slot was an atomic.Value, panicking with "store of
inconsistently typed value" as soon as two users stored different
types on the same node.
This change gives each node auxiliary values keyed by owner, set with
SetAux and read with Aux, so each user of the VFS has an independent
slot. The mounts now cache their FUSE nodes under their own key,
leaving Sys - which is read through the os.FileInfo interface -
reserved for users like serve nfs which need to control what that
returns.
Nodes with nothing attached use less memory than before (one pointer
instead of an atomic.Value) and reads remain lock free.
Bug discovered while thinking about #9617
Streamed multipart uploads had two problems when the client uploaded parts
concurrently:
Parts arriving ahead of the next part needed by the backend stream were buffered
in memory without limit, acknowledging each part as soon as it was received. A
client uploading faster than the backend could drain would therefore balloon the
server's memory to the size of the upload. Buffering is now bounded a new
--multipart-streaming-buffer-limit flag (default 256Mi, 0 for unlimited): a part
that would take the buffer over the limit is not read until the stream drains,
applying backpressure to the client instead of using unbounded memory.
A part uploaded again with the same number - typically a client retrying after
its request timed out - left a stale copy in the reorder buffer which made
CompleteMultipartUpload fail with InvalidPart, aborting the whole upload.
Re-uploaded parts are now handled properly: a copy still in the buffer is
replaced, an identical copy of an already streamed part is accepted as a no-op,
and only replacing an already streamed part with different content (which the
in-order stream cannot honour) is rejected.
Fixes#9640
The auto-derived CORS origin logic (added in #9603 for the wildcard-bind
case) ran unconditionally, so it clobbered an explicitly-passed
--rc-allow-origin too - the flag was silently ignored. That breaks the
documented Docker/remote-browser setup where the bind address and the
address the browser actually reaches the GUI at are different.
Now it only kicks in when the user hasn't set --rc-allow-origin
themselves.
Go 1.27 changes compress/flate output, which also changes archive/zip
byte output. The HTTP zip download tests currently compare raw zip bytes
against golden files, so they fail even though the generated zip archives
contain the expected files.
Compare zip entries and decompressed contents instead of the exact
compressed byte stream.
Signed-off-by: Mikel Olasagasti Uranga <mikel@olasagasti.info>
Docker may re-send Create requests for volumes that already exist,
especially after a plugin restart. Previously this returned
ErrVolumeExists which Docker surfaced as "volume name must be unique".
Now if a volume with the same name already exists, Create returns
success (no-op), matching the Docker volume plugin protocol's
expectation of idempotent operations.
When restoring multiple volumes from saved state, each volume's
filesystem setup (including connecting to the remote) was done
sequentially. Now volumes are restored concurrently, so a single
slow or unreachable remote doesn't delay the restoration of other
volumes.
When restoring volumes from saved state during plugin startup, a slow
or unreachable remote could block indefinitely in fs.NewFs. Add a
30-second per-volume timeout context so that individual volume failures
are logged and skipped rather than blocking the entire plugin.
Previously, restoreState in NewDriver would restore volumes AND perform
FUSE mounts synchronously before the Docker plugin socket was created.
This caused two problems:
1. The monChan was created after restoreState, but mount() sends on
monChan, causing a deadlock (send on nil channel blocks forever).
2. Even with the channel fix, slow or hanging mounts during state
restore would block the server socket from being created, causing
Docker to time out after ~13 seconds with "no such file or
directory" for the plugin socket.
Fix by:
- Moving monChan creation and monitor goroutine start before
restoreState
- Splitting state restore into two phases: restoreState (metadata +
filesystem setup only) and RestoreMounts (actual FUSE mounts)
- Calling RestoreMounts asynchronously after the server starts
listening
- Performing mount restoration concurrently across volumes
Fixes#9231
The volume plugin parses the remote option as a trusted connection
string, which can run local commands via backend options. Spell out
that access to the unix or TCP socket is equivalent to command
execution as the serving user, document the unix socket permissions,
and warn that the TCP socket is unauthenticated.
Previously the only way to remove an option from a remote was to set it
to an empty string, which is not the same as deleting it - a present but
empty value overrides the option's default whereas a deleted key
restores it. Editing the file by hand isn't an option for an encrypted
config either.
This adds a "config unset" command and a "config/unset" rc endpoint to
remove one or more keys from an existing remote.
Before this change a streamed multipart upload wrote its parts
straight to the object's final path on the underlying remote. That
meant an in-progress upload overwrote any object already stored under
that name, and aborting or failing the upload destroyed it. The
opposite of the S3 guarantee that an object only changes on a
successful CompleteMultipartUpload.
Remotes that upload atomically already (PartialUploads is false, e.g.
object stores) are safe to stream straight to the destination, so they
still do. Remotes where a partial upload is visible (PartialUploads is
true, e.g. local) now stream the parts to a temporary object instead
and move it, server-side, into its final place only when the upload
completes. A failed or aborted upload then just removes the temporary
object and leaves any pre-existing object untouched.
The temporary-object path needs the remote to support a server-side
move or copy in addition to PutStream uploads fall back to being
buffered in memory as before. The temporary objects are named with a
leading ".rclone_multipart_upload_" and hidden from listings.
Before this change when an aborted upload had overwritten a
pre-existing object of the same name, this left a ghost of that object
in every listing.
This invalidate the VFS cache on all the multipart upload abort paths,
so listings reflect what is actually on the underlying Fs.
A user could reach another user's private repository by sending a path
such as /<me>/../<victim>/config. The authorization check compares the
first path segment against the authenticated user, while the backend
object key was built from the raw, un-cleaned URL path.
Reject any non-canonical request path so the authorization segment and
the backend object key can no longer disagree.
Fixes GHSA-fqj9-69pf-6pjg
Archive entry names are attacker controlled. `rclone archive extract` stripped
only a leading `./` and then joined the entry name onto the destination
directory with `path.Join`, which collapses `..` segments. An entry such as
`../escaped.txt` extracted into `:s3:bucket/safe/prefix` therefore resolved to
`bucket/safe/escaped.txt`, outside the selected `prefix` directory - a path
traversal ("Zip Slip") attack that could create or overwrite sibling objects on
any destination remote.
Entry names are now validated before use: a leading `./` is still stripped (tar
archives created with `tar -czf archive.tar.gz .` rely on this), but any entry
with a `..` path component is rejected. Both `/` and `\` are treated as
separators when looking for `..`, as the local backend treats `\` as a path
separator on Windows.
Fixes: GHSA-4vr5-p2gc-h23p
S3 object keys are opaque names that may legally contain `..` segments. `serve
s3` built backend paths with `path.Join(bucket, key)`, which normalised the key
so a request such as `GET /bucket/../root-secret.txt` resolved to a file outside
the selected bucket elsewhere under the serve root. Listing prefixes and
multipart uploads were affected also.
This did not allow reading of files outside the root, but did allow reading of
files in the root which normally aren't visible; only directories are visible as
buckets normally.
Because `serve s3` maps keys to file paths it cannot represent every opaque S3
key, so rather than normalising keys (which would alias distinct keys onto one
file as well as allow traversal) it now rejects any key that is not already in
canonical path form - containing `..`, `.`, `//` or a leading or trailing slash
- with a 400 Bad Request, as MinIO does. Directory listing prefixes are
validated the same way but allow the empty bucket-root prefix and an optional
trailing slash.
Fixes: GHSA-8v25-v8p6-qf7v
The --disable-zip flag was registered manually and was missing from
OptionsInfo, so it could not be set over the rc interface. Move it
into OptionsInfo like serve webdav does, which keeps the command line
flag and also makes it settable via rc.
Per RFC 4918 section 10.6, when the Overwrite header is omitted from a
COPY or MOVE request the resource MUST treat the request as if
Overwrite: T had been sent.
The upstream golang.org/x/net/webdav library mishandles this for MOVE
by checking == "T" instead of != "F", so an absent header is treated
as Overwrite: F and the request fails with 412 Precondition Failed.
Normalise the header to T in the rclone WebDAV server before
delegating to the upstream handler when the client did not send one.
This restores RFC-compliant default behaviour and can be removed once
the upstream fix in golang/go#66059 lands and the golang.org/x/net
dependency is bumped.
Fixes#9496
Seekdir handled only a rewind to offset 0 and returned ENOTSUP otherwise.
The stateless kernel NFS server opens a fresh directory handle and seeks to
the last returned cookie on every readdir continuation, so any listing
spanning more than one readdir batch failed over NFS. dirStream is a
snapshot taken at Readdir time and go-fuse assigns each entry a sequential
offset, so seeking to off positions the stream at index off; off == 0 still
resets to the start, preserving the rewind/re-read behaviour.
Before: ls of a directory that spans more than one readdir batch failed
over NFS with "Unknown error 524".
After: it lists correctly.
Adds TestDirStreamSeekdir covering rewind, mid-stream resume and the EOF
clamp. The full NFS path was validated against a real Linux
nfs-kernel-server export over NFSv3, NFSv4.0 and NFSv4.2.
Fixes#9547
setAttr left attr.Ino unset (0) and the NewInode sites left StableAttr.Ino
unset, so the kernel saw inode 0 while the node identity differed, which
breaks NFS file-handle validation. Set both to the stable VFS inode. The
bazil cmd/mount backend does not hit this because its framework assigns
stable inodes automatically; go-fuse needs them set explicitly.
Before: chmod/chown/truncate on a just-written file through an
NFS-exported mount2 mount failed with ESTALE.
After: they succeed.
Exercising the NFS handle-validation path needs a kernel NFS server, so it
isn't covered by the local vfstest harness; validated against a real Linux
nfs-kernel-server export over NFSv3, NFSv4.0 and NFSv4.2.
#9547
The kernel NFS server creates regular files with MKNOD (it
creates-then-opens, so vfs_create routes through fuse_create -> FUSE_MKNOD
when there is no open intent), but mount2 only implemented Create
(FUSE_CREATE, used by local and SMB clients). Without Mknod every NFS file
creation failed with ENOTSUPP. This mirrors the cmd/mount mknod handler from
#2115.
Before: touch through an NFS-exported mount2 mount failed with ENOTSUPP.
After: files create normally.
Exercising this needs a kernel NFS server, so it isn't covered by the
local vfstest harness; validated against a real Linux nfs-kernel-server
export over NFSv3, NFSv4.0 and NFSv4.2.
#9547
After an upload (notably multipart) to a slow backing remote, the file
lives in the VFS and is returned by ListBucket, but node.DirEntry() stays
nil until the --vfs-write-back writeback completes. HeadObject and
GetObject returned gofakes3.KeyNotFound while it was nil, so a HEAD/GET in
that window 404'd even though the object existed.
getFileHashByte already falls back to hashing the VFS cache when the
backing object isn't available yet. Drop the early nil return, pass the
node (not the fs.Object) to getFileHashByte, and take the Content-Type
from fs.MimeTypeFromName when the backing object isn't there yet.
The Cobra generated PowerShell completion script captures rclone's output
through a pipeline with Invoke-Expression. PowerShell decodes that output
using [Console]::OutputEncoding, which on non-UTF-8 hosts (for example
PowerShell 5.1 on a Windows install with an OEM code page such as CP852)
misinterprets the UTF-8 bytes rclone emits and corrupts remote and path
names containing non-ASCII characters, so tab completion produces a path
that does not exist.
Inject "[Console]::OutputEncoding = [System.Text.Encoding]::UTF8" into the
generated script immediately before the Invoke-Expression call. This is safe
on PowerShell 7+, where UTF-8 is already the default. If the expected line is
not present (for example after a Cobra template change) the script is emitted
unmodified so we never produce a corrupted completion script.
TestS3Minio brings up a minio container via the fstest/testserver
framework, which exec's bash init.d scripts that shell out to docker.
This is not available on all platforms - Windows has no POSIX shell to
run the scripts, and macOS CI runners have no docker daemon - which
caused the build to fail there.
Add testy.SkipUnlessDocker to detect whether the framework can run and
skip the test when it cannot.
Previously serve s3 buffered every part of a multipart upload in memory
(in the gofakes3 S3 library) and concatenated them when the upload
completed, so memory use grew with the size of the upload.
serve s3 now streams the parts, in part-number order, into a single
PutStream upload to the underlying remote, which performs its own upload
with bounded memory. The whole file is never held in memory - memory use
is bounded by the parts in flight. This works for any remote that
supports PutStream (nearly all, including crypt) and for any part size,
so clients that don't produce uniform-sized parts (for example
PostgreSQL backup tools such as pgBarman and pgBackRest) work too.
Parts must arrive in ascending, contiguous part-number order; parts
uploaded out of order are buffered until their turn, and there is no
per-part retry (a failure aborts the whole upload). These trade-offs are
documented.
Passing --disable-multipart-streaming, or using a remote without
PutStream, reverts to buffering the parts in memory (the previous
behaviour); a one-off NOTICE is logged the first time this happens.
Fixes#7453
Run still uses a fresh local directory as the backing Fs that the
server wraps. RunWithBackend takes an extra remote name (e.g.
"TestS3Minio:") and uses a random subdirectory of that remote instead,
starting the matching fstest/testserver/init.d script on the way in
and tearing it down on the way out.
AuthProxy is only run for the local backend.
Before this change, if the user changed their password or public-key
and the auth proxy script returned updated config parameters for the
backend (eg a rotated api_key) rclone would continue to re-use the old
backend with the old config parameters out of the fscache.
This was because both the VFS cache and the fs/cache key were derived
from the user name only, so a change in the user's password or
public-key did not invalidate the cached backend.
Fix this by deriving the cache key from the user plus a hash of the
password/public-key, so a credential change forces a fresh backend.
The hash uses a per-process random HMAC key so the fragment that
appears in logs cannot be brute-forced offline.
Tests cover Dir size/count accounting, AttrI behaviour for files and
directories, error propagation, and Remove correctly updating ancestor
totals up the tree.
With cmd/mount2, reading a directory more than once returned the correct
entries on the first read but nothing on subsequent reads. Plain `ls`
triggers this: it does lseek(fd, 0, SEEK_SET) to rewind the directory
before a second getdents.
go-fuse v2.9.0 rewinds a directory stream by calling Seekdir on the
FileSeekdirer interface. dirStream did not implement it, so go-fuse
returned ENOTSUP and produced an empty listing on every read after the
first.
This implements Seekdir on dirStream: a rewind to offset 0 resets the
stream to the start, restoring correct listings on re-read. Non-zero
offsets are uncommon for in-memory listings and still return ENOTSUP,
matching go-fuse's own default. A compile-time interface assertion is
added so signature drift on future go-fuse updates is caught at build
time.
Before: second and subsequent reads of a directory returned no entries.
After: directories list correctly on every read.
See: https://github.com/hanwen/go-fuse/issues/549
Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
When a SETSTAT request set the access and modification times, the
handler passed the modification time for both, discarding the
requested access time. Pass the requested access time through instead.
The VFS currently ignores the atime, but it might use it one day.
The statvfs@openssh.com extension was advertised but returned an
unsupported status, so clients couldn't query the amount of free and
used space. Implement it using the VFS Statfs method, which reports the
backend's usage where the backend supports About.
The SFTP serve handler ignored the size attribute of SETSTAT/FSETSTAT
requests, only acting on the modification time. This meant a client
asking to truncate a file (eg setting the final size of an upload, or
an explicit truncate) had no effect at all.
This respects the size attribute (if present) by truncating the file
to the requested size.
The SFTP serve write handler always opened files with O_TRUNC,
ignoring the flags requested in the SFTP OPEN packet. Some clients
(notably WinSCP's "Process in Background", which resumes an upload on
a second connection) re-open the partially written file without the
truncate flag and continue writing from the offset they had reached,
relying on the existing data being preserved. Forcing O_TRUNC zeroed
that prefix, so the start of the uploaded file ended up as a block of
zero bytes.
This fix respects the requested open flags instead so a resume open
without truncate keeps the already written data intact.
See: https://forum.rclone.org/t/rclone-serve-sftp-winscp-background-mode-uploading-causes-file-corruption/53841
Lets the kernel id-map a mount2 mount into a user namespace
(e.g. Kubernetes pods with hostUsers: false). Off by default;
requires Linux 6.12+ and implies default_permissions.
Previously the Mount RPC ignored the path component of the mount
request, so `server:/sub/dir` and `server:/` both landed at the root
of the served remote. The Mount handler now cleans the requested path
with path.Clean, looks it up in the VFS and serves a billy.Filesystem
rooted at that directory, refusing the mount if the path does not
exist or is not a plain directory.
A pathRewriter cache wraps the inner handle cache so that the same
file always produces the same NFS file handle regardless of which
mount minted it (and stable across server restarts for the disk and
symlink caches). This matches the traditional NFS expectation that a
subpath mount behaves like `cd` into a subtree.
nfsmount gains a --nfs-mount-path flag (default /) so clients can
select a subpath at mount time. This replaces a latent misuse of
--volname as the NFS mount path that was previously masked by the
server ignoring it.
Fixes#9442
Leon BrocardandNick Craig-Wood
Hakan İSMAİLandGitHub
Acts1631andGitHub
GiridharandNick Craig-Wood
KyueandNick Craig-Wood
FTCHDandNick Craig-Wood
Mikel Olasagasti UrangaandNick Craig-Wood
Sanjay SanthanamandGitHub
blackflytechandNick Craig-Wood
Sandy LuppinoandNick Craig-Wood
happysnakerandGitHub
maxandNick Craig-Wood
Yash AnilandNick Craig-Wood
Valerij FredriksenandNick Craig-Wood