Mounted files showed up owned by 4294967295 (^uint32(0)) on OpenBSD.
vfsflags_unix.go, which calls unix.Geteuid()/unix.Getegid() to get the
real uid/gid, only builds for linux, darwin and freebsd; OpenBSD fell
through to vfsflags_non_unix.go's zero-value stub instead.
OpenBSD has no linux/darwin/freebsd-specific fields here, just the same
POSIX Geteuid/Getegid/Umask calls golang.org/x/sys/unix already ships
for openbsd on every arch, so this just adds openbsd to both build tags
rather than needing a separate file.
Cross-compiled for GOOS=openbsd (arm64, amd64) and go vet clean; ran the
existing vfs test suite on darwin, no regressions. Not yet verified on a
live OpenBSD mount, only that the right uid/gid syscalls now get called
in this codepath.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The mount served fine on Linux and macOS but *BSD kernel NFS clients
refused it with "Authentication error". The MOUNT reply only ever
offered AUTH_NULL, and the OpenBSD/FreeBSD/NetBSD clients won't mount a
server unless AUTH_UNIX is among the offered flavors.
Add AUTH_UNIX to the advertised list. The server still doesn't inspect
the credential (there's no per-user access control here) so the AUTH_UNIX
cred the client then sends is read as an opaque blob and ignored, exactly
as the AUTH_NULL one was. No behaviour change for existing Linux/macOS
clients, and OpenBSD now mounts and reads files over the share.
Verified on a real OpenBSD 7.9 arm64 VM: registered rclone's server in
portmap and ran "mount_nfs -T localhost:/ /mnt"; the mount now succeeds
(MOUNT and GetAttr RPCs go through) and files read back correctly through
the mount, where before it stopped at the auth stage.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The previous OpenBSD path built the right options ("-o port=N -T") but
still handed them to mount(8). On OpenBSD "-T" is a mount_nfs(8) flag,
not a mount(8) one, so mount rejected it with "mount: unknown option --
T" and the mount never ran. Call mount_nfs(8) directly on OpenBSD; the
options are already in its native syntax.
Verified on a real OpenBSD 7.9 arm64 VM: with this change the command
becomes "mount_nfs -o port=N -T localhost:/ <mnt>" and mount_nfs accepts
the flags. The mount then fails later with "Stale NFS file handle"
because the OpenBSD client and rclone's in-process NFS server disagree
on the root filehandle - that is a separate issue in the NFS server,
not in the mount options this PR is about.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
nfsmount built its mount options for GNU/Linux syntax unconditionally:
"-o port=N", "-o mountport=N" and "-o tcp". OpenBSD's mount_nfs(8)
rejects "-o mountport" outright ("option not supported", per the
reporter's log) and has no "tcp" suboption either, since it selects
TCP with the separate "-T" flag instead of an -o suboption.
Add a runtime.GOOS == "openbsd" branch that builds the option list
OpenBSD's mount_nfs actually accepts: "-o port=N" plus "-T" for TCP,
with no mountport option since OpenBSD's mountd is located via
portmap rather than a fixed, settable port. This follows the same
GOOS-branching pattern already used in this file's unmount function
(darwin) and in cmd/cmount/mount.go for openbsd/freebsd differences.
FreeBSD's mount_nfs(8) documents "port=", "mountport=" and "tcp" as
-o suboptions identical to Linux, so the existing option set is left
unchanged for freebsd and all other platforms.
Verified by cross-compiling (go build and go vet) for GOOS=openbsd,
freebsd, linux and darwin, all of which succeed. Actually mounting
via mount_nfs on OpenBSD needs a BSD machine to confirm at runtime,
which wasn't available here.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Runner image ubuntu24/20260726.254 updated podman from 4.9.3 to 5.8.4
which is installed from the podman-static bundle. This bundle ships a
non-setuid fusermount3 in /usr/local/bin which shadows the setuid
/usr/bin/fusermount3 from the fuse3 package (as /usr/local/bin comes
first in PATH). A non-setuid fusermount3 cannot mount as an
unprivileged user, so the cmd/mount and cmd/mount2 tests failed with:
fusermount3: mount failed: Operation not permitted
Fix by removing the podman bundled fusermount binaries so the distro
provided setuid one is used.
The session close flag was only computed after each append, so a
known-size upload which fits in a single chunk sent all its data and
then issued a zero-payload append purely to close the session - one
wasted round trip per small file on the default batched upload path.
Set the close flag before the first append when the size is known to
fit in one chunk.
Since batch mode became the default all uploads go through
uploadChunked, which allocated a full chunk-size retry buffer (48 MiB
by default) regardless of the file size. With the `--transfers 32`
recommended for small file uploads that is ~1.5 GiB of buffer to
upload tiny files.
Size the buffer to the file size when it is known and smaller than a
chunk.
Before this change, if an object compressed with "Content-Encoding:
gzip" was downloaded, a length and hash mismatch would occur since the
go runtime automatically decompressed the object on download, giving
errors like
corrupted on transfer: sizes differ
This change sets "Accept-Encoding: gzip" on all requests which stops
the go runtime decompressing objects on download, so compressed
objects are downloaded as-is with intact size and hash information.
If --oos-decompress is set then rclone will decompress objects with
"Content-Encoding: gzip" as they are received, at the cost of not
being able to check the length or the hash of the downloaded object.
Fixes#9694
DirMove returned fs.ErrorDirExists for any error from the destination
existence check not just when the destination actually exists. That
made sync silently fall back to file-by-file moves and masked the real
failure.
Return the underlying error instead when the check fails for any other
reason.
DirMove checked whether the destination exists using the raw path but
performed the rename with the encoded path, so for directory names
needing encoding (trailing spaces or periods, characters like
\* ? : < > | " or a literal backslash) the existence check looked at
a different server path than the rename used.
If revealing the password, creating the Kerberos client or the SMB
handshake failed after the TCP connection was established, the
connection was never closed.
A new KerberosFactory was constructed for every dial, so the client,
error and ccache modification time caches it holds were discarded
after a single use. Every new SMB connection re-read the Kerberos
config, re-parsed the ccache and did a fresh KDC exchange.
Share a single factory so clients are cached across connections as
intended, and refreshed when the ccache file changes.
Object.Open dereferenced the response's ContentLength pointer without checking
it. The OCI SDK leaves ContentLength nil when the server replies without a
Content-Length header or ContentRange which caused a nil pointer panic.
Now the size is only updated when the response actually provides one, leaving
the size from the object metadata in place otherwise.
This also fixes the same potential problem in the newObject code.
Previously, GET requests for a directory URL always returned an HTML
directory listing. There was no way to suppress this, unlike
`serve webdav` which has supported --disable-dir-list since #4191.
This adds the same flag to `serve http`. When set, GET requests for
directory URLs return 404 instead of a listing, while file downloads
continue to work normally.
Based on the approach suggested in #6306.
Fixes#4000
The multipart upload complete endpoint returns 200 with an empty body,
but rclone tried to decode that body as JSON, failed with EOF and
retried until the retries ran out, so every upload failed even though
the server had actually completed it.
Fixed by not attempting to decode the response body.
This was a regression introduced in
a4972de505 shade: retry server errors instead of failing the transfer
which started treating the JSON decode error as fatal where previously
it was accidentally ignored.
This resolves an issue where mount filters supplied to the rc API
(such as `_filter` in remote control requests) were ignored during
FUSE mounts.
By passing the request context containing the parsed filter config to
`vfs.New`, the VFS layer now correctly respects the active filter
rules.
Fixes#8838
The march janitor goroutine, which discards queued jobs when the context is
cancelled, only ever returned on context cancellation. A march that finished
normally never cancels its context, so on an async rc job (whose context
descends from context.Background and is only cancelled by job/stop) the
janitor parked forever, leaking one goroutine per run and pinning that run's
directory listings in memory. A long-running rcd driving async sync or bisync
jobs accumulated these until it ran out of memory.
Signal the janitor to exit once the march completes so it returns on both
normal completion and cancellation.
Since around mid-2026 Apple's idmsa endpoints `POST
/verify/trusteddevice/securitycode` and `POST /verify/phone/securitycode`
return HTTP 409 (instead of 2xx) even when the submitted code is accepted:
the response body carries `"securityCode": {..., "valid": true}` and the
response headers include a fresh X-Apple-Session-Token, scnt and
X-Apple-Auth-Attributes, which are only issued on successful validation.
rclone treated any 409 as failure and aborted before TrustSession, so
configuring an iclouddrive remote always failed after the 2FA step with:
validate2FACode failed: HTTP error 409 (409 ) returned body:
"{... \"securityCode\": {\"code\": \"...\", \"valid\": true} ...}"
Treat a 409 response that carries X-Apple-Session-Token as success: absorb
the session headers and continue to TrustSession. Applies to both the
trusted-device and SMS validation paths.
Fixes#9488Closes#9534
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Add a new "list_parallelism" advanced option (requires use_arrow_list) which
splits a directory's blob-name keyspace into ranges and lists them
concurrently using the Arrow startFrom/endBefore range parameters. Both
recursive (ListR) and single-directory listings benefit.
The keyspace is sharded with a single-case (digit+lowercase) character
ladder. Single case is deliberate: the service validates startFrom /
endBefore case-insensitively (while listing case-sensitively), so a
mixed-case boundary pair is rejected with a 400; its endBefore filtering is
byte-ordered, so the listing is exact.
The service only honours endBefore on the Arrow listing path of accounts with
Blob Listing with Apache Arrow enabled: elsewhere it either rejects the
request with 400 OperationNotSupportedWithFeatureMissing or, if it fell back
to XML, would ignore the shards' upper bounds. Both cases are detected and
the parallel listing falls back to a sequential listing.
Listing 100,000 objects from a well-connected server:
config time vs XML
xml 23.9s 1.0x
use_arrow_list 6.9s 3.5x
list_parallelism=2 4.4s 5.5x
list_parallelism=4 2.8s 8.4x
list_parallelism=6 2.2s 10.9x
list_parallelism=10 1.5s 15.4x
list_parallelism=20 1.4s 17.6x
list_parallelism=30 1.1s 21.7x
So Apache Arrow listing alone is ~3.5x faster than XML, and parallel listing
takes that to ~20x. The ladder caps at 37 shards, so the benefit plateaus
beyond list_parallelism=30.
The option is hidden from the configurator, command line help and docs
until the feature is officially released.
Add a new "use_arrow_list" advanced option which fetches directory listings
using the ListBlobs Apache Arrow response format instead of XML. This can be
substantially faster for very large containers.
This is EXPERIMENTAL and requires the "Blob Listing with Apache Arrow" preview
feature to be enabled on the storage account. It is not supported on accounts
with a hierarchical namespace (ADLS Gen2). If the feature is not enabled the
service returns XML and the listing transparently falls back to it. Connection
string auth is not supported - it also falls back to the normal XML listing.
The option is hidden from the configurator, command line help and docs until the
feature is officially released.
This adds a temporary subpackage implementing the experimental "Blob
Listing with Apache Arrow" feature on top of the released Azure azblob
SDK.
The Azure SDK for Go supports Arrow listing on its unreleased
feature/storage/bifrost branch (commit c6fa341ca22b) but that can't be
used in a release. This package exposes the same options and pager
interface as the experimental SDK, built on the released SDK's public
API plus verbatim copies of its Arrow decoder and of the two internal
auth policies (shared key signing and storage bearer challenge) which
the released SDK does not export. Copied files carry MIT attribution
headers pinned to their upstream source.
When Arrow listing ships in a released azblob the whole package will be
deleted and callers pointed back at the SDK - the interface is
deliberately source compatible to make that a mechanical change.
Also exports auth.Transporter so the new package's pipeline can use
rclone's fshttp transport, and adds github.com/apache/arrow-go/v18 as a
direct dependency for decoding the Arrow IPC stream.
Backend options with Hide: fs.OptionHideBoth are hidden from the config
wizard and from the command line flag listing, but "rclone help backend"
(and therefore the autogenerated backend documentation) still showed
them. Skip them there too so fully hidden options no longer appear in
the docs.
This updates `mountRc` in `cmd/mountlib/rc.go` to parse options using the
unified `rc.ParseOptions` helper. It also enforces parameter validation by
calling `rc.CheckParamsUsed`.
- Replace custom options parsing with rc.ParseOptions for vfsOpt and mountOpt.
- Delete consumed params (mountPoint, mountType, fs) and call CheckParamsUsed
before initiating FUSE mount to reject unknown parameters.
- Clean up duplication tests (TestRcFlatOptions, TestRcFlatOptionsNull) in rc_test.go.
- Update TestRc in rc_test.go to pass a clean params map to unmount.Fn.
This refactors all 8 serve protocols to use `rc.ParseOptions` for VFS
and protocol options decoding. It also implements parameter validation in the
main runner using `rc.CheckParamsUsed`.
- Update dlna, ftp, http, nfs, restic, s3, sftp, and webdav to call
rc.ParseOptions, enabling nested option block support.
- Remove unused configstruct imports.
- Update startRc in cmd/serve/rc.go to copy input parameters and
call rc.CheckParamsUsed to reject unknown parameters.
- Add TestRcStartFlatNestedAndUnknownRejection to cmd/serve/rc_test.go.
- Update inline documentation in serve/start command help to include
nested blocks information (vfsOpt, proxyOpt, opt) and a new WebDAV example.
When an HTTP/2 server retires a connection with GOAWAY after it has
already sent successful response headers, Go's http2 transport fails the
read of the response body with
http2: server sent GOAWAY and closed the connection; LastStreamID=..., ErrCode=NO_ERROR, debug=""
This was not recognised as a retriable networking error, so a transient
connection retirement aborted the whole command instead of consuming a
low level retry. It was reported against a large S3 check, where an
interrupted ListObjectsV2 page made rclone report destination objects as
missing and exit unsuccessfully.
The concrete error type is unexported by net/http, so match on the
message as we already do for the other http2 transport errors.
Rmdir checked that the directory existed with GetMetadata and then
checked it was empty with ListFolder, but ListFolder already reports a
missing path and a path that is a file, so the first request was
redundant.
Drop the GetMetadata call and map the ListFolder lookup errors onto the
same sentinel errors as before, so a missing directory still returns
fs.ErrorDirNotFound and a file still returns fs.ErrorIsFile. Limit is
set to 1 as only the presence of an entry matters, and HasMore is
checked as well so a full first page is not read as an empty directory.
Add Scality as an S3 provider covering both Scality RING (S3 Connector)
and ARTESCA, which share the same CloudServer + Vault S3 implementation.
The only quirk required is force_path_style: both products support
path-style addressing, and virtual-hosted style needs wildcard DNS that
on-prem deployments usually lack.
When --onedrive-av-override is set and the object is malware-flagged,
download via Microsoft Graph beta contentStream then /content with
Prefer: forceInfectedDownload, and keep Prefer (and AVOverride) on the
SharePoint redirect without re-encoding tempauth.
Clean files keep using the stable v1.0 /content path so permanently
enabled av_override does not put all traffic on beta APIs.
AI-assisted change; tested against OneDrive for Business with
application permissions (client_credentials).
The unauthenticated follow-up to /content redirects used Client.Do,
which does not check HTTP status. A 403 malwareDetected JSON body was
therefore written as file content and surfaced only as a size mismatch.
Check the status code, surface malware errors clearly, and avoid
suggesting --onedrive-av-override when it is already set.
The unauthenticated DLNA control endpoint decoded arbitrary SOAP bodies
into an in-memory XML field. A LAN client could send a large request
and exhaust the server's memory.
Limit SOAP request bodies to 1 MiB and return 413 when the limit is
exceeded.
ZERO-Z3 is S3-compatible object storage built on Ceph RADOS Gateway,
hosted in the EU on Zero Services' own network (AS215197), with
region-specific endpoints (zero-fra1, zero-fra2, zero-eyl1).
This commit introduced aux.go which unfortunately is illegal on windows.
e006d7c13f vfs: fix crash when multiple mounts or servers share the same VFS
Rename aux_test.go too to keep the pair together.
Add two new options, pin_host_key and host_keys, that
together provide a TOFU host-key validation mode for users who don't
maintain a known_hosts file. When --sftp-pin-host-key is used, rclone
records the server's host key into host_keys on the first successful
connection and verifies it on every subsequent connection.
host_keys is always validated when non-empty, so it can also be used
by hand to pin a known fingerprint without enabling TOFU writing.
known_hosts_file takes precedence if both are set. SSH host
certificates are rejected with a clear message pointing at
known_hosts_file. On-the-fly remotes log a warning since the captured
key cannot be persisted.
Leon BrocardandNick Craig-Wood
Hakan İSMAİLandGitHub
Yash AnilandNick Craig-Wood
phatlcandNick Craig-Wood
dependabot[bot]andNick Craig-Wood
Anupam MedirattaandGitHub
Dzmitry NianakhauandNick Craig-Wood
ifloppyandNick Craig-Wood
Acts1631andGitHub
Noah ZalevandNick Craig-Wood
Zero Services GmbHandNick Craig-Wood