Rmdir checked that the directory existed with GetMetadata and then
checked it was empty with ListFolder, but ListFolder already reports a
missing path and a path that is a file, so the first request was
redundant.
Drop the GetMetadata call and map the ListFolder lookup errors onto the
same sentinel errors as before, so a missing directory still returns
fs.ErrorDirNotFound and a file still returns fs.ErrorIsFile. Limit is
set to 1 as only the presence of an entry matters, and HasMore is
checked as well so a full first page is not read as an empty directory.
Add Scality as an S3 provider covering both Scality RING (S3 Connector)
and ARTESCA, which share the same CloudServer + Vault S3 implementation.
The only quirk required is force_path_style: both products support
path-style addressing, and virtual-hosted style needs wildcard DNS that
on-prem deployments usually lack.
When --onedrive-av-override is set and the object is malware-flagged,
download via Microsoft Graph beta contentStream then /content with
Prefer: forceInfectedDownload, and keep Prefer (and AVOverride) on the
SharePoint redirect without re-encoding tempauth.
Clean files keep using the stable v1.0 /content path so permanently
enabled av_override does not put all traffic on beta APIs.
AI-assisted change; tested against OneDrive for Business with
application permissions (client_credentials).
The unauthenticated follow-up to /content redirects used Client.Do,
which does not check HTTP status. A 403 malwareDetected JSON body was
therefore written as file content and surfaced only as a size mismatch.
Check the status code, surface malware errors clearly, and avoid
suggesting --onedrive-av-override when it is already set.
The unauthenticated DLNA control endpoint decoded arbitrary SOAP bodies
into an in-memory XML field. A LAN client could send a large request
and exhaust the server's memory.
Limit SOAP request bodies to 1 MiB and return 413 when the limit is
exceeded.
ZERO-Z3 is S3-compatible object storage built on Ceph RADOS Gateway,
hosted in the EU on Zero Services' own network (AS215197), with
region-specific endpoints (zero-fra1, zero-fra2, zero-eyl1).
This commit introduced aux.go which unfortunately is illegal on windows.
e006d7c13f vfs: fix crash when multiple mounts or servers share the same VFS
Rename aux_test.go too to keep the pair together.
Add two new options, pin_host_key and host_keys, that
together provide a TOFU host-key validation mode for users who don't
maintain a known_hosts file. When --sftp-pin-host-key is used, rclone
records the server's host key into host_keys on the first successful
connection and verifies it on every subsequent connection.
host_keys is always validated when non-empty, so it can also be used
by hand to pin a known fingerprint without enabling TOFU writing.
known_hosts_file takes precedence if both are set. SSH host
certificates are rejected with a clear message pointing at
known_hosts_file. On-the-fly remotes log a warning since the captured
key cannot be persisted.
The connection pacer in getSftpConnection used to retry every error,
so permanent failures (host key mismatch, certificate rejection, auth
failure, etc.) were looped 10 times before reporting to the user.
Switch to using fserrors.ShouldRetry which matches the pattern other
backends use so only genuinely retriable errors (timeouts, EOF,
network blips) are retried and permanent errors are surfaced
immediately.
The VFS is shared between users with the same remote and options, for
example two mounts created over the rc, or a mount and an NFS server.
Each node has a single Sys() slot which mount, mount2 and serve nfs
all used to attach their per-node data. With a shared VFS the users
overwrote each other's data: at best churning the cached FUSE nodes,
and since the slot was an atomic.Value, panicking with "store of
inconsistently typed value" as soon as two users stored different
types on the same node.
This change gives each node auxiliary values keyed by owner, set with
SetAux and read with Aux, so each user of the VFS has an independent
slot. The mounts now cache their FUSE nodes under their own key,
leaving Sys - which is read through the os.FileInfo interface -
reserved for users like serve nfs which need to control what that
returns.
Nodes with nothing attached use less memory than before (one pointer
instead of an atomic.Value) and reads remain lock free.
Bug discovered while thinking about #9617
Streamed multipart uploads had two problems when the client uploaded parts
concurrently:
Parts arriving ahead of the next part needed by the backend stream were buffered
in memory without limit, acknowledging each part as soon as it was received. A
client uploading faster than the backend could drain would therefore balloon the
server's memory to the size of the upload. Buffering is now bounded a new
--multipart-streaming-buffer-limit flag (default 256Mi, 0 for unlimited): a part
that would take the buffer over the limit is not read until the stream drains,
applying backpressure to the client instead of using unbounded memory.
A part uploaded again with the same number - typically a client retrying after
its request timed out - left a stale copy in the reorder buffer which made
CompleteMultipartUpload fail with InvalidPart, aborting the whole upload.
Re-uploaded parts are now handled properly: a copy still in the buffer is
replaced, an identical copy of an already streamed part is accepted as a no-op,
and only replacing an already streamed part with different content (which the
in-order stream cannot honour) is rejected.
Every write operation waits for the server's processing tasks which
currently take minutes each, so the full test suite takes much longer
than the timeout. Running with -short skips the FsEncoding subtests
(which are already on the ignore list) and the chunked upload tests,
removing a large number of these waits.
Multiple security vulnerabilities have been identified and addressed
in grpc-go affecting the xDS RBAC authorization engine
(internal/xds/rbac) and the HTTP/2 transport server implementation
(internal/transport).
This updates to master to add the fix which allows it to compile on
plan9.
Fixes#9640
The auto-derived CORS origin logic (added in #9603 for the wildcard-bind
case) ran unconditionally, so it clobbered an explicitly-passed
--rc-allow-origin too - the flag was silently ignored. That breaks the
documented Docker/remote-browser setup where the bind address and the
address the browser actually reaches the GUI at are different.
Now it only kicks in when the user hasn't set --rc-allow-origin
themselves.
Bump go-diskfs to v1.9.4, which contains diskfs/go-diskfs#413 handling
squashfs images that have no fragment table or no xattr table. On v1.9.3
these images fail to list at all (ReadDir errors, or a nil xattr-table
dereference inside the library).
Adapt the backend to the v1.9.4 io/fs API: ReadDir now returns
io/fs.DirEntry (metadata via DirEntry.Info), and files are opened via
FileSystem.OpenFile instead of the removed FileStat.Open. The path
conversion added in c10eb47 (toIOFS) is reused.
Add a regression test with two committed fixtures built from trivial
placeholder content: an empty directory (no fragment table) and a small
tree whose superblock has the NO_XATTRS flag set while inodes still carry
an xattr index.
Fixes#9004
CloudKit is inconsistent about how it encodes the isDeleted field on
album records, returning a JSON boolean (true/false) for some accounts
and a number (0/1) for others. The numeric form caused listing of a
photo library to fail with:
json: cannot unmarshal number into Go struct field
ckBoolField.records.fields.isDeleted.value of type bool
The encoding also varies over time, not just per account: a full HTTP
dump from the reporting user showed the server sending
"isDeleted" : { "value" : 0, "type" : "INT64" }
but the same account later reverted to the boolean encoding with no
client change. Asset records already deliver isDeleted as a number, so
both encodings are in active use server side and either may appear.
Accept both encodings when parsing CloudKit boolean fields.
See: https://forum.rclone.org/t/error-when-trying-to-list-contents-of-primarysync-directory-in-icloud-photos/54028
The old rclone-integration-test item has accumulated years of catalog
task churn and IA now deprioritises and periodically holds its tasks,
which makes the tests wait on the server's processing tasks for a
very long time. Start again with a fresh item (created with noindex
set so the test files stay out of IA's search index).
The session cache was only populated on a fresh username and password
login, so once a session ID was stored in the config every Fs
instance created its own Mega session with its own copy of the
account's node tree. The server side move code relies on all Fs
instances of a user sharing one session, and with separate sessions a
move between two rclone remotes grafted a node from one tree into
another, where the asynchronous event replays of the two sessions
raced and could detach the moved file from the destination directory
so it disappeared from listings.
Cache the session however the login was done. This also stops every
extra Fs instance re-downloading the whole account node tree.
Sync refuses to delete files when the global error stats are non-zero
so a single backend error in one transform test made every following
transform test in the same test binary fail with "not deleting files
as there were IO errors". Reset the stats at the start of each test.
The pacer callbacks only retried 429 responses, so transient server
errors like 502 Bad Gateway failed transfers immediately. Retry the
standard retryable status codes and transport errors everywhere, make
chunk upload retries resend the whole chunk, and fix a potential nil
pointer dereference when logging a bad token response.
The go-diskfs library now requires io/fs.ValidPath style paths (no
leading slash, "." for the root) so convert paths at the library
boundary. This fixes listing failures and the resulting test hangs
after the update to go-diskfs v1.9.3.
This fixes the code to compile with the updated dependencies:
- squashfs: implement the new Path method required by the go-diskfs
backend.Storage interface, returning an empty string as there is no
underlying path.
- dropbox: convert to and from the SDK's new DBXTime type (an alias
for time.Time) for ClientModified, TimeInvited and Expires.
- dropbox: remove a stray debug fmt.Printf from the shared folder
listing.
- pin go-systemd to v22.6.0 to fix netbsd builds
go-systemd v22.7.0 uses unix.ClockGettime and unix.CLOCK_MONOTONIC
which golang.org/x/sys does not define for netbsd, so the build fails.
See: https://github.com/coreos/go-systemd/issues/512
- pin google.golang.org/grpc to v1.80.0 to fix plan9 builds
grpc v1.81.0 and later use syscall.Errno, syscall.ECONNRESET and
syscall.ECONNABORTED which do not exist on plan9, so the build fails
See: https://github.com/grpc/grpc-go/issues/9253
The /file-entries/duplicate endpoint returns a 500 Server Error for
every request (reported to Drime 2026-06) which made all server side
copies fail after 10 retries. Remove the Copy method so rclone falls
back to downloading and re-uploading instead. It can be restored if
Drime fix the endpoint.
This updates go-mega to fix the handling of the event stream when a
file is moved or renamed. The server reports a move as a delete of
the old node followed by an add of the same node, and go-mega was
losing track of the node's identity in the process, which could make
the old name reappear in listings and leave the backend unable to
remove it.
Microsoft has been progressively disabling sharing invitations on
both Business and Personal accounts - the driveItem invite API
returns 400 sharingFailed for any recipient on affected accounts -
which makes the permission writing tests impossible. Probe the API
once and skip the tests which need it when it is refused.
ownCloud does not carry the checksum over to the destination of a
server side copy and refuses attempts to set it afterwards, so the
destination legitimately has no hash. The logger vs lsf check
compared the predicted hash against the empty hash and failed.
Treat an empty hash in the listing as unknown rather than wrong,
matching how sync itself compares hashes.
ImageKit can't store zero length files. The test only skips itself
when the failure is reported synchronously, but with --vfs-cache-mode
writes the writeback happens asynchronously and deliberately swallows
the error, so the test fails on the missing files. Other backends
which can't store empty files already ignore this test.
Cloudinary's search API which NewObject uses is eventually
consistent, so the --compare-dest lookup of a just uploaded file
misses it and the test fails. TestCopyFileCompareDest is already
ignored for the same reason.
Open decoded range options against an unknown size, producing a
negative offset for suffix ranges (eg the last N bytes) and sending
syntactically invalid Range headers which the server ignored. A
suffix range then returned the whole file instead of the requested
tail.
Decode the options against the known object size so the offset is
always absolute, only send a Range header when one was requested, and
honour the requested count when the server ignores the Range header.
The Linkbox API is fronted by bot protection which under load
intermittently returns an HTML challenge page with a 200 status
instead of JSON. This surfaced as a fatal JSON decode error, and the
web API path misread the empty decoded result as an expired token,
hammering the login endpoint with requests which also failed to
decode and keeping the bot protection triggered.
Treat responses which fail to decode as retryable errors so the pacer
backs off until the block lifts, and only refresh the token when the
response actually parsed as JSON and reported an error.
Yandex Disk stores the modtime in a custom property set just after
the upload completes. The server sometimes silently drops this
property (the request returns success but the property is gone when
read back), leaving the object showing the upload time instead.
Check the modtime read back after upload and set it again if it
didn't stick.
put.io serializes trash operations per account so concurrent deletes
can fail with a 400 TRASH_LOCK_TIMEOUT error saying "There is an
ongoing blocking trash operation". This transient error was treated
as fatal - retry it instead.
Objects looked up by name used the /folder/itembyname.json endpoint
which does not return the file's MD5 hash, so NewObject returned
objects with no hash. Read the metadata with /file/info.json (which
does return the hash) instead, using itembyname only to find the file
id when it isn't known.
Copy and Move returned objects with only the id and size filled in,
leaving the modtime and hash empty, and SetModTime kept nanosecond
precision in memory while the server stores seconds. These made the
returned objects differ from a fresh listing of the same objects.
The Move method compared the source and destination paths relative to
their own Fs roots, so a server side move between two differently
rooted Fs instances on the same remote (as used by --backup-dir and
sync moves between remotes) was skipped as "already there" whenever
the relative paths coincided. With --backup-dir this destroyed the
file which was supposed to be backed up.
Compare the full paths including the Fs roots instead.
dependabot[bot]andNick Craig-Wood
SocialprankerandNick Craig-Wood
Anupam MedirattaandGitHub
Dzmitry NianakhauandNick Craig-Wood
ifloppyandNick Craig-Wood
Acts1631andGitHub
Noah ZalevandNick Craig-Wood
Zero Services GmbHandNick Craig-Wood
p1andNick Craig-Wood
GiridharandNick Craig-Wood
KyueandNick Craig-Wood
maximilizeandNick Craig-Wood
Søren LindbergandNick Craig-Wood
FTCHDandNick Craig-Wood