The TransformFile tests in fs/sync call operations.TransformFile
immediately after MoveDir. On eventually-consistent backends the
internal NewObject lookup can momentarily fail with "object not
found", making the tests flaky.
This wraps the two operations.TransformFile calls in TestTransformFile
and TestManualTransformFile with fstest.Retry
When --fix-case was used (e.g. by bisync) on backends that can't set
modification times in place - such as Dropbox - files whose content
matched but whose modtimes differed would fail to rename with a
"from_lookup/not_found" error and abort the operation.
This happened because operations.NeedTransfer was called before the
fix-case rename. NeedTransfer's equality check would delete the
destination as a precursor to re-uploading it (the standard way to
update a modtime on these backends), so by the time the rename ran the
file no longer existed on the remote.
Fix by running the fix-case rename first, so that any subsequent
delete/re-upload happens at the correctly-cased destination path.
See: #8881
The test set the short idle timeout before creating the test Fs, which
made fs.NewFs fail to read the FTP welcome banner within 1s on slow CI
hosts. Restore the long timeout while NewFs dials the control
connection, then apply the short idle timeout before the upload so the
data connection still exercises the close race that shut_timeout fixes.
The previous wording "Already have a token - refresh?" was misleading
because answering yes triggers a full re-authorization flow, not an
OAuth2 refresh token grant. Updated to "Token already configured -
replace it?" to accurately describe what happens.
Also updated the SugarSync backend which has its own copy of the prompt,
and the docs for box, drive, and onedrive that reference it.
Enable on-the-fly response compression for WebDAV when the client sends
Accept-Encoding and the response content type is suitable for
compression.
This adds compression for the WebDAV responses that benefit most in
practice, notably PROPFIND XML responses and text file downloads.
I tested this with Cyberduck, which sends
`Accept-Encoding: gzip,deflate` and accepted the compressed responses.
Range requests are explicitly left uncompressed.
Fixes#5777
Before this change, the GUI server sent all static files uncompressed,
meaning the browser had to download the full size of every JS, CSS,
and HTML asset.
After this change, the GUI server uses chi's Compress middleware at
level 5, which negotiates gzip or deflate encoding based on the
client's Accept-Encoding header.
This reduces transfer sizes significantly for the web UI assets, for
example assets/index-CvfdU_RR.js is 874 KB uncompressed, and
265 KB compressed.
This is consistent with how rclone serve http, webdav, and restic
already compress their responses.
The stscreds.AssumeRoleProvider from AWS SDK Go v2 does not cache
credentials by itself. The SDK only auto-wraps providers with
aws.CredentialsCache when they are loaded via
config.LoadDefaultConfig; when assigned directly to
aws.Config.Credentials it must be wrapped manually, as documented on
stscreds.NewAssumeRoleProvider.
Without the cache, configurations using role_arn would call AssumeRole
once per S3 request, flooding STS and CloudTrail.
See: https://forum.rclone.org/t/aws-iam-roles-credentials-arent-cached/53732
When a Proton Drive file has no active revision attributes,
readMetaDataForLink returns a nil FileSystemAttrs and Object.originalSize
is left as nil. Object.Open then dereferenced this nil pointer when
calling fs.FixRangeOption, causing a SIGSEGV during copy.
Use Object.Size() instead, which already implements the correct fallback
to the link size when originalSize is unavailable.
This updates the github.com/rclone/Proton-API-Bridge package to fix a
segfault when reading files with no metadata.
Fixes#9377Fixes#9117
Previously all log output produced by Proton-API-Bridge (stdlib log)
and go-proton-api (logrus + resty's logger) bypassed rclone's
logging: it ignored -v / -vv levels and didn't reach --log-file.
Add a small adapter implementing the resty.Logger / bridge Logger
shape that calls fs.Errorf / fs.Logf / fs.Debugf, and pass it via
the new Config.Logger hook. The bridge in turn forwards the same
value to go-proton-api's WithLogger option, so HTTP-layer warnings
and the formerly-hardcoded logrus warnings inside go-proton-api
also surface through rclone's log levels.
The Proton Drive backend constructed the upstream Proton-API-Bridge
without ever passing rclone's HTTP transport. As a result none of
rclone's HTTP flags reached Proton: --dump headers, --dump bodies,
--no-check-certificate, --user-agent, --bind, --ca-cert, --header,
--tpslimit etc. all silently did nothing for this remote, and HTTP
traffic was invisible to -vv.
Pass fshttp.NewTransport(ctx) through the new Config.Transport hook on
the bridge, which forwards it to the updated go-proton-api's
WithTransport option and so to the underlying resty client.
Avoid side effects by using own logger instance
- Importing fs/log only sets rclone's private logger via fs.SetLogger,
so internal rclone logging works from the moment the package is
imported but the process-wide slog default is left untouched.
- slog.SetDefault and slog.SetLogLoggerLevel move into InitLogging,
which is called explicitly from the CLI (cmd/cmd.go), the librclone
wrapper and the integration test framework. So rclone-as-a-program
keeps capturing log.Print/log.Fatal and slog.Default() output as
before.
Library consumers that import fs/log without calling InitLogging now
keep their own slog default and can safely route rclone output back
into it via log.Handler.SetOutput without recursing.
Fixes#8907
Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
The S3 ListObjects response from `rclone serve s3` was sorting object
contents by modification time instead of object key. This made the
listing order incompatible with S3 clients which expect lexicographic
key ordering.
In particular, `aws s3 sync` assumes both source and destination
iterators are ordered by key. With the old modtime ordering it could
misidentify files as missing or outdated and re-download objects that
were already up to date.
Change the pager to sort returned objects by key and add a regression
test which uses keys and modtimes arranged so the old behaviour would
fail.
Fixes#9002
Previously `make fetch-gui` extracted the GUI release into cmd/gui/dist/
and the unpacked tree was embedded uncompressed via `//go:embed dist`.
This commits and embeds the GUI bundle (dist.zip) and its release tag
(dist.tag) to the repo so:
- the rclone binary is smaller
- `go build` works on a fresh clone without first running fetch-gui
- a given commit pins an exact GUI version
The "Fetch GUI" step was removed from .github/workflows/build.yml.
next_page is not currently being returned on listings which is causing
the rclone listing code to go wrong. This was returned so is likely a
regression in Drime.
This changes the page counter to calculate using current_page and
last_page. last_page on the first page request is just current_page+1.
drime appears to be capping per_page to 200. as more pages are
requested, last_page increments by 1 until current_page = last_page
Bump go-proton-api and Proton-API-Bridge to versions that send the new
NameSignatureEmail field and omit NodePassphraseSignature/SignatureEmail
for ordinary nodes, matching the schema accepted by the Proton Drive
API. Without this rclone moveto, --backup-dir, server-side rename and
DirMove all failed with "value cannot be empty" / "outdated app" 422.
Fixes#8512
Add read-only iCloud Photos support to the existing iclouddrive
backend via `service = photos` config option.
Also includes auth improvements on top of #9209's SRP authentication.
**Photos features:**
- 3-level hierarchy: libraries (Personal + Shared Photo Library) →
albums → photos/videos
- server-side smart albums (All Photos, Videos, Favorites,
Screenshots, Live, Bursts, Panoramas, Slo-mo, Time-lapse, Portrait,
Long Exposure, Animated, Hidden, Recently Deleted)
- User-created albums and nested album folders
- Live Photo `.MOV` companions as first-class entries
- Edited photo versions (`-edited` suffix) and RAW alternatives
- Duplicate filename dedup for camera counter wrap collisions
- Parallel cold listing for large albums
- Delta sync via CloudKit `changes/zone` - warm listings near-instant from disk cache
- Disk cache (libraries, albums, photos) with atomic writes for crash safety
- `ChangeNotify` support for FUSE mounts via `changes/zone` polling
- `ListR` support for `--fast-list` and recursive operations
- `--metadata` support - width, height, added-time, favorite, hidden
- Fresh download URLs per file - no stale URL failures on long copies
- FUSE mount documentation with recommended flags
**Auth improvements over #9209:**
- SMS 2FA fallback for users without trusted Apple devices
- Explicit push notification request - fixes iOS/macOS 26.4+ where 409
no longer auto-pushes
- Thread safety for concurrent FUSE callers (mutexes on session and client state)
- Session endpoint caching - skips ~5s `/validate` round-trip on warm start
- `Disconnect` support - clears auth state + disk cache
- PCS cookie support for Advanced Data Protection accounts, including
trusted-device approval for PCS cookies
Built on @coughlanio's Photos PoC (Closes#8734) and @mikegillan's SRP auth (#9209).
Fixes#7982
Co-authored-by: Chris Coughlan <chris@coughlan.io>
On Windows, passing "*" as mountPoint to the mount/mount RC command
auto-assigns a drive letter (e.g. "Z:"), but the resolved letter was
never propagated back to mountlib. This caused liveMounts to be keyed
on the literal "*", breaking tracking of multiple mounts and making
unmount unreliable.
Change MountFn to return the actual mount point as an additional
return value. Update MountPoint.Mount() to store the resolved value,
and mountRc() to use it as the liveMounts key. The mount/mount RC
response now returns the actual mountPoint so callers can discover
which drive letter was assigned.
The background kicker goroutine had a bare select outside a for loop,
so the 5s ticker fired at most once before the goroutine exited. The
intent was to run every 5s for the lifetime of the Downloaders.
This wraps the select in a for loop so the ticker fires repeatedly
until ctx is cancelled.
In practice this was benign because every downloader exit and every
successful Write already calls kickWaiters, so the background kicker
is only load-bearing when a waiter is queued, no downloader is
running, and _ensureDownloader failed transiently. In that state,
before this fix, the waiter would hang until another Download() call
or Close() arrived; now it gets retried every 5s and will either
recover or accumulate enough errors to trip maxErrorCount and error
out cleanly.
Nick Craig-Wood
KTibowandNick Craig-Wood
Leon BrocardandGitHub
Gustavo V. F.andGitHub
John VolkandGitHub
Yakov TillandGitHub
dlaumenandNick Craig-Wood