Zoho's WorkDrive listing API returns "401 R008 Unauthorized access"
(not a 404) when a folder id no longer resolves to a listable folder,
because it was deleted or never existed. A freshly refreshed token still
gets it, so it is not a token problem and retrying it is futile - and can
escalate to a 429 F7008 rate-limit penalty.
Handle it as a missing directory instead: shouldRetry no longer retries a
bare R008 401, listAll maps it to fs.ErrorDirNotFound, and
readMetaDataForPath flushes the stale parent from the dircache and reports
the object as not found so a later create re-resolves the parent. This lets
the VFS self-heal a stale cached directory id instead of hard-failing the
operation, and stops the VFS integration tests failing on a stale directory id.
Fixes#9578
The /stream/upload endpoint's overwrite flag is the x-prefixed header
x-override-name-exist, matching its siblings x-filename, x-parent_id and
x-streammode. uploadLargeFile sent the un-prefixed override-name-exist, which
the endpoint ignores, so overwriting a file of 10 MiB or larger created a
renamed duplicate instead of updating the existing file. The small upload API
passes the flag as a query parameter and is unaffected.
WorkDrive throttles its listing API (GET files/{id}/files) PER
folder, independently of the overall request rate: at most ~19
listings of one folder are allowed in any rolling ~60s window and
the 20th returns F7008 with a ~300s Retry-After (measured live -
every observed trip landed exactly on the 20th listing inside a
window). fstests re-lists the same working directory after almost
every sub-operation, which is why the integration suite could not
pass.
Add a per-folder listing limiter with a true per-window cap: each
window starts with --zoho-list-folder-burst listings passing
back-to-back (the burst re-arms at every window boundary, so a
sync re-listing one directory a few times never waits), the rest
of the budget is spaced evenly across the window, and a sliding
log of recent listings guarantees no rolling window ever exceeds
--zoho-list-folder-limit (default 19) per --zoho-list-folder-window
(default 60s) for any traffic pattern. The registry is
process-wide and keyed by region+folder id so every Fs instance
shares one budget per physical folder; idle entries are evicted
after a window, which is lossless because Zoho's window has also
cleared by then.
Defaults were validated against the live service: bursts of 4-6
under the 19-per-60s cap ran clean while an over-cap probe tripped
F7008 exactly at the 20th listing, and a full test_all -backends
zoho run passes cleanly.
Fixes#9570
A 429 stall was only visible as a DEBUG pacer line, so without -vv
rclone appeared to hang for 2-5 minutes. In one night's batch logs 17
job starts produced only 4 completions because the silent stalls
looked like hangs and the jobs kept getting killed, re-triggering the
throttle.
Log the first 429 of each throttle episode at NOTICE with the server
message and the wait time. An episode ends when a request succeeds
after the penalty window; retries within an episode stay at DEBUG via
the existing pacer logging. State is two atomics behind a pointer on
Fs, so shallow Fs copies share it and concurrent checkers are safe.
See #9570
Zoho throttling is account/plan-dependent; measurements show a
sustainable listing rate of ~6 requests/s on a production account -
going faster drains a token bucket and stalls ~2 minutes per
Retry-After, which is strictly slower overall. Add per-remote pacer
options (default 6/1) using the same token-bucket pacer as the Google
Drive backend. Set --zoho-tpslimit 0 to disable the cap.
See #9570
Zoho WorkDrive now sends a Retry-After header on 429 (it did not when
the backend was written). Waiting the hard-coded 60s retried too early
when the server asked for more (Retry-After: 299 is common) and the
penalty escalated (observed 84s -> 239s). Honour the header plus a 1s
margin - retrying at exactly Retry-After still finds an empty token
bucket and burns ~16 immediate 429s - and keep 60s as the fallback
when the header is absent.
shouldRetry becomes a method on *Fs so the retry decision has access
to the remote's state; later commits build on this.
See #9570
Erol Ozcan