The Zoho config system ended every interactive create, update and
reconnect by calling m.Set("root_folder_id", workspaceID) in the
workspace_end state, with fs.ConfigChoose defaulting to the first
workspace. Any existing root_folder_id was therefore overwritten and
the remote silently repointed to the first workspace root.
This matters because reconnect is the documented fix for the 401
INVALID_OAUTHSCOPE download error - tokens issued before the
ZohoFiles.files.ALL scope was added lack download access - so users are
told to reconnect and then find all subsequent list/copy/sync/delete
operations pointed at a different, often shared, workspace.
Gate the workspace selection the way the drive backend does for team
drives (#5454): if a root_folder_id is already set, ask "Change current
root folder id ...?" defaulting to No and keep it; only run workspace
selection when it is empty or the user opts in. The token type rewrite
still runs on every reconnect so the scope refresh is unaffected. Also
expose root_folder_id as a standard advanced option (Sensitive, so
config redacted masks it) so it can be set and discovered like on
drive/box/onedrive.
Fixes#9575
Zoho's WorkDrive listing API returns "401 R008 Unauthorized access"
(not a 404) when a folder id no longer resolves to a listable folder,
because it was deleted or never existed. A freshly refreshed token still
gets it, so it is not a token problem and retrying it is futile - and can
escalate to a 429 F7008 rate-limit penalty.
Handle it as a missing directory instead: shouldRetry no longer retries a
bare R008 401, listAll maps it to fs.ErrorDirNotFound, and
readMetaDataForPath flushes the stale parent from the dircache and reports
the object as not found so a later create re-resolves the parent. This lets
the VFS self-heal a stale cached directory id instead of hard-failing the
operation, and stops the VFS integration tests failing on a stale directory id.
Fixes#9578
The /stream/upload endpoint's overwrite flag is the x-prefixed header
x-override-name-exist, matching its siblings x-filename, x-parent_id and
x-streammode. uploadLargeFile sent the un-prefixed override-name-exist, which
the endpoint ignores, so overwriting a file of 10 MiB or larger created a
renamed duplicate instead of updating the existing file. The small upload API
passes the flag as a query parameter and is unaffected.
The folder list limiter granted the caller-supplied time immediately in
the burst phase, so concurrent callers observing time.Now() out of order
could record grant times that moved backwards. The sliding safety log
indexes grants as an ordered history, so out-of-order grants could also
breach the rolling-window cap. Clamp each grant to be at or after the
previous one so grant times are always monotonic.
WorkDrive throttles its listing API (GET files/{id}/files) PER
folder, independently of the overall request rate: at most ~19
listings of one folder are allowed in any rolling ~60s window and
the 20th returns F7008 with a ~300s Retry-After (measured live -
every observed trip landed exactly on the 20th listing inside a
window). fstests re-lists the same working directory after almost
every sub-operation, which is why the integration suite could not
pass.
Add a per-folder listing limiter with a true per-window cap: each
window starts with --zoho-list-folder-burst listings passing
back-to-back (the burst re-arms at every window boundary, so a
sync re-listing one directory a few times never waits), the rest
of the budget is spaced evenly across the window, and a sliding
log of recent listings guarantees no rolling window ever exceeds
--zoho-list-folder-limit (default 19) per --zoho-list-folder-window
(default 60s) for any traffic pattern. The registry is
process-wide and keyed by region+folder id so every Fs instance
shares one budget per physical folder; idle entries are evicted
after a window, which is lossless because Zoho's window has also
cleared by then.
Defaults were validated against the live service: bursts of 4-6
under the 19-per-60s cap ran clean while an over-cap probe tripped
F7008 exactly at the 20th listing, and a full test_all -backends
zoho run passes cleanly.
Fixes#9570
A 429 stall was only visible as a DEBUG pacer line, so without -vv
rclone appeared to hang for 2-5 minutes. In one night's batch logs 17
job starts produced only 4 completions because the silent stalls
looked like hangs and the jobs kept getting killed, re-triggering the
throttle.
Log the first 429 of each throttle episode at NOTICE with the server
message and the wait time. An episode ends when a request succeeds
after the penalty window; retries within an episode stay at DEBUG via
the existing pacer logging. State is two atomics behind a pointer on
Fs, so shallow Fs copies share it and concurrent checkers are safe.
See #9570
Zoho throttling is account/plan-dependent; measurements show a
sustainable listing rate of ~6 requests/s on a production account -
going faster drains a token bucket and stalls ~2 minutes per
Retry-After, which is strictly slower overall. Add per-remote pacer
options (default 6/1) using the same token-bucket pacer as the Google
Drive backend. Set --zoho-tpslimit 0 to disable the cap.
See #9570
Zoho WorkDrive now sends a Retry-After header on 429 (it did not when
the backend was written). Waiting the hard-coded 60s retried too early
when the server asked for more (Retry-After: 299 is common) and the
penalty escalated (observed 84s -> 239s). Honour the header plus a 1s
margin - retrying at exactly Retry-After still finds an empty token
bucket and burns ~16 immediate 429s - and keep 60s as the fallback
when the header is absent.
shouldRetry becomes a method on *Fs so the retry decision has access
to the remote's state; later commits build on this.
See #9570
This commit reorganises the oauth code to use our own config struct
which has all the info for the normal oauth method and also the client
credentials flow method.
It updates all backends which use lib/oauthutil to use the new config
struct which shouldn't change any functionality.
It also adds code for dealing with the client credential flow config
which doesn't require the use of a browser and doesn't have or need a
refresh token.
Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
Zoho has started returning the results from Range: requests with a 200
response code rather than the technically correct 206 error code.
Before this change this triggered workaround code to deal with Zoho
not obeying Range: requests properly.
This fix tests the returned header for a Content-Range: header and if
it exists assumes it is a valid reply to the Range: request despite
the status being 200.
This problem was spotted by the integration tests.
This is possible now that we no longer support go1.12 and brings
rclone into line with standard practices in the Go world.
This also removes errors.New and errors.Errorf from lib/errors and
prefers the stdlib errors package over lib/errors.
This changes the interface to NewObject so that if NewObject is called
on a directory then it should return fs.ErrorIsDir if possible without
doing any extra work, otherwise fs.ErrorObjectNotFound.
Tested on integration test server with:
go run integration-test.go -tests backend -run TestIntegration/FsMkdir/FsPutFiles/FsNewObjectDir -branch fix-stat -maxtries 1
This is a very large change which turns the post Config function in
backends into a state based call and response system so that
alternative user interfaces can be added.
The existing config logic has been converted, but it is quite
complicated and folloup commits will likely be needed to fix it!
Follow up commits will add a command line and API based way of using
this configuration system.
Before this change when the context was cancelled (due to
--max-duration for example) this could deadlock when uploading
multipart uploads.
This change fixes the problem by introducing another go routine to
monitor the context and close the pipe with an error when the context
errors.
This change checks the context whenever rclone might retry, and
doesn't retry if the current context has an error.
This fixes the pathological behaviour of `--max-duration` refusing to
exit because all the context deadline exceeded errors were being
retried.
This unfortunately meant changing the shouldRetry logic in every
backend and doing a lot of context propagation.
See: https://forum.rclone.org/t/add-flag-to-exit-immediately-when-max-duration-reached/22723
Nick Craig-Wood
Erol OzcanandNick Craig-Wood
Martin HassackandNick Craig-Wood
buengeseandNick Craig-Wood
yumeiyinandGitHub
albertony
Josh Sorefandalbertony
YFdyh000andGitHub
buengeseandNick Craig-Wood