When an upload failed part way through with a retryable error (eg a
502 from the block storage servers) the pacer retried the whole upload
call with the same input stream. The stream had already been partially
consumed, so the retry re-created the upload draft and committed just
the remainder of the stream as a complete file, silently truncating
it. With restic over serve restic this corrupted the repository as the
truncated pack was reported as successfully uploaded.
This fixes it by using CallNoRetry for the upload, as the other
backends do, so retryable errors are returned wrapped in a RetryError
for the caller to retry the upload with a fresh stream.
Update (which Put also uses) kept the caller's full precision modtime
in memory while the server stores second precision, so until the
object was re-read the modtime did not match what a fresh listing
would report. This broke wrappers which compare exact modtimes, such
as the hasher backend's fingerprint and the VFS cache. Truncate the
modtime to second precision to match what the server stores.
shouldRetry treated every Code=200501 API error as a transient storage
block error and retried it. Proton also returns Code=200501 with an
HTTP 422 for permanent validation failures (e.g. a content key packet
that cannot be verified, or an upload format the account is not
enabled for), so these were retried until the operation timed out
causing a multi-minute hang.
This fixes it by only retrying Code=200501 when it is not a permanent
client (4xx) error.
shouldRetry was a stub returning false unconditionally, which makes
protondrive the only rclone backend that disables pacer-level retries
entirely. Every other backend at minimum falls back to
fserrors.ShouldRetry(err) so genuine transport-level transients (TCP
resets, brief 5xx) get retried.
- Use errors.As to unwrap proton.APIError instead of string matching
- Retry transient storage block errors (Code=200501)
- Retry server errors (5xx, except 503)
- Skip 429 and 503 (handled by go-proton-api's resty retry layer
via catchTooManyRequests / catchRetryAfter, which honours Retry-After)
- Fall back to fserrors.ShouldRetry for non-API errors
Co-authored-by: tomholford <tomholford@users.noreply.github.com>
When a Proton Drive file has no active revision attributes,
readMetaDataForLink returns a nil FileSystemAttrs and Object.originalSize
is left as nil. Object.Open then dereferenced this nil pointer when
calling fs.FixRangeOption, causing a SIGSEGV during copy.
Use Object.Size() instead, which already implements the correct fallback
to the link size when originalSize is unavailable.
This updates the github.com/rclone/Proton-API-Bridge package to fix a
segfault when reading files with no metadata.
Fixes#9377Fixes#9117
Previously all log output produced by Proton-API-Bridge (stdlib log)
and go-proton-api (logrus + resty's logger) bypassed rclone's
logging: it ignored -v / -vv levels and didn't reach --log-file.
Add a small adapter implementing the resty.Logger / bridge Logger
shape that calls fs.Errorf / fs.Logf / fs.Debugf, and pass it via
the new Config.Logger hook. The bridge in turn forwards the same
value to go-proton-api's WithLogger option, so HTTP-layer warnings
and the formerly-hardcoded logrus warnings inside go-proton-api
also surface through rclone's log levels.
The Proton Drive backend constructed the upstream Proton-API-Bridge
without ever passing rclone's HTTP transport. As a result none of
rclone's HTTP flags reached Proton: --dump headers, --dump bodies,
--no-check-certificate, --user-agent, --bind, --ca-cert, --header,
--tpslimit etc. all silently did nothing for this remote, and HTTP
traffic was invisible to -vv.
Pass fshttp.NewTransport(ctx) through the new Config.Transport hook on
the bridge, which forwards it to the updated go-proton-api's
WithTransport option and so to the underlying resty client.
This updates rclone to use forks of the upstream proton drive modules
in preparation for making changes.
The go-proton-api modules has had changes from master merged so rclone
and Proton-API-Bridge are using the same version.
This change removes redundant calls to the Proton Drive Bridge when
creating Objects. Specifically, the function List() would get a
directory listing, get a link for each file, construct a remote path
from that link, then get a link for that remote path again by calling
getObjectLink() unnecessarily. This change removes that unnecessary
call, and tidies up a couple of functions around this with unused
parameters.
Related to performance issues reported in #7322 and #7413
Nick Craig-Wood
William TangeandGitHub
tdaweandGitHub
MicroscotchandGitHub
Lawrence MurrayandGitHub
Michał DzienisiewiczandNick Craig-Wood
Chun-Hung TsengandGitHub