WriteChunk copied the whole chunk, which lib/multipart already hands over
in a buffer from the global memory pool, into a bytes.Buffer so that
retries could re-send it. That doubled the per-part memory and made a
fresh chunk-sized heap allocation (64 MiB by default) for every part,
times the upload concurrency.
The chunk reader is seekable, so find its size with Seek and rewind it
inside the pacer closure instead, sending the pooled buffer directly.
Also fix the error for a part that fails to upload, which formatted the
buffer instead of the part number.
The multipart upload complete endpoint returns 200 with an empty body,
but rclone tried to decode that body as JSON, failed with EOF and
retried until the retries ran out, so every upload failed even though
the server had actually completed it.
Fixed by not attempting to decode the response body.
This was a regression introduced in
a4972de505 shade: retry server errors instead of failing the transfer
which started treating the JSON decode error as fatal where previously
it was accidentally ignored.
The pacer callbacks only retried 429 responses, so transient server
errors like 502 Bad Gateway failed transfers immediately. Retry the
standard retryable status codes and transport errors everywhere, make
chunk upload retries resend the whole chunk, and fix a potential nil
pointer dereference when logging a bad token response.
Nick Craig-Wood
jhasse-shadeandGitHub