The multipart upload allocated a fresh chunk-sized buffer (64 MiB by
default) plus a 1 MiB scratch buffer per large file, copying every byte
twice, and never returned them to rclone's memory pool.
Buffer each part with multipart.NewRW instead so the memory is reused
across uploads and part of rclone's central memory management.
The pooled buffer is seekable, so a part can now be re-sent.
uploadPart previously had no retry at all and any transient error
failed the whole upload. It is now wrapped in the pacer with the
backend's usual shouldRetry rules, seeking to the start before each
attempt. The body is wrapped in readers.NoCloser so the http transport
can't close the pooled buffer between attempts, and Content-Length is
set explicitly since net/http can't infer it from a pool.RW.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
If the source supplied fewer bytes than its declared size, the
truncated file was stored and the upload reported success with the
object claiming the declared size.
Count the bytes actually read from the source and if they do not match
the declared size, remove the truncated file and return an error.
This was found by the new TestRcatSizeShortEOF integration test.
Nick Craig-Wood
kingston125
kingston125
albertony