Each chunk of a chunked upload allocated a fresh buffer of
--hidrive-chunk-size bytes (48 MiB by default), with up to
--hidrive-upload-concurrency of them in flight, so large uploads churned
allocations and GC and a short final chunk still cost a whole chunk.
Buffer chunks with multipart.NewRW from rclone's global page pool
instead, sized to the data actually read, and return each buffer to the
pool once its PATCH request has finished. The pool.RW is seekable, so a
chunk which fails with a retryable error is re-sent from the same
buffer. Accounting is applied as a chunk is sent so bandwidth limits
and progress still track the upload. Chunk requests now carry an
explicit Content-Length rather than being sent chunked.
The prefix sent with the creating request in PutUnchecked is buffered
through the same helper.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
Nick Craig-Wood
e8034e6548