Each upload allocated a fresh 48 MiB chunk buffer, so bulk transfers of
many files churned allocations and GC, and small files paid for the full
buffer.
Buffer chunks with multipart.NewRW instead so chunk memory is reused
across uploads and is part of rclone's central memory management.
The pool.RW implements io.Closer, so the PATCH request body is wrapped in
readers.NoCloser to stop the http transport closing it after a failed
attempt and freeing its pages before the retry. The retry closure now
seeks the chunk back to the start explicitly before resending, a short
read of the source is reported as an error rather than sent as an
under-length chunk, and the request carries an explicit ContentLength.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
Nick Craig-Wood
cdabcc7cc4