Each part of a multipart upload allocated a fresh part-sized buffer
(the size is chosen by Box, typically 8-32 MiB) with up to --transfers
parts in flight, so large uploads churned allocations and GC.
Buffer parts with multipart.NewRW instead so the memory comes from
rclone's global pool, is reused across parts and files, and is part of
rclone's central memory management.
The pool.RW is seekable so the retry closure seeks back to the start
before each attempt instead of rebuilding a bytes.Reader, and the
per-part SHA1 digest is computed by reading the buffer and seeking
back. The body goes through lib/rest which already stops the transport
from closing it; the uploading goroutine owns and closes the buffer.
The whole-file SHA1 used for the commit is unchanged.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
Nick Craig-Wood
fb8783732d