Every compressed upload allocated a fresh buffer the size of
--compress-ram-cache-limit (20 MiB by default) regardless of how big
the file actually was, so uploading many small files churned large
allocations and the memory sat outside rclone's pool accounting.
Read the head of the stream into a multipart.NewRW instead, which takes
pages from the global pool only for the bytes actually read and returns
them when the upload finishes. The pool.RW is seekable, so a wrapped
backend which needs to retry a small upload can rewind the body, which
the previous bytes.Buffer did not allow. A read error while filling the
cache is returned rather than falling through to the streaming path.
Add a unit test covering the buffered, streamed and spooled paths which
checks the body handed to the wrapped remote, that it can be re-read
for a retry, and that the pool pages are returned.
Nick Craig-Wood
03c1c08a27