With speedup enabled, files up to --mailru-speedup-max-memory are read
into memory so their hash can be tried against the server before
uploading. This used io.ReadAll, which allocates a fresh heap slice per
file and grows it by doubling, so with the default 32 MiB limit and
several transfers this churned a lot of garbage outside rclone's memory
accounting.
Buffer the file with multipart.NewRW instead, hashing it in transit,
so the memory comes from the global pool and is reused.
When the hash isn't known to the server the buffered file is uploaded
from the same buffer. Previously a low level retry of that upload
resent an already drained reader, so the retry always failed. Rewind
seekable bodies at the start of each attempt so retries resend the
whole file. Add a test which drops the connection on the first attempt
and checks the retried body is complete.
The body is sent through lib/rest, which wraps it in readers.NoCloser,
so the transport can't close the pooled buffer early; Update closes it
when it returns.
Nick Craig-Wood
921c149f7e