Each resumable upload allocated a fresh chunk-sized buffer (8 MiB by
default, up to 64 MiB), so bulk transfers of large files churned
allocations and GC.
Buffer each chunk in a pool.RW from the global page pool instead so
the memory is reused across uploads and bounded by rclone's central
memory management.
The pool.RW is seekable, so the chunk is rewound at the start of each
retry rather than re-wrapped. lib/rest wraps request bodies in
readers.NoCloser so the transport cannot return the pages to the pool
between attempts - the Content-Length is set through rest.Opts because
the wrapped body is not a *bytes.Reader net/http can measure. A source
that runs dry before its declared size is reported as an unexpected EOF
rather than sending the short chunk.
Files below upload_cutoff (up to 20 MiB) were assembled into a bytes.Buffer
which grows by doubling, so each upload allocated roughly twice its size
and threw it away afterwards, churning the GC on bulk transfers. Write the
multipart/related body into a pool.RW from the global page pool instead so
the memory is reused across uploads and bounded by rclone's memory
management.
The pool.RW is seekable, so the same body is rewound at the start of each
retry rather than being re-wrapped. lib/rest already wraps request bodies
in readers.NoCloser so the transport cannot free the pages between
attempts; the Content-Length is passed explicitly because the wrapped body
is no longer a *bytes.Reader net/http can measure.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
The multipart upload copied the source into the request buffer without
checking how many bytes it had read, so a source that supplied fewer
bytes than its declared size was accepted by the server and reported as
a success with a truncated file stored.
Count the bytes actually read and fail the upload if they do not match
the declared size.
Signed-off-by: Rohit Behera <126186063+r0h1tb@users.noreply.github.com>
Nick Craig-Wood
Rohit BeheraandNick Craig-Wood
caltechustcandNick Craig-Wood
