When the source has no MD5, Update reads the whole file into memory to
hash it before uploading if it is under --jottacloud-md5-memory-limit.
This used io.ReadAll, which grows a fresh heap slice per file (up to
10 MiB by default, roughly doubled by the growth strategy), so syncs
of many files churned allocations and GC.
Buffer the data with multipart.NewRW instead so the memory comes from
rclone's global pool, is reused across uploads and is released by the
existing cleanup function.
Unknown sized streams previously took the in-memory branch regardless
of the limit, so an rcat of an arbitrarily large stream could read it
all into memory. Spool those to the temporary file instead, as is
already done for files over the limit.
The buffered body is sent through lib/rest, which wraps request bodies
in readers.NoCloser, so the transport can't close the pooled buffer
early.
Nick Craig-Wood
885e2478e7