With -l/--links the local backend faithfully recreates a source ".rclonelink" as
a real symlink at the destination. Directory metadata (chmod/chown/chtimes),
however, was applied with the raw following syscalls
os.Chmod/os.Chown/os.Chtimes rather than through the os.Root sandbox used for
content writes. A Directory is never a translatedLink, so when the destination
path already existed as a symlink planted by an untrusted source, the metadata
was applied through it to a target outside the backup destination.
Route directory metadata through os.Root when translating symlinks, so a planted
symlink can no longer redirect chmod/chown/chtimes out of the destination, while
legitimate in-tree directories are unaffected.
Single part uploads with Object Lock parameters need a Content-MD5
header, which the SDK can't compute from a stream, so the whole body
was read into memory with io.ReadAll to hash it - up to
--s3-upload-cutoff per file. prepareUpload already sets Content-MD5
from the source object's hash when it has one, so skip the buffering
entirely in that case and only buffer when the hash is unavailable.
When buffering is needed, read the body into a multipart.NewRW buffer
from the global pool, hashing in transit, so the memory is reused
across uploads and released after the request. The presigned request
path hands the body straight to http.NewRequest, so wrap it in
readers.NoCloser there to stop the transport closing the pooled buffer.
With speedup enabled, files up to --mailru-speedup-max-memory are read
into memory so their hash can be tried against the server before
uploading. This used io.ReadAll, which allocates a fresh heap slice per
file and grows it by doubling, so with the default 32 MiB limit and
several transfers this churned a lot of garbage outside rclone's memory
accounting.
Buffer the file with multipart.NewRW instead, hashing it in transit,
so the memory comes from the global pool and is reused.
When the hash isn't known to the server the buffered file is uploaded
from the same buffer. Previously a low level retry of that upload
resent an already drained reader, so the retry always failed. Rewind
seekable bodies at the start of each attempt so retries resend the
whole file. Add a test which drops the connection on the first attempt
and checks the retried body is complete.
The body is sent through lib/rest, which wraps it in readers.NoCloser,
so the transport can't close the pooled buffer early; Update closes it
when it returns.
The Linkbox API needs the MD5 of the first 10 MiB of each uploaded
file, so Update reads that prefix into memory before the upload. This
used io.ReadAll, which allocates a fresh heap slice per file and grows
it by doubling, churning well over 10 MiB of garbage per upload.
Read the prefix into a multipart.NewRW buffer instead so the memory
comes from rclone's global pool and is reused across uploads, and hash
it in transit rather than computing the same MD5 twice.
The PUT body goes through lib/rest, which stops the http transport
closing it, so Update owns the buffer and closes it on every exit path.
When the source has no MD5, Update reads the whole file into memory to
hash it before uploading if it is under --jottacloud-md5-memory-limit.
This used io.ReadAll, which grows a fresh heap slice per file (up to
10 MiB by default, roughly doubled by the growth strategy), so syncs
of many files churned allocations and GC.
Buffer the data with multipart.NewRW instead so the memory comes from
rclone's global pool, is reused across uploads and is released by the
existing cleanup function.
Unknown sized streams previously took the in-memory branch regardless
of the limit, so an rcat of an arbitrarily large stream could read it
all into memory. Spool those to the temporary file instead, as is
already done for files over the limit.
The buffered body is sent through lib/rest, which wraps request bodies
in readers.NoCloser, so the transport can't close the pooled buffer
early.
The multipart upload allocated a fresh chunk-sized buffer (64 MiB by
default) plus a 1 MiB scratch buffer per large file, copying every byte
twice, and never returned them to rclone's memory pool.
Buffer each part with multipart.NewRW instead so the memory is reused
across uploads and part of rclone's central memory management.
The pooled buffer is seekable, so a part can now be re-sent.
uploadPart previously had no retry at all and any transient error
failed the whole upload. It is now wrapped in the pacer with the
backend's usual shouldRetry rules, seeking to the start before each
attempt. The body is wrapped in readers.NoCloser so the http transport
can't close the pooled buffer between attempts, and Content-Length is
set explicitly since net/http can't infer it from a pool.RW.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
Each part of a multipart upload allocated a fresh part-sized buffer
(the size is chosen by Box, typically 8-32 MiB) with up to --transfers
parts in flight, so large uploads churned allocations and GC.
Buffer parts with multipart.NewRW instead so the memory comes from
rclone's global pool, is reused across parts and files, and is part of
rclone's central memory management.
The pool.RW is seekable so the retry closure seeks back to the start
before each attempt instead of rebuilding a bytes.Reader, and the
per-part SHA1 digest is computed by reading the buffer and seeking
back. The body goes through lib/rest which already stops the transport
from closing it; the uploading goroutine owns and closes the buffer.
The whole-file SHA1 used for the commit is unchanged.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
Each upload allocated a fresh chunk-sized buffer (10 MiB by default)
regardless of the file size, so bulk transfers of many files churned
allocations and GC.
Buffer chunks with multipart.NewRW instead so chunk memory is reused
across uploads and is part of rclone's central memory management. The
pool.RW is seekable, so the existing rewind on retry carries over.
The body goes through lib/rest which already wraps it so the transport
can't close the pool buffer. The upload loop closes it after every
chunk, on error paths included. A source which ends before the
declared size is now reported as a short read before the chunk is sent
rather than as an incomplete write afterwards.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
Each chunked upload allocated a fresh chunk-sized buffer (48 MiB by
default), so bulk transfers of many files churned allocations and GC.
Buffer chunks with multipart.NewRW instead so chunk memory is reused
across uploads and is part of rclone's central memory management. The
pool.RW is seekable, so the existing IncorrectOffset recovery which
skips already-received bytes on retry carries over unchanged, and the
"chunk received OK" check now compares against the bytes actually
buffered so a short final chunk is recognised too.
The Dropbox SDK wraps the request body in io.NopCloser, so the transport
never closes the pool buffer. The upload loop closes it after every
chunk, on error paths included.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
Each upload allocated a fresh 48 MiB chunk buffer, so bulk transfers of
many files churned allocations and GC, and small files paid for the full
buffer.
Buffer chunks with multipart.NewRW instead so chunk memory is reused
across uploads and is part of rclone's central memory management.
The pool.RW implements io.Closer, so the PATCH request body is wrapped in
readers.NoCloser to stop the http transport closing it after a failed
attempt and freeing its pages before the retry. The retry closure now
seeks the chunk back to the start explicitly before resending, a short
read of the source is reported as an error rather than sent as an
under-length chunk, and the request carries an explicit ContentLength.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
The compressibility heuristic compressed a 1 MiB sample of each upload
into a bytes.Buffer only to read its length, growing up to ~1 MiB of
garbage per file. Write the sample through a counting io.Discard-style
writer instead so no output buffer is allocated at all.
Every compressed upload allocated a fresh buffer the size of
--compress-ram-cache-limit (20 MiB by default) regardless of how big
the file actually was, so uploading many small files churned large
allocations and the memory sat outside rclone's pool accounting.
Read the head of the stream into a multipart.NewRW instead, which takes
pages from the global pool only for the bytes actually read and returns
them when the upload finishes. The pool.RW is seekable, so a wrapped
backend which needs to retry a small upload can rewind the body, which
the previous bytes.Buffer did not allow. A read error while filling the
cache is returned rather than falling through to the streaming path.
Add a unit test covering the buffered, streamed and spooled paths which
checks the body handed to the wrapped remote, that it can be re-read
for a retry, and that the pool pages are returned.
Each resumable upload allocated a fresh chunk-sized buffer (8 MiB by
default, up to 64 MiB), so bulk transfers of large files churned
allocations and GC.
Buffer each chunk in a pool.RW from the global page pool instead so
the memory is reused across uploads and bounded by rclone's central
memory management.
The pool.RW is seekable, so the chunk is rewound at the start of each
retry rather than re-wrapped. lib/rest wraps request bodies in
readers.NoCloser so the transport cannot return the pages to the pool
between attempts - the Content-Length is set through rest.Opts because
the wrapped body is not a *bytes.Reader net/http can measure. A source
that runs dry before its declared size is reported as an unexpected EOF
rather than sending the short chunk.
Files below upload_cutoff (up to 20 MiB) were assembled into a bytes.Buffer
which grows by doubling, so each upload allocated roughly twice its size
and threw it away afterwards, churning the GC on bulk transfers. Write the
multipart/related body into a pool.RW from the global page pool instead so
the memory is reused across uploads and bounded by rclone's memory
management.
The pool.RW is seekable, so the same body is rewound at the start of each
retry rather than being re-wrapped. lib/rest already wraps request bodies
in readers.NoCloser so the transport cannot free the pages between
attempts; the Content-Length is passed explicitly because the wrapped body
is no longer a *bytes.Reader net/http can measure.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
Updating a file below --hidrive-upload-cutoff copied the whole file into
an append-grown slice so the request could be retried, costing roughly
twice the file size in transient allocations for every such update.
Buffer the file in a multipart.NewRW from rclone's global page pool
instead and return it to the pool once the request has finished. The
pool.RW is seekable so retries re-send the same buffer. Accounting is
applied as the buffer is sent so bandwidth limits and progress still
track the upload. The request now carries an explicit Content-Length
rather than being sent chunked.
The upload is bounded by the size the source declares - a source that
delivers more bytes than its declared size has the excess ignored, where
previously the stream was sent to EOF.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
Each chunk of a chunked upload allocated a fresh buffer of
--hidrive-chunk-size bytes (48 MiB by default), with up to
--hidrive-upload-concurrency of them in flight, so large uploads churned
allocations and GC and a short final chunk still cost a whole chunk.
Buffer chunks with multipart.NewRW from rclone's global page pool
instead, sized to the data actually read, and return each buffer to the
pool once its PATCH request has finished. The pool.RW is seekable, so a
chunk which fails with a retryable error is re-sent from the same
buffer. Accounting is applied as a chunk is sent so bandwidth limits
and progress still track the upload. Chunk requests now carry an
explicit Content-Length rather than being sent chunked.
The prefix sent with the creating request in PutUnchecked is buffered
through the same helper.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
Every upload allocated a fresh buffer of --hidrive-upload-cutoff bytes
(96 MiB by default) to hold the part of the file sent with the creating
request, however small the file was, so copying many small files churned
large allocations and GC.
Buffer that prefix in a multipart.NewRW from rclone's global page pool
instead, sized to the smaller of the declared file size and the cutoff,
and return it to the pool once the file has been created. Accounting is
applied as the buffer is sent so bandwidth limits and progress still
track the upload. The request now carries an explicit Content-Length
rather than being sent chunked.
The upload is bounded by the size the source declares - a source that
delivers more bytes than its declared size has the excess ignored,
where previously they were read up to the cutoff.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
Upload chunks are buffered into a bytes.Reader and then, when transfer
accounting is active (always for a real copy), re-wrapped in accounting
before being handed to cachedReader. cachedReader only recognised a bare
*bytes.Reader, so the accounted chunk fell through to
readers.NewRepeatableReader, which copied the whole chunk again into an
append-grown slice. Every chunk (and the upload-cutoff prefix of every
file) therefore cost roughly twice its size in memory.
Look through the accounting wrapper when deciding whether the reader is
already a seekable buffer and, if so, seek the buffer underneath while
still reading through the accounting, so retries rewind without a copy.
Each chunk of a resumable upload was buffered in a fresh RepeatableReader
which grew by appending, so bulk transfers churned up to a chunk size (10
MiB by default) of heap per chunk and GC pressure.
Buffer chunks instead with multipart.NewRW from the global page pool
instead so memory is reused across uploads and bounded by rclone's
central memory management.
A source which delivers fewer bytes than its declared size now fails
before the chunk is sent with an unexpected EOF error rather than being
rejected by the transport.
Note that accounting now happens as the chunk is read into the buffer
rather than as it is sent, as in the drive backend.
When the source can't be re-opened, the whole input is read once to
compute its gcid before upload and held back for the upload proper. For
inputs at or below --pikpak-hash-memory-limit this used a bytes.Buffer,
a fresh heap allocation of up to the limit (and more while growing) per
file.
Hold the data in a buffer from the global memory pool instead so the
pages are reused and released on cleanup.
Inputs of unknown size were also always held in memory regardless of
their length, as only sizes above the limit chose the temp file.
Spool unknown sizes to the temp file so a large stream can't exhaust
memory.
The multipart uploader kept its own private buffer pool, a copy of the
one in lib/pool with identical settings, so its chunk memory was never
shared with the rest of rclone. Pages cached here were invisible to
other backends and vice versa, costing up to 64 MiB of extra idle cache.
Allocate chunks with multipart.NewRW, the global pool used by the
other backends instead.
WriteChunk copied the whole chunk, which lib/multipart already hands over
in a buffer from the global memory pool, into a bytes.Buffer so that
retries could re-send it. That doubled the per-part memory and made a
fresh chunk-sized heap allocation (64 MiB by default) for every part,
times the upload concurrency.
The chunk reader is seekable, so find its size with Seek and rewind it
inside the pacer closure instead, sending the pooled buffer directly.
Also fix the error for a part that fails to upload, which formatted the
buffer instead of the part number.
Each upload chunk is buffered in a pool.RW from the global memory pool
but was never closed, so its pages were never returned to the pool.
Close the buffer after each chunk is uploaded and on the read error
path.
A chunk that failed with a retryable error was also retried without
rewinding the buffer, so the retry sent an empty body with the original
Content-Length and Content-Range and failed.
Seek the chunk back to the start inside the pacer closure so each
attempt re-sends it in full.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
When both /me/drives and /me/drive fail during config (for example an
account-level 403 serviceReadOnly "Database Is Read Only"), send the
config state machine to the existing manual drive ID entry state
instead of dead-ending at choose_type with the raw error. The drive
itself remains usable when only the enumeration API is blocked.
Fixes#9794
Each resumable upload allocated a fresh chunk-sized buffer (8 MiB by
default), so bulk transfers of many files churned allocations and GC.
Buffer chunks with multipart.NewRW instead — the global page pool used
by the other backends — so chunk memory is reused across uploads and
bounded by rclone's central memory management. The pool.RW is seekable,
which also keeps chunk reads repeatable for retries.
The pool.RW implements io.Closer, so http.NewRequestWithContext upgraded
it to the request body and the transport closed it after each attempt,
returning its pages to the global pool — a chunk retried after a 5xx
then read a freed buffer and panicked in pool.(*RW).readPage. Wrap the
request body in readers.NoCloser so the transport can't take ownership
and the upload loop remains solely responsible for the buffer's
lifetime. Add a regression test that fails a chunk with a 500 and then
accepts the retry; it reproduces the panic without the fix.
Fixes#9684
The sftp backend could be used as the source of a multi-thread copy -
each chunk of a download is read on its own connection - but not as the
destination, because it did not implement OpenWriterAt. Uploading a
single large file was therefore limited to one connection while
downloading the same file was not, capping upload throughput well below
the link speed on high latency connections.
This implements OpenWriterAt for the sftp backend using lib/filepool. A
small pool of open write handles, each backed by its own connection,
lets the core write the chunks of a large file concurrently over several
connections. The file is created and truncated once up front so every
chunk offset is valid before the concurrent writes start.
Multi-thread uploads are off by default and turned on with the new
--sftp-multithread-upload flag, since many sftp servers only accept
sequential writes and would fail large uploads otherwise. Even when
enabled they fall back to a single connection when
--sftp-disable-concurrent-writes is set (a server that can't take
out-of-order packets on one handle won't take several handles either) or
when --sftp-connections caps the pool (the per-file fan-out would
otherwise deadlock waiting on it).
The OpenSSH test server (TestSFTPOpenssh) enables
--sftp-multithread-upload so the feature is exercised in CI against a
real server.
Tested against OpenSSH: a single large upload over a high latency link
went from ~12 MB/s to ~90 MB/s.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Splainte <r.wycke@hotmail.fr>
Factor the connection-backed write handle pool out of the smb backend
into a generic lib/filepool.Pool[T] with its own tests, so it can be
reused by other backends that implement fs.OpenWriterAter over a
connection pool.
The smb backend keeps its behaviour, opening and releasing handles
through small closures passed to the pool.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Splainte <r.wycke@hotmail.fr>
Nine tests fail on an ordinary Windows machine, eight in backend/local and
TestEnvironmentVariables in cmdtest, all with
symlink file.txt \?\C:\Users\...\symlink.txt: A required privilege is not
held by the client.
Windows grants SeCreateSymbolicLinkPrivilege only to an elevated process or one
running with Developer Mode enabled, and a default install gives an ordinary
user neither. CI does not see this because the windows-latest runner is
elevated, so the failures only show up on a contributor's own machine, where
AGENTS.md asks for make quicktest to pass before opening a pull request.
cmdtest already recognised the situation and attached a note to the failure
saying the test could safely be ignored. If it is safe to ignore then the test
knows it cannot run, so skip it and say why instead.
backend/local gains a helper that tries a symlink in t.TempDir() and skips if it
cannot make one, called from the six tests that need the privilege. Where a
platform can create symlinks the probe succeeds and nothing is skipped, so other
platforms are unchanged.
TestMetadata is skipped whole because it creates its symlink before anything
else and the object built from it is used throughout.
TestSymlinkEscapeConcurrent is left alone: it goes through putLink and ignores
the error, so it never needed the privilege.
Directory names which look like they have a --b2-versions version
string are now encrypted in full, so directories created by older
rclone (which left the version string in plain text) no longer
decrypt and vanished silently from listings.
DecryptDirName now falls back to the old form for such names so the
directory is listed, and logs the name it needs to be renamed to on
the underlying remote to make it accessible again. Document this in
the crypt docs.
The refresh endpoint returns a rotated token with a fresh expiry on
every successful call, but getUserInfo discarded it, so routine use
never extended the stored token's life. Once the stored token aged
out, accounts with 2FA enabled could not recover non-interactively
and required a manual reconnect.
Carry the rotated token out of getUserInfo and persist it in NewFs
via the same jwtToOAuth2Token + oauthutil.PutToken path that
refreshJWTToken uses, keeping f.cfg.Token in sync (same pattern as
refreshOrReLogin).
Fixes#9584
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The --b2-versions support added in 3fe2aaf96 strips a version string
from the last segment of a path before encrypting it, so that the
plain text version suffixes which the underlying backend appends to
encrypted file leaf names can be handled. EncryptDirName and
DecryptDirName share that code, so the last segment of a *directory*
name was version stripped too. Only file leaf names are ever given a
version string by the backend - a directory gets a
version-string-like name from the user, and such a name is encrypted
verbatim when it appears as the parent of a file name, so the same
directory ended up with two different encryptions.
Before this change, with a directory whose name matches rclone's
version format, eg dir-v2001-02-03-040506-123:
rclone copy file.txt crypt:dir-v2001-02-03-040506-123/
rclone ls crypt:dir-v2001-02-03-040506-123
# => "directory not found" - the file is invisible to listings
rclone mkdir crypt:dir-v2001-02-03-040506-123
# => creates a second directory with the same decrypted name
After this change EncryptDirName and DecryptDirName encrypt directory
names verbatim, so a directory encrypts the same way whether it is
named on its own or as the parent of a file. Version strings are only
added to file names by the underlying backend, so --b2-versions is
unaffected and the existing version tests are untouched.
A directory which was created by the old EncryptDirName will no longer
decrypt and will be reported as undecryptable in listings. Such
directories were already unusable - anything copied into one was
written to a different encrypted directory - so nothing which worked
before is broken by this.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With no_head_object set, NewObject does not read any metadata, so the
destination object returned from a server side copy had a size of 0.
The size check in operations.Copy then failed with "corrupted on
transfer: sizes differ N vs 0" and deleted the newly copied object.
This also broke Move and hence renames through rclone mount.
Populate the destination object's size and MD5 from the source object
when no_head_object is set, as a server side copy produces an object
with identical content.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Dropbox is case insensitive and the path_display it returns in
change notifications may not match the case of the configured root.
Before this change the root was trimmed with a case sensitive prefix
match, so when the cases differed the full path was passed to the
ChangeNotify callback and the notification was ignored.
This trims the root case insensitively while preserving the display
case of the remaining path.
A successful UploadPart whose response carries no ETag header made
WriteChunk panic dereferencing uout.ETag in a debug log line. The part
ETag is required by CompleteMultipartUpload, so an ETag-less 200 is
unusable: return a retryable error from inside the pacer callback so
the chunk is retried instead of crashing the transfer or completing
the upload with a broken part list.
Fixes#9822
Co-authored-by: Shurong Cao <170531907+CAOShurong@users.noreply.github.com>
Nextcloud only stores a checksum which is supplied in the OC-Checksum
header of an upload, and discards it again when the modification time
is set with PROPPATCH. Re-sending the checksum in the PROPPATCH (as is
done for ownCloud) is rejected by Nextcloud with 403 Forbidden which
made the whole PROPPATCH fail, so SetModTime returned an error on any
object which had a hash. Uploads from sources without hashes, eg
streamed uploads with `rclone rcat`, were stored with no hash at all.
Use the Nextcloud PATCH extension with the X-Recalculate-Hash header
to have the server calculate and store the SHA1 of an object after a
streamed upload and after setting the modification time. This gives
a server side hash of the stored data which also lets rclone verify
streamed uploads.
If the source supplied fewer bytes than its declared size, the single
part upload path accepted the short body and stored a truncated file
recorded with the declared size, reporting a successful upload. The
multipart path already checks for this.
Count the bytes actually read and fail the upload if they do not match
the declared size, which cancels the partially created file.
This was found by the FsPutShortEOF integration test.
listSharedFolders already decoded shared-folder names with
f.opt.Enc.ToStandardName, but listReceivedFiles stored the raw name
returned by the Dropbox API unchanged. Names that require encoding
(e.g. a trailing space, which Dropbox itself rejects, so rclone
stores it as "name␠" via EncodeRightSpace) were therefore shown under
their raw, encoded form for received files instead of being decoded
back to the standard name, and findSharedFile could not resolve such
a file by its standard name.
Apply the same ToStandardName conversion listSharedFolders uses.
On a ranged download the metadata decoder stored the response's
Content-Length (the length of the range, not the blob) in the object's
size and only corrected it from the Content-Range total afterwards.
Object.Size() is read concurrently by the VFS cache and chunked reader
while a download is in progress, so with --vfs-read-chunk-size a reader
could observe the chunk length (e.g. 67108864 for 64M chunks) as the
object size. The VFS cache then logged
vfs cache: cached file (N) is unexpectedly larger than the remote
object (67108864). The cached file is likely corrupted after an
unclean shutdown; recovering ...
and truncated the read request against the bogus size, breaking
sequential reads of large blobs with --vfs-cache-mode full.
This applies the Content-Range correction before the size is stored so
the range length is never published as the object size.
The multipart upload copied the source into the request buffer without
checking how many bytes it had read, so a source that supplied fewer
bytes than its declared size was accepted by the server and reported as
a success with a truncated file stored.
Count the bytes actually read and fail the upload if they do not match
the declared size.
Signed-off-by: Rohit Behera <126186063+r0h1tb@users.noreply.github.com>
The single-shot upload path sent the source straight to Box as a multipart
body with no Content-Length, so a source that supplied fewer bytes than its
declared size produced a short request that Box accepted and stored, and the
upload was reported as a success.
Count the bytes actually read and fail the upload if they do not match the
declared size. The multipart path already reads each chunk with io.ReadFull
and so already fails in this case.
The concurrent walker created by walk() only stopped when the callback
returned an error or the whole tree had been listed. Cancelling the
context (for example via the rc job/stop endpoint for an async
operations/size or recursive operations/list call) was therefore
ignored: the checkers kept pulling list jobs from the channel and kept
listing the entire tree, burning CPU and making job cancellation
useless for every backend without a native ListR implementation.
Make every checker select on ctx.Done() so a cancelled walk shuts down
promptly through the existing quit/drain path and reports the context
error. Also check the context between directory read chunks in the
local backend so a single huge directory does not block cancellation.
Update already wrapped the source in a counting reader but never looked at the
count, so a source that supplied fewer bytes than its declared size was
uploaded as a chunked request, accepted by the server and reported as a
success with a truncated file stored.
Compare the bytes actually read against the declared size.
Nick Craig-Wood
SillyZirandNick Craig-Wood
Mikel Olasagasti UrangaandNick Craig-Wood
DhevenddraandNick Craig-Wood
Loi NguyenandNick Craig-Wood
waterandNick Craig-Wood
machsixandNick Craig-Wood
Sanjay Kanth AandNick Craig-Wood
kingston125andNick Craig-Wood
Rohit BeheraandNick Craig-Wood
Rahman YilmazandGitHub