Validate ranged GET responses before returning their bodies. Close invalid
responses, retry malformed partial responses through the pacer, and return
without retrying when a server deterministically ignores Range.
Fixes#6980
Add response validation for calls made with Range open options. Verify
Content-Range, Content-Length, response status, and the complete
representation size before a backend accepts the response body.
Return a shared sentinel when a server ignores a partial range so callers
can avoid retrying the same unsupported request.
Before this change, when no_data_encryption was set, uploads from
local disk advertised the hash of the encrypted data even though the
data was uploaded unencrypted.
On backends which check upload hashes (eg b2) this made uploads of
small files fail with errors like "Checksum did not match data
received", and made chunked uploads store an incorrect hash so the
files failed their checksum on download with "corrupted on transfer:
SHA1 hashes differ".
See: https://forum.rclone.org/t/sha1-mismatches-on-b2-with-no-data-encryption-true/54121
Before this change the environment variable getters in fs/configmap.go
logged the option value with %q, so a password set via
RCLONE_CONFIG_remote_pass (or RCLONE_remote_pass) was printed in full
to the debug log. Values from the config file were already redacted,
which made the leak easy to miss.
This change routes both getters through fs.RedactOptionValue, which
looks up the option in the backend's option list: options marked
IsPassword or Sensitive log as XXX, unknown options are conservatively
redacted, and --dump auth still shows the value for debugging.
Fixes#5794
The NFS section under Mounting on macOS talked about serve nfs without
pointing at rclone nfsmount, which is the command that actually does the
NFS-based mount on macOS.
Fixes#7869
Signed-off-by: Dean Chen <862469039@qq.com>
rclone checksum and rclone check --checkfile hashed one file at a time
unless --download was in use. The non-download branch of CheckSum called
Object.Hash synchronously from inside the listing callback, so --checkers
had no effect there; only the --download branch used the checker token
pool.
Both branches now do their work in the same goroutine bounded by
--checkers, so --checkers means the same thing with and without
--download. Checking a sum file against 2.6 GiB of local files with
--checkers 8 went from 1.70s to 0.31s here.
As a result the "hashing" checking transfer now covers the hashing work
in both branches, rather than being finished as soon as the download
goroutine had been started.
RcatSize streams the body straight into Put behind an ObjectInfo with no
hashes, so a known-size upload had its size checked but its checksum
never verified, unlike the unknown-size path through Rcat which hashes as
it goes.
The stream can only be read once, so hash it on the way past and compare
with the destination after the upload, as operations.Copy does. A
destination which reports no usable hash is still checked by size, and one
which fails either check has the failed copy removed.
Writing any file into a third-party app container - the Obsidian, Pages or
Shortcuts folders that iCloud Drive shows alongside your own - failed with
HTTP error 412 (412 Precondition Failed) returned body:
"{ ... \"error_code\" : \"VALIDATING_REFERENCE_ERROR\", \"reason\" :
\"Request has out of order children to be chained but the parents were
missing\" }"
Reading from those paths worked, and so did creating directories in them, so
the failure looked like a missing parent when the parent was plainly there.
Items in an app container live in a different zone from ordinary iCloud Drive
folders: a folder under Documents has a drive ID like
FOLDER::com.apple.CloudDocs::<uuid>, while the Obsidian container has
FOLDER::iCloud.md.obsidian::documents#o2v. DownloadFile already accounts for
this - it deconstructs the item's own ID and addresses the zone it finds - but
CreateUpload and UpdateFile hardcoded defaultZone, and UpdateFile built the
resulting Drivewsid with a hardcoded com.apple.CloudDocs as well.
So rclone asked Apple to chain the new document to a parent in
com.apple.CloudDocs while the parent lived in iCloud.md.obsidian. The parent
really was missing from the zone being addressed, which is what the error said.
Take the zone from the parent's drive ID instead, the same way the download
path does, and build the new item's ID with ConstructDriveID. Uploads outside
an app container are unaffected: their parents are in com.apple.CloudDocs, so
the derived zone is the value that was previously hardcoded.
Verified against a real remote: files now upload into an Obsidian vault inside
the container and read back correctly with an unpatched binary afterwards.
Before this change, `NewStats` stored the context it was created from on the
`StatsInfo`. Stats groups are never freed -- they are only evicted once there are
`--max-stats-groups` of them -- so each one kept its context, and everything reachable
from it, alive for the life of the process. As the rc creates a group per call,
that included the call's filters and their compiled regexps.
The context was only ever used to get `ci.StatsFileNameLength` from the config.
`StatsInfo` already stores that same `*fs.ConfigInfo`, read from the same
context in `NewStats`.
This change fixes the issue by passing the stored ci to `transferMap.String` and
dropping the context from `StatsInfo`.
Before this change, Transfer.Done closed the account of a completed transfer but
never released it, because it assigned nil to only a local copy of the pointer.
As a result, every completed transfer continued to reference its account.
An Account holds the transfer context and the source reader, and the stats keep
completed transfers around up to `MaxCompletedTransfers` per group, with groups
discarded only at --max-stats-groups. On a long-running `rclone rcd`, this adds up.
It was noticeable when running bisync repeatedly via the rc, where the transfer
context carries `b.WriteResults` (bisync's `LoggerFn`). A `*bisyncRun` holds
Path1 and Path2 listings, which can be quite large, and are not supposed to be
retained between runs. (Naturally they aren't, when running bisync on the
command line -- which is probably why we didn't notice this issue sooner.)
This change fixes the issue by releasing `tr.acc` in Done, instead of the local
copy. `tr.Snapshot` reads the byte counts off the account, so the progress is
recorded on the transfer first. That read happens before taking `tr.mu`, because
`acc.progress()` locks `acc.values.mu`, `checkReadBefore` holds that lock while
calling `StatsInfo.GetBytes`, and `StatsInfo` locks back into `Transfer` in
`Transferred` and `_removeTransfer`.
A client which started a multipart upload and vanished without either
completing or aborting it used to hold on to its resources forever.
Incomplete multipart uploads which have had no activity for
--multipart-expiry (default 24h) are now aborted and cleaned up
exactly as if the client had called AbortMultipartUpload, with a
NOTICE logged.
An upload with a part still being received is never expired, and each
completed part restarts the clock. Late operations on an expired
upload fail with NoSuchUpload, as they do on real S3 when a lifecycle
rule has aborted the upload.
Set --multipart-expiry 0 to keep incomplete uploads forever.
Multipart uploads used to be streamed directly to the remote with their
own PutStream machinery, bypassing the VFS, a design left over from
before the VFS could abandon a streaming write.
They are now written through the VFS exactly like plain object PUTs in
every cache mode. The parts are written, in part-number order, to a
temporary object which is renamed into place server-side on
completion.
With the default --vfs-cache-mode off the parts stream through the VFS
to the remote as they arrive. With --vfs-cache-mode writes or above
they are buffered in the VFS cache and uploaded by its write-back.
User visible changes:
- Multipart uploads now show in rclone's transfer stats and obey
--bwlimit (previously they bypassed both).
- Remotes without streaming upload support now spool the upload to a
temporary file on local disk instead of buffering it in memory.
- Multipart uploads are never buffered in memory because of missing
remote capabilities - only --disable-multipart-streaming does that.
- Remotes that upload atomically now also write to a temporary object
renamed into place, so an in-progress multipart upload is no longer
briefly visible under its final key.
- On the few remotes with no server-side move or copy the parts are
written straight to the final object in all cache modes.
- With --vfs-cache-mode writes, plain PUTs and multipart uploads to the
same key go through the same cache entry, so an earlier PUT still in
the write-back window can no longer be written back over a newer
multipart upload.
- Failed write-backs are retried by the VFS without the client having
to restart the upload, and completed objects are served from the
cache for read-after-write.
The temporary objects that uploads are written to before being renamed
into place are now named .rclone_temp_put_* and .rclone_temp_multipart_*,
and the whole .rclone_temp_ prefix is reserved: any object whose name
starts with it is hidden from S3 listings. This gives a single pattern
for cleaning up leftovers from killed servers:
rclone delete --min-age 24h --include ".rclone_temp_*" remote:path
The .rclone_multipart_upload_* objects rclone v1.75 used are still
hidden from listings so leftovers from an older server stay invisible
to S3 clients.
The mtime metadata fallback was nested inside the X-Amz-Meta-Mtime
branch, so it only ran when X-Amz-Meta-Mtime was present but invalid -
and then set the modtime from the invalid value's failed parse rather
than parsing mtime. An object PUT with only mtime metadata kept the
upload time as its modtime.
Now the two keys are checked independently, as TouchObject already
does.
streamPart did not check whether the upload had been torn down, but
AbortMultipartUpload sets the reorder buffer map to nil, so an abort
arriving while a part body was still being received panicked with an
assignment to a nil map once the part was buffered.
Now a part whose upload has been aborted or completed under it is
rejected with NoSuchUpload.
A PUT which failed part way through removed the object at the
destination key. As well as differing from real S3 (where a failed PUT
never affects the stored object), this raced with the client's
automatic retry of the same PUT: the retry stored the object and
returned 200 OK, then the failed first attempt's cleanup deleted it,
silently losing an acknowledged upload. The interrupted upload could
also be committed as a truncated object, since closing the write handle
gave the streaming upload a clean end of stream.
Now a failed or interrupted PUT never disturbs the object at the key:
- The object at the key is never removed on error.
- On backends where a partial upload is visible at its final name
(PartialUploads), and when the VFS cache mode is writes or above, the
upload is written to a temporary object which is renamed into place
on success and removed on failure, as streamed multipart uploads
already do. Backends which upload atomically are still streamed
straight to the destination.
- An interrupted or short body fails the upload via
WriteFileHandle.CloseWithError instead of committing truncated data,
and a body which ends cleanly short of its declared size is rejected
with IncompleteBody.
Closing a streaming write handle sends a clean EOF to the backend
upload, so a writer which knows its data source failed part way through
had no way to stop the truncated file being stored as if it were
complete.
CloseWithError closes the handle failing the upload with the given
reason instead. EOF-like reasons are rewritten so the layers reading
the pipe can't mistake them for a clean end of stream and commit the
partial file.
Needed by serve s3 to abort interrupted PUTs - see #9718
If the source supplied fewer bytes than its declared size, the upload
request failed but a retry could report success even though the stored
file was truncated, because the retry re-sent an already exhausted
reader.
Count the bytes actually read from the source and if they do not match
the declared size return an error.
This was found by the new FsPutShortEOF integration test.
If the source supplied fewer bytes than its declared size, the upload
request failed but a retry could report success even though the stored
file was truncated, because the retry re-sent an already exhausted
reader.
Count the bytes actually read from the source and if they do not match
the declared size, remove the partially uploaded file and return an
error.
This was found by the new FsPutShortEOF integration test.
If the source supplied fewer bytes than its declared size, the
multipart upload was completed anyway, storing a truncated file and
reporting a successful upload.
Check the number of bytes read from the source against the declared
size before finalising and abort the upload with an error if they do
not match.
This was found by the new FsPutShortEOF integration test.
If the source supplied fewer bytes than its declared size, the
multipart upload was completed anyway, storing a truncated object and
reporting a successful upload.
Check the number of bytes read from the source against the declared
size before finalising and abort the upload with an error if they do
not match.
This affects all backends which use lib/multipart for chunked uploads:
azureblob, b2, drime, internxt, oracleobjectstorage, s3 and shade.
This was found by the new FsPutShortEOF and TestRcatSizeShortEOF
integration tests.
If the source supplied fewer bytes than its declared size, the
truncated file was stored and the upload reported success with the
object claiming the declared size.
Count the bytes actually read from the source and if they do not match
the declared size, remove the truncated file and return an error.
This was found by the new TestRcatSizeShortEOF integration test.
The file is created at the declared size and the data then written
with ranged writes, so if the source supplied fewer bytes than
declared, the remainder of the file was left as zeroes and the upload
reported success.
Count the bytes actually read from the source and if they do not match
the declared size, delete the partially uploaded file (if newly
created) and return an error.
This was found by the new FsPutShortEOF and TestRcatSizeShortEOF
integration tests.
Before this change, `fastCopy` created a cancellable context for the sync and
stored its cancel func on the `bisyncRun`, but only ever called it when
gracefully shutting down. On a normal run, it was never called, and until it is
cancelled, a context from `context.WithCancel` stays registered with its nearest
cancellable ancestor.
For an rc job, that ancestor is the job's own context, which the job registry
retains for `--rc-job-expire-duration`. The sync context carries bisync's
`LoggerOpt`, whose `LoggerFn` is a method value on `*bisyncRun`, so a finished
run was kept alive -- including the Path1 and Path2 listings -- for as long as the
job was.
This change fixes the issue by cancelling the sync context when `fastCopy`
returns. The cancel func is still stored on the `bisyncRun`, so a graceful
shutdown can still interrupt a sync that is in progress.
When using Microsoft Entra ID credentials, Azure Blob server-side copy
uses a user delegation SAS URL for the private copy source.
The SAS start time was set to the current local time. Azure Storage
validates the copy source from the service side, and small clock
differences can make that SAS appear not yet valid. The service then
returns 403 CannotVerifyCopySource with AuthenticationFailed.
Start the copy-source SAS 15 minutes in the past, matching Microsoft SAS
guidance for clock skew.
The precision field in the backend overview YAML files can hold
fs.ModTimeNotSupported (100 years in nanoseconds) which overflows int
on 32 bit platforms, making the YAML for those backends fail to parse
and causing rclone to log 18 internal errors on every invocation.
Use int64 for the precision field and add a test that parses every
embedded backend YAML file so this is caught on 32 bit test runs.
When an upload failed part way through with a retryable error (eg a
502 from the block storage servers) the pacer retried the whole upload
call with the same input stream. The stream had already been partially
consumed, so the retry re-created the upload draft and committed just
the remainder of the stream as a complete file, silently truncating
it. With restic over serve restic this corrupted the repository as the
truncated pack was reported as successfully uploaded.
This fixes it by using CallNoRetry for the upload, as the other
backends do, so retryable errors are returned wrapped in a RetryError
for the caller to retry the upload with a fresh stream.
Uploads through RcatSize with a known size - used by rcat --size, the
rc operations/uploadfile and the serve backends, eg serve restic - did
not check the size of the uploaded object. If the source stream ended
before the declared size worth of data had been read, the truncated
object was reported as a successful upload. This could corrupt data
for callers which trust the result, eg a restic repository accessed
via serve restic (see #9722).
This adds the same size check operations.Copy performs after a copy,
respecting --ignore-size and backends which do not report sizes.
The SFTP page said `--dump-auth`, which no longer exists: it became a
value of `--dump`, so the docs asked for a flag rclone would reject.
The same line, and a line in the Swift troubleshooting section, also
used `--dump-headers` and `--dump-bodies`. Those still parse, but
SetFlags logs "--dump-headers is obsolete - please use --dump headers
instead", so the docs were steering readers onto a deprecated form.
The generated flag listings in docs/content/flags.md and
docs/content/commands/rclone.md are left alone: those flags do still
exist, so `--help` output should keep showing them.
When the source supplied fewer bytes than its declared size, the
compressed data file was stored under a name containing the declared
size while the metadata recorded the actual number of bytes read.
NewObject looks the data file up by the size in the metadata, so the
resulting object could never be read again, and the upload reported
success.
Check that the number of bytes read matches the declared size after
uploading the data and before writing the metadata, and remove the
data file and return an error if it does not.
This was found by the new FsPutShortEOF integration test.
The append loop retries everything once the upload session has
started, so a cancelled context error was retried through all the low
level retries with exponential backoff before the upload gave up.
A source which returned EOF before supplying as many bytes as it
declared would either commit a truncated file (if the shortfall was
within the final chunk) or loop forever appending empty chunks to the
upload session. Return an error wrapping io.ErrUnexpectedEOF instead.
Note that all dropbox uploads use the chunked upload path with the
default batch_mode of sync, so this affected uploads of every size.
A source reader which returns EOF before supplying as many bytes as it
declared should not cause a backend Put to hang, panic or create an
object claiming the declared size.
The layers above detect a truncated transfer by comparing sizes after
the upload, so a backend may either return an error or create a
shorter object.
See #9704
Object.Update held its connection until the deferred putConnection ran at
function exit, so the SetModTime it does at the end of every upload had to take
a second connection from the pool, dialling a whole new SMB session when the
pool was empty. With N transfers in flight the pool grew to roughly 2N sessions
for no reason.
Return the connection as soon as the file is closed. At that point the upload
has succeeded and remove() can no longer be reached, so nothing else needs it,
and SetModTime picks the same connection straight back out of the pool.
putConnection nils the pointer, so the deferred putConnection becomes a no-op
and the connection is not returned twice.
purgeCheck listed a directory's children with includeAll set, so the Drive API
returned trashed children as well as live ones. The only thing that came of
that was the trashedFiles flag, which is used solely to decide whether to trash
the directory or delete it permanently, and when use_trash is on the directory
gets trashed either way.
That made the common case pay for nothing: after trashing N files in a
directory, removing the now empty directory paged through all N trashed entries,
about one API call and pacer delay per 1000, instead of running a single query
that returns nothing.
Ask the server to filter out trashed children when the directory is going to be
trashed anyway. Hard deletes still enumerate them, since there the distinction
decides whether the directory can be removed permanently (#1040), and
--drive-trashed-only is left alone because it deliberately lists trashed items.
MoraxandNick Craig-Wood
Pastalikek65andNick Craig-Wood
Dean ChenandNick Craig-Wood
Teddy TennantandNick Craig-Wood
phatlcandNick Craig-Wood
Jiacheng XuandNick Craig-Wood
RecoordinateandGitHub
Rodrigo RodriguesandGitHub
Christian De SantisandGitHub
nielashandNick Craig-Wood
Edward KleselandNick Craig-Wood
Anton KarpovandNick Craig-Wood
alliasgherandNick Craig-Wood