Syncs to backends which preserve directory modification times (eg
sftp, local) can update the modtime or metadata on many directories.
This count makes that work visible in the stats output, the core/stats
rc and the prometheus metrics (as dirs_updated_total).
Syncs which update lots of directories (eg to sftp) could spend a long
time setting directory modification times, making directories or
removing directories with no feedback in the --progress display or
stats, making rclone appear to have hung.
This shows directory operations (setting modtime, updating metadata,
making and removing directories) in the Checking section of the stats
and counts them as checks, in the same way file deletes are shown.
This creates a checking transfer which is shown in the progress
display while it is running but is not kept in the completed
transfers history, so it never appears in core/transferred and is not
retained in memory after it finishes.
This is for repeated bookkeeping operations (eg directory modtime
updates) which would otherwise crowd file transfers out of the
history.
This was fixed in this commit in an inelegant way
399bc6a6a6 fshttp: don't send --header values to other hosts on redirect
The current commit fixes it properly with AddConfig.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The headers set with --header and --header-download are added to
every request by the rclone transport, including redirect hops which
net/http makes to other hosts, so a credential passed with --header
for one host could be sent to any host that server chose to redirect
to.
The transport now walks the redirect chain net/http records on each
redirected request and, once the chain has visited a host other than
the one originally requested, removes the headers rather than adding
them.
Also restore the global --client-cert and --client-key config after
TestCertificates so its temporary files are not used by later tests.
The rclone core does not sanitise ".." in an object's Remote(). Such a name can
arrive from a malicious or buggy backend - an object store permits keys
containing ".." or a leading "/" - and, if acted on, lets a listing or transfer
escape the configured root. A source object named "../../other/x" is copied to
"other/x" outside the destination root, and a crafted listing name surfaces
outside the directory being listed.
Add list.RemoteEscapesRoot, which reports whether a Remote climbs above the
root when joined onto it, and list.RemoveEscaping, which drops and logs such
entries.
Apply RemoveEscaping unconditionally - independent of the include/exclude
filters - at the three per-entry filtering points every listing passes through:
filterDir, walk.listR and walk.walkRDirTree (recursive ListR).
operations.StatJSON calls List and NewObject directly, bypassing those, so it
rejects an escaping remote up front.
This confines every backend at once, so no per-backend change is needed.
Add fshttp.SetFaultInjector, which installs a function consulted by
every Transport before a request is sent. The injector can synthesise
an error status code or a transport error for chosen requests. The
request body is drained and closed as a real round trip would, but
nothing reaches the server.
This lets the integration tests check that backends cope with a
transient failure part way through an upload - in particular that a
retry re-sends the same data rather than an already consumed or
freed buffer - without needing a fake server for each backend.
Account.WriteTo wrote each buffer to the destination in full before
trimming the byte count for --max-transfer, so up to one buffer past
the limit could reach the wire and go unaccounted. This matters now
that NoCloser forwards WriteTo.
Truncate the write to the remaining allowance before writing.
The concurrent walker created by walk() only stopped when the callback
returned an error or the whole tree had been listed. Cancelling the
context (for example via the rc job/stop endpoint for an async
operations/size or recursive operations/list call) was therefore
ignored: the checkers kept pulling list jobs from the channel and kept
listing the entire tree, burning CPU and making job cancellation
useless for every backend without a native ListR implementation.
Make every checker select on ctx.Done() so a cancelled walk shuts down
promptly through the existing quit/drain path and reports the context
error. Also check the context between directory read chunks in the
local backend so a single huge directory does not block cancellation.
The dataflow analysis behind SA4023, new in the staticcheck 0.8.0
bundled with golangci-lint v2.13.0, makes linting large packages more
than 10x slower (89s vs 7s for backend/s3 alone) which took the CI
lint job past its 30 minute limit. golangci-lint no longer enforces
its run timeout during analysis so the job ran until cancelled, and
the cancellation meant the lint cache was never saved, making every
subsequent run cold and guaranteeing the timeout repeated.
The check also produces false positives (eg claiming operations.Delete
never returns nil).
This commit removes the workings of the old web ui which hasn't been
maintained for 6 years. If users supply --rc-web-gui then rclone will
exit with an error pointing users at the maintained `rclone gui`
command.
The test skipped every buffer count above 1 under -race, pointing at
golang/go#27070. That issue was closed in September 2018, so the
workaround outlived its cause: -race covered 137 of the 681 subtests.
Without the guard the race build runs all 681 and passes.
Allow --order-by to rank files using comma-separated rclone path globs. Patterns are evaluated in order, unmatched files are placed last, and path ordering makes ties deterministic.
Fixes#3975
When a multi-thread source ignores ranged reads, abort the partial
destination and retry the copy as a single stream. Reset accounting
before the fallback so transfer progress remains accurate.
Add response validation for calls made with Range open options. Verify
Content-Range, Content-Length, response status, and the complete
representation size before a backend accepts the response body.
Return a shared sentinel when a server ignores a partial range so callers
can avoid retrying the same unsupported request.
Before this change the environment variable getters in fs/configmap.go
logged the option value with %q, so a password set via
RCLONE_CONFIG_remote_pass (or RCLONE_remote_pass) was printed in full
to the debug log. Values from the config file were already redacted,
which made the leak easy to miss.
This change routes both getters through fs.RedactOptionValue, which
looks up the option in the backend's option list: options marked
IsPassword or Sensitive log as XXX, unknown options are conservatively
redacted, and --dump auth still shows the value for debugging.
Fixes#5794
rclone checksum and rclone check --checkfile hashed one file at a time
unless --download was in use. The non-download branch of CheckSum called
Object.Hash synchronously from inside the listing callback, so --checkers
had no effect there; only the --download branch used the checker token
pool.
Both branches now do their work in the same goroutine bounded by
--checkers, so --checkers means the same thing with and without
--download. Checking a sum file against 2.6 GiB of local files with
--checkers 8 went from 1.70s to 0.31s here.
As a result the "hashing" checking transfer now covers the hashing work
in both branches, rather than being finished as soon as the download
goroutine had been started.
RcatSize streams the body straight into Put behind an ObjectInfo with no
hashes, so a known-size upload had its size checked but its checksum
never verified, unlike the unknown-size path through Rcat which hashes as
it goes.
The stream can only be read once, so hash it on the way past and compare
with the destination after the upload, as operations.Copy does. A
destination which reports no usable hash is still checked by size, and one
which fails either check has the failed copy removed.
Before this change, `NewStats` stored the context it was created from on the
`StatsInfo`. Stats groups are never freed -- they are only evicted once there are
`--max-stats-groups` of them -- so each one kept its context, and everything reachable
from it, alive for the life of the process. As the rc creates a group per call,
that included the call's filters and their compiled regexps.
The context was only ever used to get `ci.StatsFileNameLength` from the config.
`StatsInfo` already stores that same `*fs.ConfigInfo`, read from the same
context in `NewStats`.
This change fixes the issue by passing the stored ci to `transferMap.String` and
dropping the context from `StatsInfo`.
Before this change, Transfer.Done closed the account of a completed transfer but
never released it, because it assigned nil to only a local copy of the pointer.
As a result, every completed transfer continued to reference its account.
An Account holds the transfer context and the source reader, and the stats keep
completed transfers around up to `MaxCompletedTransfers` per group, with groups
discarded only at --max-stats-groups. On a long-running `rclone rcd`, this adds up.
It was noticeable when running bisync repeatedly via the rc, where the transfer
context carries `b.WriteResults` (bisync's `LoggerFn`). A `*bisyncRun` holds
Path1 and Path2 listings, which can be quite large, and are not supposed to be
retained between runs. (Naturally they aren't, when running bisync on the
command line -- which is probably why we didn't notice this issue sooner.)
This change fixes the issue by releasing `tr.acc` in Done, instead of the local
copy. `tr.Snapshot` reads the byte counts off the account, so the progress is
recorded on the transfer first. That read happens before taking `tr.mu`, because
`acc.progress()` locks `acc.values.mu`, `checkReadBefore` holds that lock while
calling `StatsInfo.GetBytes`, and `StatsInfo` locks back into `Transfer` in
`Transferred` and `_removeTransfer`.
Uploads through RcatSize with a known size - used by rcat --size, the
rc operations/uploadfile and the serve backends, eg serve restic - did
not check the size of the uploaded object. If the source stream ended
before the declared size worth of data had been read, the truncated
object was reported as a successful upload. This could corrupt data
for callers which trust the result, eg a restic repository accessed
via serve restic (see #9722).
This adds the same size check operations.Copy performs after a copy,
respecting --ignore-size and backends which do not report sizes.
With --rc-serve set the root listing enumerated the names of all configured
remotes without any authentication.
Make the root listing obey the same fail-closed rule as the rest of
the rc endpoints: it now requires authentication to be configured or
an explicit opt out with --rc-no-auth.
Addresses GHSA-mfvx-7rcj-9m5g finding 2.
The pprof debug handlers were accessible without authentication disclosing the
process command line (which can carry backend credentials passed on the command
line) and runtime profiles.
Mount the pprof handlers only when when auth is configured or --rc-no-auth was
passed - so they obey the same rule as the rc endpoints.
Addresses GHSA-mfvx-7rcj-9m5g finding 1.
Before this change, rclone sent stack traces to the client on panic
capture in the rc. Stack traces can leak information which could be
useful to an attacker.
The march janitor goroutine, which discards queued jobs when the context is
cancelled, only ever returned on context cancellation. A march that finished
normally never cancels its context, so on an async rc job (whose context
descends from context.Background and is only cancelled by job/stop) the
janitor parked forever, leaking one goroutine per run and pinning that run's
directory listings in memory. A long-running rcd driving async sync or bisync
jobs accumulated these until it ran out of memory.
Signal the janitor to exit once the march completes so it returns on both
normal completion and cancellation.
When an HTTP/2 server retires a connection with GOAWAY after it has
already sent successful response headers, Go's http2 transport fails the
read of the response body with
http2: server sent GOAWAY and closed the connection; LastStreamID=..., ErrCode=NO_ERROR, debug=""
This was not recognised as a retriable networking error, so a transient
connection retirement aborted the whole command instead of consuming a
low level retry. It was reported against a large S3 check, where an
interrupted ListObjectsV2 page made rclone report destination objects as
missing and exit unsuccessfully.
The concrete error type is unexported by net/http, so match on the
message as we already do for the other http2 transport errors.
Sync refuses to delete files when the global error stats are non-zero
so a single backend error in one transform test made every following
transform test in the same test binary fail with "not deleting files
as there were IO errors". Reset the stats at the start of each test.
ownCloud does not carry the checksum over to the destination of a
server side copy and refuses attempts to set it afterwards, so the
destination legitimately has no hash. The logger vs lsf check
compared the predicted hash against the empty hash and failed.
Treat an empty hash in the listing as unknown rather than wrong,
matching how sync itself compares hashes.
Previously running rclone config (or driving it via the rc API or web
GUI) with -vv would write secrets to the debug log.
This was dangerous as users debugging a failing config flow often
paste their -vv logs into the forum or GitHub issues.
These values are now redacted from the log as "XXX". Values whose
option is known are only redacted if the option is marked IsPassword
or Sensitive, so normal answers remain visible.
Use --dump auth to see the unredacted values when debugging a config
flow - rclone prints a warning that secrets will appear in the log
when this is in effect.
This was discovered by CodeQL: https://github.com/rclone/rclone/security/code-scanning/182
The logging functions take an object which is rendered into the log
line. Rendering it with %+v dumps all its fields, which for an object
holding backend config would include decrypted credentials. Every
object currently logged is a string or has a String method, so render
anything else as just its type to keep credentials out of the logs.
See: https://github.com/rclone/rclone/security/code-scanning/183
TestRcDu relied on the default cache directory already existing. In clean
or container environments diskusage.New returned ENOENT, which the test
ignored before type asserting a nil result.
Use a temporary directory and require a successful response before
checking the disk usage values.
When doing a copy (no delete mode) without a logger, the destination
listing can be cancelled as soon as the source listing finishes, since
dst-only entries won't be processed.
This is particularly beneficial with --fast-list where the dst listing
may fetch the entire directory tree upfront via ListR. Cancelling it
early avoids waiting for a potentially large listing that won't be used.
Adds NoProcessDstOnly flag to March which, when set, cancels the dst
listing context once the source channel is exhausted in matchListings.
Fixes#9226
Add a context parameter to listDirFn so that each call site can pass
its own context. The closures in makeListDir previously captured
m.Ctx at creation time; they now use the context passed at call time
instead. This is needed so that processJob can pass a cancellable
context for the destination listing independently of the source.
Note: callers must pass m.Ctx (or a child of it) to preserve the
existing cancellation behaviour where listings stop when the march
context is cancelled.
Interactively-entered passwords were run through NFKC Unicode
normalization before being obscured, which silently rewrote characters
such as ª (U+00AA) to a. The obscured password then revealed to
something different from what the user typed confusing everyone.
Normalization is only needed for the config encryption master
password, so apply it there (in SetConfigPassword) rather than in the
shared checkPassword used for backend password options.
When `operations/stat` / StatJSON is called on a directory path it
lists the parent directory to find the target entry. If the parent has
millions of entries this is very expensive.
This fixes the problem for bucket-based backends with ListP by listing
the target directory itself first. It will stop the listing
immediately if any files are found meaning it is safe to run on
directories with millions of files.
The TestLogger/TestRepoCompare and TestLogger/TestBeforeVsAfter
testscript scenarios filled src and dst by downloading two old rclone
source archives from GitHub with `rclone copyurl`. Whenever GitHub or
the network hiccuped (eg a 502 Bad Gateway) the downloads failed and
the tests failed with it, making them flaky on CI.
Generate two overlapping trees of files in the test Setup instead.
They cover the same comparison categories the scripts exercise
(matching, differing, src-only and dst-only files) so the tests are
just as meaningful but no longer depend on the network.
Previously the only way to remove an option from a remote was to set it
to an empty string, which is not the same as deleting it - a present but
empty value overrides the option's default whereas a deleted key
restores it. Editing the file by hand isn't an option for an encrypted
config either.
This adds a "config unset" command and a "config/unset" rc endpoint to
remove one or more keys from an existing remote.
Nick Craig-Wood
Rahman YilmazandGitHub
maxandNick Craig-Wood
Shurong CaoandNick Craig-Wood
MoraxandNick Craig-Wood
Pastalikek65andNick Craig-Wood
Teddy TennantandNick Craig-Wood
phatlcandNick Craig-Wood
nielashandNick Craig-Wood
Yash AnilandNick Craig-Wood
Hakan İSMAİLandNick Craig-Wood
Søren LindbergandNick Craig-Wood
dougalandNick Craig-Wood
Cao YuhangandNick Craig-Wood