Add fshttp.SetFaultInjector, which installs a function consulted by
every Transport before a request is sent. The injector can synthesise
an error status code or a transport error for chosen requests. The
request body is drained and closed as a real round trip would, but
nothing reaches the server.
This lets the integration tests check that backends cope with a
transient failure part way through an upload - in particular that a
retry re-sends the same data rather than an already consumed or
freed buffer - without needing a fake server for each backend.
Account.WriteTo wrote each buffer to the destination in full before
trimming the byte count for --max-transfer, so up to one buffer past
the limit could reach the wire and go unaccounted. This matters now
that NoCloser forwards WriteTo.
Truncate the write to the remaining allowance before writing.
The concurrent walker created by walk() only stopped when the callback
returned an error or the whole tree had been listed. Cancelling the
context (for example via the rc job/stop endpoint for an async
operations/size or recursive operations/list call) was therefore
ignored: the checkers kept pulling list jobs from the channel and kept
listing the entire tree, burning CPU and making job cancellation
useless for every backend without a native ListR implementation.
Make every checker select on ctx.Done() so a cancelled walk shuts down
promptly through the existing quit/drain path and reports the context
error. Also check the context between directory read chunks in the
local backend so a single huge directory does not block cancellation.
The dataflow analysis behind SA4023, new in the staticcheck 0.8.0
bundled with golangci-lint v2.13.0, makes linting large packages more
than 10x slower (89s vs 7s for backend/s3 alone) which took the CI
lint job past its 30 minute limit. golangci-lint no longer enforces
its run timeout during analysis so the job ran until cancelled, and
the cancellation meant the lint cache was never saved, making every
subsequent run cold and guaranteeing the timeout repeated.
The check also produces false positives (eg claiming operations.Delete
never returns nil).
This commit removes the workings of the old web ui which hasn't been
maintained for 6 years. If users supply --rc-web-gui then rclone will
exit with an error pointing users at the maintained `rclone gui`
command.
The test skipped every buffer count above 1 under -race, pointing at
golang/go#27070. That issue was closed in September 2018, so the
workaround outlived its cause: -race covered 137 of the 681 subtests.
Without the guard the race build runs all 681 and passes.
Allow --order-by to rank files using comma-separated rclone path globs. Patterns are evaluated in order, unmatched files are placed last, and path ordering makes ties deterministic.
Fixes#3975
When a multi-thread source ignores ranged reads, abort the partial
destination and retry the copy as a single stream. Reset accounting
before the fallback so transfer progress remains accurate.
Add response validation for calls made with Range open options. Verify
Content-Range, Content-Length, response status, and the complete
representation size before a backend accepts the response body.
Return a shared sentinel when a server ignores a partial range so callers
can avoid retrying the same unsupported request.
Before this change the environment variable getters in fs/configmap.go
logged the option value with %q, so a password set via
RCLONE_CONFIG_remote_pass (or RCLONE_remote_pass) was printed in full
to the debug log. Values from the config file were already redacted,
which made the leak easy to miss.
This change routes both getters through fs.RedactOptionValue, which
looks up the option in the backend's option list: options marked
IsPassword or Sensitive log as XXX, unknown options are conservatively
redacted, and --dump auth still shows the value for debugging.
Fixes#5794
rclone checksum and rclone check --checkfile hashed one file at a time
unless --download was in use. The non-download branch of CheckSum called
Object.Hash synchronously from inside the listing callback, so --checkers
had no effect there; only the --download branch used the checker token
pool.
Both branches now do their work in the same goroutine bounded by
--checkers, so --checkers means the same thing with and without
--download. Checking a sum file against 2.6 GiB of local files with
--checkers 8 went from 1.70s to 0.31s here.
As a result the "hashing" checking transfer now covers the hashing work
in both branches, rather than being finished as soon as the download
goroutine had been started.
RcatSize streams the body straight into Put behind an ObjectInfo with no
hashes, so a known-size upload had its size checked but its checksum
never verified, unlike the unknown-size path through Rcat which hashes as
it goes.
The stream can only be read once, so hash it on the way past and compare
with the destination after the upload, as operations.Copy does. A
destination which reports no usable hash is still checked by size, and one
which fails either check has the failed copy removed.
Before this change, `NewStats` stored the context it was created from on the
`StatsInfo`. Stats groups are never freed -- they are only evicted once there are
`--max-stats-groups` of them -- so each one kept its context, and everything reachable
from it, alive for the life of the process. As the rc creates a group per call,
that included the call's filters and their compiled regexps.
The context was only ever used to get `ci.StatsFileNameLength` from the config.
`StatsInfo` already stores that same `*fs.ConfigInfo`, read from the same
context in `NewStats`.
This change fixes the issue by passing the stored ci to `transferMap.String` and
dropping the context from `StatsInfo`.
Before this change, Transfer.Done closed the account of a completed transfer but
never released it, because it assigned nil to only a local copy of the pointer.
As a result, every completed transfer continued to reference its account.
An Account holds the transfer context and the source reader, and the stats keep
completed transfers around up to `MaxCompletedTransfers` per group, with groups
discarded only at --max-stats-groups. On a long-running `rclone rcd`, this adds up.
It was noticeable when running bisync repeatedly via the rc, where the transfer
context carries `b.WriteResults` (bisync's `LoggerFn`). A `*bisyncRun` holds
Path1 and Path2 listings, which can be quite large, and are not supposed to be
retained between runs. (Naturally they aren't, when running bisync on the
command line -- which is probably why we didn't notice this issue sooner.)
This change fixes the issue by releasing `tr.acc` in Done, instead of the local
copy. `tr.Snapshot` reads the byte counts off the account, so the progress is
recorded on the transfer first. That read happens before taking `tr.mu`, because
`acc.progress()` locks `acc.values.mu`, `checkReadBefore` holds that lock while
calling `StatsInfo.GetBytes`, and `StatsInfo` locks back into `Transfer` in
`Transferred` and `_removeTransfer`.
Uploads through RcatSize with a known size - used by rcat --size, the
rc operations/uploadfile and the serve backends, eg serve restic - did
not check the size of the uploaded object. If the source stream ended
before the declared size worth of data had been read, the truncated
object was reported as a successful upload. This could corrupt data
for callers which trust the result, eg a restic repository accessed
via serve restic (see #9722).
This adds the same size check operations.Copy performs after a copy,
respecting --ignore-size and backends which do not report sizes.
With --rc-serve set the root listing enumerated the names of all configured
remotes without any authentication.
Make the root listing obey the same fail-closed rule as the rest of
the rc endpoints: it now requires authentication to be configured or
an explicit opt out with --rc-no-auth.
Addresses GHSA-mfvx-7rcj-9m5g finding 2.
The pprof debug handlers were accessible without authentication disclosing the
process command line (which can carry backend credentials passed on the command
line) and runtime profiles.
Mount the pprof handlers only when when auth is configured or --rc-no-auth was
passed - so they obey the same rule as the rc endpoints.
Addresses GHSA-mfvx-7rcj-9m5g finding 1.
Before this change, rclone sent stack traces to the client on panic
capture in the rc. Stack traces can leak information which could be
useful to an attacker.
The march janitor goroutine, which discards queued jobs when the context is
cancelled, only ever returned on context cancellation. A march that finished
normally never cancels its context, so on an async rc job (whose context
descends from context.Background and is only cancelled by job/stop) the
janitor parked forever, leaking one goroutine per run and pinning that run's
directory listings in memory. A long-running rcd driving async sync or bisync
jobs accumulated these until it ran out of memory.
Signal the janitor to exit once the march completes so it returns on both
normal completion and cancellation.
When an HTTP/2 server retires a connection with GOAWAY after it has
already sent successful response headers, Go's http2 transport fails the
read of the response body with
http2: server sent GOAWAY and closed the connection; LastStreamID=..., ErrCode=NO_ERROR, debug=""
This was not recognised as a retriable networking error, so a transient
connection retirement aborted the whole command instead of consuming a
low level retry. It was reported against a large S3 check, where an
interrupted ListObjectsV2 page made rclone report destination objects as
missing and exit unsuccessfully.
The concrete error type is unexported by net/http, so match on the
message as we already do for the other http2 transport errors.
Sync refuses to delete files when the global error stats are non-zero
so a single backend error in one transform test made every following
transform test in the same test binary fail with "not deleting files
as there were IO errors". Reset the stats at the start of each test.
ownCloud does not carry the checksum over to the destination of a
server side copy and refuses attempts to set it afterwards, so the
destination legitimately has no hash. The logger vs lsf check
compared the predicted hash against the empty hash and failed.
Treat an empty hash in the listing as unknown rather than wrong,
matching how sync itself compares hashes.
Previously running rclone config (or driving it via the rc API or web
GUI) with -vv would write secrets to the debug log.
This was dangerous as users debugging a failing config flow often
paste their -vv logs into the forum or GitHub issues.
These values are now redacted from the log as "XXX". Values whose
option is known are only redacted if the option is marked IsPassword
or Sensitive, so normal answers remain visible.
Use --dump auth to see the unredacted values when debugging a config
flow - rclone prints a warning that secrets will appear in the log
when this is in effect.
This was discovered by CodeQL: https://github.com/rclone/rclone/security/code-scanning/182
The logging functions take an object which is rendered into the log
line. Rendering it with %+v dumps all its fields, which for an object
holding backend config would include decrypted credentials. Every
object currently logged is a string or has a String method, so render
anything else as just its type to keep credentials out of the logs.
See: https://github.com/rclone/rclone/security/code-scanning/183
TestRcDu relied on the default cache directory already existing. In clean
or container environments diskusage.New returned ENOENT, which the test
ignored before type asserting a nil result.
Use a temporary directory and require a successful response before
checking the disk usage values.
When doing a copy (no delete mode) without a logger, the destination
listing can be cancelled as soon as the source listing finishes, since
dst-only entries won't be processed.
This is particularly beneficial with --fast-list where the dst listing
may fetch the entire directory tree upfront via ListR. Cancelling it
early avoids waiting for a potentially large listing that won't be used.
Adds NoProcessDstOnly flag to March which, when set, cancels the dst
listing context once the source channel is exhausted in matchListings.
Fixes#9226
Add a context parameter to listDirFn so that each call site can pass
its own context. The closures in makeListDir previously captured
m.Ctx at creation time; they now use the context passed at call time
instead. This is needed so that processJob can pass a cancellable
context for the destination listing independently of the source.
Note: callers must pass m.Ctx (or a child of it) to preserve the
existing cancellation behaviour where listings stop when the march
context is cancelled.
Interactively-entered passwords were run through NFKC Unicode
normalization before being obscured, which silently rewrote characters
such as ª (U+00AA) to a. The obscured password then revealed to
something different from what the user typed confusing everyone.
Normalization is only needed for the config encryption master
password, so apply it there (in SetConfigPassword) rather than in the
shared checkPassword used for backend password options.
When `operations/stat` / StatJSON is called on a directory path it
lists the parent directory to find the target entry. If the parent has
millions of entries this is very expensive.
This fixes the problem for bucket-based backends with ListP by listing
the target directory itself first. It will stop the listing
immediately if any files are found meaning it is safe to run on
directories with millions of files.
The TestLogger/TestRepoCompare and TestLogger/TestBeforeVsAfter
testscript scenarios filled src and dst by downloading two old rclone
source archives from GitHub with `rclone copyurl`. Whenever GitHub or
the network hiccuped (eg a 502 Bad Gateway) the downloads failed and
the tests failed with it, making them flaky on CI.
Generate two overlapping trees of files in the test Setup instead.
They cover the same comparison categories the scripts exercise
(matching, differing, src-only and dst-only files) so the tests are
just as meaningful but no longer depend on the network.
Previously the only way to remove an option from a remote was to set it
to an empty string, which is not the same as deleting it - a present but
empty value overrides the option's default whereas a deleted key
restores it. Editing the file by hand isn't an option for an encrypted
config either.
This adds a "config unset" command and a "config/unset" rc endpoint to
remove one or more keys from an existing remote.
Closing a stream in the parallel chunked reader cancels the stream's
context, so the in-flight read returns context.Canceled. This was wrapped
and returned as "failed to read stream", which the VFS cache downloader
treats as a real download error - it only recognises
asyncreader.ErrorStreamAbandoned as a benign teardown, as returned by the
sequential reader.
Return asyncreader.ErrorStreamAbandoned for a cancellation so tearing down
the parallel reader (on close, seek or reposition) is recognised as benign,
matching the sequential reader, instead of logging download errors and
retrying when --vfs-read-chunk-streams is used with --vfs-cache-mode full.
When creating or updating a remote through the rc api (config/create,
config/update), parameters whose name starts with the ephemeral prefix
"config_" (for example config_template_file and config_template used to
customise the OAuth success page) were silently ignored.
updateRemote sets each supplied parameter into the config mapper, but
skips the "config_" prefixed keys so they are never written to the
config file. That guard is correct, because the mapper's setter writes
to the config file and these values are ephemeral. However backends read
these values back from the mapper (oauthutil reads config_template_file
and config_template via m.Get), so dropping them entirely meant the
values could never reach the backend and the default template was always
used.
Collect the ephemeral parameters into a separate map and add it to the
mapper as a getter overlay at PriorityNormal after the loop. The values
are now readable through m.Get without being written to the config file,
which is the same approach rclone authorize already uses to expose a
template supplied on the command line.
Fixes#9572
Co-authored-by: Hakanbaban53 <93117749+Hakanbaban53@users.noreply.github.com>
Co-authored-by: maximilize <3752128+maximilize@users.noreply.github.com>
A Range header requesting a suffix longer than the object (e.g.
"bytes=-90407" against a 5 byte object) caused RangeOption.Decode to
compute a negative offset (size - End), which serve.Object then used
directly as a slice/seek offset and panicked with "slice bounds out of
range". FixRangeOption (used by backends like OneDrive/Box that lack
native suffix-range support) had the same root cause: it produced a
RangeOption with a negative Start, which Header() silently dropped,
turning the request into the wrong byte range instead of erroring or
serving the whole object.
Per RFC 7233 section 2.1, when the suffix-length exceeds the
representation size, the entire representation should be served.
Clamp the computed offset/start to 0 in both places.
Fixes#6310
NewStatsGroup started the averageLoop goroutine unconditionally at
group creation. In an rcd daemon driven by many short rc sync/move
calls (a common pattern for scheduled spool flushes), each call gets
a fresh job/N stats group. When such a job transferred zero files
the loop was never stopped, because _stopAverageLoop is only reached
via DoneTransferring once transferring and checking both go from
non-empty back to empty, which never happens if nothing was ever
transferring in the first place. The result was one leaked goroutine
per rc call, growing unbounded until the daemon was OOM-killed
(reported: ~61k goroutines and ~640 MB RSS after ~7 days from a
per-minute timer over 6 mappings).
This is the same class of leak as #8571, which fixed the equivalent
auto-start in NewStats. Fix it the same way: do not start the average
loop at group creation. NewTransfer and NewTransferRemoteSize already
call startAverageLoop when real transfer activity begins, and
DoneTransferring already stops it when the last transfer completes,
so on-demand behaviour is unchanged for groups that actually do work.
Groups that never transfer anything now cost zero goroutines.
Adds a regression test that fails without the fix.
Fixes#9567
Nick Craig-Wood
Rahman YilmazandGitHub
maxandNick Craig-Wood
Shurong CaoandNick Craig-Wood
MoraxandNick Craig-Wood
Pastalikek65andNick Craig-Wood
Teddy TennantandNick Craig-Wood
phatlcandNick Craig-Wood
nielashandNick Craig-Wood
Yash AnilandNick Craig-Wood
Hakan İSMAİLandNick Craig-Wood
Søren LindbergandNick Craig-Wood
dougalandNick Craig-Wood
Cao YuhangandNick Craig-Wood
Amit MishraandNick Craig-Wood
blackflytechandNick Craig-Wood
GauravandGitHub
Sanjays2402andNick Craig-Wood