Set built the timetable with *x = append(*x, ts), so setting a bandwidth
timetable on a value that already held one kept both schedules. The single-value
branch of the same function has always done *x = BwTimetable{ts}, and the other
multi-token Set methods in this package build into a local and assign at the end.
The visible effect is through the rc API. The "main" options block registered in
fs.RegisterGlobalOptions is the live globalConfig, and options/set reshapes JSON
straight into it, so
rclone rc options/set --json '{"main": {"BwLimit": "Mon-10:00,1Mi"}}'
added to the running daemon's timetable rather than replacing it, and the older
slot kept winning: LimitAt for a Sunday returned the previous 10Mi. The same
applies to a _config override on a single call, since AddConfig shallow-copies
the global.
Building into a local also stops a failed parse from leaving the previous
timetable partly overwritten, which the existing error cases already expect.
Ceph RGW (and Linode Object Storage, which is Ceph-backed) can break
SigV4 when Accept-Encoding is included in the signature, especially
when a reverse proxy rewrites that header. GCS already sets this quirk;
apply the same default for Ceph and Linode as suggested in #8206.
Set the OCI source SSE-C request headers when using a customer key.
Server-side copies need these headers to decrypt the source object, in
addition to the existing headers that encrypt the destination.
The test asks InstallUpdate to install the latest beta into an
unwritable file and expects an error. When the binary under test
reports exactly the latest beta version (as make quicktest does right
after a beta is published from the same commit), InstallUpdate
correctly decides there is nothing to do and returns nil, and the test
then dereferences the nil error and panics.
Pin fs.Version to a fixed old value for the duration of the test so an
update is always attempted, and use require.Error so a missing error
fails the test instead of crashing it.
Decrypting the config with --daemon runs --password-command twice, which
means two authentications when the command needs one, such as a hardware
key touch for `pass show`.
SetConfigPassword saves the obscured key to the temp file named by
_RCLONE_CONFIG_KEY_FILE so the daemon process can pick it up, but the
process that wrote it then read and deleted that file itself before
daemonizing. The daemon started with the variable pointing at a file that
was already gone, found no key, and ran the password command again.
Skip acquiring a password when _RCLONE_CONFIG_KEY_FILE is set, as the
PassConfigKeyForDaemonization documentation already describes, and only
consume the key file in a process that has no key of its own. The parent
then leaves the key for the daemon, and the daemon uses it.
Fixes#7341
applyOptions consumes the "path" option into vol.Path rather than leaving
it in vol.Options, but restoreState rebuilt the options with only fs and
type. The explicit path was therefore dropped when the plugin restarted,
and since fsString is rebuilt from those options the volume was remounted
at the root of the remote instead of at its subpath.
Before this change a volume created with type + path lost its path
completely, and one created with remote + path silently fell back to the
path of the connection string. With a backend whose credentials are
scoped to the subpath the restored mount then failed every operation
rather than serving the wrong directory.
Feed the persisted path back like fs and type, so applyOptions applies
the same precedence on restore that it applies when the volume is
first created.
Fixes#9853
The Dropbox backend advertises CaseInsensitive: true, but the two
shared-mode lookup helpers compared names with an exact, case-sensitive
==, so a shared folder or received file named "Project" could not be
found when requested as "project". Use strings.EqualFold in both
findSharedFolder and findSharedFile to honour the advertised
case-insensitivity.
Fixes#9706
In shared_folders mode NewFs derived the shared folder name with
path.Dir(f.root), which returns the parent path rather than the first
path component. For a root like "SharedFolder/subdir/deeper" this yielded
"SharedFolder/subdir", which findSharedFolder cannot match, so NewFs
failed with ErrorDirNotFound. Use the first path component of the root,
as the shared_folders option documents, so deeply nested roots mount.
Fixes#9705
The flag applies to out of space errors while writing and while creating
files or directories, not only while writing. Describe those operations
without naming ENOSPC, which is a Unix error that Windows never reports, so
the help is accurate on every platform.
KickCleaner sets the out of space flag, kicks the cleaner and then waits for
that flag to clear. Only the cleaner clears it, and the cleaner returns
immediately when the cache poll interval is not positive, so when it is
disabled nothing ever reads the kick or clears the flag and the caller waits
forever.
It now returns straight away in that case, under the same condition the
cleaner itself uses to decide it is disabled. Callers already retry a bounded
number of times and then report the error, which is the right outcome when
nothing is going to free space.
IsErrNoSpace compared against syscall.ENOSPC. Go defines that constant on
Windows as a value in its application reserved range which no Windows API
returns, so the comparison could never be true there. A full disk on Windows
reports ERROR_DISK_FULL or ERROR_HANDLE_DISK_FULL instead.
Preallocation failures were still caught, because those return a separate
sentinel, but a disk that is already full fails at the directory creation or
at the open long before preallocation is reached. That is the case reported.
The errors are now held in a list which platform specific files add to in
their init, which is the shape retriable_errors already uses in this package,
and the comparison itself is unchanged. Windows appends the two codes that
lib/file already recognises when preallocation fails. Every other platform
keeps exactly the behaviour it had.
This also reaches the VFS cache, which uses the same helper and has no
preallocation path of its own, so its out of space handling has been inert
on Windows.
Google now requires an app homepage URL and privacy policy URL to be
set on the OAuth consent screen's "Branding" page before the "PUBLISH
APP" button becomes clickable, even for a personal single-user app.
The existing instructions jumped straight to publishing in step 9
without mentioning this, leaving the button greyed out with no
explanation of why.
Fixes#9854
Add operations/cat endpoint to the Remote Control (RC) API to allow reading
and streaming file contents in-process over librclone / FFI and HTTP RC.
Supports range options (offset, count, head, tail), separator, optional maxSize
buffer limit, and returns both string and base64 encoded results.
The code which opens the --combined, --differ etc report files (or
stdout for "-") and closes them afterwards was duplicated between the
check command and the sync logger flags. This moves it into
operations.OpenReportFiles which both now use. It also closes any
files already opened if a later one fails to open.
The sync report flags (--combined, --missing-on-src, --missing-on-dst,
--match, --differ, --error and --dest-after) were only wired up in the
CLI commands so there was no way to get these reports over the rc or
from librclone.
This adds boolean parameters of the same names as operations/check
(combined, missingOnSrc, missingOnDst, match, differ, error and
destAfter) to sync/sync, sync/copy and sync/move. Each requested
report is returned as an array of strings in the output, just as
operations/check does. All default to off so existing callers see no
change in the output.
To share the code between the CLI and the rc the report writer helper
from operations/check is exported as operations.RcReportWriter, the
lsf defaults for --dest-after are moved into
operations.NewSyncLoggerOpt and the listing setup and --no-traverse
warnings from operationsflags.ConfigureLoggers into LoggerOpt.Init.
Before this change `rclone dedupe --dedupe-mode rename` probed the
backend for each candidate `name-N.ext` in turn and gave up when it
had tried 100 names for a given object. With daily runs against the
same duplicated filename this ceiling was eventually reached and
rclone logged "Could not find an available new name". Each probe was
also a backend lookup, so a run against 99 existing names took
minutes on Google Drive.
The rename now uses the listing dedupe has already made to skip names
known to be taken without asking the backend, and only confirms the
final candidate with NewObject (the listing may be incomplete because
of filters). The suffix counter is shared between the objects being
renamed so no name is checked twice. The safety limit is raised to
10000 which, thanks to the listing, no longer costs a lookup per name.
The deferred cleanup in the constructor checked a local error variable
rather than the error being returned, so failures after the VFS was
created (such as an invalid --passive-port) never shut it down. Name
the error return so the cleanup sees the returned error.
When the HTTP server failed to initialise, for example because the
listen address was already in use, rclone panicked with a nil pointer
dereference instead of reporting the error.
The deferred cleanup in the constructor read the provider from the
named return value, but `return nil, err` sets that to nil before the
deferred function runs. Use a local variable for the server instead.
Writing to the mount with os.WriteFile made the Go runtime register
the file with its poller so the kernel then polled the file from
epoll_ctl and epoll_wait, sending POLL requests to the FUSE server
running in this same process. A thread waiting inside epoll cannot be
preempted by the runtime, so a garbage collection starting while such
a POLL was outstanding stopped the world for good - the test binary
could not be killed even with SIGKILL and the mount was left behind,
wedging anything that touched it.
Now we write through the mount with a descriptor straight from
open(2), which os.NewFile keeps out of the poller, check that it
really is out of the poller with SetDeadline, and check at the end of
the test that the mountpoint is unmounted.
In this commit we fixed the same problem for mount by running in a
subprocess however changing one write file routine here was much
easier than re-arranging the tests.
4a382c09ec mount: run tests in a subprocess to fix deadlock - #3259
Note that go-fuse (and hence mount2) works around this problem it by
forcing an early POLL it can answer with ENOSYS.
See: https://github.com/golang/go/issues/21014
The check that an entry returned by an archiver is a direct child of
the directory being listed normalised a parent of "/" to the root, so
an entry named "/x" passed as a child of the root while "x/" and
"dir//x" were rejected.
Decide by stripping the directory prefix and checking what is left
with sanitize.Leaf, which rejects an empty name, ".", ".." and any
name containing a "/". This also covers the leading slash case.
A file entry in a zip whose name refers to a directory, such as ".",
"/", "" or "sub/.", was only skipped when it named the root of an
archive which was itself the root of the remote. When the archive was
found by listing its parent directory the entry appeared as a file
with the same name as the archive alongside the directory for it, and
copying the archive tried to write both. When the entry named a
subdirectory it appeared as a file alongside that directory, and with
that subdirectory mounted as the archive root the entry was taken to
be the single file the root points at, hiding every real entry.
Skip any file entry whose last path component is "", "." or "..",
checked on the raw name before it is cleaned or joined on the prefix.
The zip archiver handed out its cached directory tree directly. Any
caller which filters a listing in place (as the core listing code
does) altered the cache, so later listings of the same directory could
be corrupted.
Return a copy of the cached listing instead.
When an upload failed with a 500 error the upload was retried with a
new upload link but the same input stream. The stream had already been
consumed by the first attempt so the retry uploaded an empty file.
This fixes it by returning a RetryError instead so the caller retries
the upload with a fresh stream, which will fetch a new upload link.
When an upload failed with a retryable error the pacer retried the
whole upload with the same input stream. The stream had already been
consumed by the first attempt so the retry uploaded an empty file.
This fixes it by using CallNoRetry for the upload, as the other
backends do, so retryable errors are returned wrapped in a RetryError
for the caller to retry the upload with a fresh stream.
It also makes 5xx errors from the upload storage servers retryable.
These come back from the SDK as a different error type to API errors
so were not being retried at all.
When an upload failed with a retryable error the pacer retried the
whole PUT with the same input stream. The stream had already been
consumed by the first attempt so the retry uploaded an empty file.
This fixes it by using CallNoRetry for the upload, as the other
backends do, so retryable errors are returned wrapped in a RetryError
for the caller to retry the upload with a fresh stream.
MkdirModTime checked --interactive/--dry-run itself and then called
MkdirMetadata or Mkdir which check again, so --interactive asked twice
about making the same directory and --dry-run skipped before the
operation could be shown in the progress display or counted as a
check.
Now MkdirModTime decides how to make the directory first and delegates
entirely to MkdirMetadata, or Mkdir followed by SetDirModTime, each of
which does its own --interactive/--dry-run check exactly once. This
also means the modtime setting fallback shows in the progress display,
respects --no-update-dir-modtime and has its errors counted.
When syncing to a backend which supports directory modtimes, a
directory which needed its modtime updated and which had files
transferred into it would get its modtime set twice - once when the
directory was checked and once in the pass at the end of the sync.
Now, when the end of sync pass is in use (which it is for all default
syncs), directories which need their modtime (or metadata) updating
are marked for that pass instead of being updated immediately. This
halves the number of directory modtime updates in a typical sync and
makes the "Updated dirs" stat count each directory once.
Syncs to backends which preserve directory modification times (eg
sftp, local) can update the modtime or metadata on many directories.
This count makes that work visible in the stats output, the core/stats
rc and the prometheus metrics (as dirs_updated_total).
Syncs which update lots of directories (eg to sftp) could spend a long
time setting directory modification times, making directories or
removing directories with no feedback in the --progress display or
stats, making rclone appear to have hung.
This shows directory operations (setting modtime, updating metadata,
making and removing directories) in the Checking section of the stats
and counts them as checks, in the same way file deletes are shown.
This creates a checking transfer which is shown in the progress
display while it is running but is not kept in the completed
transfers history, so it never appears in core/transferred and is not
retained in memory after it finishes.
This is for repeated bookkeeping operations (eg directory modtime
updates) which would otherwise crowd file transfers out of the
history.
This was fixed in this commit in an inelegant way
399bc6a6a6 fshttp: don't send --header values to other hosts on redirect
The current commit fixes it properly with AddConfig.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The headers set with --header and --header-download are added to
every request by the rclone transport, including redirect hops which
net/http makes to other hosts, so a credential passed with --header
for one host could be sent to any host that server chose to redirect
to.
The transport now walks the redirect chain net/http records on each
redirected request and, once the chain has visited a host other than
the one originally requested, removes the headers rather than adding
them.
Also restore the global --client-cert and --client-key config after
TestCertificates so its temporary files are not used by later tests.
The headers set with --http-headers are documented for passing
credentials such as Authorization or Cookie. The backend used the
default net/http redirect policy which copies all but a handful of
well known headers to any redirect target, so a redirect from the
configured server to another host would send those credentials to
that host, and a redirect from https to http would send them in
plaintext.
When headers are configured this installs a CheckRedirect function
which:
- removes the configured headers from every hop once the redirect
chain has left the originally requested host
- refuses a redirect from https to http with an error
SameHost compares two URLs by host name (case insensitively) and port
(treating the scheme's default port as no port) so redirect policies
can tell a real change of host from a server spelling its own host
differently, e.g. redirecting "https://example.com/" to
"https://EXAMPLE.com:443/".
The HTTPS downgrade check now compares the redirect target against
the original request rather than the previous hop, so a chain which
started on plaintext http, passed through an https server and came
back to http is no longer refused - nothing is being downgraded
relative to what the user asked for. A chain which started on https
and reaches http via any number of hops is still refused.
Whether an archive entry name can escape the archive's namespace was
left entirely to each archiver. Enforce it in the archive backend too.
List only passes on direct children of the directory listed and
NewObject only returns the object asked for, so a future archiver
which forgets to validate names cannot expose a traversal to fs/sync
and fs/operations.
The path inside the archive was compared against the cleaned entry
names without being cleaned itself, so `archive.zip/sub/./dir` or
`archive.zip/sub//dir` failed to list even though `archive.zip/sub/dir`
worked.
A zip containing a file entry whose name refers to the archive's own
root (".", "/" or "") was presented as a single file called "." and
all its other entries disappeared. A file at the root can only be the
archive member the backend was pointed at, so with no root such an
entry is skipped like any other unsafe name.
Entry names read from a squashfs directory are not sanitized by
go-diskfs. The squashfs backend joined each leaf name onto its
directory to form the object's remote, so a crafted image could escape
its directory.
Use sanitize.Leaf to skip unsafe entries in List. A "\" is an
ordinary character in a file name on the systems squashfs images are
made on and in an rclone remote path, so it is deliberately not
rejected; making it safe for the destination is the destination
backend's job.
Skipped entries are logged at DEBUG with a single NOTICE count per
listing so a crafted image under a mount cannot flood the log.
When a zip archive was mounted at a subdirectory root, readZip used a bare
strings.HasPrefix to decide which entries fell inside the root. This
matched on a raw string prefix rather than a path boundary, so mounting
root "foo" also exposed sibling entries such as "foobar/..." with their
names left uncorrected.
Require a path boundary when filtering by root.
The zip backend mounts a zip file as a browsable Fs. Go's archive/zip
does not sanitize entry names, and readZip applied path.Clean but did
not reject a cleaned name that still pointed outside the archive. A
crafted zip could make rclone copy/sync attempt writes outside the
intended destination.
Sanitize entry names with sanitize.Path - the same check used by
rclone archive extract - skipping any entry with a ".." path
component, whether separated by "/" or "\". A backslash is otherwise
kept as an ordinary character in the name, as archive extract does. It
is up to the destination backend to make names safe for its storage.
Skipped entries are logged as a single count per archive so a crafted
archive with many escaping entries cannot flood the log.
youdie006andNick Craig-Wood
Nicholas VeltenandNick Craig-Wood
AdityaandNick Craig-Wood
elioteeandNick Craig-Wood
tomaszniandGitHub
Murat TopcuandNick Craig-Wood
phatlcandNick Craig-Wood
Vaibhav MashalandGitHub
Sanjay Kanth AandGitHub
ferrumclaudepilgrimandNick Craig-Wood
Hakan İSMAİLandGitHub
Dhevenddra K GandGitHub
