Add a ctx field to the VFS struct, initialized in New() from the
existing cancellable context. Propagate this through the cache
subsystem hierarchy.
This ensures proper context cancellation when a VFS shuts down, rather
than using disconnected context.TODO() or context.Background() calls
throughout and paves the way for VFS to have its own config.
The flag.Lookup("test.v") check existed to skip opening a browser
during tests, but the tests don't exercise RunE, so this was never
used. The --no-open-browser flag is sufficient on its own.
Bind the RC server to localhost:0 and read the bound URL back via a
new rcserver.Server.URLs() accessor instead of pre-allocating a port
in cmd/gui. This removes the small TOCTOU race window between
freePort() closing its listener and rcserver claiming the same port.
- Fail gracefully if `make fetch-gui` hasn't been run
- Return errors instead of panic or fatal errrors
- Don't run `make fetch-gui` on every make since we have it in the workflow
The fetch-gui-dist.sh script calls the GitHub releases API
unauthenticated, which is limited to 60 requests/hour per source IP.
GitHub Actions runners share outbound IPs, so this quota is regularly
exhausted.
Pass GITHUB_TOKEN (or GH_TOKEN) as an Authorization header when
present, raising the limit to 1000/hour, and wire secrets.GITHUB_TOKEN
into the workflow step. Local unauthenticated runs still work.
These Debugf calls in NewFilter() ran during GlobalOptionsInit(), before
InitLogging() configured the JSON log format. This caused plain-text
debug lines to leak to stderr when --use-json-log was set, breaking
tooling that expected only JSON output.
The resolved time values are already available via --dump filters so
this commit removes the debug messages.
Previously ssh.InsecureIgnoreHostKey() was set unconditionally as the
default HostKeyCallback with no indication to the user.
This logs a warning pointing users to the documentation on how to
enable host key validation.
See: https://github.com/rclone/rclone/security/code-scanning/167
Before this change when a cache item was in its grace period (with
HandleCaching) and the file is reopened, _checkObject runs before the
grace timer recovery check. If the remote object's fingerprint changed
_checkObject removes the cache file from disk. However the grace
recovery path still reused the now-stale fd pointing to a deleted
inode, skipping _createFile entirely. This left no cache file on disk,
causing cache.Exists() to return false and breaking
rename-while-writing logic.
Fix this by checking the cache file still exists before reusing the fd
in grace recovery. If the file was removed, close the stale fd and
downloaders and fall through to _createFile.
Also update the fingerprint in item.rename after setting the new object,
preventing unnecessary cache invalidation when a file is reopened after
a rename.
This was discovered in the integration tests on backends that update
modtime on rename (like mailru).
The Linkbox open API (/api/open/file_search) no longer returns download
URLs, breaking all downloads. This switches to using the web API
(/api/file/my_file_list/web) which requires email+password authentication
but returns working download URLs.
This will unfortunately require changing your existing rclone config.
- Add email, password, and web_token config options
- Add web API login via /api/user/login_email with token caching and retry
- Create separate CDN HTTP client with HTTP/2 disabled and browser
User-Agent to avoid CDN fingerprint blocking
- Remove searchOK regex and name-filtering (web API doesn't support it)
In this commit
0db3e7a2a0 vfs: fix slow nfs serve by adding --vfs-handle-caching
We added --vfs-handle-caching but unfortunately forgot to disable it
for the TestRWCacheUpdate test.
Add a configurable grace period (default 5s) that delays closing file
handles and downloaders when the last handle closes. If a new handle
opens within the grace period, it reuses the existing resources.
This fixes 40x performance degradation with serve nfs vs serve sftp
caused by go-nfs opening/reading/closing on every NFS READ RPC, which
destroyed read-ahead prefetch before it could accumulate.
The grace period only applies to non-dirty files so that writeback
proceeds immediately on close.
Fixes#9251
- replace Bootstrap/jQuery with purpose-built CSS and JS
- remove backend icons from navbar and content pages
- replace remaining FontAwesome icons with inline SVGs, remove FontAwesome
- modernize CSS styling for menus, typography, cards, tables, and code blocks
- add copy-to-clipboard buttons on code blocks using SVG icon
- move TOC to left sidebar with responsive overlay drawer
- add sticky header, top scrollbar and first column for wide tables
- add left/right arrow buttons to scrollable tables
- hide homepage logo on mobile
- make wide menus with filter for Commands and Storage Systems
- add dark mode support based on browser preference
- fix CSS/JS cache busting to use build time
When extsort.Strings() cannot create temporary files (e.g. due to
apparmor restrictions or permission denied), it returns a nil sorter
with the error on errChan. The code then called Sort() on the nil
sorter, causing a panic.
Check for nil sorter and return the error instead of panicking.
Fixes#9244
The format field was read in Handle() without synchronization while
setFormat() could write it concurrently from InitLogging(). This
caused a data race detected by the race detector, failing
TestListBucketsAuthProxy in cmd/serve/s3.
Fix by protecting all access to format with the existing mutex.
This reduces the number of go routines which can get out of hand when
using large --transfers and --multi-thread-streams from potentially
--multi-thread-streams * --transfers Go routines to --max-memory /
--multi-thread-chunk-size
It serializes the memory allocator in each transfer which should be
good for performance and reduce lock contention.
Before this change server side copies would show at 0% until they were
done then show at 100%.
With support from the backend, server side copies can now be accounted
in real time. This will only work for backends which have been
modified and themselves get feedback about how copies are going.
If the transfer fails, the bytes accounted will be reversed.
Before this change when doing a sync with `--no-traverse` and
`--files-from` we could call `NewObject` a total of `--checkers` *
`--checkers` times simultaneously.
With `--checkers 128` this can exceed the 10,000 thread limit and
fails when run on a local to local transfer because `NewObject` calls
`lstat` which is a syscall which needs an OS thread of its own.
This patch uses a weighted semaphore to limit the number of
simultaneous calls to `NewObject` to `--checkers` instead which won't
blow the 10,000 thread limit and is far more sensible use of OS
resources.
Fixes#9073
These stats weren't being updated in the global stats read by rc
core/stats:
- transferQueue
- deletesSize
- serverSideCopies
- serverSideCopyBytes
- serverSideMoves
- serverSideMoveBytes
Before this change we read sleepTime before acquiring the pacer token
and uses that possibly stale value to schedule the token return. When
many goroutines enter while sleepTime is high (e.g., 10s), each
goroutine caches this 10s value. Even if successful calls rapidly
decay the pacer state to 0, the queued goroutines still schedule 10s
token returns, so the queue drains at 1 req/10s for the entire herd.
This can create multi‑minute delays even after the pacer has dropped
to 0.
After this change we refresh the sleep time after getting the token.
This problem was introduced by the desire to skip reading the pacer
token entirely when sleepTime is 0 in high performance backends (eg
s3, azure blob).
It was possible in the presence of --max-connections and recursive
calls to the pacer to deadlock it leaving all connections waiting on
either a max connection token or a pacer token.
This fixes the problem by making sure we return the pacer token on
schedule if we take it.
This also short circuits the pacer token if sleepTime is 0.
Before this change, we uses an unguarded type assertion which failed
when NewObject returned a google document instead of an Object.
This change uses the ID() method to read the id which works for all
types of file instead and returns a sensible error if one isn't found.
Fixes#8990
Nick Craig-Wood