Decrypting the config with --daemon runs --password-command twice, which
means two authentications when the command needs one, such as a hardware
key touch for `pass show`.
SetConfigPassword saves the obscured key to the temp file named by
_RCLONE_CONFIG_KEY_FILE so the daemon process can pick it up, but the
process that wrote it then read and deleted that file itself before
daemonizing. The daemon started with the variable pointing at a file that
was already gone, found no key, and ran the password command again.
Skip acquiring a password when _RCLONE_CONFIG_KEY_FILE is set, as the
PassConfigKeyForDaemonization documentation already describes, and only
consume the key file in a process that has no key of its own. The parent
then leaves the key for the daemon, and the daemon uses it.
Fixes#7341
Interactively-entered passwords were run through NFKC Unicode
normalization before being obscured, which silently rewrote characters
such as ª (U+00AA) to a. The obscured password then revealed to
something different from what the user typed confusing everyone.
Normalization is only needed for the config encryption master
password, so apply it there (in SetConfigPassword) rather than in the
shared checkPassword used for backend password options.
Previously the only way to remove an option from a remote was to set it
to an empty string, which is not the same as deleting it - a present but
empty value overrides the option's default whereas a deleted key
restores it. Editing the file by hand isn't an option for an encrypted
config either.
This adds a "config unset" command and a "config/unset" rc endpoint to
remove one or more keys from an existing remote.
When creating or updating a remote through the rc api (config/create,
config/update), parameters whose name starts with the ephemeral prefix
"config_" (for example config_template_file and config_template used to
customise the OAuth success page) were silently ignored.
updateRemote sets each supplied parameter into the config mapper, but
skips the "config_" prefixed keys so they are never written to the
config file. That guard is correct, because the mapper's setter writes
to the config file and these values are ephemeral. However backends read
these values back from the mapper (oauthutil reads config_template_file
and config_template via m.Get), so dropping them entirely meant the
values could never reach the backend and the default template was always
used.
Collect the ephemeral parameters into a separate map and add it to the
mapper as a getter overlay at PriorityNormal after the loop. The values
are now readable through m.Get without being written to the config file,
which is the same approach rclone authorize already uses to expose a
template supplied on the command line.
Fixes#9572
Co-authored-by: Hakanbaban53 <93117749+Hakanbaban53@users.noreply.github.com>
Co-authored-by: maximilize <3752128+maximilize@users.noreply.github.com>
This commit allows global config options to be set as flat parameters on all rc
commands. This makes the rc commands much more like command line parameters and
will aid understanding of how the rc is used.
It is backwards compatible with the old method using _config and _filter.
A backend flag set on the command line to a value that happened to
equal its default was silently ignored, letting the config file win
instead.
For example --sftp-user defaults to the current user, so connecting as
that same user with --sftp-user=USER had no effect and caused rclone
to use the value from the config file.
See: https://forum.rclone.org/t/sftp-user-parsing-as-cli-argument-broken/53955
Option help strings are also used to generate the website documentation,
so some contain markdown links with root-relative targets such as
[encoding section in the overview](/overview/#encoding). These render
correctly on rclone.org but are confusing in the interactive config
prompt, where the user sees the raw markdown and the link has no
reachable root.
Rewrite such links to text (https://rclone.org/path) when showing an
option's help in the interactive config. The raw help is left unchanged
so documentation generation is unaffected.
Replace AuthRequired bool with NoAuth bool on the rc.Call struct and
flip the auth check logic. Previously endpoints were unauthenticated
by default and had to opt in with AuthRequired: true, which led to
security vulnerabilities when developers forgot to set the flag.
Now all endpoints require authentication by default. Only explicitly
safe read-only endpoints are marked with NoAuth: true:
- rc/noop
- rc/error
- rc/list
- core/version
- core/stats
- core/group-list
- core/transferred
- core/du
- cache/stats
- vfs/list
- vfs/stats
- vfs/queue
- job/status
- job/list
See GHSA-25qr-6mpr-f7qx, GHSA-jfwf-28xr-xw6q
We accidentally added a non `camelCase` parameter to the rc
(`config_password`)- this fixes it (to `configPassword`) but accepts
the old name too as it has been in a release.
Before this change, you had to modify a fragile data-structure
containing all providers. This often led to things being out of order,
duplicates and conflicts whilst merging. As well as the changes for
one provider being in different places across the file.
After this change, new providers are defined in an easy to edit YAML file,
one per provider.
The config output has been tested before and after for all providers
and any changes are cosmetic only.
Before this, String() quoted every part of the config map even if it
wasn't necessary.
The new Human() method removes the quoting and adds the special case
for "true" values.
Before this change we were reading input from stdin using the terminal
in the default line mode which has a limit of 4095 characters.
The typical culprit was onedrive tokens (which are very long) giving the error
Couldn't decode response: invalid character 'e' looking for beginning of value
This change swaps over to use the github.com/peterh/liner read line
library which does not have that limitation and also enables more
sensible cursor editing.
Fixes#8688#8323#5835
Now that we have unified the config, we can make a much more
convenient rc interface which mirrors the command line exactly, rather
than using the structure of the internal Go structs.
Before this would have Output "FieldName": "ListenAddr" where it
actually needs to be set in a sub object "HTTP".
After this fix it outputs "FieldName": "HTTP.ListenAddr" to indicate
"ListenAddr" needs to be set in the object "HTTP".
This commit modernizes Go usage. This was done with:
go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./...
Then files needed to be `go fmt`ed and a few comments needed to be
restored.
The modernizations include replacing
- if/else conditional assignment by a call to the built-in min or max functions added in go1.21
- sort.Slice(x, func(i, j int) bool) { return s[i] < s[j] } by a call to slices.Sort(s), added in go1.21
- interface{} by the 'any' type added in go1.18
- append([]T(nil), s...) by slices.Clone(s) or slices.Concat(s), added in go1.21
- loop around an m[k]=v map update by a call to one of the Collect, Copy, Clone, or Insert functions from the maps package, added in go1.21
- []byte(fmt.Sprintf...) by fmt.Appendf(nil, ...), added in go1.19
- append(s[:i], s[i+1]...) by slices.Delete(s, i, i+1), added in go1.21
- a 3-clause for i := 0; i < n; i++ {} loop by for i := range n {}, added in go1.22
This commit reorganises the oauth code to use our own config struct
which has all the info for the normal oauth method and also the client
credentials flow method.
It updates all backends which use lib/oauthutil to use the new config
struct which shouldn't change any functionality.
It also adds code for dealing with the client credential flow config
which doesn't require the use of a browser and doesn't have or need a
refresh token.
Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
After the config re-organisation, the setting of stringArray config
values (eg `--exclude` set with `RCLONE_EXCLUDE`) was broken and gave
a message like this for `RCLONE_EXCLUDE=*.jpg`:
Failed to load "filter" default values: failed to initialise "filter" options:
couldn't parse config item "exclude" = "*.jpg" as []string: parsing "*.jpg" as []string failed:
invalid character '/' looking for beginning of value
This was caused by the parser trying to parse the input string as a
JSON value.
When the config was re-organised it was thought that the internal
representation of stringArray values was not important as it was never
visible externally, however this turned out not to be true.
A defined representation was chosen - a comma separated string and
this was documented and tests were introduced in this patch.
This potentially introduces a very small backwards incompatibility. In
rclone v1.67.0
RCLONE_EXCLUDE=a,b
Would be interpreted as
--exclude "a,b"
Whereas this new code will interpret it as
--exclude "a" --exclude "b"
The benefit of being able to set multiple values with an environment
variable was deemed to outweigh the very small backwards compatibility
risk.
If a value with a `,` is needed, then use CSV escaping, eg
RCLONE_EXCLUDE="a,b"
(Note this needs to have the quotes in so at the unix shell that would be
RCLONE_EXCLUDE='"a,b"'
Fixes#8063
This changes log statements from log to fs package, which is required for --use-json-log
to properly make log output in JSON format. The recently added custom linting rule,
handled by ruleguard via gocritic via golangci-lint, warns about these and suggests
the alternative. Fixing was therefore basically running "golangci-lint run --fix",
although some manual fixup of mainly imports are necessary following that.
Before this change, rclone ignored the --password-command on the
rclone config setting except when decrypting an existing config file.
This change allows for offloading the password storage/generation into
external hardware key or other protected password storage.
Fixes#7859
Before this change we used the repo with an initial uppercase `U`. However it is now canonically spelled with a lower case `u`.
This package is too old to have a go.mod but the README clearly states the desired capitalization.
In https://github.com/unknwon/goconfig/commit/4b0d4b818a95f1242556e08cbf9912d169cbade7 the
recommended capitalization was changed to lower case.
Co-authored-by: John Oxley <joxley@meta.com>
phatlcandNick Craig-Wood
dougalandNick Craig-Wood
yashanil98andGitHub
yyandGitHub
Jonas Tingebornanddougal
albertony
russcossandGitHub
n4n5andGitHub
Samantha BowenandNick Craig-Wood
JessandNick Craig-Wood
Martin HassackandNick Craig-Wood
Oleg KunitsynandGitHub
crystalstallandGitHub
