CEPH uses a special bucket form `tenant:bucket` for multitentant
access using S3 as documented here:
https://docs.ceph.com/en/reef/radosgw/multitenancy/#s3
However when doing multipart uploads, in the reply from
`CreateMultipart` the `tenant:` was missing from the `Bucket` response
rclone was using to build the `UploadPart` request. This caused a 404
failure return. This may be a CEPH bug, but it is easy to work around.
This changes the code to use the `Bucket` and `Key` that we used in
`CreateMultipart` in `UploadPart` rather than the one returned from
`CreateMultipart` which fixes the problem.
See: https://forum.rclone.org/t/rclone-zcat-does-not-work-with-a-multitenant-ceph-backend/48618
Before this change attempting to download a file with
`Content-Encoding: gzip` from Cloudflare R2 gave this error
corrupted on transfer: sizes differ src 0 vs dst 999
This was caused by the SDK v2 overriding our attempt to set
`Accept-Encoding: gzip`.
This fixes the problem by disabling the middleware that does that
overriding.
Before this change, if writing to a local backend with --metadata and
--links, if the incoming metadata contained mode or ownership
information then rclone would apply the mode/ownership to the
destination of the link not the link itself.
This fixes the problem by using the link safe sycall variants
lchown/fchmodat when --links and --metadata is in use. Note that Linux
does not support setting permissions on symlinks, so rclone emits a
debug message in this case.
This also fixes setting times on symlinks on Windows which wasn't
implemented for atime, mtime and was incorrectly setting the target of
the symlink for btime.
See: https://github.com/rclone/rclone/security/advisories/GHSA-hrxh-9w67-g4cv
Before this change, if writing to a local backend with --metadata and
--links, if the incoming metadata contained mode or ownership
information then rclone would apply the mode/ownership to the
destination of the link not the link itself.
This fixes the problem by using the link safe sycall variants
lchown/fchmodat when --links and --metadata is in use. Note that Linux
does not support setting permissions on symlinks, so rclone emits a
debug message in this case.
This also fixes setting times on symlinks on Windows which wasn't
implemented for atime, mtime and was incorrectly setting the target of
the symlink for btime.
See: https://github.com/rclone/rclone/security/advisories/GHSA-hrxh-9w67-g4cv
We changed the precision of the onedrive personal backend in
c053429b9c from 1mS to 1S.
However the tests did not get updated. This changes the time tests to
use `fstest.AssertTimeEqualWithPrecision` which compares with
precision so hopefully won't break again.
Before this change, upgrading to v1.13.7 caused a deadlock in the tests.
This was caused by additional locking in the sftp package exposing a
bad choice by the rclone code.
See https://github.com/pkg/sftp/issues/603 and thanks to @puellanivis
for the fix suggestion.
The Mailru backend integration tests have been failing due to new rate
limits on the backend.
This patch
- Removes Mailru from the chunker tests
- Adds the flag so we only run one Mailru test at once
Before this testing any backend which implemented the OpenChunkWriter
gave this error:
ERROR : writer-at-subdir/writer-at-file: Don't know how to set key "chunkSize" on upload
This was due to the ChunkOption incorrectly rendering into HTTP
headers which weren't understood by the backend.
This reduces the precision advertised by the backend from 1ms to 1s
for OneDrive personal accounts.
The precision was set to 1ms as part of:
1473de3f04 onedrive: add metadata support
which was released in v1.66.0.
However it appears not all OneDrive personal accounts support 1ms time
precision and that Microsoft may be migrating accounts away from this
to backends which only support 1s precision.
Fixes#8101
This will ensure no Content-Md5 headers are sent and ensure ETags are not
interpreted as MD5 sums. X-Amz-Meta-Md5chksum will be set on all objects
whether single or multipart uploaded.
This also sets "no_check_bucket = true".
This is enough to make the integration tests pass, but there are some
limitations as noted in the docs.
See: https://forum.rclone.org/t/support-s3-directory-bucket/47653/
When uploading chunked files to nextcloud, it gives a 423 error while
it is merging files.
This waits for an exponentially increasing amount of time for it to
clear.
If after we have received a 423 error we receive a 404 error then we
assume all is good as this is what appears to happen in practice.
Fixes#7109
The SDKv2 conversion introduced a regression to do with setting
credentials with env_auth=true. The rclone documentation explicitly
states that env_auth only applies if secret_access_key and
access_key_id are blank and users had been relying on that.
However after the SDKv2 conversion we were ignoring static credentials
if env_auth=true.
This fixes the problem by ignoring env_auth=true if secret_access_key
and access_key_id are both provided. This brings rclone back into line
with the documentation and users expectations.
Fixes#8067
After the config re-organisation, the setting of stringArray config
values (eg `--exclude` set with `RCLONE_EXCLUDE`) was broken and gave
a message like this for `RCLONE_EXCLUDE=*.jpg`:
Failed to load "filter" default values: failed to initialise "filter" options:
couldn't parse config item "exclude" = "*.jpg" as []string: parsing "*.jpg" as []string failed:
invalid character '/' looking for beginning of value
This was caused by the parser trying to parse the input string as a
JSON value.
When the config was re-organised it was thought that the internal
representation of stringArray values was not important as it was never
visible externally, however this turned out not to be true.
A defined representation was chosen - a comma separated string and
this was documented and tests were introduced in this patch.
This potentially introduces a very small backwards incompatibility. In
rclone v1.67.0
RCLONE_EXCLUDE=a,b
Would be interpreted as
--exclude "a,b"
Whereas this new code will interpret it as
--exclude "a" --exclude "b"
The benefit of being able to set multiple values with an environment
variable was deemed to outweigh the very small backwards compatibility
risk.
If a value with a `,` is needed, then use CSV escaping, eg
RCLONE_EXCLUDE="a,b"
(Note this needs to have the quotes in so at the unix shell that would be
RCLONE_EXCLUDE='"a,b"'
Fixes#8063
Before this fix, we initialised the options blocks in a random order.
This meant that there was a 50/50 chance whether --dump filters would
show the filters or not as it was depending on the "main" block having
being read first to set the Dump flags.
This initialises the options blocks in a defined order which is
alphabetically but with main first which fixes the problem.
The upload routine no longer returns a url to download the object.
This fixes the problem by fetching it if necessary when we attempt to
Open the object.
For some reason the parent ID got out of date in the Object (exact
reason not known - but the fact that this was OK before suggests a
change in the provider).
However we know the parent ID as it is in the directory cache, so use
that instead.
This gives the error
> Update second step failed: Linkbox error 500: The file name needs to include a suffix, such as xxx.mp4
As linkbox can't have files starting with "." and we are trying to save a file called ".ignore".
The server side move had a combination of bugs
- Fichier changed the API disallowing a move to the same name
- Rclone was using the wrong object for some operations
The original problem was introduced here
bcdfad3c83 build: update logging statements to make json log work #6038
And this was fixed non-optimally here
f1a84d171e build: fix build after update
Before this change, rclone ignored the --password-command on the
rclone config setting except when decrypting an existing config file.
This change allows for offloading the password storage/generation into
external hardware key or other protected password storage.
Fixes#7859
Before this change all exports were exported as root and the --uid and
--gid flags of the VFS were ignored.
This fixes the issue by exporting the UID and GID correctly which
default to the current user and group unless set explicitly.
This problem was caused by the defaults not being set for the options
after the conversion to the new config system in
28ba4b832d serve nfs: convert options to new style
This makes the nfs serve options globally available so nfsmount can
use them directly.
Fixes#8029
Nick Craig-Wood
