Lockfiles with invalid JSON content caused bisync to fail permanently
because lockFileIsExpired() logged the decode error but still fell
through to the "valid lock file" path with zero-value TimeExpires.
Now when a JSON decode error is detected:
- If --max-lock is set (< basicallyforever): treat garbled lockfile as
expired, mark listings failed, and proceed (safe assumption: the
previous bisync run crashed and left garbage).
- If --max-lock is not set (default): log a clear error telling the
user the lockfile needs manual inspection, and return false.
AWS S3 requires Content-MD5 for PutObject with Object Lock parameters.
Since rclone passes a non-seekable io.Reader, the SDK cannot compute
checksums automatically. Buffer the body and compute MD5 manually for
singlepart PutObject and presigned request uploads when Object Lock
parameters are set. Multipart uploads are unaffected as Object Lock
headers go on CreateMultipartUpload which has no body.
Add object_lock_supported provider quirk (default true) to allow
skipping Object Lock integration tests on providers with incomplete
S3 API support. Set to false for GCS which uses non-standard
x-goog-bypass-governance-retention header and doesn't implement
PutObjectLegalHold/GetObjectLegalHold.
Add Multipart and Presigned subtests to Object Lock integration tests
to cover all three upload paths.
Fixes#9199
Fixes#5063 by documenting that S3 object keys containing
consecutive forward slashes (//) are not supported by rclone.
The issue occurs because rclone normalizes paths like "a//b" to "a/b",
causing "object not found" errors when trying to access the original
object. This documentation addition explicitly warns users about this
limitation and provides workarounds.
Changes:
- Added new subsection "Important note about double slashes (//)"
under "Restricted filename characters" in S3 documentation
- Explains the normalization behavior and its consequences
- Provides clear examples and workarounds
AI Model/Tool Attribution:
- Implemented using opencode AI assistant
- Issue analysis and documentation update performed by AI tools
Resolves: #5063
Add support for S3 Object Lock with the following new options:
- --s3-object-lock-mode: set retention mode (GOVERNANCE/COMPLIANCE/copy)
- --s3-object-lock-retain-until-date: set retention date (RFC3339/duration/copy)
- --s3-object-lock-legal-hold-status: set legal hold (ON/OFF/copy)
- --s3-bypass-governance-retention: bypass GOVERNANCE lock on delete
- --s3-bucket-object-lock-enabled: enable Object Lock on bucket creation
- --s3-object-lock-set-after-upload: apply lock via separate API calls
The special value "copy" preserves the source object's setting when used
with --metadata flag, enabling scenarios like cloning objects from
COMPLIANCE to GOVERNANCE mode while preserving the original retention date.
Includes integration tests that create a temporary Object Lock bucket covering:
- Retention Mode and Date
- Legal Hold
- Apply settings after upload
- Override protections using bypass-governance flag
The tests are gracefully skipped on providers that do not support Object Lock.
Fixes#4683Closes#7894#7893#8866
Nick Craig-Wood
lifandGitHub
kapitainskyandGitHub
albertony
JasonandNick Craig-Wood
ChrisandNick Craig-Wood
Marco FerrettiandGitHub
nielash
razorlovesandGitHub
Adam KasztennyandNick Craig-Wood
Shlomi AvihouandNick Craig-Wood