Nick Craig-Wood
7dd42c93cf
Layers which wrap backends, such as the hasher backend and the VFS cache, use fs.Fingerprint to detect whether an object has changed. This only works if the object a backend hands back from a write operation is identical to the object a fresh listing returns. If a backend keeps the source's nanosecond precision modtime in memory while the server stores milliseconds, or omits a hash the server knows, every fingerprint comparison fails and cached hashes or files are silently invalidated. The existing tests compared modtimes within f.Precision() so they could not detect these sub-precision divergences. Add a checkFingerprint helper which asserts the fast and slow fs.Fingerprint of the object returned from a write operation exactly match those of the object read back from the remote, and wire it into all the write paths which return or mutate an object: Put (all variants), PutStream, large and streamed uploads, Update, SetModTime, server side Copy and Move, and the metadata mutating Copy/Move subtests. Rework the ObjectOpenFingerprint test to use the same helper against a fully read reference object, with subtests reporting which kind of read is being checked.
Website | Documentation | Download | Contributing | Changelog | Installation | Forum
Rclone
Rclone ("rsync for cloud storage") is a command-line program to sync files and directories to and from different cloud storage providers.
Storage providers
- 1Fichier 📄
- Akamai Netstorage 📄
- Alibaba Cloud (Aliyun) Object Storage System (OSS) 📄
- Amazon S3 📄
- ArvanCloud Object Storage (AOS) 📄
- Bizfly Cloud Simple Storage 📄
- Backblaze B2 📄
- Box 📄
- Ceph 📄
- China Mobile Ecloud Elastic Object Storage (EOS) 📄
- Citrix ShareFile 📄
- Cloudflare R2 📄
- Cloudinary 📄
- Cubbit DS3 📄
- DigitalOcean Spaces 📄
- Digi Storage 📄
- Dreamhost 📄
- Drime 📄
- Dropbox 📄
- Enterprise File Fabric 📄
- Exaba 📄
- Fastly Object Storage 📄
- Fastmail Files 📄
- FileLu 📄
- Filen 📄
- Files.com 📄
- FlashBlade 📄
- FTP 📄
- GoFile 📄
- Google Cloud Storage 📄
- Google Drive 📄
- Google Photos 📄
- HDFS (Hadoop Distributed Filesystem) 📄
- Hetzner Object Storage 📄
- Hetzner Storage Box 📄
- HiDrive 📄
- Hitachi Content Platform (HCP) 📄
- HTTP 📄
- Huawei Cloud Object Storage Service(OBS) 📄
- Huawei Drive 📄
- iCloud Drive 📄
- ImageKit 📄
- Internet Archive 📄
- Internxt 📄
- Jottacloud 📄
- IBM COS S3 📄
- Impossible Cloud 📄
- Intercolo Object Storage 📄
- IONOS Cloud 📄
- Koofr 📄
- Leviia Object Storage 📄
- Liara Object Storage 📄
- Linkbox 📄
- Linode Object Storage 📄
- Magalu Object Storage 📄
- Mail.ru Cloud 📄
- Memset Memstore 📄
- MEGA 📄
- MEGA S4 Object Storage 📄
- Memory 📄
- Microsoft Azure Blob Storage 📄
- Microsoft Azure Files Storage 📄
- Microsoft OneDrive 📄
- Minio 📄
- Nextcloud 📄
- Blomp Cloud Storage 📄
- OpenDrive 📄
- OpenStack Swift 📄
- Oracle Cloud Storage 📄
- Oracle Object Storage 📄
- Outscale 📄
- OVHcloud Object Storage (Swift) 📄
- OVHcloud Object Storage (S3-compatible) 📄
- ownCloud 📄
- pCloud 📄
- Petabox 📄
- PikPak 📄
- Pixeldrain 📄
- premiumize.me 📄
- put.io 📄
- Proton Drive 📄
- QingStor 📄
- Qiniu Cloud Object Storage (Kodo) 📄
- Rabata Cloud Storage 📄
- Quatrix 📄
- Rackspace Cloud Files 📄
- RackCorp Object Storage 📄
- rsync.net 📄
- Scaleway 📄
- Seafile 📄
- Seagate Lyve Cloud 📄
- SeaweedFS 📄
- Selectel Object Storage 📄
- Servercore Object Storage 📄
- SFTP 📄
- Shade 📄
- SMB / CIFS 📄
- Spectra Logic 📄
- Storj 📄
- SugarSync 📄
- Synology C2 Object Storage 📄
- Tencent Cloud Object Storage (COS) 📄
- Uloz.to 📄
- US3 Object Storage 📄
- Wasabi 📄
- WebDAV 📄
- Yandex Disk 📄
- Zadara Object Storage 📄
- Zoho WorkDrive 📄
- Zata.ai 📄
- The local filesystem 📄
Please see the full list of all storage providers and their features
Virtual storage providers
These backends adapt or modify other storage providers
- Alias: rename existing remotes 📄
- Archive: read archive files 📄
- Cache: cache remotes (DEPRECATED) 📄
- Chunker: split large files 📄
- Combine: combine multiple remotes into a directory tree 📄
- Compress: compress files 📄
- Crypt: encrypt files 📄
- Hasher: hash files 📄
- Union: join multiple remotes to work together 📄
Features
- MD5/SHA-1 hashes checked at all times for file integrity
- Timestamps preserved on files
- Partial syncs supported on a whole file basis
- Copy mode to just copy new/changed files
- Sync (one way) mode to make a directory identical
- Bisync (two way) to keep two directories in sync bidirectionally
- Check mode to check for file hash equality
- Can sync to and from network, e.g. two different cloud accounts
- Optional large file chunking (Chunker)
- Optional transparent compression (Compress)
- Optional encryption (Crypt)
- Optional FUSE mount (rclone mount)
- Multi-threaded downloads to local disk
- Can serve local or remote files over HTTP/WebDAV/FTP/SFTP/DLNA
Installation & documentation
Please see the rclone website for:
Downloads
License
This is free software under the terms of the MIT license (check the COPYING file included in this package).