Version v1.33
This commit is contained in:
@@ -1,7 +1,7 @@
|
||||
.\"t
|
||||
.\" Automatically generated by Pandoc 1.16.0.2
|
||||
.\"
|
||||
.TH "rclone" "1" "Aug 04, 2016" "User Manual" ""
|
||||
.TH "rclone" "1" "Aug 24, 2016" "User Manual" ""
|
||||
.hy
|
||||
.SH Rclone
|
||||
.PP
|
||||
@@ -645,7 +645,7 @@ rclone\ dedupe\ [mode]\ remote:path
|
||||
.IP
|
||||
.nf
|
||||
\f[C]
|
||||
\ \ \ \ \ \ \-\-dedupe\-mode\ value\ \ \ Dedupe\ mode\ interactive|skip|first|newest|oldest|rename.\ (default\ "interactive")
|
||||
\ \ \ \ \ \ \-\-dedupe\-mode\ string\ \ \ Dedupe\ mode\ interactive|skip|first|newest|oldest|rename.
|
||||
\f[]
|
||||
.fi
|
||||
.SS rclone authorize
|
||||
@@ -662,6 +662,42 @@ browser \- use as instructed by rclone config.
|
||||
rclone\ authorize
|
||||
\f[]
|
||||
.fi
|
||||
.SS rclone cat
|
||||
.PP
|
||||
Concatenates any files and sends them to stdout.
|
||||
.SS Synopsis
|
||||
.PP
|
||||
rclone cat sends any files to standard output.
|
||||
.PP
|
||||
You can use it like this to output a single file
|
||||
.IP
|
||||
.nf
|
||||
\f[C]
|
||||
rclone\ cat\ remote:path/to/file
|
||||
\f[]
|
||||
.fi
|
||||
.PP
|
||||
Or like this to output any file in dir or subdirectories.
|
||||
.IP
|
||||
.nf
|
||||
\f[C]
|
||||
rclone\ cat\ remote:path/to/dir
|
||||
\f[]
|
||||
.fi
|
||||
.PP
|
||||
Or like this to output any .txt files in dir or subdirectories.
|
||||
.IP
|
||||
.nf
|
||||
\f[C]
|
||||
rclone\ \-\-include\ "*.txt"\ cat\ remote:path/to/dir
|
||||
\f[]
|
||||
.fi
|
||||
.IP
|
||||
.nf
|
||||
\f[C]
|
||||
rclone\ cat\ remote:path
|
||||
\f[]
|
||||
.fi
|
||||
.SS rclone genautocomplete
|
||||
.PP
|
||||
Output bash completion script for rclone.
|
||||
@@ -709,6 +745,99 @@ website.
|
||||
rclone\ gendocs\ output_directory
|
||||
\f[]
|
||||
.fi
|
||||
.SS rclone mount
|
||||
.PP
|
||||
Mount the remote as a mountpoint.
|
||||
\f[B]EXPERIMENTAL\f[]
|
||||
.SS Synopsis
|
||||
.PP
|
||||
rclone mount allows Linux, FreeBSD and macOS to mount any of
|
||||
Rclone\[aq]s cloud storage systems as a file system with FUSE.
|
||||
.PP
|
||||
This is \f[B]EXPERIMENTAL\f[] \- use with care.
|
||||
.PP
|
||||
First set up your remote using \f[C]rclone\ config\f[].
|
||||
Check it works with \f[C]rclone\ ls\f[] etc.
|
||||
.PP
|
||||
Start the mount like this
|
||||
.IP
|
||||
.nf
|
||||
\f[C]
|
||||
rclone\ mount\ remote:path/to/files\ /path/to/local/mount\ &
|
||||
\f[]
|
||||
.fi
|
||||
.PP
|
||||
Stop the mount with
|
||||
.IP
|
||||
.nf
|
||||
\f[C]
|
||||
fusermount\ \-u\ /path/to/local/mount
|
||||
\f[]
|
||||
.fi
|
||||
.PP
|
||||
Or with OS X
|
||||
.IP
|
||||
.nf
|
||||
\f[C]
|
||||
umount\ \-u\ /path/to/local/mount
|
||||
\f[]
|
||||
.fi
|
||||
.SS Limitations
|
||||
.PP
|
||||
This can only read files seqentially, or write files sequentially.
|
||||
It can\[aq]t read and write or seek in files.
|
||||
.PP
|
||||
rclonefs inherits rclone\[aq]s directory handling.
|
||||
In rclone\[aq]s world directories don\[aq]t really exist.
|
||||
This means that empty directories will have a tendency to disappear once
|
||||
they fall out of the directory cache.
|
||||
.PP
|
||||
The bucket based FSes (eg swift, s3, google compute storage, b2)
|
||||
won\[aq]t work from the root \- you will need to specify a bucket, or a
|
||||
path within the bucket.
|
||||
So \f[C]swift:\f[] won\[aq]t work whereas \f[C]swift:bucket\f[] will as
|
||||
will \f[C]swift:bucket/path\f[].
|
||||
.PP
|
||||
Only supported on Linux, FreeBSD and OS X at the moment.
|
||||
.SS rclone mount vs rclone sync/copy
|
||||
.PP
|
||||
File systems expect things to be 100% reliable, whereas cloud storage
|
||||
systems are a long way from 100% reliable.
|
||||
The rclone sync/copy commands cope with this with lots of retries.
|
||||
However rclone mount can\[aq]t use retries in the same way without
|
||||
making local copies of the uploads.
|
||||
This might happen in the future, but for the moment rclone mount
|
||||
won\[aq]t do that, so will be less reliable than the rclone command.
|
||||
.SS Bugs
|
||||
.IP \[bu] 2
|
||||
All the remotes should work for read, but some may not for write
|
||||
.RS 2
|
||||
.IP \[bu] 2
|
||||
those which need to know the size in advance won\[aq]t \- eg B2
|
||||
.IP \[bu] 2
|
||||
maybe should pass in size as \-1 to mean work it out
|
||||
.RE
|
||||
.SS TODO
|
||||
.IP \[bu] 2
|
||||
Check hashes on upload/download
|
||||
.IP \[bu] 2
|
||||
Preserve timestamps
|
||||
.IP \[bu] 2
|
||||
Move directories
|
||||
.IP
|
||||
.nf
|
||||
\f[C]
|
||||
rclone\ mount\ remote:path\ /path/to/mountpoint
|
||||
\f[]
|
||||
.fi
|
||||
.SS Options
|
||||
.IP
|
||||
.nf
|
||||
\f[C]
|
||||
\ \ \ \ \ \ \-\-debug\-fuse\ \ \ Debug\ the\ FUSE\ internals\ \-\ needs\ \-v.
|
||||
\ \ \ \ \ \ \-\-no\-modtime\ \ \ Don\[aq]t\ read\ the\ modification\ time\ (can\ speed\ things\ up).
|
||||
\f[]
|
||||
.fi
|
||||
.SS Copying single files
|
||||
.PP
|
||||
rclone normally syncs or copies directories.
|
||||
@@ -2415,6 +2544,47 @@ This causes rclone to be limited to transferring about 2 files per
|
||||
second only.
|
||||
Individual files may be transferred much faster at 100s of MBytes/s but
|
||||
lots of small files can take a long time.
|
||||
.SS Making your own client_id
|
||||
.PP
|
||||
When you use rclone with Google drive in its default configuration you
|
||||
are using rclone\[aq]s client_id.
|
||||
This is shared between all the rclone users.
|
||||
There is a global rate limit on the number of queries per second that
|
||||
each client_id can do set by Google.
|
||||
rclone already has a high quota and I will continue to make sure it is
|
||||
high enough by contacting Google.
|
||||
.PP
|
||||
However you might find you get better performance making your own
|
||||
client_id if you are a heavy user.
|
||||
Or you may not depending on exactly how Google have been raising
|
||||
rclone\[aq]s rate limit.
|
||||
.PP
|
||||
Here is how to create your own Google Drive client ID for rclone:
|
||||
.IP "1." 3
|
||||
Log into the Google API Console (https://console.developers.google.com/)
|
||||
with your Google account.
|
||||
It doesn\[aq]t matter what Google account you use.
|
||||
(It need not be the same account as the Google Drive you want to access)
|
||||
.IP "2." 3
|
||||
Select a project or create a new project.
|
||||
.IP "3." 3
|
||||
Under Overview, Google APIs, Google Apps APIs, click "Drive API", then
|
||||
"Enable".
|
||||
.IP "4." 3
|
||||
Click "Credentials" in the left\-side panel (not "Go to credentials",
|
||||
which opens the wizard), then "Create credentials", then "OAuth client
|
||||
ID".
|
||||
It will prompt you to set the OAuth consent screen product name, if you
|
||||
haven\[aq]t set one already.
|
||||
.IP "5." 3
|
||||
Choose an application type of "other", and click "Create".
|
||||
(the default name is fine)
|
||||
.IP "6." 3
|
||||
It will show you a client ID and client secret.
|
||||
Use these values in rclone config to add a new remote or edit an
|
||||
existing remote.
|
||||
.PP
|
||||
(Thanks to \@balazer on github for these instructions.)
|
||||
.SS Amazon S3
|
||||
.PP
|
||||
Paths are specified as \f[C]remote:bucket\f[] (or \f[C]remote:\f[] for
|
||||
@@ -2538,6 +2708,25 @@ Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
||||
\ 9\ /\ South\ America\ (Sao\ Paulo)\ Region.
|
||||
\ \ \ \\\ "sa\-east\-1"
|
||||
location_constraint>\ 1
|
||||
Canned\ ACL\ used\ when\ creating\ buckets\ and/or\ storing\ objects\ in\ S3.
|
||||
For\ more\ info\ visit\ http://docs.aws.amazon.com/AmazonS3/latest/dev/acl\-overview.html#canned\-acl
|
||||
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
||||
\ 1\ /\ Owner\ gets\ FULL_CONTROL.\ No\ one\ else\ has\ access\ rights\ (default).
|
||||
\ \ \ \\\ "private"
|
||||
\ 2\ /\ Owner\ gets\ FULL_CONTROL.\ The\ AllUsers\ group\ gets\ READ\ access.
|
||||
\ \ \ \\\ "public\-read"
|
||||
\ \ \ /\ Owner\ gets\ FULL_CONTROL.\ The\ AllUsers\ group\ gets\ READ\ and\ WRITE\ access.
|
||||
\ 3\ |\ Granting\ this\ on\ a\ bucket\ is\ generally\ not\ recommended.
|
||||
\ \ \ \\\ "public\-read\-write"
|
||||
\ 4\ /\ Owner\ gets\ FULL_CONTROL.\ The\ AuthenticatedUsers\ group\ gets\ READ\ access.
|
||||
\ \ \ \\\ "authenticated\-read"
|
||||
\ \ \ /\ Object\ owner\ gets\ FULL_CONTROL.\ Bucket\ owner\ gets\ READ\ access.
|
||||
\ 5\ |\ If\ you\ specify\ this\ canned\ ACL\ when\ creating\ a\ bucket,\ Amazon\ S3\ ignores\ it.
|
||||
\ \ \ \\\ "bucket\-owner\-read"
|
||||
\ \ \ /\ Both\ the\ object\ owner\ and\ the\ bucket\ owner\ get\ FULL_CONTROL\ over\ the\ object.
|
||||
\ 6\ |\ If\ you\ specify\ this\ canned\ ACL\ when\ creating\ a\ bucket,\ Amazon\ S3\ ignores\ it.
|
||||
\ \ \ \\\ "bucket\-owner\-full\-control"
|
||||
acl>\ private
|
||||
The\ server\-side\ encryption\ algorithm\ used\ when\ storing\ this\ object\ in\ S3.
|
||||
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
||||
\ 1\ /\ None
|
||||
@@ -3451,6 +3640,14 @@ The default for this is 9GB which shouldn\[aq]t need to be changed.
|
||||
To download files above this threshold, rclone requests a
|
||||
\f[C]tempLink\f[] which downloads the file through a temporary URL
|
||||
directly from the underlying S3 storage.
|
||||
.SS \-\-acd\-upload\-wait\-time=TIME
|
||||
.PP
|
||||
Sometimes Amazon Drive gives an error when a file has been fully
|
||||
uploaded but the file appears anyway after a little while.
|
||||
This controls the time rclone waits \- 2 minutes by default.
|
||||
You might want to increase the time if you are having problems with very
|
||||
big files.
|
||||
Upload with the \f[C]\-v\f[] flag for more info.
|
||||
.SS Limitations
|
||||
.PP
|
||||
Note that Amazon Drive is case insensitive so you can\[aq]t have a file
|
||||
@@ -4186,6 +4383,320 @@ format.
|
||||
.SS MD5 checksums
|
||||
.PP
|
||||
MD5 checksums are natively supported by Yandex Disk.
|
||||
.SS Crypt
|
||||
.PP
|
||||
The \f[C]crypt\f[] remote encrypts and decrypts another remote.
|
||||
.PP
|
||||
To use it first set up the underlying remote following the config
|
||||
instructions for that remote.
|
||||
You can also use a local pathname instead of a remote which will encrypt
|
||||
and decrypt from that directory which might be useful for encrypting
|
||||
onto a USB stick for example.
|
||||
.PP
|
||||
First check your chosen remote is working \- we\[aq]ll call it
|
||||
\f[C]remote:path\f[] in these docs.
|
||||
Note that anything inside \f[C]remote:path\f[] will be encrypted and
|
||||
anything outside won\[aq]t.
|
||||
This means that if you are using a bucket based remote (eg S3, B2,
|
||||
swift) then you should probably put the bucket in the remote
|
||||
\f[C]s3:bucket\f[].
|
||||
If you just use \f[C]s3:\f[] then rclone will make encrypted bucket
|
||||
names too (if using file name encryption) which may or may not be what
|
||||
you want.
|
||||
.PP
|
||||
Now configure \f[C]crypt\f[] using \f[C]rclone\ config\f[].
|
||||
We will call this one \f[C]secret\f[] to differentiate it from the
|
||||
\f[C]remote\f[].
|
||||
.IP
|
||||
.nf
|
||||
\f[C]
|
||||
No\ remotes\ found\ \-\ make\ a\ new\ one
|
||||
n)\ New\ remote
|
||||
s)\ Set\ configuration\ password
|
||||
q)\ Quit\ config
|
||||
n/s/q>\ n\ \ \
|
||||
name>\ secret
|
||||
Type\ of\ storage\ to\ configure.
|
||||
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
||||
\ 1\ /\ Amazon\ Drive
|
||||
\ \ \ \\\ "amazon\ cloud\ drive"
|
||||
\ 2\ /\ Amazon\ S3\ (also\ Dreamhost,\ Ceph,\ Minio)
|
||||
\ \ \ \\\ "s3"
|
||||
\ 3\ /\ Backblaze\ B2
|
||||
\ \ \ \\\ "b2"
|
||||
\ 4\ /\ Dropbox
|
||||
\ \ \ \\\ "dropbox"
|
||||
\ 5\ /\ Encrypt/Decrypt\ a\ remote
|
||||
\ \ \ \\\ "crypt"
|
||||
\ 6\ /\ Google\ Cloud\ Storage\ (this\ is\ not\ Google\ Drive)
|
||||
\ \ \ \\\ "google\ cloud\ storage"
|
||||
\ 7\ /\ Google\ Drive
|
||||
\ \ \ \\\ "drive"
|
||||
\ 8\ /\ Hubic
|
||||
\ \ \ \\\ "hubic"
|
||||
\ 9\ /\ Local\ Disk
|
||||
\ \ \ \\\ "local"
|
||||
10\ /\ Microsoft\ OneDrive
|
||||
\ \ \ \\\ "onedrive"
|
||||
11\ /\ Openstack\ Swift\ (Rackspace\ Cloud\ Files,\ Memset\ Memstore,\ OVH)
|
||||
\ \ \ \\\ "swift"
|
||||
12\ /\ Yandex\ Disk
|
||||
\ \ \ \\\ "yandex"
|
||||
Storage>\ 5
|
||||
Remote\ to\ encrypt/decrypt.
|
||||
remote>\ remote:path
|
||||
How\ to\ encrypt\ the\ filenames.
|
||||
Choose\ a\ number\ from\ below,\ or\ type\ in\ your\ own\ value
|
||||
\ 1\ /\ Don\[aq]t\ encrypt\ the\ file\ names.\ \ Adds\ a\ ".bin"\ extension\ only.
|
||||
\ \ \ \\\ "off"
|
||||
\ 2\ /\ Encrypt\ the\ filenames\ see\ the\ docs\ for\ the\ details.
|
||||
\ \ \ \\\ "standard"
|
||||
filename_encryption>\ 2
|
||||
Password\ or\ pass\ phrase\ for\ encryption.
|
||||
y)\ Yes\ type\ in\ my\ own\ password
|
||||
g)\ Generate\ random\ password
|
||||
y/g>\ y
|
||||
Enter\ the\ password:
|
||||
password:
|
||||
Confirm\ the\ password:
|
||||
password:
|
||||
Password\ or\ pass\ phrase\ for\ salt.\ Optional\ but\ recommended.
|
||||
Should\ be\ different\ to\ the\ previous\ password.
|
||||
y)\ Yes\ type\ in\ my\ own\ password
|
||||
g)\ Generate\ random\ password
|
||||
n)\ No\ leave\ this\ optional\ password\ blank
|
||||
y/g/n>\ g
|
||||
Password\ strength\ in\ bits.
|
||||
64\ is\ just\ about\ memorable
|
||||
128\ is\ secure
|
||||
1024\ is\ the\ maximum
|
||||
Bits>\ 128
|
||||
Your\ password\ is:\ JAsJvRcgR\-_veXNfy_sGmQ
|
||||
Use\ this\ password?
|
||||
y)\ Yes
|
||||
n)\ No
|
||||
y/n>\ y
|
||||
Remote\ config
|
||||
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
||||
[secret]
|
||||
remote\ =\ remote:path
|
||||
filename_encryption\ =\ standard
|
||||
password\ =\ CfDxopZIXFG0Oo\-ac7dPLWWOHkNJbw
|
||||
password2\ =\ HYUpfuzHJL8qnX9fOaIYijq0xnVLwyVzp3y4SF3TwYqAU6HLysk
|
||||
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
|
||||
y)\ Yes\ this\ is\ OK
|
||||
e)\ Edit\ this\ remote
|
||||
d)\ Delete\ this\ remote
|
||||
y/e/d>\ y
|
||||
\f[]
|
||||
.fi
|
||||
.PP
|
||||
\f[B]Important\f[] The password is stored in the config file is lightly
|
||||
obscured so it isn\[aq]t immediately obvious what it is.
|
||||
It is in no way secure unless you use config file encryption.
|
||||
.PP
|
||||
A long passphrase is recommended, or you can use a random one.
|
||||
Note that if you reconfigure rclone with the same passwords/passphrases
|
||||
elsewhere it will be compatible \- all the secrets used are derived from
|
||||
those two passwords/passphrases.
|
||||
.PP
|
||||
Note that rclone does not encrypt * file length \- this can be calcuated
|
||||
within 16 bytes * modification time \- used for syncing
|
||||
.SS Example
|
||||
.PP
|
||||
To test I made a little directory of files using "standard" file name
|
||||
encryption.
|
||||
.IP
|
||||
.nf
|
||||
\f[C]
|
||||
plaintext/
|
||||
├──\ file0.txt
|
||||
├──\ file1.txt
|
||||
└──\ subdir
|
||||
\ \ \ \ ├──\ file2.txt
|
||||
\ \ \ \ ├──\ file3.txt
|
||||
\ \ \ \ └──\ subsubdir
|
||||
\ \ \ \ \ \ \ \ └──\ file4.txt
|
||||
\f[]
|
||||
.fi
|
||||
.PP
|
||||
Copy these to the remote and list them back
|
||||
.IP
|
||||
.nf
|
||||
\f[C]
|
||||
$\ rclone\ \-q\ copy\ plaintext\ secret:
|
||||
$\ rclone\ \-q\ ls\ secret:
|
||||
\ \ \ \ \ \ \ \ 7\ file1.txt
|
||||
\ \ \ \ \ \ \ \ 6\ file0.txt
|
||||
\ \ \ \ \ \ \ \ 8\ subdir/file2.txt
|
||||
\ \ \ \ \ \ \ 10\ subdir/subsubdir/file4.txt
|
||||
\ \ \ \ \ \ \ \ 9\ subdir/file3.txt
|
||||
\f[]
|
||||
.fi
|
||||
.PP
|
||||
Now see what that looked like when encrypted
|
||||
.IP
|
||||
.nf
|
||||
\f[C]
|
||||
$\ rclone\ \-q\ ls\ remote:path
|
||||
\ \ \ \ \ \ \ 55\ hagjclgavj2mbiqm6u6cnjjqcg
|
||||
\ \ \ \ \ \ \ 54\ v05749mltvv1tf4onltun46gls
|
||||
\ \ \ \ \ \ \ 57\ 86vhrsv86mpbtd3a0akjuqslj8/dlj7fkq4kdq72emafg7a7s41uo
|
||||
\ \ \ \ \ \ \ 58\ 86vhrsv86mpbtd3a0akjuqslj8/7uu829995du6o42n32otfhjqp4/b9pausrfansjth5ob3jkdqd4lc
|
||||
\ \ \ \ \ \ \ 56\ 86vhrsv86mpbtd3a0akjuqslj8/8njh1sk437gttmep3p70g81aps
|
||||
\f[]
|
||||
.fi
|
||||
.PP
|
||||
Note that this retains the directory structure which means you can do
|
||||
this
|
||||
.IP
|
||||
.nf
|
||||
\f[C]
|
||||
$\ rclone\ \-q\ ls\ secret:subdir
|
||||
\ \ \ \ \ \ \ \ 8\ file2.txt
|
||||
\ \ \ \ \ \ \ \ 9\ file3.txt
|
||||
\ \ \ \ \ \ \ 10\ subsubdir/file4.txt
|
||||
\f[]
|
||||
.fi
|
||||
.PP
|
||||
If don\[aq]t use file name encryption then the remote will look like
|
||||
this \- note the \f[C]\&.bin\f[] extensions added to prevent the cloud
|
||||
provider attempting to interpret the data.
|
||||
.IP
|
||||
.nf
|
||||
\f[C]
|
||||
$\ rclone\ \-q\ ls\ remote:path
|
||||
\ \ \ \ \ \ \ 54\ file0.txt.bin
|
||||
\ \ \ \ \ \ \ 57\ subdir/file3.txt.bin
|
||||
\ \ \ \ \ \ \ 56\ subdir/file2.txt.bin
|
||||
\ \ \ \ \ \ \ 58\ subdir/subsubdir/file4.txt.bin
|
||||
\ \ \ \ \ \ \ 55\ file1.txt.bin
|
||||
\f[]
|
||||
.fi
|
||||
.SS File name encryption modes
|
||||
.PP
|
||||
Here are some of the features of the file name encryption modes
|
||||
.PP
|
||||
Off * doesn\[aq]t hide file names or directory structure * allows for
|
||||
longer file names (~246 characters) * can use sub paths and copy single
|
||||
files
|
||||
.PP
|
||||
Standard * file names encrypted * file names can\[aq]t be as long (~156
|
||||
characters) * can use sub paths and copy single files * directory
|
||||
structure visibile * identical files names will have identical uploaded
|
||||
names * can use shortcuts to shorten the directory recursion
|
||||
.PP
|
||||
Cloud storage systems have various limits on file name length and total
|
||||
path length which you are more likely to hit using "Standard" file name
|
||||
encryption.
|
||||
If you keep your file names to below 156 characters in length then you
|
||||
should be OK on all providers.
|
||||
.PP
|
||||
There may be an even more secure file name encryption mode in the future
|
||||
which will address the long file name problem.
|
||||
.SS File formats
|
||||
.SS File encryption
|
||||
.PP
|
||||
Files are encrypted 1:1 source file to destination object.
|
||||
The file has a header and is divided into chunks.
|
||||
.SS Header
|
||||
.IP \[bu] 2
|
||||
8 bytes magic string \f[C]RCLONE\\x00\\x00\f[]
|
||||
.IP \[bu] 2
|
||||
24 bytes Nonce (IV)
|
||||
.PP
|
||||
The initial nonce is generated from the operating systems crypto strong
|
||||
random number genrator.
|
||||
The nonce is incremented for each chunk read making sure each nonce is
|
||||
unique for each block written.
|
||||
The chance of a nonce being re\-used is miniscule.
|
||||
If you wrote an exabyte of data (10¹⁸ bytes) you would have a
|
||||
probability of approximately 2×10⁻³² of re\-using a nonce.
|
||||
.SS Chunk
|
||||
.PP
|
||||
Each chunk will contain 64kB of data, except for the last one which may
|
||||
have less data.
|
||||
The data chunk is in standard NACL secretbox format.
|
||||
Secretbox uses XSalsa20 and Poly1305 to encrypt and authenticate
|
||||
messages.
|
||||
.PP
|
||||
Each chunk contains:
|
||||
.IP \[bu] 2
|
||||
16 Bytes of Poly1305 authenticator
|
||||
.IP \[bu] 2
|
||||
1 \- 65536 bytes XSalsa20 encrypted data
|
||||
.PP
|
||||
64k chunk size was chosen as the best performing chunk size (the
|
||||
authenticator takes too much time below this and the performance drops
|
||||
off due to cache effects above this).
|
||||
Note that these chunks are buffered in memory so they can\[aq]t be too
|
||||
big.
|
||||
.PP
|
||||
This uses a 32 byte (256 bit key) key derived from the user password.
|
||||
.SS Examples
|
||||
.PP
|
||||
1 byte file will encrypt to
|
||||
.IP \[bu] 2
|
||||
32 bytes header
|
||||
.IP \[bu] 2
|
||||
17 bytes data chunk
|
||||
.PP
|
||||
49 bytes total
|
||||
.PP
|
||||
1MB (1048576 bytes) file will encrypt to
|
||||
.IP \[bu] 2
|
||||
32 bytes header
|
||||
.IP \[bu] 2
|
||||
16 chunks of 65568 bytes
|
||||
.PP
|
||||
1049120 bytes total (a 0.05% overhead).
|
||||
This is the overhead for big files.
|
||||
.SS Name encryption
|
||||
.PP
|
||||
File names are encrypted segment by segment \- the path is broken up
|
||||
into \f[C]/\f[] separated strings and these are encrypted individually.
|
||||
.PP
|
||||
File segments are padded using using PKCS#7 to a multiple of 16 bytes
|
||||
before encryption.
|
||||
.PP
|
||||
They are then encrypted with EME using AES with 256 bit key.
|
||||
EME (ECB\-Mix\-ECB) is a wide\-block encryption mode presented in the
|
||||
2003 paper "A Parallelizable Enciphering Mode" by Halevi and Rogaway.
|
||||
.PP
|
||||
This makes for determinstic encryption which is what we want \- the same
|
||||
filename must encrypt to the same thing otherwise we can\[aq]t find it
|
||||
on the cloud storage system.
|
||||
.PP
|
||||
This means that
|
||||
.IP \[bu] 2
|
||||
filenames with the same name will encrypt the same
|
||||
.IP \[bu] 2
|
||||
filenames which start the same won\[aq]t have a common prefix
|
||||
.PP
|
||||
This uses a 32 byte key (256 bits) and a 16 byte (128 bits) IV both of
|
||||
which are derived from the user password.
|
||||
.PP
|
||||
After encryption they are written out using a modified version of
|
||||
standard \f[C]base32\f[] encoding as described in RFC4648.
|
||||
The standard encoding is modified in two ways:
|
||||
.IP \[bu] 2
|
||||
it becomes lower case (no\-one likes upper case filenames!)
|
||||
.IP \[bu] 2
|
||||
we strip the padding character \f[C]=\f[]
|
||||
.PP
|
||||
\f[C]base32\f[] is used rather than the more efficient \f[C]base64\f[]
|
||||
so rclone can be used on case insensitive remotes (eg Windows, Amazon
|
||||
Drive).
|
||||
.SS Key derivation
|
||||
.PP
|
||||
Rclone uses \f[C]scrypt\f[] with parameters \f[C]N=16384,\ r=8,\ p=1\f[]
|
||||
with a an optional user supplied salt (password2) to derive the 32+32+16
|
||||
= 80 bytes of key material required.
|
||||
If the user doesn\[aq]t supply a salt then rclone uses an internal one.
|
||||
.PP
|
||||
\f[C]scrypt\f[] makes it impractical to mount a dictionary attack on
|
||||
rclone encrypted data.
|
||||
For full protection agains this you should always use a salt.
|
||||
.SS Local Filesystem
|
||||
.PP
|
||||
Local paths are specified as normal filesystem paths, eg
|
||||
@@ -4272,6 +4783,76 @@ Of course this will cause problems if the absolute path length of a file
|
||||
exceeds 258 characters on z, so only use this option if you have to.
|
||||
.SS Changelog
|
||||
.IP \[bu] 2
|
||||
v1.33 \- 2016\-08\-24
|
||||
.RS 2
|
||||
.IP \[bu] 2
|
||||
New Features
|
||||
.IP \[bu] 2
|
||||
Implement encryption
|
||||
.RS 2
|
||||
.IP \[bu] 2
|
||||
data encrypted in NACL secretbox format
|
||||
.IP \[bu] 2
|
||||
with optional file name encryption
|
||||
.RE
|
||||
.IP \[bu] 2
|
||||
New commands
|
||||
.RS 2
|
||||
.IP \[bu] 2
|
||||
rclone mount \- implements FUSE mounting of remotes (EXPERIMENTAL)
|
||||
.IP \[bu] 2
|
||||
works on Linux, FreeBSD and OS X (need testers for the last 2!)
|
||||
.IP \[bu] 2
|
||||
rclone cat \- outputs remote file or files to the terminal
|
||||
.IP \[bu] 2
|
||||
rclone genautocomplete \- command to make a bash completion script for
|
||||
rclone
|
||||
.RE
|
||||
.IP \[bu] 2
|
||||
Editing a remote using \f[C]rclone\ config\f[] now goes through the
|
||||
wizard
|
||||
.IP \[bu] 2
|
||||
Compile with go 1.7 \- this fixes rclone on macOS Sierra and on 386
|
||||
processors
|
||||
.IP \[bu] 2
|
||||
Use cobra for sub commands and docs generation
|
||||
.IP \[bu] 2
|
||||
drive
|
||||
.IP \[bu] 2
|
||||
Document how to make your own client_id
|
||||
.IP \[bu] 2
|
||||
s3
|
||||
.IP \[bu] 2
|
||||
User\-configurable Amazon S3 ACL (thanks Radek Šenfeld)
|
||||
.IP \[bu] 2
|
||||
b2
|
||||
.IP \[bu] 2
|
||||
Fix stats accounting for upload \- no more jumping to 100% done
|
||||
.IP \[bu] 2
|
||||
On cleanup delete hide marker if it is the current file
|
||||
.IP \[bu] 2
|
||||
New B2 API endpoint (thanks Per Cederberg)
|
||||
.IP \[bu] 2
|
||||
Set maximum backoff to 5 Minutes
|
||||
.IP \[bu] 2
|
||||
onedrive
|
||||
.IP \[bu] 2
|
||||
Fix URL escaping in file names \- eg uploading files with \f[C]+\f[] in
|
||||
them.
|
||||
.IP \[bu] 2
|
||||
amazon cloud drive
|
||||
.IP \[bu] 2
|
||||
Fix token expiry during large uploads
|
||||
.IP \[bu] 2
|
||||
Work around 408 REQUEST_TIMEOUT and 504 GATEWAY_TIMEOUT errors
|
||||
.IP \[bu] 2
|
||||
local
|
||||
.IP \[bu] 2
|
||||
Fix filenames with invalid UTF\-8 not being uploaded
|
||||
.IP \[bu] 2
|
||||
Fix problem with some UTF\-8 characters on OS X
|
||||
.RE
|
||||
.IP \[bu] 2
|
||||
v1.32 \- 2016\-07\-13
|
||||
.RS 2
|
||||
.IP \[bu] 2
|
||||
@@ -5480,6 +6061,10 @@ Antonio Messina <antonio.s.messina@gmail.com>
|
||||
.IP \[bu] 2
|
||||
Stefan G.
|
||||
Weichinger <office@oops.co.at>
|
||||
.IP \[bu] 2
|
||||
Per Cederberg <cederberg@gmail.com>
|
||||
.IP \[bu] 2
|
||||
Radek Šenfeld <rush@logic.cz>
|
||||
.SS Contact the rclone project
|
||||
.PP
|
||||
The project website is at:
|
||||
|
||||
Reference in New Issue
Block a user