f-denkena/rclone
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
3,715 Commits 1 Branch 0 Tags
dd150efdd74829f86c8a4eb09331cfef7e8b4eff
Commit Graph
71 Commits
Author SHA1 Message Date
Aleksandar JankovicandNick Craig-Wood 6b55b8b133 s3: add option for multipart failiure behaviour 
This is needed for resuming uploads across different sessions.
 2019-10-02 16:49:16 +01:00
Fabian MöllerandNick Craig-Wood 33f129fbbc s3: use lib/encoder 
Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
 2019-09-30 22:00:24 +01:00
Nick Craig-Wood 73e010aff9 docs: make the config walkthroughs consistent for each backend 2019-08-26 10:47:17 +01:00
Nick Craig-Wood 93810a739d docs: update fontawesome free to 5.10.2 and fixup broken images 2019-08-25 11:09:19 +01:00
Nick Craig-Wood 57d5de6fba build: fix up package paths after repo move 
git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g'
goimports -w `find . -name \*.go`
 2019-07-28 18:47:38 +01:00
Russell DavisandNick Craig-Wood 6a3c3d9b89 Update docs on S3 policy to include ListAllMyBuckets permission 
This permission is required for `rclone lsd`.
 2019-07-02 07:56:54 +01:00
Matti NiemenmaaandNick Craig-Wood a6dca4c13f s3: Add INTELLIGENT_TIERING storage class 
For Intelligent-Tiering:
https://aws.amazon.com/s3/storage-classes/#Unknown_or_changing_access
 2019-07-01 18:17:48 +01:00
Nick Craig-Wood 245fed513a Version v1.48.0 2019-06-15 13:55:41 +01:00
Nick Craig-Wood d533de0f5c docs: clarify that s3 can't access Glacier Vault 2019-06-15 09:56:17 +01:00
Philip HarveyandNick Craig-Wood 1a2fb52266 s3: make SetModTime work for GLACIER while syncing - Fixes #3224 
Before this change rclone would fail with

    Failed to set modification time: InvalidObjectState: Operation is not valid for the source object's storage class

when attempting to set the modification time of an object in GLACIER.

After this change rclone will re-upload the object as part of a sync if it needs to change the modification time.

See: https://forum.rclone.org/t/suspected-bug-in-s3-or-compatible-sync-logic-to-glacier/10187
 2019-06-03 15:28:19 +01:00
Animosity022andNick Craig-Wood b88e50cc36 docs: Typo fixes with "a existing" 
Fixed a typo with "a existing" to "an existing"
 2019-05-11 16:49:48 +01:00
Nick Craig-Wood a0d4fdb2fa Version v1.47.0 2019-04-13 11:01:58 +01:00
ManuandNick Craig-Wood 6e86526c9d s3: add support for "Glacier Deep Archive" storage class - fixes #3088 2019-04-11 10:21:41 +01:00
Ben BoeckelandNick Craig-Wood 3d424c6e08 docs: fix various typos 2019-03-28 15:12:51 +00:00
Nick Craig-Wood 27714e29c3 s3: note incompatibility with CEPH Jewel - fixes #3015 2019-03-06 11:50:37 +00:00
Nick Craig-Wood eb85ecc9c4 Version v1.46 2019-02-09 10:42:57 +00:00
Rémy LéoneandNick Craig-Wood 1e6cbaa355 s3: Add Scaleway to s3 documentation 2019-02-01 17:09:57 +00:00
Nick Craig-Wood 1318c6aec8 s3: Add Alibaba OSS to integration tests and fix storage classes 2019-01-12 20:41:47 +00:00
Nick Craig-Wood ff0b8e10af s3: Support Alibaba Cloud (Aliyun) OSS 
The existing s3 backend passed all integration tests with OSS provided
`force_path_style = false`.

This makes sure that is so and adds documentation and configuration
for OSS.

Thanks to @luolibin for their work on the OSS backend which we ended
up not needing.

Fixes #1641
Fixes #1237
 2019-01-12 17:28:04 +00:00
William CockerandNick Craig-Wood dda23baf42 s3 : update doc for Glacier storage class 
s3 : update doc for Glacier storage class : related to #923
 2018-12-06 21:53:38 +00:00
Nick Craig-Wood d99ffde7c0 s3: change --s3-upload-concurrency default to 4 to increase perfomance #2772 
Increasing the --s3-upload-concurrency to 4 (from 2) gives an
additional 45% throughput at the cost of 10MB extra memory per transfer.

After testing the upload perfoc
 2018-12-02 17:58:34 +00:00
Nick Craig-Wood 198c34ce21 s3: implement --s3-upload-cutoff for single part uploads below this - fixes #2772 
Before this change rclone would use multipart uploads for any size of
file.  However multipart uploads are less efficient for smaller files
and don't have MD5 checksums so it is advantageous to use single part
uploads if possible.

This implements single part uploads for all files smaller than the
upload_cutoff size.  Streamed files must be uploaded as multipart
files though.
 2018-12-02 17:58:34 +00:00
Nick Craig-Wood d35bd15762 Version v1.45 2018-11-24 13:44:25 +00:00
Nick Craig-Wood 0f02c9540c s3: make --s3-v2-auth flag 
This is an alternative to setting the region to "other-v2-signature"
which is inconvenient for multi-region providers.
 2018-10-14 00:10:29 +01:00
Nick Craig-Wood 78b9bd77f5 docs: auto generate backend options documentation 
This inserts the output of "rclone help backend xxx" into the help
pages for each backend.
 2018-10-06 11:47:46 +01:00
Nick Craig-Wood a9273c5da5 docs: move documentation for options from docs/content into backends 
In the following commit, the documentation will be autogenerated.
 2018-10-06 11:47:46 +01:00
Nick Craig-Wood 01af8e2905 s3: docs for how to configure Aliyun OSS / Netease NOS - thanks @xiaolei0125 2018-07-20 15:49:07 +01:00
Nick Craig-Wood f06ba393b8 s3: Add --s3-force-path-style - fixes #2401 2018-07-20 15:41:40 +01:00
themyloginandNick Craig-Wood 7d2861ead6 Adjust S3 upload concurrency with --s3-upload-concurrency 2018-06-14 16:15:17 +01:00
Nick Craig-Wood 311a962011 s3: Look in S3 named profile files for credentials - fixes #2243 2018-04-21 09:00:20 +01:00
Nick Craig-Wood dc247d21ff s3: add in config for all the supported S3 providers #2140 
These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other.

This configures endpoints where known and makes sure config doesn't
appear where it isn't valid where possible.
 2018-04-13 16:33:26 +01:00
Giri BadanahattiandNick Craig-Wood acd5d4377e config,s3: hierarchical configuration support #2140 
This introduces a method of making provider specific configuration
within a remote.  This is useful particularly in s3.

This commit does the basic configuration in S3 for IBM COS.
 2018-04-13 16:05:35 +01:00
Craig RachelandNick Craig-Wood 0e6faa2313 s3: add One Zone Infrequent Access storage class - fixes #2240 2018-04-13 13:36:25 +01:00
Peter BaumgartnerandNick Craig-Wood 1db68571fd s3,swift: Add --use-server-modtime 
`--use-server-modtime` stops s3 and swift retrieving the modtime from metadata which enables a fast sync mode with the `--update` flag.
 2018-04-13 13:32:17 +01:00
Nick Craig-Wood 92c5aa3786 s3: add --s3-chunk-size option - fixes #2203 2018-04-05 15:40:08 +01:00
gbadanahattiandNick Craig-Wood e5aa5fe7d8 s3: docs: Minor format and URL changes to IBM COS Documentation content 2018-03-26 20:49:53 +01:00
Nick Craig-Wood e62fe06763 s3: document --ignore-checksum workaround for KMS #1824 2018-03-17 10:51:45 +00:00
Giri BadanahattiandNick Craig-Wood aba43cd3a4 Documention for IBM COS (S3) configuration. 2018-03-15 20:20:43 +00:00
Nick Craig-Wood 89748feaa5 s3: update docs to discourage use of v2 auth - fixes #2120 
From testing it appears that CEPH no longer works properly with v2
auth and neither does Dreamhost, so update the docs anc configuration
to recommend v4 auth.
 2018-03-13 20:47:29 +00:00
Chris RedekopandNick Craig-Wood 4dd1e507f4 s3: set/get the hash for multipart files - #523 2018-01-06 14:30:10 +00:00
David MinorandNick Craig-Wood 4c116af1d0 s3: add support for ECS task IAM roles 
ECS container IAM metadata is in a different place than EC2 IAM metadata.
Use defaults' RemoteCredProvider function to query the standard locations
for the credentials.

Give the ECS role precedence over the role available from the underlying
EC2 instance.
 2017-11-23 08:56:56 +00:00
Corban RaunandNick Craig-Wood a8dfc5ce3b Fix spelling in some documentation 2017-11-05 21:38:59 +00:00
Andrew Starr-BochicchioandNick Craig-Wood e6fb96cfd4 Initial docs for usage with DigitalOcean Spaces. 2017-10-26 21:34:42 +01:00
Nick Craig-Wood 3e9861eecf docs: improve links to cloud providers 2017-09-30 09:28:49 +01:00
Nick Craig-Wood 1d994f7330 s3: add Wasabi instructions 2017-09-30 09:00:56 +01:00
Girish RamakrishnanandNick Craig-Wood 2f0551074c s3: set session token when using STS 2017-09-12 22:59:29 +01:00
Jan VarhoandNick Craig-Wood 5a1351f141 s3: Document glacier transitions and behavior 2017-09-12 09:27:32 +01:00
HarshavardhanaandNick Craig-Wood 8d77e48190 Minio supports ETags and metadata. 
Current doc mentioned lack of ETag and metadata
support which since has been long fixed in many
upstream Minio releases.

Also cleanup the doc to show new startup banner etc.
 2017-06-20 08:21:02 +01:00
Edward Q. BridgesandNick Craig-Wood 9876ba53f8 Updated permissions 
As it happens, after testing the `GetObject` permission is also required to do `HEAD` requests on a given object.
 2017-06-14 17:29:21 +01:00
Nick Craig-Wood 50928a5027 Implement --fast-list flag. 
This is supported remotes which can do a recursive listing.  It will
use more memory.

This is related to #1277 but doesn't fix that issue yet.
 2017-06-14 16:49:40 +01:00