f-denkena/rclone
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
6,024 Commits 1 Branch 0 Tags
fd453f2c7b5e52deb21d1e90c374985c950417b1
Commit Graph
87 Commits
Author SHA1 Message Date
Nick Craig-Wood 0c03aa3a8b dropbox: speed up directory listings by specifying 1000 items in a chunk 2021-11-11 11:14:22 +00:00
Nick Craig-Wood caa2b8bf40 dropbox: save an API request when at the root 
Before this change, rclone always emitted an API request to discover
what type of thing the root is.

This is unecessary as it is always a directory.
 2021-11-11 11:14:22 +00:00
Nick Craig-Wood e43b5ce5e5 Remove github.com/pkg/errors and replace with std library version 
This is possible now that we no longer support go1.12 and brings
rclone into line with standard practices in the Go world.

This also removes errors.New and errors.Errorf from lib/errors and
prefers the stdlib errors package over lib/errors.
 2021-11-07 11:53:30 +00:00
albertony e2f47ecdeb docs: punctuation cleanup 
See #5538
 2021-10-20 22:56:19 +02:00
Rajat Goel db6002952e dropbox: upgrade sdk version 2021-10-16 10:55:02 +01:00
Nick Craig-Wood 9ce0df3242 dropbox: add --dropbox-batch-commit-timeout to control batch timeout 
This also adds an Debug message log showing how long each batch took

See: #5491
 2021-10-15 15:32:40 +01:00
Nick Craig-Wood 3fbaa4c0b0 backends: make NewObject return fs.ErrorIsDir if possible 
This changes the interface to NewObject so that if NewObject is called
on a directory then it should return fs.ErrorIsDir if possible without
doing any extra work, otherwise fs.ErrorObjectNotFound.

Tested on integration test server with:

go run integration-test.go -tests backend -run TestIntegration/FsMkdir/FsPutFiles/FsNewObjectDir -branch fix-stat -maxtries 1
 2021-10-14 17:15:50 +01:00
albertony f1f923a986 Change byte unit format from MiByte to MiB 2021-08-31 09:57:27 +02:00
Ivan Andreev 5b6f637461 fs/hash: align hashsum names and update documentation (#5339) 
- Unify all hash names as lowercase alphanumerics without punctuation.
- Legacy names continue to work but disappear from docs, they can be depreciated or dropped later.
- Make rclone hashsum print supported hash list in case of wrong spelling.
- Update documentation.

Fixes #5071
Fixes #4841
 2021-05-21 17:32:33 +03:00
Alexey Ivanov 2fa7a3c0fb dropbox: simplify chunked uploads 
Signed-off-by: Alexey Ivanov <rbtz@dropbox.com>
 2021-05-14 14:07:44 +01:00
Nick Craig-Wood 75c417ad93 dropbox: fix async batch missing the last few entries 2021-05-14 14:07:44 +01:00
Nick Craig-Wood 5ee646f264 dropbox: make batcher retry all errors so it doesn't exit early 
See: https://forum.rclone.org/t/dropbox-too-many-requests-or-write-operations-trying-again-in-15-seconds/23316/18
 2021-05-14 14:07:44 +01:00
Nick Craig-Wood 4a4aca4da7 dropbox: fix deadlock in batch Commit 2021-05-14 14:07:44 +01:00
Nick Craig-Wood 2e4b65f888 dropbox: add --dropbox-batch-mode flag to speed up uploading #5156 
This adds 3 upload modes for dropbox off, sync and async and makes
sync the default.

This should improve uploads (especially for small files) greatly.
 2021-05-14 14:07:44 +01:00
Nick Craig-Wood 94dbfa4ea6 fs: change Config callback into state based callback #3455 
This is a very large change which turns the post Config function in
backends into a state based call and response system so that
alternative user interfaces can be added.

The existing config logic has been converted, but it is quite
complicated and folloup commits will likely be needed to fix it!

Follow up commits will add a command line and API based way of using
this configuration system.
 2021-05-14 14:07:44 +01:00
Nick Craig-Wood b78c9a65fa backends: remove log.Fatal and replace with error returns #5234 
This changes the Config interface so that it returns an error.
 2021-04-27 18:18:08 +01:00
albertony 2925e1384c Use binary prefixes for size and rate units 
Includes adding support for additional size input suffix Mi and MiB, treated equivalent to M.
Extends binary suffix output with letter i, e.g. Ki and Mi.
Centralizes creation of bit/byte unit strings.
 2021-04-27 02:25:52 +03:00
Nick Craig-Wood 5d85e6bc9c dropbox: fix Unable to decrypt returned paths from changeNotify - fixes #5165 
This was caused by incorrect use of strings.TrimLeft where
strings.TrimPrefix was required.
 2021-04-21 10:52:05 +01:00
Nick Craig-Wood 3e96752079 dropbox: add missing team_data.member scope for use with --impersonate 
See: https://forum.rclone.org/t/dropbox-business-not-accepting-oauth2/23390/32
 2021-04-17 17:40:08 +01:00
Nick Craig-Wood dcd4edc9f5 dropbox: fix About after scopes changes - rclone config reconnect needed 
This adds the missing scope for the About call. To use it it will be
necessary to refresh the token with `rclone config reconnect`.

See: https://forum.rclone.org/t/dropbox-too-many-requests-or-write-operations-trying-again-in-15-seconds/23316/33
 2021-04-16 15:07:03 +01:00
Nick Craig-Wood fcba51557f dropbox: set visibility in link sharing when --expire is set 
Note that due to a bug in the dropbox SDK you'll need to set --expire
to access this.

See: https://github.com/dropbox/dropbox-sdk-go-unofficial/issues/75
See: https://forum.rclone.org/t/rclone-link-dropbox-permissions/23211
 2021-04-07 13:58:37 +01:00
Nick Craig-Wood 9deab5a563 dropbox: raise priority of rate limited message to INFO to make it more noticeable 
If you exceed rate limits, dropbox tells you to wait for 300 seconds -
this is rather a long time for the user to be waiting for rclone to
finish, so emit a NOTICE level log instead of a DEBUG.
 2021-03-22 09:04:25 +00:00
Nick Craig-Wood 4013bc4a4c Fix excessive retries missing --max-duration timeout - fixes #4504 
This change checks the context whenever rclone might retry, and
doesn't retry if the current context has an error.

This fixes the pathological behaviour of `--max-duration` refusing to
exit because all the context deadline exceeded errors were being
retried.

This unfortunately meant changing the shouldRetry logic in every
backend and doing a lot of context propagation.

See: https://forum.rclone.org/t/add-flag-to-exit-immediately-when-max-duration-reached/22723
 2021-03-13 09:25:44 +00:00
Robert Thomas c3e2392f2b dropbox: fix polling support for scoped apps - fixes #5089 (#5092) 
This fixes the polling implementation for Dropbox, particularly
when using a scoped app. This also adds a lower end check for the
timeout, as I forgot to include that in the original implementation.
 2021-03-05 17:44:47 +00:00
Nick Craig-Wood 0ad38dd6fa dropbox,ftp,onedrive,yandex: make --timeout 0 work properly 
See: https://forum.rclone.org/t/an-issue-about-ftp-backend-in-2-different-systems/22551
 2021-03-01 12:08:58 +00:00
Robert Thomas e5aa92c922 dropbox: add polling support - fixes #2949 
This implements polling support for the Dropbox backend. The Dropbox SDK dependency had to be updated due to an auth issue, which was fixed on Jan 12 2021. A secondary internal Dropbox service was created to handle unauthorized SDK requests, as is necessary when using the ListFolderLongpoll function/endpoint. The config variable was renamed to cfg to avoid potential conflicts with the imported config package.
 2021-02-24 09:33:31 +00:00
Nick Craig-Wood 37e630178e dropbox: add scopes to oauth request and optionally "members.read" 
This change adds the scopes rclone wants during the oauth request.
Previously rclone left these blank to get a default set.

This allows rclone to add the "members.read" scope which is necessary
for "impersonate" to work, but only when it is in use as it require
authorisation from a Team Admin.

See: https://forum.rclone.org/t/dropbox-no-members-read/22223/3
 2021-02-13 12:35:24 +00:00
Nick Craig-Wood e073720a8f dropbox: enable short lived access tokens #4792 
Starting September 30th, 2021, the Dropbox OAuth flow will no longer
return long-lived access tokens. It will instead return short-lived
access tokens, and optionally return refresh tokens.

This patch adds the token_access_type=offline parameter which causes
dropbox to return short lived tokens now.
 2020-12-02 16:50:16 +00:00
Nick Craig-Wood 584523672c dropbox: test file name length before upload to fix upload loop 
Before this change rclone would upload the whole of multipart files
before receiving a message from dropbox that the path was too long.

This change hard codes the 255 rune limit and checks that before
uploading any files.

Fixes #4805
 2020-12-01 17:56:36 +00:00
Nick Craig-Wood a9585efd64 dropbox: make malformed_path errors from too long files not retriable 
Before this change, rclone would retry files with filenames that were
too long again and again.

This changed recognises the malformed_path error that is returned and
marks it not to be retried which stops unnecessary retrying of the file.

See #4805
 2020-12-01 17:56:36 +00:00
Nick Craig-Wood f6b1f05e0f dropbox: tidy repeated error message 2020-12-01 17:56:36 +00:00
Nick Craig-Wood 7fde48a805 dropbox: set Features.ReadMimeType=false as Object.MimeType not supported 2020-11-29 17:22:43 +00:00
Nick Craig-Wood 2e21c58e6a fs: deglobalise the config #4685 
This is done by making fs.Config private and attaching it to the
context instead.

The Config should be obtained with fs.GetConfig and fs.AddConfig
should be used to get a new mutable config that can be changed.
 2020-11-26 16:40:12 +00:00
Nick Craig-Wood f7efce594b config: add context.Context #3257 #4685 
This add config to the Config callback in the backends and the related
config functions.
 2020-11-09 18:05:54 +00:00
Nick Craig-Wood 8b96933e58 fs: Add context to fs.Features.Fill & fs.Features.Mask #3257 #4685 2020-11-09 18:05:54 +00:00
Nick Craig-Wood d846210978 fs: Add context to NewFs #3257 #4685 
This adds a context.Context parameter to NewFs and related calls.

This is necessary as part of reading config from the context -
backends need to be able to read the global config.
 2020-11-09 18:05:54 +00:00
Josh Soref bbe7eb35f1 docs: spelling: server-side 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref d0888edc0a Spelling fixes 
Fix spelling of: above, already, anonymous, associated,
authentication, bandwidth, because, between, blocks, calculate,
candidates, cautious, changelog, cleaner, clipboard, command,
completely, concurrently, considered, constructs, corrupt, current,
daemon, dependencies, deprecated, directory, dispatcher, download,
eligible, ellipsis, encrypter, endpoint, entrieslist, essentially,
existing writers, existing, expires, filesystem, flushing, frequently,
hierarchy, however, implementation, implements, inaccurate,
individually, insensitive, longer, maximum, metadata, modified,
multipart, namedirfirst, nextcloud, obscured, opened, optional,
owncloud, pacific, passphrase, password, permanently, persimmon,
positive, potato, protocol, quota, receiving, recommends, referring,
requires, revisited, satisfied, satisfies, satisfy, semver,
serialized, session, storage, strategies, stringlist, successful,
supported, surprise, temporarily, temporary, transactions, unneeded,
update, uploads, wrapped

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-14 15:21:31 +01:00
buengese 233bed6a73 dropbox: implement IDer - fixes #2928 2020-09-08 19:04:32 +02:00
buengese 575f061629 dropbox: add support for viewing shared files and folders 2020-09-08 19:02:35 +02:00
Tim Gallant 30eb094f28 oauthutil: adds SharedOptions for OAuth backends 
1. adds SharedOptions data structure to oauthutil
2. adds config.ConfigToken option to oauthutil.SharedOptions
3. updates the backends that have oauth functionality

Fixes #2849
 2020-08-07 16:32:01 +01:00
Nick Craig-Wood a2afa9aadd fs: Add directory to optional Purge interface - fixes #1891 
- add a directory to the optional Purge interface
- fix up all the backends
- add an additional integration test to test for the feature
- use the new feature in operations.Purge

Many of the backends had been prepared in advance for this so the
change was trivial for them.
 2020-07-31 17:43:17 +01:00
Nick Craig-Wood df9c930581 dropbox: fix public link by removing expires parameter 
Adding the expires parameter gives settings_error/not_authorized/.. errors.

The expires setting isn't in the documentation so this commit removes
it for now.
 2020-06-18 18:40:33 +01:00
Roman Kredentser 55ad1354b6 link: Add --expire and --unlink flags 
This adds expire and unlink fields to the PublicLink interface.

This fixes up the affected backends and removes unlink parameters
where they are present.
 2020-06-05 14:51:05 +01:00
Nick Craig-Wood 49ba4eeb86 oauthutil: tidy interface to Config to add Options struct 
The interface was getting so that a new function was needed for every
Config variant. Adding an Options struct fixes this.
 2020-05-26 11:27:01 +01:00
harry d91a547d59 dropbox: make error insufficient space to be fatal 2020-03-26 16:19:50 +00:00
Nick Craig-Wood 4b9da601be dropbox: treat insufficient_space errors as non retriable errors 
Before this change rclone would keep trying to upload files after
dropbox had signalled it was full.

This change makes the relevant error a non-retriable error.

See: https://forum.rclone.org/t/why-does-a-file-transfer-continue-when-there-is-no-available-storage/13677
 2020-01-18 11:10:18 +00:00
Nick Craig-Wood bafe7d5a73 backends: move encoding definitions from fs/encodings 2020-01-16 14:40:36 +00:00
Nick Craig-Wood 3c620d521d backend: adjust backends to have encoding parameter 
Fixes #3761
Fixes #3836
Fixes #3841
 2020-01-16 14:40:36 +00:00
Nick Craig-Wood 1ce1ea34aa hash: fix hash names for DropboxHash and CRC-32 
These were unintentionally renamed as part of 1dc8bcd48c

Fixes #3679
 2019-10-30 12:20:10 +00:00