f-denkena/rclone
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
10,258 Commits 1 Branch 0 Tags
cdabcc7cc42d3735ed1ffaf75ce5561d484fbebc
Commit Graph
184 Commits
Author SHA1 Message Date
Josh Soref bbe7eb35f1 docs: spelling: server-side 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref d0888edc0a Spelling fixes 
Fix spelling of: above, already, anonymous, associated,
authentication, bandwidth, because, between, blocks, calculate,
candidates, cautious, changelog, cleaner, clipboard, command,
completely, concurrently, considered, constructs, corrupt, current,
daemon, dependencies, deprecated, directory, dispatcher, download,
eligible, ellipsis, encrypter, endpoint, entrieslist, essentially,
existing writers, existing, expires, filesystem, flushing, frequently,
hierarchy, however, implementation, implements, inaccurate,
individually, insensitive, longer, maximum, metadata, modified,
multipart, namedirfirst, nextcloud, obscured, opened, optional,
owncloud, pacific, passphrase, password, permanently, persimmon,
positive, potato, protocol, quota, receiving, recommends, referring,
requires, revisited, satisfied, satisfies, satisfy, semver,
serialized, session, storage, strategies, stringlist, successful,
supported, surprise, temporarily, temporary, transactions, unneeded,
update, uploads, wrapped

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-14 15:21:31 +01:00
Nick Craig-Wood db4bbf9521 operations: fix use of --suffix without --backup-dir 
As part of the original work adding this feature it was overlooked
that this didn't actually work for full rclone copy/sync.

This commit fixes the problem and adds a test to make sure it stays
working.

See: https://forum.rclone.org/t/suffix-not-working-on-folder-upload-via-ssh-sftp/19526
 2020-10-04 16:40:20 +01:00
Nick Craig-Wood 75de30cfa8 sync: fix --cutoff-mode soft & cautious so it doesn't end the transfer early 
Before ths fix --cutoff-mode soft and cautious would emit a Fatal
error which stopped the sync immediately.

This fix introduces a new error which is checked in the sync error
processing which stops the sync gracefully.

Fixes #4576
 2020-09-09 12:53:21 +01:00
Nick Craig-Wood 122a47fba6 accounting: Allow transfers to be canceled with context #3257 
This makes all transfers cancelable even if the backend doesn't
support context as all transfers are done using the Accounting
framework.
 2020-07-28 16:41:17 +01:00
Nick Craig-Wood 8f42532b6d sync: add --track-renames-strategy leaf 
See: https://forum.rclone.org/t/how-to-minimize-bandwith-w-r-t-renames-during-sync/16928/22
 2020-07-28 11:34:27 +01:00
Nick Craig-Wood 811b30d116 sync: fix deadlock with --track-renames-strategy modtime - fixes #4427 
Before this change we could exit the popRenameMap function with the
lock held.

This fixes the problem by defer-ring the unlock.

See: https://forum.rclone.org/t/track-renames-strategy-modtime-doesnt-work/16992
 2020-07-17 17:09:58 +01:00
Nick Craig-Wood fb06427c69 sync: fix --track-renames-strategy modtime 
Before this change `--track-renames-strategy` was broken. The hashing
method it used could declare times that were very close together to be
different.

The time hash was discarded and instead we check the modification time
window on every hash match.

Provided that the user doesn't use `--track-renames-strategy` on a
huge number of identically sized files this will perform just fine.

See: https://forum.rclone.org/t/track-renames-strategy-modtime-doesnt-work/16992/5
 2020-06-12 15:38:35 +01:00
Nick Craig-Wood b23cf58a41 operations: Add skip all, do all, quit operations to --interactive - fixes #3886 
This also adds SkipDestructive into all the remaing places --dry-run
was used and adds documentation.
 2020-06-10 12:33:53 +01:00
Martin Michlmayr 041b201abd doc: fix typos throughout docs and code 2020-05-25 11:23:58 +01:00
Nick Craig-Wood 518d39815c sync,copy,move: add --check-first to do all checking before starting transfers 
See: https://forum.rclone.org/t/rclone-sync-doing-transfer-and-checking-in-paralel/16352/
 2020-05-22 17:50:07 +01:00
Nick Craig-Wood 147f97d1f7 sync: allow --max-backlog to be -ve meaning as large as possible 2020-05-22 17:50:07 +01:00
Martin Michlmayr 4aee962233 doc: fix typos throughout docs and code 2020-05-20 15:54:51 +01:00
Ben Zenker 899c8e0697 march: added flag to allow Unicode filenames to remain unique 
If your filenames contain two near-identical Unicode characters,
rclone will normalize these, making them identical. This flag
gives you the ability to keep them unique. This might
create unintended side effects, such as duplicating files that
contain certain Unicode characters, when downloading them from
certain cloud providers to a macOS filesystem.

Fixes #4228
 2020-05-15 12:28:01 +01:00
Nick Craig-Wood b2c9ef23fa sync: make --track-renames tests only check rename count if expecting renames 2020-03-31 10:58:49 +01:00
Nick Craig-Wood 5f9be3dd05 sync: make --track-renames tests less fragile by using rename stat 
Before this change these tests attempted to measure transfers and
checks in lieu of having a rename statistic with a very complicated
heuristic.

The change switches over to using the rename statistic which should be
100% reliable.
 2020-03-30 18:30:33 +01:00
Nick Craig-Wood ad9c7ff7ed sync: Fix incorrect "nothing to transfer" message using --delete-before 
Before this change the first pass of --delete-before would output
"There was nothing to transfer" and then proceed to transfer things.

This makes sure the message isn't printed in the delete phase.

See: https://forum.rclone.org/t/incorrect-debug-output/15267
 2020-03-30 16:45:02 +01:00
Nick Craig-Wood 32df634cb6 sync: fix --track-renames-strategy modtime test on remotes which don't support modtime 2020-03-22 11:52:40 +00:00
Nick Craig-Wood 93f5125f51 sync: fix --track-renames-strategy tests 
This commit corrects the logic for --track-renames-strategy which
broke the integration tests.

It also improves the parsing of the argument and adds a test for that.
 2020-03-21 17:39:51 +00:00
Bernd Schoolmann 158870bcdb fs: Add --track-renames-strategy for configurable matching criteria for --track-renames 
This commit adds the `--track-renames-strategy` flag which allows the
user to choose the strategy for tracking renames when using the
`--track-renames` flag.

This can be "hash" or "modtime" or both currently.

This, when used with `--track-renames-strategy modtime` enables
support for tracking renames in encrypted remotes.

Fixes #3696
Fixes #2721
 2020-03-20 13:04:56 +00:00
Nick Craig-Wood c227a90b52 sync: implement --order-by xxx,mixed 2020-03-16 15:50:04 +00:00
Nick Craig-Wood 1e3d899db8 sync: replace container/heap with github.com/aalpar/deheap 2020-03-16 15:50:04 +00:00
Anuar Serdaliyev f14871caf7 accounting: Correct exitcode on Transfer Limit Exceeded flag. Fixes #3203 
Before this change the exit code for transfer limit exceeded was
incorrect. This was because the `resolveExitCode` function unwraps the
error thus reading the underlying error which is not the same as the
error it was comparing to (`ErrorMaxTransferLimitReached`).

This change fixes it by splitting the error definition in two so that
when the Fatal error is unwrapped we match against
`ErrorMaxTransferLimitReached` however when we return the error we
return `ErrorMaxTransferLimitReachedFatal`.
 2020-03-10 12:00:10 +00:00
Ishuah Kariuki 0c6f14c694 copy/sync: only create empty directories when they don't exist on the remote 
Sync/copy now only creates empty directories when they don't exist on the remote (--create-empty-src-dirs flag) - fixes #2800
 2020-03-03 16:24:22 +00:00
Nick Craig-Wood 2b268f9724 build: fixup formatting after go1.14 go fmt changes 2020-02-28 16:58:33 +00:00
Tim Gallant 1d73f071f6 fs: improve log output when no changes are made - fixes #3454 
- changes a few log messages to debug level
- adds a log message for when 0 bytes are transferred
 2020-02-11 12:16:15 +00:00
Nick Craig-Wood 9c9cdf1712 fs: don't run tests for --max-duration on remote backends 
This is a timing dependent test and to make it long enough so that it
would work with the remotes would make it too long for local tests.

The code paths are identical for local vs non-local so just run on
local.

This fixes the integration tests.
 2020-01-26 09:23:03 +00:00
boosh 0d7573dd81 fs: Add --max-duration flag to control the maximum duration of a transfer session 
This gives you more control over how long rclone will run for, making
it easier to script backups, e.g. via cron. Once the `--max-duration`
time limit is reached, no new transfers will be initiated, but those
already in-flight will be allowed to complete.

Fixes #985
 2020-01-24 13:28:56 +00:00
Nick Craig-Wood 375d25f158 sync: implement --order-by flag to order transfers - fixes #1205 2020-01-16 15:24:36 +00:00
Nick Craig-Wood 207474abab sync: add --no-check-dest flag - fixes #3616 2019-12-29 16:47:57 +00:00
Ankur Gupta 75a6c49f87 Fix error counter - fixes #3650 
For few commands, RClone counts a error multiple times. This was fixed by
creating a new error type which keeps a flag to remember if the error has
already been counted or not. The CountError function now wraps the original
error eith the above new error type and returns it.
 2019-11-18 14:13:02 +00:00
Nick Craig-Wood 543100070a sync: free objects after they come out of the transfer pipe 
This reduces memory when the transfer pipe shrinks

See: https://forum.rclone.org/t/rclone-memory-consumption-increasing-linearly/12244
 2019-10-16 10:27:07 +01:00
Nick Craig-Wood f3b0f8a9f0 sync: --update/-u not transfer files that haven't changed - fixes #3232 
Before this change --update would transfer any file which was newer
than the destination regardless of whether it had changed or not.
This is needlessly wasteful of bandwidth.

After this change --update will only transfer files if they are newer
**and** they are different (checked with checksum and size).
 2019-10-12 11:54:56 +01:00
Ivan Andreev ccecfa9cb1 chunker: finish meta-format before release 
changes:
- chunker: remove GetTier and SetTier
- remove wdmrcompat metaformat
- remove fastopen strategy
- make hash_type option non-advanced
- adverise hash support when possible
- add metadata field "ver", run strict checks
- describe internal behavior in comments
- improve documentation

note:
wdmrcompat used to write file name in the metadata, so maximum metadata
size was 1K; removing it allows to cap size by 200 bytes now.
 2019-09-25 11:03:33 +01:00
Ivan Andreev 1e4691f951 tests/sync: adjust transfer counts for chunker 2019-09-24 12:45:12 +01:00
Nick Craig-Wood af3c47d282 fstest: remove -subdir flag as it no longer tests anything useful #3421 2019-08-17 09:54:19 +01:00
Nick Craig-Wood 57d5de6fba build: fix up package paths after repo move 
git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g'
goimports -w `find . -name \*.go`
 2019-07-28 18:47:38 +01:00
Aleksandar Jankovic 53a1a0e3ef accounting: add reference to completed transfers 
Add core/transferred call that lists completed transfers and their
status.
 2019-07-28 14:48:19 +01:00
Aleksandar Jankovic 8243ff8bc8 accounting: isolate stats to groups 
Introduce stats groups that will isolate accounting for logically
different transferring operations. That way multiple accounting
operations can be done in parallel without interfering with each other
stats.

Using groups is optional. There is dedicated global stats that will be
used by default if no group is specified. This is operating mode for CLI
usage which is just fire and forget operation.

For running rclone as rc http server each request will create it's own
group. Also there is an option to specify your own group.
 2019-07-28 14:48:19 +01:00
yparitcher 8e8b78d7e5 Implement --compare-dest & --copy-dest Fixes #3278 2019-07-22 19:42:29 +01:00
yparitcher b4266da4eb sync: fix SyncSuffix tests #3272 2019-07-03 17:36:22 +01:00
Aleksandar Jankovic cc0800a72e march: return errors when listing dirs 
Partially fixes #3172
 2019-07-01 15:32:46 +01:00
yparitcher de14378734 Implement --suffix without --backup-dir for current dir 
Fixes #2801
 2019-07-01 10:46:26 +01:00
yparitcher eea1b6de32 Abstract --Backup-dir checks so can be applied across Sync, Copy, Move 2019-07-01 10:46:26 +01:00
Laura 38ebdf54be sync/operations: don't use zero length files in tests 
We now have a backend (fichier) which doesn't support 0 length
files. Therefore all 0 length files in the tests have been replaced
with length 1.

In a future commit we will implement a test for 0 length files.
 2019-06-30 18:11:45 +01:00
Aleksandar Jankovic f78cd1e043 Add context propagation to rclone 
- Change rclone/fs interfaces to accept context.Context
- Update interface implementations to use context.Context
- Change top level usage to propagate context to lover level functions

Context propagation is needed for stopping transfers and passing other
request-scoped values.
 2019-06-19 11:59:46 +01:00
Nick Craig-Wood 75606dcc27 sync: fix tests on union remote 2019-06-15 10:53:36 +01:00
garry415 1124c423ee fs: add --ignore-case-sync for forced case insensitivity - fixes #2773 2019-06-03 21:12:10 +01:00
Nick Craig-Wood 8ee6034b23 Look for Fs in the cache rather than calling NewFs directly 
This will save operations when rclone is used in remote control mode
or with the same remote multiple times in the command line.
 2019-05-27 16:14:55 +01:00
Nick Craig-Wood 206e1caa99 fs/cache: factor Fs caching from fs/rc into its own package 2019-05-27 16:14:55 +01:00