Commit Graph
76 Commits
Author SHA1 Message Date
Fabian Möller 15ce0ae57c fstests: fix maximum tested size in TestFsPutChunked
Before this it was possible hat maxChunkSize was incorrectly set to 200.
2018-10-16 11:50:47 +02:00
Fabian Möller 57273d364b fstests: add TestFsPutChunked 2018-10-11 14:47:58 +01:00
Fabian Möller 84289d1d69 readers: add NewPatternReader 2018-10-11 14:47:58 +01:00
Fabian Möller 98e2746e31 backend: add fstests.ChunkedUploadConfig
- azureblob
- b2
- drive
- dropbox
- onedrive
- s3
- swift
2018-10-11 14:47:58 +01:00
Fabian Möller c00ec0cbe4 fstests: add ChunkedUploadConfig 2018-10-11 14:47:58 +01:00
Fabian Möller 1a40bceb1d backend: unify NewFs path handling for wrapping remotes
Use the same function to join the root paths for the wrapping remotes
alias, cache and crypt.
The new function fspath.JoinRootPath is equivalent to path.Join, but if
the first non empty element starts with "//", this is preserved to allow
Windows network path to be used in these remotes.
2018-10-10 17:50:27 +01:00
Fabian Möller 1e2676df26 union: fix ChangeNotify to support multiple remotes
To correctly support multiple remotes, each remote has to receive a
value on the input channel.
2018-10-07 11:13:37 +02:00
Fabian Möller 1d14972e41 vfs: reduce directory cache cleared by poll-interval
Reduce the number of nodes purged from the dir-cache when ForgetPath is
called. This is done by only forgetting the cache of the received path
and invalidating the parent folder cache by resetting *Dir.read.

The parent will read the listing on the next access and reuse the
dir-cache of entries in *Dir.items.
2018-10-02 10:21:14 +01:00
Fabian Möller 05fa9cb379 drive: improve directory notifications in ChangeNotify
When moving a directory in drive, most of the time only a notification
for the directory itself is created, not the old or new parents.

This tires to find the old path in the dirCache and the new path with
the dirCache of the new parent, which can result in two notifications
for a moved directory.
2018-10-02 10:14:14 +01:00
Fabian Möller 527c0af1c3 drive: cleanup changeNotifyRunner 2018-09-25 17:54:48 +02:00
Fabian Möller a20fae0364 drive: code cleanup 2018-09-25 15:20:23 +01:00
Fabian Möller 15b1a1f909 drive: add support for apps-script to json export 2018-09-25 15:20:23 +01:00
Fabian Möller 80b25daac7 drive: add support for multipart document extensions 2018-09-25 15:20:23 +01:00
Fabian Möller 70b30d5ca4 drive: add document links 2018-09-25 15:20:23 +01:00
Fabian Möller 0b2fc621fc drive: restructure Object type 2018-09-25 15:20:23 +01:00
Fabian Möller 171e39b230 drive: add --drive-import-formats
Add a new flag to the drive backend to allow document conversions oni upload.
The existing --drive-formats flag has been renamed to --drive-export-formats.
The old flag is still working to be backward compatible.
2018-09-25 15:20:23 +01:00
Fabian Möller 690a44e40e drive: rewrite mime type and extension handling
Make use of the mime package to find matching extensions and mime types.
For simplicity, all extensions are now prefixed with "." to match the
mime package requirements.

Parsed extensions get converted if needed.
2018-09-25 15:20:23 +01:00
Fabian Möller d9a3b26e47 vfs: add vfs/poll-interval rc command
This command can be used to query the current status of the
poll-interval option and also update the value.
2018-09-25 14:01:13 +02:00
Fabian Möller 1eec59e091 fs: update ChangeNotifier interface
This introduces a channel to the ChangeNotify function, which can be
used to update the poll-interval and cleanly exit the polling function.
2018-09-25 14:01:13 +02:00
Fabian Möller 03ea05b860 drive: add workaround for slow downloads
Add --drive-v2-download-min-size flag to allow downloading files via the
drive v2 API. If files are greater than this flag, a download link is
generated when needed. The flag is disabled by default.
2018-09-18 15:55:50 +01:00
Fabian Möller b8678c9d4b vendor: add google.golang.org/api/drive/v2 2018-09-18 15:55:50 +01:00
Fabian Möller 13823a7743 drive: fix escaped chars in documents during list
Fixes #2591
2018-09-18 15:53:44 +01:00
Fabian Möller 3c44ef788a cache: add plex_insecure option to skip certificate validation
Fixes #2215
2018-09-10 21:19:25 +01:00
Fabian Möller e5663de09e docs: add section for .plex.direct urls to cache 2018-09-10 21:15:18 +01:00
Fabian Möller f6ee0795ac cache: preserve leading / in wrapped remote path
When combining the remote value and the root path, preserve the absence
or presence of the / at the beginning of the wrapped remote path.

e.g. a remote "cloud:" and root path "dir" becomes "cloud:dir" instead
of "cloud:/dir".

Fixes #2553
2018-09-10 20:35:50 +01:00
Fabian Möller eb5a95e7de crypt: preserve leading / in wrapped remote path
When combining the remote value and the root path, preserve the absence
or presence of the / at the beginning of the wrapped remote path.

e.g. a remote "cloud:" and root path "dir" becomes "cloud:dir" instead
of "cloud:/dir".
2018-09-10 20:35:50 +01:00
Fabian Möller cc23ad71ce ncdu: return error instead of log.Fatal in Show 2018-09-07 10:06:37 +01:00
Fabian Möller 6b8b9d19f3 googlecloudstorage: fix service_account_file been ignored - Fixes #2523 2018-09-04 15:31:20 +01:00
Fabian Möller deda093637 cache: fix error return value of cache/fetch rc method 2018-09-03 17:32:11 +02:00
Fabian Möller a0b3fd3a33 cache: fix worker scale down
Ensure that calling scaleWorkers will create/destroy the right amount of
workers.
2018-09-03 12:29:35 +02:00
Fabian Möller cdbe3691b7 cache: add cache/fetch rc function 2018-09-03 12:29:35 +02:00
Fabian Möller 3a0b3b0f6e drive: reformat long API call lines 2018-09-03 12:22:05 +02:00
Fabian Möller f818df52b8 config: add List type 2018-08-20 17:38:51 +01:00
Fabian Möller 6dc5aa7454 docs: clearify buffer-size is per transfer/filehandle 2018-08-17 18:11:40 +01:00
Fabian Möller 552eb8e06b vfs: try to seek buffer on read only files 2018-08-17 18:10:28 +01:00
Fabian Möller 2e80d4c18e vfs: update vfs/refresh rc command documentation 2018-08-07 09:31:12 +01:00
Fabian Möller 6349147af4 vfs: add non recursive mode to vfs/refresh rc command 2018-08-07 09:31:12 +01:00
Fabian Möller 782972088d vfs: add the vfs/refresh rc command
vfs/refresh will walk the directory tree for the given paths and
freshen the directory cache. It will use the fast-list capability
of the remote when enabled.
2018-08-07 09:31:12 +01:00
Fabian Möller 38381d3786 lsjson: add option to show the original object IDs 2018-08-07 09:28:55 +01:00
Fabian Möller eb6aafbd14 cache: implement fs.ObjectUnWrapper 2018-08-07 09:28:55 +01:00
Fabian Möller dc5a734522 drive: implement ListR 2018-08-06 21:31:47 +01:00
Fabian Möller dd670ad1db drive: handle gdocs when filtering file names in list
Fixes #2399
2018-07-30 13:01:16 +01:00
Fabian Möller 7983b6bdca vfs: enable vfs-read-chunk-size by default 2018-07-29 18:17:05 +01:00
Fabian Möller 9815b09d90 fs: add multipliers for SizeSuffix 2018-07-29 18:17:05 +01:00
Fabian Möller 9c90b5e77c stats: use appropriate Lock func's 2018-07-22 11:33:19 +02:00
Fabian Möller 6791cf7d7f atexit: prevent Run from being called on nil signal 2018-05-12 18:59:25 +02:00
Fabian Möller d022c81d99 mount: ensure atexit gets run on interrupt
When running `rclone mount`, there were 2 signal handlers for `os.Interrupt`.

Those handlers would run concurrently and in some cases cause either unmount or `atexit.Run()` being skipped.

In addition `atexit.Run()` will get called in `resolveExitCode` to ensure cleanup on errors.
2018-05-12 10:40:44 +01:00
Fabian Möller 98bf65c43b vfs: fix ChangeNotify for new or changed folders
Fixes #2251
2018-05-05 12:54:03 +01:00
Fabian Möller 3b5e70c8c6 drive: fix ChangeNotify for folders 2018-05-05 12:54:03 +01:00
Fabian Möller bd3ad1ac3e vfs: add option to read source files in chunks 2018-05-05 12:49:42 +01:00
Fabian Möller 9fdf273614 fs: improve ChunkedReader
- make Close permanent and return errors afterwards
- use RangeSeek from the wrapped reader if present
- add a limit to chunk growth
- correct RangeSeek interface behavior
- add tests
2018-05-05 12:49:42 +01:00
Fabian Möller 21faac6e6c Add David0rk to contributors 2018-02-27 10:06:56 +01:00
Fabian Möller 167a4396c7 drive: remove debug binary 2018-02-27 09:59:06 +01:00
Fabian Möller 00adf40f9f cryptdecode: use Cipher instead of NewFs (#2087)
* crypt: extract NewCipher out of NewFs
* cryptdecode: make use of crypt.NewCipher

Fixes #2075
2018-02-25 12:57:14 +01:00
Fabian Möller aeefa34f62 fstests: add TestInternal (#2085)
TestInternal allows to perform a custom test on the backend using the
optional InternalTester interface.
2018-02-25 10:58:06 +01:00
Fabian Möller 451cd6d971 fs: add ChunkedReader 2018-02-19 15:45:24 +00:00
Fabian Möller a647c54888 fs: add RangeSeeker interface 2018-02-19 15:45:24 +00:00
Fabian Möller 62e72801be vfs: fix race between multiple RWFileHandle (#2052)
Fixes #2034
2018-02-18 14:12:26 +01:00
Fabian Möller b183bd7f00 alias: add new backend to create aliases for remote names #1049
The alias backend is a wrapper for an existing remote.
It allows you to name a "remote:path" as an "alias:".
2018-02-06 18:23:47 +00:00
Fabian Möller 3c7a755631 lsjson: explain the Path value in the docs 2018-01-31 20:06:01 +00:00
Fabian Möller cf6d522d2f drive: fix upload to existing file (#2032)
This fixes uploads to existing files for Google Drive introduced by #2007.
Instead of updating the old file a new "Untitled" file would be created
in the root folder.
2018-01-30 14:37:06 +01:00
Fabian Möller 29d428040c cache: clean root path (#2023)
Trim "/" from the root path to fix "slice bounds out of range" panic
in cache.go:1272.

Fixes #1945
2018-01-30 14:35:40 +01:00
Fabian Möller 1aa482c333 drive: fix chunked upload (#2030) 2018-01-29 23:36:39 +01:00
Fabian Möller f622017539 drive: use contains for name matching in list
Use contains for name matching in list to work around #1675.
2018-01-29 14:18:49 +00:00
Fabian Möller 07f20dd1fd drive: migrate to api v3 2018-01-29 12:00:02 +00:00
Fabian Möller 29286cc8b3 drive: fix single Drive Document as FS root
Allow using Drive Documents as FS root by doing a direcoty list during NewFS.

Fixes #1772
2018-01-26 09:59:36 +00:00
Fabian Möller 1f5e23aedb scripts: make absolute paths consistent
Change absolute binary paths in scripts to /usr/bin/env or make them
relative.
This allows the scripts to be used on linux distributions
like NixOS, where binaries are not located in /usr/ or /bin/.
2018-01-26 09:39:05 +00:00
Fabian Möller 17dabf7a99 ftp: fix RangeOption support in Open #1825 2018-01-25 10:21:00 +00:00
Fabian Möller 9520992a54 sftp: fix RangeOption support in Open #1825 2018-01-25 10:20:43 +00:00
Fabian Möller a3dd2c691e amazonclouddrive: remove unnecessary notifies from DirChangeNotify
It is unnecessary to notify the node.Parents, because a cahnge event is
generated for all involved files and folders in a move from d1/f1 to
d2/f1. There will be a event for d1, d2 and f1.

Additionally a duplicate notification is resolved when them empty string
is in pathsToClear.

Related to #2006
2018-01-25 10:19:06 +00:00
Fabian Möller 186f78d44f local: fix RangeOption support in Open #1825 2018-01-21 19:50:26 +00:00
Fabian Möller 115d24e1f7 amazonclouddrive: implement DirChangeNotify
Use the Changes API to invalidate cache entries.
The latest retrieved checkpoint is stored in the config file to allow
fast resumption after restart.
2018-01-20 18:48:52 +00:00
Fabian Möller d8b3bf014d mount: use sdnotify to signal systemd the mount is ready
When the NOTIFY_SOCKET environment variable is set notify systemd after
the mount is ready.
2017-11-20 11:03:10 +00:00
Fabian Möller 0bfa29cbcf vendor: add github.com/okzk/sdnotify 2017-11-20 11:03:10 +00:00
Fabian Möller ce5b3a531d crypt: implement DirChangeNotify
crypt now implements the DirChangeNotify if the wrapped FS provides it.
2017-11-19 20:09:52 +00:00
Fabian Möller 558f014d43 migrate Gopkg.toml and Gopkg.lock to new format
Update Gopkg.toml and Gopkg.lock to follow the breaking changes
introduced by https://github.com/golang/dep/pull/644
2017-06-01 16:23:13 +01:00