f-denkena/rclone
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
4,418 Commits 1 Branch 0 Tags
7e48ee8758c4b049015f856e0a9801b9ff561323
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick Craig-Wood 7e48ee8758 cache: fix dedupe on caches wrapping drives - fixes #4320 
This implements the MergeDirs optional method.
 2020-06-10 21:52:52 +01:00
Nick Craig-Wood d55053098f check: make check do --checkers files concurrently - fixes #4318 2020-06-10 17:20:54 +01:00
Nick Craig-Wood 63cf0b1cdd check: make check command obey --dry-run/-i/--interactive - fixes #4325 2020-06-10 17:20:54 +01:00
Nick Craig-Wood 5866b1b017 bin: test-repeat.sh - script to run tests many times with individual logs 2020-06-10 17:08:31 +01:00
Nick Craig-Wood 8493f3939c bin: not-in-stable.go - script to help with merging fixes to the stable branch 2020-06-10 17:07:43 +01:00
Nick Craig-Wood 095f4e9b9d build: fix docker release build action 2020-06-10 17:00:33 +01:00
Nick Craig-Wood a1382a03aa Start v1.52.1-DEV development 2020-06-10 16:49:55 +01:00
Nick Craig-Wood 844b903595 docs: promote the use of -i/--interactive and "rclone sync -i" everywhere #1574 2020-06-10 12:33:53 +01:00
Nick Craig-Wood a3b3e1f646 tree: remove -i shorthand for --noindent as it conflicts with -i/--interactive 2020-06-10 12:33:53 +01:00
Nick Craig-Wood b23cf58a41 operations: Add skip all, do all, quit operations to --interactive - fixes #3886 
This also adds SkipDestructive into all the remaing places --dry-run
was used and adds documentation.
 2020-06-10 12:33:53 +01:00
Nick Craig-Wood 2ea15a72bc s3: fix --header-upload - Fixes #4303 
Before this change we were setting the headers on the PUT
request for normal and multipart uploads. For normal uploads this caused the error

    403 Forbidden: There were headers present in the request which were not signed

After this fix we set the headers in the object upload request itself
as the s3 SDK expects.

This means that we only support a limited range of headers

- Cache-Control
- Content-Disposition
- Content-Encoding
- Content-Language
- Content-Type
- X-Amz-Tagging
- X-Amz-Meta-

Note for the last of those are for setting custom metadata in the form
"X-Amz-Meta-Key: value".

This now works for multipart uploads and single part uploads

See also #59
 2020-06-10 12:28:48 +01:00
Nick Craig-Wood b5c654a100 lib/structs: factor reflection based structure manipulation into a library 2020-06-10 12:28:48 +01:00
Nick Craig-Wood 6807b0e42f Add Kamil Trzciński to contributors 2020-06-10 12:16:27 +01:00
Nick Craig-Wood 88df5927f9 vfs: funnel all read/write calls through ReadAt/WriteAt 
This is in preparation for partial reads for read/write files
 2020-06-09 18:07:41 +01:00
Nick Craig-Wood 8c37262e05 vfs: don't use embedded methods for read/write handles for clarity 2020-06-09 18:07:23 +01:00
Nick Craig-Wood 3c14a893fb asyncreader: Make StopBuffer as well as Abandon and fix confusion in callers 2020-06-09 18:05:12 +01:00
Nick Craig-Wood 05bc19c331 vfs: Remove uneeded locking from read write handle String() 2020-06-09 18:04:50 +01:00
Nick Craig-Wood c4110780bf lib/file: fix SetSparse on Windows 7 which fixes downloads of files > 250MB 
Before this change we passed both lpOverlapped and lpBytesReturned as NULL.

> If lpOverlapped is NULL, lpBytesReturned cannot be NULL. Even when
> an operation produces no output data, and lpOutBuffer can be NULL,
> the DeviceIoControl function makes use of the variable pointed to by
> lpBytesReturned. After such an operation, the value of the variable
> is without meaning.

After this change we set lpBytesReturned to a valid pointer.

See: https://forum.rclone.org/t/errors-when-downloading-any-file-over-250mb-from-google-drive-windows-sparse-files/16889
 2020-06-06 13:13:15 +01:00
Nick Craig-Wood d729004554 Add Roman Kredentser to contributors 2020-06-05 14:51:52 +01:00
Nick Craig-Wood fb61ed8506 b2: Implement server side copy for files > 5GB - fixes #3991 
This factors copy out of SetModTime and Copy so it can be called from
both places.

This also reworks all the multipart uploading to use sync.Errgroup and
memory pooling like the other backends. This makes it more memory
efficient and handle errors better.

See: https://forum.rclone.org/t/copying-files-within-a-b2-bucket/16680/10
 2020-06-05 13:27:53 +01:00
Nick Craig-Wood 4c7f7582fd obscure: write more help as we are referencing it elsewhere 2020-06-05 12:48:28 +01:00
Nick Craig-Wood 973e3d6a7b backends: make sure backends expand ~ and environment vars in file names they use 
See: https://forum.rclone.org/t/relative-path-in-rclone-config-service-account-json/16693
 2020-06-03 17:39:08 +01:00
Nick Craig-Wood b62d08d136 config: set RCLONE_CONFIG_DIR for use in config files and subprocesses 
See: https://forum.rclone.org/t/relative-path-in-rclone-config-service-account-json/16693
 2020-06-03 17:39:08 +01:00
Nick Craig-Wood 50e31c6636 vfs: fix OS vs Unix path confusion - fixes ChangeNotify on Windows 
See: https://forum.rclone.org/t/windows-mount-polling-not-recognising-all-changes-made-by-another-box/16708
 2020-06-03 17:05:58 +01:00
Nick Craig-Wood 151f03378f s3: fix upload of single files into buckets without create permission 
Before this change, attempting to upload a single file into an s3
bucket which did not have create permission gave AccessDenied: Access
Denied error when it tried to create the bucket.

This was masked until e2bf91452a was
fixed.

This fix marks the bucket as OK if a fetch on an object indicates it
is OK. This stops rclone thinking it has to create the bucket in the
first place.

Fixes #4297
 2020-06-02 14:33:21 +01:00
Nick Craig-Wood 26fb9007da build: fix xgo build after go1.14 go.mod update 
Before this change xgo was getting added to go.mod - the build then failed with

    go: inconsistent vendoring in /usr/src/rclone:
    github.com/karalabe/xgo@v0.0.0-20191115072854-c5ccff8648a7: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt

This change gets xgo in GOPATH mode to avoid it getting added to go.mod
 2020-06-02 13:41:54 +01:00
Nick Craig-Wood 3b20335d2a docs: remove leading slash in page reference in footer when present 2020-06-02 12:09:45 +01:00
Nick Craig-Wood 8d55367a6a build: set user_allow_other in /etc/fuse.conf in the Docker image 
This allows non root mounts to use the --allow-other flag

See: https://forum.rclone.org/t/trying-utilize-docker-for-the-first-time-having-some-issues-with-an-rclone-mount-user-allow-other-error-etc-fuse-conf-has-been-updated-to-allow/16393
 2020-06-01 21:33:48 +01:00
Nick Craig-Wood 187ee62e3d Add edwardxml to contributors 2020-06-01 21:33:48 +01:00
Nick Craig-Wood 10e2ec1fbb Add Matteo Pietro Dazzi to contributors 2020-06-01 21:33:48 +01:00
Nick Craig-Wood fef90ef0a9 build: update Docker build workflows 
- prune docker images to ones we normally build binaries for
- add fixed versions
- add fetch-depth to fetch the tags so the version number is correct
- rename the job names
 2020-06-01 16:13:23 +01:00
Nick Craig-Wood eee28d0d39 build: remove quicktest from Dockerfile 
This is making docker builds take too long and it isn't the place of
the Docker file to be running unit tests.
 2020-06-01 16:13:23 +01:00
Nick Craig-Wood b59999dd59 build: update go.mod to go1.14 to enable -mod=vendor build 
When the main module contains a top-level vendor directory and its
go.mod file specifies go 1.14 or higher, the go command now defaults
to -mod=vendor for operations that accept that flag.
 2020-06-01 16:13:23 +01:00
Nick Craig-Wood e62c032184 docs: remove manually set dates and use git dates instead 2020-06-01 13:07:46 +01:00
Nick Craig-Wood 1635b37ff1 docs: Add link to source and modified time to footer of every page 2020-06-01 13:07:46 +01:00
Nick Craig-Wood 8774381e2e cmd: Note commands which need obscured input in the docs - fixes #4252 2020-05-31 12:59:09 +01:00
Nick Craig-Wood cbfe7a405b Add Alex Guerrero to contributors 2020-05-31 12:59:09 +01:00
Nick Craig-Wood cbf3d43561 drive: fix missing items when listing using --fast-list / ListR 
This is caused by a bug in Google drive where, in some circumstances
querying for "(A in parents) or (B in parents)" returns nothing
whereas querying for "A in parents" and "B in parents" separately
works fine.

This has been reported here:

https://issuetracker.google.com/issues/149522397

This workaround detects this condition by seeing if a listing for more
than one directory at once returns nothing.

If it does then it retries each one individually.

This can potentially have a false positive if the user has multiple
empty directories which are queried at once. The consequence of this
will be that ListR is disabled for a while until the directories are
found to be actually empty in which case it will be re-enabled.

Fixes #3114 and Fixes #4289
 2020-05-31 11:44:15 +01:00
Nick Craig-Wood 764b90a519 Update release notes with Docker build instructions 2020-05-28 13:12:13 +01:00
Nick Craig-Wood d785942ed5 mountlib: fix rc tests when mount does not work 
This fixed the Docker 1.52 build
 2020-05-28 13:11:42 +01:00
Nick Craig-Wood 1cceadaf7c Start v1.52.0-DEV development 2020-05-27 18:36:32 +01:00
Nick Craig-Wood 6882aeff97 Version v1.52.0 2020-05-27 17:31:10 +01:00
Nick Craig-Wood a0922643e6 build: fix build after nfpm change to drop bindir 2020-05-27 17:31:09 +01:00
Nick Craig-Wood 4d7254f88f Add funding links to rclone repo 2020-05-27 17:31:09 +01:00
Nick Craig-Wood 8499090038 docs: Updates to the front page from Edward Barker 2020-05-27 17:31:09 +01:00
Nick Craig-Wood ca0c4c7585 docs: update index and donate pages after review 2020-05-27 17:31:09 +01:00
Nick Craig-Wood 5aec01100e docs: make primary buttons rclone colors 2020-05-27 17:31:09 +01:00
Nick Craig-Wood 4d6af44045 docs: update all auto generated docs 2020-05-27 17:31:09 +01:00
Nick Craig-Wood 256d17eba8 docs: markdownify the strings in the commands index 2020-05-27 17:31:09 +01:00
Nick Craig-Wood 5777f4481f docs: add missing shortcode 2020-05-27 17:31:09 +01:00
Nick Craig-Wood 16e89706d3 docs: add a status notice to the cache backend 2020-05-27 17:31:09 +01:00
Nick Craig-Wood 41a58d7f6e docs: tweaks to index and donate page 2020-05-27 17:31:09 +01:00
Nick Craig-Wood 2a7f3eecf4 docs: reduce height of provider entries 2020-05-27 17:31:09 +01:00
Nick Craig-Wood 41e12114a8 docs: fix validation errors 2020-05-27 17:31:09 +01:00
Nick Craig-Wood 1b12d5d346 docs: move most of the chrome into baseof.html as per a modern hugo install 2020-05-27 17:31:09 +01:00
Nick Craig-Wood 74b8cbfb84 docs: set unsafe HTML parsing to false and fix raw HTML insertion 
This means that markdown files can't contain <thing> any more.
 2020-05-27 17:31:09 +01:00
Nick Craig-Wood 06427371eb docs: add "make validate_website" and fix validation errors 2020-05-27 17:31:09 +01:00
Nick Craig-Wood d08e1616a7 docs: add autogenerated date to footer 2020-05-27 17:31:09 +01:00
Nick Craig-Wood 79b011305e docs: increase space around titles for better visual feel 2020-05-27 17:31:09 +01:00
Nick Craig-Wood 050879f0ca docs: make command docs titles be one higher 2020-05-27 17:31:09 +01:00
Nick Craig-Wood 0a74f8022e docs: tweak rendering of tables to match the new theme 2020-05-27 17:31:09 +01:00
Nick Craig-Wood 0db20c9c56 docs: Remove spurious styles from donate page 
Correct link when not running on real URL
 2020-05-27 17:31:09 +01:00
Nick Craig-Wood 33e9db9745 docs: Use versioned css/jss 2020-05-27 17:31:09 +01:00
Nick Craig-Wood d0145d4359 docs: rename css and js includes to have their version numbers in 2020-05-27 17:31:09 +01:00
Nick Craig-Wood 80dab10ec9 docs: Fix generated docs 
In distributed docs
- Make img tags absolute
- Add donate link in docs
- Fix provider table
 2020-05-27 17:31:08 +01:00
Nick Craig-Wood 022cda3109 docs: update provider list on front page 2020-05-27 17:31:08 +01:00
Nick Craig-Wood 4030672b66 docs: Add upload_test_website target to Makefile 2020-05-27 17:31:08 +01:00
Nick Craig-Wood 348379625c docs: update donations page 2020-05-27 17:31:08 +01:00
Nick Craig-Wood ab4a2275eb docs: add copy to clipboard javascript 
add popper library for popup menus
 2020-05-27 17:31:08 +01:00
Nick Craig-Wood c8683fc916 docs: bring README up to date 2020-05-27 17:31:08 +01:00
Nick Craig-Wood 273ee0d696 docs: remove unused page types 2020-05-27 17:31:08 +01:00
Nick Craig-Wood 6d19bbba73 docs: add description to commands and index page 2020-05-27 17:31:08 +01:00
Nick Craig-Wood d6c31b51c6 docs: new content for index and donate pages 2020-05-27 17:31:08 +01:00
Nick Craig-Wood af19f924ff docs: layout tweaks 2020-05-27 17:31:08 +01:00
Nick Craig-Wood 80572c544b docs: more config fixes for hugo 0.7 2020-05-27 17:31:08 +01:00
Nick Craig-Wood 11f8cb32d1 docs: new flags 2020-05-27 17:31:08 +01:00
Nick Craig-Wood 62781d0925 docs: fix layouts 2020-05-27 17:31:08 +01:00
Nick Craig-Wood 2bd786a452 docs: fix rc docs and update anchors for new Hugo version 2020-05-27 17:31:08 +01:00
Nick Craig-Wood 038648aaca docs: rename about.md to _index.md after Hugo upgrade 
Use the official way of including markdown content into index.html
 2020-05-27 17:31:08 +01:00
Nick Craig-Wood 0e3ac6b13c docs: updated donate page 2020-05-27 17:31:08 +01:00
Nick Craig-Wood 1e274b01fe docs: fix layouts after bootstrap upgrade 
- use bootstrap recommended viewport
 2020-05-27 17:31:08 +01:00
Nick Craig-Wood c4f6439715 docs: update Hugo config after updating to 0.69.2 2020-05-27 17:31:08 +01:00
Nick Craig-Wood cdecb44789 docs: update bootstrap to 4.4.1 and jQuery to 3.5.1 2020-05-27 17:31:08 +01:00
Nick Craig-Wood 78ca08ba8a pcloud: fix initial config "Auth state doesn't match" message #4210 
pCloud should be passing back the state parameter that rclone passed
in on config but it seems to have got lost somewhere.

This sets a work-around for the pCloud backend allowing an empty state
parameter.

See: https://forum.rclone.org/t/cannot-connect-to-pcloud/16592
See: https://forum.rclone.org/t/cannot-create-pcloud-config-file-on-osx/16583
 2020-05-26 11:27:01 +01:00
Nick Craig-Wood 49ba4eeb86 oauthutil: tidy interface to Config to add Options struct 
The interface was getting so that a new function was needed for every
Config variant. Adding an Options struct fixes this.
 2020-05-26 11:27:01 +01:00
Nick Craig-Wood c08617c70f box: Calculate Free amount in About call 2020-05-25 16:47:34 +01:00
Nick Craig-Wood 1bd3365868 vfs: fix TestVFSStatfs with known total, used and unknown free 2020-05-25 16:46:56 +01:00
Nick Craig-Wood 31f21551bf mount: change maximum leaf name length to 1024 bytes - fixes #3884 
This limit was previously 4k set in 59026c4761 however leaf
names above 1k now produce an IO error.

WinFSP seems to have its own method for dropping too long file names
above 255 long.
 2020-05-25 15:41:11 +01:00
Nick Craig-Wood baaec5b126 testserver: don't allow starting of an empty remote 2020-05-25 15:26:06 +01:00
Nick Craig-Wood 9db8ecbc32 box: implement About to read size used - fixes #4264 2020-05-23 18:46:44 +01:00
Nick Craig-Wood 518d39815c sync,copy,move: add --check-first to do all checking before starting transfers 
See: https://forum.rclone.org/t/rclone-sync-doing-transfer-and-checking-in-paralel/16352/
 2020-05-22 17:50:07 +01:00
Nick Craig-Wood 147f97d1f7 sync: allow --max-backlog to be -ve meaning as large as possible 2020-05-22 17:50:07 +01:00
Nick Craig-Wood 7f44735709 Add Daniel Slyman to contributors 2020-05-22 17:50:07 +01:00
Nick Craig-Wood cf5d0f5c1f Revert "drive: server side copy docs use default description if empty" 
This reverts commit 9e4b68a364.

This does not work as intended - it only changes docs files and to
make it change drive files would take an extra roundtrip.

I think the sematics of server side copy are now correct - additional
features should be added with a new flag.

See #4230
 2020-05-19 16:48:02 +01:00
Nick Craig-Wood 4d431e94b9 oauth2: try to make token expiry messages more helpful - fixes #4250 
See also: #4251
 2020-05-19 16:19:35 +01:00
Nick Craig-Wood bdafbad61e cache: fix tests writing to empty path 
This meant the tests were writing to the current directory instead of
a temporary directory.
 2020-05-19 16:01:35 +01:00
Nick Craig-Wood eb6e9b194a fspath: Stop empty strings being a valid path - fixes #4239 
Before this change you could use "" as a valid remote, so `rclone lsf
""` would work. This was treated as the current directory.

This is unexpected and creates a footgun for scripting when an empty
variable is passed to rclone by accident.

This fix returns the error "can't use empty string as a path" instead
of allowing it.
 2020-05-19 12:34:23 +01:00
Nick Craig-Wood ecdfd80459 Add Brandon McNama to contributors 2020-05-19 12:22:59 +01:00
Nick Craig-Wood a268eedb1d Add Martin Michlmayr to contributors 2020-05-19 12:22:59 +01:00
Nick Craig-Wood 610f40f700 local: implement --local-no-sparse flag for disabling sparse files #2469 
This also introduces a one time warning for sparse files and updates
the docs to warn about them.
 2020-05-19 10:16:43 +01:00