Commit Graph
52 Commits
Author SHA1 Message Date
Ivan Andreev ef5c212f9b version: show build tags and type of executable
This patch modifies the output of `rclone version`.
The `os/arch` line is split into `os/type` and `os/arch`.
The `go version` line is now tagged as `go/version` for consistency.

Additionally the `go/linking` line tells whether the rclone
was linked as a static or dynamic executable.
The new `go/tags` line shows a space separated list of build tags.

The info about linking and build tags is also added to the output
of the `core/version` RC endpoint.
2021-03-22 22:55:24 +03:00
Ivan AndreevandGitHub 2bdbf00fa3 selfupdate: add instructions on reverting the update (#5141) 2021-03-18 23:11:16 +03:00
Ivan AndreevandGitHub e91f2e342a docs: mention rclone selfupdate in quickstart (#5122) 2021-03-13 23:02:40 +03:00
Ivan AndreevandGitHub 6fa74340a0 cmd: rclone selfupdate (#5080)
Implements self-update command
Fixes #548
Fixes #5076
2021-03-11 22:39:30 +03:00
Ivan AndreevandGitHub 75056dc9b2 ftp: update dependency jlaffaye/ftp (#5097) 2021-03-05 15:58:04 +03:00
Ivan AndreevandGitHub 7aa7acd926 address stringent ineffectual assignment check in golangci-lint (#5093) 2021-03-04 14:26:48 +03:00
Ivan Andreev 0dd3ae5e0d Add Robert Thomas to contributors 2021-02-24 19:40:54 +03:00
Ivan Andreev f6265fbeff Add pvalls to contributors 2021-02-24 03:35:24 +03:00
Ivan Andreev 1397b85214 Add Georg Neugschwandtner to contributors 2021-02-24 03:28:15 +03:00
Ivan Andreev 86a0dae632 Add Rauno Ots to contributors 2021-02-24 03:27:16 +03:00
Ivan Andreev 076ff96f6b webdav: check that purged directory really exists (#2921)
Sharepoint 2016 returns status 204 to the purge request
even if the directory to purge does not really exist.
This change adds an extra check to detect this condition
and returns a proper error code.
2021-02-23 23:27:30 +00:00
Ivan Andreev 985011e73b webdav: fix sharepoint-ntlm error 401 for parallel actions (#2921)
The go-ntlmssp NTLM negotiator has to try various authentication methods.
Intermediate responses from Sharepoint have status code 401, only the
final one is different. When rclone runs a large operation in parallel
goroutines according to --checkers or --transfers, one of threads can
receive intermediate 401 response targeted for another one and returns
the 401 authentication error to the user.
This patch fixes that.
2021-02-23 23:27:30 +00:00
Ivan Andreev 9ca6bf59c6 webdav: enforce encoding to fix errors with sharepoint-ntlm (#2921)
On-premises Sharepoint returns HTTP errors 400 or 500 in
reply to attempts to use file names with special characters
like hash, percent, tilde, invalid UTF-7 and so on.
This patch activates transparent encoding of such characters.
2021-02-23 23:27:30 +00:00
Ivan AndreevandGeorg Neugschwandtner ac6bb222f9 webdav: improve terminology in sharepoint-ntlm docs (#2921)
The most popular keyword for the Sharepoint in-house or company
installations is "On-Premises".
"Microsoft OneDrive account" is in fact just a Microsoft account.

Co-authored-by: Georg Neugschwandtner <georg.neugschwandtner@gmx.net>
2021-02-23 23:27:30 +00:00
Ivan AndreevandGitHub 5834020316 docker.bash: work correctly with multi-ip containers (#5028)
Currently if container under test has multiple IP addresses,
the `docker_ip` function from `docker.sh` will return a gibberish.
This patch makes it return the first address found.
Additionally, I apply shellcheck on `docker.sh`.
2021-02-17 03:38:02 +03:00
Ivan AndreevandGitHub f5066a09cd build: replace go 1.16-rc1 by 1.16.x (#5036) 2021-02-17 03:37:30 +03:00
Ivan Andreev 20578f3f89 Add Sakuragawa Misty to contributors 2021-01-29 23:05:49 +03:00
Ivan AndreevandGitHub 2bddba118e fstest: apply shellcheck on rclone-serve.bash (#4975) 2021-01-29 19:07:17 +03:00
Ivan AndreevandGitHub c7e5976e11 build: replace go 1.16-beta1 by 1.16-rc1 (#4974) 2021-01-29 19:05:46 +03:00
Ivan Andreev 35a4de2030 chunker: fix case-insensitive NewObject, test metadata detection #4902
- fix test case FsNewObjectCaseInsensitive (PR #4830)
- continue PR #4917, add comments in metadata detection code
- add warning about metadata detection in user documentation
- change metadata size limits, make room for future development
- hide critical chunker parameters from command line
2021-01-10 22:29:24 +03:00
Ivan Andreev 847625822f chunker: improve detection of incompatible metadata #4917
Before this patch chunker required that there is at least one
data chunk to start checking for a composite object.

Now if chunker finds at least one potential temporary or control
chunk, it marks found files as a suspected composite object.
When later rclone tries a concrete operation on the object,
it performs postponed metadata read and decides: is this a native
composite object, incompatible metadata or just garbage.
2021-01-10 21:55:15 +03:00
Ivan AndreevandNick Craig-Wood 768e4c4735 build: Temporary fix for Windows build errors
Applies a temporary fix similar to https://github.com/grafana/grafana/pull/28557
before go 1.15.6+ fixes https://github.com/golang/go/issues/40795
2021-01-07 09:48:51 +00:00
Ivan Andreev be6a888e50 chunker: skip long local hashing, hash in-transit (fixes #4021)
PR 4614
2020-10-26 20:18:07 +03:00
Ivan Andreev dad8447423 mailru: avoid prehashing of large local files
PR 4617
2020-10-26 20:16:52 +03:00
Ivan Andreev 65ff109065 mailru: accept special folders eg camera-upload
Fixes #4025
PR 4690
2020-10-26 20:04:31 +03:00
Ivan Andreev d8239e0194 mailru: remove deprecated protocol quirks 2020-09-26 15:38:32 +03:00
Ivan Andreev 004c3796de chunker: disable ListR to fix missing files on GDrive (workaround #3972) 2020-09-26 15:19:16 +03:00
Ivan Andreev 18c7549770 mailru: fix invalid timestamp on corrupted files (fixes #4229) 2020-09-26 15:12:30 +03:00
Ivan AndreevandGitHub c797494d88 Merge pull request #4608 from ivandeex/pr-chunker-crypt
chunker: fix upload over crypt (fixes #4570)
2020-09-18 17:58:44 +03:00
Ivan Andreev e2a57182be mailru: re-enable fixed chunker tests
This reverts commit 9d3d397f50.
2020-09-18 17:56:34 +03:00
Ivan Andreev 8928441466 mailru: fix range requests after june changes on server 2020-09-18 17:56:34 +03:00
Ivan Andreev 0e8965060f mailru: fix uploads after recent changes on server
similar fix: https://github.com/yar229/WebDavMailRuCloud/commit/5efa9958f1f8fae568908e3f547c2580f48d49dc
2020-09-18 17:56:34 +03:00
Ivan AndreevandNick Craig-Wood 41ba1bba2b chunker: reduce length of temporary suffix 2019-12-09 16:56:32 +00:00
Ivan Andreev 2e50543053 Add Ivan Andreev to maintainers 2019-10-20 00:33:16 +03:00
Ivan AndreevandNick Craig-Wood 77b42aa33a chunker: fix integration tests and hashsum issues 2019-10-13 10:43:46 +01:00
Ivan AndreevandNick Craig-Wood 910c80bd02 chunker: option to hash all files 2019-10-13 10:43:46 +01:00
Ivan AndreevandNick Craig-Wood 9049bb62ca chunker: prevent chunk corruption, survive meta-like input 2019-10-13 10:43:46 +01:00
Ivan AndreevandNick Craig-Wood 7aa2b4191c chunker: reservations for future extensions 2019-10-13 10:43:46 +01:00
Ivan AndreevandNick Craig-Wood ccecfa9cb1 chunker: finish meta-format before release
changes:
- chunker: remove GetTier and SetTier
- remove wdmrcompat metaformat
- remove fastopen strategy
- make hash_type option non-advanced
- adverise hash support when possible
- add metadata field "ver", run strict checks
- describe internal behavior in comments
- improve documentation

note:
wdmrcompat used to write file name in the metadata, so maximum metadata
size was 1K; removing it allows to cap size by 200 bytes now.
2019-09-25 11:03:33 +01:00
Ivan AndreevandNick Craig-Wood c41812fc88 tests: bring memory hungry tests close to end 2019-09-24 12:45:12 +01:00
Ivan AndreevandNick Craig-Wood d98d1be3fe accounting: fix panic due to server-side copy fallback 2019-09-24 12:45:12 +01:00
Ivan AndreevandNick Craig-Wood 661dc568f3 fstest: let backends advertise maximum file size 2019-09-24 12:45:12 +01:00
Ivan AndreevandNick Craig-Wood 1e4691f951 tests/sync: adjust transfer counts for chunker 2019-09-24 12:45:12 +01:00
Ivan AndreevandNick Craig-Wood be674faff1 tests/config: integration tests for chunker
Recommended `rclone.conf` snippets for this `config.yaml`:
```
[TestChunkerLocal]
type = chunker
meta_format = simplejson
remote = /tmp/rclone-chunker-test

[TestChunkerChunk3bLocal]
type = chunker
chunk_size = 3b
meta_format = simplejson
remote = /tmp/rclone-chunker-test

[TestChunkerNometaLocal]
type = chunker
meta_format = none
remote = /tmp/rclone-chunker-test

[TestChunkerChunk3bNometaLocal]
type = chunker
chunk_size = 3b
meta_format = none
remote = /tmp/rclone-chunker-test

[TestChunkerCompatLocal]
type = chunker
meta_format = wdmrcompat
remote = /tmp/rclone-chunker-test
```
2019-09-24 12:45:12 +01:00
Ivan AndreevandNick Craig-Wood c68c919cea docs: chunker documentation 2019-09-24 12:45:12 +01:00
Ivan AndreevandNick Craig-Wood 59dba1de88 chunker: implementation + required fstest patch
Note: chunker implements many irrelevant methods (UserInfo, Disconnect etc),
but they are required by TestIntegration/FsCheckWrap and cannot be removed.

Dropped API methods: MergeDirs DirCacheFlush PublicLink UserInfo Disconnect OpenWriterAt

Meta formats:
- renamed old simplejson format to wdmrcompat.
- new simplejson format supports hash sums and verification of chunk size/count.

Change list:
- split-chunking overlay for mailru
- add to all
- fix linter errors
- fix integration tests
- support chunks without meta object
- fix package paths
- propagate context
- fix formatting
- implement new required wrapper interfaces
- also test large file uploads
- simplify options
- user friendly name pattern
- set default chunk size 2G
- fix building with golang 1.9
- fix ci/cd on a separate branch
- fix updated object name (SyncUTFNorm failed)
- fix panic in Box overlay
- workaround: Box rename failed if name taken
- enhance comments in unit test
- fix formatting
- embed wrapped remote rather than inherit
- require wrapped remote to support move (or copy)
- implement 3 (keep fstest)
- drop irrelevant file system interfaces
- factor out Object.mainChunk
- refactor TestLargeUpload as InternalTest
- add unit test for chunk name formats
- new improved simplejson meta format
- tricky case in test FsIsFile (fix+ignore)
- remove debugging print
- hide temporary objects from listings
- fix bugs in chunking reader:
  - return EOF immediately when all data is sent
  - handle case when wrapped remote puts by hash (bug detected by TestRcat)
- chunked file hashing (feature)
- server-side copy across configs (feature)
- robust cleanup of temporary chunks in Put
- linear download strategy (no read-ahead, feature)
- fix unexpected EOF in the box multipart uploader
- throw error if destination ignores data
2019-09-24 12:45:12 +01:00
Ivan AndreevandNick Craig-Wood 8fe87c8157 mailru: skip extra http request if data fits in hash 2019-09-17 10:04:51 +01:00
Ivan AndreevandNick Craig-Wood 8fb44a822d mailru: fix rare nil pointer panic 2019-09-17 10:04:51 +01:00
Ivan AndreevandNick Craig-Wood ba1daea072 mailru: backend for mail.ru 2019-09-09 21:56:16 +01:00
Ivan AndreevandNick Craig-Wood bdcd0b4c64 Add mailru hash (mrhash) 2019-09-09 21:34:15 +01:00
Ivan AndreevandNick Craig-Wood d0c6e5cf5a vfs: skip TestCaseSensitivity on case insensitive backends 2019-09-06 10:44:59 +01:00
Ivan AndreevandNick Craig-Wood 1c4e33d4ad vfs: add flag --vfs-case-insensitive for windows/macOS mounts
rclone mount when run on Windows & macOS will now default to `--vfs-case-insensitive`.
This means that
2019-09-04 21:30:48 +01:00