4722b94d1a
The sftp backend could be used as the source of a multi-thread copy - each chunk of a download is read on its own connection - but not as the destination, because it did not implement OpenWriterAt. Uploading a single large file was therefore limited to one connection while downloading the same file was not, capping upload throughput well below the link speed on high latency connections. This implements OpenWriterAt for the sftp backend using lib/filepool. A small pool of open write handles, each backed by its own connection, lets the core write the chunks of a large file concurrently over several connections. The file is created and truncated once up front so every chunk offset is valid before the concurrent writes start. Multi-thread uploads are off by default and turned on with the new --sftp-multithread-upload flag, since many sftp servers only accept sequential writes and would fail large uploads otherwise. Even when enabled they fall back to a single connection when --sftp-disable-concurrent-writes is set (a server that can't take out-of-order packets on one handle won't take several handles either) or when --sftp-connections caps the pool (the per-file fan-out would otherwise deadlock waiting on it). The OpenSSH test server (TestSFTPOpenssh) enables --sftp-multithread-upload so the feature is exercised in CI against a real server. Tested against OpenSSH: a single large upload over a high latency link went from ~12 MB/s to ~90 MB/s. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Splainte <r.wycke@hotmail.fr>
30 lines
649 B
Bash
Executable File
30 lines
649 B
Bash
Executable File
#!/usr/bin/env bash
|
|
|
|
set -e
|
|
|
|
NAME=rclone-sftp-openssh
|
|
USER=rclone
|
|
PASS=password
|
|
PORT=28627
|
|
|
|
. $(dirname "$0")/docker.bash
|
|
|
|
start() {
|
|
docker run --rm -d --name ${NAME} \
|
|
-p 127.0.0.1:${PORT}:22 \
|
|
rclone/test-sftp-openssh
|
|
|
|
echo type=sftp
|
|
echo host=127.0.0.1
|
|
echo port=$PORT
|
|
echo user=$USER
|
|
echo pass=$(rclone obscure $PASS)
|
|
echo copy_is_hardlink=true
|
|
# OpenSSH supports concurrent writes at arbitrary offsets, so exercise
|
|
# multi-thread uploads here (off by default) against a real server.
|
|
echo multithread_upload=true
|
|
echo _connect=127.0.0.1:${PORT}
|
|
}
|
|
|
|
. $(dirname "$0")/run.bash
|