Factor the connection-backed write handle pool out of the smb backend
into a generic lib/filepool.Pool[T] with its own tests, so it can be
reused by other backends that implement fs.OpenWriterAter over a
connection pool.
The smb backend keeps its behaviour, opening and releasing handles
through small closures passed to the pool.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Splainte <r.wycke@hotmail.fr>
Object.Update held its connection until the deferred putConnection ran at
function exit, so the SetModTime it does at the end of every upload had to take
a second connection from the pool, dialling a whole new SMB session when the
pool was empty. With N transfers in flight the pool grew to roughly 2N sessions
for no reason.
Return the connection as soon as the file is closed. At that point the upload
has succeeded and remove() can no longer be reached, so nothing else needs it,
and SetModTime picks the same connection straight back out of the pool.
putConnection nils the pointer, so the deferred putConnection becomes a no-op
and the connection is not returned twice.
DirMove returned fs.ErrorDirExists for any error from the destination
existence check not just when the destination actually exists. That
made sync silently fall back to file-by-file moves and masked the real
failure.
Return the underlying error instead when the check fails for any other
reason.
DirMove checked whether the destination exists using the raw path but
performed the rename with the encoded path, so for directory names
needing encoding (trailing spaces or periods, characters like
\* ? : < > | " or a literal backslash) the existence check looked at
a different server path than the rename used.
If revealing the password, creating the Kerberos client or the SMB
handshake failed after the TCP connection was established, the
connection was never closed.
A new KerberosFactory was constructed for every dial, so the client,
error and ccache modification time caches it holds were discarded
after a single use. Every new SMB connection re-read the Kerberos
config, re-parsed the ccache and did a fresh KDC exchange.
Share a single factory so clients are cached across connections as
intended, and refreshed when the ccache file changes.
In the current design, OpenWriterAt provides the interface for random-access
writes, and openChunkWriterFromOpenWriterAt wraps this interface to enable
parallel chunk uploads using multiple goroutines. A global connection pool is
already in place to manage SMB connections across files.
However, currently only one connection is used per file, which makes multiple
goroutines compete for the connection during multithreaded writes.
This changes create separate connections for each goroutine, which allows true
parallelism by giving each goroutine its own SMB connection
Signed-off-by: sudipto baral <sudiptobaral.me@gmail.com>
Before this fix it was possible for an about call in various backends
to exceed an int64 and wrap.
This patch causes it to clip to the max int64 value instead.
This change enhances the SMB backend in Rclone to automatically refresh
Kerberos credentials when the associated ccache file is updated.
Previously, credentials were only loaded once per path and cached
indefinitely, which caused issues when service tickets expired or the
cache was renewed on the server.
* Lower pacer minSleep to establish new connections faster
* Use Echo requests to check whether connections are working (required an upgrade of go-smb2)
* Only remount shares when needed
* Use context for connection establishment
* When returning a connection to the pool, only check the ones that encountered errors
* Close connections in parallel
Before this fix the smb backend could panic if a stat call failed.
This fix makes it return an error instead.
It should have the side effect that we do one less stat call on upload
too.
Fixes#8106
Before this change the IP address of the server was used in the SMB
connect request (see CloudSoda/go-smb2#18).
The updated library now can pass the hostname instead.
The update requires a small change in the dial method call.
Fixes rclone#6672
Before this change PartialUploads was not set. This is clearly wrong
since incoming files are visible on the smb server.
Setting PartialUploads fixes the multithread upload modtime problem as
it uses the PartialUploads flag as an indication that it needs to set
the modtime explicitly.
This problem was detected by the new TestMultithreadCopy integration
tests
Fixes#7411
Before this change smb drives sometimes showed a fraction of the
correct size using `rclone about`.
This fixes the problem by switching the upstream library from
github.com/hirochachacha/go-smb2 to github.com/cloudsoda/go-smb2 which
has a fix for the problem.
The new library passes the integration tests.
Fixes#6733
smb2.File implements the WriterAtCloser interface defined in
fs/types.go. Expose it via a OpenWriterAt method on
the fs struct to support multi-threaded writes.
Before this change, if you mounted the root of the smb then it would
give an error on rclone about and periodically in the mount logs:
Statfs failed: bucket or container name is needed in remote
This fix makes the smb backend return empty usage in this case which
will stop the errors and show the default 1P of free space.
See: https://forum.rclone.org/t/error-statfs-failed-bucket-or-container-name-is-needed-in-remote/39631
This introduces a new fs.Option flag, Sensitive and uses this along
with IsPassword to redact the info in the config file for support
purposes.
It adds this flag into backends where appropriate. It was necessary to
add oauthutil.SharedOptions to some backends as they were missing
them.
Fixes#5209
Before this change we were putting connections into the connection
pool which had a local context in.
This meant that when the operation had finished the context was
cancelled and the connection became unusable.
See: https://forum.rclone.org/t/failed-to-sync-context-canceled/34017/
alliasgherandNick Craig-Wood
Loi NguyenandGitHub
Sudipto BaralandGitHub
Jonathan GiannuzziandNick Craig-Wood
halmsandNick Craig-Wood
Roberto RicciandNick Craig-Wood
Edwin Mackenzie-OwenandGitHub
albertony
happyxhwandGitHub
LesmiscoreandGitHub