Make checking and uploading run in parallel using channels

Runs much faster and uses much less memory (thanks to walking the
filesystem into a channel too).
This commit is contained in:
Nick Craig-Wood
2012-11-29 22:13:58 +00:00
parent ba24b84396
commit 4d23e29274
2 changed files with 126 additions and 59 deletions
+16 -1
View File
@@ -16,6 +16,10 @@ Need an iterate all objects routine... Could use a channel
FIXME progress meter would be nice! Do this by wrapping the Reader with a progress bar
Do bandwidth limit by wrapping the Reader too
Maybe using https://jra-go.googlecode.com/hg/linkio/ which will work for multiple
uploads or downloads.
* code.google.com/p/mxk/go1/flowcontrol - only does one flow at once
Or maybe put into swift library.
Could have an integrity check mode where we check the MD5sums of the local vs the remote
@@ -23,4 +27,15 @@ Some stats would be nice!
Windows paths? Do we need to translate / and \?
Make swift timeouts be settable
Make swift timeouts be settable with command line parameters
Add bandwidth limit?
Make a wrapper in connection which
* measures bandwidth and reports it
* limits bandwidth using Reader and Writer
* for a pool of all individual connectinos
* does timeouts by setting a limit, seeing whether io has happened
and resetting it if it has
Check the locking in swift module!