compress: fix corrupted objects being created when the source ends early
When the source supplied fewer bytes than its declared size, the compressed data file was stored under a name containing the declared size while the metadata recorded the actual number of bytes read. NewObject looks the data file up by the size in the metadata, so the resulting object could never be read again, and the upload reported success. Check that the number of bytes read matches the declared size after uploading the data and before writing the metadata, and remove the data file and return an error if it does not. This was found by the new FsPutShortEOF integration test.
This commit is contained in:
@@ -698,6 +698,16 @@ func (f *Fs) putWithCustomFunctions(ctx context.Context, in io.Reader, src fs.Ob
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Check the source supplied the number of bytes it declared -
|
||||
// compressed data is stored under a name containing the
|
||||
// original size so a mismatch makes the object unreadable.
|
||||
if size := src.Size(); size >= 0 && meta.Size != size {
|
||||
if removeErr := dataObject.Remove(ctx); removeErr != nil {
|
||||
fs.Errorf(dataObject, "Failed to remove partially transferred object: %v", removeErr)
|
||||
}
|
||||
return nil, fmt.Errorf("expected %d bytes in input, but got %d: %w", size, meta.Size, io.ErrUnexpectedEOF)
|
||||
}
|
||||
|
||||
mo, err := f.putMetadata(ctx, meta, src, options, putMeta)
|
||||
|
||||
// meta data upload may fail. in this case we try to remove the original object
|
||||
|
||||
Reference in New Issue
Block a user