compress: fix corrupted objects being created when the source ends early

When the source supplied fewer bytes than its declared size, the
compressed data file was stored under a name containing the declared
size while the metadata recorded the actual number of bytes read.
NewObject looks the data file up by the size in the metadata, so the
resulting object could never be read again, and the upload reported
success.

Check that the number of bytes read matches the declared size after
uploading the data and before writing the metadata, and remove the
data file and return an error if it does not.

This was found by the new FsPutShortEOF integration test.
This commit is contained in:
Nick Craig-Wood
2026-08-03 20:58:14 +01:00
parent e0701daea0
commit 18fa445ffc
+10
View File
@@ -698,6 +698,16 @@ func (f *Fs) putWithCustomFunctions(ctx context.Context, in io.Reader, src fs.Ob
return nil, err
}
// Check the source supplied the number of bytes it declared -
// compressed data is stored under a name containing the
// original size so a mismatch makes the object unreadable.
if size := src.Size(); size >= 0 && meta.Size != size {
if removeErr := dataObject.Remove(ctx); removeErr != nil {
fs.Errorf(dataObject, "Failed to remove partially transferred object: %v", removeErr)
}
return nil, fmt.Errorf("expected %d bytes in input, but got %d: %w", size, meta.Size, io.ErrUnexpectedEOF)
}
mo, err := f.putMetadata(ctx, meta, src, options, putMeta)
// meta data upload may fail. in this case we try to remove the original object