yandex: add --yandex-upload-wait to fix 500 errors when uploading

In this commit we attempted to wait for the success report of an
upload to fix the 500 error:

fe78b559d1 yandex: fix 500 errors by waiting for uploads to complete before setting modtime

However Yandex Disk finalizes an upload asynchronously on its servers.
Waiting for the upload operation to report success is not enough -
under load the server reports the operation as successful slightly
before the file is fully finalized, so setting the modification time
straight after an upload can still fail with 500 Internal Server
Error.

Yandex support recommend waiting 1.5s - 3s after the upload before
modifying the file's metadata, so add an --yandex-upload-wait option
(default off) to insert a delay between the upload completing and the
modification time being set.

backend/yandex/yandex.go

@@ -85,6 +85,21 @@ func init() {
 			Default:  true,
 			Advanced: true,
 			Hide:     fs.OptionHideConfigurator,
+		}, {
+			Name: "upload_wait",
+			Help: `Wait this long after an upload before setting the modification time.
+
+Yandex Disk finalizes an upload asynchronously on its servers after
+the upload has completed. If the modification time is set while this
+finalization is still in progress the server returns 500 Internal
+Server Error errors.
+
+If you are getting 500 errors on upload then setting this to 2s is
+normally enough to stop them, at the cost of slowing down uploads.
+
+Yandex support recommend a value of 1.5s - 3s.`,
+			Default:  fs.Duration(0),
+			Advanced: true,
 		}}...),
 	})
 }
@@ -95,6 +110,7 @@ type Options struct {
 	HardDelete     bool                 `config:"hard_delete"`
 	Enc            encoder.MultiEncoder `config:"encoding"`
 	SpoofUserAgent bool                 `config:"spoof_ua"`
+	UploadWait     fs.Duration          `config:"upload_wait"`
 }
 
 // Fs represents a remote yandex
@@ -1131,9 +1147,24 @@ func (o *Object) upload(ctx context.Context, in io.Reader, overwrite bool, mimeT
 	// Wait for PUT to be committed
 	if ur.OperationID != "" {
 		err = o.fs.waitForJob(ctx, rootURL+"/operations/"+ur.OperationID)
+		if err != nil {
+			return err
+		}
 	}
 
-	return err
+	// Wait for the server to finalize the upload before the file's
+	// metadata is accessed. The operation status above can report
+	// success before the finalization has completed, so give the
+	// server some extra time if configured.
+	if o.fs.opt.UploadWait > 0 {
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+		case <-time.After(time.Duration(o.fs.opt.UploadWait)):
+		}
+	}
+
+	return nil
 }
 
 // Update the already existing object

docs/content/yandex.md

@@ -273,6 +273,14 @@ to twice the max size of file in GiB should be enough, so if you want
 to upload a 30 GiB file set a timeout of `2 * 30 = 60m`, that is
 `--timeout 60m`.
 
+If you get `500 Internal Server Error` errors just after uploads,
+particularly when uploading many files in parallel, then try setting
+`--yandex-upload-wait 2s`. Yandex Disk finalizes uploads
+asynchronously on its servers and can report an upload as complete
+slightly before the file is ready, so accessing the file's metadata
+too soon causes these errors. Yandex support recommend waiting
+1.5s - 3s after each upload.
+
 Having a Yandex Mail account is mandatory to use the Yandex.Disk subscription.
 Token generation will work without a mail account, but Rclone won't be able to
 complete any actions.