If parallel uploads are enabled it is possible for the fileset to be uploaded before all dblocks are uploaded.
When the fileset is placed in the queue to be uploaded place it in a separate list and only upload it after all dblocks have been uploaded.
Fixes#4341
When support for parallel uploads was implemented, we overlooked a case
where a failed put of a dblock file is retried with a different
filename. In this case, an instance of TemporaryIndexVolume was created
in the DataBlockProcessor or SpillCollectorProcess prior to the
attempted put. The put would fail, and then retried with a different
filename. While the Remotevolume table in the database would reflect
the new filename, the TemporaryIndexVolume still contained a reference
to the old dblock filename. This would cause issues in direct restores,
etc., when the referenced dblock file could not be found.
Now, the TemporaryIndexVolume is only created after the dblock put has
completed.
This addresses issue #3932.
The only code in the finally block was to set the result of the
FlushRequest if the completed task was not faulted. This could be done
without the try-finally.
The FlushBackend method uses Task.WhenAny to await the first completion
of the uploader task or the flush request. Since the uploader was
setting the status of the flush request in the finally block, the flush
request would complete first when an exception was thrown, causing the
uploader's exception to be unobserved. To resolve this, we check that
the uploader's status is not faulted before setting the flush request
status.
This addresses issue #3673.
- Rename indexVolume to indexVolumeWriter in DataBlockProcessor to reduce
confusion.
- When flushing the remaining uploads, check the tasks for exceptions in
completion order rather than waiting for them all at once. If there is an
exception all other uploads are cancelled anyway.
- Remove setting the operation progress as Backup_WaitForUpload in the
BackendUploader since the BackupHandler sets it.
- Use an exception filter in BackendUploader.
- Use Options property MaxUploadPrSecond instead of getting the raw value.
If a backend throws an exception while uploading, cancel other uploads and
throw an exception from the BackendUploader. Exceptions from cancelled
uploads are caught and not thrown so as to not spam the user with useless
information.
Changed AddBackendEvent() in ResultClasses to allow the option to not update
the backend progress. This allows the BackendUploader to log the act of starting
an upload without changing the backend progress. The backend progress is now
handled by a new class, FileProgressThrottler, which will determine the transfer
rate of all uploads currently in progress.
This class also handles throttling the upload streams. The algorithm will try
to decrease or increase the streams throttle rates as appropriate by changing
the value based on the over/under amount or based on a percentage to prevent
throttling one stream directly to zero.
Encrypting, hashing, and creating index volumes are now done
in the DataBlockProcessor and the SpillCollectorProcess. This allows
uploads to always be transferring data and not have to stop to create
a file, encrypt data, etc.
Kenneth Skovhede