According to the current implementation, the progress bar does not advance
when files are skipped by FilePreFilterProcess. This causes users'
confusion since the web frontend will get stuck at a certain stage for a
long time. This commit addresses this issue.
When support for parallel uploads was implemented, we overlooked a case
where a failed put of a dblock file is retried with a different
filename. In this case, an instance of TemporaryIndexVolume was created
in the DataBlockProcessor or SpillCollectorProcess prior to the
attempted put. The put would fail, and then retried with a different
filename. While the Remotevolume table in the database would reflect
the new filename, the TemporaryIndexVolume still contained a reference
to the old dblock filename. This would cause issues in direct restores,
etc., when the referenced dblock file could not be found.
Now, the TemporaryIndexVolume is only created after the dblock put has
completed.
This addresses issue #3932.
This fixes a regression introduced around revisions e0899a2a91 ("add
method to add the filelist file") and 3b5af1cf01 ("add method to add
filelist file"), where the filelist was not written by all usages of the
FilesetVolumeWriter. Previously, one would have to invoke
AddFilelistFile with each usage. Now, we simply do so when the
FilesetVolumeWriter is closed.
We also modified the UploadSyntheticFilelist so that the usage of the
FilesetVolumeWriter is contained in a using statement, which will ensure
that it is disposed.
This fixes issue #3924.
- add backup state to DB table 'fileset' (job database upgrade to version 10)
- modify the Restore page dropdown to display if a backup is "partial"
- modify retention logic to remove partial backups only when the next recent full-backup has been removed
The only code in the finally block was to set the result of the
FlushRequest if the completed task was not faulted. This could be done
without the try-finally.
The FlushBackend method uses Task.WhenAny to await the first completion
of the uploader task or the flush request. Since the uploader was
setting the status of the flush request in the finally block, the flush
request would complete first when an exception was thrown, causing the
uploader's exception to be unobserved. To resolve this, we check that
the uploader's status is not faulted before setting the flush request
status.
This addresses issue #3673.
As part of the parallel upload changes a bug slipped in where a check was
being made if the index volume writer exists before creating said index
volume writer. Fixed by changing the test to check if the temporary index
volume exists and then creating the writer for the index file.
- Rename indexVolume to indexVolumeWriter in DataBlockProcessor to reduce
confusion.
- When flushing the remaining uploads, check the tasks for exceptions in
completion order rather than waiting for them all at once. If there is an
exception all other uploads are cancelled anyway.
- Remove setting the operation progress as Backup_WaitForUpload in the
BackendUploader since the BackupHandler sets it.
- Use an exception filter in BackendUploader.
- Use Options property MaxUploadPrSecond instead of getting the raw value.
If a backend throws an exception while uploading, cancel other uploads and
throw an exception from the BackendUploader. Exceptions from cancelled
uploads are caught and not thrown so as to not spam the user with useless
information.
Changed AddBackendEvent() in ResultClasses to allow the option to not update
the backend progress. This allows the BackendUploader to log the act of starting
an upload without changing the backend progress. The backend progress is now
handled by a new class, FileProgressThrottler, which will determine the transfer
rate of all uploads currently in progress.
This class also handles throttling the upload streams. The algorithm will try
to decrease or increase the streams throttle rates as appropriate by changing
the value based on the over/under amount or based on a percentage to prevent
throttling one stream directly to zero.
Encrypting, hashing, and creating index volumes are now done
in the DataBlockProcessor and the SpillCollectorProcess. This allows
uploads to always be transferring data and not have to stop to create
a file, encrypt data, etc.
Fish