- Rename indexVolume to indexVolumeWriter in DataBlockProcessor to reduce
confusion.
- When flushing the remaining uploads, check the tasks for exceptions in
completion order rather than waiting for them all at once. If there is an
exception all other uploads are cancelled anyway.
- Remove setting the operation progress as Backup_WaitForUpload in the
BackendUploader since the BackupHandler sets it.
- Use an exception filter in BackendUploader.
- Use Options property MaxUploadPrSecond instead of getting the raw value.
If a backend throws an exception while uploading, cancel other uploads and
throw an exception from the BackendUploader. Exceptions from cancelled
uploads are caught and not thrown so as to not spam the user with useless
information.
Changed AddBackendEvent() in ResultClasses to allow the option to not update
the backend progress. This allows the BackendUploader to log the act of starting
an upload without changing the backend progress. The backend progress is now
handled by a new class, FileProgressThrottler, which will determine the transfer
rate of all uploads currently in progress.
This class also handles throttling the upload streams. The algorithm will try
to decrease or increase the streams throttle rates as appropriate by changing
the value based on the over/under amount or based on a percentage to prevent
throttling one stream directly to zero.
Encrypting, hashing, and creating index volumes are now done
in the DataBlockProcessor and the SpillCollectorProcess. This allows
uploads to always be transferring data and not have to stop to create
a file, encrypt data, etc.
This reverts a change made in revision 31ac40083a ("Prevent temp files
in VolumeWriterBase from being deleted prematurely" in pull request #2637).
It's unclear what issue was being worked around by preventing the
removal of the temporary files (based on comments in pull request #2637,
it appears to be related to tests that failed only on Travis). However,
this workaround prevented the removal of temporary files created during
compact operations, which caused several users' temporary directories to
be filled up with files that should have been removed.
This concerns issue #3652.
This ensures the removal of the temporary file containing the
filelist.json that remained regardless if files needed to be uploaded or
not.
This concerns issue #3652.
There is only one usage of this method that did not provide value for
the filter parameter. By making this parameter required, we can avoid
the many issues that arise with optional parameters (default values
being embedded at the call sites, ambiguous overload resolution, etc.).
This will hopefully make it clearer that the files being selected for
verification are those on the remote backend, and not user files that
are being backed up.
Kenneth Hsu