When support for parallel uploads was implemented, we overlooked a case
where a failed put of a dblock file is retried with a different
filename. In this case, an instance of TemporaryIndexVolume was created
in the DataBlockProcessor or SpillCollectorProcess prior to the
attempted put. The put would fail, and then retried with a different
filename. While the Remotevolume table in the database would reflect
the new filename, the TemporaryIndexVolume still contained a reference
to the old dblock filename. This would cause issues in direct restores,
etc., when the referenced dblock file could not be found.
Now, the TemporaryIndexVolume is only created after the dblock put has
completed.
This addresses issue #3932.
As part of the parallel upload changes a bug slipped in where a check was
being made if the index volume writer exists before creating said index
volume writer. Fixed by changing the test to check if the temporary index
volume exists and then creating the writer for the index file.
- Rename indexVolume to indexVolumeWriter in DataBlockProcessor to reduce
confusion.
- When flushing the remaining uploads, check the tasks for exceptions in
completion order rather than waiting for them all at once. If there is an
exception all other uploads are cancelled anyway.
- Remove setting the operation progress as Backup_WaitForUpload in the
BackendUploader since the BackupHandler sets it.
- Use an exception filter in BackendUploader.
- Use Options property MaxUploadPrSecond instead of getting the raw value.
Encrypting, hashing, and creating index volumes are now done
in the DataBlockProcessor and the SpillCollectorProcess. This allows
uploads to always be transferring data and not have to stop to create
a file, encrypt data, etc.
Changed a few things on the way, removing the log process
# Conflicts:
# Duplicati/Library/Main/Controller.cs
# Duplicati/Library/Main/Database/ExtensionMethods.cs
# Duplicati/Library/Main/Database/LocalBackupDatabase.cs
# Duplicati/Library/Main/Duplicati.Library.Main.csproj
# Duplicati/Library/Main/Operation/BackupHandler.cs
# Duplicati/Library/Main/Operation/FilelistProcessor.cs
# Duplicati/Library/Main/Operation/PurgeBrokenFilesHandler.cs
# Duplicati/Library/Main/Operation/TestFilterHandler.cs
# Duplicati/Library/Main/Options.cs
# Duplicati/Library/Main/ResultClasses.cs
# Duplicati/UnitTest/BasicSetupHelper.cs
Previously, empty files were associasted with empty blocks. These blocks were added to volumes which could result in empty, but valid volumes being garbage collected and valid file entries deleted.
This change changes the logic so that empty blocks are not stored at all.
Kenneth Hsu