* Rewrite the backend manager.
This creates a new more logic backend manager that handles all backend operations.
For each top-level operation, there is now a single backend manager instance that is passed to sub-commands.
The internals of the backend manager are now rewritten to use async logic instead of threading.
The design uses a single task runner that dispatches operations and honors concurrent settings and limits from one place.
The logic for each operation has been moved into a separate files/classes to make it easier to understand each operation.
This fixes#5804
* Fixed a few issues with not awaiting tasks
* Fixed not creating empty databases
* Fixed an issue with hashes not being recorded on download
* Reworked the download logic to avoid attempting to decrypt the file if it has been modified.
* Review fixes
* Renamed db collector to better reflect the purpose.
* Review fixes
* Simplified task control by having all logic in a single class and using async control mechanism
* Fixed the tests
* Made the marking of a backup partial explicitly communicated if the enumeration process is stopped.
* Added support for pausing active transfers
* Introduced transfer token to abort operations.
* Updated the way LiveControl is emitting events.
* Retains the paused state of the server across reboots, this fixes#5760.
* Reworked the way throttle speeds are propagated through LiveControls.
* Added option to pause transfers in the UI.
* Removed throttle settings from the LiveControls class so it is exclusively handled by the ApplicationSettings.
* Removed `thread-priority` as it does not work at all when tasks are used.
* Fixed the stop dialog and API to only support stop and abort.
* Removed unused variables
The LocalDatabase class was deleting from the BlocklistHash table based on the volume id of blocks. The deletes for the Blockset and BlocksetEntry tables were using a union query which included the same query the BlocklistHash delete was using. Changed the BlocklistHash delete query to use the union as well to ensure that all three tables would use the same query to delete the necessary rows. This fixes the "Detected non-empty blocksets with no associated blocks!" error.
Moved a write call in the DataBlockProcessor to send a volume for upload after setting blockvolume to null. This prevents a null reference exception because otherwise it would be sent to the SpillCollector and it tries to add a block, but the volume has already been closed.
Added the ? operator to the RegistryUtility class to prevent a null reference exception while debugging. It was annoying catching it in the debugger.
Fixes#4037
When support for parallel uploads was implemented, we overlooked a case
where a failed put of a dblock file is retried with a different
filename. In this case, an instance of TemporaryIndexVolume was created
in the DataBlockProcessor or SpillCollectorProcess prior to the
attempted put. The put would fail, and then retried with a different
filename. While the Remotevolume table in the database would reflect
the new filename, the TemporaryIndexVolume still contained a reference
to the old dblock filename. This would cause issues in direct restores,
etc., when the referenced dblock file could not be found.
Now, the TemporaryIndexVolume is only created after the dblock put has
completed.
This addresses issue #3932.
As part of the parallel upload changes a bug slipped in where a check was
being made if the index volume writer exists before creating said index
volume writer. Fixed by changing the test to check if the temporary index
volume exists and then creating the writer for the index file.
- Rename indexVolume to indexVolumeWriter in DataBlockProcessor to reduce
confusion.
- When flushing the remaining uploads, check the tasks for exceptions in
completion order rather than waiting for them all at once. If there is an
exception all other uploads are cancelled anyway.
- Remove setting the operation progress as Backup_WaitForUpload in the
BackendUploader since the BackupHandler sets it.
- Use an exception filter in BackendUploader.
- Use Options property MaxUploadPrSecond instead of getting the raw value.
Encrypting, hashing, and creating index volumes are now done
in the DataBlockProcessor and the SpillCollectorProcess. This allows
uploads to always be transferring data and not have to stop to create
a file, encrypt data, etc.
Changed a few things on the way, removing the log process
# Conflicts:
# Duplicati/Library/Main/Controller.cs
# Duplicati/Library/Main/Database/ExtensionMethods.cs
# Duplicati/Library/Main/Database/LocalBackupDatabase.cs
# Duplicati/Library/Main/Duplicati.Library.Main.csproj
# Duplicati/Library/Main/Operation/BackupHandler.cs
# Duplicati/Library/Main/Operation/FilelistProcessor.cs
# Duplicati/Library/Main/Operation/PurgeBrokenFilesHandler.cs
# Duplicati/Library/Main/Operation/TestFilterHandler.cs
# Duplicati/Library/Main/Options.cs
# Duplicati/Library/Main/ResultClasses.cs
# Duplicati/UnitTest/BasicSetupHelper.cs
Previously, empty files were associasted with empty blocks. These blocks were added to volumes which could result in empty, but valid volumes being garbage collected and valid file entries deleted.
This change changes the logic so that empty blocks are not stored at all.
