When creating a new controller, the options dictionary was used by-reference.
This meant that code could update the options dictionary *after* creating the controller, which would not be thread-safe.
It does not have much effect on regular operations as the normal method is to create the dictionary, pass it, and not modify it.
But in the tests, we frequently re-use the dictionary and update it with new values to test various cases.
Due to the dictionary being a reference, the controller would update the dictionary to fit the operations, but this would reflect back into the tests, meaning that some tests were being run with options that were not supposed to be there.
This meant that at least two edge cases that we test for were never found to have errors, even though they do.
The first issue is the listing of files using a non-default blocksize. The second issue is recreating the database with a non-default file-hash algorithm.
This PR removes the reference passing, and fixes the issues that were uncovered.
This PR has a large blast radius because it takes the final step and bumps up the Controller to be fully async.
We have historically done a piece-by-piece update, so all operations were already async but the controller interface was kept synchronous.
With this update, the controller is now fully async and all tests are updated.
Most places where the new C# compiler warns about function names not ending in `Async` were also adressed, giving a massive refactor change.
Functionally, no changes are done.
This PR adds support for locking files if the backend supports it.
To activate locking, set the option `--file-lock-duration=30D` and the backup will lock the files.
If the database is rebuilt with the intention of continuing the backups, use the option `--repair-refresh-lock-info` which will update lock information in the database after recreating the database.
The locking works by asking the backend to lock files after a backup has completed.
The implementation keeps track of which files are currently assigned a lock and prevents attempting to delete the files that are currently locked.
Note that the bucket should not have a default lock policy as Duplicati needs to finish the backup before the locking is applied.
In this initial version, Azure Blob Storage, B2, S3 and iDrive are supported with locking.
The CLI is updated to allow setting locks on a specific version. The backend tool is updated to allow setting locks on specific files.
This PR adds the option to disable throttle on a backup, so it will ignore throttle settings both from the server and the job.
Additionally, a list of backend keys can be provided, where backups will disable throttling. This can be used to set advanced options with a set of backends that should not be throttled.
By default, the `file` backend is now exempt from throttling by default.
This fixes#2685
This ensures that files cannot appear on the remote destination, despite terminating the host program.
This fixes#4485
There is still work needed to handle the repair and compact operations so they use transactions and can commit changes during transfer progress.
This moves the throttle support to the StreamUtil throttled stream, which allows multiple streams to share a throttle value, such that the throttle setting is applied to the entire communication from Duplicati, and not for each connection.
This also cleans up a bit in the progress reporting, where the backend manager now keeps track of the "active" transfer and only emits progress on that. Once it finishes, a new transfer is promoted a "active".
While investigating issue [#6103](https://github.com/duplicati/duplicati/issues/6103), found that connections were not disposed due to the backend pool in BackendManager's handler. Set up a FileZilla server to test the issue before and after the change, confirming that connections now close as expected with this fix.
This commit adds extra logic to prevent cases where the backup reports success but leaves behind partial files.
A new flag is recorded to track if partial files are possible. If no partial files are possible, the verification will fail if any are found.
* Rewrite the backend manager.
This creates a new more logic backend manager that handles all backend operations.
For each top-level operation, there is now a single backend manager instance that is passed to sub-commands.
The internals of the backend manager are now rewritten to use async logic instead of threading.
The design uses a single task runner that dispatches operations and honors concurrent settings and limits from one place.
The logic for each operation has been moved into a separate files/classes to make it easier to understand each operation.
This fixes#5804
* Fixed a few issues with not awaiting tasks
* Fixed not creating empty databases
* Fixed an issue with hashes not being recorded on download
* Reworked the download logic to avoid attempting to decrypt the file if it has been modified.
* Review fixes
* Renamed db collector to better reflect the purpose.
* Review fixes
Kenneth Skovhede