When creating a new controller, the options dictionary was used by-reference.
This meant that code could update the options dictionary *after* creating the controller, which would not be thread-safe.
It does not have much effect on regular operations as the normal method is to create the dictionary, pass it, and not modify it.
But in the tests, we frequently re-use the dictionary and update it with new values to test various cases.
Due to the dictionary being a reference, the controller would update the dictionary to fit the operations, but this would reflect back into the tests, meaning that some tests were being run with options that were not supposed to be there.
This meant that at least two edge cases that we test for were never found to have errors, even though they do.
The first issue is the listing of files using a non-default blocksize. The second issue is recreating the database with a non-default file-hash algorithm.
This PR removes the reference passing, and fixes the issues that were uncovered.
This PR has a large blast radius because it takes the final step and bumps up the Controller to be fully async.
We have historically done a piece-by-piece update, so all operations were already async but the controller interface was kept synchronous.
With this update, the controller is now fully async and all tests are updated.
Most places where the new C# compiler warns about function names not ending in `Async` were also adressed, giving a massive refactor change.
Functionally, no changes are done.
This PR adds support for using the `quota-disable` option to diable the quota check performed before executing the remote sync operation.
This is needed if the quota information returned is unreliable.
If the delete call fails, the file will be marked as "deleted" in the database even though it may not actually be deleted.
Once the delete grace time expires, Duplicati will refuse to run backups as a "new" file is found in the destination folder.
This PR changes the logic to track if the file was actually deleted, and will not mark the file deleted until the delete call reports success.
This fixes#6539
This PR adds an option to give a soft-delete prefix. If the prefix is set, files are renamed instead of actually being deleted. For backends that support life-cycle rules, this can be used to provide protection agains accidental deletion.
The intended use is that the client credentials do not have permissions to delete files, but can rename files. When a file is soft-deleted, it is then renamed and no longer "visible" to Duplicati. The destination can then have lifecycle rules that deletes files with the designated prefix after a set interval, or by another service that has permissions to actually delete files.
This PR adds support for locking files if the backend supports it.
To activate locking, set the option `--file-lock-duration=30D` and the backup will lock the files.
If the database is rebuilt with the intention of continuing the backups, use the option `--repair-refresh-lock-info` which will update lock information in the database after recreating the database.
The locking works by asking the backend to lock files after a backup has completed.
The implementation keeps track of which files are currently assigned a lock and prevents attempting to delete the files that are currently locked.
Note that the bucket should not have a default lock policy as Duplicati needs to finish the backup before the locking is applied.
In this initial version, Azure Blob Storage, B2, S3 and iDrive are supported with locking.
The CLI is updated to allow setting locks on a specific version. The backend tool is updated to allow setting locks on specific files.
This PR adds a dynamic property so a backend can signal if it supports streaming, based on the settings.
This is currently used for the File backend, so that toggling `--use-move-for-put` will disable streaming on the backend instead of relying on the `--disable-streaming-transfers` flag.
This PR changes the reported filename to use the remote filename instead of the random temporary filename that has little value to the user as the file is deleted immediately after the error is raised.
This PR adds the ability to subscribe to messages over a websocket.
To prevent polling data, the server can now push messages through the websocket, so the client can instantly update when new data is available.
The change is backwards compatible, still serving the status updates over the socket. If the client is providing the authentication token, it is auto-subscried to the legacy status message.
If the client is using the new authentication message, it needs to subscribe to get the status updates (and any other services it needs).
The event system has been extended to support new, and more accurate, events. This is the first step towards removing the general status update and the long-poll mechanism.
This also re-introduces the blocking marker, so the client can know if the operation is halted due to too many pending transfers.
Some endpoints have been moved to service implementations, to allow serving the exact same data from both endpoints and websocket.
This PR also updates the way the progress is handled, so that all transfers are returned to the client, and the transfer speeds for each transfer is calculated based on a small sample buffer, so the values are more accurate even if the program is paused during transfers.
This PR adds the option to disable throttle on a backup, so it will ignore throttle settings both from the server and the job.
Additionally, a list of backend keys can be provided, where backups will disable throttling. This can be used to set advanced options with a set of backends that should not be throttled.
By default, the `file` backend is now exempt from throttling by default.
This fixes#2685
This change now flushes all pending messages from the backendmanager to the database, even if the main operation crashes (or is stopped).
This ensures that the database and remote storage are more in sync.
Also removed a redundant warning from the backendmanager during shutdown, and simplified a check for active operations.
This fixes#6200
Kenneth Skovhede