This PR adds support for retries in the backend tester.
The CI tests have been updated to retry operations 3 times, which will hopefully make the CI tests more stable.
This moves the throttle support to the StreamUtil throttled stream, which allows multiple streams to share a throttle value, such that the throttle setting is applied to the entire communication from Duplicati, and not for each connection.
This also cleans up a bit in the progress reporting, where the backend manager now keeps track of the "active" transfer and only emits progress on that. Once it finishes, a new transfer is promoted a "active".
This PR is the final change, where all backends are now purely implemented with async interfaces.
The code changes here replace the LIST call with an async version. This affect all backends (and then some) but the changes are mostly mechanical rewrites of the code.
In many places, the backeds were already prepared for async output, in other some glue was needed.
To reduce the scope of this change, some backends simply report a synchronous result as an async list.
This change will allow an environment variable READ_WRITE_TIMEOUT_SECONDS to control the read write timeout used during the tests on streaming backends.
-1 will set it to infinite.
Additionally all console logs during test execution have a timestamp with millisecond precision to aid in benchmarking.
All CLI tools and executables now accept a unified set of help command variants:
"help", "/help","--help", "usage", "/usage", "--usage", "/h", "-h"
Affected executables:
Duplicati.Agent
Duplicati.CommandLine
Duplicati.CommandLine.AutoUpdater
Duplicati.CommandLine.BackendTester
Duplicati.CommandLine.BackendTool
Duplicati.CommandLine.ConfigurationImporter
Duplicati.CommandLine.SecretTool
Duplicati.CommandLine.Snapshots
Duplicati.GUI.TrayIcon
Duplicati.Server
Duplicati.Service
Duplicati.WindowsService
Note that SharpAesCrypt command is not included in this because the parameter parsing is part of the SharpAesCrypt code base.
* Added options to insert delays in backend tester
* Added wait delays to Azure and GoogleDrive to see it the CI is more stable with small delays
* Added more debug output to the backend tester
* Fixed dropbox reporting OAuth quota errors, when it is likely Dropbox API quota errors
* Implemented exception detection on AzureBlob backend
* Also showing sleep times in output
* Fixed a bug with the dropbox quota response
* Fixed error message to report correct number of files
* Fixed suppressed warning
* Fixed issues with timeout in msgraph
Fixed a place where the cancellation token was not applied.
Fixed a problem with msgraph not sending correct DNS information.
Fixed a problem with msgraph downloading the entire file before streaming it, bypassing throttle and tripping the timeout.
Added `ITimeoutExemptBackend` to selectively disable timeouts for backends that do not support it.
Updated the backendmanager to support `ITimeoutExemptBackend`.
Updated the backendmanager to abort as soon as a timeout happens.
* Updated BackendTester to use the `TimeoutObservingStream` and support the `--read-write-timeout` setting.
Updated backend tester code to use `var`
* Fixed a bug reading the `read-write-timeout`
* Updated to using constants with names instead of `-1`.
This commit adds the option to provide preload/preset options that change the default values for Duplicati's commandline or environment variables.
This fixes#5480
Add Storj DCS backend and deprecated Tardigrade DCS backend.
Storj renamed their "Tardigrade Decentralized Cloud Storage" product to "Storj DCS". This adds a new Storj DCS backend and deprecates the old Tardigrade one. A warning is issued to inform Tardigrade users to modify their configurations to use the new backend.
This also adds a passphrase verification field to ensure that the user has provided the correct encryption passphrase.
Kenneth Skovhede