This hand-picks the changes from #5049 by @gpatel-fr.
The change is to use a left-join to include all broken files, even if there are no entries in the Blockset table.
The update is only visual but makes the results consistent with the actual operation performed by purge-broken-files.
This close#5049
This moves the throttle support to the StreamUtil throttled stream, which allows multiple streams to share a throttle value, such that the throttle setting is applied to the entire communication from Duplicati, and not for each connection.
This also cleans up a bit in the progress reporting, where the backend manager now keeps track of the "active" transfer and only emits progress on that. Once it finishes, a new transfer is promoted a "active".
The delete call would select the same fileset multiple times if multiple retention options were present.
This fixes the selection to a distinct set.
This fixes#6127
This addresses [Issue](https://github.com/duplicati/duplicati/issues/6114).
Whenever in non-write context, the permissions of the folder are not set.
Changing pattern from static constructor execution.
This comes from suggestion on issue https://github.com/duplicati/duplicati/issues/6056
The chosen argument was --json which produces a json that wraps the result with extended properties such as in example:
``
{
"Timestamp": "2025-03-28T15:47:01.3368160-03:00",
"UnixTimestamp": 1743187621,
"Command": "import",
"Success": true,
"ExitCode": 0,
"Messages": [
"Importing backup configuration from ../2-firstbacku.json",
"Connecting to http://127.0.0.1:8200/...",
"No database found in../data/",
"Imported \"firstbackup (5)\" with ID 8"
],
"Exceptions": [],
"Imported": {
"Id": "8",
"Name": "firstbackup (5)"
}
}
``
The documentation will reflect all commands and schemas as this makes its way into Canary
It should be noted that I was unable to completely remove IPv4 on Windows 11 to a state where it would not provide the 127.0.0.1 loopback, even when disabling Ipv4 in various ways, including with registry flags.
On ubuntu/linux, after completely disabling ipv4 the ipv6 loopback was returned.
This enables a fail-early approach, where the remote source loads the initial results.
By storing the `IAsyncEnumerable` it is possible to not perform any additional remote calls, even with the initial probing call.
This makes it possible to detect the absence of sources, so the backup can be aborted if sources are missing.
This fixes the restore issues as all sources will now appear as if they were originally part of the source file system.
The backend implementation has now been simplified so the backends only return names of entries with no path information. Most backends already do this, but the S3 backends did not, so this was fixed as well.
This also clears up the interfaces, so the backend does not know about the `ISourceProviderEntry`. All mapping is done by the `BackendSourceProvider`, making it simpler to support other backends as sources.
This PR is the final change, where all backends are now purely implemented with async interfaces.
The code changes here replace the LIST call with an async version. This affect all backends (and then some) but the changes are mostly mechanical rewrites of the code.
In many places, the backeds were already prepared for async output, in other some glue was needed.
To reduce the scope of this change, some backends simply report a synchronous result as an async list.
This change will allow an environment variable READ_WRITE_TIMEOUT_SECONDS to control the read write timeout used during the tests on streaming backends.
-1 will set it to infinite.
Additionally all console logs during test execution have a timestamp with millisecond precision to aid in benchmarking.
Added support for `--portable-mode` and `--server-datafolder` for the CLI, ServerUtil, and Agent.
Fixed a few places where file reads were not cached properly.
Moved some responsibilities of UpdaterManager into DataFolderManager.
This now supports storing `machineid.txt` and `installation.txt` inside the path pointed to by `--server-datafolder` or alternatively with `--portable-mode`.
It is possible that some fringe backwards compatibility is lost with this update.
Fixed an issue with AutoUpdater crashing on check/download.
This fixes#5902
All CLI tools and executables now accept a unified set of help command variants:
"help", "/help","--help", "usage", "/usage", "--usage", "/h", "-h"
Affected executables:
Duplicati.Agent
Duplicati.CommandLine
Duplicati.CommandLine.AutoUpdater
Duplicati.CommandLine.BackendTester
Duplicati.CommandLine.BackendTool
Duplicati.CommandLine.ConfigurationImporter
Duplicati.CommandLine.SecretTool
Duplicati.CommandLine.Snapshots
Duplicati.GUI.TrayIcon
Duplicati.Server
Duplicati.Service
Duplicati.WindowsService
Note that SharpAesCrypt command is not included in this because the parameter parsing is part of the SharpAesCrypt code base.
* Added options to insert delays in backend tester
* Added wait delays to Azure and GoogleDrive to see it the CI is more stable with small delays
* Added more debug output to the backend tester
* Fixed dropbox reporting OAuth quota errors, when it is likely Dropbox API quota errors
* Implemented exception detection on AzureBlob backend
* Also showing sleep times in output
* Fixed a bug with the dropbox quota response
* Fixed error message to report correct number of files
* Fixed suppressed warning
* Fixed issues with timeout in msgraph
Fixed a place where the cancellation token was not applied.
Fixed a problem with msgraph not sending correct DNS information.
Fixed a problem with msgraph downloading the entire file before streaming it, bypassing throttle and tripping the timeout.
Added `ITimeoutExemptBackend` to selectively disable timeouts for backends that do not support it.
Updated the backendmanager to support `ITimeoutExemptBackend`.
Updated the backendmanager to abort as soon as a timeout happens.
* Updated BackendTester to use the `TimeoutObservingStream` and support the `--read-write-timeout` setting.
Updated backend tester code to use `var`
* Fixed a bug reading the `read-write-timeout`
* Updated to using constants with names instead of `-1`.
