This fixes an issue with formatting for invariant values that was caused by incrorrect function overload selection.
To avoid future issues, the two similar functions have been renamed to clarify what they are working for.
This fixes pre-calculating the SHA256 hash for AWS SDK requests to S3.
This fixes an issue where a throttled stream would not be throttled correctly, because the AWS library would read the throttled stream twice (once for hashing, and once for transmitting).
To ensure compatibility with Wasabi S3, the hash is also lowercased.
This fixes the encoding of hostnames with `.` characters in them. Without this fix, the hostname would be encoded making it impossible to use regular DNS hostnames.
The URLs passed to the custom URL parser were not decoding the path part, but passing it directly. This caused problems if the input was correctly URL encoded.
The updated code now correctly decodes the hostname+path parts, and encodes them again.
This PR removes the `OAuthContextSettings` class, and makes the `--oauth-url` option available on each backend that uses it, so it can be configured as part of the destination.
To assemble everything related to configuring OAuth, the code was moved to the `AuthIdOptionsHelper`, such that the usage can be shared between implementations.
This PR also adds the option to set the default OAuth url from the environment variable `DUPLICATI_OAUTH_SERVICE`.
The server can then be set in the following locations (most important last):
- Environment variable
- Server-wide advanced settings
- Backup job advanced settings (or commandline)
- Destination url
This PR adds the ability to subscribe to messages over a websocket.
To prevent polling data, the server can now push messages through the websocket, so the client can instantly update when new data is available.
The change is backwards compatible, still serving the status updates over the socket. If the client is providing the authentication token, it is auto-subscried to the legacy status message.
If the client is using the new authentication message, it needs to subscribe to get the status updates (and any other services it needs).
The event system has been extended to support new, and more accurate, events. This is the first step towards removing the general status update and the long-poll mechanism.
This also re-introduces the blocking marker, so the client can know if the operation is halted due to too many pending transfers.
Some endpoints have been moved to service implementations, to allow serving the exact same data from both endpoints and websocket.
This PR also updates the way the progress is handled, so that all transfers are returned to the client, and the transfer speeds for each transfer is calculated based on a small sample buffer, so the values are more accurate even if the program is paused during transfers.
This PR moves the check from the FTP backend into shared code, so all backends will now check for read/write permissions when using the "Test" button.
For situations where write permissions are not required (like restore or verify) there is a flag on the v2 API that indicates if the connection was established, so the UI can show a message suggesting that it is possible to proceed, if the current UI operation does not require write permissions.
This fixes#2473
If th user has the SeBackupPrivilege, set the default snapshot policy to `Auto` and otherwise use `Off` (the current default).
This ensures that if the process is running with the correct permissions, there will be fewer warnings.
This only changes the default value on Windows.
This fixes#2833
This adds nullability to FilterExpression and Options.
It also updates almost all properties in Options to use a common format that enforces uniform parsing of the values.
A few options have been updates to have the default values specified as a constant, to avoid replicating the value in two places.
This PR adds the option to suppress warnings by their log id. Warning IDs that are supplied to `--suppress-warnings` will be converted to information messages before being logged.
If a specific warning is disabled, such as `CompressionReadErrorFallback`, this will then no longer count as a warning for the job, and the log file will see the warning as an information message.
This PR also adds two simpler filter options that makes it possible to filter log messages by supplying the log IDs. This can already be achieved with log filters, but the ID filter is a bit simpler to apply, as you only need to know the ID.
Finally, this PR also adds common logging for errors in the categories:
- Permission denied, id = `PermissionDenied`
- File locked, id = `FileLocked`
- Path not found, id = `PathNotFound`
- Path too long, id = `PathTooLong`
These new log ids makes it simpler to ignore warnings about locked or inaccesible files.
Some providers, B2, Idrivee2, JottaCloud and S3 (AWS version) needs a checksum for the content.
To calculate this content the libraries will calculate the hash by reading the stream. But since the stream can be throttled, the hash calculation is throttled.
In some cases the contents are buffered, so the transfer is done at full speed, but the hashing at throttled speed. The average speed is the same, but network traffic is bursty, which it should not be with a throttle.
Also added option to disable signing with AWS Sigv4 as that interferes with the throttled stream. It is not currently possible to pass the sha-256 hash to the Sigv4 code, so it will always read the entire (potentially throttled) stream before sending the contents, causing unwanted delays. Use the new option `--s3-disable-payload-signing` to disable v4 signing.
This removes the WorkerThread class and many of the complications around it.
Instead of having a generic worker, there is now the `QueueRunnerService` which takes on all the resposibilities for handling queued tasks.
This is one step towards removing `FIXMEGlobal`
This moves the throttle support to the StreamUtil throttled stream, which allows multiple streams to share a throttle value, such that the throttle setting is applied to the entire communication from Duplicati, and not for each connection.
This also cleans up a bit in the progress reporting, where the backend manager now keeps track of the "active" transfer and only emits progress on that. Once it finishes, a new transfer is promoted a "active".
This adds an extra option to allow setting the SQLite page cache size as a regular option.
Prior to this commit it was only possible to set the SQLite page cache size via environment variables.
The option to use environment variables is preserved, and as options from the environment variable are applied after the new setting, environment variables take precedence.
The default value for the new option is to use 1% of the system memory for the page cache. For the restore process, a connection per worker may be made which defaults to half the number of cores, so the maximum amount of memory is 1% * half the CPU cores.
This is just the upper limit, and SQLite may choose not to use all of it.
This fixes#6178
On some locales, odd strings would parse as valid full dates, causing relative dates to be incorrectly calculated.
This commit fixes the issues and revisits the DST aware calculations with more tests.
This commit adds the `DateTime` option to support validation of the `--time` parameter, which can both be an absolute date and time, or a relative value.
This fixes#6107
Kenneth Skovhede