This PR updates existing backends to implement the `IRenameEnabledBackend` interface, allowing for more efficient file renaming operations. The implementation varies by backend, with some using native rename operations and others falling back to copy-and-delete patterns.
This PR adds support for locking files if the backend supports it.
To activate locking, set the option `--file-lock-duration=30D` and the backup will lock the files.
If the database is rebuilt with the intention of continuing the backups, use the option `--repair-refresh-lock-info` which will update lock information in the database after recreating the database.
The locking works by asking the backend to lock files after a backup has completed.
The implementation keeps track of which files are currently assigned a lock and prevents attempting to delete the files that are currently locked.
Note that the bucket should not have a default lock policy as Duplicati needs to finish the backup before the locking is applied.
In this initial version, Azure Blob Storage, B2, S3 and iDrive are supported with locking.
The CLI is updated to allow setting locks on a specific version. The backend tool is updated to allow setting locks on specific files.
This PR adds a dynamic property so a backend can signal if it supports streaming, based on the settings.
This is currently used for the File backend, so that toggling `--use-move-for-put` will disable streaming on the backend instead of relying on the `--disable-streaming-transfers` flag.
This PR adds an explicit check for the status code for a B2 upload to ensure that non-success status codes are treated as failures.
Prior to this PR an upload would be considered a success if it has a failed status code but returns a valid JSON document with a `contentLength` property.
The URLs passed to the custom URL parser were not decoding the path part, but passing it directly. This caused problems if the input was correctly URL encoded.
The updated code now correctly decodes the hostname+path parts, and encodes them again.
This PR moves the check from the FTP backend into shared code, so all backends will now check for read/write permissions when using the "Test" button.
For situations where write permissions are not required (like restore or verify) there is a flag on the v2 API that indicates if the connection was established, so the UI can show a message suggesting that it is possible to proceed, if the current UI operation does not require write permissions.
This fixes#2473
Some providers, B2, Idrivee2, JottaCloud and S3 (AWS version) needs a checksum for the content.
To calculate this content the libraries will calculate the hash by reading the stream. But since the stream can be throttled, the hash calculation is throttled.
In some cases the contents are buffered, so the transfer is done at full speed, but the hashing at throttled speed. The average speed is the same, but network traffic is bursty, which it should not be with a throttle.
Also added option to disable signing with AWS Sigv4 as that interferes with the throttled stream. It is not currently possible to pass the sha-256 hash to the Sigv4 code, so it will always read the entire (potentially throttled) stream before sending the contents, causing unwanted delays. Use the new option `--s3-disable-payload-signing` to disable v4 signing.
This PR is the final change, where all backends are now purely implemented with async interfaces.
The code changes here replace the LIST call with an async version. This affect all backends (and then some) but the changes are mostly mechanical rewrites of the code.
In many places, the backeds were already prepared for async output, in other some glue was needed.
To reduce the scope of this change, some backends simply report a synchronous result as an async list.
This will make listing buckets with large count of files more reliable, if a 429 response is received it will conform to retry-after headers or if not present (as is the case today) will default to a 5 second delay.
Modernize HTTP client implementation across backend services by:
- Replace legacy HttpWebClient with modern HttpClient
- Implement proper async/await patterns throughout the codebase
- Add cancellation token support for improved request handling
- Refactor related code for better maintainability
This change is part of our broader initiative to modernize the codebase
and improve its resilience through proper asynchronous operations.
Without await, the using statement can dispose the Stream before the
call to PutAsync completes, resulting in an ObjectDisposedException.
This fixes#4556.
Kenneth Skovhede