If the delete call fails, the file will be marked as "deleted" in the database even though it may not actually be deleted.
Once the delete grace time expires, Duplicati will refuse to run backups as a "new" file is found in the destination folder.
This PR changes the logic to track if the file was actually deleted, and will not mark the file deleted until the delete call reports success.
This fixes#6539
This PR adds an option to give a soft-delete prefix. If the prefix is set, files are renamed instead of actually being deleted. For backends that support life-cycle rules, this can be used to provide protection agains accidental deletion.
The intended use is that the client credentials do not have permissions to delete files, but can rename files. When a file is soft-deleted, it is then renamed and no longer "visible" to Duplicati. The destination can then have lifecycle rules that deletes files with the designated prefix after a set interval, or by another service that has permissions to actually delete files.
This PR adds support for locking files if the backend supports it.
To activate locking, set the option `--file-lock-duration=30D` and the backup will lock the files.
If the database is rebuilt with the intention of continuing the backups, use the option `--repair-refresh-lock-info` which will update lock information in the database after recreating the database.
The locking works by asking the backend to lock files after a backup has completed.
The implementation keeps track of which files are currently assigned a lock and prevents attempting to delete the files that are currently locked.
Note that the bucket should not have a default lock policy as Duplicati needs to finish the backup before the locking is applied.
In this initial version, Azure Blob Storage, B2, S3 and iDrive are supported with locking.
The CLI is updated to allow setting locks on a specific version. The backend tool is updated to allow setting locks on specific files.
This PR adds a dynamic property so a backend can signal if it supports streaming, based on the settings.
This is currently used for the File backend, so that toggling `--use-move-for-put` will disable streaming on the backend instead of relying on the `--disable-streaming-transfers` flag.
This PR changes the reported filename to use the remote filename instead of the random temporary filename that has little value to the user as the file is deleted immediately after the error is raised.
This PR adds the ability to subscribe to messages over a websocket.
To prevent polling data, the server can now push messages through the websocket, so the client can instantly update when new data is available.
The change is backwards compatible, still serving the status updates over the socket. If the client is providing the authentication token, it is auto-subscried to the legacy status message.
If the client is using the new authentication message, it needs to subscribe to get the status updates (and any other services it needs).
The event system has been extended to support new, and more accurate, events. This is the first step towards removing the general status update and the long-poll mechanism.
This also re-introduces the blocking marker, so the client can know if the operation is halted due to too many pending transfers.
Some endpoints have been moved to service implementations, to allow serving the exact same data from both endpoints and websocket.
This PR also updates the way the progress is handled, so that all transfers are returned to the client, and the transfer speeds for each transfer is calculated based on a small sample buffer, so the values are more accurate even if the program is paused during transfers.
This PR adds the option to disable throttle on a backup, so it will ignore throttle settings both from the server and the job.
Additionally, a list of backend keys can be provided, where backups will disable throttling. This can be used to set advanced options with a set of backends that should not be throttled.
By default, the `file` backend is now exempt from throttling by default.
This fixes#2685
This change now flushes all pending messages from the backendmanager to the database, even if the main operation crashes (or is stopped).
This ensures that the database and remote storage are more in sync.
Also removed a redundant warning from the backendmanager during shutdown, and simplified a check for active operations.
This fixes#6200
This ensures that files cannot appear on the remote destination, despite terminating the host program.
This fixes#4485
There is still work needed to handle the repair and compact operations so they use transactions and can commit changes during transfer progress.
This moves the throttle support to the StreamUtil throttled stream, which allows multiple streams to share a throttle value, such that the throttle setting is applied to the entire communication from Duplicati, and not for each connection.
This also cleans up a bit in the progress reporting, where the backend manager now keeps track of the "active" transfer and only emits progress on that. Once it finishes, a new transfer is promoted a "active".
While investigating issue [#6103](https://github.com/duplicati/duplicati/issues/6103), found that connections were not disposed due to the backend pool in BackendManager's handler. Set up a FileZilla server to test the issue before and after the change, confirming that connections now close as expected with this fix.
This commit adds extra logic to prevent cases where the backup reports success but leaves behind partial files.
A new flag is recorded to track if partial files are possible. If no partial files are possible, the verification will fail if any are found.
