This PR fixes a bug that caused backup searching to be case-sensitive.
The PR also exposes the case sensitive-flag in the API endpoint so the UI can toggle case-sensitive searching.
This PR fixes a bug where a script could write a new filter, but the filter was never applied to the backup.
This also fixes an issue where the filter string was not available in scripts or emails, except on the initial call.
This PR updates the logic so the `run-script-after` and `run-script-post-backup` also have access to the filter string, if needed for reporting or other.
This fixes#5143
This PR flips the logic so filesystem parsing is not enabled by default. This means that any disk source will be treated as an opaque stream of blocks.
Brave users can still try out the filesystem parsing, but we need more time to ensure the logic works across variations on filesystems.
This PR has a large blast radius because it takes the final step and bumps up the Controller to be fully async.
We have historically done a piece-by-piece update, so all operations were already async but the controller interface was kept synchronous.
With this update, the controller is now fully async and all tests are updated.
Most places where the new C# compiler warns about function names not ending in `Async` were also adressed, giving a massive refactor change.
Functionally, no changes are done.
This PR deprecates the `mssp` and `od4b` backends as they stopped working when the APIs were turned off.
The upgrade path is to use `sharepoint` instead (which used to be named "SharePoint v2").
This PR also adds a small amount of metadata to modules so they can be marked as "untested" or "deprecated", and the configuration for marking modules has been set up in `BackendModules.cs`.
The backends that we currently cannot test (usualy due to not having public signups) are now marked as untested.
When queries are slow on end-user machines it is hard to debug what the exact problem is.
The typical flow is to discover a "stuck" process. But since profiling logging was not enabled there is no output.
The user then has to re-run the problematic operation with profiling on, and locate the last output where the call gets stuck.
This PR introducces a monitor mechanism where all queries register themselves prior to execution, and then de-register once done.
At a fixed interval, the list of active queries is examined, and if a query is found to be running longer than the threshold, it will be logged as a warning.
This means the user can simply go view the live log for warnings, and the offending query will be emitted.
The default threshold is set at a conservative 30 minutes. Since the timer only checks at the interval, it can take at most 2x the interval before a message is logged. After this, a message is logged each time the interval is reached (e.g., a new warning is logged every 30 min).
The option `--long-database-query-threshold` can be used to set a different threshold.
This PR adds experimental support for using either ZStandard or GZip compression. Unlike the Zip module, which also supports these methods, the new module does a full-volume compression which is both much faster and compresses better with modern compression algorithms.
Because the compressor can see the full volume (and not just the blocks) it can compress data across blocks.
The downside to this is that it is not possible to take a single entry out of the compressed stream, but instead, Duplicati needs to decompress the whole stream and can then access the contents.
To make the implementation compatible with standard tools, the inner format is Tar (using ustar format). To allow faster reading, a small end-of-file header is added to the Tar file, emulating the concept from Zip files.
This small addition allows Duplicati to have random access to files in a Tar volume without needing to scan the whole thing.
To make sure data is always recoverable, the format is 100% compatible with regular tools, so `untar -xf` will work on the created volumes.
The downside is an extra pass for decompression when reading the volumes. When writing, each entry (a block of data most commonly) is written to a temporary file before being added to the output stream. It is possible to toggle this to use a memory buffer for even more speed up.
The ZStandard compression is from `ZstdSharp.Port` which has excellent performance.
When we move to .NET11 this can easily be changed to the new built-in module.
Since this is the first attempt to add this, it will log a warning on each use, explaining that the feature is currently just for testing.
Co-authored-by: Copilot <copilot@github.com>
This is done by making the operation update class public so they can be accessed outside of the main library, and the progress updater is passed down from the controller.
When an error occurred, the log would be written to the database, but the transaction would then be rolled back, causing the log to disappear.
This adds explicit commit to the log when handling error cases.
This fixes#6538
This PR adds checks on limited restores, where the user has asked for files to be restored within a specific folder.
This PR adds explicit checks to ensure that the restore data cannot accidentially or maliciously constructed in such a way that it will affect files outside of the restore target folder.
This PR adds support for locking files if the backend supports it.
To activate locking, set the option `--file-lock-duration=30D` and the backup will lock the files.
If the database is rebuilt with the intention of continuing the backups, use the option `--repair-refresh-lock-info` which will update lock information in the database after recreating the database.
The locking works by asking the backend to lock files after a backup has completed.
The implementation keeps track of which files are currently assigned a lock and prevents attempting to delete the files that are currently locked.
Note that the bucket should not have a default lock policy as Duplicati needs to finish the backup before the locking is applied.
In this initial version, Azure Blob Storage, B2, S3 and iDrive are supported with locking.
The CLI is updated to allow setting locks on a specific version. The backend tool is updated to allow setting locks on specific files.
This adds the options `--log-http-requests` and `--log-socket-data` which can be activated to capture diagnostics about HTTP requests and socket operations.
Kenneth Skovhede