This PR fixes a bug that caused backup searching to be case-sensitive.
The PR also exposes the case sensitive-flag in the API endpoint so the UI can toggle case-sensitive searching.
This PR fixes a bug where a script could write a new filter, but the filter was never applied to the backup.
This also fixes an issue where the filter string was not available in scripts or emails, except on the initial call.
This PR updates the logic so the `run-script-after` and `run-script-post-backup` also have access to the filter string, if needed for reporting or other.
This fixes#5143
This PR implements support for reading and writing alternate data streams on Windows.
For now, the option is disabled by default and needs to be enabled with `--enable-ads-backup`. After this option is set, Duplicati will also extract any alternate datastreams from the filesystem and include them in the backup.
By default, ADS elements will be restored, but `--disable-ads-restore` can be used to ignore any ADS elements during the restore process.
This fixes#189
This adds a check that prevents restoring files outside the specified target folder when using the recovery tool.
When choosing to restore to a specific folder, the expectation is that files are not restored outside the folder, which could happen with symlinks or manually crafted entries. This blocks it the same way the regular restore does and also supports `--allow-restore-outside-target-directory` to disable the check.
This fixes a case where an unmodified file that was renamed, but only had changes to the casing (i.e., `file.txt` to `FILE.txt`) would ignore the new name.
If the file contents or metadata were changed, the filename would be updated, but for pure renames, the new name would not be picked up.
This issue is only present if USN was also applied, and the bug is only present on Windows.
This PR adds a flag to the test method so we can choose to test for read-only access (needed for restore) or read-write (needed for backups).
Additionally, this PR fixes a logic bug in that the BackendSourceProvider would not actually test the connection, but just check the placeholder element returned (which is always returned). With this PR the backend's ListAsync command is tested fully.
Also renamed the methods on ISourceProvider to have the `...Async` prefix.
This replaces #6939
This PR flips the logic so filesystem parsing is not enabled by default. This means that any disk source will be treated as an opaque stream of blocks.
Brave users can still try out the filesystem parsing, but we need more time to ensure the logic works across variations on filesystems.
This PR adds experimental support for running the backup in an isolated process.
The motivation for this is to protect the server from a faulty process and later enable running server and processing with different security levels.
This needs to be toggled on, and then it will run the process isolated. When running as an isolated process, the "Abort" call will kill the process so it can no longer become "stuck".
When creating a new controller, the options dictionary was used by-reference.
This meant that code could update the options dictionary *after* creating the controller, which would not be thread-safe.
It does not have much effect on regular operations as the normal method is to create the dictionary, pass it, and not modify it.
But in the tests, we frequently re-use the dictionary and update it with new values to test various cases.
Due to the dictionary being a reference, the controller would update the dictionary to fit the operations, but this would reflect back into the tests, meaning that some tests were being run with options that were not supposed to be there.
This meant that at least two edge cases that we test for were never found to have errors, even though they do.
The first issue is the listing of files using a non-default blocksize. The second issue is recreating the database with a non-default file-hash algorithm.
This PR removes the reference passing, and fixes the issues that were uncovered.
This PR has a large blast radius because it takes the final step and bumps up the Controller to be fully async.
We have historically done a piece-by-piece update, so all operations were already async but the controller interface was kept synchronous.
With this update, the controller is now fully async and all tests are updated.
Most places where the new C# compiler warns about function names not ending in `Async` were also adressed, giving a massive refactor change.
Functionally, no changes are done.
This adds an extra clause when building a synthetic filelist to avoid cases where the synthetic filelist fails to be generated because some entries are orphans.
The tests that reproduce this are quite slow and require some tampering with the database to get into the faulty state, so the tests will not run in CI testing but can manually be executed.
This PR fixes an issue with using a non-default compression method. The recreate database will detect the compression module correctly and restore everything. But when recreating the database it will look at the `compression-module` from the given options and persist this value into the database.
If the recreate was made without passing in the compression module in the options, the default compression module would be persisted to the database, which would later cause problems for operations that rely on this value.
With this PR there is a test for the behavior, and the compression module is extracted from the first volume, similar to how blocksize and other options are written back.
In some cases it is possible for the index volumes to be created with empty block lists. If this happens, the reader would fail as the strict JSON parsing would encounter an unexpected token.
With the fix, the code now correctly reads such JSON documents and some variants.
Added a number of tests to make sure these cases are correctly reproduced.
This fixes#6892
This PR adds a test that generates a synthentic filelist of a variable size, and then measures the time it takes to recreate the fileset.
As there was a reported slowdown, analysis indicated that the query would grow in the order of O=N^2, where N is the number of paths.
The code is then updated to instead use some temporary on-disk space to lay out the paths and then insert them as a new fileset. This should bring the time down to O=N.
Based on a test with 10k files, the query speed up is ~900x, but will be more on larger filesets.
This PR adds a small clause the prevents the delete query from failing if there are inconsistencies where deleted blocks can exist and a volume has no blocks.
I was not able to reproduce the issue without manipulating the database, but it was reported happening on the forum.
Co-authored-by: Copilot <copilot@github.com>
This PR adds experimental support for using either ZStandard or GZip compression. Unlike the Zip module, which also supports these methods, the new module does a full-volume compression which is both much faster and compresses better with modern compression algorithms.
Because the compressor can see the full volume (and not just the blocks) it can compress data across blocks.
The downside to this is that it is not possible to take a single entry out of the compressed stream, but instead, Duplicati needs to decompress the whole stream and can then access the contents.
To make the implementation compatible with standard tools, the inner format is Tar (using ustar format). To allow faster reading, a small end-of-file header is added to the Tar file, emulating the concept from Zip files.
This small addition allows Duplicati to have random access to files in a Tar volume without needing to scan the whole thing.
To make sure data is always recoverable, the format is 100% compatible with regular tools, so `untar -xf` will work on the created volumes.
The downside is an extra pass for decompression when reading the volumes. When writing, each entry (a block of data most commonly) is written to a temporary file before being added to the output stream. It is possible to toggle this to use a memory buffer for even more speed up.
The ZStandard compression is from `ZstdSharp.Port` which has excellent performance.
When we move to .NET11 this can easily be changed to the new built-in module.
Since this is the first attempt to add this, it will log a warning on each use, explaining that the feature is currently just for testing.
Co-authored-by: Copilot <copilot@github.com>
Kenneth Skovhede