This PR adds a new option `--disable-default-secret-provider` that will fully disable probing for the default secret provider.
This is to assist in cases where the secret provider is broken or shows unwanted popup dialogs.
This PR extends the handling of excluded file attributes on Windows to include additional values that are present in Windows 10+11, but are not yet part of the .NET file attributes.
Specifically, this handles the `FILE_ATTRIBUTE_RECALL_ON_OPEN` and `FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS` attributes which are used to signal files are not available locally and will be fetched on access.
On Windows, a new option `--exclude-non-local-files` is added which will simply update `--exclude-files-attributes` with `Offline,RecallOnOpen,RecallOnDataAccess` such that setting the option will exclude offline files from the backup, without requiring that the user is aware of the attribute names or effects.
For all platforms, this PR also updates the parsing of the `--exclude-files-attributes` so it is now a flags-type enum and will validate the input as well as show the possible options in help and dropdowns.
This PR adds a flag to the test method so we can choose to test for read-only access (needed for restore) or read-write (needed for backups).
Additionally, this PR fixes a logic bug in that the BackendSourceProvider would not actually test the connection, but just check the placeholder element returned (which is always returned). With this PR the backend's ListAsync command is tested fully.
Also renamed the methods on ISourceProvider to have the `...Async` prefix.
This replaces #6939
This fixes a locked database issue on Windows.
The issue was caused by the backup process having an open database, and then not passing that connection to the lock handler.
The lock handler would then attempt to open a new connection to the same database, which would fail.
This PR fixes the issue by passing an instance of the database to the lock handler.
Also addressed a logic issue where the dbpath was checked, even if an existing database connection was found.
Also fixed the lock results not setting the EndTime property, causing the reported lelapsed time to be off.
This PR flips the logic so filesystem parsing is not enabled by default. This means that any disk source will be treated as an opaque stream of blocks.
Brave users can still try out the filesystem parsing, but we need more time to ensure the logic works across variations on filesystems.
This PR adds experimental support for running the backup in an isolated process.
The motivation for this is to protect the server from a faulty process and later enable running server and processing with different security levels.
This needs to be toggled on, and then it will run the process isolated. When running as an isolated process, the "Abort" call will kill the process so it can no longer become "stuck".
When creating a new controller, the options dictionary was used by-reference.
This meant that code could update the options dictionary *after* creating the controller, which would not be thread-safe.
It does not have much effect on regular operations as the normal method is to create the dictionary, pass it, and not modify it.
But in the tests, we frequently re-use the dictionary and update it with new values to test various cases.
Due to the dictionary being a reference, the controller would update the dictionary to fit the operations, but this would reflect back into the tests, meaning that some tests were being run with options that were not supposed to be there.
This meant that at least two edge cases that we test for were never found to have errors, even though they do.
The first issue is the listing of files using a non-default blocksize. The second issue is recreating the database with a non-default file-hash algorithm.
This PR removes the reference passing, and fixes the issues that were uncovered.
This PR hides the `--unittest-mode` option that is used to treat all warnings as errors which is helpful when testing things, but is not intended to be used elsewhere.
The option is simply not shown in the list of supported options but still works in case anyone wants to use it.
This PR has a large blast radius because it takes the final step and bumps up the Controller to be fully async.
We have historically done a piece-by-piece update, so all operations were already async but the controller interface was kept synchronous.
With this update, the controller is now fully async and all tests are updated.
Most places where the new C# compiler warns about function names not ending in `Async` were also adressed, giving a massive refactor change.
Functionally, no changes are done.
This PR deprecates the `mssp` and `od4b` backends as they stopped working when the APIs were turned off.
The upgrade path is to use `sharepoint` instead (which used to be named "SharePoint v2").
This PR also adds a small amount of metadata to modules so they can be marked as "untested" or "deprecated", and the configuration for marking modules has been set up in `BackendModules.cs`.
The backends that we currently cannot test (usualy due to not having public signups) are now marked as untested.
This PR changes the restore message when no files are restored.
The logic now keeps track of existing files that do not need to be restored, as well as the actual number of bytes written.
The size of restored files was tracked previously, but was instead reporting the size of the written data.
With this update the legacy restore and new restore report the same numbers.
The warning that is shown if no files are restored has been updated to check if existing files were present, and if so, the restore is assumed to have worked as expected, and an information message is logged, instead of the warning message.
This PR adds a small change when writing the `filelist.json`. The code now *requires* that a metahash is present when writing an entry.
Previously, the code would handle cases where there was no metahash and output the `filelist.json` without these fields.
However, the recreate fails if the fields are missing due to database consistency requirements.
Also, there appear to be no regular code paths that should end up creating an entry without the metahash.
This may need to be reverted, but for now it will help track down if there are any cases where a missing metahash is needed, and if we find that, we can fix the relevant functions and revert this change.
This adds an extra clause when building a synthetic filelist to avoid cases where the synthetic filelist fails to be generated because some entries are orphans.
The tests that reproduce this are quite slow and require some tampering with the database to get into the faulty state, so the tests will not run in CI testing but can manually be executed.
This PR fixes an issue with using a non-default compression method. The recreate database will detect the compression module correctly and restore everything. But when recreating the database it will look at the `compression-module` from the given options and persist this value into the database.
If the recreate was made without passing in the compression module in the options, the default compression module would be persisted to the database, which would later cause problems for operations that rely on this value.
With this PR there is a test for the behavior, and the compression module is extracted from the first volume, similar to how blocksize and other options are written back.
When queries are slow on end-user machines it is hard to debug what the exact problem is.
The typical flow is to discover a "stuck" process. But since profiling logging was not enabled there is no output.
The user then has to re-run the problematic operation with profiling on, and locate the last output where the call gets stuck.
This PR introducces a monitor mechanism where all queries register themselves prior to execution, and then de-register once done.
At a fixed interval, the list of active queries is examined, and if a query is found to be running longer than the threshold, it will be logged as a warning.
This means the user can simply go view the live log for warnings, and the offending query will be emitted.
The default threshold is set at a conservative 30 minutes. Since the timer only checks at the interval, it can take at most 2x the interval before a message is logged. After this, a message is logged each time the interval is reached (e.g., a new warning is logged every 30 min).
The option `--long-database-query-threshold` can be used to set a different threshold.
In some cases it is possible for the index volumes to be created with empty block lists. If this happens, the reader would fail as the strict JSON parsing would encounter an unexpected token.
With the fix, the code now correctly reads such JSON documents and some variants.
Added a number of tests to make sure these cases are correctly reproduced.
This fixes#6892
This PR adds a test that generates a synthentic filelist of a variable size, and then measures the time it takes to recreate the fileset.
As there was a reported slowdown, analysis indicated that the query would grow in the order of O=N^2, where N is the number of paths.
The code is then updated to instead use some temporary on-disk space to lay out the paths and then insert them as a new fileset. This should bring the time down to O=N.
Based on a test with 10k files, the query speed up is ~900x, but will be more on larger filesets.
This PR adds a try/catch around initializing the default secret provider, to avoid crashes on startup where the setup is not correct for the provider.
If an explicit provider is set, this will crash as the user explicitly asked for it, but it is not working.
With this change, the implicit secret provider does not cause crashes.
This PR adds support for using the `quota-disable` option to diable the quota check performed before executing the remote sync operation.
This is needed if the quota information returned is unreliable.
This PR adds a small clause the prevents the delete query from failing if there are inconsistencies where deleted blocks can exist and a volume has no blocks.
I was not able to reproduce the issue without manipulating the database, but it was reported happening on the forum.
Co-authored-by: Copilot <copilot@github.com>
This PR adds a new `--run-script-post-backup` option that allows running a script after the main backup operation has completed, but before locking, compacting and verification is done.
The use for this is intended for cases where you want to stop or pause some service, run a backup, and then resume the service. Prior to this PR you would need to wait for the entire backup operation to complete, but with the new option you can resume as soon as the source data is no longer needed.
