This PR fixes a bug that caused backup searching to be case-sensitive.
The PR also exposes the case sensitive-flag in the API endpoint so the UI can toggle case-sensitive searching.
This PR implements support for reading and writing alternate data streams on Windows.
For now, the option is disabled by default and needs to be enabled with `--enable-ads-backup`. After this option is set, Duplicati will also extract any alternate datastreams from the filesystem and include them in the backup.
By default, ADS elements will be restored, but `--disable-ads-restore` can be used to ignore any ADS elements during the restore process.
This fixes#189
This fixes a locked database issue on Windows.
The issue was caused by the backup process having an open database, and then not passing that connection to the lock handler.
The lock handler would then attempt to open a new connection to the same database, which would fail.
This PR fixes the issue by passing an instance of the database to the lock handler.
Also addressed a logic issue where the dbpath was checked, even if an existing database connection was found.
Also fixed the lock results not setting the EndTime property, causing the reported lelapsed time to be off.
This PR has a large blast radius because it takes the final step and bumps up the Controller to be fully async.
We have historically done a piece-by-piece update, so all operations were already async but the controller interface was kept synchronous.
With this update, the controller is now fully async and all tests are updated.
Most places where the new C# compiler warns about function names not ending in `Async` were also adressed, giving a massive refactor change.
Functionally, no changes are done.
This adds an extra clause when building a synthetic filelist to avoid cases where the synthetic filelist fails to be generated because some entries are orphans.
The tests that reproduce this are quite slow and require some tampering with the database to get into the faulty state, so the tests will not run in CI testing but can manually be executed.
When queries are slow on end-user machines it is hard to debug what the exact problem is.
The typical flow is to discover a "stuck" process. But since profiling logging was not enabled there is no output.
The user then has to re-run the problematic operation with profiling on, and locate the last output where the call gets stuck.
This PR introducces a monitor mechanism where all queries register themselves prior to execution, and then de-register once done.
At a fixed interval, the list of active queries is examined, and if a query is found to be running longer than the threshold, it will be logged as a warning.
This means the user can simply go view the live log for warnings, and the offending query will be emitted.
The default threshold is set at a conservative 30 minutes. Since the timer only checks at the interval, it can take at most 2x the interval before a message is logged. After this, a message is logged each time the interval is reached (e.g., a new warning is logged every 30 min).
The option `--long-database-query-threshold` can be used to set a different threshold.
This PR adds a test that generates a synthentic filelist of a variable size, and then measures the time it takes to recreate the fileset.
As there was a reported slowdown, analysis indicated that the query would grow in the order of O=N^2, where N is the number of paths.
The code is then updated to instead use some temporary on-disk space to lay out the paths and then insert them as a new fileset. This should bring the time down to O=N.
Based on a test with 10k files, the query speed up is ~900x, but will be more on larger filesets.
This PR adds a small clause the prevents the delete query from failing if there are inconsistencies where deleted blocks can exist and a volume has no blocks.
I was not able to reproduce the issue without manipulating the database, but it was reported happening on the forum.
Co-authored-by: Copilot <copilot@github.com>
When performing queries with large inputs, there is a possibility that we reach the maximum number of possible parameters. If the query exceeds the maximum number of parameters the query fails.
This change uses a temporary table that will be created in the case the inputs exceed the total number available parameters, making sure the calls proceed as before with small inputs, but uses temporary tables on larger inputs.
Some tests were added to ensure the update works as expected, even with large inputs.
This PR adds guards to prevent creating filesets with multiple files that have the same path.
While it should technically be impossible to have multiple entries that have the same path, it could happend either due to glitches or because the source data (manual lists, remote sources, etc) returns duplicates.
This PR adds a simple check for each folder to ensure that on a folder-level, duplicate paths cannot be introduced.
There is also a post-backup check to evict any duplicates, and the recreate process will reject duplicate paths.
Finally, the repair command will remove duplicates if they somehow manage to get into the database anyway.
A database-level prevention is not currently feasible as it needs a cross-table check for uniqueness, which requires more work from the database to check for each added file. Since this is expected to be a very rare event, the added processing was not justified.
This PR guards agains inserting entries into the DeletedVolume if they reference non-existing volumes, which can happen in error scenarios.
This fixes#6553
This PR handles a case where the `dindex` files may reference non-existing `dblock` files. In this case the recreate may register missing (and unused) blocks in the `DeletedBlock` table, which will later cause issues if the blocks are attempted used.
The fix is to remove missing blocks before the cleanup so there is no entry that references the missing volumes.
This fixes issue #6552
This PR improves the query used to calculate the size of removed files when issuing the purge-broken-files command.
This PR also adds the option `--reduced-purge-statistics` which will fully skip the size calculation, in the event the query is still performing poorly.
This fixes#6366
When an error occurred, the log would be written to the database, but the transaction would then be rolled back, causing the log to disappear.
This adds explicit commit to the log when handling error cases.
This fixes#6538
Renamed the options to be more descriptive.
Rewrote the dependency query to be more readable and also check for metadata dependency.
Fixed an edge case where there were no modified files, and no new backup is created.
Fixed some issues with local/utc datetime compares.
Fixed logic for deleting filesets to only look at the filesets themselves. This makes the version lock follow the dlist file, and can create "dangling" volumes that have no filesets, but these will be picked up by compaction.
Updated tests to work correctly.
Added lock-info update option for the backup recreate call, so the UI can also re-create lock information on database rebuilds.
This PR adds support for locking files if the backend supports it.
To activate locking, set the option `--file-lock-duration=30D` and the backup will lock the files.
If the database is rebuilt with the intention of continuing the backups, use the option `--repair-refresh-lock-info` which will update lock information in the database after recreating the database.
The locking works by asking the backend to lock files after a backup has completed.
The implementation keeps track of which files are currently assigned a lock and prevents attempting to delete the files that are currently locked.
Note that the bucket should not have a default lock policy as Duplicati needs to finish the backup before the locking is applied.
In this initial version, Azure Blob Storage, B2, S3 and iDrive are supported with locking.
The CLI is updated to allow setting locks on a specific version. The backend tool is updated to allow setting locks on specific files.
This PR adds a check before entering the RemoveRemoteVolumes that checks if there are dangling FilesetEntry records when calling the function.
This makes it possible to figure out if the problem is created by the RemoveRemoteVolumes code, or an already existing problem in the database.
Also added a few tests that check that the condition is correctly detected.
This fixes an issue with formatting for invariant values that was caused by incrorrect function overload selection.
To avoid future issues, the two similar functions have been renamed to clarify what they are working for.
Kenneth Skovhede