When performing queries with large inputs, there is a possibility that we reach the maximum number of possible parameters. If the query exceeds the maximum number of parameters the query fails.
This change uses a temporary table that will be created in the case the inputs exceed the total number available parameters, making sure the calls proceed as before with small inputs, but uses temporary tables on larger inputs.
Some tests were added to ensure the update works as expected, even with large inputs.
This PR adds guards to prevent creating filesets with multiple files that have the same path.
While it should technically be impossible to have multiple entries that have the same path, it could happend either due to glitches or because the source data (manual lists, remote sources, etc) returns duplicates.
This PR adds a simple check for each folder to ensure that on a folder-level, duplicate paths cannot be introduced.
There is also a post-backup check to evict any duplicates, and the recreate process will reject duplicate paths.
Finally, the repair command will remove duplicates if they somehow manage to get into the database anyway.
A database-level prevention is not currently feasible as it needs a cross-table check for uniqueness, which requires more work from the database to check for each added file. Since this is expected to be a very rare event, the added processing was not justified.
This PR guards agains inserting entries into the DeletedVolume if they reference non-existing volumes, which can happen in error scenarios.
This fixes#6553
This PR handles a case where the `dindex` files may reference non-existing `dblock` files. In this case the recreate may register missing (and unused) blocks in the `DeletedBlock` table, which will later cause issues if the blocks are attempted used.
The fix is to remove missing blocks before the cleanup so there is no entry that references the missing volumes.
This fixes issue #6552
This PR improves the query used to calculate the size of removed files when issuing the purge-broken-files command.
This PR also adds the option `--reduced-purge-statistics` which will fully skip the size calculation, in the event the query is still performing poorly.
This fixes#6366
When an error occurred, the log would be written to the database, but the transaction would then be rolled back, causing the log to disappear.
This adds explicit commit to the log when handling error cases.
This fixes#6538
Renamed the options to be more descriptive.
Rewrote the dependency query to be more readable and also check for metadata dependency.
Fixed an edge case where there were no modified files, and no new backup is created.
Fixed some issues with local/utc datetime compares.
Fixed logic for deleting filesets to only look at the filesets themselves. This makes the version lock follow the dlist file, and can create "dangling" volumes that have no filesets, but these will be picked up by compaction.
Updated tests to work correctly.
Added lock-info update option for the backup recreate call, so the UI can also re-create lock information on database rebuilds.
This PR adds support for locking files if the backend supports it.
To activate locking, set the option `--file-lock-duration=30D` and the backup will lock the files.
If the database is rebuilt with the intention of continuing the backups, use the option `--repair-refresh-lock-info` which will update lock information in the database after recreating the database.
The locking works by asking the backend to lock files after a backup has completed.
The implementation keeps track of which files are currently assigned a lock and prevents attempting to delete the files that are currently locked.
Note that the bucket should not have a default lock policy as Duplicati needs to finish the backup before the locking is applied.
In this initial version, Azure Blob Storage, B2, S3 and iDrive are supported with locking.
The CLI is updated to allow setting locks on a specific version. The backend tool is updated to allow setting locks on specific files.
This PR adds a check before entering the RemoveRemoteVolumes that checks if there are dangling FilesetEntry records when calling the function.
This makes it possible to figure out if the problem is created by the RemoveRemoteVolumes code, or an already existing problem in the database.
Also added a few tests that check that the condition is correctly detected.
This fixes an issue with formatting for invariant values that was caused by incrorrect function overload selection.
To avoid future issues, the two similar functions have been renamed to clarify what they are working for.
Kenneth Skovhede