This PR adds a small clause the prevents the delete query from failing if there are inconsistencies where deleted blocks can exist and a volume has no blocks.
I was not able to reproduce the issue without manipulating the database, but it was reported happening on the forum.
Co-authored-by: Copilot <copilot@github.com>
Renamed the options to be more descriptive.
Rewrote the dependency query to be more readable and also check for metadata dependency.
Fixed an edge case where there were no modified files, and no new backup is created.
Fixed some issues with local/utc datetime compares.
Fixed logic for deleting filesets to only look at the filesets themselves. This makes the version lock follow the dlist file, and can create "dangling" volumes that have no filesets, but these will be picked up by compaction.
Updated tests to work correctly.
Added lock-info update option for the backup recreate call, so the UI can also re-create lock information on database rebuilds.
This PR adds support for locking files if the backend supports it.
To activate locking, set the option `--file-lock-duration=30D` and the backup will lock the files.
If the database is rebuilt with the intention of continuing the backups, use the option `--repair-refresh-lock-info` which will update lock information in the database after recreating the database.
The locking works by asking the backend to lock files after a backup has completed.
The implementation keeps track of which files are currently assigned a lock and prevents attempting to delete the files that are currently locked.
Note that the bucket should not have a default lock policy as Duplicati needs to finish the backup before the locking is applied.
In this initial version, Azure Blob Storage, B2, S3 and iDrive are supported with locking.
The CLI is updated to allow setting locks on a specific version. The backend tool is updated to allow setting locks on specific files.
This adds an extra option to allow setting the SQLite page cache size as a regular option.
Prior to this commit it was only possible to set the SQLite page cache size via environment variables.
The option to use environment variables is preserved, and as options from the environment variable are applied after the new setting, environment variables take precedence.
The default value for the new option is to use 1% of the system memory for the page cache. For the restore process, a connection per worker may be made which defaults to half the number of cores, so the maximum amount of memory is 1% * half the CPU cores.
This is just the upper limit, and SQLite may choose not to use all of it.
This fixes#6178
The exception could occur due to the existing delete for FilesetEntry
deleting based on volume ids, which would only match a filelist.
The bsIdsSubQuery would return BlocksetIds that should
be deleted and would delete from the FileLookup table.
FilesetEntry would then have a higher count of rows
than what would be returned by the LEFT OUTER JOINS on all of the
other tables in VerifyConsistency() since more rows had been deleted
from FileLookup.
To fix this when rows are deleted in FileLookup, use the associated metadata id
to delete from FilesetEntry and other tables based on the blockset id for the
metadata id.
Some existing delete statements were moved, none were deleted.
The existing FileLookup delete had the match on metadata id removed as
a blockset id does not equal a metadata id.
The ROW_NUMBER window function is not available in SQLite versions prior
to 3.25. In some cases, I think we do fall back to the system SQLite
libraries, and many distributions do not have versions that support the
window functions. To be safe, we will generate the backup versions
manually.
The ID in the Fileset table is the auto-incremented integer. When
considering filesets for deletion (in particular, when deleting specific
versions), we need to reference the backup version number that is
exposed to the user (integers where the newest backup version is 0).
This extracts the logic for various fileset deletion strategies into
separate classes. This simplifies the code and makes it easier to test
the individual functionalities.
This fixes an occurrence of the "Unexpected difference in fileset
version" error due to a deleted block having the same hash as one that
is currently in-use.
This fixes issue #3800.
Kenneth Skovhede