This repair will leave the database in a still-broken state, but if this is run prior to purge-broken-files, less data will be lost.
Also updated the purge-broken-files operation to keep files and directories in the set, if they are only missing metadata.
This hand-picks the changes from #5049 by @gpatel-fr.
The change is to use a left-join to include all broken files, even if there are no entries in the Blockset table.
The update is only visual but makes the results consistent with the actual operation performed by purge-broken-files.
This close#5049
This adds an extra option to allow setting the SQLite page cache size as a regular option.
Prior to this commit it was only possible to set the SQLite page cache size via environment variables.
The option to use environment variables is preserved, and as options from the environment variable are applied after the new setting, environment variables take precedence.
The default value for the new option is to use 1% of the system memory for the page cache. For the restore process, a connection per worker may be made which defaults to half the number of cores, so the maximum amount of memory is 1% * half the CPU cores.
This is just the upper limit, and SQLite may choose not to use all of it.
This fixes#6178
Bug #3210 (and several other possibly related ones) had cases where a recreated database would have these invalid blockset IDs.
It seems like they were caused by filesets referring to a blockset by hash, but not having any index or block file actually containing the block for that hash.
Also fixes countMissingInformation query - it should join with missingBlocklistVolumes, rather than missingBlocklistEntries, as it is selecting the VolumeID.
Carl Johnsen