This PR extends the allowed characters in names for secret keys to also include `-` and `_`, allowing keys like `$my-secret-key` which would previously not be matched.
This PR also adds detection of partial matches, so keys that start with the prefix, but do not match are reported as a warning. This could happen if the input key is `$my:secret` because the `:` is not matched. To prevent logging sensitive information, only the number of partial matches is reported, not the values.
It is possible that the user intended for such a value to be a matched key, but it is also possible that this value was meant to be verbatim. We can consider if this should stop the process instead of just emitting a warning, as the user should fix the issue in either case.
The fix for this would be to use the secret provider pattern and change to something that is not partially matched.
Since index files only contains information related to recreating the database, they can be recreated directly from a working database.
This PR adds an automatic repair feature that replaces index files if they are missing content. This will gradually repair remote index files if they are affected by the compact bug that re-wrote index files without the blocklists.
To fully repair, it is possible to run the `Test` command with the option `--full-remote-verification=indexonly` and a large number of samples. Since the new option `--replace-faulty-index-files` is default set to `true` this will repair any defective index files and ignore all others.
This update uncovered that the Test method would previously not verify the presence of blocklists in the index files. This is likely a very old bug, caused by the fact that the original implementation did not place blocklisthashes in the index files. The omission of this check is the reason the extent of the compact issue was not discovered earlier.
With this PR it is now also visible that there is ample room for error in creating the index files during the backup process. This is caused by the parallel processing and carry-over, where the index files are created on-the-go, so they are ready to upload once the blocks are filled.
While this is likely good for performance, it has some drawbacks.
- A failed block upload will cause a rewrite of the index file
- An elaborate callback system is needed to update the index file
- It is possible to race against the database and create extra blocklist hashes, bloating the index files (causes problems on verification)
A subsequent task is to rewrite the logic to not touch the index files outside the backend manager, so the backend manager will just use the database to create the index file. This means the same code will be invoked for both the create, the recreate, and the replacement.
For now, extra content in index files is logged with the verbose log level.
This fixes#6296
