This adds an extra option to allow setting the SQLite page cache size as a regular option.
Prior to this commit it was only possible to set the SQLite page cache size via environment variables.
The option to use environment variables is preserved, and as options from the environment variable are applied after the new setting, environment variables take precedence.
The default value for the new option is to use 1% of the system memory for the page cache. For the restore process, a connection per worker may be made which defaults to half the number of cores, so the maximum amount of memory is 1% * half the CPU cores.
This is just the upper limit, and SQLite may choose not to use all of it.
This fixes#6178
This effectively caused the retention time to be ignored when reading from the database.
Also moved some code related to finding an unfinished fileset into the common database class.
This fixes the restore issues as all sources will now appear as if they were originally part of the source file system.
The backend implementation has now been simplified so the backends only return names of entries with no path information. Most backends already do this, but the S3 backends did not, so this was fixed as well.
This also clears up the interfaces, so the backend does not know about the `ISourceProviderEntry`. All mapping is done by the `BackendSourceProvider`, making it simpler to support other backends as sources.
Made the SMBShareConnection slightly more robust.
Introduced v2 List API to S3 backend, currently it retains v1 as the default.
Changed S3 backends to not recurse, potentially making smaller listings. This option can be disabled if it breaks things.
Added support for relative paths in the SSH backend.
The option has no effect.
This closes#3566
The block cache was added originally to work around perceived slow hash table lookups in the database.
By having an in-memory dictionary, lookups would be much faster.
It has no effect on disk wear, as the lookups are read-only.
The impact of such a cache was measured to be minimal with recent versions of SQLite.
As the check is done on the hot-path for the backup, it is split into two queries, where the first is done in the table only and the other is joining two tables.
This is based on the solution in #4982
Carl Johnsen