Carl Johnsen
5680fa3e15
If the restore operation threw an exception, the database wouldn't be closed properly, leading to subsequent tests failing due to the file being locked.
2024-12-05 08:49:36 +01:00
Carl Johnsen
ab7fcb2e4a
Opening a filestream doesn't work as expected on windows (caught in ProblematicPathTests) - changed to SystemIO.IO_OS.FileOpen*
2024-12-05 08:48:54 +01:00
Kenneth Skovhede
f5cabd4c7f
Remove --use-block-cache
...
The option has no effect.
This closes #3566
The block cache was added originally to work around perceived slow hash table lookups in the database.
By having an in-memory dictionary, lookups would be much faster.
It has no effect on disk wear, as the lookups are read-only.
The impact of such a cache was measured to be minimal with recent versions of SQLite.
2024-12-04 10:06:49 +01:00
Carl Johnsen
0e36c4393a
Added progress printing during restore
2024-12-03 10:35:33 +01:00
Carl Johnsen
2b0dc0613f
Added handling of existing symlinks
2024-12-03 10:04:08 +01:00
Carl Johnsen
c2e7275e1e
Removed unused metadatastorage in the new restore flow
2024-12-03 09:39:46 +01:00
Carl Johnsen
3e7b505ed2
Merge branch 'master' into performance/restoreopt
2024-12-03 09:21:14 +01:00
Carl Johnsen
f0c3fddfa2
Found runaway TODO
2024-12-03 09:11:42 +01:00
Carl Johnsen
76240550ed
Add preallocation option for restored files and update related documentation
2024-12-03 09:10:36 +01:00
Carl Johnsen
6ba956dfcd
Moved all database operations to the LocalRestoreDatabase.
...
Also changed the old changes, so that the functions related to the legacy restore remains untouched.
2024-12-03 08:35:14 +01:00
Carl Johnsen
6413014c7d
Added internal profiling of VolumeManager alongside proper exception handling.
2024-12-02 20:58:46 +01:00
Carl Johnsen
1e3e67c053
Update the construction of the processes within RestoreHandler
2024-12-02 20:48:20 +01:00
Carl Johnsen
99adcdd17c
Added license, docstrings and logging to VolumeManager
2024-12-02 20:41:56 +01:00
Carl Johnsen
7436606455
Cleanup VolumeDownloader
2024-12-02 20:32:38 +01:00
Carl Johnsen
d0df0c054e
Cleanup VolumeDecryptor
2024-12-02 20:29:40 +01:00
Carl Johnsen
1284adec3b
Cleanup VolumeDecompressor
2024-12-02 20:28:45 +01:00
Carl Johnsen
6c66fee921
Cleanup FileProcessor and rework docstrings
2024-12-02 20:27:21 +01:00
Carl Johnsen
9070ea2382
Added docstring and cleaned up FileLister
2024-12-02 20:12:06 +01:00
Carl Johnsen
a663c46178
Removed debug depth for channels
2024-12-02 20:07:49 +01:00
Carl Johnsen
46983ee5cd
Added proper handling of the internal profiling timers in BlockManagers SleepableDictionary
2024-12-02 19:59:56 +01:00
Carl Johnsen
4ae37292a1
Changed the count dictionaries in BlockManager to be normal dictionaries, as they are faster and the accesses to them are serialized anyways.
2024-12-02 17:56:33 +01:00
Carl Johnsen
b44934ac4b
Removed database from BlockManagers cache
2024-12-02 17:48:29 +01:00
Carl Johnsen
9e4e1e246b
Added docstrings to BlockManager
2024-12-02 15:49:20 +01:00
Carl Johnsen
3d8330bc58
Cleaned up BlockManager
2024-12-02 15:44:01 +01:00
Carl Johnsen
e7b2338214
Renamed VolumeCache to VolumeManager for consistency
2024-12-02 15:43:52 +01:00
Carl Johnsen
1bbe26330f
Bump CoCoL package version to 1.8.1
2024-12-02 15:18:09 +01:00
Carl Johnsen
7aa19f851e
Add configurable volume downloaders for restore process
2024-12-02 15:16:21 +01:00
Carl Johnsen
533088684b
Consistent naming of decryptors
2024-12-02 15:12:50 +01:00
Carl Johnsen
b6ed72bd6c
Reworked the restore CSP network
2024-12-02 15:09:23 +01:00
Carl Johnsen
3156d43adc
BlockVolumeReaders are expensive to create - cache them instead.
2024-12-02 10:55:45 +01:00
Carl Johnsen
f85ffabe3e
Linux SQLite doesn't want values in quotes
2024-12-02 10:45:52 +01:00
Carl Johnsen
cdf0c28fd8
Added internal profiling as an option. Currently only implemented in the restore process.
2024-11-29 05:49:03 +01:00
Carl Johnsen
87747b5580
The metadata SQL query is now prepared in the fileprocessor
2024-11-29 05:46:17 +01:00
Carl Johnsen
079598c2e8
Added database function for getting files alongside symlinks to restore
2024-11-29 03:41:41 +01:00
Carl Johnsen
6aa202fbea
Already present files were incorrectly truncated
2024-11-29 03:38:53 +01:00
Carl Johnsen
deb98e3322
CreateDirectoryStructure wouldn't always create the target folders, if the file database doesn't list the folder. Empty files should create the directory, if it doesn't exist.
2024-11-28 10:43:51 +01:00
Carl Johnsen
9bcda7d6cf
Symlinks are now handled correctly
2024-11-28 09:22:09 +01:00
Carl Johnsen
1ecb6e7c35
Fixed the metadata block query so that it finds the correct blocks
2024-11-27 11:28:27 +01:00
Carl Johnsen
4128ea3ff3
Empty files can also not have a data block associated
2024-11-27 11:27:55 +01:00
Carl Johnsen
7aa9db533e
Added preliminary metadata restoration
2024-11-26 20:01:19 +01:00
Carl Johnsen
164b2a212c
Already restored blocks are now tracked according to the cache tracking strategy as well
2024-11-25 17:32:56 +01:00
Carl Johnsen
952aebf685
VolumeDownloader handles when an eviction requests relates to a volume that is not in cache
2024-11-25 17:32:00 +01:00
Carl Johnsen
baaa17a738
Make cache_max and eviction_ratio fields readonly
2024-11-25 17:23:42 +01:00
Carl Johnsen
823b896da1
Removed redundant TODOs
2024-11-25 15:23:59 +01:00
Carl Johnsen
c26c54c5f7
Fixed naming inconsistencies in the new restore options
2024-11-25 15:17:10 +01:00
Carl Johnsen
7c79535572
VolumeDownloader now deletes temporary volumes from both cache and disk when they are no longer needed during restore.
2024-11-25 15:16:03 +01:00
Carl Johnsen
7ee174b6f8
Execute the blockcount check and decrement in a single SQL statement
2024-11-25 09:33:32 +01:00
Carl Johnsen
27c4f93af2
The FileProcessor bursts as much as possible by default
2024-11-25 08:53:27 +01:00
Carl Johnsen
ebac24ce04
Rather than specifying multiple guids for a restore, use the one that the restore database has.
2024-11-25 08:52:51 +01:00
Carl Johnsen
5625c4c1ce
Add configurable options for scaling the number of restore file processors, decryptors, and decompressors
2024-11-22 16:53:16 +01:00
Carl Johnsen