Carl Johnsen
dbe413d889
Fixed the restore BlockManager block counting logic
2025-02-28 10:26:07 +01:00
Carl Johnsen
2ebb021174
Cleaned up BlockManager
2025-02-27 14:40:29 +01:00
Carl Johnsen
2fae2d510e
Removed old block counters from BlockManager
2025-02-27 14:16:06 +01:00
Carl Johnsen
038e78ad06
The block cache is now always present, as it keeps track of blocks. In case of a cache size 0, every Set() will trigger an evict.
2025-02-27 11:17:11 +01:00
Carl Johnsen
bdfdc7e92a
MemoryCache evict now correctly handles the ArrayPool
2025-02-27 10:54:47 +01:00
Carl Johnsen
7109155127
Additional block count reporting of the block manager
2025-02-26 16:11:13 +01:00
Carl Johnsen
0e6e1bcbd8
Currently, the block cache cannot compact while using the arraypool, as this doesn't properly return the buffers to the pool
2025-02-26 16:10:51 +01:00
Carl Johnsen
e773907eed
Changed from allocating a new byte[] buffer in the decompressor to use the Buffers.ArrayPool
2025-02-25 16:45:55 +01:00
Carl Johnsen
93756cd74b
Whitespace fixes
2025-02-18 09:17:08 +01:00
Carl Johnsen
6600e418a7
Fixed a bug during profile logging
2025-02-18 09:14:17 +01:00
Kenneth Skovhede and GitHub
d99f74ed94
Rewrite the backend manager. ( #5896 )
...
* Rewrite the backend manager.
This creates a new more logic backend manager that handles all backend operations.
For each top-level operation, there is now a single backend manager instance that is passed to sub-commands.
The internals of the backend manager are now rewritten to use async logic instead of threading.
The design uses a single task runner that dispatches operations and honors concurrent settings and limits from one place.
The logic for each operation has been moved into a separate files/classes to make it easier to understand each operation.
This fixes #5804
* Fixed a few issues with not awaiting tasks
* Fixed not creating empty databases
* Fixed an issue with hashes not being recorded on download
* Reworked the download logic to avoid attempting to decrypt the file if it has been modified.
* Review fixes
* Renamed db collector to better reflect the purpose.
* Review fixes
2025-01-28 08:54:50 +01:00
Kenneth Skovhede and GitHub
9464caf622
Feature/update license 2025 ( #5851 )
...
* Fixed some minor whitespace issues
* Updated all copyright to 2025
2025-01-07 09:40:39 +01:00
Carl Johnsen
ee8a82c230
Moved logging outside of the lock on blockcount in restore block manager
2024-12-10 06:14:55 +01:00
Carl Johnsen
f3afd4c682
Reintroduced the restore block cache eviction logic
2024-12-10 06:03:08 +01:00
Carl Johnsen
7f37c49292
Changed the VolumeManager to use a single input channel of type object, and then switching on the message received, rather than ReadFromAny().
...
Because it seems like the amount of readers in queue for one of the channels became much higher (70) than the actual number of readers (1).
2024-12-10 04:12:33 +01:00
Carl Johnsen
2d6e1d7ff7
Changed the new restore version to use explicit CoCoL channels
2024-12-10 04:10:34 +01:00
Carl Johnsen
6ba956dfcd
Moved all database operations to the LocalRestoreDatabase.
...
Also changed the old changes, so that the functions related to the legacy restore remains untouched.
2024-12-03 08:35:14 +01:00
Carl Johnsen
46983ee5cd
Added proper handling of the internal profiling timers in BlockManagers SleepableDictionary
2024-12-02 19:59:56 +01:00
Carl Johnsen
4ae37292a1
Changed the count dictionaries in BlockManager to be normal dictionaries, as they are faster and the accesses to them are serialized anyways.
2024-12-02 17:56:33 +01:00
Carl Johnsen
b44934ac4b
Removed database from BlockManagers cache
2024-12-02 17:48:29 +01:00
Carl Johnsen
9e4e1e246b
Added docstrings to BlockManager
2024-12-02 15:49:20 +01:00
Carl Johnsen
3d8330bc58
Cleaned up BlockManager
2024-12-02 15:44:01 +01:00
Carl Johnsen
b6ed72bd6c
Reworked the restore CSP network
2024-12-02 15:09:23 +01:00
Carl Johnsen
3156d43adc
BlockVolumeReaders are expensive to create - cache them instead.
2024-12-02 10:55:45 +01:00
Carl Johnsen
cdf0c28fd8
Added internal profiling as an option. Currently only implemented in the restore process.
2024-11-29 05:49:03 +01:00
Carl Johnsen
7aa9db533e
Added preliminary metadata restoration
2024-11-26 20:01:19 +01:00
Carl Johnsen
164b2a212c
Already restored blocks are now tracked according to the cache tracking strategy as well
2024-11-25 17:32:56 +01:00
Carl Johnsen
952aebf685
VolumeDownloader handles when an eviction requests relates to a volume that is not in cache
2024-11-25 17:32:00 +01:00
Carl Johnsen
baaa17a738
Make cache_max and eviction_ratio fields readonly
2024-11-25 17:23:42 +01:00
Carl Johnsen
7c79535572
VolumeDownloader now deletes temporary volumes from both cache and disk when they are no longer needed during restore.
2024-11-25 15:16:03 +01:00
Carl Johnsen
7ee174b6f8
Execute the blockcount check and decrement in a single SQL statement
2024-11-25 09:33:32 +01:00
Carl Johnsen
ebac24ce04
Rather than specifying multiple guids for a restore, use the one that the restore database has.
2024-11-25 08:52:51 +01:00
Carl Johnsen
5436b2f9f5
Add eviction ratio option for restore cache management
2024-11-22 16:32:53 +01:00
Carl Johnsen
de24a9f15a
Added the option to control the size of the data block cache during restore
2024-11-22 16:24:27 +01:00
Carl Johnsen
0cd9374bed
Added comments and docstrings
2024-11-22 15:26:04 +01:00
Carl Johnsen
890fa2ce52
Add copyright notices
2024-11-22 11:47:07 +01:00
Carl Johnsen
8a1bf22e27
Shutting down and cleaning up the block directory could be done in an incorrect order where some task was trying to access the removed table.
...
Disposing has now been moved to after all of the accessors have finished.
2024-11-22 07:05:20 +01:00
Carl Johnsen
0d7b9403c9
The SleepableDictionary now keeps track of which blocks to restore, so that it only caches the blocks that needs to be cached.
2024-11-19 10:35:57 +01:00
Carl Johnsen
d47d775a34
Added logging and error handling to BlockManager
2024-11-15 15:00:27 +01:00
Carl Johnsen
0e2be45418
Changed the shutdown strategy of BlockManager as it could lead to a deadlock since the channels weren't retired properly.
2024-11-15 08:55:13 +01:00
Carl Johnsen
dc73380fe8
To minimize database requests (or at least where they emerge), query the database once per block per file, once per newly downloaded volume, and once to build the file list. The results are then "shipped" around the network along with the previous messages.
2024-11-13 16:55:47 +01:00
Carl Johnsen
284755c6b1
Rather than uncompressing everything, only decompress the requested block.
2024-11-12 13:47:25 +01:00
Carl Johnsen
09e04e76d0
Now the restore chain also forwards the block id
2024-11-12 13:45:32 +01:00
Carl Johnsen
b686cd4934
Changed from ConcurrentDictionary to MemoryCache, to utilize their cache eviction logic.
2024-11-11 13:02:24 +01:00
Carl Johnsen
ec738f4063
Added handling of retired exception to close down the process network
2024-11-08 14:35:44 +01:00
Carl Johnsen
4d7c8e2455
Volumes in flight should be a concurrent collection, as it's accessed through multiple Tasks
2024-11-08 13:01:05 +01:00
Carl Johnsen
f6839160f2
Channels now send the volume / block id around alongside the volume / block
2024-11-08 10:42:54 +01:00
Carl Johnsen
cb5b973883
Added a TODO in BlockManager
2024-11-08 09:59:39 +01:00
Carl Johnsen
50aa530883
db wasn't properly set in the BlockManager's internal cache
2024-11-08 08:45:20 +01:00
Carl Johnsen
a6f6cb7f39
CoCoL did not like the array of ChannerMarkers
2024-11-08 08:45:00 +01:00
Carl Johnsen