Carl Johnsen
ebb04ddd08
Made waking up a waiting block request on the BlockManagers cache async to lower time spent by the volume consumer task
2025-03-11 14:23:37 +01:00
Carl Johnsen
b61d594180
Added higher resolution internal profiling timers for the BlockManager
2025-03-11 14:22:49 +01:00
Kenneth Skovhede
6f7f2757a9
Rewrote timeout monitoring
2025-03-10 10:22:34 +01:00
Carl Johnsen
83dc334069
Improve error handling in restore operations by retiring inputs and outputs on exceptions
2025-03-10 05:44:31 +01:00
Carl Johnsen
63464342cd
Changed the request timeout to issue a warning, rather than an error
2025-03-09 08:16:10 +01:00
Carl Johnsen
60fa777a8e
Added a timeout for block requests
2025-03-03 17:34:27 +01:00
Carl Johnsen
1650399fe6
Added explicit logging to trace a block request through the new restore network
2025-03-03 17:33:20 +01:00
Carl Johnsen
96149bbcf3
The MemoryCache in BlockManager cannot be null any more.
2025-03-03 17:09:46 +01:00
Carl Johnsen
0ea45b0023
Rather than sleeping on the active readers spinlock, the post-eviction callback should yield to whichever other thread is trying to take the lock.
2025-03-03 17:09:20 +01:00
Carl Johnsen
805ad42472
The MemoryCache in the BlockManager was removing its own entry in its post-eviction callback
2025-03-03 17:08:36 +01:00
Carl Johnsen
dbe413d889
Fixed the restore BlockManager block counting logic
2025-02-28 10:26:07 +01:00
Carl Johnsen
2ebb021174
Cleaned up BlockManager
2025-02-27 14:40:29 +01:00
Carl Johnsen
2fae2d510e
Removed old block counters from BlockManager
2025-02-27 14:16:06 +01:00
Carl Johnsen
038e78ad06
The block cache is now always present, as it keeps track of blocks. In case of a cache size 0, every Set() will trigger an evict.
2025-02-27 11:17:11 +01:00
Carl Johnsen
bdfdc7e92a
MemoryCache evict now correctly handles the ArrayPool
2025-02-27 10:54:47 +01:00
Carl Johnsen
7109155127
Additional block count reporting of the block manager
2025-02-26 16:11:13 +01:00
Carl Johnsen
0e6e1bcbd8
Currently, the block cache cannot compact while using the arraypool, as this doesn't properly return the buffers to the pool
2025-02-26 16:10:51 +01:00
Carl Johnsen
e773907eed
Changed from allocating a new byte[] buffer in the decompressor to use the Buffers.ArrayPool
2025-02-25 16:45:55 +01:00
Carl Johnsen
93756cd74b
Whitespace fixes
2025-02-18 09:17:08 +01:00
Carl Johnsen
6600e418a7
Fixed a bug during profile logging
2025-02-18 09:14:17 +01:00
Kenneth Skovhede and GitHub
d99f74ed94
Rewrite the backend manager. ( #5896 )
...
* Rewrite the backend manager.
This creates a new more logic backend manager that handles all backend operations.
For each top-level operation, there is now a single backend manager instance that is passed to sub-commands.
The internals of the backend manager are now rewritten to use async logic instead of threading.
The design uses a single task runner that dispatches operations and honors concurrent settings and limits from one place.
The logic for each operation has been moved into a separate files/classes to make it easier to understand each operation.
This fixes #5804
* Fixed a few issues with not awaiting tasks
* Fixed not creating empty databases
* Fixed an issue with hashes not being recorded on download
* Reworked the download logic to avoid attempting to decrypt the file if it has been modified.
* Review fixes
* Renamed db collector to better reflect the purpose.
* Review fixes
2025-01-28 08:54:50 +01:00
Kenneth Skovhede and GitHub
9464caf622
Feature/update license 2025 ( #5851 )
...
* Fixed some minor whitespace issues
* Updated all copyright to 2025
2025-01-07 09:40:39 +01:00
Carl Johnsen
ee8a82c230
Moved logging outside of the lock on blockcount in restore block manager
2024-12-10 06:14:55 +01:00
Carl Johnsen
f3afd4c682
Reintroduced the restore block cache eviction logic
2024-12-10 06:03:08 +01:00
Carl Johnsen
7f37c49292
Changed the VolumeManager to use a single input channel of type object, and then switching on the message received, rather than ReadFromAny().
...
Because it seems like the amount of readers in queue for one of the channels became much higher (70) than the actual number of readers (1).
2024-12-10 04:12:33 +01:00
Carl Johnsen
2d6e1d7ff7
Changed the new restore version to use explicit CoCoL channels
2024-12-10 04:10:34 +01:00
Carl Johnsen
6ba956dfcd
Moved all database operations to the LocalRestoreDatabase.
...
Also changed the old changes, so that the functions related to the legacy restore remains untouched.
2024-12-03 08:35:14 +01:00
Carl Johnsen
46983ee5cd
Added proper handling of the internal profiling timers in BlockManagers SleepableDictionary
2024-12-02 19:59:56 +01:00
Carl Johnsen
4ae37292a1
Changed the count dictionaries in BlockManager to be normal dictionaries, as they are faster and the accesses to them are serialized anyways.
2024-12-02 17:56:33 +01:00
Carl Johnsen
b44934ac4b
Removed database from BlockManagers cache
2024-12-02 17:48:29 +01:00
Carl Johnsen
9e4e1e246b
Added docstrings to BlockManager
2024-12-02 15:49:20 +01:00
Carl Johnsen
3d8330bc58
Cleaned up BlockManager
2024-12-02 15:44:01 +01:00
Carl Johnsen
b6ed72bd6c
Reworked the restore CSP network
2024-12-02 15:09:23 +01:00
Carl Johnsen
3156d43adc
BlockVolumeReaders are expensive to create - cache them instead.
2024-12-02 10:55:45 +01:00
Carl Johnsen
cdf0c28fd8
Added internal profiling as an option. Currently only implemented in the restore process.
2024-11-29 05:49:03 +01:00
Carl Johnsen
7aa9db533e
Added preliminary metadata restoration
2024-11-26 20:01:19 +01:00
Carl Johnsen
164b2a212c
Already restored blocks are now tracked according to the cache tracking strategy as well
2024-11-25 17:32:56 +01:00
Carl Johnsen
952aebf685
VolumeDownloader handles when an eviction requests relates to a volume that is not in cache
2024-11-25 17:32:00 +01:00
Carl Johnsen
baaa17a738
Make cache_max and eviction_ratio fields readonly
2024-11-25 17:23:42 +01:00
Carl Johnsen
7c79535572
VolumeDownloader now deletes temporary volumes from both cache and disk when they are no longer needed during restore.
2024-11-25 15:16:03 +01:00
Carl Johnsen
7ee174b6f8
Execute the blockcount check and decrement in a single SQL statement
2024-11-25 09:33:32 +01:00
Carl Johnsen
ebac24ce04
Rather than specifying multiple guids for a restore, use the one that the restore database has.
2024-11-25 08:52:51 +01:00
Carl Johnsen
5436b2f9f5
Add eviction ratio option for restore cache management
2024-11-22 16:32:53 +01:00
Carl Johnsen
de24a9f15a
Added the option to control the size of the data block cache during restore
2024-11-22 16:24:27 +01:00
Carl Johnsen
0cd9374bed
Added comments and docstrings
2024-11-22 15:26:04 +01:00
Carl Johnsen
890fa2ce52
Add copyright notices
2024-11-22 11:47:07 +01:00
Carl Johnsen
8a1bf22e27
Shutting down and cleaning up the block directory could be done in an incorrect order where some task was trying to access the removed table.
...
Disposing has now been moved to after all of the accessors have finished.
2024-11-22 07:05:20 +01:00
Carl Johnsen
0d7b9403c9
The SleepableDictionary now keeps track of which blocks to restore, so that it only caches the blocks that needs to be cached.
2024-11-19 10:35:57 +01:00
Carl Johnsen
d47d775a34
Added logging and error handling to BlockManager
2024-11-15 15:00:27 +01:00
Carl Johnsen
0e2be45418
Changed the shutdown strategy of BlockManager as it could lead to a deadlock since the channels weren't retired properly.
2024-11-15 08:55:13 +01:00
Carl Johnsen