Carl Johnsen
c5ebf4fa71
Added the restore channel depth option
2025-02-25 16:19:04 +01:00
Carl Johnsen
a0d5096ebc
Reworked the "Volume loop" of the restore network to handle Backends themselves, rather than using the BackendManager.
...
This allows for tighter control over which process performs which action, allowing them more easily to run in parallel.
2025-02-18 09:25:20 +01:00
Carl Johnsen
93756cd74b
Whitespace fixes
2025-02-18 09:17:08 +01:00
Kenneth Skovhede and GitHub
d99f74ed94
Rewrite the backend manager. ( #5896 )
...
* Rewrite the backend manager.
This creates a new more logic backend manager that handles all backend operations.
For each top-level operation, there is now a single backend manager instance that is passed to sub-commands.
The internals of the backend manager are now rewritten to use async logic instead of threading.
The design uses a single task runner that dispatches operations and honors concurrent settings and limits from one place.
The logic for each operation has been moved into a separate files/classes to make it easier to understand each operation.
This fixes #5804
* Fixed a few issues with not awaiting tasks
* Fixed not creating empty databases
* Fixed an issue with hashes not being recorded on download
* Reworked the download logic to avoid attempting to decrypt the file if it has been modified.
* Review fixes
* Renamed db collector to better reflect the purpose.
* Review fixes
2025-01-28 08:54:50 +01:00
Kenneth Skovhede and GitHub
9464caf622
Feature/update license 2025 ( #5851 )
...
* Fixed some minor whitespace issues
* Updated all copyright to 2025
2025-01-07 09:40:39 +01:00
Carl Johnsen
2d6e1d7ff7
Changed the new restore version to use explicit CoCoL channels
2024-12-10 04:10:34 +01:00
Carl Johnsen
6ba956dfcd
Moved all database operations to the LocalRestoreDatabase.
...
Also changed the old changes, so that the functions related to the legacy restore remains untouched.
2024-12-03 08:35:14 +01:00
Carl Johnsen
a663c46178
Removed debug depth for channels
2024-12-02 20:07:49 +01:00
Carl Johnsen
e7b2338214
Renamed VolumeCache to VolumeManager for consistency
2024-12-02 15:43:52 +01:00
Carl Johnsen
b6ed72bd6c
Reworked the restore CSP network
2024-12-02 15:09:23 +01:00
Carl Johnsen
3156d43adc
BlockVolumeReaders are expensive to create - cache them instead.
2024-12-02 10:55:45 +01:00
Carl Johnsen
27c4f93af2
The FileProcessor bursts as much as possible by default
2024-11-25 08:53:27 +01:00
Carl Johnsen
0cd9374bed
Added comments and docstrings
2024-11-22 15:26:04 +01:00
Carl Johnsen
890fa2ce52
Add copyright notices
2024-11-22 11:47:07 +01:00
Carl Johnsen
8c4f5cebf0
The size of the channels are being used by the FileProcessor so that it knows how much it can burst requests.
2024-11-21 20:57:01 +01:00
Carl Johnsen
effd50985f
The real deadlock on process network initialization was found: the volume downloader did not write asynchronously, which would result it waiting on writing to a process that would never be created.
2024-11-15 12:57:55 +01:00
Carl Johnsen
dc73380fe8
To minimize database requests (or at least where they emerge), query the database once per block per file, once per newly downloaded volume, and once to build the file list. The results are then "shipped" around the network along with the previous messages.
2024-11-13 16:55:47 +01:00
Carl Johnsen
66df6afc34
Changed the downloader to move the download waiting to the decrypter
2024-11-13 14:31:37 +01:00
Carl Johnsen
09e04e76d0
Now the restore chain also forwards the block id
2024-11-12 13:45:32 +01:00
Carl Johnsen
459664d33b
Added restore channel buffer sizes as a variable
2024-11-08 14:38:24 +01:00
Carl Johnsen
f6839160f2
Channels now send the volume / block id around alongside the volume / block
2024-11-08 10:42:54 +01:00
Carl Johnsen
d4a073cc8d
Fixed typo
2024-11-08 10:06:00 +01:00
Carl Johnsen
bd52be6f6b
Added dummy decrypter process
2024-11-08 09:55:17 +01:00
Carl Johnsen
5955cbc036
Modified the VolumeDownloader so that it works
2024-11-08 09:13:54 +01:00
Carl Johnsen
b41b43f430
Now the volume ID is found alongside the block ID
2024-11-08 08:02:17 +01:00
Carl Johnsen
adf9d4acf4
Added the draft block manager
2024-11-08 07:20:18 +01:00
Carl Johnsen
027d777fe0
First draft of the Restore.FileLister process
2024-11-08 03:31:30 +01:00
Carl Johnsen
b5745a3be0
Added initial volume downloader process
2024-11-07 10:04:52 +01:00
Carl Johnsen