Carl Johnsen
9919d86e9b
Updated VolumeDownloader to use the new BackendManager.GetDirectAsync() method
2025-02-19 16:12:42 +01:00
Carl Johnsen
a0d5096ebc
Reworked the "Volume loop" of the restore network to handle Backends themselves, rather than using the BackendManager.
...
This allows for tighter control over which process performs which action, allowing them more easily to run in parallel.
2025-02-18 09:25:20 +01:00
Carl Johnsen
93756cd74b
Whitespace fixes
2025-02-18 09:17:08 +01:00
Kenneth Skovhede and GitHub
d99f74ed94
Rewrite the backend manager. ( #5896 )
...
* Rewrite the backend manager.
This creates a new more logic backend manager that handles all backend operations.
For each top-level operation, there is now a single backend manager instance that is passed to sub-commands.
The internals of the backend manager are now rewritten to use async logic instead of threading.
The design uses a single task runner that dispatches operations and honors concurrent settings and limits from one place.
The logic for each operation has been moved into a separate files/classes to make it easier to understand each operation.
This fixes #5804
* Fixed a few issues with not awaiting tasks
* Fixed not creating empty databases
* Fixed an issue with hashes not being recorded on download
* Reworked the download logic to avoid attempting to decrypt the file if it has been modified.
* Review fixes
* Renamed db collector to better reflect the purpose.
* Review fixes
2025-01-28 08:54:50 +01:00
Kenneth Skovhede and GitHub
9464caf622
Feature/update license 2025 ( #5851 )
...
* Fixed some minor whitespace issues
* Updated all copyright to 2025
2025-01-07 09:40:39 +01:00
Carl Johnsen
5bc12e3d46
Changed the reporting of broken local and remote files during restore to only report the first 10 filenames.
2024-12-10 06:57:56 +01:00
Carl Johnsen
2d6e1d7ff7
Changed the new restore version to use explicit CoCoL channels
2024-12-10 04:10:34 +01:00
Carl Johnsen
7436606455
Cleanup VolumeDownloader
2024-12-02 20:32:38 +01:00
Carl Johnsen
533088684b
Consistent naming of decryptors
2024-12-02 15:12:50 +01:00
Carl Johnsen
b6ed72bd6c
Reworked the restore CSP network
2024-12-02 15:09:23 +01:00
Carl Johnsen
3156d43adc
BlockVolumeReaders are expensive to create - cache them instead.
2024-12-02 10:55:45 +01:00
Carl Johnsen
cdf0c28fd8
Added internal profiling as an option. Currently only implemented in the restore process.
2024-11-29 05:49:03 +01:00
Carl Johnsen
952aebf685
VolumeDownloader handles when an eviction requests relates to a volume that is not in cache
2024-11-25 17:32:00 +01:00
Carl Johnsen
7c79535572
VolumeDownloader now deletes temporary volumes from both cache and disk when they are no longer needed during restore.
2024-11-25 15:16:03 +01:00
Carl Johnsen
0cd9374bed
Added comments and docstrings
2024-11-22 15:26:04 +01:00
Carl Johnsen
890fa2ce52
Add copyright notices
2024-11-22 11:47:07 +01:00
Carl Johnsen
7717f32e77
Added tracking of errors on remote files during restoration
2024-11-21 20:55:20 +01:00
Carl Johnsen
09132cd509
Changed the VolumeDownloader to use prepared statements
2024-11-19 10:34:13 +01:00
Carl Johnsen
f228cc0791
Added logging and error handling to VolumeDownloader
2024-11-15 14:21:41 +01:00
Carl Johnsen
effd50985f
The real deadlock on process network initialization was found: the volume downloader did not write asynchronously, which would result it waiting on writing to a process that would never be created.
2024-11-15 12:57:55 +01:00
Carl Johnsen
62e96935ad
Added TODOs for future fixes
2024-11-15 08:57:57 +01:00
Carl Johnsen
dc73380fe8
To minimize database requests (or at least where they emerge), query the database once per block per file, once per newly downloaded volume, and once to build the file list. The results are then "shipped" around the network along with the previous messages.
2024-11-13 16:55:47 +01:00
Carl Johnsen
66df6afc34
Changed the downloader to move the download waiting to the decrypter
2024-11-13 14:31:37 +01:00
Carl Johnsen
284755c6b1
Rather than uncompressing everything, only decompress the requested block.
2024-11-12 13:47:25 +01:00
Carl Johnsen
09e04e76d0
Now the restore chain also forwards the block id
2024-11-12 13:45:32 +01:00
Carl Johnsen
6f354e6d08
Added some progress tracking and removed some debug printing
2024-11-08 15:24:56 +01:00
Carl Johnsen
ec738f4063
Added handling of retired exception to close down the process network
2024-11-08 14:35:44 +01:00
Carl Johnsen
f6839160f2
Channels now send the volume / block id around alongside the volume / block
2024-11-08 10:42:54 +01:00
Carl Johnsen
5955cbc036
Modified the VolumeDownloader so that it works
2024-11-08 09:13:54 +01:00
Carl Johnsen
b5745a3be0
Added initial volume downloader process
2024-11-07 10:04:52 +01:00
Carl Johnsen