 Carl Johnsen
|
fe6f91dc9a
|
Add option for using legacy restore method
|
2024-11-22 16:40:54 +01:00 |
|
|
Carl Johnsen
|
5436b2f9f5
|
Add eviction ratio option for restore cache management
|
2024-11-22 16:32:53 +01:00 |
|
|
Carl Johnsen
|
de24a9f15a
|
Added the option to control the size of the data block cache during restore
|
2024-11-22 16:24:27 +01:00 |
|
|
Carl Johnsen
|
0cd9374bed
|
Added comments and docstrings
|
2024-11-22 15:26:04 +01:00 |
|
|
Carl Johnsen
|
890fa2ce52
|
Add copyright notices
|
2024-11-22 11:47:07 +01:00 |
|
|
Carl Johnsen
|
43a392f139
|
The tempfiletable name is also required externally from the database.
|
2024-11-22 11:26:25 +01:00 |
|
|
Carl Johnsen
|
7094786e23
|
Removed unnessecarily complex nested ternary statements.
|
2024-11-22 11:26:01 +01:00 |
|
|
Carl Johnsen
|
c798910658
|
If the target file already exists and is verified, do nothing more
|
2024-11-22 11:25:35 +01:00 |
|
|
Carl Johnsen
|
393cfcf5b2
|
Broken files tracker wasn't initialized
|
2024-11-22 11:24:55 +01:00 |
|
|
Carl Johnsen
|
c79793dbe2
|
Using the field result rather than passing it as an argument
|
2024-11-22 11:09:47 +01:00 |
|
|
Carl Johnsen
|
1fc3212144
|
Added logging of the local and remote files thay may fail during restoration
|
2024-11-22 11:09:17 +01:00 |
|
|
Carl Johnsen
|
8a1bf22e27
|
Shutting down and cleaning up the block directory could be done in an incorrect order where some task was trying to access the removed table.
Disposing has now been moved to after all of the accessors have finished.
|
2024-11-22 07:05:20 +01:00 |
|
|
Carl Johnsen
|
8c4f5cebf0
|
The size of the channels are being used by the FileProcessor so that it knows how much it can burst requests.
|
2024-11-21 20:57:01 +01:00 |
|
|
Carl Johnsen
|
b18504864f
|
Changed to simple using statement on metadatastorage
|
2024-11-21 20:56:12 +01:00 |
|
|
Carl Johnsen
|
7717f32e77
|
Added tracking of errors on remote files during restoration
|
2024-11-21 20:55:20 +01:00 |
|
|
Carl Johnsen
|
cacc5d18a0
|
Added tracking of errors of local files during restoration
|
2024-11-21 15:14:29 +01:00 |
|
|
Carl Johnsen
|
916d35e928
|
Added logic for utilizing the original data when restoring, if it matches the hashes in the database.
|
2024-11-21 10:39:48 +01:00 |
|
|
Carl Johnsen
|
1cbed23603
|
Ordering by file size ensures that we start with the files that'll take the longest
|
2024-11-21 10:38:47 +01:00 |
|
|
Carl Johnsen
|
b9a4b8c934
|
Moved local file verification to its own function
|
2024-11-21 07:19:13 +01:00 |
|
|
Carl Johnsen
|
d109bf5db1
|
The FileProcessor now verifies whether the target file has already been restored.
|
2024-11-20 16:33:33 +01:00 |
|
|
Carl Johnsen
|
9955647768
|
Changed the bursting strategy of the FileProcessor to fill up the pipeline, and then issue a new request after every response
|
2024-11-19 11:34:10 +01:00 |
|
|
Carl Johnsen
|
0d7b9403c9
|
The SleepableDictionary now keeps track of which blocks to restore, so that it only caches the blocks that needs to be cached.
|
2024-11-19 10:35:57 +01:00 |
|
|
Carl Johnsen
|
09132cd509
|
Changed the VolumeDownloader to use prepared statements
|
2024-11-19 10:34:13 +01:00 |
|
|
Carl Johnsen
|
4d368f5a17
|
Changed the FileProcessor to use prepared statements
|
2024-11-19 10:33:50 +01:00 |
|
|
Carl Johnsen
|
878780dfe4
|
Added Database.ExecuteReaderEnumerable for prepared statements
|
2024-11-19 10:32:54 +01:00 |
|
|
Carl Johnsen
|
c1751879b2
|
Started restructuring to return to the original structure and filelister shouldn't be doing the prepatory database work.
|
2024-11-15 20:32:46 +01:00 |
|
|
Carl Johnsen
|
75bda5f6c7
|
Added dryrun logic and logging to the restore processes
|
2024-11-15 19:02:30 +01:00 |
|
|
Carl Johnsen
|
d47d775a34
|
Added logging and error handling to BlockManager
|
2024-11-15 15:00:27 +01:00 |
|
|
Carl Johnsen
|
5eb6f63c6b
|
Added logging and error handling to FileLister
|
2024-11-15 14:57:36 +01:00 |
|
|
Carl Johnsen
|
339e06ce9c
|
Added logging and error handling to FileProcessor
|
2024-11-15 14:50:52 +01:00 |
|
|
Carl Johnsen
|
0cee612255
|
Added logging and error handling to VolumeDecrypter
|
2024-11-15 14:22:58 +01:00 |
|
|
Carl Johnsen
|
f228cc0791
|
Added logging and error handling to VolumeDownloader
|
2024-11-15 14:21:41 +01:00 |
|
|
Carl Johnsen
|
4f1869557d
|
Added logging to VolumeDecompressor
|
2024-11-15 14:18:24 +01:00 |
|
|
Carl Johnsen
|
b7aa1a3856
|
Added on-the-fly file hash verification
|
2024-11-15 13:50:45 +01:00 |
|
|
Carl Johnsen
|
9a8e3ab90c
|
Added on-the-fly block hash verification
|
2024-11-15 13:50:13 +01:00 |
|
|
Carl Johnsen
|
5fcdaf334a
|
FileProcessor now bursts their block requests
|
2024-11-15 13:40:21 +01:00 |
|
|
Carl Johnsen
|
effd50985f
|
The real deadlock on process network initialization was found: the volume downloader did not write asynchronously, which would result it waiting on writing to a process that would never be created.
|
2024-11-15 12:57:55 +01:00 |
|
|
Carl Johnsen
|
62e96935ad
|
Added TODOs for future fixes
|
2024-11-15 08:57:57 +01:00 |
|
|
Carl Johnsen
|
a2ca7239b6
|
Split up statement to allow for easier debugging.
|
2024-11-15 08:56:58 +01:00 |
|
|
Carl Johnsen
|
b30f19929b
|
Rather than creating a buffer and then resize to a specific size, just create it with the target size
|
2024-11-15 08:56:15 +01:00 |
|
|
Carl Johnsen
|
0e2be45418
|
Changed the shutdown strategy of BlockManager as it could lead to a deadlock since the channels weren't retired properly.
|
2024-11-15 08:55:13 +01:00 |
|
|
Carl Johnsen
|
d65937bab7
|
Restructuring process network generation to allow for easier debugging.
|
2024-11-15 08:53:54 +01:00 |
|
|
Carl Johnsen
|
dc73380fe8
|
To minimize database requests (or at least where they emerge), query the database once per block per file, once per newly downloaded volume, and once to build the file list. The results are then "shipped" around the network along with the previous messages.
|
2024-11-13 16:55:47 +01:00 |
|
|
Carl Johnsen
|
66df6afc34
|
Changed the downloader to move the download waiting to the decrypter
|
2024-11-13 14:31:37 +01:00 |
|
|
Carl Johnsen
|
d0639ee0ae
|
Fixed bug that would leave the restore database in a locked state
|
2024-11-13 14:30:35 +01:00 |
|
|
Carl Johnsen
|
569e927e13
|
Added an explicit wait for backend to complete after running the restore
|
2024-11-13 14:27:46 +01:00 |
|
|
Carl Johnsen
|
c5eaf409c9
|
Added preliminary scaling parameters
|
2024-11-13 14:27:19 +01:00 |
|
|
Carl Johnsen
|
284755c6b1
|
Rather than uncompressing everything, only decompress the requested block.
|
2024-11-12 13:47:25 +01:00 |
|
|
Carl Johnsen
|
09e04e76d0
|
Now the restore chain also forwards the block id
|
2024-11-12 13:45:32 +01:00 |
|
|
Carl Johnsen
|
b686cd4934
|
Changed from ConcurrentDictionary to MemoryCache, to utilize their cache eviction logic.
|
2024-11-11 13:02:24 +01:00 |
|