Carl-Johannes Johnsen and GitHub
476774640a
Merge pull request #5991 from carljohnsen/performance/restoreopt2
2025-02-28 13:03:25 +01:00
Kenneth Skovhede
f77e529eb6
Merge commit '2339aeff28adba15442d0be60c56568ae642ad0b' into feature/make-backends-async
2025-02-26 15:23:48 +01:00
Carl Johnsen
c5ebf4fa71
Added the restore channel depth option
2025-02-25 16:19:04 +01:00
Carl Johnsen
9919d86e9b
Updated VolumeDownloader to use the new BackendManager.GetDirectAsync() method
2025-02-19 16:12:42 +01:00
Carl Johnsen
2c06820281
Made the BackendManager.GetOperations' decryption methods static and public so the restores VolumeDecryptor can use the same code.
2025-02-19 16:09:32 +01:00
Carl Johnsen
a0d5096ebc
Reworked the "Volume loop" of the restore network to handle Backends themselves, rather than using the BackendManager.
...
This allows for tighter control over which process performs which action, allowing them more easily to run in parallel.
2025-02-18 09:25:20 +01:00
Carl Johnsen
ce60b807c8
Added additional profiling logging
2025-02-18 09:18:46 +01:00
Kenneth Skovhede
7031bc57b6
Adding ConfigureAwait(false) to all async enumerations
2025-02-18 09:18:36 +01:00
Carl Johnsen
93756cd74b
Whitespace fixes
2025-02-18 09:17:08 +01:00
Kenneth Skovhede and GitHub
d99f74ed94
Rewrite the backend manager. ( #5896 )
...
* Rewrite the backend manager.
This creates a new more logic backend manager that handles all backend operations.
For each top-level operation, there is now a single backend manager instance that is passed to sub-commands.
The internals of the backend manager are now rewritten to use async logic instead of threading.
The design uses a single task runner that dispatches operations and honors concurrent settings and limits from one place.
The logic for each operation has been moved into a separate files/classes to make it easier to understand each operation.
This fixes #5804
* Fixed a few issues with not awaiting tasks
* Fixed not creating empty databases
* Fixed an issue with hashes not being recorded on download
* Reworked the download logic to avoid attempting to decrypt the file if it has been modified.
* Review fixes
* Renamed db collector to better reflect the purpose.
* Review fixes
2025-01-28 08:54:50 +01:00
Kenneth Skovhede and GitHub
9464caf622
Feature/update license 2025 ( #5851 )
...
* Fixed some minor whitespace issues
* Updated all copyright to 2025
2025-01-07 09:40:39 +01:00
Carl Johnsen
88846251b1
Restoring a symlink that already exists triggers an overwrite of the file, which should count towards number of files restored.
2025-01-06 08:14:32 +01:00
Kenneth Skovhede and GitHub
ff1b2b7990
Simplified task control ( #5753 )
...
* Simplified task control by having all logic in a single class and using async control mechanism
* Fixed the tests
* Made the marking of a backup partial explicitly communicated if the enumeration process is stopped.
* Added support for pausing active transfers
* Introduced transfer token to abort operations.
* Updated the way LiveControl is emitting events.
* Retains the paused state of the server across reboots, this fixes #5760 .
* Reworked the way throttle speeds are propagated through LiveControls.
* Added option to pause transfers in the UI.
* Removed throttle settings from the LiveControls class so it is exclusively handled by the ApplicationSettings.
* Removed `thread-priority` as it does not work at all when tasks are used.
* Fixed the stop dialog and API to only support stop and abort.
* Removed unused variables
2024-12-18 08:27:59 +01:00
Carl Johnsen
c13482cf4c
Removed unnecessary CoCoL channel scope
2024-12-10 10:28:44 +01:00
Carl Johnsen
5bc12e3d46
Changed the reporting of broken local and remote files during restore to only report the first 10 filenames.
2024-12-10 06:57:56 +01:00
Carl Johnsen
ce9a85c3ff
Reintroduced comment and version fix, which shouldn't have been removed.
2024-12-10 06:43:51 +01:00
Carl Johnsen
2d6e1d7ff7
Changed the new restore version to use explicit CoCoL channels
2024-12-10 04:10:34 +01:00
Carl Johnsen
7b12eb8a67
When deleting a symlink, we should check whether it is a file or a directory, as each have their own function call
2024-12-05 16:39:01 +01:00
Carl Johnsen
5680fa3e15
If the restore operation threw an exception, the database wouldn't be closed properly, leading to subsequent tests failing due to the file being locked.
2024-12-05 08:49:36 +01:00
Carl Johnsen
0e36c4393a
Added progress printing during restore
2024-12-03 10:35:33 +01:00
Carl Johnsen
2b0dc0613f
Added handling of existing symlinks
2024-12-03 10:04:08 +01:00
Carl Johnsen
c2e7275e1e
Removed unused metadatastorage in the new restore flow
2024-12-03 09:39:46 +01:00
Carl Johnsen
f0c3fddfa2
Found runaway TODO
2024-12-03 09:11:42 +01:00
Carl Johnsen
1e3e67c053
Update the construction of the processes within RestoreHandler
2024-12-02 20:48:20 +01:00
Carl Johnsen
6c66fee921
Cleanup FileProcessor and rework docstrings
2024-12-02 20:27:21 +01:00
Carl Johnsen
e7b2338214
Renamed VolumeCache to VolumeManager for consistency
2024-12-02 15:43:52 +01:00
Carl Johnsen
7aa19f851e
Add configurable volume downloaders for restore process
2024-12-02 15:16:21 +01:00
Carl Johnsen
533088684b
Consistent naming of decryptors
2024-12-02 15:12:50 +01:00
Carl Johnsen
b6ed72bd6c
Reworked the restore CSP network
2024-12-02 15:09:23 +01:00
Carl Johnsen
cdf0c28fd8
Added internal profiling as an option. Currently only implemented in the restore process.
2024-11-29 05:49:03 +01:00
Carl Johnsen
7aa9db533e
Added preliminary metadata restoration
2024-11-26 20:01:19 +01:00
Carl Johnsen
823b896da1
Removed redundant TODOs
2024-11-25 15:23:59 +01:00
Carl Johnsen
5625c4c1ce
Add configurable options for scaling the number of restore file processors, decryptors, and decompressors
2024-11-22 16:53:16 +01:00
Carl Johnsen
fe6f91dc9a
Add option for using legacy restore method
2024-11-22 16:40:54 +01:00
Carl Johnsen
de24a9f15a
Added the option to control the size of the data block cache during restore
2024-11-22 16:24:27 +01:00
Carl Johnsen
0cd9374bed
Added comments and docstrings
2024-11-22 15:26:04 +01:00
Carl Johnsen
890fa2ce52
Add copyright notices
2024-11-22 11:47:07 +01:00
Carl Johnsen
c79793dbe2
Using the field result rather than passing it as an argument
2024-11-22 11:09:47 +01:00
Carl Johnsen
1fc3212144
Added logging of the local and remote files thay may fail during restoration
2024-11-22 11:09:17 +01:00
Carl Johnsen
b18504864f
Changed to simple using statement on metadatastorage
2024-11-21 20:56:12 +01:00
Carl Johnsen
7717f32e77
Added tracking of errors on remote files during restoration
2024-11-21 20:55:20 +01:00
Carl Johnsen
0d7b9403c9
The SleepableDictionary now keeps track of which blocks to restore, so that it only caches the blocks that needs to be cached.
2024-11-19 10:35:57 +01:00
Carl Johnsen
878780dfe4
Added Database.ExecuteReaderEnumerable for prepared statements
2024-11-19 10:32:54 +01:00
Carl Johnsen
c1751879b2
Started restructuring to return to the original structure and filelister shouldn't be doing the prepatory database work.
2024-11-15 20:32:46 +01:00
Carl Johnsen
effd50985f
The real deadlock on process network initialization was found: the volume downloader did not write asynchronously, which would result it waiting on writing to a process that would never be created.
2024-11-15 12:57:55 +01:00
Carl Johnsen
d65937bab7
Restructuring process network generation to allow for easier debugging.
2024-11-15 08:53:54 +01:00
Carl Johnsen
dc73380fe8
To minimize database requests (or at least where they emerge), query the database once per block per file, once per newly downloaded volume, and once to build the file list. The results are then "shipped" around the network along with the previous messages.
2024-11-13 16:55:47 +01:00
Carl Johnsen
569e927e13
Added an explicit wait for backend to complete after running the restore
2024-11-13 14:27:46 +01:00
Carl Johnsen
c5eaf409c9
Added preliminary scaling parameters
2024-11-13 14:27:19 +01:00
Carl Johnsen
afa2e9bf07
Added more progress keeping and running the original verification step.
2024-11-08 16:05:57 +01:00
