Commit Graph
2233 Commits
Author SHA1 Message Date
Carl Johnsen 6afe959c56 Changed the default number of file processors to match the other concurrent counts 2024-12-23 06:50:24 +01:00
Carl Johnsen ec55580f5c Updated the call signature of FileBlockProcessor.Run() to match the new format 2024-12-23 06:49:04 +01:00
Carl Johnsen a566d1a49f Merge remote-tracking branch 'upstream/master' into fileprocessors 2024-12-23 06:45:43 +01:00
Kenneth SkovhedeandGitHub 161b5708d4 Added missing timeouts to uploads. (#5803)
* Added missing timeouts to uploads.

Since much of the BackendManager is replicated into the BackendUploader, this code needs to be updated as well.

* Fixed namespace issue
2024-12-20 21:03:48 +01:00
Kenneth SkovhedeandGitHub 86f3f6b178 Fixed issues with timeout in msgraph (#5790)
* Fixed issues with timeout in msgraph

Fixed a place where the cancellation token was not applied.

Fixed a problem with msgraph not sending correct DNS information.

Fixed a problem with msgraph downloading the entire file before streaming it, bypassing throttle and tripping the timeout.

Added `ITimeoutExemptBackend` to selectively disable timeouts for backends that do not support it.

Updated the backendmanager to support `ITimeoutExemptBackend`.

Updated the backendmanager to abort as soon as a timeout happens.

* Updated BackendTester to use the `TimeoutObservingStream` and support the `--read-write-timeout` setting.

Updated backend tester code to use `var`

* Fixed a bug reading the `read-write-timeout`

* Updated to using constants with names instead of `-1`.
2024-12-19 09:31:06 +01:00
Kenneth SkovhedeandGitHub ff1b2b7990 Simplified task control (#5753)
* Simplified task control by having all logic in a single class and using async control mechanism
* Fixed the tests
* Made the marking of a backup partial explicitly communicated if the enumeration process is stopped.
* Added support for pausing active transfers
* Introduced transfer token to abort operations.
* Updated the way LiveControl is emitting events.
* Retains the paused state of the server across reboots, this fixes #5760.
* Reworked the way throttle speeds are propagated through LiveControls.
* Added option to pause transfers in the UI.
* Removed throttle settings from the LiveControls class so it is exclusively handled by the ApplicationSettings.
* Removed `thread-priority` as it does not work at all when tasks are used.
* Fixed the stop dialog and API to only support stop and abort.
* Removed unused variables
2024-12-18 08:27:59 +01:00
Kenneth Skovhede 20a6973ffd Updated documentation for --synchronous-upload.
This fixes #4207
2024-12-17 12:13:34 +01:00
Kenneth Skovhede 5363bf0d96 Implemented read-write timeout.
This fixes #5725
2024-12-10 22:12:20 +01:00
Carl Johnsen c13482cf4c Removed unnecessary CoCoL channel scope 2024-12-10 10:28:44 +01:00
Carl Johnsen 5bc12e3d46 Changed the reporting of broken local and remote files during restore to only report the first 10 filenames. 2024-12-10 06:57:56 +01:00
Carl Johnsen eed626298d Changed the reporting of broken files to be a filebackedlist, to ensure that there's enough memory in case of a massive failure. 2024-12-10 06:44:55 +01:00
Carl Johnsen ce9a85c3ff Reintroduced comment and version fix, which shouldn't have been removed. 2024-12-10 06:43:51 +01:00
Carl Johnsen ee8a82c230 Moved logging outside of the lock on blockcount in restore block manager 2024-12-10 06:14:55 +01:00
Carl Johnsen caeb119a5b Removed incorrect hints about default parameters 2024-12-10 06:08:02 +01:00
Carl Johnsen f3afd4c682 Reintroduced the restore block cache eviction logic 2024-12-10 06:03:08 +01:00
Carl Johnsen 95887b5841 Set the default cache eviction ratio to be 4gb.
Also set the default value to be parsed from a string.
2024-12-10 06:02:51 +01:00
Carl Johnsen 405e73b2d2 Changed the 'restore-cache-evict' option to be an integer representing percentage instead of a float between 0 and 1 2024-12-10 05:38:57 +01:00
Carl Johnsen f491fa93fc Changed restore scaling parameters to use the default value to produce the default string, rather than a string literal. 2024-12-10 05:32:40 +01:00
Carl Johnsen 9440565706 Changed from hardcoded restore scaling constants to scale with the number of processors on the target machine. 2024-12-10 05:17:59 +01:00
Carl Johnsen 1a2f58b6c8 Merge branch 'master' into performance/restoreopt 2024-12-10 04:14:07 +01:00
Carl Johnsen 7f37c49292 Changed the VolumeManager to use a single input channel of type object, and then switching on the message received, rather than ReadFromAny().
Because it seems like the amount of readers in queue for one of the channels became much higher (70) than the actual number of readers (1).
2024-12-10 04:12:33 +01:00
Carl Johnsen 2d6e1d7ff7 Changed the new restore version to use explicit CoCoL channels 2024-12-10 04:10:34 +01:00
Carl Johnsen b2288cb804 Removed named CoCoL channels in favor of passing around instances. 2024-12-09 13:55:22 +01:00
Carl Johnsen e642ef8bef Merge branch 'master' into performance/restoreopt 2024-12-06 09:22:27 +01:00
Carl Johnsen 0c58de82b5 Added a check for whether the file exists, rather than throwing exceptions. 2024-12-05 16:42:09 +01:00
Carl Johnsen 4a523efd24 Sometimes, the temporary empty volumes aren't removed from the database 2024-12-05 16:41:28 +01:00
Carl Johnsen 7b12eb8a67 When deleting a symlink, we should check whether it is a file or a directory, as each have their own function call 2024-12-05 16:39:01 +01:00
Carl Johnsen 345701576c Merge branch 'master' into performance/restoreopt 2024-12-05 12:33:30 +01:00
Carl Johnsen 5680fa3e15 If the restore operation threw an exception, the database wouldn't be closed properly, leading to subsequent tests failing due to the file being locked. 2024-12-05 08:49:36 +01:00
Carl Johnsen ab7fcb2e4a Opening a filestream doesn't work as expected on windows (caught in ProblematicPathTests) - changed to SystemIO.IO_OS.FileOpen* 2024-12-05 08:48:54 +01:00
Kenneth Skovhede f5cabd4c7f Remove --use-block-cache
The option has no effect.
This closes #3566
The block cache was added originally to work around perceived slow hash table lookups in the database.
By having an in-memory dictionary, lookups would be much faster.
It has no effect on disk wear, as the lookups are read-only.
The impact of such a cache was measured to be minimal with recent versions of SQLite.
2024-12-04 10:06:49 +01:00
Carl Johnsen 0e36c4393a Added progress printing during restore 2024-12-03 10:35:33 +01:00
Carl Johnsen 2b0dc0613f Added handling of existing symlinks 2024-12-03 10:04:08 +01:00
Carl Johnsen c2e7275e1e Removed unused metadatastorage in the new restore flow 2024-12-03 09:39:46 +01:00
Carl Johnsen 3e7b505ed2 Merge branch 'master' into performance/restoreopt 2024-12-03 09:21:14 +01:00
Carl Johnsen f0c3fddfa2 Found runaway TODO 2024-12-03 09:11:42 +01:00
Carl Johnsen 76240550ed Add preallocation option for restored files and update related documentation 2024-12-03 09:10:36 +01:00
Carl Johnsen 6ba956dfcd Moved all database operations to the LocalRestoreDatabase.
Also changed the old changes, so that the functions related to the legacy restore remains untouched.
2024-12-03 08:35:14 +01:00
Carl Johnsen 6413014c7d Added internal profiling of VolumeManager alongside proper exception handling. 2024-12-02 20:58:46 +01:00
Carl Johnsen 1e3e67c053 Update the construction of the processes within RestoreHandler 2024-12-02 20:48:20 +01:00
Carl Johnsen 99adcdd17c Added license, docstrings and logging to VolumeManager 2024-12-02 20:41:56 +01:00
Carl Johnsen 7436606455 Cleanup VolumeDownloader 2024-12-02 20:32:38 +01:00
Carl Johnsen d0df0c054e Cleanup VolumeDecryptor 2024-12-02 20:29:40 +01:00
Carl Johnsen 1284adec3b Cleanup VolumeDecompressor 2024-12-02 20:28:45 +01:00
Carl Johnsen 6c66fee921 Cleanup FileProcessor and rework docstrings 2024-12-02 20:27:21 +01:00
Carl Johnsen 9070ea2382 Added docstring and cleaned up FileLister 2024-12-02 20:12:06 +01:00
Carl Johnsen a663c46178 Removed debug depth for channels 2024-12-02 20:07:49 +01:00
Carl Johnsen 46983ee5cd Added proper handling of the internal profiling timers in BlockManagers SleepableDictionary 2024-12-02 19:59:56 +01:00
Carl Johnsen 4ae37292a1 Changed the count dictionaries in BlockManager to be normal dictionaries, as they are faster and the accesses to them are serialized anyways. 2024-12-02 17:56:33 +01:00
Carl Johnsen b44934ac4b Removed database from BlockManagers cache 2024-12-02 17:48:29 +01:00