Commit Graph
2357 Commits
Author SHA1 Message Date
Kenneth SkovhedeandGitHub 9464caf622 Feature/update license 2025 (#5851)
* Fixed some minor whitespace issues

* Updated all copyright to 2025
2025-01-07 09:40:39 +01:00
Carl Johnsen 46b05b94ef Limit file name conflict attempts to 100 and log an error if exceeded 2025-01-06 16:01:53 +01:00
Carl Johnsen 6f263ad874 Added a note about potential memory consumption of restores FileProcessor 2025-01-06 16:00:11 +01:00
Carl Johnsen 284b14b72f Merge branch 'master' into bugfix/restore-overwrite 2025-01-06 14:43:03 +01:00
Kenneth SkovhedeandGitHub 2a757dea0f Quota check disabled fix (#5843)
The QuotaCheck is done in two places, and only one place was checking if the quota check was supposed to be disabled.
This adds the check in the other place.
2025-01-06 14:21:53 +01:00
Carl Johnsen 88846251b1 Restoring a symlink that already exists triggers an overwrite of the file, which should count towards number of files restored. 2025-01-06 08:14:32 +01:00
Carl Johnsen 1bea0f3696 Restored empty files should count towards the restoration file count. 2025-01-05 12:46:08 +01:00
Carl Johnsen 4cd565d592 Added handling of already restored timestamp-retargeted files 2025-01-05 12:09:38 +01:00
Carl Johnsen b759b4564d Added docstrings to new functions 2025-01-05 12:09:04 +01:00
Carl Johnsen eb64320a49 Changed the reporting of restored files to match the legacy restore's reporting. 2025-01-05 11:51:52 +01:00
Carl Johnsen 637bfcefc8 Forgot to check whether the file exists 2025-01-05 11:25:19 +01:00
Carl Johnsen eba477c271 Added handling of the 'overwrite' option in the new restore flow 2025-01-05 10:44:08 +01:00
Carl Johnsen f76bf90e68 Added additional dry run logging 2025-01-05 10:43:23 +01:00
Carl Johnsen 8e92754efd VerifyLocalBlocks now keeps track of internally verified blocks for logging 2025-01-05 10:43:01 +01:00
Carl Johnsen be99f03bac VerifyLocalBlocks wasn't respecting dry run properly 2025-01-05 10:41:33 +01:00
Carl Johnsen a14d29da93 Modified VerifyTargetBlocks to also return the blocks that were verified. 2025-01-05 10:40:14 +01:00
Carl Johnsen a7507fb765 Simplified variable assignment 2025-01-05 10:35:50 +01:00
Carl Johnsen 83d1617984 Fixed the IO calls to also work on Windows 2025-01-03 23:27:35 +01:00
Carl Johnsen 52018db0bb Added handling and warning throwing for when the folder for restored files aren't created properly 2025-01-03 20:48:58 +01:00
Carl Johnsen 198e4d8e6e Refactor SQL query formatting 2025-01-03 11:06:49 +01:00
Carl Johnsen 661a73ff2a Merge remote-tracking branch 'upstream/master' into SlowBigRestore 2025-01-03 10:11:50 +01:00
Kenneth SkovhedeandGitHub bc6114323f Workaround for multi-backend. (#5815)
* Workaround for multi-backend.

There is currently an issue with creating the BackendManager which has a backend instance. This instance is not used by the uploader, so they will create new instances.
The BackendManager instance can then time out, but is used later.

This commit will allow the uploader to "borrow" the instance from the backend manager, so only a single instance is needed.

This is not a clean fix, but doing so requires a full rewrite of the backend manager and uploader, so the controller creates a single instance of the manager, which is then used for all operations.

This would allow the backend manager to keep track of instances across all requests.

* Improved reuse after shutdown
2024-12-28 21:25:49 +01:00
Carl Johnsen 6afe959c56 Changed the default number of file processors to match the other concurrent counts 2024-12-23 06:50:24 +01:00
Carl Johnsen ec55580f5c Updated the call signature of FileBlockProcessor.Run() to match the new format 2024-12-23 06:49:04 +01:00
Carl Johnsen a566d1a49f Merge remote-tracking branch 'upstream/master' into fileprocessors 2024-12-23 06:45:43 +01:00
Kenneth SkovhedeandGitHub 161b5708d4 Added missing timeouts to uploads. (#5803)
* Added missing timeouts to uploads.

Since much of the BackendManager is replicated into the BackendUploader, this code needs to be updated as well.

* Fixed namespace issue
2024-12-20 21:03:48 +01:00
Kenneth SkovhedeandGitHub 86f3f6b178 Fixed issues with timeout in msgraph (#5790)
* Fixed issues with timeout in msgraph

Fixed a place where the cancellation token was not applied.

Fixed a problem with msgraph not sending correct DNS information.

Fixed a problem with msgraph downloading the entire file before streaming it, bypassing throttle and tripping the timeout.

Added `ITimeoutExemptBackend` to selectively disable timeouts for backends that do not support it.

Updated the backendmanager to support `ITimeoutExemptBackend`.

Updated the backendmanager to abort as soon as a timeout happens.

* Updated BackendTester to use the `TimeoutObservingStream` and support the `--read-write-timeout` setting.

Updated backend tester code to use `var`

* Fixed a bug reading the `read-write-timeout`

* Updated to using constants with names instead of `-1`.
2024-12-19 09:31:06 +01:00
Kenneth SkovhedeandGitHub ff1b2b7990 Simplified task control (#5753)
* Simplified task control by having all logic in a single class and using async control mechanism
* Fixed the tests
* Made the marking of a backup partial explicitly communicated if the enumeration process is stopped.
* Added support for pausing active transfers
* Introduced transfer token to abort operations.
* Updated the way LiveControl is emitting events.
* Retains the paused state of the server across reboots, this fixes #5760.
* Reworked the way throttle speeds are propagated through LiveControls.
* Added option to pause transfers in the UI.
* Removed throttle settings from the LiveControls class so it is exclusively handled by the ApplicationSettings.
* Removed `thread-priority` as it does not work at all when tasks are used.
* Fixed the stop dialog and API to only support stop and abort.
* Removed unused variables
2024-12-18 08:27:59 +01:00
Kenneth Skovhede 20a6973ffd Updated documentation for --synchronous-upload.
This fixes #4207
2024-12-17 12:13:34 +01:00
Kenneth Skovhede 5363bf0d96 Implemented read-write timeout.
This fixes #5725
2024-12-10 22:12:20 +01:00
Carl Johnsen c13482cf4c Removed unnecessary CoCoL channel scope 2024-12-10 10:28:44 +01:00
Carl Johnsen 5bc12e3d46 Changed the reporting of broken local and remote files during restore to only report the first 10 filenames. 2024-12-10 06:57:56 +01:00
Carl Johnsen eed626298d Changed the reporting of broken files to be a filebackedlist, to ensure that there's enough memory in case of a massive failure. 2024-12-10 06:44:55 +01:00
Carl Johnsen ce9a85c3ff Reintroduced comment and version fix, which shouldn't have been removed. 2024-12-10 06:43:51 +01:00
Carl Johnsen ee8a82c230 Moved logging outside of the lock on blockcount in restore block manager 2024-12-10 06:14:55 +01:00
Carl Johnsen caeb119a5b Removed incorrect hints about default parameters 2024-12-10 06:08:02 +01:00
Carl Johnsen f3afd4c682 Reintroduced the restore block cache eviction logic 2024-12-10 06:03:08 +01:00
Carl Johnsen 95887b5841 Set the default cache eviction ratio to be 4gb.
Also set the default value to be parsed from a string.
2024-12-10 06:02:51 +01:00
Carl Johnsen 405e73b2d2 Changed the 'restore-cache-evict' option to be an integer representing percentage instead of a float between 0 and 1 2024-12-10 05:38:57 +01:00
Carl Johnsen f491fa93fc Changed restore scaling parameters to use the default value to produce the default string, rather than a string literal. 2024-12-10 05:32:40 +01:00
Carl Johnsen 9440565706 Changed from hardcoded restore scaling constants to scale with the number of processors on the target machine. 2024-12-10 05:17:59 +01:00
Carl Johnsen 1a2f58b6c8 Merge branch 'master' into performance/restoreopt 2024-12-10 04:14:07 +01:00
Carl Johnsen 7f37c49292 Changed the VolumeManager to use a single input channel of type object, and then switching on the message received, rather than ReadFromAny().
Because it seems like the amount of readers in queue for one of the channels became much higher (70) than the actual number of readers (1).
2024-12-10 04:12:33 +01:00
Carl Johnsen 2d6e1d7ff7 Changed the new restore version to use explicit CoCoL channels 2024-12-10 04:10:34 +01:00
Carl Johnsen b2288cb804 Removed named CoCoL channels in favor of passing around instances. 2024-12-09 13:55:22 +01:00
Carl Johnsen e642ef8bef Merge branch 'master' into performance/restoreopt 2024-12-06 09:22:27 +01:00
Carl Johnsen 0c58de82b5 Added a check for whether the file exists, rather than throwing exceptions. 2024-12-05 16:42:09 +01:00
Carl Johnsen 4a523efd24 Sometimes, the temporary empty volumes aren't removed from the database 2024-12-05 16:41:28 +01:00
Carl Johnsen 7b12eb8a67 When deleting a symlink, we should check whether it is a file or a directory, as each have their own function call 2024-12-05 16:39:01 +01:00
Carl Johnsen 345701576c Merge branch 'master' into performance/restoreopt 2024-12-05 12:33:30 +01:00