Commit Graph
356 Commits
Author SHA1 Message Date
Carl-Johannes JohnsenandGitHub 476774640a Merge pull request #5991 from carljohnsen/performance/restoreopt2 2025-02-28 13:03:25 +01:00
Carl Johnsen 260786a91a Merge branch 'feature/restore-buffer-size-option' into performance/restoreopt2 2025-02-27 14:03:01 +01:00
Carl Johnsen 038e78ad06 The block cache is now always present, as it keeps track of blocks. In case of a cache size 0, every Set() will trigger an evict. 2025-02-27 11:17:11 +01:00
Carl Johnsen 6d7faf97bb Whitespace 2025-02-25 17:41:57 +01:00
Carl Johnsen 42ec5b633c Restore block cache max option didn't match the description 2025-02-25 17:41:35 +01:00
Carl Johnsen c5ebf4fa71 Added the restore channel depth option 2025-02-25 16:19:04 +01:00
Carl Johnsen 123c12e2c3 Whitespace 2025-02-25 16:18:13 +01:00
Kenneth Skovhede 26bbe6a577 Added setup step for source providers.
This enables a fail-early approach, where the remote source loads the initial results.
By storing the `IAsyncEnumerable` it is possible to not perform any additional remote calls, even with the initial probing call.

This makes it possible to detect the absence of sources, so the backup can be aborted if sources are missing.
2025-02-24 12:50:58 +01:00
Kenneth Skovhede cf5b4bb24d Added support for Wmic based Windows VSS snapshots.
Refactored code a bit to be easier to understand.
2025-02-19 14:00:14 +01:00
Kenneth Skovhede fac12ba772 Further consolidated the data folder.
Added support for `--portable-mode` and `--server-datafolder` for the CLI, ServerUtil, and Agent.

Fixed a few places where file reads were not cached properly.

Moved some responsibilities of UpdaterManager into DataFolderManager.

This now supports storing `machineid.txt` and `installation.txt` inside the path pointed to by `--server-datafolder` or alternatively with `--portable-mode`.

It is possible that some fringe backwards compatibility is lost with this update.

Fixed an issue with AutoUpdater crashing on check/download.

This fixes #5902
2025-01-28 13:01:35 +01:00
Kenneth SkovhedeandGitHub 610f3b47d1 Update reporting module (#5910)
* Updated reporting values.

The reporter module can now emit the following extra values:
- operating-system
- installation-type
- destination-type
- next-scheduled-run

When running from the built-in scheduler, it will automatically set `--next-scheduled-run` so it is applied to the reporting data.

* Implemented an improved machine name reader.
This fixes #5907

* Added a guard against sending too much of the destination url as the destination.

* Fixed reading Windows Machine name

* Removed extra Windows logic as it has no effect
2025-01-28 08:56:42 +01:00
Kenneth SkovhedeandGitHub 9464caf622 Feature/update license 2025 (#5851)
* Fixed some minor whitespace issues

* Updated all copyright to 2025
2025-01-07 09:40:39 +01:00
Carl Johnsen 6afe959c56 Changed the default number of file processors to match the other concurrent counts 2024-12-23 06:50:24 +01:00
Carl Johnsen a566d1a49f Merge remote-tracking branch 'upstream/master' into fileprocessors 2024-12-23 06:45:43 +01:00
Kenneth SkovhedeandGitHub 86f3f6b178 Fixed issues with timeout in msgraph (#5790)
* Fixed issues with timeout in msgraph

Fixed a place where the cancellation token was not applied.

Fixed a problem with msgraph not sending correct DNS information.

Fixed a problem with msgraph downloading the entire file before streaming it, bypassing throttle and tripping the timeout.

Added `ITimeoutExemptBackend` to selectively disable timeouts for backends that do not support it.

Updated the backendmanager to support `ITimeoutExemptBackend`.

Updated the backendmanager to abort as soon as a timeout happens.

* Updated BackendTester to use the `TimeoutObservingStream` and support the `--read-write-timeout` setting.

Updated backend tester code to use `var`

* Fixed a bug reading the `read-write-timeout`

* Updated to using constants with names instead of `-1`.
2024-12-19 09:31:06 +01:00
Kenneth SkovhedeandGitHub ff1b2b7990 Simplified task control (#5753)
* Simplified task control by having all logic in a single class and using async control mechanism
* Fixed the tests
* Made the marking of a backup partial explicitly communicated if the enumeration process is stopped.
* Added support for pausing active transfers
* Introduced transfer token to abort operations.
* Updated the way LiveControl is emitting events.
* Retains the paused state of the server across reboots, this fixes #5760.
* Reworked the way throttle speeds are propagated through LiveControls.
* Added option to pause transfers in the UI.
* Removed throttle settings from the LiveControls class so it is exclusively handled by the ApplicationSettings.
* Removed `thread-priority` as it does not work at all when tasks are used.
* Fixed the stop dialog and API to only support stop and abort.
* Removed unused variables
2024-12-18 08:27:59 +01:00
Kenneth Skovhede 5363bf0d96 Implemented read-write timeout.
This fixes #5725
2024-12-10 22:12:20 +01:00
Carl Johnsen 95887b5841 Set the default cache eviction ratio to be 4gb.
Also set the default value to be parsed from a string.
2024-12-10 06:02:51 +01:00
Carl Johnsen 405e73b2d2 Changed the 'restore-cache-evict' option to be an integer representing percentage instead of a float between 0 and 1 2024-12-10 05:38:57 +01:00
Carl Johnsen f491fa93fc Changed restore scaling parameters to use the default value to produce the default string, rather than a string literal. 2024-12-10 05:32:40 +01:00
Carl Johnsen 9440565706 Changed from hardcoded restore scaling constants to scale with the number of processors on the target machine. 2024-12-10 05:17:59 +01:00
Carl Johnsen 345701576c Merge branch 'master' into performance/restoreopt 2024-12-05 12:33:30 +01:00
Kenneth Skovhede f5cabd4c7f Remove --use-block-cache
The option has no effect.
This closes #3566
The block cache was added originally to work around perceived slow hash table lookups in the database.
By having an in-memory dictionary, lookups would be much faster.
It has no effect on disk wear, as the lookups are read-only.
The impact of such a cache was measured to be minimal with recent versions of SQLite.
2024-12-04 10:06:49 +01:00
Carl Johnsen 3e7b505ed2 Merge branch 'master' into performance/restoreopt 2024-12-03 09:21:14 +01:00
Carl Johnsen 76240550ed Add preallocation option for restored files and update related documentation 2024-12-03 09:10:36 +01:00
Carl Johnsen 7aa19f851e Add configurable volume downloaders for restore process 2024-12-02 15:16:21 +01:00
Carl Johnsen 533088684b Consistent naming of decryptors 2024-12-02 15:12:50 +01:00
Carl Johnsen cdf0c28fd8 Added internal profiling as an option. Currently only implemented in the restore process. 2024-11-29 05:49:03 +01:00
Carl Johnsen c26c54c5f7 Fixed naming inconsistencies in the new restore options 2024-11-25 15:17:10 +01:00
Carl Johnsen 5625c4c1ce Add configurable options for scaling the number of restore file processors, decryptors, and decompressors 2024-11-22 16:53:16 +01:00
Carl Johnsen fe6f91dc9a Add option for using legacy restore method 2024-11-22 16:40:54 +01:00
Carl Johnsen 5436b2f9f5 Add eviction ratio option for restore cache management 2024-11-22 16:32:53 +01:00
Carl Johnsen de24a9f15a Added the option to control the size of the data block cache during restore 2024-11-22 16:24:27 +01:00
Kenneth Skovhede d2f615f442 Added possible values to cpu-intensity for better documentation 2024-11-07 08:10:32 +01:00
Kenneth Skovhede a72bf27aa3 Merge remote-tracking branch 'origin/master' into feature/add-secret-provider 2024-11-05 11:48:30 +01:00
Carl Johnsen 8927dbb81a Added the commandline option for controlling the CPU intensity 2024-11-01 13:32:18 +01:00
Kenneth Skovhede 9a89ca5a15 Cleaned up the use of the default pattern to use a single const value.
Added support for inheriting the secret pattern as it is more logical that all settings are carried through to the application.
2024-10-26 08:37:26 +02:00
Kenneth Skovhede 4b84135622 Merge remote-tracking branch 'origin/master' into feature/add-secret-provider 2024-10-24 16:11:41 +02:00
Kenneth Skovhede 34d13a20e2 Initial version of the external secret provider 2024-10-24 16:06:59 +02:00
Kenneth Skovhede f251d34b57 Implemented support for including the machine hostname in reports for simpler identification. 2024-10-16 16:22:47 +02:00
Kenneth SkovhedeandGitHub dec473eab3 Merge pull request #5588 from marceloduplicati/feature/webdav-ignore-invalid-certificates
Solves Webdav issue with self signed certificates and accepts appropriate options
2024-10-15 11:10:25 +02:00
Marcelo Ceccon fd591d9af2 Fix to Webdav issue around certificate validation
Fix to Webdav to process the options accept-specified-ssl-hash and accept-any-ssl-certificate

Minor refactoring
2024-10-10 10:36:47 +02:00
Kenneth Skovhede 91ea81f537 Added commandline option to toggle locks 2024-10-08 09:35:00 +02:00
Kenneth Skovhede 3c144f5481 Merge remote-tracking branch 'origin/master' into feature/remove-thread-culture-dependence 2024-10-02 14:56:11 +02:00
Kenneth Skovhede 627bccbb94 Removed piped download as it was slower than not piping.
It can be re-added later if a better way is found.
2024-09-26 08:43:01 +02:00
Kenneth Skovhede f3e2a0a902 Avoid setting Thread.CurrentCulture
Avoids setting deprecated `Thread.CurrentCulture` and `Thread.CurrentUICulture`, using `CultureInfo.CurrentCulture` instead.
Fixed a few legacy places where incorrect string compares were used.
2024-09-18 09:24:55 +02:00
Suguru Hirahara 13ca8f6fca Add ConsoleloglevelLong to console-log-level 2024-07-30 02:39:00 -04:00
Suguru Hirahara 6a7bf8fd34 Add LogfileloglevelLong to log-file-log-level 2024-07-30 02:38:52 -04:00
Suguru Hirahara ed0a9c746b Add ListverifyuploadsLong to list-verify-uploads 2024-07-30 02:38:39 -04:00
Suguru Hirahara 5b65562ea6 Update strings to indicate deprecation
- DisablefilepathcacheDeprecated
- FilereadbuffersizeDeprecated

This also makes "DEPRECATED: " translatable.
2024-07-24 00:56:05 -04:00