Commit Graph
979 Commits
Author SHA1 Message Date
Carl Johnsen dbd3520d9c Formatting 2025-02-10 12:18:16 +01:00
Kenneth Skovhede 3caaa78336 Merge remote-tracking branch 'origin/master' into feature/alternate-source-provider 2025-02-09 16:15:06 +01:00
Kenneth Skovhede 8c935a7344 This creates a new ISourceProvider interface for describing a data source.
With this interface it is possible to wrap the current support for snapshots and USN and simplify the enumeration.

This makes a number of assumptions into fields such as `IsMetaEntry` and `IsRootEntry` that removes code that was checking for various properties.

This first commit has support for VSS and USN as well as "naked" files, keeping feature parity.

There is also an experimental support for loading a backend-as-a-source. The support for this is currently limited as the backends need to be extended to support a folder concept.
2025-02-09 16:14:22 +01:00
Kenneth Skovhede 536e62275f Changed the logic in recreate to avoid opening the same zip file multiple times. 2025-02-06 21:34:02 +01:00
Kenneth Skovhede 1690871d10 Implemented fix that pushes updated timestamps to previous version, in the case that no files were changed, and no new backup is recorded in remote storage. 2025-02-06 15:31:20 +01:00
Kenneth Skovhede bae82feeae Add a check for stale database
This commit adds a check to verify that the remote database is up-to-date before proceeding to repair the remote storage.

Without this check it is possible to start repairs with an old database, and this will cause the remote data to be deleted.

This fixes #4579
2025-02-06 13:20:32 +01:00
Kenneth Skovhede fac12ba772 Further consolidated the data folder.
Added support for `--portable-mode` and `--server-datafolder` for the CLI, ServerUtil, and Agent.

Fixed a few places where file reads were not cached properly.

Moved some responsibilities of UpdaterManager into DataFolderManager.

This now supports storing `machineid.txt` and `installation.txt` inside the path pointed to by `--server-datafolder` or alternatively with `--portable-mode`.

It is possible that some fringe backwards compatibility is lost with this update.

Fixed an issue with AutoUpdater crashing on check/download.

This fixes #5902
2025-01-28 13:01:35 +01:00
Kenneth SkovhedeandGitHub 039e1a08fb Merge branch 'master' into feature/prevent-windows-subfolder-storage 2025-01-28 08:58:26 +01:00
Kenneth SkovhedeandGitHub d99f74ed94 Rewrite the backend manager. (#5896)
* Rewrite the backend manager.

This creates a new more logic backend manager that handles all backend operations.

For each top-level operation, there is now a single backend manager instance that is passed to sub-commands.

The internals of the backend manager are now rewritten to use async logic instead of threading.

The design uses a single task runner that dispatches operations and honors concurrent settings and limits from one place.

The logic for each operation has been moved into a separate files/classes to make it easier to understand each operation.

This fixes #5804

* Fixed a few issues with not awaiting tasks

* Fixed not creating empty databases

* Fixed an issue with hashes not being recorded on download

* Reworked the download logic to avoid attempting to decrypt the file if it has been modified.

* Review fixes

* Renamed db collector to better reflect the purpose.

* Review fixes
2025-01-28 08:54:50 +01:00
Kenneth Skovhede 27f0bb4ea0 Unified code to find the data folder.
The `machineid.txt` and `installation.txt` files are now default located in same folder where the database is default located.
2025-01-27 22:51:45 +01:00
Kenneth SkovhedeandGitHub 6c493186a7 Changed MissingFileDetected message to Warning (#5893)
* Changed `MissingFileDetected` message to `Warning`

* Updated tests to match errors/warning counts
2025-01-22 21:18:33 +01:00
Kenneth SkovhedeandGitHub 997891525a Updated license-updater to update csproj files (#5898)
* Updated license-updater to update csproj files

* License updates
2025-01-22 21:04:19 +01:00
Carl Johnsen fe7cb13fde Fixed issue with stream not being open in the correct mode during restore 2025-01-17 12:27:05 +01:00
Kenneth SkovhedeandGitHub edd69bfd59 Improve cancel robustness 2 (#5882)
* Attempt to avoid comitting transaction without any queries being made

* Guard internal logic
2025-01-16 16:01:54 +01:00
Kenneth SkovhedeandGitHub 6bbd64a655 Fixed nullability warnings in Utility.cs (#5861)
* Fixed nullability warnings in `Utility.cs`

* Fixed a number of nullability issues and warnings

* More nullable fixes

* Fixed the overwrite issue

* Fixed additional nullability issues

* Removed additional nullability issues

* More nullability fixes and added some xmldoc

* Added more xmldocs and removed an unused argument

* Removed more warnings
2025-01-14 14:03:48 +01:00
Kenneth SkovhedeandGitHub d64c7e721f Handle duplicated missing blocks (#5858)
* Implemented test for #5845

* Removed unittests not executing cleanup.

* Cleanup commit

* Fixed the query that causes the constraint violation
2025-01-10 09:24:54 +01:00
Kenneth SkovhedeandGitHub 382ba172b3 Fixed duplicate entries in index files (#5856)
* Added test to verify that compression modules will mask duplicate entries, if present

* Because there are likely many, otherwise valid, dindex zip archives with duplicated blocklists out there, the log message is now verbose.

To make sure we no longer create archives with duplicated entries, the `--unittest-mode` option makes this a warning that is checked in most tests.

* Added guard in `IndexVolumeCreator` to prevent creating zip files with duplicated block lists.

* Added debug check for duplicates

* Fixed the duplicate test
2025-01-10 09:24:31 +01:00
Kenneth SkovhedeandGitHub 9464caf622 Feature/update license 2025 (#5851)
* Fixed some minor whitespace issues

* Updated all copyright to 2025
2025-01-07 09:40:39 +01:00
Carl Johnsen 46b05b94ef Limit file name conflict attempts to 100 and log an error if exceeded 2025-01-06 16:01:53 +01:00
Carl Johnsen 6f263ad874 Added a note about potential memory consumption of restores FileProcessor 2025-01-06 16:00:11 +01:00
Carl Johnsen 284b14b72f Merge branch 'master' into bugfix/restore-overwrite 2025-01-06 14:43:03 +01:00
Kenneth SkovhedeandGitHub 2a757dea0f Quota check disabled fix (#5843)
The QuotaCheck is done in two places, and only one place was checking if the quota check was supposed to be disabled.
This adds the check in the other place.
2025-01-06 14:21:53 +01:00
Carl Johnsen 88846251b1 Restoring a symlink that already exists triggers an overwrite of the file, which should count towards number of files restored. 2025-01-06 08:14:32 +01:00
Carl Johnsen 1bea0f3696 Restored empty files should count towards the restoration file count. 2025-01-05 12:46:08 +01:00
Carl Johnsen 4cd565d592 Added handling of already restored timestamp-retargeted files 2025-01-05 12:09:38 +01:00
Carl Johnsen b759b4564d Added docstrings to new functions 2025-01-05 12:09:04 +01:00
Carl Johnsen eb64320a49 Changed the reporting of restored files to match the legacy restore's reporting. 2025-01-05 11:51:52 +01:00
Carl Johnsen 637bfcefc8 Forgot to check whether the file exists 2025-01-05 11:25:19 +01:00
Carl Johnsen eba477c271 Added handling of the 'overwrite' option in the new restore flow 2025-01-05 10:44:08 +01:00
Carl Johnsen f76bf90e68 Added additional dry run logging 2025-01-05 10:43:23 +01:00
Carl Johnsen 8e92754efd VerifyLocalBlocks now keeps track of internally verified blocks for logging 2025-01-05 10:43:01 +01:00
Carl Johnsen be99f03bac VerifyLocalBlocks wasn't respecting dry run properly 2025-01-05 10:41:33 +01:00
Carl Johnsen a14d29da93 Modified VerifyTargetBlocks to also return the blocks that were verified. 2025-01-05 10:40:14 +01:00
Carl Johnsen a7507fb765 Simplified variable assignment 2025-01-05 10:35:50 +01:00
Carl Johnsen 83d1617984 Fixed the IO calls to also work on Windows 2025-01-03 23:27:35 +01:00
Carl Johnsen 52018db0bb Added handling and warning throwing for when the folder for restored files aren't created properly 2025-01-03 20:48:58 +01:00
Kenneth SkovhedeandGitHub bc6114323f Workaround for multi-backend. (#5815)
* Workaround for multi-backend.

There is currently an issue with creating the BackendManager which has a backend instance. This instance is not used by the uploader, so they will create new instances.
The BackendManager instance can then time out, but is used later.

This commit will allow the uploader to "borrow" the instance from the backend manager, so only a single instance is needed.

This is not a clean fix, but doing so requires a full rewrite of the backend manager and uploader, so the controller creates a single instance of the manager, which is then used for all operations.

This would allow the backend manager to keep track of instances across all requests.

* Improved reuse after shutdown
2024-12-28 21:25:49 +01:00
Carl Johnsen ec55580f5c Updated the call signature of FileBlockProcessor.Run() to match the new format 2024-12-23 06:49:04 +01:00
Carl Johnsen a566d1a49f Merge remote-tracking branch 'upstream/master' into fileprocessors 2024-12-23 06:45:43 +01:00
Kenneth SkovhedeandGitHub 161b5708d4 Added missing timeouts to uploads. (#5803)
* Added missing timeouts to uploads.

Since much of the BackendManager is replicated into the BackendUploader, this code needs to be updated as well.

* Fixed namespace issue
2024-12-20 21:03:48 +01:00
Kenneth SkovhedeandGitHub ff1b2b7990 Simplified task control (#5753)
* Simplified task control by having all logic in a single class and using async control mechanism
* Fixed the tests
* Made the marking of a backup partial explicitly communicated if the enumeration process is stopped.
* Added support for pausing active transfers
* Introduced transfer token to abort operations.
* Updated the way LiveControl is emitting events.
* Retains the paused state of the server across reboots, this fixes #5760.
* Reworked the way throttle speeds are propagated through LiveControls.
* Added option to pause transfers in the UI.
* Removed throttle settings from the LiveControls class so it is exclusively handled by the ApplicationSettings.
* Removed `thread-priority` as it does not work at all when tasks are used.
* Fixed the stop dialog and API to only support stop and abort.
* Removed unused variables
2024-12-18 08:27:59 +01:00
Carl Johnsen c13482cf4c Removed unnecessary CoCoL channel scope 2024-12-10 10:28:44 +01:00
Carl Johnsen 5bc12e3d46 Changed the reporting of broken local and remote files during restore to only report the first 10 filenames. 2024-12-10 06:57:56 +01:00
Carl Johnsen ce9a85c3ff Reintroduced comment and version fix, which shouldn't have been removed. 2024-12-10 06:43:51 +01:00
Carl Johnsen ee8a82c230 Moved logging outside of the lock on blockcount in restore block manager 2024-12-10 06:14:55 +01:00
Carl Johnsen f3afd4c682 Reintroduced the restore block cache eviction logic 2024-12-10 06:03:08 +01:00
Carl Johnsen 1a2f58b6c8 Merge branch 'master' into performance/restoreopt 2024-12-10 04:14:07 +01:00
Carl Johnsen 7f37c49292 Changed the VolumeManager to use a single input channel of type object, and then switching on the message received, rather than ReadFromAny().
Because it seems like the amount of readers in queue for one of the channels became much higher (70) than the actual number of readers (1).
2024-12-10 04:12:33 +01:00
Carl Johnsen 2d6e1d7ff7 Changed the new restore version to use explicit CoCoL channels 2024-12-10 04:10:34 +01:00
Carl Johnsen b2288cb804 Removed named CoCoL channels in favor of passing around instances. 2024-12-09 13:55:22 +01:00