Commit Graph
2288 Commits
Author SHA1 Message Date
Carl Johnsen 043b9230b5 VerifyLocalBlocks opened file in incorrect mode, and the target file may not exist. 2025-02-10 13:44:02 +01:00
Carl Johnsen 82019ea28e Modified VerifyTargetBlocks to not evict block from cache. It is now handled outside, in case of file name retargeting when overwrite is false. 2025-02-10 13:42:27 +01:00
Carl Johnsen 0b3ff33716 Moved retire on exception to final, so regardless of how the FileProcessor exits, it retires the channels. 2025-02-10 12:38:24 +01:00
Carl Johnsen 3b9fc188e1 Retire block request and response on exception in FileProcessor 2025-02-10 12:34:44 +01:00
Carl Johnsen 259322f0d9 CopyOldTargetBlocksToNewTarget was made async 2025-02-10 12:34:16 +01:00
Carl Johnsen 6e2f8156f8 VerifyLocalBlocks should only perform checks when the target differs from the original. 2025-02-10 12:33:45 +01:00
Carl Johnsen dbd3520d9c Formatting 2025-02-10 12:18:16 +01:00
Kenneth SkovhedeandGitHub 35179d9966 Merge pull request #5946 from duplicati/feature/prevent-dataloss-from-outdated-database
Add a check for stale database
2025-02-07 16:15:40 +01:00
Marcelo C. 11643c2413 Removing TimeoutObservingStream from backend manager 2025-02-06 11:07:17 -03:00
Kenneth Skovhede bae82feeae Add a check for stale database
This commit adds a check to verify that the remote database is up-to-date before proceeding to repair the remote storage.

Without this check it is possible to start repairs with an old database, and this will cause the remote data to be deleted.

This fixes #4579
2025-02-06 13:20:32 +01:00
Kenneth SkovhedeandGitHub 28e3fa1063 Merge pull request #5914 from duplicati/feature/prevent-windows-subfolder-storage
Improve data folder assignment
2025-01-31 14:52:45 +01:00
Kenneth Skovhede b1e23da40c Implemented correct shutdown of backend manager handler 2025-01-29 11:04:13 +01:00
Kenneth Skovhede fac12ba772 Further consolidated the data folder.
Added support for `--portable-mode` and `--server-datafolder` for the CLI, ServerUtil, and Agent.

Fixed a few places where file reads were not cached properly.

Moved some responsibilities of UpdaterManager into DataFolderManager.

This now supports storing `machineid.txt` and `installation.txt` inside the path pointed to by `--server-datafolder` or alternatively with `--portable-mode`.

It is possible that some fringe backwards compatibility is lost with this update.

Fixed an issue with AutoUpdater crashing on check/download.

This fixes #5902
2025-01-28 13:01:35 +01:00
Kenneth SkovhedeandGitHub 039e1a08fb Merge branch 'master' into feature/prevent-windows-subfolder-storage 2025-01-28 08:58:26 +01:00
Kenneth SkovhedeandGitHub 610f3b47d1 Update reporting module (#5910)
* Updated reporting values.

The reporter module can now emit the following extra values:
- operating-system
- installation-type
- destination-type
- next-scheduled-run

When running from the built-in scheduler, it will automatically set `--next-scheduled-run` so it is applied to the reporting data.

* Implemented an improved machine name reader.
This fixes #5907

* Added a guard against sending too much of the destination url as the destination.

* Fixed reading Windows Machine name

* Removed extra Windows logic as it has no effect
2025-01-28 08:56:42 +01:00
Kenneth SkovhedeandGitHub f7b7555e87 Fixes for tests (#5918)
* Two small test fixes

* Ignore exceptions when comitting an empty transaction

* Attempt to fix FTP connection issue

* Unified test by having a single download function.
Added DNS probing call to attempt fixing sporadic DNS errors.

* Fixed argument order

* Avoid creating temp file during tests, if not needed
2025-01-28 08:55:28 +01:00
Kenneth SkovhedeandGitHub d99f74ed94 Rewrite the backend manager. (#5896)
* Rewrite the backend manager.

This creates a new more logic backend manager that handles all backend operations.

For each top-level operation, there is now a single backend manager instance that is passed to sub-commands.

The internals of the backend manager are now rewritten to use async logic instead of threading.

The design uses a single task runner that dispatches operations and honors concurrent settings and limits from one place.

The logic for each operation has been moved into a separate files/classes to make it easier to understand each operation.

This fixes #5804

* Fixed a few issues with not awaiting tasks

* Fixed not creating empty databases

* Fixed an issue with hashes not being recorded on download

* Reworked the download logic to avoid attempting to decrypt the file if it has been modified.

* Review fixes

* Renamed db collector to better reflect the purpose.

* Review fixes
2025-01-28 08:54:50 +01:00
Kenneth Skovhede 27f0bb4ea0 Unified code to find the data folder.
The `machineid.txt` and `installation.txt` files are now default located in same folder where the database is default located.
2025-01-27 22:51:45 +01:00
Kenneth Skovhede c1d89e32e9 Guard against name clashes for Windows folder 2025-01-27 13:09:35 +01:00
Kenneth Skovhede 805d3b1c31 Added special handling of Linux to avoid storing data in the root of the filesystem 2025-01-27 11:51:03 +01:00
Kenneth Skovhede c05b00ae0b Implemented support for avoiding storing data in the Windows folder 2025-01-27 11:49:17 +01:00
Kenneth Skovhede 97d62163c5 Renamed methods for database path to better distinguish what they do 2025-01-27 11:46:24 +01:00
Kenneth SkovhedeandGitHub 6c493186a7 Changed MissingFileDetected message to Warning (#5893)
* Changed `MissingFileDetected` message to `Warning`

* Updated tests to match errors/warning counts
2025-01-22 21:18:33 +01:00
Kenneth SkovhedeandGitHub 997891525a Updated license-updater to update csproj files (#5898)
* Updated license-updater to update csproj files

* License updates
2025-01-22 21:04:19 +01:00
Carl Johnsen fe7cb13fde Fixed issue with stream not being open in the correct mode during restore 2025-01-17 12:27:05 +01:00
Kenneth SkovhedeandGitHub edd69bfd59 Improve cancel robustness 2 (#5882)
* Attempt to avoid comitting transaction without any queries being made

* Guard internal logic
2025-01-16 16:01:54 +01:00
Kenneth SkovhedeandGitHub 52a73c3418 Fixed issue 5845 (#5880)
* Fixed issue 5845

This commit introduces the test that makes the error.

* Fixed query cleaning up duplicates, in the case that where a block has multiple valid source candidates.

* Reduced number of replicas for the test
2025-01-16 14:27:15 +01:00
Kenneth Skovhede fb1d15fb9d Avoid optimize after vacuum 2025-01-14 19:42:03 +01:00
Kenneth SkovhedeandGitHub 6bbd64a655 Fixed nullability warnings in Utility.cs (#5861)
* Fixed nullability warnings in `Utility.cs`

* Fixed a number of nullability issues and warnings

* More nullable fixes

* Fixed the overwrite issue

* Fixed additional nullability issues

* Removed additional nullability issues

* More nullability fixes and added some xmldoc

* Added more xmldocs and removed an unused argument

* Removed more warnings
2025-01-14 14:03:48 +01:00
Kenneth SkovhedeandGitHub d64c7e721f Handle duplicated missing blocks (#5858)
* Implemented test for #5845

* Removed unittests not executing cleanup.

* Cleanup commit

* Fixed the query that causes the constraint violation
2025-01-10 09:24:54 +01:00
Kenneth SkovhedeandGitHub 382ba172b3 Fixed duplicate entries in index files (#5856)
* Added test to verify that compression modules will mask duplicate entries, if present

* Because there are likely many, otherwise valid, dindex zip archives with duplicated blocklists out there, the log message is now verbose.

To make sure we no longer create archives with duplicated entries, the `--unittest-mode` option makes this a warning that is checked in most tests.

* Added guard in `IndexVolumeCreator` to prevent creating zip files with duplicated block lists.

* Added debug check for duplicates

* Fixed the duplicate test
2025-01-10 09:24:31 +01:00
Kenneth SkovhedeandGitHub 9464caf622 Feature/update license 2025 (#5851)
* Fixed some minor whitespace issues

* Updated all copyright to 2025
2025-01-07 09:40:39 +01:00
Carl Johnsen 46b05b94ef Limit file name conflict attempts to 100 and log an error if exceeded 2025-01-06 16:01:53 +01:00
Carl Johnsen 6f263ad874 Added a note about potential memory consumption of restores FileProcessor 2025-01-06 16:00:11 +01:00
Carl Johnsen 284b14b72f Merge branch 'master' into bugfix/restore-overwrite 2025-01-06 14:43:03 +01:00
Kenneth SkovhedeandGitHub 2a757dea0f Quota check disabled fix (#5843)
The QuotaCheck is done in two places, and only one place was checking if the quota check was supposed to be disabled.
This adds the check in the other place.
2025-01-06 14:21:53 +01:00
Carl Johnsen 88846251b1 Restoring a symlink that already exists triggers an overwrite of the file, which should count towards number of files restored. 2025-01-06 08:14:32 +01:00
Carl Johnsen 1bea0f3696 Restored empty files should count towards the restoration file count. 2025-01-05 12:46:08 +01:00
Carl Johnsen 4cd565d592 Added handling of already restored timestamp-retargeted files 2025-01-05 12:09:38 +01:00
Carl Johnsen b759b4564d Added docstrings to new functions 2025-01-05 12:09:04 +01:00
Carl Johnsen eb64320a49 Changed the reporting of restored files to match the legacy restore's reporting. 2025-01-05 11:51:52 +01:00
Carl Johnsen 637bfcefc8 Forgot to check whether the file exists 2025-01-05 11:25:19 +01:00
Carl Johnsen eba477c271 Added handling of the 'overwrite' option in the new restore flow 2025-01-05 10:44:08 +01:00
Carl Johnsen f76bf90e68 Added additional dry run logging 2025-01-05 10:43:23 +01:00
Carl Johnsen 8e92754efd VerifyLocalBlocks now keeps track of internally verified blocks for logging 2025-01-05 10:43:01 +01:00
Carl Johnsen be99f03bac VerifyLocalBlocks wasn't respecting dry run properly 2025-01-05 10:41:33 +01:00
Carl Johnsen a14d29da93 Modified VerifyTargetBlocks to also return the blocks that were verified. 2025-01-05 10:40:14 +01:00
Carl Johnsen a7507fb765 Simplified variable assignment 2025-01-05 10:35:50 +01:00
Carl Johnsen 83d1617984 Fixed the IO calls to also work on Windows 2025-01-03 23:27:35 +01:00
Carl Johnsen 52018db0bb Added handling and warning throwing for when the folder for restored files aren't created properly 2025-01-03 20:48:58 +01:00