Commit Graph
230 Commits
Author SHA1 Message Date
Kenneth Skovhede 6a82da0e81 Fix null versions array 2026-03-14 19:17:51 +01:00
Kenneth Skovhede ec6e32be39 Fixed usage of filelist filter 2026-03-14 18:26:37 +01:00
Kenneth Skovhede a01a7ff5f7 Fixed filtering to return a null delegate when not filtering 2026-03-14 13:56:26 +01:00
Kenneth Skovhede 9bd4d228f3 Minor polish to the new restore flow
This PR adds explicit validation to the size parameters so we ensure that we throw a meaningful error message instead of fail on some later step.

This also fixes an error in an exception message. Should the error scenario ever trigger, it would previously show incorrect data or cause out-of-bounds error instead of the intended error message.
2026-03-04 14:44:09 +01:00
Carl Johnsen 4d1e388e33 Less intrusive legacy restore changes
Due to the legacy restore flow being volume-centric, restoring has to become two phases: 1 for the priority files, and 2 for the "rest" of the files.
2026-02-19 17:52:53 +01:00
Carl Johnsen 594efeca04 Merge branch 'master' into feature/full-disk-backup 2026-02-15 16:42:08 +01:00
Carl Johnsen ced4ed8ea4 Legacy restore now also restores priority files first 2026-02-09 12:41:55 +01:00
Carl Johnsen e9012ada05 Added synchronization of priority files prior to restoring regular files 2026-02-09 12:28:21 +01:00
Carl Johnsen 3fe57b308d Added functionality for priority files for restore providers 2026-02-09 11:36:15 +01:00
Kenneth SkovhedeandGitHub 18c12b66c3 Merge pull request #6753 from duplicati/feature/check-temp-space
Check free temporary space
2026-02-06 15:56:29 +01:00
Kenneth SkovhedeandGitHub baca571ef8 Merge pull request #6727 from duplicati/feature/prevent-restore-target-escape
Prevent restore target escape
2026-02-06 14:33:01 +01:00
Kenneth Skovhede 05ea1dfafe Check free temporary space
This PR adds checks for free temporary space when starting the server and when restoring.

On systems that have limited space in the temp folder a warning will now be shown.

This is intended to capture issues on some Docker systems where /tmp is mounted in memory instead of being disk backed.

It will also detect the issue on other systems that are space constrained.
2026-02-06 10:37:27 +01:00
Kenneth Skovhede cf8ef437c9 Fixed an edge case where the code would attempt to create a parent folder outside the restore target prior to applying metadata 2026-02-05 15:17:14 +01:00
Kenneth Skovhede 56f74803fc Fixed edge cases for path traversal protection 2026-02-05 09:21:24 +01:00
Kenneth Skovhede 2bb31d8273 Update to use destination restore path to avoid leaking credentials in the log 2026-01-29 20:22:09 +01:00
Kenneth SkovhedeandGitHub 26f7dd8586 Merge branch 'master' into feature/add-office-365-backup 2026-01-29 20:04:42 +01:00
WhiteElephantIsNotARobotandClaude a1a22efafa Add RestorePath property to restore results
This adds the output folder path to the restore log/results, addressing
issue #6698. When performing a restore, especially long-running ones,
users may forget where files were restored to. This change ensures the
restore path is included in the JSON output.

Changes:
- Added RestorePath property to IRestoreResults interface
- Added RestorePath property to RestoreResults class
- Set RestorePath in RestoreHandler at the start of the restore operation

When no custom restore path is specified (files restored to original
locations), RestorePath will be null. Otherwise, it contains the
absolute path where files were restored.

Co-Authored-By: Claude (mimo-v2-flash) <noreply@anthropic.com>
2026-01-29 14:32:01 +00:00
Kenneth Skovhede c0dd3ccea2 Fixed another failing test 2026-01-26 14:58:58 +01:00
Kenneth Skovhede 27dc64121b Implemented cross-tenant email restore 2026-01-16 11:09:13 +01:00
Kenneth Skovhede ca7dff57f3 Updated restore flow to use the IRestoreDestinationProvider interface for accessing files 2026-01-14 10:11:24 +01:00
Carl Johnsen d48afd7496 Check the database for volume sizes reported in remotevolume 2025-11-10 14:56:20 +01:00
Carl Johnsen 77f3000e4f Reduced the timing of a block to the lifetime of the block request as it roundtrips from/to the BlockManager 2025-11-10 13:28:59 +01:00
Carl Johnsen 57e40a6290 Initial deadlock detection threshold is now set to 1 minute per 10 MB of volume size. 2025-11-10 12:54:33 +01:00
Carl Johnsen 96921bc5fb RestoreHandler now sets up the DeadLock timer alongside the restore process network 2025-10-28 14:44:13 +01:00
Carl Johnsen 6402e1b4a7 Fixed deadlock when the restore-channel-size option was set to 0 2025-09-30 13:00:52 +02:00
Carl Johnsen 4826b800a5 Instead of keeping track of how many that are actively accessing the arrays, we now have an object wrapping the array.
When this object is collected by the garbage collector, the array is returned to the array pool. This moves the responsibility to the garbage collector leading to a more thread-safe approach with less locks, at the cost of potentially using memory for a longer time until the garbage collector picks up the objects.
2025-09-30 12:55:13 +02:00
Carl Johnsen dfc92c71b1 Added more rendevouz points to allow for aborting during deadlocks 2025-08-10 20:54:13 +02:00
Carl Johnsen 5129958f4f Changed the restore channels to use a constructor to avoid static variables 2025-08-10 20:50:02 +02:00
Carl Johnsen f9111a9dde Made RestoreHandler call the DisposeAsync instead of Dispose on the LocalRestoreDatabase 2025-06-24 12:05:15 +02:00
Carl Johnsen 2412800c0c Removed the sqlite page cache option. It should be set through the environment variable, or some other way of providing a custom database configurotion 2025-06-19 11:41:39 +02:00
Carl Johnsen 31c5441bdb Forwarded cancellation tokens, added some missing .ConfigureAwait() and added some missing docstrings 2025-06-19 11:04:09 +02:00
Carl Johnsen 32efa4b845 Updated all of the handler calls to database methods to forward their cancellation token 2025-06-19 09:47:33 +02:00
Carl Johnsen 9ac583db94 Made all relevant using statements use await to use DisposeAsync 2025-06-18 11:53:11 +02:00
Carl Johnsen 859b65054d Simplified using statements that uses databases or accesses their methods 2025-06-18 11:52:05 +02:00
Carl Johnsen 3d11a05400 Added .ConfigureAwait(false) to all of the await statements 2025-06-12 08:34:29 +02:00
Carl Johnsen ee40767412 Merge branch 'master' into performance/mssqlite-async 2025-05-27 12:04:50 +02:00
Carl Johnsen da487182ae Connection pool in RestoreDatabase now has a transaction associated with each connection 2025-05-23 15:46:30 +02:00
Kenneth Skovhede d34b20d38a Remove Sia backend
The Sia backend implementation is based on the previous version of Sia and will no longer work after the hard fork later this year.
2025-05-22 15:28:59 +02:00
Carl Johnsen cdba7ef6e9 Updated Utility to use the new async database methods 2025-05-22 08:25:25 +02:00
Carl Johnsen 6a0b836a79 Updated RestoreHandler to use the new async database methods 2025-05-22 06:47:16 +02:00
Kenneth Skovhede 0a7a1cd689 Added option to set the SQLite page cache
This adds an extra option to allow setting the SQLite page cache size as a regular option.

Prior to this commit it was only possible to set the SQLite page cache size via environment variables.

The option to use environment variables is preserved, and as options from the environment variable are applied after the new setting, environment variables take precedence.

The default value for the new option is to use 1% of the system memory for the page cache. For the restore process, a connection per worker may be made which defaults to half the number of cores, so the maximum amount of memory is 1% * half the CPU cores.

This is just the upper limit, and SQLite may choose not to use all of it.

This fixes #6178
2025-04-22 14:26:46 +02:00
Carl Johnsen f73d3390cd The FileProcessor now works in two phases: first it restores the files, then they restore the folder metadata, unless specified otherwise. 2025-03-24 20:24:37 +01:00
Kenneth Skovhede a163f53dba Handle stop exceptions
This adds a common way to catch stop exceptions and handles them so error resilient logic does not continue when a stop exception is thrown.

Previously, the logic would only check for `ThreadAbortException` which was used to signal a stop prior to .NET8.
2025-03-20 22:32:04 +01:00
Kenneth Skovhede 7f008341df Untangle the results and database
The code was using a two-way reference linking the database to the results and the results to the database.

However, this was not really used so it ended up just cluttering the code. With this commit, the results and databases are no longer keeping track of each other, and the only interaction is when the results are written to the database at the end of the operation.

To ensure result data is always written, this code has been moved into the controller, which will now flush the results for all operations. Prior to this, only some operations would write result data to the log.

The controller now also handles logic for cleaning old log data and invoking the vacuum command.

Fixed a bug due to using System.Text.Json where some attributes from NewtonSoft.Json were being applied and not picked up.
2025-03-20 09:51:49 +01:00
Kenneth Skovhede c1bdaa069e Async operation handlers
Rewrote all operation handlers to be async.
This moves the async/sync threshold out into the controller.

Fixed all the uses of cancellation token in the handlers to use the provided cancellation token.

Introduced a new ReusableTransaction that can be passed around, avoiding a ref parameter passing of the transaction instance.
2025-03-14 10:41:14 +01:00
Kenneth SkovhedeandGitHub eac394f0a3 Merge pull request #6011 from duplicati/feature/improved-failure-resilience
Improved failure resilience
2025-03-12 14:48:14 +01:00
Kenneth Skovhede 6f7f2757a9 Rewrote timeout monitoring 2025-03-10 10:22:34 +01:00
Kenneth Skovhede 1321488527 Improve failure detection
This commit adds extra logic to prevent cases where the backup reports success but leaves behind partial files.

A new flag is recorded to track if partial files are possible. If no partial files are possible, the verification will fail if any are found.
2025-03-06 12:02:11 +01:00
Carl-Johannes JohnsenandGitHub 476774640a Merge pull request #5991 from carljohnsen/performance/restoreopt2 2025-02-28 13:03:25 +01:00
Kenneth Skovhede f77e529eb6 Merge commit '2339aeff28adba15442d0be60c56568ae642ad0b' into feature/make-backends-async 2025-02-26 15:23:48 +01:00