Updated the Duplicati backend to support direct backups where the storage server returns keys to an AWS compatible storage and lets the client communicate directly with the storage.
The gateway mode is stil supported.
This removes a built-in HTTP timeout of 100s that is applied to all requests.
Since Duplicati sometimes need more than 100s to transfer a file, this could cause problems.
Duplicati already handles timeouts, but this timeout was happening regardless of Duplicati's other timeouts.
This PR adds experimental support for running the backup in an isolated process.
The motivation for this is to protect the server from a faulty process and later enable running server and processing with different security levels.
This needs to be toggled on, and then it will run the process isolated. When running as an isolated process, the "Abort" call will kill the process so it can no longer become "stuck".
When creating a new controller, the options dictionary was used by-reference.
This meant that code could update the options dictionary *after* creating the controller, which would not be thread-safe.
It does not have much effect on regular operations as the normal method is to create the dictionary, pass it, and not modify it.
But in the tests, we frequently re-use the dictionary and update it with new values to test various cases.
Due to the dictionary being a reference, the controller would update the dictionary to fit the operations, but this would reflect back into the tests, meaning that some tests were being run with options that were not supposed to be there.
This meant that at least two edge cases that we test for were never found to have errors, even though they do.
The first issue is the listing of files using a non-default blocksize. The second issue is recreating the database with a non-default file-hash algorithm.
This PR removes the reference passing, and fixes the issues that were uncovered.
This PR hides the `--unittest-mode` option that is used to treat all warnings as errors which is helpful when testing things, but is not intended to be used elsewhere.
The option is simply not shown in the list of supported options but still works in case anyone wants to use it.
This PR has a large blast radius because it takes the final step and bumps up the Controller to be fully async.
We have historically done a piece-by-piece update, so all operations were already async but the controller interface was kept synchronous.
With this update, the controller is now fully async and all tests are updated.
Most places where the new C# compiler warns about function names not ending in `Async` were also adressed, giving a massive refactor change.
Functionally, no changes are done.
This PR deprecates the `mssp` and `od4b` backends as they stopped working when the APIs were turned off.
The upgrade path is to use `sharepoint` instead (which used to be named "SharePoint v2").
This PR also adds a small amount of metadata to modules so they can be marked as "untested" or "deprecated", and the configuration for marking modules has been set up in `BackendModules.cs`.
The backends that we currently cannot test (usualy due to not having public signups) are now marked as untested.
This PR changes the restore message when no files are restored.
The logic now keeps track of existing files that do not need to be restored, as well as the actual number of bytes written.
The size of restored files was tracked previously, but was instead reporting the size of the written data.
With this update the legacy restore and new restore report the same numbers.
The warning that is shown if no files are restored has been updated to check if existing files were present, and if so, the restore is assumed to have worked as expected, and an information message is logged, instead of the warning message.
This PR adds a small change when writing the `filelist.json`. The code now *requires* that a metahash is present when writing an entry.
Previously, the code would handle cases where there was no metahash and output the `filelist.json` without these fields.
However, the recreate fails if the fields are missing due to database consistency requirements.
Also, there appear to be no regular code paths that should end up creating an entry without the metahash.
This may need to be reverted, but for now it will help track down if there are any cases where a missing metahash is needed, and if we find that, we can fix the relevant functions and revert this change.
This adds an extra clause when building a synthetic filelist to avoid cases where the synthetic filelist fails to be generated because some entries are orphans.
The tests that reproduce this are quite slow and require some tampering with the database to get into the faulty state, so the tests will not run in CI testing but can manually be executed.
Kenneth Skovhede