This updates VSS to default use Vanara in favor of AlphaVSS which is no longer maintained.
The build for Vanara requires targeting `net8.0-windows7.0`, which will cause significant build overhead and complexity for cross platform builds.
To counter this, the setup is to have a single project, `Duplicati.Library.WindowsModules`, that is targeting `net8.0-windows7.0`.
The output from this project is then hoisted into the TrayIcon project for Windows builds so the files are available when debugging on Windows.
A top-level dummy executable project is added to ensure the project always builds.
The built modules are then loaded with reflection when requested.
With the use of Vanara there is now also support for using BackupRead to read files without making a VSS snapshot.
With BackupRead, it is possible to read locked files, but it still requires the SeBackupPrivilege as VSS does as well.
Unfortunately, the `vssapi.dll` file is not shipped for Arm64 on Windows, so even with Vanara this will not work, and only WMIC is supported on Arm64.
A workaround is to run Duplicati with x64 emulation if more advanced VSS features are needed (HyperV and MSSQL support).
This PR also updates options and filters out unsupported options for each operating system, so options that are not supported by the current OS are not reported and will give warnings if they are used, as opposed to just being ignored.
The release builder project has been updated to exclude the unused project, and purge unwanted outputs.
This PR adds the ability to subscribe to messages over a websocket.
To prevent polling data, the server can now push messages through the websocket, so the client can instantly update when new data is available.
The change is backwards compatible, still serving the status updates over the socket. If the client is providing the authentication token, it is auto-subscried to the legacy status message.
If the client is using the new authentication message, it needs to subscribe to get the status updates (and any other services it needs).
The event system has been extended to support new, and more accurate, events. This is the first step towards removing the general status update and the long-poll mechanism.
This also re-introduces the blocking marker, so the client can know if the operation is halted due to too many pending transfers.
Some endpoints have been moved to service implementations, to allow serving the exact same data from both endpoints and websocket.
This PR also updates the way the progress is handled, so that all transfers are returned to the client, and the transfer speeds for each transfer is calculated based on a small sample buffer, so the values are more accurate even if the program is paused during transfers.
The code was using a two-way reference linking the database to the results and the results to the database.
However, this was not really used so it ended up just cluttering the code. With this commit, the results and databases are no longer keeping track of each other, and the only interaction is when the results are written to the database at the end of the operation.
To ensure result data is always written, this code has been moved into the controller, which will now flush the results for all operations. Prior to this, only some operations would write result data to the log.
The controller now also handles logic for cleaning old log data and invoking the vacuum command.
Fixed a bug due to using System.Text.Json where some attributes from NewtonSoft.Json were being applied and not picked up.
Rewrote all operation handlers to be async.
This moves the async/sync threshold out into the controller.
Fixed all the uses of cancellation token in the handlers to use the provided cancellation token.
Introduced a new ReusableTransaction that can be passed around, avoiding a ref parameter passing of the transaction instance.
This commit adds extra logic to prevent cases where the backup reports success but leaves behind partial files.
A new flag is recorded to track if partial files are possible. If no partial files are possible, the verification will fail if any are found.
This enables a fail-early approach, where the remote source loads the initial results.
By storing the `IAsyncEnumerable` it is possible to not perform any additional remote calls, even with the initial probing call.
This makes it possible to detect the absence of sources, so the backup can be aborted if sources are missing.
This fixes the restore issues as all sources will now appear as if they were originally part of the source file system.
The backend implementation has now been simplified so the backends only return names of entries with no path information. Most backends already do this, but the S3 backends did not, so this was fixed as well.
This also clears up the interfaces, so the backend does not know about the `ISourceProviderEntry`. All mapping is done by the `BackendSourceProvider`, making it simpler to support other backends as sources.
Made the SMBShareConnection slightly more robust.
Introduced v2 List API to S3 backend, currently it retains v1 as the default.
Changed S3 backends to not recurse, potentially making smaller listings. This option can be disabled if it breaks things.
Added support for relative paths in the SSH backend.
With this interface it is possible to wrap the current support for snapshots and USN and simplify the enumeration.
This makes a number of assumptions into fields such as `IsMetaEntry` and `IsRootEntry` that removes code that was checking for various properties.
This first commit has support for VSS and USN as well as "naked" files, keeping feature parity.
There is also an experimental support for loading a backend-as-a-source. The support for this is currently limited as the backends need to be extended to support a folder concept.
* Rewrite the backend manager.
This creates a new more logic backend manager that handles all backend operations.
For each top-level operation, there is now a single backend manager instance that is passed to sub-commands.
The internals of the backend manager are now rewritten to use async logic instead of threading.
The design uses a single task runner that dispatches operations and honors concurrent settings and limits from one place.
The logic for each operation has been moved into a separate files/classes to make it easier to understand each operation.
This fixes#5804
* Fixed a few issues with not awaiting tasks
* Fixed not creating empty databases
* Fixed an issue with hashes not being recorded on download
* Reworked the download logic to avoid attempting to decrypt the file if it has been modified.
* Review fixes
* Renamed db collector to better reflect the purpose.
* Review fixes
* Fixed nullability warnings in `Utility.cs`
* Fixed a number of nullability issues and warnings
* More nullable fixes
* Fixed the overwrite issue
* Fixed additional nullability issues
* Removed additional nullability issues
* More nullability fixes and added some xmldoc
* Added more xmldocs and removed an unused argument
* Removed more warnings
* Workaround for multi-backend.
There is currently an issue with creating the BackendManager which has a backend instance. This instance is not used by the uploader, so they will create new instances.
The BackendManager instance can then time out, but is used later.
This commit will allow the uploader to "borrow" the instance from the backend manager, so only a single instance is needed.
This is not a clean fix, but doing so requires a full rewrite of the backend manager and uploader, so the controller creates a single instance of the manager, which is then used for all operations.
This would allow the backend manager to keep track of instances across all requests.
* Improved reuse after shutdown
* Simplified task control by having all logic in a single class and using async control mechanism
* Fixed the tests
* Made the marking of a backup partial explicitly communicated if the enumeration process is stopped.
* Added support for pausing active transfers
* Introduced transfer token to abort operations.
* Updated the way LiveControl is emitting events.
* Retains the paused state of the server across reboots, this fixes#5760.
* Reworked the way throttle speeds are propagated through LiveControls.
* Added option to pause transfers in the UI.
* Removed throttle settings from the LiveControls class so it is exclusively handled by the ApplicationSettings.
* Removed `thread-priority` as it does not work at all when tasks are used.
* Fixed the stop dialog and API to only support stop and abort.
* Removed unused variables
Kenneth Skovhede