This PR adds experimental support for using either ZStandard or GZip compression. Unlike the Zip module, which also supports these methods, the new module does a full-volume compression which is both much faster and compresses better with modern compression algorithms.
Because the compressor can see the full volume (and not just the blocks) it can compress data across blocks.
The downside to this is that it is not possible to take a single entry out of the compressed stream, but instead, Duplicati needs to decompress the whole stream and can then access the contents.
To make the implementation compatible with standard tools, the inner format is Tar (using ustar format). To allow faster reading, a small end-of-file header is added to the Tar file, emulating the concept from Zip files.
This small addition allows Duplicati to have random access to files in a Tar volume without needing to scan the whole thing.
To make sure data is always recoverable, the format is 100% compatible with regular tools, so `untar -xf` will work on the created volumes.
The downside is an extra pass for decompression when reading the volumes. When writing, each entry (a block of data most commonly) is written to a temporary file before being added to the output stream. It is possible to toggle this to use a memory buffer for even more speed up.
The ZStandard compression is from `ZstdSharp.Port` which has excellent performance.
When we move to .NET11 this can easily be changed to the new built-in module.
Since this is the first attempt to add this, it will log a warning on each use, explaining that the feature is currently just for testing.
Co-authored-by: Copilot <copilot@github.com>
This updates VSS to default use Vanara in favor of AlphaVSS which is no longer maintained.
The build for Vanara requires targeting `net8.0-windows7.0`, which will cause significant build overhead and complexity for cross platform builds.
To counter this, the setup is to have a single project, `Duplicati.Library.WindowsModules`, that is targeting `net8.0-windows7.0`.
The output from this project is then hoisted into the TrayIcon project for Windows builds so the files are available when debugging on Windows.
A top-level dummy executable project is added to ensure the project always builds.
The built modules are then loaded with reflection when requested.
With the use of Vanara there is now also support for using BackupRead to read files without making a VSS snapshot.
With BackupRead, it is possible to read locked files, but it still requires the SeBackupPrivilege as VSS does as well.
Unfortunately, the `vssapi.dll` file is not shipped for Arm64 on Windows, so even with Vanara this will not work, and only WMIC is supported on Arm64.
A workaround is to run Duplicati with x64 emulation if more advanced VSS features are needed (HyperV and MSSQL support).
This PR also updates options and filters out unsupported options for each operating system, so options that are not supported by the current OS are not reported and will give warnings if they are used, as opposed to just being ignored.
The release builder project has been updated to exclude the unused project, and purge unwanted outputs.
Kept the infrastructure to allow re-introducing it later if a better source for a device key is found.
Added support for not having any valid settings encryption key.
Added warnings if the user is running without a settings encryption key.
Added additional checks during startup to help with troubleshooting and configuring a setup with a settings encryption key.
Docker builds will now warn if no encryption key is set.
This fixes#5518
Re-implemented everything using ASP.NET.
Changed some requests to use JSON instead of FORM data.
Some work towards deleting the FIXMEGlobal instance.
Auth is missing, XSRF does not work correctly.
