This PR adds experimental support for using either ZStandard or GZip compression. Unlike the Zip module, which also supports these methods, the new module does a full-volume compression which is both much faster and compresses better with modern compression algorithms.
Because the compressor can see the full volume (and not just the blocks) it can compress data across blocks.
The downside to this is that it is not possible to take a single entry out of the compressed stream, but instead, Duplicati needs to decompress the whole stream and can then access the contents.
To make the implementation compatible with standard tools, the inner format is Tar (using ustar format). To allow faster reading, a small end-of-file header is added to the Tar file, emulating the concept from Zip files.
This small addition allows Duplicati to have random access to files in a Tar volume without needing to scan the whole thing.
To make sure data is always recoverable, the format is 100% compatible with regular tools, so `untar -xf` will work on the created volumes.
The downside is an extra pass for decompression when reading the volumes. When writing, each entry (a block of data most commonly) is written to a temporary file before being added to the output stream. It is possible to toggle this to use a memory buffer for even more speed up.
The ZStandard compression is from `ZstdSharp.Port` which has excellent performance.
When we move to .NET11 this can easily be changed to the new built-in module.
Since this is the first attempt to add this, it will log a warning on each use, explaining that the feature is currently just for testing.
Co-authored-by: Copilot <copilot@github.com>
* Fixed nullability warnings in `Utility.cs`
* Fixed a number of nullability issues and warnings
* More nullable fixes
* Fixed the overwrite issue
* Fixed additional nullability issues
* Removed additional nullability issues
* More nullability fixes and added some xmldoc
* Added more xmldocs and removed an unused argument
* Removed more warnings
* Added test to verify that compression modules will mask duplicate entries, if present
* Because there are likely many, otherwise valid, dindex zip archives with duplicated blocklists out there, the log message is now verbose.
To make sure we no longer create archives with duplicated entries, the `--unittest-mode` option makes this a warning that is checked in most tests.
* Added guard in `IndexVolumeCreator` to prevent creating zip files with duplicated block lists.
* Added debug check for duplicates
* Fixed the duplicate test
