Adds error-correction (parity) data for remote data volumes so they can be
repaired after bit-rot or corruption on the backend, addressing issue #314
(and reviving the intent of the stale draft PR #4574, redesigned for the
current architecture).
A pluggable IParity module system is introduced, mirroring the existing
compression/encryption modules (IParity + ParityLoader + a new
Duplicati.Library.Parity project). The default Par2Parity implementation
shells out to the "par2" (par2cmdline) program. par2 emits an index file
plus recovery volume files; these are concatenated into a single parity
file (the PAR2 format is a stream of self-describing packets, so this is a
valid PAR2 file). All par2 operations run in a temp working directory using
a fixed canonical name so create and repair stay symmetric despite differing
temp file names.
Behavior:
- Upload (Put): after a data volume (dblock/dlist) is uploaded, a parity
companion "<name>.par2" is generated from the final encrypted bytes and
uploaded unencrypted and untracked, inline on the held backend. dindex is
skipped (regenerable). Generated post-upload so it always keys off the
settled filename (retry/rename safe).
- Download (Get): on a hash/size mismatch, the parity companion is fetched
and used to repair the file in place before decryption. This single choke
point covers Restore/Compact/Test/Repair.
- Delete: the parity companion is removed (best-effort) with its data volume.
- Parsing: parity files are recognized by ParsedVolume.Parse (avoiding the
greedy encryption-group trap) and excluded from remote-list reconciliation
so they are never flagged as extra.
Parity is opt-in via --parity-redundancy-level (>0), --parity-module
(default par2), with --no-parity and par2-specific options. It is best-effort:
if the par2 program is absent the backup still succeeds with a single warning,
and repair falls through to the normal hash-mismatch error. Not tracked in the
database and requires no schema migration; the parity name is always derivable
from the data volume name.
Adds NUnit tests: parity filename generation/parsing and loader discovery
(no external dependency), plus end-to-end Par2Parity create/verify/repair
tests that are skipped when the par2 program is not available.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This improves the secret provider by allowing a value of `default://` to get the system default provider (if one exists).
The license checker now supports multiple formats for the on-disk file.
This updates some minor issues that would cause warnings or make things appear to not work, even though they would work fine when actually running a backup.
This PR adds a flag to the test method so we can choose to test for read-only access (needed for restore) or read-write (needed for backups).
Additionally, this PR fixes a logic bug in that the BackendSourceProvider would not actually test the connection, but just check the placeholder element returned (which is always returned). With this PR the backend's ListAsync command is tested fully.
Also renamed the methods on ISourceProvider to have the `...Async` prefix.
This replaces #6939
This PR adds the option to limit the available modules to just the set of specified choices.
If no option is set, all modules are available (default).
If the option is set, only those in the list are available (whitelisting).
The options are:
- `--allowed-backend-modules`
- `--allowed-encryption-modules`
- `--allowed-compression-modules`
This PR fixes the issues with not correctly loading and disposing modules.
Prior to this PR, there would be only a single instance of each module loaded, which would be configured and later disposed, meaning that the same instance would be used after being disposed, and disposed multiple times.
This mostly worked, but it could carry configuration details over between unrelated runs.
This fixes#6314
This enables a fail-early approach, where the remote source loads the initial results.
By storing the `IAsyncEnumerable` it is possible to not perform any additional remote calls, even with the initial probing call.
This makes it possible to detect the absence of sources, so the backup can be aborted if sources are missing.
This fixes the restore issues as all sources will now appear as if they were originally part of the source file system.
The backend implementation has now been simplified so the backends only return names of entries with no path information. Most backends already do this, but the S3 backends did not, so this was fixed as well.
This also clears up the interfaces, so the backend does not know about the `ISourceProviderEntry`. All mapping is done by the `BackendSourceProvider`, making it simpler to support other backends as sources.
With this interface it is possible to wrap the current support for snapshots and USN and simplify the enumeration.
This makes a number of assumptions into fields such as `IsMetaEntry` and `IsRootEntry` that removes code that was checking for various properties.
This first commit has support for VSS and USN as well as "naked" files, keeping feature parity.
There is also an experimental support for loading a backend-as-a-source. The support for this is currently limited as the backends need to be extended to support a folder concept.
