Adds error-correction (parity) data for remote data volumes so they can be
repaired after bit-rot or corruption on the backend, addressing issue #314
(and reviving the intent of the stale draft PR #4574, redesigned for the
current architecture).
A pluggable IParity module system is introduced, mirroring the existing
compression/encryption modules (IParity + ParityLoader + a new
Duplicati.Library.Parity project). The default Par2Parity implementation
shells out to the "par2" (par2cmdline) program. par2 emits an index file
plus recovery volume files; these are concatenated into a single parity
file (the PAR2 format is a stream of self-describing packets, so this is a
valid PAR2 file). All par2 operations run in a temp working directory using
a fixed canonical name so create and repair stay symmetric despite differing
temp file names.
Behavior:
- Upload (Put): after a data volume (dblock/dlist) is uploaded, a parity
companion "<name>.par2" is generated from the final encrypted bytes and
uploaded unencrypted and untracked, inline on the held backend. dindex is
skipped (regenerable). Generated post-upload so it always keys off the
settled filename (retry/rename safe).
- Download (Get): on a hash/size mismatch, the parity companion is fetched
and used to repair the file in place before decryption. This single choke
point covers Restore/Compact/Test/Repair.
- Delete: the parity companion is removed (best-effort) with its data volume.
- Parsing: parity files are recognized by ParsedVolume.Parse (avoiding the
greedy encryption-group trap) and excluded from remote-list reconciliation
so they are never flagged as extra.
Parity is opt-in via --parity-redundancy-level (>0), --parity-module
(default par2), with --no-parity and par2-specific options. It is best-effort:
if the par2 program is absent the backup still succeeds with a single warning,
and repair falls through to the normal hash-mismatch error. Not tracked in the
database and requires no schema migration; the parity name is always derivable
from the data volume name.
Adds NUnit tests: parity filename generation/parsing and loader discovery
(no external dependency), plus end-to-end Par2Parity create/verify/repair
tests that are skipped when the par2 program is not available.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This refactors a part of the database and adds a new "sync" command that reuses a lot of the backup logic to take a set of sources and apply them to a backend.
Unlike the backup process, the sync copies files verbatim without any encryption.
This PR has a large blast radius because it takes the final step and bumps up the Controller to be fully async.
We have historically done a piece-by-piece update, so all operations were already async but the controller interface was kept synchronous.
With this update, the controller is now fully async and all tests are updated.
Most places where the new C# compiler warns about function names not ending in `Async` were also adressed, giving a massive refactor change.
Functionally, no changes are done.
If there are extra unknown files, this can be a sign that multiple backups are using the same folder.
It can also be a sign that an older database is used for some reason.
This PR updates the error message to mention these two common cases, and spell out what the repair process will do.
This commit adds extra logic to prevent cases where the backup reports success but leaves behind partial files.
A new flag is recorded to track if partial files are possible. If no partial files are possible, the verification will fail if any are found.
* Rewrite the backend manager.
This creates a new more logic backend manager that handles all backend operations.
For each top-level operation, there is now a single backend manager instance that is passed to sub-commands.
The internals of the backend manager are now rewritten to use async logic instead of threading.
The design uses a single task runner that dispatches operations and honors concurrent settings and limits from one place.
The logic for each operation has been moved into a separate files/classes to make it easier to understand each operation.
This fixes#5804
* Fixed a few issues with not awaiting tasks
* Fixed not creating empty databases
* Fixed an issue with hashes not being recorded on download
* Reworked the download logic to avoid attempting to decrypt the file if it has been modified.
* Review fixes
* Renamed db collector to better reflect the purpose.
* Review fixes
The QuotaCheck is done in two places, and only one place was checking if the quota check was supposed to be disabled.
This adds the check in the other place.
The assigned quota size is checked in addition to the reported backend quota (if available). It can be used to limit the backup size on the remote. Warnings are created according to the same quota warning threshold.
Previously, this option had no function.
If set, the backend is treated as if it does not report a quota at all. No warnings and errors are created and no quota values are set in the backup log.
This option is intended to be used if a backend reports a wrong quota, so it needs to be disabled.
Previously, we would catch any Exception thrown by VerifyRemoteList and
perform an auto-cleanup (if specified in the options). However, we
should only perform the auto-cleanup if VerifyRemoteList detected an
issue with the uploaded files. Otherwise, an unrelated exception can
cause the database repair to be performed unnecessarily. In the case of
a connection issue, this can leave the database in a corrupted state.
This fixes#4516.
When a backup is interrupted it will now show as a partial backup in the list of backups on the restore files page.
Temporary filelist volumes are protected from deletion in several places. All but the RestoreHandler
only protect the most recent filelist. The RestoreHandler protects all temporary filelist volumes as the user
might be restoring from one that would be deleted.
Prevent the CoCoL BackendRequests channel is retired exception from appearing when the backup is stopped by checking
if the channel is retired before trying to add the FlushRequest to the channel. Any exceptions from the BackendUploader
task will still be thrown.
Changed a few things on the way, removing the log process
# Conflicts:
# Duplicati/Library/Main/Controller.cs
# Duplicati/Library/Main/Database/ExtensionMethods.cs
# Duplicati/Library/Main/Database/LocalBackupDatabase.cs
# Duplicati/Library/Main/Duplicati.Library.Main.csproj
# Duplicati/Library/Main/Operation/BackupHandler.cs
# Duplicati/Library/Main/Operation/FilelistProcessor.cs
# Duplicati/Library/Main/Operation/PurgeBrokenFilesHandler.cs
# Duplicati/Library/Main/Operation/TestFilterHandler.cs
# Duplicati/Library/Main/Options.cs
# Duplicati/Library/Main/ResultClasses.cs
# Duplicati/UnitTest/BasicSetupHelper.cs
