Adds error-correction (parity) data for remote data volumes so they can be
repaired after bit-rot or corruption on the backend, addressing issue #314
(and reviving the intent of the stale draft PR #4574, redesigned for the
current architecture).
A pluggable IParity module system is introduced, mirroring the existing
compression/encryption modules (IParity + ParityLoader + a new
Duplicati.Library.Parity project). The default Par2Parity implementation
shells out to the "par2" (par2cmdline) program. par2 emits an index file
plus recovery volume files; these are concatenated into a single parity
file (the PAR2 format is a stream of self-describing packets, so this is a
valid PAR2 file). All par2 operations run in a temp working directory using
a fixed canonical name so create and repair stay symmetric despite differing
temp file names.
Behavior:
- Upload (Put): after a data volume (dblock/dlist) is uploaded, a parity
companion "<name>.par2" is generated from the final encrypted bytes and
uploaded unencrypted and untracked, inline on the held backend. dindex is
skipped (regenerable). Generated post-upload so it always keys off the
settled filename (retry/rename safe).
- Download (Get): on a hash/size mismatch, the parity companion is fetched
and used to repair the file in place before decryption. This single choke
point covers Restore/Compact/Test/Repair.
- Delete: the parity companion is removed (best-effort) with its data volume.
- Parsing: parity files are recognized by ParsedVolume.Parse (avoiding the
greedy encryption-group trap) and excluded from remote-list reconciliation
so they are never flagged as extra.
Parity is opt-in via --parity-redundancy-level (>0), --parity-module
(default par2), with --no-parity and par2-specific options. It is best-effort:
if the par2 program is absent the backup still succeeds with a single warning,
and repair falls through to the normal hash-mismatch error. Not tracked in the
database and requires no schema migration; the parity name is always derivable
from the data volume name.
Adds NUnit tests: parity filename generation/parsing and loader discovery
(no external dependency), plus end-to-end Par2Parity create/verify/repair
tests that are skipped when the par2 program is not available.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
With this interface it is possible to wrap the current support for snapshots and USN and simplify the enumeration.
This makes a number of assumptions into fields such as `IsMetaEntry` and `IsRootEntry` that removes code that was checking for various properties.
This first commit has support for VSS and USN as well as "naked" files, keeping feature parity.
There is also an experimental support for loading a backend-as-a-source. The support for this is currently limited as the backends need to be extended to support a folder concept.
This updates all projects to target .NET Framework 4.7.1. The
TencentCOS and Tardigrade backends depend on .NET Standard 2.0. When a
.NET Framework prior to 4.7.1 is targeted, the system cannot be sure
that all the dependencies exist, so it copies all dependent assemblies
to the output directory. This causes many assemblies from the System
namespace to become bundled in the release.
https://stackoverflow.com/a/48875007
We had previously attempted to make individual projects target 4.7.1
(see pull request #4242), but this can cause compatibility issues when
4.6.2 projects depend on 4.7.1. projects.
This will require Mono 5.10.0 or greater (previously, we required 5.0.0
or greater).
https://www.mono-project.com/docs/about-mono/releases/5.10.0/#class-libraries
This fixes issue #4234.
- no code changes except those noted below
- projects upgrade to 4.6.2
- wixinstaller project upgraded automatically by VisualStudio
- wixinstaller updated to require 4.6.2
- Library.Encryption changed to Standard2.0 so accommodate update to SharpAesCrypt
Using strong-named assemblies can cause difficulties with the GNU LGPL
license, which allows for one to recombine or relink their application
with modified versions of the code. While one solution is to share the
private key so that people can sign the assemblies themselves, this
would break the trust that is expected from signed assemblies. For now,
the easiest fix is to simply not sign the assemblies. Note that by
doing so, we prevent the code from being referenced from other signed
assemblies.
This also fixes an issue introduced in revision ba94d36a80 ("Added
auto-update for WindowsService and Service."), where the WindowsService
project (signed) referenced the AutoUpdater project (not signed).
We also removed instances of <SignAssembly>false</SignAssembly> to be
consistent with newly created .csproj files that do not contain the
SignAssembly element.
This was motivated by the discussion in issue #2814.
Status: Fixed
The new strategy is to provide general assembly redirects.
These are generated during the build procedure and added to the output.
Using these redirects it is possible to load a new/old version of a backend
(or other module) into Duplicati.
To ease testing, it is now also possible to place a dll for a backend in
a subfolder named "backends" and it will override the normal backend.
To remove the test backend, simply stop Duplicati and delete the "backends" folder.
The same works for compression, encryption, generic modules, and settingcontrols
with the folders "compression", "encryption", "modules", and "settingcontrols"
respectively.
git-svn-id: https://duplicati.googlecode.com/svn/trunk@536 59da171f-624f-0410-aa54-27559c288bec
This was based on the idea that the interfaces should remain static.
I hope this change is enough to ensure stable interfaces until release 1.2.
Overview of changes:
Moved all interfaces into the same dll.
Encryption and compression is now plugable modules, just as the backends.
Encryption/compression can now register an UI.
Encryption now uses AESCrypt as a default.
GPG does not default to using the --armor option.
Added support for generic modules, but none are written yet.
Added support for plugable settings pages in the "Options" dialog.
Fixed issue #148.
Duplicati now uses AESCrypt as the default encryption format.
Fixed issue #199.
GPG now supports custom commandline options.
Fixed issue #207.
Encryption modules are now plugable.
Fixed issue #118.
S3 credentials are now stored.
Fixed issue #151.
Backends can now register system wide options.
git-svn-id: https://duplicati.googlecode.com/svn/trunk@427 59da171f-624f-0410-aa54-27559c288bec
