This adds a number of additional features to the repair process so it can actually fully recover if all data is present.
Also added a several tests to ensure that the functionality works in multiple scenarios.
This fixes#5987
* Rewrite the backend manager.
This creates a new more logic backend manager that handles all backend operations.
For each top-level operation, there is now a single backend manager instance that is passed to sub-commands.
The internals of the backend manager are now rewritten to use async logic instead of threading.
The design uses a single task runner that dispatches operations and honors concurrent settings and limits from one place.
The logic for each operation has been moved into a separate files/classes to make it easier to understand each operation.
This fixes#5804
* Fixed a few issues with not awaiting tasks
* Fixed not creating empty databases
* Fixed an issue with hashes not being recorded on download
* Reworked the download logic to avoid attempting to decrypt the file if it has been modified.
* Review fixes
* Renamed db collector to better reflect the purpose.
* Review fixes
* Added test to verify that compression modules will mask duplicate entries, if present
* Because there are likely many, otherwise valid, dindex zip archives with duplicated blocklists out there, the log message is now verbose.
To make sure we no longer create archives with duplicated entries, the `--unittest-mode` option makes this a warning that is checked in most tests.
* Added guard in `IndexVolumeCreator` to prevent creating zip files with duplicated block lists.
* Added debug check for duplicates
* Fixed the duplicate test
Update the SQL to also group on size (just in case).
Renamed the normal read method to signal that it is unverified.
Made the verified reader use enumeration state machine to avoid materializing the list.
`var`-ified some of the test code.
When the file list is over 4GB one needs to have the zip64 option
enabled. This commit catches the exception and reports it as an error so
that the user knows what the problem is.
Fixes duplicati/duplicati#/4206
One of the first things UploadSyntheticFilelist does is throw an exception if the volume is not in the Uploaded state. Or if it is in the Uploaded state, the next check would prevent a synthetic filelist from being created if the volume was not in the Uploading or Temporary states.
Removed the check for the volume not being in the Uploaded state as the warning in the second check handles that case.
Removed the using statement as it would dispose of the FilesetVolumeWriter after passing it to the UploadChannel. The UploadChannel could not use it since it had been disposed.
Fixed the GetRemoteVolumeFromIDAsync taking a fileset id and matching it up against a volume id. Now it will use fileset id to get the volume that matches the fileset.
Change the StopNow disruption test to not use a synthetic file list. The test was written with it not working and therefore fails a check on the number of filesets after stopping a backup.
Bonus fix for a possible null exception in Dispose for VolumeWriterBase.
This fixes a regression introduced around revisions e0899a2a91 ("add
method to add the filelist file") and 3b5af1cf01 ("add method to add
filelist file"), where the filelist was not written by all usages of the
FilesetVolumeWriter. Previously, one would have to invoke
AddFilelistFile with each usage. Now, we simply do so when the
FilesetVolumeWriter is closed.
We also modified the UploadSyntheticFilelist so that the usage of the
FilesetVolumeWriter is contained in a using statement, which will ensure
that it is disposed.
This fixes issue #3924.
This reverts a change made in revision 31ac40083a ("Prevent temp files
in VolumeWriterBase from being deleted prematurely" in pull request #2637).
It's unclear what issue was being worked around by preventing the
removal of the temporary files (based on comments in pull request #2637,
it appears to be related to tests that failed only on Travis). However,
this workaround prevented the removal of temporary files created during
compact operations, which caused several users' temporary directories to
be filled up with files that should have been removed.
This concerns issue #3652.
This ensures the removal of the temporary file containing the
filelist.json that remained regardless if files needed to be uploaded or
not.
This concerns issue #3652.
Kenneth Skovhede