Commit Graph
93 Commits
Author SHA1 Message Date
Carl Johnsen 32efa4b845 Updated all of the handler calls to database methods to forward their cancellation token 2025-06-19 09:47:33 +02:00
Carl Johnsen baab370046 Added the final docstrings 2025-06-18 08:26:20 +02:00
Carl Johnsen 3d11a05400 Added .ConfigureAwait(false) to all of the await statements 2025-06-12 08:34:29 +02:00
Carl Johnsen 996f2b16d8 Updated the FilelistProcessor operation to use the new async database methods 2025-05-21 15:34:00 +02:00
Carl Johnsen 63141d4645 Removed whitespace 2025-05-21 15:33:19 +02:00
Kenneth Skovhede 0df9d67a6a Implemented handling for cases where the previous fileset ends up being empty and removed.
This fixes #5862
2025-05-14 23:36:48 +02:00
Kenneth SkovhedeandGitHub 39b12b958d Merge pull request #6191 from duplicati/feature/warn-more-on-extra-files
Warn more on extra files
2025-04-25 13:59:20 +02:00
Kenneth Skovhede 69df1d5eec Fixed logic to correctly see if ArchiveTime is set 2025-04-24 20:30:56 +02:00
Kenneth Skovhede 75f7582014 Fixed passing transaction to more places 2025-04-24 20:28:26 +02:00
Kenneth Skovhede 8272ea3370 Warn more on extra files
If there are extra unknown files, this can be a sign that multiple backups are using the same folder.

It can also be a sign that an older database is used for some reason.

This PR updates the error message to mention these two common cases, and spell out what the repair process will do.
2025-04-23 14:10:15 +02:00
Kenneth Skovhede dd07dabee1 Merge commit 'b1f705f71a39e4ea79c7dae56bfd0fb345af5110' into feature/support-glacier-style-non-readable-files 2025-03-18 20:44:05 +01:00
Kenneth Skovhede 32762bec34 Fixed comment and added warning message 2025-03-10 10:59:53 +01:00
Kenneth Skovhede 9a82beaf8b Fixed log messages to correspond to actual operation performed 2025-03-10 10:55:07 +01:00
Kenneth Skovhede 12cabf8da7 Merge branch 'feature/improved-failure-resilience' of github.com:duplicati/duplicati into feature/improved-failure-resilience 2025-03-08 15:28:24 +01:00
Kenneth Skovhede 5f8ff74bd5 Streamlined handling of previous backup's fileset and dlist file if it failed to upload. 2025-03-08 15:25:55 +01:00
Kenneth Skovhede c3762a57ca Added docs 2025-03-07 15:23:06 +01:00
Kenneth Skovhede ca236f03e3 Renamed flag to more precisely describe the purpose 2025-03-06 14:56:35 +01:00
Kenneth Skovhede 7beeb27ade Extended tests and fixed a case where a left-over upload could cause errors. 2025-03-06 14:53:46 +01:00
Kenneth Skovhede 4e60a7ad49 Merge remote-tracking branch 'origin/master' into feature/improved-failure-resilience 2025-03-06 12:09:42 +01:00
Kenneth Skovhede 1321488527 Improve failure detection
This commit adds extra logic to prevent cases where the backup reports success but leaves behind partial files.

A new flag is recorded to track if partial files are possible. If no partial files are possible, the verification will fail if any are found.
2025-03-06 12:02:11 +01:00
Kenneth Skovhede c61cc778a8 Log remote filesets
This commit adds the number of known remote filesets to the log output.
2025-02-28 11:18:23 +01:00
Kenneth Skovhede bb5fd80a29 Implemented support for some backends as sources: SSH, CIFS, and S3 2025-02-17 14:01:08 +01:00
Kenneth Skovhede 14ea722e53 Treat archived files with size zero as correct 2025-02-11 17:25:19 +01:00
Kenneth Skovhede 5ae6e3d623 Added support for destinations that do not return the size of archived files 2025-02-11 10:07:09 +01:00
Kenneth Skovhede 9806f5cf74 Support for un-marking an archived file 2025-01-29 21:44:06 +01:00
Kenneth Skovhede 71b3152f71 Implemented support for not testing archived files 2025-01-29 11:36:45 +01:00
Kenneth SkovhedeandGitHub d99f74ed94 Rewrite the backend manager. (#5896)
* Rewrite the backend manager.

This creates a new more logic backend manager that handles all backend operations.

For each top-level operation, there is now a single backend manager instance that is passed to sub-commands.

The internals of the backend manager are now rewritten to use async logic instead of threading.

The design uses a single task runner that dispatches operations and honors concurrent settings and limits from one place.

The logic for each operation has been moved into a separate files/classes to make it easier to understand each operation.

This fixes #5804

* Fixed a few issues with not awaiting tasks

* Fixed not creating empty databases

* Fixed an issue with hashes not being recorded on download

* Reworked the download logic to avoid attempting to decrypt the file if it has been modified.

* Review fixes

* Renamed db collector to better reflect the purpose.

* Review fixes
2025-01-28 08:54:50 +01:00
Kenneth SkovhedeandGitHub 9464caf622 Feature/update license 2025 (#5851)
* Fixed some minor whitespace issues

* Updated all copyright to 2025
2025-01-07 09:40:39 +01:00
Kenneth SkovhedeandGitHub 2a757dea0f Quota check disabled fix (#5843)
The QuotaCheck is done in two places, and only one place was checking if the quota check was supposed to be disabled.
This adds the check in the other place.
2025-01-06 14:21:53 +01:00
Kenneth Skovhede 586c08522e Implemented async method for Test, CreateFolder and Quota in all backends.
Still missing the List and DNSHostnames.
The AsyncRequest class is also preventing many backends from fully upgrading.
2024-10-01 09:30:02 +02:00
Kenneth Skovhede 757dbcade2 Implemented different take on PreBackupVerify 2024-09-10 11:38:38 +02:00
Jojo-1000 50808c866b Merge branch 'master' into quota-disable 2024-05-11 23:46:19 +02:00
Kenneth Skovhede 40dd3c6816 Updated all license mentions to MIT 2024-02-28 15:45:30 +01:00
Jojo-1000 aa365f7f34 Update --quota-size option to assign a backup size.
The assigned quota size is checked in addition to the reported backend quota (if available). It can be used to limit the backup size on the remote. Warnings are created according to the same quota warning threshold.
Previously, this option had no function.
2023-07-16 19:49:54 +02:00
Jojo-1000 8af1dbb6fd Add --quota-disable option to disable reported backend quota.
If set, the backend is treated as if it does not report a quota at all. No warnings and errors are created and no quota values are set in the backup log.
This option is intended to be used if a backend reports a wrong quota, so it needs to be disabled.
2023-07-16 19:16:11 +02:00
Kenneth Hsu e90eb3b2a3 Create custom Exception for remote list verification failures.
Previously, we would catch any Exception thrown by VerifyRemoteList and
perform an auto-cleanup (if specified in the options).  However, we
should only perform the auto-cleanup if VerifyRemoteList detected an
issue with the uploaded files.  Otherwise, an unrelated exception can
cause the database repair to be performed unnecessarily.  In the case of
a connection issue, this can leave the database in a corrupted state.

This fixes #4516.
2021-05-20 13:42:41 -07:00
Kenneth Hsu de1b89167b Rename local variable. 2020-04-05 12:00:51 -07:00
Kenneth Hsu b53770b589 Add VerifyRemoteList overload to reduce duplicated code. 2020-04-05 11:46:08 -07:00
Sean Templeton d5e064ae45 Fix a few more issues with stop now
When a backup is interrupted it will now show as a partial backup in the list of backups on the restore files page.

Temporary filelist volumes are protected from deletion in several places. All but the RestoreHandler
only protect the most recent filelist. The RestoreHandler protects all temporary filelist volumes as the user
might be restoring from one that would be deleted.

Prevent the CoCoL BackendRequests channel is retired exception from appearing when the backup is stopped by checking
if the channel is retired before trying to add the FlushRequest to the channel. Any exceptions from the BackendUploader
task will still be thrown.
2020-03-01 21:19:07 -06:00
Kenneth Hsu 83ac03843d Fix spelling errors in comments.
In doing so, we also normalized some line endings.
2019-12-14 09:52:55 -08:00
Kenneth Hsu afab61a05b Use pattern matching to simplify casts. 2019-10-19 10:15:38 -07:00
BlueBlock 5614c3ef81 change where gui backup version count gets the count
get the count from the db so we can filter on isFullBackup
2019-08-18 14:42:39 -04:00
Kenneth Hsu d72128ec83 Remove unused method parameters. 2018-10-06 15:09:43 -07:00
Kenneth Skovhede 79f7b22e0f Merge branch 'master' into concurrent_processing
Changed a few things on the way, removing the log process

# Conflicts:
#	Duplicati/Library/Main/Controller.cs
#	Duplicati/Library/Main/Database/ExtensionMethods.cs
#	Duplicati/Library/Main/Database/LocalBackupDatabase.cs
#	Duplicati/Library/Main/Duplicati.Library.Main.csproj
#	Duplicati/Library/Main/Operation/BackupHandler.cs
#	Duplicati/Library/Main/Operation/FilelistProcessor.cs
#	Duplicati/Library/Main/Operation/PurgeBrokenFilesHandler.cs
#	Duplicati/Library/Main/Operation/TestFilterHandler.cs
#	Duplicati/Library/Main/Options.cs
#	Duplicati/Library/Main/ResultClasses.cs
#	Duplicati/UnitTest/BasicSetupHelper.cs
2018-04-11 23:02:47 +02:00
Kenneth Skovhede 842fd96543 Implemented a new logging system that is more transparent and allows a more granular way of picking log messages.
Added ID's to each log message and each exception to allow later introduction of a Knowledgebase service that explains each error in more detail.
2018-03-15 09:12:34 +01:00
Tyler Gill 0a1e98712a Adds warnings and errors if the reported available quota either reaches a configurable percentage of the backup size, or when it is completely exhausted.
This is configurable via the --quota-warning-threshold command line parameter.
2018-02-07 11:23:52 -07:00
Kenneth Skovhede 75bf6857c1 Merge branch 'master' into concurrent_processing
# Conflicts:
#	.travis.yml
#	Duplicati/Library/Main/Database/LocalBackupDatabase.cs
#	Duplicati/Library/Main/Operation/BackupHandler.cs
#	Duplicati/Library/Main/ResultClasses.cs
#	Duplicati/Library/Main/Volumes/VolumeWriterBase.cs
#	Duplicati/Library/Utility/TempFile.cs
2018-01-20 15:02:41 +01:00
Kenneth Hsu 6abe087c18 Use helper method to get unique items and duplicates. 2017-09-28 21:55:15 -07:00
Kenneth Skovhede 7ed1362e4c Merge branch 'master' into concurrent_processing
# Conflicts:
#	Duplicati/Library/Main/Operation/BackupHandler.cs
2017-09-27 09:58:29 +02:00
Kenneth SkovhedeandGitHub 436dd7f017 Merge pull request #2761 from warwickmm/refactor/simplify_duplicate_file_check
Use simpler data structures when detecting duplicate filenames
2017-09-26 15:42:46 +02:00