Commit Graph
2357 Commits
Author SHA1 Message Date
gpatel-frandGitHub b08b59228c report all filesets in error in VerifyConsistency (#5052)
only reporting the first one could incite to deleting
it while many filesets can be impacted by a file error
2023-12-12 18:25:14 +01:00
gpatel-frandGitHub 79b87566dd set default log level to Dryrun if dryrun option is set (#5059) 2023-12-11 14:42:45 +01:00
gpatel-frandGitHub 0de4cb7968 allow full-remote-verification to target only list and index files (#5037)
The option is turned into an enum with values True, False, ListIndexes
True and False are for compatibility with the existing backups.
ListIndexes is equivalent to True but don't handle block volumes.
This is to make more workable a strategy of full remote testing
after each backup.
2023-12-10 00:24:07 +01:00
gpatel-frandGitHub af4fd5af05 change error reporting for full backup remote verification (#5004)
Before the change, it could happen that when the postbackup test
was finding problems in the backend, the job would be reported
as having completed successfully, the found problems being displayed
only in the complete log. After the change, an error is logged
that cause the job to be failed.
To repro, the following procedure can be used in a test backup:
change a hash for a block in the block table database and run a backup having set
--full-remote-verification and --full-block-verification to true and
--backup-test-percentage to 100.
2023-12-10 00:23:16 +01:00
gpatel-frandGitHub ec9d49d684 Experiment/rebuildfaster (#5043)
* experimental faster db rebuild (#4955)

* try run update db query for every block file only when needed
add repair-force-block-use option
more appropriate naming for some routines
more tracing
more comments
2023-11-05 18:19:08 +01:00
gpatel-frandGitHub 43dea88b4f enable to perform tests without checking file list (#5054)
following the option --disable-filelist-consistency-checks
2023-11-04 14:25:59 +01:00
Jojo-1000 71cf250450 Fix missing index-block link in interrupted backup.
See issue #5023.
The index-block link should be created before the upload is started, otherwise an interrupted backup could cause an index file without a link.
2023-10-08 22:34:15 +02:00
Jojo-1000 2a2dd92ba4 Fix index block link for extra index files in repair.
The link was created before the volume was added to the database, resulting in ID -1.
2023-10-08 22:34:15 +02:00
Jojo-1000 33aab2aec2 Add test for repair with extra index files. 2023-10-08 22:34:09 +02:00
gpatel-fr b8b1540a5c add simultaneous file processors
the --concurrent-fileprocessors (default 1 for compatibility) set the value for a job
experimental.
2023-09-22 00:28:11 +02:00
Janandgpatel-fr b894a197a9 Fix missing file error caused by interrupted compact.
Before the dindex files are deleted on the remote, update the state in the database to Deleting. Previously, only dblock files were handled this way.
Patch by warwickmm
Closes #4129
2023-09-21 22:56:52 +02:00
Janandgpatel-fr 71eb03130a Improve result reporting for interrupted/failed backups (Fix #4829)
Direct import of PR#4978
Update web UI for new result reports.
For operations with fatal errors, write logs to same operation ID.
Test that Interrupted flag is correct in RunScriptTests.
Update backup log display for new result reporting.
Hide file statistics for fatal errors and change fatal icon.
2023-09-21 20:01:17 +02:00
Jojo-1000 aa365f7f34 Update --quota-size option to assign a backup size.
The assigned quota size is checked in addition to the reported backend quota (if available). It can be used to limit the backup size on the remote. Warnings are created according to the same quota warning threshold.
Previously, this option had no function.
2023-07-16 19:49:54 +02:00
Jojo-1000 8af1dbb6fd Add --quota-disable option to disable reported backend quota.
If set, the backend is treated as if it does not report a quota at all. No warnings and errors are created and no quota values are set in the backup log.
This option is intended to be used if a backend reports a wrong quota, so it needs to be disabled.
2023-07-16 19:16:11 +02:00
Jojo-1000 b6643fa473 Add documentation comments for LocaleChange. 2023-06-23 14:22:36 +02:00
Jojo-1000 708715e705 Fix force-locale option for CLI and GUI.
- Localization culture is set from multiple threads on the server, so use current local context for every new call instead of global context
- Change locale before executing help commands if flag is present
- When force-locale is set, also change locale for log messages

Closes #4176
2023-06-23 13:30:46 +02:00
Russell Smith 6fb3206b22 Use 1MB default block as suggested by users.
This still allows 50 blocks for the default upload size.
It reduces the blocks to track by a factor of 10.
If you are using larger backups, you want a larger block size.

If you have small backups and it's a problem, recreating those
will take less time.
2023-04-22 16:50:48 +10:00
gpatel-fr cf17f488d1 update Newtonsoft lib to 13.02 2023-03-24 23:17:11 +01:00
Thomas Suckow 78197453a1 Merge remote-tracking branch master into feature/net5-avalonia
# Conflicts:
#	Duplicati CommandLine Only.sln
#	Duplicati.sln
#	Duplicati/CommandLine/BackendTester/Duplicati.CommandLine.BackendTester.csproj
#	Duplicati/CommandLine/BackendTool/Duplicati.CommandLine.BackendTool.csproj
#	Duplicati/CommandLine/Duplicati.CommandLine.csproj
#	Duplicati/CommandLine/RecoveryTool/Duplicati.CommandLine.RecoveryTool.csproj
#	Duplicati/GUI/Duplicati.GUI.TrayIcon/Duplicati.GUI.TrayIcon.csproj
#	Duplicati/GUI/Duplicati.GUI.TrayIcon/app.config
#	Duplicati/Library/AutoUpdater/UpdaterManager.cs
#	Duplicati/Library/Backend/Jottacloud/Duplicati.Library.Backend.Jottacloud.csproj
#	Duplicati/Library/Main/Duplicati.Library.Main.csproj
#	Duplicati/Library/Utility/Duplicati.Library.Utility.csproj
#	Duplicati/Library/Utility/Utility.cs
#	Duplicati/Server/Duplicati.Server.csproj
#	Duplicati/Server/app.config
#	Duplicati/Service/app.config
#	Duplicati/UnitTest/BackendToolTests.cs
#	Duplicati/UnitTest/Duplicati.UnitTest.csproj
#	guiTests/guiTest.py
2022-11-11 21:40:08 -08:00
dgasawayandGitHub 0ba4da94a6 Add a few more compressed, audio, and video exclusions
Supporting information:
https://wiki.xiph.org/MIME_Types_and_File_Extensions
https://en.wikipedia.org/wiki/Opus_(audio_format)
https://en.wikipedia.org/wiki/Zstd
https://en.wikipedia.org/wiki/MPEG-2#Filename_extensions
2022-09-06 14:52:51 -07:00
Kenneth SkovhedeandGitHub 1935e3f642 Merge pull request #4601 from warwickmm/fix_patch_with_local_blocks
Fix implementation of Options.PatchWithLocalBlocks
2022-06-14 22:12:54 +02:00
martengooz 7f9daa6041 Add additional compression exclusions 2022-06-12 17:16:35 +02:00
Thomas Suckow 8c518f7d3a .net6 2022-04-12 21:43:50 -07:00
albertony c490009f20 Delay creation of database directory until we know it is needed.
This avoids creating directory %LocalAppData%\Duplicati when it would just be left
empty, and it makes running portable mode (e.g. debug builds) stealthier.
2022-04-01 21:18:19 +02:00
Thomas Suckow bf5b9537d8 Merge remote-tracking branch 'thomas/feature/net5-freebsd' into feature/net5-unsplit
# Conflicts:
#	Duplicati.sln
#	Duplicati/CommandLine/BackendTester/Duplicati.CommandLine.BackendTester.csproj
#	Duplicati/CommandLine/BackendTool/Duplicati.CommandLine.BackendTool.csproj
#	Duplicati/CommandLine/CLI/Commands.cs
#	Duplicati/CommandLine/Duplicati.CommandLine.csproj
#	Duplicati/CommandLine/RecoveryTool/Duplicati.CommandLine.RecoveryTool.csproj
#	Duplicati/GUI/Duplicati.GUI.TrayIcon/Duplicati.GUI.TrayIcon.csproj
#	Duplicati/GUI/Duplicati.GUI.TrayIcon/ImageLoader.cs
#	Duplicati/Library/Backend/SSHv2/Duplicati.Library.Backend.SSHv2.csproj
#	Duplicati/Library/Backend/SSHv2/packages.config
#	Duplicati/Library/Backend/Tardigrade/Duplicati.Library.Backend.Tardigrade.csproj
#	Duplicati/Library/Common/Duplicati.Library.Common.csproj
#	Duplicati/Library/SQLiteHelper/Duplicati.Library.SQLiteHelper.csproj
#	Duplicati/Library/Snapshots/Duplicati.Library.Snapshots.csproj
#	Duplicati/License/Duplicati.License.csproj
#	Duplicati/Server/Duplicati.Server.csproj
#	Duplicati/UnitTest/Duplicati.UnitTest.csproj
#	Duplicati/UnitTest/TestUtils.cs
#	Duplicati/UnixSupport/UnixSupport.csproj
#	Installer/Makefile/Makefile
#	guiTests/guiTest.py
2022-03-03 21:20:34 -08:00
tariandeath d44496f6d4 Add semicolons 2022-02-27 13:29:58 -07:00
JEDandGitHub 662bad4526 Add newline 2022-02-27 00:13:37 -07:00
tariandeath f6ebcd6262 Fix typo 2022-02-26 20:45:00 -07:00
tariandeath 8b24536d4e Add Upgraded Version to binary 2022-02-26 20:39:14 -07:00
tariandeath 396119aa40 Add indexes to improve backup query performance. 2022-02-26 20:24:14 -07:00
Kenneth SkovhedeandGitHub f0dfe67c89 Merge pull request #4660 from warwickmm/download_error_log_filename
Always log filename if error occurs during retrieval
2022-02-23 22:07:16 +01:00
Thomas Suckow d528703fe7 Merge branch 'master' into experiment/net5-split
# Conflicts:
#	.github/workflows/tests.yml
#	Duplicati.sln
#	Duplicati/CommandLine/BackendTester/Duplicati.CommandLine.BackendTester.csproj
#	Duplicati/CommandLine/BackendTool/Duplicati.CommandLine.BackendTool.csproj
#	Duplicati/CommandLine/Duplicati.CommandLine.csproj
#	Duplicati/CommandLine/RecoveryTool/Duplicati.CommandLine.RecoveryTool.csproj
#	Duplicati/GUI/Duplicati.GUI.TrayIcon/Duplicati.GUI.TrayIcon.csproj
#	Duplicati/Library/Backend/AzureBlob/AzureBlobWrapper.cs
#	Duplicati/Library/Backend/Mega/Duplicati.Library.Backend.Mega.csproj
#	Duplicati/Library/Backend/Mega/packages.config
#	Duplicati/Library/Backend/OAuthHelper/Duplicati.Library.OAuthHelper.csproj
#	Duplicati/Library/Backend/Tardigrade/Duplicati.Library.Backend.Tardigrade.csproj
#	Duplicati/Library/Backend/Tardigrade/Properties/AssemblyInfo.cs
#	Duplicati/Library/Backend/Tardigrade/packages.config
#	Duplicati/Library/Common/Duplicati.Library.Common.csproj
#	Duplicati/Library/Common/IO/SystemIOWindows.cs
#	Duplicati/Library/Modules/Builtin/Duplicati.Library.Modules.Builtin.csproj
#	Duplicati/Library/Modules/Builtin/packages.config
#	Duplicati/Library/Utility/Utility.cs
#	Duplicati/License/Duplicati.License.csproj
#	Duplicati/Server/Duplicati.Server.csproj
#	Duplicati/Server/packages.config
#	Duplicati/UnitTest/Duplicati.UnitTest.csproj
2022-02-22 19:04:01 -08:00
snamds 468d672d87 Improve performance of big restore datasets 2022-01-31 14:26:21 +01:00
snamds f9ea467939 Improve performance of big restore datasets 2022-01-31 14:22:50 +01:00
Riccardo Paolo Bestetti 528478fda4 Implement exponential backoff for backend errors 2022-01-20 23:22:08 +01:00
Kenneth Hsu 6a84e9f972 Always log filename if error occurs during retrieval.
This fixes #4658.
2022-01-08 12:06:33 -08:00
Kenneth Hsu 712588f1d6 Fix implementation of PatchWithLocalBlocks.
Previously, this would return true as long as the key was present in the
Dictionary, regardless of what the value parsed to.
2021-09-03 16:23:28 -07:00
Kenneth SkovhedeandGitHub 6dcbccab28 Merge pull request #4520 from warwickmm/remote_list_verification_exception
Create custom Exception for remote list verification failures
2021-08-11 10:32:18 +02:00
warwickmmandGitHub 184dfeae78 Merge pull request #4492 from TopperDEL/storj-dcs-update
Add Storj DCS backend and deprecated Tardigrade DCS backend.

Storj renamed their "Tardigrade Decentralized Cloud Storage" product to "Storj DCS".  This adds a new Storj DCS backend and deprecates the old Tardigrade one.  A warning is issued to inform Tardigrade users to modify their configurations to use the new backend.

This also adds a passphrase verification field to ensure that the user has provided the correct encryption passphrase.
2021-07-20 08:28:54 -07:00
TopperDEL fc55b16992 fix: More info in deprecation-warning 2021-07-16 08:34:01 +02:00
TopperDEL 5d36791aa5 feat: Inform user about deprecated Tardigrade-Backend with a warning 2021-07-15 17:37:22 +02:00
Kenneth Hsu e90eb3b2a3 Create custom Exception for remote list verification failures.
Previously, we would catch any Exception thrown by VerifyRemoteList and
perform an auto-cleanup (if specified in the options).  However, we
should only perform the auto-cleanup if VerifyRemoteList detected an
issue with the uploaded files.  Otherwise, an unrelated exception can
cause the database repair to be performed unnecessarily.  In the case of
a connection issue, this can leave the database in a corrupted state.

This fixes #4516.
2021-05-20 13:42:41 -07:00
warwickmmandGitHub 7b4f475762 Merge pull request #4502 from samuel-w/master
Make file extension compression hints case-insensitive.
2021-05-09 14:29:59 -07:00
samuel-wandGitHub fa79b406a2 Improve comment 2021-05-09 15:16:31 -05:00
samuel-w d05d83cf4b Add comment 2021-05-08 15:51:10 -05:00
samuel-w bfa24581c1 Add Disk Archive files to compression hints 2021-05-08 01:45:07 -05:00
samuel-w d5c83d2af3 Compare file extensions case insensitively.
Fixes https://github.com/duplicati/duplicati/issues/4493
2021-05-08 01:14:08 -05:00
Thomas Suckow ecb0249259 dotnet no longer provides a method for a string to hashalgorithm
* Also fix up not diposing of hashers
2021-04-04 11:17:13 -07:00
Thomas Suckow d8ddd6982c Refactor VolumeHash 2021-04-04 08:28:44 -07:00
Thomas Suckow 898193fcef For now, just interrupt since its better than NOP. 2021-04-03 20:54:47 -07:00