This should fix a number of cases where the backup returns a consistency error because the deletion and the backend check happens before the backend is in sync.
A simple backup that specifies --deleted-files causes a Fatal error.
Shortened repro steps:
(See Github comment for full repro steps)
>duplicati.commandline.exe backup file://C:\data\local\testchangedfiles\destination "C:\data\local\testchangedfiles\source\\" --encryption-module="none" --no-encryption="true" --dbpath="testchangedfiles.sqlite" --compression-module="zip" --keep-versions=-1 --no-auto-compact="true"
>del source\file6.txt
>duplicati.commandline.exe backup file://C:\data\local\testchangedfiles\destination "C:\data\local\testchangedfiles\source\\" --changed-files="C:\data\local\testchangedfiles\source\file1.txt;C:\data\local\testchangedfiles\source\file2.txt" --deleted-files="C:\data\local\testchangedfiles\source\file6.txt" --encryption-module="none" --no-encryption="true" --dbpath="testchangedfiles.sqlite" --compression-module="zip" --keep-versions=-1 --no-auto-compact="true"
Results in:
Fatal error => Input string was not in a correct format.
System.FormatException: Input string was not in a correct format.
at System.Number.StringToNumber
at System.Number.ParseInt64
at System.String.System.IConvertible.ToInt64
at System.Convert.ToInt64
at System.Data.SQLite.SQLiteStatement.BindParameter
at System.Data.SQLite.SQLiteStatement.BindParameters
at System.Data.SQLite.SQLiteCommand.BuildNextCommand
at System.Data.SQLite.SQLiteCommand.GetStatement
at System.Data.SQLite.SQLiteDataReader.NextResult
at System.Data.SQLite.SQLiteDataReader..ctor
at System.Data.SQLite.SQLiteCommand.ExecuteReader
at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery
at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery
at Duplicati.Library.Main.Database.LocalBackupDatabase.AppendFilesFromPreviousSet
at Duplicati.Library.Main.Database.LocalBackupDatabase.AppendFilesFromPreviousSet
at Duplicati.Library.Main.Operation.BackupHandler.RunMainOperation
at Duplicati.Library.Main.Operation.BackupHandler.Run
at Duplicati.Library.Main.Controller.RunAction[T]
at Duplicati.Library.Main.Controller.Backup
at Duplicati.CommandLine.Commands.Backup
at Duplicati.CommandLine.Program.RunCommandLine
>Exit code: 100
With this patch, backup completes successfully.
- Don't use Vacuum() just before PurgeLogData (which performs vacuum() already)
- Removed unncessary poorly performning query (after optimizing it I found out it's unused)
- Correctly log calls for ExecuteScalarInt64
Previously, the output of the first replace was never used and therefore the absolute Windows paths were not restored
correctly.
The two patterns (i.e. ?: and \\) are disjoint so rewriting the previous result in case a match is found
for the second one is OK.
This is a less experimental rewrite of the query that fixes the exact problem.
The speedups from this query fix are also substantial, going from 2.5 hours to 11 minutes.
This is likely a fix to these issues:
#1699, #2048, #2140, #2178
This could help with issues:
#1391, #2050
Now we only look at the timestamp from the previous fileset, instead of for the last time the file was seen.
This should improve lookup times, and thus the running time for backups with many files.
This change means that the following sequence of operations will react differently:
- backup
- delete file
- backup
- undelete file
- backup
Prior to this commit, the undeleted file would be seen with the original timestamp and thus can avoid being scanned for changes.
After this commit, the file will be treated as a new file, even if it has the previous timestamp and will thus be scanned for changes.
An additional switch has been introduced, --check-filetime-only that improves the lookup further by not looking at the metadata or filesize, and thus makes a much lighter call to the database, where fewer tables need to be touched.
Kenneth Hsu