haeaeae/duplicati
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
4,307 Commits 140 Branches 416 Tags
4d77fd7ebcc54d8255b8bad730ef7e14dc36ec87
Commit Graph
77 Commits
Author SHA1 Message Date
Kenneth Skovhede 4d77fd7ebc Flipped the check for existing metadata 2017-05-30 10:29:18 +02:00
Kenneth Skovhede d18431b385 Merge branch 'master' into concurrent_processing 
# Conflicts:
#	Duplicati/Library/Main/Controller.cs
#	Duplicati/Library/Main/Database/ExtensionMethods.cs
#	Duplicati/Library/Main/Database/LocalBackupDatabase.cs
#	Duplicati/Library/Main/Duplicati.Library.Main.csproj
#	Duplicati/Library/Main/Operation/BackupHandler.cs
#	Duplicati/Library/Main/Operation/RecreateDatabaseHandler.cs
#	Duplicati/Library/Main/ResultClasses.cs

First attempt at merging all changes from 2016-10-08 to now :/
 2017-05-30 00:35:31 +02:00
Kenneth Skovhede 6cadaad56a Rewrote the query to use CROSS JOIN to force the SQLite query planner to consider the File table first. 2017-05-19 08:07:39 +02:00
Kenneth Skovhede d9014ee09c Changed query as suggested by @sjonez in issue #2442. 
This fixes #2442.
 2017-05-18 23:40:24 +02:00
Dan Stahr 27c9e15875 Improved database performance and logging 
- Don't use Vacuum() just before PurgeLogData (which performs vacuum() already)
 - Removed unncessary poorly performning query (after optimizing it I found out it's unused)
 - Correctly log calls for ExecuteScalarInt64
 2017-03-23 15:22:58 +00:00
Kenneth Skovhede f5a1bcd32c Reverted the SQL query to use the Timestamp sorting instead of explicit fileset. 
This fixes #2244
 2017-01-11 23:48:41 +01:00
Kenneth Skovhede 9a849a2b58 Improved support for cross-os restores. 
This fixes #1302
This fixes #1983
 2017-01-06 23:01:54 +01:00
Kenneth Skovhede 179290d521 Added more extensive listing to further pinpoint the source of the error in #1400 2017-01-05 22:15:59 +01:00
Kenneth Skovhede 76aae5de7b Changed the logic for getting the last-modified timestamp from the database. 
Now we only look at the timestamp from the previous fileset, instead of for the last time the file was seen.
This should improve lookup times, and thus the running time for backups with many files.

This change means that the following sequence of operations will react differently:

- backup
- delete file
- backup
- undelete file
- backup

Prior to this commit, the undeleted file would be seen with the original timestamp and thus can avoid being scanned for changes.
After this commit, the file will be treated as a new file, even if it has the previous timestamp and will thus be scanned for changes.

An additional switch has been introduced, --check-filetime-only that improves the lookup further by not looking at the metadata or filesize, and thus makes a much lighter call to the database, where fewer tables need to be touched.
 2017-01-05 10:03:32 +01:00
Kenneth Skovhede 9a227ada6b Reworked the cleanup process to allow a dangling filelist, such that this dangling entry can be used as the basis for a synthetic filelist. 
Also added an option to disable synthetic filelists.
This fixes #2038.
This fixes #2101
 2016-12-24 11:42:26 +01:00
Kenneth Skovhede 3182a227b9 Added additional debug info for tracing the cause of issue #1400 2016-12-23 17:04:12 +01:00
Kenneth Skovhede d8972b0f2b Removed unused memory cache code 2016-12-16 17:30:39 +01:00
Kenneth Skovhede c1c8c9a1aa Merge branch 'master' into concurrent_processing 
Some changes were required for this merge,
now the StreamBlockSplitter process will take in any stream, store the blocks and return the blockset ID.
This makes it compatible with the changes for metadata handling where they support more than a single block of metadata.

# Conflicts:
#	Duplicati/Library/Main/Database/LocalDatabase.cs
#	Duplicati/Library/Main/Database/LocalRepairDatabase.cs
#	Duplicati/Library/Main/Operation/BackupHandler.cs
#	Duplicati/Library/Main/Operation/RecreateDatabaseHandler.cs
 2016-10-18 13:27:36 +02:00
Kenneth Skovhede 309bf1648d Merge branch 'master' into feature/fix_for_different_blockhash_and_filehash_plus_multiblocks_metadata 
# Conflicts:
#	Duplicati/Library/Main/Database/LocalDatabase.cs
#	Duplicati/Library/Main/Database/LocalRecreateDatabase.cs
#	Duplicati/Library/Main/Operation/RecreateDatabaseHandler.cs
#	Duplicati/UnitTest/BorderTests.cs
 2016-10-17 13:28:27 +02:00
Kenneth Skovhede 94c5e10480 Merge branch 'master' into concurrent_processing 
# Conflicts:
#	Duplicati/Library/Main/Database/LocalBackupDatabase.cs
#	Duplicati/Library/Main/Database/LocalDatabase.cs
#	Duplicati/Library/Main/Database/RemoteVolumeEntry.cs
#	Duplicati/Library/Main/Operation/BackupHandler.cs
#	Duplicati/Library/Main/Operation/RecreateDatabaseHandler.cs
#	Duplicati/Library/Main/Operation/RepairHandler.cs
#	Duplicati/Library/Main/Operation/RestoreControlFilesHandler.cs
 2016-09-16 10:33:07 +02:00
Kenneth Skovhede 0c68daff6c Fixed all whitespace to be 4 spaces instead of tabs 2016-09-15 11:39:27 +02:00
Kenneth Skovhede f0c0fc3a2e Merge branch 'master' into concurrent_processing 
# Conflicts:
#	Duplicati/Library/Main/Database/LocalDatabase.cs
#	Duplicati/Library/Main/Duplicati.Library.Main.csproj
#	Duplicati/Library/Main/Operation/BackupHandler.cs
#	thirdparty/CoCoL/CoCoL.dll
 2016-08-08 15:03:42 +02:00
FootStark 568cc60856 Fixed lookup of a file's last entry during backup 
- Fixed SQL to create temp table for lookup of last backup's file
entries
- Removed temp table by writing directly to internal lookup struct.
- Direct lookup of single file entries from DB
 2016-05-06 00:14:03 +02:00
Kenneth Skovhede 742eab0913 Fixed multiple cases where connections and command instances were not disposed. 
This should fix all cases of "File is locked" on windows.
 2016-04-06 20:40:34 +02:00
Kenneth Skovhede 66eb681541 Fixed issues that arose when using multi-threaded execution 2016-04-05 23:29:14 +02:00
Kenneth Skovhede 72b33bcf93 Extra fixes to support larger metadata (up to appx 300 mb with default blocksize) 2016-04-03 23:54:52 +02:00
Kenneth Skovhede e40bd7f02a Proposed version, not complete with handling large metadata chunks 2016-04-03 23:10:47 +02:00
Kenneth Skovhede 537fe20838 Added a lot of comments. 
Fixed a few issues that caused parallel uploads.
Fixed the use of the encryption module so the encryption module is not re-used for multiple jobs.
 2016-02-25 19:15:51 +01:00
Kenneth Skovhede 8abdf87188 More cleanup and fixes for the concurrent processing 2016-02-20 15:06:29 +01:00
Kenneth Skovhede 086547e655 Implemented the missing details for the concurrent setup, it now compiles 2016-02-18 17:20:52 +01:00
Kenneth Skovhede ec877696b0 More work on concurrent processing 2016-02-18 16:46:53 +01:00
Kenneth Skovhede 0de9adc606 Added grace-period for incomplete uploads to avoid backups stopping due to the Apache WebDAV issues. 
This fixes #1243.
 2015-04-05 14:33:13 +02:00
Kenneth Skovhede 4eefccc4f2 Fixed an issue where changed metadata was not stored if the file data had not changed. 
This also includes a fix for a path-lookup that was larger than needed.
 2015-01-25 21:55:15 +01:00
Kenneth Skovhede 4846ffa103 Now using the file modification timestamp and the filesize as indicators that trigger file scanning. 
This fixes #1099.
 2015-01-24 21:59:53 +01:00
Kenneth Skovhede 4f9de30936 Added upload of detected missing index files 2014-12-30 18:29:07 +01:00
Kenneth Skovhede 3bcf7cf695 Added fixes to prevent the bare-metal recovery issue discussed in issue #1218 and #1214. 2014-12-30 16:15:21 +01:00
Kenneth Skovhede e6f68eac67 Fixed one more speed issue when running with low-memory, it is now only marginally slower when not using memory caches. 2014-10-31 11:01:23 +01:00
Kenneth Skovhede d05f6f1324 Improved performance when running with all in-memory caches disabled by a factor of 16. 
On a smaller dataset, the performance is now only a little more than twice as slow with a low memory footprint compared to the default with all memory caches enabled.
 2014-10-30 23:45:36 +01:00
Kenneth Skovhede 4fab98f0f1 Fixed another bug with inserting duplicate metadatasets 2014-10-30 21:36:27 +01:00
Kenneth Skovhede 1547d0fd1e Fixed a bug with creating duplicate metadataset entries 2014-10-30 15:43:24 +01:00
Kenneth Skovhede c030e321c6 Improved error messages when a faulty database is discovered 2014-10-30 15:41:51 +01:00
Kenneth Skovhede c53f43a083 Fixed a bug with metadata lookup. 2014-10-30 00:15:25 +01:00
Kenneth Skovhede 0c1afd15a5 Improved a performance issue with counting the number of changed files 2014-08-12 11:04:57 +02:00
Kenneth Skovhede 796459df5d Fix for hash collisions with different size inputs 2014-05-14 14:44:56 +02:00
Kenneth Skovhede 6df9fba2f3 Fixed handling of paths that change from file to folder, and folder to file. 
Renamed FilesetId to FileID as that is what it is.
 2013-11-20 19:18:12 +01:00
Kenneth Skovhede a712d969c4 Fixed issue with duplicate files inserted in filelist 2013-11-01 23:51:00 +01:00
Kenneth Skovhede 04bd3ac9c9 Improved speed for creating the scantime lookup table 2013-09-25 23:30:19 +02:00
Kenneth Skovhede 32a617b1e2 Rewrote the changed/added/deleted files to count using temporary table (minor speedup) 2013-09-10 20:42:54 +02:00
Kenneth Skovhede 4913a27cac Added some extra profiling output 2013-09-10 08:29:12 +02:00
Kenneth Skovhede 1f48d43515 Removed the HashDatabaseProtector and added the HashLookupHelper instead. 2013-09-09 22:50:18 +02:00
Kenneth Skovhede c2a10f268a Rewrote the changed file query SQL 2013-09-09 22:48:26 +02:00
Kenneth Skovhede 555945eae4 Implemented a better cache for filenames. 2013-09-07 21:33:59 +02:00
Kenneth Skovhede 818554de4e Bugfix for the block cache 2013-09-04 22:10:36 +02:00
Kenneth Skovhede ffcd62a412 Added lookup table to avoid searching the database when adding a blocksetentry 2013-09-03 23:48:42 +02:00
Kenneth Skovhede a75f8e087d Rewrote the logic that fixes missing filelist uploads 2013-08-31 15:30:34 +02:00