Commit Graph
586 Commits
Author SHA1 Message Date
Carl-Johannes JohnsenandGitHub 476774640a Merge pull request #5991 from carljohnsen/performance/restoreopt2 2025-02-28 13:03:25 +01:00
Carl Johnsen 6e08affd1a Rather than having the tempfile database create new ID's for each entry, reuse the File.ID so the connections that do not have access to the temp table can access the same data. 2025-02-28 10:27:02 +01:00
Carl Johnsen a0eae4131b Cleaned up LocalRestoreDatabase 2025-02-27 14:14:07 +01:00
Carl Johnsen 692954ee97 Made the local restore database use a connection pool for getting block data 2025-02-26 16:08:55 +01:00
Carl Johnsen a49c66bcb5 Added caching of localrestoredatabase data 2025-02-25 17:42:30 +01:00
Carl Johnsen 6d7faf97bb Whitespace 2025-02-25 17:41:57 +01:00
Kenneth Skovhede 9dfe977d3f Changed logic so the source providers are now logically mounted into the source directory structure.
This fixes the restore issues as all sources will now appear as if they were originally part of the source file system.

The backend implementation has now been simplified so the backends only return names of entries with no path information.  Most backends already do this, but the S3 backends did not, so this was fixed as well.

This also clears up the interfaces, so the backend does not know about the `ISourceProviderEntry`. All mapping is done by the `BackendSourceProvider`, making it simpler to support other backends as sources.
2025-02-21 16:06:55 +01:00
Kenneth Skovhede 1716d09b51 Fixed source providers for S3, CIFS, and SSH.
Made the SMBShareConnection slightly more robust.

Introduced v2 List API to S3 backend, currently it retains v1 as the default.

Changed S3 backends to not recurse, potentially making smaller listings. This option can be disabled if it breaks things.

Added support for relative paths in the SSH backend.
2025-02-21 13:20:52 +01:00
Kenneth Skovhede 1690871d10 Implemented fix that pushes updated timestamps to previous version, in the case that no files were changed, and no new backup is recorded in remote storage. 2025-02-06 15:31:20 +01:00
Kenneth SkovhedeandGitHub f7b7555e87 Fixes for tests (#5918)
* Two small test fixes

* Ignore exceptions when comitting an empty transaction

* Attempt to fix FTP connection issue

* Unified test by having a single download function.
Added DNS probing call to attempt fixing sporadic DNS errors.

* Fixed argument order

* Avoid creating temp file during tests, if not needed
2025-01-28 08:55:28 +01:00
Kenneth SkovhedeandGitHub d99f74ed94 Rewrite the backend manager. (#5896)
* Rewrite the backend manager.

This creates a new more logic backend manager that handles all backend operations.

For each top-level operation, there is now a single backend manager instance that is passed to sub-commands.

The internals of the backend manager are now rewritten to use async logic instead of threading.

The design uses a single task runner that dispatches operations and honors concurrent settings and limits from one place.

The logic for each operation has been moved into a separate files/classes to make it easier to understand each operation.

This fixes #5804

* Fixed a few issues with not awaiting tasks

* Fixed not creating empty databases

* Fixed an issue with hashes not being recorded on download

* Reworked the download logic to avoid attempting to decrypt the file if it has been modified.

* Review fixes

* Renamed db collector to better reflect the purpose.

* Review fixes
2025-01-28 08:54:50 +01:00
Kenneth SkovhedeandGitHub 52a73c3418 Fixed issue 5845 (#5880)
* Fixed issue 5845

This commit introduces the test that makes the error.

* Fixed query cleaning up duplicates, in the case that where a block has multiple valid source candidates.

* Reduced number of replicas for the test
2025-01-16 14:27:15 +01:00
Kenneth Skovhede fb1d15fb9d Avoid optimize after vacuum 2025-01-14 19:42:03 +01:00
Kenneth SkovhedeandGitHub 6bbd64a655 Fixed nullability warnings in Utility.cs (#5861)
* Fixed nullability warnings in `Utility.cs`

* Fixed a number of nullability issues and warnings

* More nullable fixes

* Fixed the overwrite issue

* Fixed additional nullability issues

* Removed additional nullability issues

* More nullability fixes and added some xmldoc

* Added more xmldocs and removed an unused argument

* Removed more warnings
2025-01-14 14:03:48 +01:00
Kenneth SkovhedeandGitHub d64c7e721f Handle duplicated missing blocks (#5858)
* Implemented test for #5845

* Removed unittests not executing cleanup.

* Cleanup commit

* Fixed the query that causes the constraint violation
2025-01-10 09:24:54 +01:00
Kenneth SkovhedeandGitHub 9464caf622 Feature/update license 2025 (#5851)
* Fixed some minor whitespace issues

* Updated all copyright to 2025
2025-01-07 09:40:39 +01:00
Carl Johnsen 198e4d8e6e Refactor SQL query formatting 2025-01-03 11:06:49 +01:00
Carl Johnsen 661a73ff2a Merge remote-tracking branch 'upstream/master' into SlowBigRestore 2025-01-03 10:11:50 +01:00
Carl Johnsen 345701576c Merge branch 'master' into performance/restoreopt 2024-12-05 12:33:30 +01:00
Kenneth Skovhede f5cabd4c7f Remove --use-block-cache
The option has no effect.
This closes #3566
The block cache was added originally to work around perceived slow hash table lookups in the database.
By having an in-memory dictionary, lookups would be much faster.
It has no effect on disk wear, as the lookups are read-only.
The impact of such a cache was measured to be minimal with recent versions of SQLite.
2024-12-04 10:06:49 +01:00
Carl Johnsen 6ba956dfcd Moved all database operations to the LocalRestoreDatabase.
Also changed the old changes, so that the functions related to the legacy restore remains untouched.
2024-12-03 08:35:14 +01:00
Carl Johnsen f85ffabe3e Linux SQLite doesn't want values in quotes 2024-12-02 10:45:52 +01:00
Carl Johnsen 079598c2e8 Added database function for getting files alongside symlinks to restore 2024-11-29 03:41:41 +01:00
Carl Johnsen 7aa9db533e Added preliminary metadata restoration 2024-11-26 20:01:19 +01:00
Carl Johnsen ebac24ce04 Rather than specifying multiple guids for a restore, use the one that the restore database has. 2024-11-25 08:52:51 +01:00
Carl Johnsen 0cd9374bed Added comments and docstrings 2024-11-22 15:26:04 +01:00
Carl Johnsen 43a392f139 The tempfiletable name is also required externally from the database. 2024-11-22 11:26:25 +01:00
Carl Johnsen 878780dfe4 Added Database.ExecuteReaderEnumerable for prepared statements 2024-11-19 10:32:54 +01:00
Carl Johnsen 50c6fb434c Got the list of block IDs per file 2024-11-08 05:56:48 +01:00
Kenneth SkovhedeandGitHub 3e3258b081 Merge pull request #5615 from duplicati/feature/fix-issue-5038
Feature/fix issue 5038
2024-11-05 10:07:00 +01:00
Kenneth Skovhede cae5b218d4 Potential fix for the missing blocklist hash 2024-11-04 21:46:12 +01:00
Kenneth SkovhedeandGitHub e959542536 Merge pull request #5623 from duplicati/feature/fix-compact-disruption-test
Fix issues with extra blocks and files being recorded after an interrupted `Compact()`
2024-11-04 16:59:23 +01:00
Kenneth SkovhedeandGitHub ababf2f84e Merge pull request #5610 from duplicati/feature/fix-leftover-index
Feature/fix leftover index
2024-11-04 16:50:05 +01:00
Kenneth Skovhede 448b6e0e03 Fixed typo 2024-11-01 16:54:39 +01:00
Kenneth Skovhede a7e57c2ffb Added DuplicateBlock to the list of items purged. 2024-11-01 16:06:41 +01:00
Kenneth Skovhede 1809ffac09 Added a fix from #4982 that deletes additional information from the database when removing broken files. 2024-11-01 16:06:20 +01:00
Kenneth Skovhede dedee8fd35 Hand-picked the recreate fixes from #4982.
This fixes that the recreate will move unused blocks into the `DeletedBlock` table after the database has been recreated.
Due to this, the disruption test can now be verified.
2024-11-01 15:56:14 +01:00
Kenneth Skovhede 1dc2acb2bf Ensure repair also works with the index 2024-11-01 15:02:48 +01:00
Kenneth Skovhede db517a300b Implemented a fix for compaction issues that could cause inconsistent views of the remote storage. 2024-11-01 14:50:26 +01:00
Kenneth Skovhede 30a3410180 Changed strategy to check if block is known in a previous blockset.
If this fileset is the first containing the block in a blockset, we add it to the index file.
2024-10-31 15:30:25 +01:00
Kenneth Skovhede e9c256c7f2 Merge remote-tracking branch 'origin/master' into feature/fix-leftover-index 2024-10-29 16:40:06 +01:00
Kenneth Skovhede 300752652f Added detection of the SQLite journal file for the active database.
This fixes #5603
2024-10-28 09:00:40 +01:00
Kenneth Skovhede be0e4d2ed1 Fixed an incorrect string quote 2024-10-05 17:36:42 +02:00
Kenneth Skovhede ff6757877f Fixing the inconsistency check on repair.
Update the SQL to also group on size (just in case).
Renamed the normal read method to signal that it is unverified.
Made the verified reader use enumeration state machine to avoid materializing the list.
`var`-ified some of the test code.
2024-09-09 15:27:37 +02:00
Kenneth Skovhede 17d9b89fbb Merge branch 'master' into fix-repair-index-consistency
# Conflicts:
#	Duplicati/Library/Main/Operation/RecreateDatabaseHandler.cs
2024-09-09 11:49:34 +02:00
Suguru Hirahara ae57dee0f0 Remove redundant space characters from *.cs and *.html 2024-07-26 00:10:31 -04:00
Kenneth SkovhedeandGitHub bfc3d61bc2 Merge branch 'master' into fix-leftover-index 2024-07-19 13:53:06 +02:00
Kenneth Skovhede 3f29331dc6 Fixed a case where listing would fail with empty filters 2024-06-21 07:52:07 +02:00
Jojo-1000 e316944a50 Fix compare on linux showing 0 bytes when exclude filters are used.
Closes #5229
2024-06-10 16:00:39 +02:00
Kenneth SkovhedeandGitHub fdc891aa36 Merge pull request #5208 from Jojo-1000/fix-sql-quotes
Fix SQL string quotes
2024-06-04 20:58:20 +02:00