Kenneth Skovhede
1f7415cc8e
Simplified backend manager handler code
2025-03-04 15:42:31 +01:00
Carl-Johannes Johnsen and GitHub
476774640a
Merge pull request #5991 from carljohnsen/performance/restoreopt2
2025-02-28 13:03:25 +01:00
Carl Johnsen
6e08affd1a
Rather than having the tempfile database create new ID's for each entry, reuse the File.ID so the connections that do not have access to the temp table can access the same data.
2025-02-28 10:27:02 +01:00
Carl Johnsen
dbe413d889
Fixed the restore BlockManager block counting logic
2025-02-28 10:26:07 +01:00
Carl Johnsen
97df02c57c
Fixed missing docstrings
2025-02-27 14:48:08 +01:00
Carl Johnsen
b9eaa224b8
Renamed variables in VolumeDecompressor
2025-02-27 14:46:04 +01:00
Carl Johnsen
6ef151f4af
Cleaned up FileProcessor
2025-02-27 14:45:05 +01:00
Carl Johnsen
a539af58ca
Removed unused code from FileLister
2025-02-27 14:44:54 +01:00
Carl Johnsen
2ebb021174
Cleaned up BlockManager
2025-02-27 14:40:29 +01:00
Carl Johnsen
2fae2d510e
Removed old block counters from BlockManager
2025-02-27 14:16:06 +01:00
Carl Johnsen
a0eae4131b
Cleaned up LocalRestoreDatabase
2025-02-27 14:14:07 +01:00
Carl Johnsen
260786a91a
Merge branch 'feature/restore-buffer-size-option' into performance/restoreopt2
2025-02-27 14:03:01 +01:00
Carl Johnsen
038e78ad06
The block cache is now always present, as it keeps track of blocks. In case of a cache size 0, every Set() will trigger an evict.
2025-02-27 11:17:11 +01:00
Carl Johnsen
bdfdc7e92a
MemoryCache evict now correctly handles the ArrayPool
2025-02-27 10:54:47 +01:00
Kenneth Skovhede
11a27042bc
Merge remote-tracking branch 'origin/master' into feature/alternate-source-provider
2025-02-26 19:25:46 +01:00
Carl Johnsen
7109155127
Additional block count reporting of the block manager
2025-02-26 16:11:13 +01:00
Carl Johnsen
0e6e1bcbd8
Currently, the block cache cannot compact while using the arraypool, as this doesn't properly return the buffers to the pool
2025-02-26 16:10:51 +01:00
Carl Johnsen
6873162002
Additional internal profiling resolution
2025-02-26 16:09:23 +01:00
Carl Johnsen
692954ee97
Made the local restore database use a connection pool for getting block data
2025-02-26 16:08:55 +01:00
Kenneth Skovhede
f77e529eb6
Merge commit '2339aeff28adba15442d0be60c56568ae642ad0b' into feature/make-backends-async
2025-02-26 15:23:48 +01:00
Carl Johnsen
a49c66bcb5
Added caching of localrestoredatabase data
2025-02-25 17:42:30 +01:00
Carl Johnsen
6d7faf97bb
Whitespace
2025-02-25 17:41:57 +01:00
Carl Johnsen
42ec5b633c
Restore block cache max option didn't match the description
2025-02-25 17:41:35 +01:00
Carl Johnsen
bf3a26ffcd
Added more internal profiling timers to the FileProcessor to get more high-resolution profiling
2025-02-25 16:49:46 +01:00
Carl Johnsen
e773907eed
Changed from allocating a new byte[] buffer in the decompressor to use the Buffers.ArrayPool
2025-02-25 16:45:55 +01:00
Carl Johnsen
d7b8d13bc0
VolumeDecryptor and VolumeManager were unnecessarily sending around and keeping track of disposed encrypted volumes
2025-02-25 16:39:28 +01:00
Carl Johnsen
5c7a933722
FileLister did not report its internal profling properly
2025-02-25 16:37:49 +01:00
Carl Johnsen
b62a1f7600
Added missing comments in filelister
2025-02-25 16:36:30 +01:00
Carl Johnsen
b86794791b
Reallocating byte[] buffer takes up too long time if done for each file
2025-02-25 16:35:46 +01:00
Carl Johnsen
c5ebf4fa71
Added the restore channel depth option
2025-02-25 16:19:04 +01:00
Carl Johnsen
123c12e2c3
Whitespace
2025-02-25 16:18:13 +01:00
Kenneth Skovhede
1428d8cc6f
Preserve order of input sources
2025-02-24 12:53:48 +01:00
Kenneth Skovhede
26bbe6a577
Added setup step for source providers.
...
This enables a fail-early approach, where the remote source loads the initial results.
By storing the `IAsyncEnumerable` it is possible to not perform any additional remote calls, even with the initial probing call.
This makes it possible to detect the absence of sources, so the backup can be aborted if sources are missing.
2025-02-24 12:50:58 +01:00
Kenneth Skovhede
9dfe977d3f
Changed logic so the source providers are now logically mounted into the source directory structure.
...
This fixes the restore issues as all sources will now appear as if they were originally part of the source file system.
The backend implementation has now been simplified so the backends only return names of entries with no path information. Most backends already do this, but the S3 backends did not, so this was fixed as well.
This also clears up the interfaces, so the backend does not know about the `ISourceProviderEntry`. All mapping is done by the `BackendSourceProvider`, making it simpler to support other backends as sources.
2025-02-21 16:06:55 +01:00
Carl-Johannes Johnsen and GitHub
5aa06f3cd4
Merge pull request #5983 from carljohnsen/performance/restore-parallel-download
...
Performance/restore parallel download
2025-02-21 14:29:54 +01:00
Kenneth Skovhede
1716d09b51
Fixed source providers for S3, CIFS, and SSH.
...
Made the SMBShareConnection slightly more robust.
Introduced v2 List API to S3 backend, currently it retains v1 as the default.
Changed S3 backends to not recurse, potentially making smaller listings. This option can be disabled if it breaks things.
Added support for relative paths in the SSH backend.
2025-02-21 13:20:52 +01:00
Kenneth Skovhede
5a4fcaa6d7
Added tool to work with remote sources
2025-02-21 11:51:45 +01:00
Carl Johnsen
98e572714d
Added a comment about downloaders in the teardown of BackendManager.Handler
2025-02-21 10:08:21 +01:00
Carl Johnsen
7612b5cce3
Ensured proper synchronization of operations in BackendManager.Handler
2025-02-21 10:07:41 +01:00
Carl Johnsen
c164181ec2
BackendManager.GetOperation's were unintentionally still synchronized.
2025-02-21 10:07:09 +01:00
Carl Johnsen
beefbf2a46
Removed some unused variables from the BackendManager
2025-02-19 16:12:58 +01:00
Carl Johnsen
9919d86e9b
Updated VolumeDownloader to use the new BackendManager.GetDirectAsync() method
2025-02-19 16:12:42 +01:00
Carl Johnsen
374741fc6d
Added BackendManager.GetDirectAsync(), which downloads a file without decrypting it
2025-02-19 16:11:55 +01:00
Carl Johnsen
2c06820281
Made the BackendManager.GetOperations' decryption methods static and public so the restores VolumeDecryptor can use the same code.
2025-02-19 16:09:32 +01:00
Carl Johnsen
e2e930b2d7
Reworked BackendManager.Handler to handle multiple downloads in parallel
2025-02-19 15:55:59 +01:00
Carl Johnsen
864ce59c4a
Reworded pending uploads to active to better reflect that the uploads are in progress
2025-02-19 15:55:13 +01:00
Kenneth Skovhede
cf5b4bb24d
Added support for Wmic based Windows VSS snapshots.
...
Refactored code a bit to be easier to understand.
2025-02-19 14:00:14 +01:00
Kenneth Skovhede
73d0ad905f
Fixed USN support
2025-02-18 22:12:37 +01:00
Carl Johnsen
a0d5096ebc
Reworked the "Volume loop" of the restore network to handle Backends themselves, rather than using the BackendManager.
...
This allows for tighter control over which process performs which action, allowing them more easily to run in parallel.
2025-02-18 09:25:20 +01:00
Carl Johnsen
17e2c0b9cf
Added ConfigureAwait(false)
2025-02-18 09:20:28 +01:00
Kenneth Skovhede