Carl-Johannes Johnsen and GitHub
476774640a
Merge pull request #5991 from carljohnsen/performance/restoreopt2
2025-02-28 13:03:25 +01:00
Carl Johnsen
dbe413d889
Fixed the restore BlockManager block counting logic
2025-02-28 10:26:07 +01:00
Carl Johnsen
97df02c57c
Fixed missing docstrings
2025-02-27 14:48:08 +01:00
Carl Johnsen
b9eaa224b8
Renamed variables in VolumeDecompressor
2025-02-27 14:46:04 +01:00
Carl Johnsen
6ef151f4af
Cleaned up FileProcessor
2025-02-27 14:45:05 +01:00
Carl Johnsen
a539af58ca
Removed unused code from FileLister
2025-02-27 14:44:54 +01:00
Carl Johnsen
2ebb021174
Cleaned up BlockManager
2025-02-27 14:40:29 +01:00
Carl Johnsen
2fae2d510e
Removed old block counters from BlockManager
2025-02-27 14:16:06 +01:00
Carl Johnsen
260786a91a
Merge branch 'feature/restore-buffer-size-option' into performance/restoreopt2
2025-02-27 14:03:01 +01:00
Carl Johnsen
038e78ad06
The block cache is now always present, as it keeps track of blocks. In case of a cache size 0, every Set() will trigger an evict.
2025-02-27 11:17:11 +01:00
Carl Johnsen
bdfdc7e92a
MemoryCache evict now correctly handles the ArrayPool
2025-02-27 10:54:47 +01:00
Kenneth Skovhede
11a27042bc
Merge remote-tracking branch 'origin/master' into feature/alternate-source-provider
2025-02-26 19:25:46 +01:00
Carl Johnsen
7109155127
Additional block count reporting of the block manager
2025-02-26 16:11:13 +01:00
Carl Johnsen
0e6e1bcbd8
Currently, the block cache cannot compact while using the arraypool, as this doesn't properly return the buffers to the pool
2025-02-26 16:10:51 +01:00
Carl Johnsen
6873162002
Additional internal profiling resolution
2025-02-26 16:09:23 +01:00
Kenneth Skovhede
f77e529eb6
Merge commit '2339aeff28adba15442d0be60c56568ae642ad0b' into feature/make-backends-async
2025-02-26 15:23:48 +01:00
Carl Johnsen
a49c66bcb5
Added caching of localrestoredatabase data
2025-02-25 17:42:30 +01:00
Carl Johnsen
bf3a26ffcd
Added more internal profiling timers to the FileProcessor to get more high-resolution profiling
2025-02-25 16:49:46 +01:00
Carl Johnsen
e773907eed
Changed from allocating a new byte[] buffer in the decompressor to use the Buffers.ArrayPool
2025-02-25 16:45:55 +01:00
Carl Johnsen
d7b8d13bc0
VolumeDecryptor and VolumeManager were unnecessarily sending around and keeping track of disposed encrypted volumes
2025-02-25 16:39:28 +01:00
Carl Johnsen
5c7a933722
FileLister did not report its internal profling properly
2025-02-25 16:37:49 +01:00
Carl Johnsen
b62a1f7600
Added missing comments in filelister
2025-02-25 16:36:30 +01:00
Carl Johnsen
b86794791b
Reallocating byte[] buffer takes up too long time if done for each file
2025-02-25 16:35:46 +01:00
Carl Johnsen
c5ebf4fa71
Added the restore channel depth option
2025-02-25 16:19:04 +01:00
Kenneth Skovhede
26bbe6a577
Added setup step for source providers.
...
This enables a fail-early approach, where the remote source loads the initial results.
By storing the `IAsyncEnumerable` it is possible to not perform any additional remote calls, even with the initial probing call.
This makes it possible to detect the absence of sources, so the backup can be aborted if sources are missing.
2025-02-24 12:50:58 +01:00
Kenneth Skovhede
9dfe977d3f
Changed logic so the source providers are now logically mounted into the source directory structure.
...
This fixes the restore issues as all sources will now appear as if they were originally part of the source file system.
The backend implementation has now been simplified so the backends only return names of entries with no path information. Most backends already do this, but the S3 backends did not, so this was fixed as well.
This also clears up the interfaces, so the backend does not know about the `ISourceProviderEntry`. All mapping is done by the `BackendSourceProvider`, making it simpler to support other backends as sources.
2025-02-21 16:06:55 +01:00
Carl-Johannes Johnsen and GitHub
5aa06f3cd4
Merge pull request #5983 from carljohnsen/performance/restore-parallel-download
...
Performance/restore parallel download
2025-02-21 14:29:54 +01:00
Kenneth Skovhede
1716d09b51
Fixed source providers for S3, CIFS, and SSH.
...
Made the SMBShareConnection slightly more robust.
Introduced v2 List API to S3 backend, currently it retains v1 as the default.
Changed S3 backends to not recurse, potentially making smaller listings. This option can be disabled if it breaks things.
Added support for relative paths in the SSH backend.
2025-02-21 13:20:52 +01:00
Kenneth Skovhede
5a4fcaa6d7
Added tool to work with remote sources
2025-02-21 11:51:45 +01:00
Carl Johnsen
9919d86e9b
Updated VolumeDownloader to use the new BackendManager.GetDirectAsync() method
2025-02-19 16:12:42 +01:00
Carl Johnsen
2c06820281
Made the BackendManager.GetOperations' decryption methods static and public so the restores VolumeDecryptor can use the same code.
2025-02-19 16:09:32 +01:00
Kenneth Skovhede
cf5b4bb24d
Added support for Wmic based Windows VSS snapshots.
...
Refactored code a bit to be easier to understand.
2025-02-19 14:00:14 +01:00
Kenneth Skovhede
73d0ad905f
Fixed USN support
2025-02-18 22:12:37 +01:00
Carl Johnsen
a0d5096ebc
Reworked the "Volume loop" of the restore network to handle Backends themselves, rather than using the BackendManager.
...
This allows for tighter control over which process performs which action, allowing them more easily to run in parallel.
2025-02-18 09:25:20 +01:00
Carl Johnsen
17e2c0b9cf
Added ConfigureAwait(false)
2025-02-18 09:20:28 +01:00
Carl Johnsen
ce60b807c8
Added additional profiling logging
2025-02-18 09:18:46 +01:00
Kenneth Skovhede
7031bc57b6
Adding ConfigureAwait(false) to all async enumerations
2025-02-18 09:18:36 +01:00
Carl Johnsen
93756cd74b
Whitespace fixes
2025-02-18 09:17:08 +01:00
Carl Johnsen
6600e418a7
Fixed a bug during profile logging
2025-02-18 09:14:17 +01:00
Kenneth Skovhede
bb5fd80a29
Implemented support for some backends as sources: SSH, CIFS, and S3
2025-02-17 14:01:08 +01:00
Kenneth Skovhede
a621bb3d14
Fixed issue after merge
2025-02-17 08:19:19 +01:00
Kenneth Skovhede
6bbc93b8df
Merge remote-tracking branch 'origin/master' into feature/alternate-source-provider
2025-02-16 16:37:12 +01:00
Kenneth Skovhede and GitHub
e76bbc478f
Merge pull request #5950 from duplicati/feature/fix-zip-fallback
...
Handle automatic fallback when reading zip files
2025-02-10 21:06:39 +01:00
Kenneth Skovhede and GitHub
6693b77d85
Merge pull request #5948 from duplicati/feature/fix-not-storing-updated-timestamps
...
Fix not storing updated timestamps
2025-02-10 21:05:46 +01:00
Carl Johnsen
043b9230b5
VerifyLocalBlocks opened file in incorrect mode, and the target file may not exist.
2025-02-10 13:44:02 +01:00
Carl Johnsen
82019ea28e
Modified VerifyTargetBlocks to not evict block from cache. It is now handled outside, in case of file name retargeting when overwrite is false.
2025-02-10 13:42:27 +01:00
Carl Johnsen
0b3ff33716
Moved retire on exception to final, so regardless of how the FileProcessor exits, it retires the channels.
2025-02-10 12:38:24 +01:00
Carl Johnsen
3b9fc188e1
Retire block request and response on exception in FileProcessor
2025-02-10 12:34:44 +01:00
Carl Johnsen
259322f0d9
CopyOldTargetBlocksToNewTarget was made async
2025-02-10 12:34:16 +01:00
Carl Johnsen
6e2f8156f8
VerifyLocalBlocks should only perform checks when the target differs from the original.
2025-02-10 12:33:45 +01:00
