Commit Graph
21 Commits
Author SHA1 Message Date
Kenneth SkovhedeandGitHub d99f74ed94 Rewrite the backend manager. (#5896)
* Rewrite the backend manager.

This creates a new more logic backend manager that handles all backend operations.

For each top-level operation, there is now a single backend manager instance that is passed to sub-commands.

The internals of the backend manager are now rewritten to use async logic instead of threading.

The design uses a single task runner that dispatches operations and honors concurrent settings and limits from one place.

The logic for each operation has been moved into a separate files/classes to make it easier to understand each operation.

This fixes #5804

* Fixed a few issues with not awaiting tasks

* Fixed not creating empty databases

* Fixed an issue with hashes not being recorded on download

* Reworked the download logic to avoid attempting to decrypt the file if it has been modified.

* Review fixes

* Renamed db collector to better reflect the purpose.

* Review fixes
2025-01-28 08:54:50 +01:00
Kenneth SkovhedeandGitHub 9464caf622 Feature/update license 2025 (#5851)
* Fixed some minor whitespace issues

* Updated all copyright to 2025
2025-01-07 09:40:39 +01:00
Kenneth SkovhedeandGitHub ff1b2b7990 Simplified task control (#5753)
* Simplified task control by having all logic in a single class and using async control mechanism
* Fixed the tests
* Made the marking of a backup partial explicitly communicated if the enumeration process is stopped.
* Added support for pausing active transfers
* Introduced transfer token to abort operations.
* Updated the way LiveControl is emitting events.
* Retains the paused state of the server across reboots, this fixes #5760.
* Reworked the way throttle speeds are propagated through LiveControls.
* Added option to pause transfers in the UI.
* Removed throttle settings from the LiveControls class so it is exclusively handled by the ApplicationSettings.
* Removed `thread-priority` as it does not work at all when tasks are used.
* Fixed the stop dialog and API to only support stop and abort.
* Removed unused variables
2024-12-18 08:27:59 +01:00
Carl Johnsen b2288cb804 Removed named CoCoL channels in favor of passing around instances. 2024-12-09 13:55:22 +01:00
Kenneth Skovhede 734659f3f7 Implemented a fix for 5038 2024-10-31 13:42:07 +01:00
Kenneth Skovhede 40dd3c6816 Updated all license mentions to MIT 2024-02-28 15:45:30 +01:00
Kenneth Hsu 0cdc5b9159 Ensure that dindex contains correct dblock file after renames.
When support for parallel uploads was implemented, we overlooked a case
where a failed put of a dblock file is retried with a different
filename.  In this case, an instance of TemporaryIndexVolume was created
in the DataBlockProcessor or SpillCollectorProcess prior to the
attempted put.  The put would fail, and then retried with a different
filename.  While the Remotevolume table in the database would reflect
the new filename, the TemporaryIndexVolume still contained a reference
to the old dblock filename.  This would cause issues in direct restores,
etc., when the referenced dblock file could not be found.

Now, the TemporaryIndexVolume is only created after the dblock put has
completed.

This addresses issue #3932.
2019-10-05 10:46:39 -07:00
Kenneth Hsu 614ccaf79d Convert duplicated static methods into instance method. 2019-10-04 21:37:33 -07:00
Sean Templeton 8bd4b825c8 Fix several Codacy issues 2019-03-13 21:51:05 -05:00
Sean Templeton 05c3f4808c Move encrypting, hashing, and creating index volumes out of BackendUploader
Encrypting, hashing, and creating index volumes are now done
in the DataBlockProcessor and the SpillCollectorProcess. This allows
uploads to always be transferring data and not have to stop to create
a file, encrypt data, etc.
2019-03-03 21:41:09 -06:00
Kenneth Hsu b5d9451b33 Remove redundant casts. 2019-01-21 19:30:52 -08:00
Kenneth Skovhede c50721e0c4 Added input validation to the IndexVolumeCreator to make sure it does not create invalid temporary lists 2018-05-09 17:37:30 +02:00
Kenneth Skovhede 3081447271 Added an on-the-fly index builder to avoid querying the database for each volume 2018-04-30 14:04:30 +02:00
Kenneth Skovhede 860924bb62 Updated main functions to new CoCoL library 2017-09-20 21:03:24 +02:00
Kenneth Skovhede a3f2b39d23 Implemented handling of pause/stop/abort in the concurrent code.
Implemented the dry-run feature for backups.
2016-02-29 21:52:11 +01:00
Kenneth Skovhede 537fe20838 Added a lot of comments.
Fixed a few issues that caused parallel uploads.
Fixed the use of the encryption module so the encryption module is not re-used for multiple jobs.
2016-02-25 19:15:51 +01:00
Kenneth Skovhede 20f7b42945 Fixed channels to use type-safe initializations.
Updated logging channel to use a wrapper for later early-filter optimizations.
Various bug fixes and optimizations.
2016-02-24 23:48:42 +01:00
Kenneth Skovhede 3ff2f1eb98 The concurrent processing now seems to work, at least in the base case.
Rewrote the handling of index volumes to create them on-demand instead of in-advance.
This makes it much simpler to handle upload failures.
2016-02-22 21:27:12 +01:00
Kenneth Skovhede ec877696b0 More work on concurrent processing 2016-02-18 16:46:53 +01:00
Kenneth Skovhede 0287c10f91 Wrote the spil collector 2016-02-15 09:01:05 +01:00
Kenneth Skovhede 01c3b3f5c0 More work on a concurrent implementation of the backup process 2016-02-09 09:17:31 +01:00