This PR adds the ability to subscribe to messages over a websocket.
To prevent polling data, the server can now push messages through the websocket, so the client can instantly update when new data is available.
The change is backwards compatible, still serving the status updates over the socket. If the client is providing the authentication token, it is auto-subscried to the legacy status message.
If the client is using the new authentication message, it needs to subscribe to get the status updates (and any other services it needs).
The event system has been extended to support new, and more accurate, events. This is the first step towards removing the general status update and the long-poll mechanism.
This also re-introduces the blocking marker, so the client can know if the operation is halted due to too many pending transfers.
Some endpoints have been moved to service implementations, to allow serving the exact same data from both endpoints and websocket.
This PR also updates the way the progress is handled, so that all transfers are returned to the client, and the transfer speeds for each transfer is calculated based on a small sample buffer, so the values are more accurate even if the program is paused during transfers.
This ensures that files cannot appear on the remote destination, despite terminating the host program.
This fixes#4485
There is still work needed to handle the repair and compact operations so they use transactions and can commit changes during transfer progress.
* Rewrite the backend manager.
This creates a new more logic backend manager that handles all backend operations.
For each top-level operation, there is now a single backend manager instance that is passed to sub-commands.
The internals of the backend manager are now rewritten to use async logic instead of threading.
The design uses a single task runner that dispatches operations and honors concurrent settings and limits from one place.
The logic for each operation has been moved into a separate files/classes to make it easier to understand each operation.
This fixes#5804
* Fixed a few issues with not awaiting tasks
* Fixed not creating empty databases
* Fixed an issue with hashes not being recorded on download
* Reworked the download logic to avoid attempting to decrypt the file if it has been modified.
* Review fixes
* Renamed db collector to better reflect the purpose.
* Review fixes
* Added test to verify that compression modules will mask duplicate entries, if present
* Because there are likely many, otherwise valid, dindex zip archives with duplicated blocklists out there, the log message is now verbose.
To make sure we no longer create archives with duplicated entries, the `--unittest-mode` option makes this a warning that is checked in most tests.
* Added guard in `IndexVolumeCreator` to prevent creating zip files with duplicated block lists.
* Added debug check for duplicates
* Fixed the duplicate test
* Simplified task control by having all logic in a single class and using async control mechanism
* Fixed the tests
* Made the marking of a backup partial explicitly communicated if the enumeration process is stopped.
* Added support for pausing active transfers
* Introduced transfer token to abort operations.
* Updated the way LiveControl is emitting events.
* Retains the paused state of the server across reboots, this fixes#5760.
* Reworked the way throttle speeds are propagated through LiveControls.
* Added option to pause transfers in the UI.
* Removed throttle settings from the LiveControls class so it is exclusively handled by the ApplicationSettings.
* Removed `thread-priority` as it does not work at all when tasks are used.
* Fixed the stop dialog and API to only support stop and abort.
* Removed unused variables
In code the throttling generally works backwards than what one would expect.
The streams that are throttled are the streams that operate on files from the local machine.
This fixes the download throttling not working and the upload throttling applying to
both uploads and downloads.
When doing a Put operation the stream on the local machine is reading from the
local filesystem and hence the upload throttle bytes per second needs to be
applied to the reading of the stream, not the writing.
Similarly when doing a Get operation the download throttle bytes per second needs
to be applied to the writing of the stream.
Fixes#4115
According to the current implementation, the progress bar does not advance
when files are skipped by FilePreFilterProcess. This causes users'
confusion since the web frontend will get stuck at a certain stage for a
long time. This commit addresses this issue.
Changed AddBackendEvent() in ResultClasses to allow the option to not update
the backend progress. This allows the BackendUploader to log the act of starting
an upload without changing the backend progress. The backend progress is now
handled by a new class, FileProgressThrottler, which will determine the transfer
rate of all uploads currently in progress.
This class also handles throttling the upload streams. The algorithm will try
to decrease or increase the streams throttle rates as appropriate by changing
the value based on the over/under amount or based on a percentage to prevent
throttling one stream directly to zero.
Encrypting, hashing, and creating index volumes are now done
in the DataBlockProcessor and the SpillCollectorProcess. This allows
uploads to always be transferring data and not have to stop to create
a file, encrypt data, etc.
When the code following the await can be executed on any thread, it's
recommended to use ConfigureAwait(false) to avoid unnecessary context
switching and potential deadlocks.
This simplifies things and speeds up database query queuing, at the expense of not guaranteeing that the same thread will access the database, but just guaranteeing single threaded database access.
Carl Johnsen