This adds the options `--log-http-requests` and `--log-socket-data` which can be activated to capture diagnostics about HTTP requests and socket operations.
This PR fixes the issues with not correctly loading and disposing modules.
Prior to this PR, there would be only a single instance of each module loaded, which would be configured and later disposed, meaning that the same instance would be used after being disposed, and disposed multiple times.
This mostly worked, but it could carry configuration details over between unrelated runs.
This fixes#6314
This PR changes `--replace-faulty-index-files` to `--dont-replace-faulty-index-files`, and swaps the default value.
This PR also adds a check that triggers a warning if any boolean values are introduced with a default value that is `true`.
This PR adds the ability to subscribe to messages over a websocket.
To prevent polling data, the server can now push messages through the websocket, so the client can instantly update when new data is available.
The change is backwards compatible, still serving the status updates over the socket. If the client is providing the authentication token, it is auto-subscried to the legacy status message.
If the client is using the new authentication message, it needs to subscribe to get the status updates (and any other services it needs).
The event system has been extended to support new, and more accurate, events. This is the first step towards removing the general status update and the long-poll mechanism.
This also re-introduces the blocking marker, so the client can know if the operation is halted due to too many pending transfers.
Some endpoints have been moved to service implementations, to allow serving the exact same data from both endpoints and websocket.
This PR also updates the way the progress is handled, so that all transfers are returned to the client, and the transfer speeds for each transfer is calculated based on a small sample buffer, so the values are more accurate even if the program is paused during transfers.
This PR adds the option to suppress warnings by their log id. Warning IDs that are supplied to `--suppress-warnings` will be converted to information messages before being logged.
If a specific warning is disabled, such as `CompressionReadErrorFallback`, this will then no longer count as a warning for the job, and the log file will see the warning as an information message.
This PR also adds two simpler filter options that makes it possible to filter log messages by supplying the log IDs. This can already be achieved with log filters, but the ID filter is a bit simpler to apply, as you only need to know the ID.
Finally, this PR also adds common logging for errors in the categories:
- Permission denied, id = `PermissionDenied`
- File locked, id = `FileLocked`
- Path not found, id = `PathNotFound`
- Path too long, id = `PathTooLong`
These new log ids makes it simpler to ignore warnings about locked or inaccesible files.
This change now flushes all pending messages from the backendmanager to the database, even if the main operation crashes (or is stopped).
This ensures that the database and remote storage are more in sync.
Also removed a redundant warning from the backendmanager during shutdown, and simplified a check for active operations.
This fixes#6200
This moves the throttle support to the StreamUtil throttled stream, which allows multiple streams to share a throttle value, such that the throttle setting is applied to the entire communication from Duplicati, and not for each connection.
This also cleans up a bit in the progress reporting, where the backend manager now keeps track of the "active" transfer and only emits progress on that. Once it finishes, a new transfer is promoted a "active".
This adds an extra option to allow setting the SQLite page cache size as a regular option.
Prior to this commit it was only possible to set the SQLite page cache size via environment variables.
The option to use environment variables is preserved, and as options from the environment variable are applied after the new setting, environment variables take precedence.
The default value for the new option is to use 1% of the system memory for the page cache. For the restore process, a connection per worker may be made which defaults to half the number of cores, so the maximum amount of memory is 1% * half the CPU cores.
This is just the upper limit, and SQLite may choose not to use all of it.
This fixes#6178
The code was using a two-way reference linking the database to the results and the results to the database.
However, this was not really used so it ended up just cluttering the code. With this commit, the results and databases are no longer keeping track of each other, and the only interaction is when the results are written to the database at the end of the operation.
To ensure result data is always written, this code has been moved into the controller, which will now flush the results for all operations. Prior to this, only some operations would write result data to the log.
The controller now also handles logic for cleaning old log data and invoking the vacuum command.
Fixed a bug due to using System.Text.Json where some attributes from NewtonSoft.Json were being applied and not picked up.
This PR adds a unified way of supplying timeouts to each of the backends.
It also adds an opt-in way of adding commonly used options, so parsing and usage can be standardized across multiple backends.
Rewrote all operation handlers to be async.
This moves the async/sync threshold out into the controller.
Fixed all the uses of cancellation token in the handlers to use the provided cancellation token.
Introduced a new ReusableTransaction that can be passed around, avoiding a ref parameter passing of the transaction instance.
If the user selects the blocksize to be the same as the remote volume size, this will cause each block to be in a separate volume and will trigger repeated compact calls.
This commit adds a check that ensures the volume size is large enough to hold at least two blocks.
Also fixed an issue where the compact could create volumes larger than the set limit (at most one block larger).
This fixes the restore issues as all sources will now appear as if they were originally part of the source file system.
The backend implementation has now been simplified so the backends only return names of entries with no path information. Most backends already do this, but the S3 backends did not, so this was fixed as well.
This also clears up the interfaces, so the backend does not know about the `ISourceProviderEntry`. All mapping is done by the `BackendSourceProvider`, making it simpler to support other backends as sources.
Made the SMBShareConnection slightly more robust.
Introduced v2 List API to S3 backend, currently it retains v1 as the default.
Changed S3 backends to not recurse, potentially making smaller listings. This option can be disabled if it breaks things.
Added support for relative paths in the SSH backend.
Added support for `--portable-mode` and `--server-datafolder` for the CLI, ServerUtil, and Agent.
Fixed a few places where file reads were not cached properly.
Moved some responsibilities of UpdaterManager into DataFolderManager.
This now supports storing `machineid.txt` and `installation.txt` inside the path pointed to by `--server-datafolder` or alternatively with `--portable-mode`.
It is possible that some fringe backwards compatibility is lost with this update.
Fixed an issue with AutoUpdater crashing on check/download.
This fixes#5902
* Rewrite the backend manager.
This creates a new more logic backend manager that handles all backend operations.
For each top-level operation, there is now a single backend manager instance that is passed to sub-commands.
The internals of the backend manager are now rewritten to use async logic instead of threading.
The design uses a single task runner that dispatches operations and honors concurrent settings and limits from one place.
The logic for each operation has been moved into a separate files/classes to make it easier to understand each operation.
This fixes#5804
* Fixed a few issues with not awaiting tasks
* Fixed not creating empty databases
* Fixed an issue with hashes not being recorded on download
* Reworked the download logic to avoid attempting to decrypt the file if it has been modified.
* Review fixes
* Renamed db collector to better reflect the purpose.
* Review fixes
* Simplified task control by having all logic in a single class and using async control mechanism
* Fixed the tests
* Made the marking of a backup partial explicitly communicated if the enumeration process is stopped.
* Added support for pausing active transfers
* Introduced transfer token to abort operations.
* Updated the way LiveControl is emitting events.
* Retains the paused state of the server across reboots, this fixes#5760.
* Reworked the way throttle speeds are propagated through LiveControls.
* Added option to pause transfers in the UI.
* Removed throttle settings from the LiveControls class so it is exclusively handled by the ApplicationSettings.
* Removed `thread-priority` as it does not work at all when tasks are used.
* Fixed the stop dialog and API to only support stop and abort.
* Removed unused variables
Kenneth Skovhede