Commit Graph
4937 Commits
Author SHA1 Message Date
Carl Johnsen 52018db0bb Added handling and warning throwing for when the folder for restored files aren't created properly 2025-01-03 20:48:58 +01:00
Kenneth SkovhedeandGitHub 2cbf62d02b Update Storj options and docs (#5838)
This introduces a few constants in favor of multiple strings with the same value.
The supported arguments in the module have been augmented to better report possible values and default values.
The options parser has been updated to work better with CLI where the options are not default filled.
2025-01-03 17:21:00 +01:00
Kenneth SkovhedeandGitHub 228a472b07 Changed Sia redundancy to decimal (#5837) 2025-01-03 17:20:37 +01:00
Kenneth SkovhedeandGitHub d0924c3e01 Auto-select version for OpenStack (#5836)
This adds a check to see if the give url contains the standard `/v3` suffix, and if so selects the v3 protocol.
2025-01-03 17:20:19 +01:00
Kenneth SkovhedeandGitHub c70b593fed Corrected documentation for OpenStack (#5835)
The region was not described correctly.
2025-01-03 17:19:57 +01:00
Kenneth SkovhedeandGitHub 78da4527be Removed unused region from Aliyun (#5833) 2025-01-03 14:08:29 +01:00
Carl Johnsen 198e4d8e6e Refactor SQL query formatting 2025-01-03 11:06:49 +01:00
Carl Johnsen 661a73ff2a Merge remote-tracking branch 'upstream/master' into SlowBigRestore 2025-01-03 10:11:50 +01:00
Kenneth SkovhedeandGitHub 2fac31bd60 Clean Azure Blob options (#5829)
This removes Azure Blob options that has been deprecated for years.
The option `--azure-blob-container-name` was not used, so it was removed as well.
2025-01-02 18:51:00 +01:00
Kenneth SkovhedeandGitHub 6fdb01b52a This commit ensures the client connects on first use, instead of relying on the library to connect. (#5828) 2025-01-01 14:06:35 +01:00
Kenneth SkovhedeandGitHub 31503ea595 Added option to log FTP diagnostics data to regular log (#5821) 2024-12-31 16:47:28 +01:00
Kenneth SkovhedeandGitHub ba645ddb5f Clean up options for file backend (#5820)
This removes unsupported arguments from non-windows clients, so the reported options fit what is supported.
2024-12-31 16:37:07 +01:00
Kenneth SkovhedeandGitHub bc6114323f Workaround for multi-backend. (#5815)
* Workaround for multi-backend.

There is currently an issue with creating the BackendManager which has a backend instance. This instance is not used by the uploader, so they will create new instances.
The BackendManager instance can then time out, but is used later.

This commit will allow the uploader to "borrow" the instance from the backend manager, so only a single instance is needed.

This is not a clean fix, but doing so requires a full rewrite of the backend manager and uploader, so the controller creates a single instance of the manager, which is then used for all operations.

This would allow the backend manager to keep track of instances across all requests.

* Improved reuse after shutdown
2024-12-28 21:25:49 +01:00
Kenneth SkovhedeandGitHub 22bdcecfd4 Fixed a minor race where the thread could start before the worker is initialized. (#5817) 2024-12-28 19:58:20 +01:00
Kenneth SkovhedeandGitHub 8f853cfa7f Google drive and GCS do not work with timeout (#5812) 2024-12-26 10:42:58 +01:00
Kenneth SkovhedeandGitHub 394b876985 Enable noop daemon for FTP (#5813)
This fixes an issue with timeouts on large file.
2024-12-26 10:41:06 +01:00
Carl Johnsen 6afe959c56 Changed the default number of file processors to match the other concurrent counts 2024-12-23 06:50:24 +01:00
Carl Johnsen ec55580f5c Updated the call signature of FileBlockProcessor.Run() to match the new format 2024-12-23 06:49:04 +01:00
Carl Johnsen a566d1a49f Merge remote-tracking branch 'upstream/master' into fileprocessors 2024-12-23 06:45:43 +01:00
Kenneth SkovhedeandGitHub 161b5708d4 Added missing timeouts to uploads. (#5803)
* Added missing timeouts to uploads.

Since much of the BackendManager is replicated into the BackendUploader, this code needs to be updated as well.

* Fixed namespace issue
2024-12-20 21:03:48 +01:00
Kenneth SkovhedeandGitHub 447c2b3238 Feature/improve ci testing (#5781)
* Added options to insert delays in backend tester

* Added wait delays to Azure and GoogleDrive to see it the CI is more stable with small delays

* Added more debug output to the backend tester

* Fixed dropbox reporting OAuth quota errors, when it is likely Dropbox API quota errors

* Implemented exception detection on AzureBlob backend

* Also showing sleep times in output

* Fixed a bug with the dropbox quota response

* Fixed error message to report correct number of files

* Fixed suppressed warning
2024-12-20 16:10:08 +01:00
Kenneth SkovhedeandGitHub 68627c06bf Feature/alternate windows eventlog (#5784)
* Improved Windows Eventlog support.

Added support for logging to a different log with a Log:Source format.
Backwards compatible by treating "Source" as "Application:Source", where "Source" is default "Duplicati".

* Changed default log-level for eventlog to `Warning`.
This fixes #5770

* Fixed issue where checking for event log would crash the process.
Changed default log to `Duplicati` (was `Application`).
New default source names for service, agent and server.

* Reverted windows log-level back to information, now that there is a separate log for it

* Using "Duplicati 2" as the log name because "Duplicati" may be registered as a source name.

* Attempt to stop service before uninstalling

* Added additional diagnostics messages

* Fixed bug with starting the agent as a service

* Default to not use WindowsService for Agent as it requires permissions to create the event log

* Reduced log message levels for ongoing communication in agent

* Prevent double-logging from agent into Windows EventLog

* Made the agent log under a separate log source name
2024-12-20 16:09:42 +01:00
Kenneth SkovhedeandGitHub 0023dfd19f Removed the use of WebClient in MicrosoftGraphBackend in favor of HttpClient. (#5799) 2024-12-19 22:40:18 +01:00
Kenneth SkovhedeandGitHub 8593887184 Rename OneDrive v2 (#5795)
* Change displayname of OneDrive v2

Since OneDrive v1 is removed, there is no need to name the current backend `v2`.

* Renamed everything but the protocol key
2024-12-19 22:38:35 +01:00
Kenneth SkovhedeandGitHub 86f3f6b178 Fixed issues with timeout in msgraph (#5790)
* Fixed issues with timeout in msgraph

Fixed a place where the cancellation token was not applied.

Fixed a problem with msgraph not sending correct DNS information.

Fixed a problem with msgraph downloading the entire file before streaming it, bypassing throttle and tripping the timeout.

Added `ITimeoutExemptBackend` to selectively disable timeouts for backends that do not support it.

Updated the backendmanager to support `ITimeoutExemptBackend`.

Updated the backendmanager to abort as soon as a timeout happens.

* Updated BackendTester to use the `TimeoutObservingStream` and support the `--read-write-timeout` setting.

Updated backend tester code to use `var`

* Fixed a bug reading the `read-write-timeout`

* Updated to using constants with names instead of `-1`.
2024-12-19 09:31:06 +01:00
marceloduplicatiandGitHub e8c352e5db Merge pull request #5782 from marceloduplicati/feature/ftp-folder-escaping-fix
Fix to re-introduce Unscaping directory names for FTP Client
2024-12-18 09:57:21 -03:00
Kenneth SkovhedeandGitHub ff1b2b7990 Simplified task control (#5753)
* Simplified task control by having all logic in a single class and using async control mechanism
* Fixed the tests
* Made the marking of a backup partial explicitly communicated if the enumeration process is stopped.
* Added support for pausing active transfers
* Introduced transfer token to abort operations.
* Updated the way LiveControl is emitting events.
* Retains the paused state of the server across reboots, this fixes #5760.
* Reworked the way throttle speeds are propagated through LiveControls.
* Added option to pause transfers in the UI.
* Removed throttle settings from the LiveControls class so it is exclusively handled by the ApplicationSettings.
* Removed `thread-priority` as it does not work at all when tasks are used.
* Fixed the stop dialog and API to only support stop and abort.
* Removed unused variables
2024-12-18 08:27:59 +01:00
Marcelo C. c8c79d09eb Fix to re-introduce Unscaping directory names for FTP Client
The need for this was detected by forum post: https://forum.duplicati.com/t/ftp-without-ssl-fails-with-2-1-0-2-beta-2024-11-29/19691/12
2024-12-17 10:11:45 -03:00
Kenneth Skovhede 20a6973ffd Updated documentation for --synchronous-upload.
This fixes #4207
2024-12-17 12:13:34 +01:00
Kenneth Skovhede b610e412d7 Updated preload probing logic.
This updates the logic to not emit any console messags unless the environment variable `DUPLICATI_PRELOAD_SETTINGS_DEBUG=1` is set.
This changes the paths used on MacOS to not use the `/usr/share/Duplicati` folder which is write procted, but rather the `/usr/share/local/Duplicati`.

Added the `/usr/share/local/Duplicati` path for Linux as well.

Added detection for portable mode and only using relative/specific paths for preload in portable mode.

Added support for also probing `%LOCALAPPDATA% folder, which is where other settings are loaded, to allow "all settings in one place". Note that it is possible to use the preload to choose a different folder for anything but the `preload.json` file.

Added an option for specifying `DUPLICATI_PRELOAD_SETTINGS=!<path to file>`, where the leading `!` signifies that **only** that file is loaded and all other locations are ignored.
2024-12-13 11:07:40 +01:00
Kenneth Skovhede c9665da91b Reworked the tokens to only support emitting a single token per server startup 2024-12-12 11:05:09 +01:00
Kenneth Skovhede 7d4f9ffab3 Implemented a forever-token feature 2024-12-12 08:59:59 +01:00
Kenneth SkovhedeandGitHub 4eef98857f Merge pull request #5754 from duplicati/feature/transfer-readwrite-timeout
Implemented read-write timeout.
2024-12-11 10:59:51 +01:00
Kenneth Skovhede 4dfc4f586e Adding e2 url format to description.
This fixes #5755
2024-12-11 08:38:06 +01:00
Kenneth Skovhede 5363bf0d96 Implemented read-write timeout.
This fixes #5725
2024-12-10 22:12:20 +01:00
Carl Johnsen c13482cf4c Removed unnecessary CoCoL channel scope 2024-12-10 10:28:44 +01:00
Carl Johnsen 5bc12e3d46 Changed the reporting of broken local and remote files during restore to only report the first 10 filenames. 2024-12-10 06:57:56 +01:00
Carl Johnsen eed626298d Changed the reporting of broken files to be a filebackedlist, to ensure that there's enough memory in case of a massive failure. 2024-12-10 06:44:55 +01:00
Carl Johnsen ce9a85c3ff Reintroduced comment and version fix, which shouldn't have been removed. 2024-12-10 06:43:51 +01:00
Carl Johnsen ee8a82c230 Moved logging outside of the lock on blockcount in restore block manager 2024-12-10 06:14:55 +01:00
Carl Johnsen caeb119a5b Removed incorrect hints about default parameters 2024-12-10 06:08:02 +01:00
Carl Johnsen f3afd4c682 Reintroduced the restore block cache eviction logic 2024-12-10 06:03:08 +01:00
Carl Johnsen 95887b5841 Set the default cache eviction ratio to be 4gb.
Also set the default value to be parsed from a string.
2024-12-10 06:02:51 +01:00
Carl Johnsen 405e73b2d2 Changed the 'restore-cache-evict' option to be an integer representing percentage instead of a float between 0 and 1 2024-12-10 05:38:57 +01:00
Carl Johnsen f491fa93fc Changed restore scaling parameters to use the default value to produce the default string, rather than a string literal. 2024-12-10 05:32:40 +01:00
Carl Johnsen 9440565706 Changed from hardcoded restore scaling constants to scale with the number of processors on the target machine. 2024-12-10 05:17:59 +01:00
Carl Johnsen 1a2f58b6c8 Merge branch 'master' into performance/restoreopt 2024-12-10 04:14:07 +01:00
Carl Johnsen 7f37c49292 Changed the VolumeManager to use a single input channel of type object, and then switching on the message received, rather than ReadFromAny().
Because it seems like the amount of readers in queue for one of the channels became much higher (70) than the actual number of readers (1).
2024-12-10 04:12:33 +01:00
Carl Johnsen 2d6e1d7ff7 Changed the new restore version to use explicit CoCoL channels 2024-12-10 04:10:34 +01:00
Carl Johnsen 91ccc3e3c1 Merge branch 'master' into bugfix/explicit-cocol-channels 2024-12-09 19:07:33 +01:00