Commit Graph
4722 Commits
Author SHA1 Message Date
Kenneth SkovhedeandGitHub 22bdcecfd4 Fixed a minor race where the thread could start before the worker is initialized. (#5817) 2024-12-28 19:58:20 +01:00
Kenneth SkovhedeandGitHub 8f853cfa7f Google drive and GCS do not work with timeout (#5812) 2024-12-26 10:42:58 +01:00
Kenneth SkovhedeandGitHub 394b876985 Enable noop daemon for FTP (#5813)
This fixes an issue with timeouts on large file.
2024-12-26 10:41:06 +01:00
Carl Johnsen 6afe959c56 Changed the default number of file processors to match the other concurrent counts 2024-12-23 06:50:24 +01:00
Carl Johnsen ec55580f5c Updated the call signature of FileBlockProcessor.Run() to match the new format 2024-12-23 06:49:04 +01:00
Carl Johnsen a566d1a49f Merge remote-tracking branch 'upstream/master' into fileprocessors 2024-12-23 06:45:43 +01:00
Kenneth SkovhedeandGitHub 161b5708d4 Added missing timeouts to uploads. (#5803)
* Added missing timeouts to uploads.

Since much of the BackendManager is replicated into the BackendUploader, this code needs to be updated as well.

* Fixed namespace issue
2024-12-20 21:03:48 +01:00
Kenneth SkovhedeandGitHub 447c2b3238 Feature/improve ci testing (#5781)
* Added options to insert delays in backend tester

* Added wait delays to Azure and GoogleDrive to see it the CI is more stable with small delays

* Added more debug output to the backend tester

* Fixed dropbox reporting OAuth quota errors, when it is likely Dropbox API quota errors

* Implemented exception detection on AzureBlob backend

* Also showing sleep times in output

* Fixed a bug with the dropbox quota response

* Fixed error message to report correct number of files

* Fixed suppressed warning
2024-12-20 16:10:08 +01:00
Kenneth SkovhedeandGitHub 68627c06bf Feature/alternate windows eventlog (#5784)
* Improved Windows Eventlog support.

Added support for logging to a different log with a Log:Source format.
Backwards compatible by treating "Source" as "Application:Source", where "Source" is default "Duplicati".

* Changed default log-level for eventlog to `Warning`.
This fixes #5770

* Fixed issue where checking for event log would crash the process.
Changed default log to `Duplicati` (was `Application`).
New default source names for service, agent and server.

* Reverted windows log-level back to information, now that there is a separate log for it

* Using "Duplicati 2" as the log name because "Duplicati" may be registered as a source name.

* Attempt to stop service before uninstalling

* Added additional diagnostics messages

* Fixed bug with starting the agent as a service

* Default to not use WindowsService for Agent as it requires permissions to create the event log

* Reduced log message levels for ongoing communication in agent

* Prevent double-logging from agent into Windows EventLog

* Made the agent log under a separate log source name
2024-12-20 16:09:42 +01:00
Kenneth SkovhedeandGitHub 0023dfd19f Removed the use of WebClient in MicrosoftGraphBackend in favor of HttpClient. (#5799) 2024-12-19 22:40:18 +01:00
Kenneth SkovhedeandGitHub 8593887184 Rename OneDrive v2 (#5795)
* Change displayname of OneDrive v2

Since OneDrive v1 is removed, there is no need to name the current backend `v2`.

* Renamed everything but the protocol key
2024-12-19 22:38:35 +01:00
Kenneth SkovhedeandGitHub 86f3f6b178 Fixed issues with timeout in msgraph (#5790)
* Fixed issues with timeout in msgraph

Fixed a place where the cancellation token was not applied.

Fixed a problem with msgraph not sending correct DNS information.

Fixed a problem with msgraph downloading the entire file before streaming it, bypassing throttle and tripping the timeout.

Added `ITimeoutExemptBackend` to selectively disable timeouts for backends that do not support it.

Updated the backendmanager to support `ITimeoutExemptBackend`.

Updated the backendmanager to abort as soon as a timeout happens.

* Updated BackendTester to use the `TimeoutObservingStream` and support the `--read-write-timeout` setting.

Updated backend tester code to use `var`

* Fixed a bug reading the `read-write-timeout`

* Updated to using constants with names instead of `-1`.
2024-12-19 09:31:06 +01:00
marceloduplicatiandGitHub e8c352e5db Merge pull request #5782 from marceloduplicati/feature/ftp-folder-escaping-fix
Fix to re-introduce Unscaping directory names for FTP Client
2024-12-18 09:57:21 -03:00
Kenneth SkovhedeandGitHub ff1b2b7990 Simplified task control (#5753)
* Simplified task control by having all logic in a single class and using async control mechanism
* Fixed the tests
* Made the marking of a backup partial explicitly communicated if the enumeration process is stopped.
* Added support for pausing active transfers
* Introduced transfer token to abort operations.
* Updated the way LiveControl is emitting events.
* Retains the paused state of the server across reboots, this fixes #5760.
* Reworked the way throttle speeds are propagated through LiveControls.
* Added option to pause transfers in the UI.
* Removed throttle settings from the LiveControls class so it is exclusively handled by the ApplicationSettings.
* Removed `thread-priority` as it does not work at all when tasks are used.
* Fixed the stop dialog and API to only support stop and abort.
* Removed unused variables
2024-12-18 08:27:59 +01:00
Marcelo C. c8c79d09eb Fix to re-introduce Unscaping directory names for FTP Client
The need for this was detected by forum post: https://forum.duplicati.com/t/ftp-without-ssl-fails-with-2-1-0-2-beta-2024-11-29/19691/12
2024-12-17 10:11:45 -03:00
Kenneth Skovhede 20a6973ffd Updated documentation for --synchronous-upload.
This fixes #4207
2024-12-17 12:13:34 +01:00
Kenneth Skovhede b610e412d7 Updated preload probing logic.
This updates the logic to not emit any console messags unless the environment variable `DUPLICATI_PRELOAD_SETTINGS_DEBUG=1` is set.
This changes the paths used on MacOS to not use the `/usr/share/Duplicati` folder which is write procted, but rather the `/usr/share/local/Duplicati`.

Added the `/usr/share/local/Duplicati` path for Linux as well.

Added detection for portable mode and only using relative/specific paths for preload in portable mode.

Added support for also probing `%LOCALAPPDATA% folder, which is where other settings are loaded, to allow "all settings in one place". Note that it is possible to use the preload to choose a different folder for anything but the `preload.json` file.

Added an option for specifying `DUPLICATI_PRELOAD_SETTINGS=!<path to file>`, where the leading `!` signifies that **only** that file is loaded and all other locations are ignored.
2024-12-13 11:07:40 +01:00
Kenneth Skovhede c9665da91b Reworked the tokens to only support emitting a single token per server startup 2024-12-12 11:05:09 +01:00
Kenneth Skovhede 7d4f9ffab3 Implemented a forever-token feature 2024-12-12 08:59:59 +01:00
Kenneth SkovhedeandGitHub 4eef98857f Merge pull request #5754 from duplicati/feature/transfer-readwrite-timeout
Implemented read-write timeout.
2024-12-11 10:59:51 +01:00
Kenneth Skovhede 4dfc4f586e Adding e2 url format to description.
This fixes #5755
2024-12-11 08:38:06 +01:00
Kenneth Skovhede 5363bf0d96 Implemented read-write timeout.
This fixes #5725
2024-12-10 22:12:20 +01:00
Carl Johnsen c13482cf4c Removed unnecessary CoCoL channel scope 2024-12-10 10:28:44 +01:00
Carl Johnsen 5bc12e3d46 Changed the reporting of broken local and remote files during restore to only report the first 10 filenames. 2024-12-10 06:57:56 +01:00
Carl Johnsen eed626298d Changed the reporting of broken files to be a filebackedlist, to ensure that there's enough memory in case of a massive failure. 2024-12-10 06:44:55 +01:00
Carl Johnsen ce9a85c3ff Reintroduced comment and version fix, which shouldn't have been removed. 2024-12-10 06:43:51 +01:00
Carl Johnsen ee8a82c230 Moved logging outside of the lock on blockcount in restore block manager 2024-12-10 06:14:55 +01:00
Carl Johnsen caeb119a5b Removed incorrect hints about default parameters 2024-12-10 06:08:02 +01:00
Carl Johnsen f3afd4c682 Reintroduced the restore block cache eviction logic 2024-12-10 06:03:08 +01:00
Carl Johnsen 95887b5841 Set the default cache eviction ratio to be 4gb.
Also set the default value to be parsed from a string.
2024-12-10 06:02:51 +01:00
Carl Johnsen 405e73b2d2 Changed the 'restore-cache-evict' option to be an integer representing percentage instead of a float between 0 and 1 2024-12-10 05:38:57 +01:00
Carl Johnsen f491fa93fc Changed restore scaling parameters to use the default value to produce the default string, rather than a string literal. 2024-12-10 05:32:40 +01:00
Carl Johnsen 9440565706 Changed from hardcoded restore scaling constants to scale with the number of processors on the target machine. 2024-12-10 05:17:59 +01:00
Carl Johnsen 1a2f58b6c8 Merge branch 'master' into performance/restoreopt 2024-12-10 04:14:07 +01:00
Carl Johnsen 7f37c49292 Changed the VolumeManager to use a single input channel of type object, and then switching on the message received, rather than ReadFromAny().
Because it seems like the amount of readers in queue for one of the channels became much higher (70) than the actual number of readers (1).
2024-12-10 04:12:33 +01:00
Carl Johnsen 2d6e1d7ff7 Changed the new restore version to use explicit CoCoL channels 2024-12-10 04:10:34 +01:00
Carl Johnsen 91ccc3e3c1 Merge branch 'master' into bugfix/explicit-cocol-channels 2024-12-09 19:07:33 +01:00
Carl Johnsen b2288cb804 Removed named CoCoL channels in favor of passing around instances. 2024-12-09 13:55:22 +01:00
Kenneth Skovhede ac61b2afbe Updated FluentFTP to latest.
This should fix #5715
2024-12-09 10:20:27 +01:00
Carl Johnsen e642ef8bef Merge branch 'master' into performance/restoreopt 2024-12-06 09:22:27 +01:00
Carl Johnsen c8c437f9aa Merge branch 'master' into bugfix/faulty_temp_volumes 2024-12-06 08:29:36 +01:00
Kenneth SkovhedeandGitHub 5e03e3084b Merge pull request #5745 from duplicati/feature/remove-unused-dependency-ssh
Feature/remove unused dependency ssh
2024-12-05 21:11:01 +01:00
Carl Johnsen c372af8e6f Merge branch 'master' into bugfix/faulty_temp_volumes 2024-12-05 16:43:51 +01:00
Carl Johnsen 0c58de82b5 Added a check for whether the file exists, rather than throwing exceptions. 2024-12-05 16:42:09 +01:00
Carl Johnsen 4a523efd24 Sometimes, the temporary empty volumes aren't removed from the database 2024-12-05 16:41:28 +01:00
Carl Johnsen 7b12eb8a67 When deleting a symlink, we should check whether it is a file or a directory, as each have their own function call 2024-12-05 16:39:01 +01:00
Kenneth SkovhedeandGitHub 834aceefa0 Merge pull request #5744 from duplicati/feature/upgrade-ssh-net
Bumped SSH.NET to latest
2024-12-05 15:01:06 +01:00
Kenneth Skovhede e738221bb2 Removed unused dependency 2024-12-05 14:42:53 +01:00
Kenneth SkovhedeandGitHub 642d8908f8 Merge pull request #5740 from marceloduplicati/feature/http-sender-new-parameters
Adding support for self-signed or specific by hash certificates
2024-12-05 14:37:33 +01:00
Kenneth Skovhede 6156024e8f Bumped SSH.NET to latest
This fixes #4674
2024-12-05 14:33:44 +01:00