Commit Graph
4755 Commits
Author SHA1 Message Date
Carl Johnsen 46b05b94ef Limit file name conflict attempts to 100 and log an error if exceeded 2025-01-06 16:01:53 +01:00
Carl Johnsen 6f263ad874 Added a note about potential memory consumption of restores FileProcessor 2025-01-06 16:00:11 +01:00
Carl Johnsen 284b14b72f Merge branch 'master' into bugfix/restore-overwrite 2025-01-06 14:43:03 +01:00
marceloduplicatiandGitHub 99ee234640 Primary functionality implemented for CIFS (#5780)
* Primary functionality implemented for CIFS

This will add CIFS support to Duplicati.

One of the advantages is that the resource does not need to be mounted as the backend connects and authenticates directly to the resource via TCP/or Netbios over TCP.

The authentication supported is domain/username/password. At present integrated Windows authentication is not supported.

This implementation relies on SMBLibrary library (https://github.com/TalAloni/SMBLibrary).

* Fixes to XML documentation

* Implementing single CIFS connection + Fix on PutAsync

Single CIFS connection to avoid re-authentication on every operation.
Fix to PutAsync which only because apparent after file sizes were larger than _smb2Client.MaxWriteSize

* Adding CI tests for CIFS

The CI test for CIFS is self contained, requiring no environment variables. Also updated README.
2025-01-06 14:22:23 +01:00
Kenneth SkovhedeandGitHub 2a757dea0f Quota check disabled fix (#5843)
The QuotaCheck is done in two places, and only one place was checking if the quota check was supposed to be disabled.
This adds the check in the other place.
2025-01-06 14:21:53 +01:00
Carl Johnsen 88846251b1 Restoring a symlink that already exists triggers an overwrite of the file, which should count towards number of files restored. 2025-01-06 08:14:32 +01:00
Carl Johnsen 1bea0f3696 Restored empty files should count towards the restoration file count. 2025-01-05 12:46:08 +01:00
Carl Johnsen 4cd565d592 Added handling of already restored timestamp-retargeted files 2025-01-05 12:09:38 +01:00
Carl Johnsen b759b4564d Added docstrings to new functions 2025-01-05 12:09:04 +01:00
Carl Johnsen eb64320a49 Changed the reporting of restored files to match the legacy restore's reporting. 2025-01-05 11:51:52 +01:00
Carl Johnsen 637bfcefc8 Forgot to check whether the file exists 2025-01-05 11:25:19 +01:00
Carl Johnsen eba477c271 Added handling of the 'overwrite' option in the new restore flow 2025-01-05 10:44:08 +01:00
Carl Johnsen f76bf90e68 Added additional dry run logging 2025-01-05 10:43:23 +01:00
Carl Johnsen 8e92754efd VerifyLocalBlocks now keeps track of internally verified blocks for logging 2025-01-05 10:43:01 +01:00
Carl Johnsen be99f03bac VerifyLocalBlocks wasn't respecting dry run properly 2025-01-05 10:41:33 +01:00
Carl Johnsen a14d29da93 Modified VerifyTargetBlocks to also return the blocks that were verified. 2025-01-05 10:40:14 +01:00
Carl Johnsen a7507fb765 Simplified variable assignment 2025-01-05 10:35:50 +01:00
Carl Johnsen 83d1617984 Fixed the IO calls to also work on Windows 2025-01-03 23:27:35 +01:00
Carl Johnsen 52018db0bb Added handling and warning throwing for when the folder for restored files aren't created properly 2025-01-03 20:48:58 +01:00
Kenneth SkovhedeandGitHub 2cbf62d02b Update Storj options and docs (#5838)
This introduces a few constants in favor of multiple strings with the same value.
The supported arguments in the module have been augmented to better report possible values and default values.
The options parser has been updated to work better with CLI where the options are not default filled.
2025-01-03 17:21:00 +01:00
Kenneth SkovhedeandGitHub 228a472b07 Changed Sia redundancy to decimal (#5837) 2025-01-03 17:20:37 +01:00
Kenneth SkovhedeandGitHub d0924c3e01 Auto-select version for OpenStack (#5836)
This adds a check to see if the give url contains the standard `/v3` suffix, and if so selects the v3 protocol.
2025-01-03 17:20:19 +01:00
Kenneth SkovhedeandGitHub c70b593fed Corrected documentation for OpenStack (#5835)
The region was not described correctly.
2025-01-03 17:19:57 +01:00
Kenneth SkovhedeandGitHub 78da4527be Removed unused region from Aliyun (#5833) 2025-01-03 14:08:29 +01:00
Carl Johnsen 198e4d8e6e Refactor SQL query formatting 2025-01-03 11:06:49 +01:00
Carl Johnsen 661a73ff2a Merge remote-tracking branch 'upstream/master' into SlowBigRestore 2025-01-03 10:11:50 +01:00
Kenneth SkovhedeandGitHub 2fac31bd60 Clean Azure Blob options (#5829)
This removes Azure Blob options that has been deprecated for years.
The option `--azure-blob-container-name` was not used, so it was removed as well.
2025-01-02 18:51:00 +01:00
Kenneth SkovhedeandGitHub 6fdb01b52a This commit ensures the client connects on first use, instead of relying on the library to connect. (#5828) 2025-01-01 14:06:35 +01:00
Kenneth SkovhedeandGitHub 31503ea595 Added option to log FTP diagnostics data to regular log (#5821) 2024-12-31 16:47:28 +01:00
Kenneth SkovhedeandGitHub ba645ddb5f Clean up options for file backend (#5820)
This removes unsupported arguments from non-windows clients, so the reported options fit what is supported.
2024-12-31 16:37:07 +01:00
Kenneth SkovhedeandGitHub bc6114323f Workaround for multi-backend. (#5815)
* Workaround for multi-backend.

There is currently an issue with creating the BackendManager which has a backend instance. This instance is not used by the uploader, so they will create new instances.
The BackendManager instance can then time out, but is used later.

This commit will allow the uploader to "borrow" the instance from the backend manager, so only a single instance is needed.

This is not a clean fix, but doing so requires a full rewrite of the backend manager and uploader, so the controller creates a single instance of the manager, which is then used for all operations.

This would allow the backend manager to keep track of instances across all requests.

* Improved reuse after shutdown
2024-12-28 21:25:49 +01:00
Kenneth SkovhedeandGitHub 22bdcecfd4 Fixed a minor race where the thread could start before the worker is initialized. (#5817) 2024-12-28 19:58:20 +01:00
Kenneth SkovhedeandGitHub 8f853cfa7f Google drive and GCS do not work with timeout (#5812) 2024-12-26 10:42:58 +01:00
Kenneth SkovhedeandGitHub 394b876985 Enable noop daemon for FTP (#5813)
This fixes an issue with timeouts on large file.
2024-12-26 10:41:06 +01:00
Carl Johnsen 6afe959c56 Changed the default number of file processors to match the other concurrent counts 2024-12-23 06:50:24 +01:00
Carl Johnsen ec55580f5c Updated the call signature of FileBlockProcessor.Run() to match the new format 2024-12-23 06:49:04 +01:00
Carl Johnsen a566d1a49f Merge remote-tracking branch 'upstream/master' into fileprocessors 2024-12-23 06:45:43 +01:00
Kenneth SkovhedeandGitHub 161b5708d4 Added missing timeouts to uploads. (#5803)
* Added missing timeouts to uploads.

Since much of the BackendManager is replicated into the BackendUploader, this code needs to be updated as well.

* Fixed namespace issue
2024-12-20 21:03:48 +01:00
Kenneth SkovhedeandGitHub 447c2b3238 Feature/improve ci testing (#5781)
* Added options to insert delays in backend tester

* Added wait delays to Azure and GoogleDrive to see it the CI is more stable with small delays

* Added more debug output to the backend tester

* Fixed dropbox reporting OAuth quota errors, when it is likely Dropbox API quota errors

* Implemented exception detection on AzureBlob backend

* Also showing sleep times in output

* Fixed a bug with the dropbox quota response

* Fixed error message to report correct number of files

* Fixed suppressed warning
2024-12-20 16:10:08 +01:00
Kenneth SkovhedeandGitHub 68627c06bf Feature/alternate windows eventlog (#5784)
* Improved Windows Eventlog support.

Added support for logging to a different log with a Log:Source format.
Backwards compatible by treating "Source" as "Application:Source", where "Source" is default "Duplicati".

* Changed default log-level for eventlog to `Warning`.
This fixes #5770

* Fixed issue where checking for event log would crash the process.
Changed default log to `Duplicati` (was `Application`).
New default source names for service, agent and server.

* Reverted windows log-level back to information, now that there is a separate log for it

* Using "Duplicati 2" as the log name because "Duplicati" may be registered as a source name.

* Attempt to stop service before uninstalling

* Added additional diagnostics messages

* Fixed bug with starting the agent as a service

* Default to not use WindowsService for Agent as it requires permissions to create the event log

* Reduced log message levels for ongoing communication in agent

* Prevent double-logging from agent into Windows EventLog

* Made the agent log under a separate log source name
2024-12-20 16:09:42 +01:00
Kenneth SkovhedeandGitHub 0023dfd19f Removed the use of WebClient in MicrosoftGraphBackend in favor of HttpClient. (#5799) 2024-12-19 22:40:18 +01:00
Kenneth SkovhedeandGitHub 8593887184 Rename OneDrive v2 (#5795)
* Change displayname of OneDrive v2

Since OneDrive v1 is removed, there is no need to name the current backend `v2`.

* Renamed everything but the protocol key
2024-12-19 22:38:35 +01:00
Kenneth SkovhedeandGitHub 86f3f6b178 Fixed issues with timeout in msgraph (#5790)
* Fixed issues with timeout in msgraph

Fixed a place where the cancellation token was not applied.

Fixed a problem with msgraph not sending correct DNS information.

Fixed a problem with msgraph downloading the entire file before streaming it, bypassing throttle and tripping the timeout.

Added `ITimeoutExemptBackend` to selectively disable timeouts for backends that do not support it.

Updated the backendmanager to support `ITimeoutExemptBackend`.

Updated the backendmanager to abort as soon as a timeout happens.

* Updated BackendTester to use the `TimeoutObservingStream` and support the `--read-write-timeout` setting.

Updated backend tester code to use `var`

* Fixed a bug reading the `read-write-timeout`

* Updated to using constants with names instead of `-1`.
2024-12-19 09:31:06 +01:00
marceloduplicatiandGitHub e8c352e5db Merge pull request #5782 from marceloduplicati/feature/ftp-folder-escaping-fix
Fix to re-introduce Unscaping directory names for FTP Client
2024-12-18 09:57:21 -03:00
Kenneth SkovhedeandGitHub ff1b2b7990 Simplified task control (#5753)
* Simplified task control by having all logic in a single class and using async control mechanism
* Fixed the tests
* Made the marking of a backup partial explicitly communicated if the enumeration process is stopped.
* Added support for pausing active transfers
* Introduced transfer token to abort operations.
* Updated the way LiveControl is emitting events.
* Retains the paused state of the server across reboots, this fixes #5760.
* Reworked the way throttle speeds are propagated through LiveControls.
* Added option to pause transfers in the UI.
* Removed throttle settings from the LiveControls class so it is exclusively handled by the ApplicationSettings.
* Removed `thread-priority` as it does not work at all when tasks are used.
* Fixed the stop dialog and API to only support stop and abort.
* Removed unused variables
2024-12-18 08:27:59 +01:00
Marcelo C. c8c79d09eb Fix to re-introduce Unscaping directory names for FTP Client
The need for this was detected by forum post: https://forum.duplicati.com/t/ftp-without-ssl-fails-with-2-1-0-2-beta-2024-11-29/19691/12
2024-12-17 10:11:45 -03:00
Kenneth Skovhede 20a6973ffd Updated documentation for --synchronous-upload.
This fixes #4207
2024-12-17 12:13:34 +01:00
Kenneth Skovhede b610e412d7 Updated preload probing logic.
This updates the logic to not emit any console messags unless the environment variable `DUPLICATI_PRELOAD_SETTINGS_DEBUG=1` is set.
This changes the paths used on MacOS to not use the `/usr/share/Duplicati` folder which is write procted, but rather the `/usr/share/local/Duplicati`.

Added the `/usr/share/local/Duplicati` path for Linux as well.

Added detection for portable mode and only using relative/specific paths for preload in portable mode.

Added support for also probing `%LOCALAPPDATA% folder, which is where other settings are loaded, to allow "all settings in one place". Note that it is possible to use the preload to choose a different folder for anything but the `preload.json` file.

Added an option for specifying `DUPLICATI_PRELOAD_SETTINGS=!<path to file>`, where the leading `!` signifies that **only** that file is loaded and all other locations are ignored.
2024-12-13 11:07:40 +01:00
Kenneth Skovhede c9665da91b Reworked the tokens to only support emitting a single token per server startup 2024-12-12 11:05:09 +01:00
Kenneth Skovhede 7d4f9ffab3 Implemented a forever-token feature 2024-12-12 08:59:59 +01:00