Commit Graph
1378 Commits
Author SHA1 Message Date
Kenneth Skovhede bc8f6dd0c8 This adds the option to parse flags, where an enum can have multiple values.
It is only used by the FTP backend.
2025-01-30 10:52:17 +01:00
Kenneth SkovhedeandGitHub b6981d8668 This fixes errors with the Minio backend not re-throwing exceptions (#5897)
* This fixes errors with the Minio backend not re-throwing exceptions

* Throwing the correct exception ...
2025-01-22 23:29:27 +01:00
Kenneth SkovhedeandGitHub 997891525a Updated license-updater to update csproj files (#5898)
* Updated license-updater to update csproj files

* License updates
2025-01-22 21:04:19 +01:00
Kenneth SkovhedeandGitHub 045c6b9147 Feature/adjust pcloud exception (#5883)
* Fixed pCloud exception type

Changed pCloud exception to `FileMissingException` (was `FileNotFoundException`)

* Fixed a label in tests
2025-01-16 22:21:16 +01:00
marceloduplicatiandGitHub 08f231dd74 Fixes to timeouts on WebDav (#5878)
* Removing timeout from Webdav

The timeout will be controlled externally by the TimeoutObservingStream and or the cancellation token.

Timeout for List set to default of 10 minutes

Since filesystems may take quite sometime to list large number of files, the timeout was set to a high value.

Previous timeout of 30 seconds was triggering in a case reported by user.

Timeout for CreateFolder and DeleteAsync was kept to default of 30 seconds.

Minor formmating/refactoring was done.

* Removing timeout from Webdav

The timeout will be controlled externally by the TimeoutObservingStream and or the cancellation token.

Timeout for List set to default of 10 minutes

Since filesystems may take quite sometime to list large number of files, the timeout was set to a high value.

Previous timeout of 30 seconds was triggering in a case reported by user.

Timeout for CreateFolder and DeleteAsync was kept to default of 30 seconds.

Minor formating/refactoring was done.

* Fixes to Integrated Authentication

This is a fix to the webdav when using integrated authentication, the hint for the underlying issue was found on an old [thread] (https://github.com/dotnet/wcf/issues/2923). By setting up a vm environment with Windows 2022 Server+IIS/Webdav it was possible to reproduce the [issue](https://github.com/duplicati/duplicati/issues/5872) and fix it.

Change to validation script is to avoid the requirement of filling the username when integrated authentication is true.
2025-01-16 16:29:40 +01:00
marceloduplicatiandGitHub 5e477e7e87 Migrate from HttpWebClient to HttpClient with async/await + Refactoring (#5875)
Modernize HTTP client implementation across backend services by:
- Replace legacy HttpWebClient with modern HttpClient
- Implement proper async/await patterns throughout the codebase
- Add cancellation token support for improved request handling
- Refactor related code for better maintainability

This change is part of our broader initiative to modernize the codebase
and improve its resilience through proper asynchronous operations.
2025-01-16 16:29:16 +01:00
Kenneth SkovhedeandGitHub 58bf21306a Improve S3 logic for permission tests (#5879) 2025-01-16 14:28:20 +01:00
Kenneth SkovhedeandGitHub 15856fd46f Fixed 2 compile warnings (#5873) 2025-01-14 19:42:50 +01:00
Kenneth SkovhedeandGitHub 6bbd64a655 Fixed nullability warnings in Utility.cs (#5861)
* Fixed nullability warnings in `Utility.cs`

* Fixed a number of nullability issues and warnings

* More nullable fixes

* Fixed the overwrite issue

* Fixed additional nullability issues

* Removed additional nullability issues

* More nullability fixes and added some xmldoc

* Added more xmldocs and removed an unused argument

* Removed more warnings
2025-01-14 14:03:48 +01:00
Marcelo C. cffba06749 Add configurable read and write buffer sizes for SMB operations
Adds support for user-configurable read and write buffer sizes while enforcing protocol-negotiated maximums. When not specified, operations default to using the maximum sizes negotiated during the SMB session setup.
2025-01-10 13:53:51 -03:00
Marcelo C. 3ce67e529a Remove no-reuse-connection Option
Tests revealed that by not using a single connection throttling issues appeared when connecting to Windows shares (possibly due to multiple authentications in a short period of time)
2025-01-10 12:10:03 -03:00
Marcelo C. a1263cd48c Path normalization
Path separators are converted to \ which is Windows & Linux compatible, overwriting the Path.Combine separator (which would be host dependant)
2025-01-10 12:05:40 -03:00
Marcelo C. f598b478f9 Add 'no-reuse-connection' option for SMB/CIFS operations
The 'no-reuse-connection' option controls whether SMB/CIFS connections are reused:

When enabled, creates a new connection and authentication for each file operation
When not specified, defaults to reusing the same connection across operations
2025-01-08 22:25:34 -03:00
Kenneth SkovhedeandGitHub 9464caf622 Feature/update license 2025 (#5851)
* Fixed some minor whitespace issues

* Updated all copyright to 2025
2025-01-07 09:40:39 +01:00
marceloduplicatiandGitHub 99ee234640 Primary functionality implemented for CIFS (#5780)
* Primary functionality implemented for CIFS

This will add CIFS support to Duplicati.

One of the advantages is that the resource does not need to be mounted as the backend connects and authenticates directly to the resource via TCP/or Netbios over TCP.

The authentication supported is domain/username/password. At present integrated Windows authentication is not supported.

This implementation relies on SMBLibrary library (https://github.com/TalAloni/SMBLibrary).

* Fixes to XML documentation

* Implementing single CIFS connection + Fix on PutAsync

Single CIFS connection to avoid re-authentication on every operation.
Fix to PutAsync which only because apparent after file sizes were larger than _smb2Client.MaxWriteSize

* Adding CI tests for CIFS

The CI test for CIFS is self contained, requiring no environment variables. Also updated README.
2025-01-06 14:22:23 +01:00
Kenneth SkovhedeandGitHub 2cbf62d02b Update Storj options and docs (#5838)
This introduces a few constants in favor of multiple strings with the same value.
The supported arguments in the module have been augmented to better report possible values and default values.
The options parser has been updated to work better with CLI where the options are not default filled.
2025-01-03 17:21:00 +01:00
Kenneth SkovhedeandGitHub 228a472b07 Changed Sia redundancy to decimal (#5837) 2025-01-03 17:20:37 +01:00
Kenneth SkovhedeandGitHub d0924c3e01 Auto-select version for OpenStack (#5836)
This adds a check to see if the give url contains the standard `/v3` suffix, and if so selects the v3 protocol.
2025-01-03 17:20:19 +01:00
Kenneth SkovhedeandGitHub c70b593fed Corrected documentation for OpenStack (#5835)
The region was not described correctly.
2025-01-03 17:19:57 +01:00
Kenneth SkovhedeandGitHub 78da4527be Removed unused region from Aliyun (#5833) 2025-01-03 14:08:29 +01:00
Kenneth SkovhedeandGitHub 2fac31bd60 Clean Azure Blob options (#5829)
This removes Azure Blob options that has been deprecated for years.
The option `--azure-blob-container-name` was not used, so it was removed as well.
2025-01-02 18:51:00 +01:00
Kenneth SkovhedeandGitHub 6fdb01b52a This commit ensures the client connects on first use, instead of relying on the library to connect. (#5828) 2025-01-01 14:06:35 +01:00
Kenneth SkovhedeandGitHub 31503ea595 Added option to log FTP diagnostics data to regular log (#5821) 2024-12-31 16:47:28 +01:00
Kenneth SkovhedeandGitHub ba645ddb5f Clean up options for file backend (#5820)
This removes unsupported arguments from non-windows clients, so the reported options fit what is supported.
2024-12-31 16:37:07 +01:00
Kenneth SkovhedeandGitHub 8f853cfa7f Google drive and GCS do not work with timeout (#5812) 2024-12-26 10:42:58 +01:00
Kenneth SkovhedeandGitHub 394b876985 Enable noop daemon for FTP (#5813)
This fixes an issue with timeouts on large file.
2024-12-26 10:41:06 +01:00
Kenneth SkovhedeandGitHub 447c2b3238 Feature/improve ci testing (#5781)
* Added options to insert delays in backend tester

* Added wait delays to Azure and GoogleDrive to see it the CI is more stable with small delays

* Added more debug output to the backend tester

* Fixed dropbox reporting OAuth quota errors, when it is likely Dropbox API quota errors

* Implemented exception detection on AzureBlob backend

* Also showing sleep times in output

* Fixed a bug with the dropbox quota response

* Fixed error message to report correct number of files

* Fixed suppressed warning
2024-12-20 16:10:08 +01:00
Kenneth SkovhedeandGitHub 0023dfd19f Removed the use of WebClient in MicrosoftGraphBackend in favor of HttpClient. (#5799) 2024-12-19 22:40:18 +01:00
Kenneth SkovhedeandGitHub 8593887184 Rename OneDrive v2 (#5795)
* Change displayname of OneDrive v2

Since OneDrive v1 is removed, there is no need to name the current backend `v2`.

* Renamed everything but the protocol key
2024-12-19 22:38:35 +01:00
Kenneth SkovhedeandGitHub 86f3f6b178 Fixed issues with timeout in msgraph (#5790)
* Fixed issues with timeout in msgraph

Fixed a place where the cancellation token was not applied.

Fixed a problem with msgraph not sending correct DNS information.

Fixed a problem with msgraph downloading the entire file before streaming it, bypassing throttle and tripping the timeout.

Added `ITimeoutExemptBackend` to selectively disable timeouts for backends that do not support it.

Updated the backendmanager to support `ITimeoutExemptBackend`.

Updated the backendmanager to abort as soon as a timeout happens.

* Updated BackendTester to use the `TimeoutObservingStream` and support the `--read-write-timeout` setting.

Updated backend tester code to use `var`

* Fixed a bug reading the `read-write-timeout`

* Updated to using constants with names instead of `-1`.
2024-12-19 09:31:06 +01:00
Marcelo C. c8c79d09eb Fix to re-introduce Unscaping directory names for FTP Client
The need for this was detected by forum post: https://forum.duplicati.com/t/ftp-without-ssl-fails-with-2-1-0-2-beta-2024-11-29/19691/12
2024-12-17 10:11:45 -03:00
Kenneth Skovhede 4dfc4f586e Adding e2 url format to description.
This fixes #5755
2024-12-11 08:38:06 +01:00
Kenneth Skovhede ac61b2afbe Updated FluentFTP to latest.
This should fix #5715
2024-12-09 10:20:27 +01:00
Kenneth Skovhede e738221bb2 Removed unused dependency 2024-12-05 14:42:53 +01:00
Kenneth Skovhede 6156024e8f Bumped SSH.NET to latest
This fixes #4674
2024-12-05 14:33:44 +01:00
Kenneth SkovhedeandGitHub f3a71fa1c5 Merge pull request #5732 from marceloduplicati/feature/fix-pcloud-subfolders
Implement nested folder support and improve file operations on pCloud Backend
2024-12-05 14:03:20 +01:00
Marcelo C. ce73e0f1f6 Implementation of fixes suggested on code review
Post changes tests:

- Folder creation and discovery checked.
- Backup and restore operations checked.
- Tests with backend tests performed.
2024-12-04 09:29:14 -03:00
Kenneth SkovhedeandGitHub 471ff7ede8 Merge pull request #5734 from duplicati/feature/fix-quota-info-on-file-backend
Fixed issue with quota info.
2024-12-04 09:39:42 +01:00
Kenneth Skovhede 17cc382325 The legacy flag --use-ssl toggles explicit TLS 2024-12-04 08:34:09 +01:00
Kenneth Skovhede 356498414c Fixed issue with quota info.
This fixes #5733
2024-12-03 22:12:38 +01:00
Marcelo C. 356d0c1bfd Implement nested folder support and improve file operations on pCloud Backend
- Add support for subfolders
- Implement robust file selection using fileID for Download/Delete operations instead of concatenation of path + filename
- Improved validation for folder existence during backend connectivity tests supporting subfolders.
2024-12-03 12:16:17 -03:00
Marcelo C. 2eab4611de Fix for the streaming error on BackendTester interaction
This issue does not affect the pCloud backend per se and its functioning as expected.

The issue is related to a symptom the BackendTester revealed as discussed here:

https://github.com/duplicati/duplicati/pull/5709

It turns out the backends should assume the caller has set the input stream at the correct position and take it from there.

Therefore the fix is simply to remove the .Position = 0; which was well intended as a way to make sure the stream was at the beginning, but  thats not the pattern to follow when integrating backends.

A future blog post/tutorial on new backend integrations will mention these premises.
2024-11-27 19:34:18 +01:00
Kenneth SkovhedeandGitHub 79d6dc0cad Merge branch 'master' into feature/backends-add-pcloud 2024-11-25 11:08:55 +01:00
Marcelo C. 068ae42aa1 Implementation of changes requested in review 2024-11-25 11:01:59 +01:00
Kenneth Skovhede c915b70697 Added support for ftp/aftp treating url paths as before the update 2024-11-21 14:37:25 +01:00
Kenneth Skovhede 9dadd514bf Inverted the flag for relative paths, as that is more compatible with the previous library. 2024-11-21 13:13:44 +01:00
Kenneth Skovhede 5712264e6a Updated some parsing to be simpler.
Changed the strategy to use absolute paths for all operations, to avoid cases where a silent reconnect would end up in a different folder.
Add backwards compatible option to use CWD style relative operations.
Added support for detecting the SSL certificate hash when there is a certificate error.
2024-11-19 13:55:23 +01:00
Kenneth Skovhede 2828729a59 Changed logic for how error messages are constructed 2024-11-19 06:52:17 +01:00
Marcelo Ceccon 4beafc0cc7 Added pCloud Integration
Added pCloud backend with custom interface for editing destination.
2024-11-18 13:07:01 +01:00
Kenneth Skovhede 88ae1d18b1 Fixed output from S3 options to mimic previous version 2024-11-15 14:18:39 +01:00