This fixes the restore issues as all sources will now appear as if they were originally part of the source file system.
The backend implementation has now been simplified so the backends only return names of entries with no path information. Most backends already do this, but the S3 backends did not, so this was fixed as well.
This also clears up the interfaces, so the backend does not know about the `ISourceProviderEntry`. All mapping is done by the `BackendSourceProvider`, making it simpler to support other backends as sources.
Made the SMBShareConnection slightly more robust.
Introduced v2 List API to S3 backend, currently it retains v1 as the default.
Changed S3 backends to not recurse, potentially making smaller listings. This option can be disabled if it breaks things.
Added support for relative paths in the SSH backend.
This PR is the final change, where all backends are now purely implemented with async interfaces.
The code changes here replace the LIST call with an async version. This affect all backends (and then some) but the changes are mostly mechanical rewrites of the code.
In many places, the backeds were already prepared for async output, in other some glue was needed.
To reduce the scope of this change, some backends simply report a synchronous result as an async list.
With this interface it is possible to wrap the current support for snapshots and USN and simplify the enumeration.
This makes a number of assumptions into fields such as `IsMetaEntry` and `IsRootEntry` that removes code that was checking for various properties.
This first commit has support for VSS and USN as well as "naked" files, keeping feature parity.
There is also an experimental support for loading a backend-as-a-source. The support for this is currently limited as the backends need to be extended to support a folder concept.
This will make listing buckets with large count of files more reliable, if a 429 response is received it will conform to retry-after headers or if not present (as is the case today) will default to a 5 second delay.
* Removing timeout from Webdav
The timeout will be controlled externally by the TimeoutObservingStream and or the cancellation token.
Timeout for List set to default of 10 minutes
Since filesystems may take quite sometime to list large number of files, the timeout was set to a high value.
Previous timeout of 30 seconds was triggering in a case reported by user.
Timeout for CreateFolder and DeleteAsync was kept to default of 30 seconds.
Minor formmating/refactoring was done.
* Removing timeout from Webdav
The timeout will be controlled externally by the TimeoutObservingStream and or the cancellation token.
Timeout for List set to default of 10 minutes
Since filesystems may take quite sometime to list large number of files, the timeout was set to a high value.
Previous timeout of 30 seconds was triggering in a case reported by user.
Timeout for CreateFolder and DeleteAsync was kept to default of 30 seconds.
Minor formating/refactoring was done.
* Fixes to Integrated Authentication
This is a fix to the webdav when using integrated authentication, the hint for the underlying issue was found on an old [thread] (https://github.com/dotnet/wcf/issues/2923). By setting up a vm environment with Windows 2022 Server+IIS/Webdav it was possible to reproduce the [issue](https://github.com/duplicati/duplicati/issues/5872) and fix it.
Change to validation script is to avoid the requirement of filling the username when integrated authentication is true.
Modernize HTTP client implementation across backend services by:
- Replace legacy HttpWebClient with modern HttpClient
- Implement proper async/await patterns throughout the codebase
- Add cancellation token support for improved request handling
- Refactor related code for better maintainability
This change is part of our broader initiative to modernize the codebase
and improve its resilience through proper asynchronous operations.
* Fixed nullability warnings in `Utility.cs`
* Fixed a number of nullability issues and warnings
* More nullable fixes
* Fixed the overwrite issue
* Fixed additional nullability issues
* Removed additional nullability issues
* More nullability fixes and added some xmldoc
* Added more xmldocs and removed an unused argument
* Removed more warnings
Adds support for user-configurable read and write buffer sizes while enforcing protocol-negotiated maximums. When not specified, operations default to using the maximum sizes negotiated during the SMB session setup.
Tests revealed that by not using a single connection throttling issues appeared when connecting to Windows shares (possibly due to multiple authentications in a short period of time)
The 'no-reuse-connection' option controls whether SMB/CIFS connections are reused:
When enabled, creates a new connection and authentication for each file operation
When not specified, defaults to reusing the same connection across operations
* Primary functionality implemented for CIFS
This will add CIFS support to Duplicati.
One of the advantages is that the resource does not need to be mounted as the backend connects and authenticates directly to the resource via TCP/or Netbios over TCP.
The authentication supported is domain/username/password. At present integrated Windows authentication is not supported.
This implementation relies on SMBLibrary library (https://github.com/TalAloni/SMBLibrary).
* Fixes to XML documentation
* Implementing single CIFS connection + Fix on PutAsync
Single CIFS connection to avoid re-authentication on every operation.
Fix to PutAsync which only because apparent after file sizes were larger than _smb2Client.MaxWriteSize
* Adding CI tests for CIFS
The CI test for CIFS is self contained, requiring no environment variables. Also updated README.
This introduces a few constants in favor of multiple strings with the same value.
The supported arguments in the module have been augmented to better report possible values and default values.
The options parser has been updated to work better with CLI where the options are not default filled.
* Added options to insert delays in backend tester
* Added wait delays to Azure and GoogleDrive to see it the CI is more stable with small delays
* Added more debug output to the backend tester
* Fixed dropbox reporting OAuth quota errors, when it is likely Dropbox API quota errors
* Implemented exception detection on AzureBlob backend
* Also showing sleep times in output
* Fixed a bug with the dropbox quota response
* Fixed error message to report correct number of files
* Fixed suppressed warning
* Change displayname of OneDrive v2
Since OneDrive v1 is removed, there is no need to name the current backend `v2`.
* Renamed everything but the protocol key
* Fixed issues with timeout in msgraph
Fixed a place where the cancellation token was not applied.
Fixed a problem with msgraph not sending correct DNS information.
Fixed a problem with msgraph downloading the entire file before streaming it, bypassing throttle and tripping the timeout.
Added `ITimeoutExemptBackend` to selectively disable timeouts for backends that do not support it.
Updated the backendmanager to support `ITimeoutExemptBackend`.
Updated the backendmanager to abort as soon as a timeout happens.
* Updated BackendTester to use the `TimeoutObservingStream` and support the `--read-write-timeout` setting.
Updated backend tester code to use `var`
* Fixed a bug reading the `read-write-timeout`
* Updated to using constants with names instead of `-1`.
Kenneth Skovhede