This PR adds support for using Box, Dropbox, GoogleDrive, SharepointV2, MicrosoftGroups, and OneDrive as remote sources.
For GoogleDrive, this requires full permissions and not just the regular OAuth access token.
Also extended S3 and SMB to actually look up a file if requested, which makes it possible to rely on the backup exclusion files, even when using remote sources. (SSH/SFTP does not currently support this).
This PR adds a dynamic property so a backend can signal if it supports streaming, based on the settings.
This is currently used for the File backend, so that toggling `--use-move-for-put` will disable streaming on the backend instead of relying on the `--disable-streaming-transfers` flag.
This fixes the reported timeout issue with MicrosoftGraphBackend cause by each fragment being required to be uploaded within the `--short-timeout` value.
With this PR the overall progress is monitored with `--read-write-timeout` (as before) and each fragment's speed is not measured.
Also updated the fragment numbers with +1 so the first fragment is "1/5" instead of "0/5", such that the last fragment is "5/5".
This PR handles empty responses with the status code 204 - No Content.
Prior to this PR a 204 response when cancelling an upload session would cause an error due to not being able to parse the empty body.
This should get us one step closer to figuring out what is triggering the errors.
This PR updates the parsing code for errors/messags to avoid trying to read the stream twice.
This also removes the support for the unused HttpWebResponse (which is deprecated).
This PR removes the `OAuthContextSettings` class, and makes the `--oauth-url` option available on each backend that uses it, so it can be configured as part of the destination.
To assemble everything related to configuring OAuth, the code was moved to the `AuthIdOptionsHelper`, such that the usage can be shared between implementations.
This PR also adds the option to set the default OAuth url from the environment variable `DUPLICATI_OAUTH_SERVICE`.
The server can then be set in the following locations (most important last):
- Environment variable
- Server-wide advanced settings
- Backup job advanced settings (or commandline)
- Destination url
This PR moves the check from the FTP backend into shared code, so all backends will now check for read/write permissions when using the "Test" button.
For situations where write permissions are not required (like restore or verify) there is a flag on the v2 API that indicates if the connection was established, so the UI can show a message suggesting that it is possible to proceed, if the current UI operation does not require write permissions.
This fixes#2473
This PR is the final change, where all backends are now purely implemented with async interfaces.
The code changes here replace the LIST call with an async version. This affect all backends (and then some) but the changes are mostly mechanical rewrites of the code.
In many places, the backeds were already prepared for async output, in other some glue was needed.
To reduce the scope of this change, some backends simply report a synchronous result as an async list.
* Change displayname of OneDrive v2
Since OneDrive v1 is removed, there is no need to name the current backend `v2`.
* Renamed everything but the protocol key
* Fixed issues with timeout in msgraph
Fixed a place where the cancellation token was not applied.
Fixed a problem with msgraph not sending correct DNS information.
Fixed a problem with msgraph downloading the entire file before streaming it, bypassing throttle and tripping the timeout.
Added `ITimeoutExemptBackend` to selectively disable timeouts for backends that do not support it.
Updated the backendmanager to support `ITimeoutExemptBackend`.
Updated the backendmanager to abort as soon as a timeout happens.
* Updated BackendTester to use the `TimeoutObservingStream` and support the `--read-write-timeout` setting.
Updated backend tester code to use `var`
* Fixed a bug reading the `read-write-timeout`
* Updated to using constants with names instead of `-1`.
Kenneth Skovhede