This PR adds support for using Box, Dropbox, GoogleDrive, SharepointV2, MicrosoftGroups, and OneDrive as remote sources.
For GoogleDrive, this requires full permissions and not just the regular OAuth access token.
Also extended S3 and SMB to actually look up a file if requested, which makes it possible to rely on the backup exclusion files, even when using remote sources. (SSH/SFTP does not currently support this).
This adds support for the option `s3-authentication-region` which can be used to control the authentication region, which is required for some Minio setups.
This fixes#6562
- Added Cloudflare to list of known S3 provider domains.
- Removed AWS specific naming on the help texts.
- Fixed an edge case where calculating how many files to test with an empty list would crash
- Fixed a bug with getting a safe display url for very short urls
This PR adds support for locking files if the backend supports it.
To activate locking, set the option `--file-lock-duration=30D` and the backup will lock the files.
If the database is rebuilt with the intention of continuing the backups, use the option `--repair-refresh-lock-info` which will update lock information in the database after recreating the database.
The locking works by asking the backend to lock files after a backup has completed.
The implementation keeps track of which files are currently assigned a lock and prevents attempting to delete the files that are currently locked.
Note that the bucket should not have a default lock policy as Duplicati needs to finish the backup before the locking is applied.
In this initial version, Azure Blob Storage, B2, S3 and iDrive are supported with locking.
The CLI is updated to allow setting locks on a specific version. The backend tool is updated to allow setting locks on specific files.
This adds support for supplying a url for the hostname which will also toggle the default SSL setting.
Removed providers MyCloudyPlace and dinCloud as they appear to no longer operate.
This PR adds a dynamic property so a backend can signal if it supports streaming, based on the settings.
This is currently used for the File backend, so that toggling `--use-move-for-put` will disable streaming on the backend instead of relying on the `--disable-streaming-transfers` flag.
This PR updates the S3 backend to automatically remove leading and trailing `/` characters so upgrades are less likely to encounter the error, as the previous version did not fail on such hostnames.
The check still captures cases where the user attempts to input a path or URL into the hostname field.
This fixes pre-calculating the SHA256 hash for AWS SDK requests to S3.
This fixes an issue where a throttled stream would not be throttled correctly, because the AWS library would read the throttled stream twice (once for hashing, and once for transmitting).
To ensure compatibility with Wasabi S3, the hash is also lowercased.
This PR moves the check from the FTP backend into shared code, so all backends will now check for read/write permissions when using the "Test" button.
For situations where write permissions are not required (like restore or verify) there is a flag on the v2 API that indicates if the connection was established, so the UI can show a message suggesting that it is possible to proceed, if the current UI operation does not require write permissions.
This fixes#2473
Some providers, B2, Idrivee2, JottaCloud and S3 (AWS version) needs a checksum for the content.
To calculate this content the libraries will calculate the hash by reading the stream. But since the stream can be throttled, the hash calculation is throttled.
In some cases the contents are buffered, so the transfer is done at full speed, but the hashing at throttled speed. The average speed is the same, but network traffic is bursty, which it should not be with a throttle.
Also added option to disable signing with AWS Sigv4 as that interferes with the throttled stream. It is not currently possible to pass the sha-256 hash to the Sigv4 code, so it will always read the entire (potentially throttled) stream before sending the contents, causing unwanted delays. Use the new option `--s3-disable-payload-signing` to disable v4 signing.
Kenneth Skovhede