This PR adds support for using Box, Dropbox, GoogleDrive, SharepointV2, MicrosoftGroups, and OneDrive as remote sources.
For GoogleDrive, this requires full permissions and not just the regular OAuth access token.
Also extended S3 and SMB to actually look up a file if requested, which makes it possible to rely on the backup exclusion files, even when using remote sources. (SSH/SFTP does not currently support this).
This adds support for the option `s3-authentication-region` which can be used to control the authentication region, which is required for some Minio setups.
This fixes#6562
This PR adds support for locking files if the backend supports it.
To activate locking, set the option `--file-lock-duration=30D` and the backup will lock the files.
If the database is rebuilt with the intention of continuing the backups, use the option `--repair-refresh-lock-info` which will update lock information in the database after recreating the database.
The locking works by asking the backend to lock files after a backup has completed.
The implementation keeps track of which files are currently assigned a lock and prevents attempting to delete the files that are currently locked.
Note that the bucket should not have a default lock policy as Duplicati needs to finish the backup before the locking is applied.
In this initial version, Azure Blob Storage, B2, S3 and iDrive are supported with locking.
The CLI is updated to allow setting locks on a specific version. The backend tool is updated to allow setting locks on specific files.
This fixes pre-calculating the SHA256 hash for AWS SDK requests to S3.
This fixes an issue where a throttled stream would not be throttled correctly, because the AWS library would read the throttled stream twice (once for hashing, and once for transmitting).
To ensure compatibility with Wasabi S3, the hash is also lowercased.
Some providers, B2, Idrivee2, JottaCloud and S3 (AWS version) needs a checksum for the content.
To calculate this content the libraries will calculate the hash by reading the stream. But since the stream can be throttled, the hash calculation is throttled.
In some cases the contents are buffered, so the transfer is done at full speed, but the hashing at throttled speed. The average speed is the same, but network traffic is bursty, which it should not be with a throttle.
Also added option to disable signing with AWS Sigv4 as that interferes with the throttled stream. It is not currently possible to pass the sha-256 hash to the Sigv4 code, so it will always read the entire (potentially throttled) stream before sending the contents, causing unwanted delays. Use the new option `--s3-disable-payload-signing` to disable v4 signing.
This fixes the restore issues as all sources will now appear as if they were originally part of the source file system.
The backend implementation has now been simplified so the backends only return names of entries with no path information. Most backends already do this, but the S3 backends did not, so this was fixed as well.
This also clears up the interfaces, so the backend does not know about the `ISourceProviderEntry`. All mapping is done by the `BackendSourceProvider`, making it simpler to support other backends as sources.
Made the SMBShareConnection slightly more robust.
Introduced v2 List API to S3 backend, currently it retains v1 as the default.
Changed S3 backends to not recurse, potentially making smaller listings. This option can be disabled if it breaks things.
Added support for relative paths in the SSH backend.
Delay was caused by AWS library performing network queries when reading properties on the configuration object.
On top of this, the properties were read twice.
Unified reading+application of options between iDrive and AWS S3.
This adds a method to get a default instance of AmazonS3Config. This
accomplishes two key things:
1. Previously, the documentation for the default buffer size (8192) was
inconsistent with the actual default value (65536). This was due to
differences in the property values betwen the instance used to
generate the documentation, and that used in the actual backend
client.
2. Disables the use of Amazon Resource Names (ARN). This does not seem
critical to our application. Moreover, if the UseArnRegion property
is not set, accessing it triggers an expensive operation (~30
seconds) to get the region endpoint. This causes the loading of the
backend modules to take a long time and run afoul of HTTP request
timeouts, causing the application to think that no backends are
available.
The previous version of the AWSSDK.S3 library that we used did not
have this property, so this was not an issue until we started
migrating away from .NET Framework.
This fixes#4543.
Carl Johnsen