Kenneth Skovhede
11a27042bc
Merge remote-tracking branch 'origin/master' into feature/alternate-source-provider
2025-02-26 19:25:46 +01:00
Kenneth Skovhede and GitHub
4822bc4550
Merge pull request #5978 from duplicati/feature/make-backends-async
...
Final conversion to async backends
2025-02-26 18:02:05 +01:00
Olivier Chaze
395a5cb64f
Adding two Infomaiak Cloud locations
2025-02-26 14:47:20 +01:00
Kenneth Skovhede
6238067f32
Updated logic for S3 backend, so the client implementations return the relative (non-prefixed) path
2025-02-24 13:17:08 +01:00
Kenneth Skovhede
4e73be1806
Fixed path conversion on Windows
2025-02-21 16:25:18 +01:00
Kenneth Skovhede
9dfe977d3f
Changed logic so the source providers are now logically mounted into the source directory structure.
...
This fixes the restore issues as all sources will now appear as if they were originally part of the source file system.
The backend implementation has now been simplified so the backends only return names of entries with no path information. Most backends already do this, but the S3 backends did not, so this was fixed as well.
This also clears up the interfaces, so the backend does not know about the `ISourceProviderEntry`. All mapping is done by the `BackendSourceProvider`, making it simpler to support other backends as sources.
2025-02-21 16:06:55 +01:00
Kenneth Skovhede
1716d09b51
Fixed source providers for S3, CIFS, and SSH.
...
Made the SMBShareConnection slightly more robust.
Introduced v2 List API to S3 backend, currently it retains v1 as the default.
Changed S3 backends to not recurse, potentially making smaller listings. This option can be disabled if it breaks things.
Added support for relative paths in the SSH backend.
2025-02-21 13:20:52 +01:00
Kenneth Skovhede
36ba547a92
Added the @ prefix to backend paths
2025-02-19 14:20:03 +01:00
Kenneth Skovhede
7031bc57b6
Adding ConfigureAwait(false) to all async enumerations
2025-02-18 09:18:36 +01:00
Kenneth Skovhede
0f9669b273
Final conversion to async backends
...
This PR is the final change, where all backends are now purely implemented with async interfaces.
The code changes here replace the LIST call with an async version. This affect all backends (and then some) but the changes are mostly mechanical rewrites of the code.
In many places, the backeds were already prepared for async output, in other some glue was needed.
To reduce the scope of this change, some backends simply report a synchronous result as an async list.
2025-02-17 16:45:51 +01:00
Kenneth Skovhede
bb5fd80a29
Implemented support for some backends as sources: SSH, CIFS, and S3
2025-02-17 14:01:08 +01:00
Kenneth Skovhede and GitHub
9464caf622
Feature/update license 2025 ( #5851 )
...
* Fixed some minor whitespace issues
* Updated all copyright to 2025
2025-01-07 09:40:39 +01:00
Kenneth Skovhede
38a255eb00
Fixed startup delay on Windows.
...
Delay was caused by AWS library performing network queries when reading properties on the configuration object.
On top of this, the properties were read twice.
Unified reading+application of options between iDrive and AWS S3.
2024-11-15 14:04:42 +01:00
Kenneth Skovhede
81c0b2209e
Fixed a warning when using S3
2024-11-11 21:20:21 +01:00
Kenneth Skovhede
a9cd488bf2
Change the s3-client option to an enumeration-type as it has two possible values.
...
Fixed the parser to handle empty string and null as the same.
2024-11-06 22:07:27 +01:00
Kenneth Skovhede
454b1bb93b
Merge remote-tracking branch 'origin/master' into feature/backend-async-get
2024-10-04 09:11:50 +02:00
Kenneth Skovhede
66ec130761
Added endpoints for Seagate Lyve S3
2024-10-03 21:47:14 +02:00
Kenneth Skovhede
61ea36091a
Added DNSNames as async
2024-10-02 09:37:52 +02:00
Kenneth Skovhede
586c08522e
Implemented async method for Test, CreateFolder and Quota in all backends.
...
Still missing the List and DNSHostnames.
The AsyncRequest class is also preventing many backends from fully upgrading.
2024-10-01 09:30:02 +02:00
Kenneth Skovhede
8bdb85e261
Implemented RenameAsync on all backends supporting rename
2024-09-29 23:21:50 +02:00
Kenneth Skovhede
ecf2e8b551
Implemented DeleteAsync on all backends
2024-09-29 23:04:54 +02:00
Kenneth Skovhede
e3e3d1305f
Implemented async version of Get for all backends.
2024-09-29 22:00:57 +02:00
a289093eec
Update Duplicati/Library/Backend/S3/S3Backend.cs
...
Co-authored-by: Suguru Hirahara <luixxiul@users.noreply.github.com >
2024-08-01 08:04:28 +02:00
Kenneth Skovhede
2fe5115dd6
Added Sakura Cloud to list of S3 providers
2024-07-29 16:03:23 +02:00
Suguru Hirahara
254bcc63a7
Rename DescriptionS3ClientLong to S3ClientDescriptionLong
2024-07-26 22:25:40 -04:00
Poli and GitHub
9d7fad14cf
Update S3Backend.cs - Poli Systems GmbH
...
Updated the S3 backends of Poli Systems.
S3 became S3-02 and there is a new backend S3-03.
2024-06-15 01:57:45 +02:00
Kenneth Skovhede and GitHub
f20d7b9321
Merge pull request #5222 from duplicati/feature/some-s3-option-cleanup
...
Removed S3 options that have been deprecated for years.
2024-06-11 10:42:32 +02:00
Kenneth Skovhede
b11c558ba2
Removed options that have been deprecated for years.
...
Simplified the parsing of options slightly.
2024-06-04 20:54:57 +02:00
Kenneth Skovhede
c0a49dec2d
Removed unused US region, and fixed mapping of location names
2024-06-04 20:13:02 +02:00
Kenneth Skovhede
dbdd9a0022
Added additional AWS S3 regions:
...
- Africa (Cape Town): `af-south-1`
- Asia Pacific (Hyderabad): `ap-south-2`
- Asia Pacific (Jakarta): `ap-southeast-3`
- Asia Pacific (Melbourne): `ap-southeast-4`
- Canada West (Calgary): `ca-west-1`
- Europe (Spain): `eu-south-2`
- Europe (Zurich): `eu-central-2`
- Israel (Tel Aviv): `il-central-1`
- Middle East (UAE): `me-central-1`
- AWS GovCloud (US-East): `us-gov-east-1`
- AWS GovCloud (US-West): `us-gov-west-1`
List of region endpoints is now generated from region table.
2024-06-04 20:00:04 +02:00
Kenneth Skovhede
473c6cbbb8
Merge commit '4f577c65a4d8806f79637c50c21ca3a683c5a07a' into feature/kestrel-avalonia-upgrade
2024-03-04 12:21:53 +01:00
Thomas Suckow
a3bbf63561
Merge remote-tracking branch 'origin/feature/upgrade-to-sdk-style-projects' into feature/net5-kestrel
...
# Conflicts:
# .appveyor.yml
# .github/workflows/tests.yml
# BuildTools/AutoUpdateBuilder/AutoUpdateBuilder.csproj
# BuildTools/DependencyFinder/DependencyFinder.csproj
# BuildTools/GnupgSigningTool/GnupgSigningTool.csproj
# BuildTools/UpdateVersionStamp/UpdateVersionStamp.csproj
# Duplicati CommandLine Only.sln
# Duplicati.Library.RestAPI/RESTMethods/RequestInfo.cs
# Duplicati.Library.RestAPI/Runner.cs
# Duplicati.sln
# Duplicati/CommandLine/BackendTester/Duplicati.CommandLine.BackendTester.csproj
# Duplicati/CommandLine/BackendTester/app.config
# Duplicati/CommandLine/BackendTool/Duplicati.CommandLine.BackendTool.csproj
# Duplicati/CommandLine/BackendTool/app.config
# Duplicati/CommandLine/ConfigurationImporter/Duplicati.CommandLine.ConfigurationImporter.csproj
# Duplicati/CommandLine/Duplicati.CommandLine.csproj
# Duplicati/CommandLine/RecoveryTool/Duplicati.CommandLine.RecoveryTool.csproj
# Duplicati/CommandLine/app.config
# Duplicati/GUI/Duplicati.GUI.TrayIcon/Assets/icons/normal-warning.png
# Duplicati/GUI/Duplicati.GUI.TrayIcon/Duplicati.GUI.TrayIcon.csproj
# Duplicati/GUI/Duplicati.GUI.TrayIcon/Program.cs
# Duplicati/GUI/Duplicati.GUI.TrayIcon/app.config
# Duplicati/Library/AutoUpdater/Duplicati.Library.AutoUpdater.csproj
# Duplicati/Library/Backend/AlternativeFTP/Duplicati.Library.Backend.AlternativeFTP.csproj
# Duplicati/Library/Backend/AzureBlob/Duplicati.Library.Backend.AzureBlob.csproj
# Duplicati/Library/Backend/Backblaze/Duplicati.Library.Backend.Backblaze.csproj
# Duplicati/Library/Backend/Box/Duplicati.Library.Backend.Box.csproj
# Duplicati/Library/Backend/CloudFiles/Duplicati.Library.Backend.CloudFiles.csproj
# Duplicati/Library/Backend/Dropbox/Duplicati.Library.Backend.Dropbox.csproj
# Duplicati/Library/Backend/FTP/Duplicati.Library.Backend.FTP.csproj
# Duplicati/Library/Backend/File/Duplicati.Library.Backend.File.csproj
# Duplicati/Library/Backend/GoogleServices/Duplicati.Library.Backend.GoogleServices.csproj
# Duplicati/Library/Backend/HubiC/Duplicati.Library.Backend.HubiC.csproj
# Duplicati/Library/Backend/Idrivee2/Duplicati.Library.Backend.Idrivee2.csproj
# Duplicati/Library/Backend/Jottacloud/Duplicati.Library.Backend.Jottacloud.csproj
# Duplicati/Library/Backend/Mega/Duplicati.Library.Backend.Mega.csproj
# Duplicati/Library/Backend/OAuthHelper/Duplicati.Library.OAuthHelper.csproj
# Duplicati/Library/Backend/OneDrive/Duplicati.Library.Backend.OneDrive.csproj
# Duplicati/Library/Backend/OpenStack/Duplicati.Library.Backend.OpenStack.csproj
# Duplicati/Library/Backend/Rclone/Duplicati.Library.Backend.Rclone.csproj
# Duplicati/Library/Backend/S3/Duplicati.Library.Backend.S3.csproj
# Duplicati/Library/Backend/SSHv2/Duplicati.Library.Backend.SSHv2.csproj
# Duplicati/Library/Backend/SharePoint/Duplicati.Library.Backend.SharePoint.csproj
# Duplicati/Library/Backend/Sia/Duplicati.Library.Backend.Sia.csproj
# Duplicati/Library/Backend/Storj/Duplicati.Library.Backend.Storj.csproj
# Duplicati/Library/Backend/TahoeLAFS/Duplicati.Library.Backend.TahoeLAFS.csproj
# Duplicati/Library/Backend/Tardigrade/Duplicati.Library.Backend.Tardigrade.csproj
# Duplicati/Library/Backend/TencentCOS/Duplicati.Library.Backend.TencentCOS.csproj
# Duplicati/Library/Backend/WEBDAV/Duplicati.Library.Backend.WEBDAV.csproj
# Duplicati/Library/Common/Duplicati.Library.Common.csproj
# Duplicati/Library/Compression/Duplicati.Library.Compression.csproj
# Duplicati/Library/DynamicLoader/Duplicati.Library.DynamicLoader.csproj
# Duplicati/Library/Encryption/Duplicati.Library.Encryption.csproj
# Duplicati/Library/Interface/Duplicati.Library.Interface.csproj
# Duplicati/Library/Localization/Duplicati.Library.Localization.csproj
# Duplicati/Library/Logging/Duplicati.Library.Logging.csproj
# Duplicati/Library/Main/Duplicati.Library.Main.csproj
# Duplicati/Library/Modules/Builtin/CheckMonoSSL.cs
# Duplicati/Library/Modules/Builtin/Duplicati.Library.Modules.Builtin.csproj
# Duplicati/Library/Modules/Builtin/SendJabberMessage.cs
# Duplicati/Library/SQLiteHelper/Duplicati.Library.SQLiteHelper.csproj
# Duplicati/Library/Snapshots/Duplicati.Library.Snapshots.csproj
# Duplicati/Library/UsageReporter/Duplicati.Library.UsageReporter.csproj
# Duplicati/Library/Utility/Duplicati.Library.Utility.csproj
# Duplicati/Library/Utility/HashAlgorithmHelper.cs
# Duplicati/Library/Utility/Power/WindowsPowerSupplyState.cs
# Duplicati/License/Duplicati.License.csproj
# Duplicati/Server/Duplicati.Server.Serialization/Duplicati.Server.Serialization.csproj
# Duplicati/Server/Duplicati.Server.csproj
# Duplicati/Server/app.config
# Duplicati/Service/Duplicati.Service.csproj
# Duplicati/Service/app.config
# Duplicati/Tools/Duplicati.Tools.csproj
# Duplicati/UnitTest/Duplicati.UnitTest.csproj
# Duplicati/UnitTest/GeneralBlackBoxTesting.cs
# Duplicati/UnitTest/app.config
# Duplicati/WindowsService/WindowsService.csproj
# Installer/debian/docker/debian/control
# Tools/SQLiteTool/SQLiteTool.csproj
# Tools/ZipFileDebugger/ZipFileDebugger.csproj
# guiTests/guiTest.py
# thirdparty/UnixSupport/UnixSupport.csproj
# thirdparty/alphavss/Homepage.txt
# thirdparty/alphavss/License.txt
# thirdparty/alphavss/licensedata.json
2024-02-29 21:55:17 -08:00
Kenneth Skovhede
40dd3c6816
Updated all license mentions to MIT
2024-02-28 15:45:30 +01:00
Jan and GitHub
3a0667dc64
Add option to disable chunked encoding for AWS S3. ( #4999 )
...
Some S3 providers do not support MultiChunkedEncoding (#4994 ).
2023-09-25 02:02:55 +02:00
adithajay and GitHub
16c13f4fef
Update S3Backend.cs
...
Added endpoints for S3 compatible storage from Scaleway. I feel adding this provider is important as they provide 75GB of free storage and many users would utilize that.
2023-04-12 02:31:15 +05:30
Daniel Baker and GitHub
a8053ce702
Update S3Backend.cs
...
Added Impossible Cloud US endpoint.
2023-03-15 16:27:58 +01:00
Thomas Suckow
78197453a1
Merge remote-tracking branch master into feature/net5-avalonia
...
# Conflicts:
# Duplicati CommandLine Only.sln
# Duplicati.sln
# Duplicati/CommandLine/BackendTester/Duplicati.CommandLine.BackendTester.csproj
# Duplicati/CommandLine/BackendTool/Duplicati.CommandLine.BackendTool.csproj
# Duplicati/CommandLine/Duplicati.CommandLine.csproj
# Duplicati/CommandLine/RecoveryTool/Duplicati.CommandLine.RecoveryTool.csproj
# Duplicati/GUI/Duplicati.GUI.TrayIcon/Duplicati.GUI.TrayIcon.csproj
# Duplicati/GUI/Duplicati.GUI.TrayIcon/app.config
# Duplicati/Library/AutoUpdater/UpdaterManager.cs
# Duplicati/Library/Backend/Jottacloud/Duplicati.Library.Backend.Jottacloud.csproj
# Duplicati/Library/Main/Duplicati.Library.Main.csproj
# Duplicati/Library/Utility/Duplicati.Library.Utility.csproj
# Duplicati/Library/Utility/Utility.cs
# Duplicati/Server/Duplicati.Server.csproj
# Duplicati/Server/app.config
# Duplicati/Service/app.config
# Duplicati/UnitTest/BackendToolTests.cs
# Duplicati/UnitTest/Duplicati.UnitTest.csproj
# guiTests/guiTest.py
2022-11-11 21:40:08 -08:00
linitio
1ee2214b8d
FIx missing ',' on S3Backend.cs
2022-06-14 16:34:36 +02:00
linitio
7560a65f04
Add all Infomaniak clusters Swiss Backup & Public Cloud
2022-06-14 16:20:29 +02:00
linitio
8ade3abe8c
Add all Infomaniak clusters Swiss Backup & Public Cloud
2022-06-14 16:17:16 +02:00
Kenneth Skovhede
63e7c4e693
Minor naming fix
2022-06-12 20:25:22 +02:00
Thomas Suckow
d528703fe7
Merge branch 'master' into experiment/net5-split
...
# Conflicts:
# .github/workflows/tests.yml
# Duplicati.sln
# Duplicati/CommandLine/BackendTester/Duplicati.CommandLine.BackendTester.csproj
# Duplicati/CommandLine/BackendTool/Duplicati.CommandLine.BackendTool.csproj
# Duplicati/CommandLine/Duplicati.CommandLine.csproj
# Duplicati/CommandLine/RecoveryTool/Duplicati.CommandLine.RecoveryTool.csproj
# Duplicati/GUI/Duplicati.GUI.TrayIcon/Duplicati.GUI.TrayIcon.csproj
# Duplicati/Library/Backend/AzureBlob/AzureBlobWrapper.cs
# Duplicati/Library/Backend/Mega/Duplicati.Library.Backend.Mega.csproj
# Duplicati/Library/Backend/Mega/packages.config
# Duplicati/Library/Backend/OAuthHelper/Duplicati.Library.OAuthHelper.csproj
# Duplicati/Library/Backend/Tardigrade/Duplicati.Library.Backend.Tardigrade.csproj
# Duplicati/Library/Backend/Tardigrade/Properties/AssemblyInfo.cs
# Duplicati/Library/Backend/Tardigrade/packages.config
# Duplicati/Library/Common/Duplicati.Library.Common.csproj
# Duplicati/Library/Common/IO/SystemIOWindows.cs
# Duplicati/Library/Modules/Builtin/Duplicati.Library.Modules.Builtin.csproj
# Duplicati/Library/Modules/Builtin/packages.config
# Duplicati/Library/Utility/Utility.cs
# Duplicati/License/Duplicati.License.csproj
# Duplicati/Server/Duplicati.Server.csproj
# Duplicati/Server/packages.config
# Duplicati/UnitTest/Duplicati.UnitTest.csproj
2022-02-22 19:04:01 -08:00
Mickael Asseline
81d6d1e40a
Add Infomaniak SwissBackup to backend
2022-01-31 15:17:52 +01:00
Poli and GitHub
8fda8caafb
Update S3Backend.cs
2021-09-19 15:11:58 +02:00
Kenneth Hsu
f2bcf470d2
Add factory method to get default AmazonS3Config instance.
...
This adds a method to get a default instance of AmazonS3Config. This
accomplishes two key things:
1. Previously, the documentation for the default buffer size (8192) was
inconsistent with the actual default value (65536). This was due to
differences in the property values betwen the instance used to
generate the documentation, and that used in the actual backend
client.
2. Disables the use of Amazon Resource Names (ARN). This does not seem
critical to our application. Moreover, if the UseArnRegion property
is not set, accessing it triggers an expensive operation (~30
seconds) to get the region endpoint. This causes the loading of the
backend modules to take a long time and run afoul of HTTP request
timeouts, causing the application to think that no backends are
available.
The previous version of the AWSSDK.S3 library that we used did not
have this property, so this was not an issue until we started
migrating away from .NET Framework.
This fixes #4543 .
2021-06-28 09:52:13 -07:00
Kenneth Hsu
de2a651763
Fix bug in PutAsync implementations.
...
Without await, the using statement can dispose the Stream before the
call to PutAsync completes, resulting in an ObjectDisposedException.
This fixes #4556 .
2021-06-12 15:12:13 -07:00
Kenneth Hsu
29f1d2c3cb
Fix bug handling deprecated Azure and S3 options.
...
In pull request #4443 , we renamed some of the Azure and S3 options to
use dashes '-' instead of underscores '_' for consistency. However, our
treatment of the options in the backend constructors still required that
the underscore versions be present even if the dashed version was
provided.
Now, we allow for either option to be present, but the value for the
dashed version will take precedence.
2021-05-15 14:37:53 -07:00
Kenneth Hsu
c8261320e8
Do not alias S3 username and password to deprecated options.
...
The Controller class iterates over all supported options and will issue
a warning if any of the provided options are deprecated (or aliased to a
deprecated option). If the provided auth-username option is aliased to
the deprecated aws_access_key_id, then a warning notification will
appear, even if the user hasn't explicitly provided the deprecated
value.
This fixes #4462 .
2021-03-07 15:38:13 -08:00
Juliano Hubel
8933d6cd2a
#3748|Spelling Correction
2021-02-16 17:10:02 +00:00
Juliano Hubel
6a569b80d2
#3748|Adding_Deprecated_messages_to_old_variables
2021-02-16 10:41:55 +00:00
Kenneth Skovhede