Commit Graph
3678 Commits
Author SHA1 Message Date
Valentin c2dc922866 Mega Strings const review 2021-07-05 21:10:23 +02:00
Valentin eda3666bc7 Add support for mega.nz two fa authentication 2021-06-26 13:39:05 +02:00
Serge 5404fed634 Do not use StreamWriter 2021-06-02 19:35:40 +10:00
Serge 955dbebc03 Check contents of written test file
Before performing test write, reset stream Position to 0, otherwise test performs zero-byte write.
When performing test read, verify that file was written correctly.
2021-06-01 02:51:42 +10:00
warwickmmandGitHub ed86cca3e4 Merge pull request #4509 from sergethedev17/feature/azure-storage-sastoken
Add ability to provide SAS token for Azure backend
2021-05-30 10:15:43 -07:00
Kenneth SkovhedeandGitHub 8ee90fdf64 Merge pull request #4506 from warwickmm/mono_ssh_avoid_ecdsa_algorithms
Avoid ECDSA algorithms when using SFTP with Mono
2021-05-29 12:22:52 +02:00
Kenneth SkovhedeandGitHub 368a3a9a9b Merge branch 'master' into feature/azure-storage-sastoken 2021-05-29 12:20:30 +02:00
Kenneth SkovhedeandGitHub 2bcb0f511c Merge pull request #4513 from warwickmm/fix_bug_handlings_deprecated_options
Fix bug in handling deprecated Azure and S3 options
2021-05-29 12:17:44 +02:00
Kenneth Hsu a5fbee0335 Simplify exception handling when testing for ECDSA support.
.NET 5 moves the responsibility to the operating system, and since
SSH.NET currently relies on the ECDsaCng class, we will simply interpret
all exceptions to mean lack of ECDSA support.
2021-05-18 12:19:49 -07:00
Kenneth Hsu c246976fbb Cache knowledge of ECDSA support using static field.
This is simpler, and would potentially allow us to query this without an
instance.
2021-05-18 10:18:14 -07:00
Kenneth Hsu 22564bb2a3 Cache knowledge of ECDSA support. 2021-05-18 10:07:33 -07:00
Kenneth Hsu bfa240c8ec Assume we have ECDSA support if our test throws exception.
It's possible that our test will no longer be valid (e.g., due to
changes in SSH.NET), so we don't want the backend to fail if our test
is no longer valid.
2021-05-18 09:06:00 -07:00
Kenneth Hsu 90dfce81b2 Construct instance of ECDsaCng to test for ECDSA support.
This more closely reflects the SSH.NET code.
2021-05-18 09:00:40 -07:00
Kenneth Hsu 3d8817608e Test for ECDSA support explicitly.
The previous implementation could possibly fail for reasons unrelated to
ECDSA support.
2021-05-16 17:29:00 -07:00
marodev a156c511da Reorder parameters of previously wrongly refactored string interpolation 2021-05-16 09:12:00 +02:00
Kenneth Hsu 29f1d2c3cb Fix bug handling deprecated Azure and S3 options.
In pull request #4443, we renamed some of the Azure and S3 options to
use dashes '-' instead of underscores '_' for consistency.  However, our
treatment of the options in the backend constructors still required that
the underscore versions be present even if the dashed version was
provided.

Now, we allow for either option to be present, but the value for the
dashed version will take precedence.
2021-05-15 14:37:53 -07:00
marodev e87688d9d3 Fix ReSharper's "Use string interpolation expression" / Code Smell 2021-05-15 17:02:57 +02:00
Serge 5b3d85fc25 Add authentication using Shared access signature token
Allow users to authenticate using a Shared access signature (SAS) token. SAS tokens are more secure because they:
 * have a start and expiry date/time
 * can be restricted to specified IP addresses
 * can limit access scope to a single Storage blob container
2021-05-15 21:22:50 +10:00
Serge 5172119864 Bugfix read non-present key from dictionary
The lines with null-coalescing operator (??) throw an exception when the dictionary does not contain the first key (eg azure_account_name). Exception thrown: "The given key was not present in the dictionary".
2021-05-15 21:03:26 +10:00
Kenneth Hsu b80845a176 Enumerate items to list before modifying collection. 2021-05-11 15:40:33 -07:00
Kenneth Hsu 3e2103335b SSH: Avoid advertising support for ECDSA algorithms with Mono.
SSH.NET relies on the System.Security.Cryptography.ECDsaCng class for
ECDSA algorithms, which is not implemented in Mono (as of 6.12.0.144).
This prevents clients from connecting if one of the ECDSA algorithms is
chosen as the host key algorithm.  In the event that this causes a
connection failure, we will prevent the client from advertising support
for ECDSA algorithms and make another connection attempt.

Related forum discussion:

	https://forum.duplicati.com/t/release-2-0-6-1-beta-2021-sftp-failure-synology/12358
2021-05-11 12:42:34 -07:00
warwickmmandGitHub 7b4f475762 Merge pull request #4502 from samuel-w/master
Make file extension compression hints case-insensitive.
2021-05-09 14:29:59 -07:00
samuel-wandGitHub fa79b406a2 Improve comment 2021-05-09 15:16:31 -05:00
samuel-w d05d83cf4b Add comment 2021-05-08 15:51:10 -05:00
Kenneth SkovhedeandGitHub a2fba718ba Merge pull request #4501 from warwickmm/fix_typo_in_script_examples
Fix typo in example scripts
2021-05-08 11:50:19 +02:00
Kenneth SkovhedeandGitHub a43f7c8c91 Merge pull request #4503 from samuel-w/dar-sql
Add Disk Archive files to compression hints
2021-05-08 11:48:59 +02:00
Kenneth SkovhedeandGitHub 8ce34c084b Merge pull request #4488 from warwickmm/file_backend_should_truncate
Make File backend truncate existing files when overwriting
2021-05-08 11:40:23 +02:00
samuel-w bfa24581c1 Add Disk Archive files to compression hints 2021-05-08 01:45:07 -05:00
samuel-w d5c83d2af3 Compare file extensions case insensitively.
Fixes https://github.com/duplicati/duplicati/issues/4493
2021-05-08 01:14:08 -05:00
Kenneth Hsu 2d5d300234 Fix typo in example scripts. 2021-05-06 14:03:00 -07:00
warwickmmandGitHub 50ed059e11 Merge pull request #4469 from tygill/fix/ms-graph-throttling
Fix OneDrive throttling and respect retry-after header.
2021-05-04 09:42:52 -07:00
Kenneth Hsu c14ace0d68 Make File backend truncate existing files when overwriting.
The implementations of ISystemIO.FileOpenWrite call File.OpenWrite,
which does not truncate the existing file.  As such, if we overwrite a
larger file with a smaller one, tail remnants of the existing file will
remain.

Since most of the put operations involve unique filenames, this is only
a problem when overwriting existing files (e.g.,
duplicati-verification.json).

This fixes a regression introduced in revision 71c6aca8a7 ("Fixed IO
calls in Filebackend") from pull request #3456.  The previous
implementation called File.Open with FileMode.Create, which truncates
existing files.

This fixes #4486.
2021-04-20 16:16:00 -07:00
Tyler Gill fa33e5f135 Remove CopyStream buffering logic (will try to make a separate pull request for that) 2021-03-28 23:20:20 -06:00
Tyler Gill b64c2f1df0 Move Retry-After logic to RetryAfterHelper to make it easier for other backends to incorporate it. 2021-03-23 17:28:39 -06:00
Tyler Gill dadd30380c Add some logging to Microsoft Graph backends. 2021-03-23 17:08:58 -06:00
Tyler Gill 1ec1e18634 OAuthHttpClient should behave the same as the WebRequest code path and treat no http-operation-timeout override as meaning to use an infinite timeout. 2021-03-22 10:47:25 -06:00
Tyler Gill 3539112cca Store the Utility buffer in a WeakReference so the GC can collect it if necessary. 2021-03-22 09:47:17 -06:00
Tyler Gill 06c28387d5 Address some more feedback 2021-03-20 16:09:23 -06:00
Tyler Gill 47dccff835 Fix a few more things. 2021-03-20 13:37:41 -06:00
aureliandevel baf489e8a9 Fix column name typo 2021-03-20 13:26:53 +11:00
Tyler Gill 719adbf1f1 Clean up a few more bugs and inconsistencies in ReadLimitLengthStream. 2021-03-19 13:59:09 -06:00
Tyler Gill 7f1b6bc7a2 Fix some bugs in ReadLimitLengthStream changes, and add a unit test for some of the new behavior.
Also make NonSeekableStream block seeks via the Position setter.
2021-03-18 12:24:11 -06:00
Tyler Gill 9a4632d551 Remove SubStream and add functionality to ReadLimitLengthStream. Switch to using that.
Also add some hardening around the Position and make sure the stream starts at relative Position 0.
2021-03-17 20:37:46 -06:00
Tyler Gill 27b526daa8 Responding to some comments. 2021-03-17 17:25:19 -06:00
Tyler Gill b0793d3e49 Refactor the Retry-After logic. Now it tracks Retry-After headers on all requests, and blocks any future requests until the time from the prior request has passed. 2021-03-16 11:40:38 -06:00
Tyler Gill 7894152489 A few more improvements to the MicrosoftGraphBackends.
First, when performing an upload using an upload session, data is streamed directly from the input stream to the request stream. (It is no longer read into an intermediate buffer first.)
Hopefully this will help these uploads respond to throttling behavior better.
Second, these backends now check for and respect a Retry-After header if it is present.
On 5xx errors, the existing exponential backoff is used for the intermediate retries.
A few other minor things have also been cleaned up (e.g., using Task.Delay instead of Thread.Sleep)
2021-03-15 18:04:39 -06:00
warwickmmandGitHub b9db1ab149 Merge pull request #4464 from duplicati/warwickmm-replace_deprecated_s3_options
Fix S3 warnings about deprecated options.
2021-03-08 08:39:51 -08:00
Kenneth Hsu 18893e8e20 Fix URL to curl in CheckMonoSSL error message.
Starting in November 2020, curl.se is the new official home site for the
curl project.  By default, curl does not follow redirects, so this fixes
the URL in the error message and includes the option to follow redirects
in case the URL changes again in the future.
2021-03-07 16:39:41 -08:00
Kenneth Hsu c8261320e8 Do not alias S3 username and password to deprecated options.
The Controller class iterates over all supported options and will issue
a warning if any of the provided options are deprecated (or aliased to a
deprecated option).  If the provided auth-username option is aliased to
the deprecated aws_access_key_id, then a warning notification will
appear, even if the user hasn't explicitly provided the deprecated
value.

This fixes #4462.
2021-03-07 15:38:13 -08:00
Kenneth SkovhedeandGitHub ecc2cc57ef Merge pull request #4401 from warwickmm/cache_dir_environment_variable
Add $XDG_CACHE_HOME to Linux CacheFiles filter
2021-03-06 15:02:01 +01:00