Commit Graph
113 Commits
Author SHA1 Message Date
Kenneth SkovhedeandGitHub bc6114323f Workaround for multi-backend. (#5815)
* Workaround for multi-backend.

There is currently an issue with creating the BackendManager which has a backend instance. This instance is not used by the uploader, so they will create new instances.
The BackendManager instance can then time out, but is used later.

This commit will allow the uploader to "borrow" the instance from the backend manager, so only a single instance is needed.

This is not a clean fix, but doing so requires a full rewrite of the backend manager and uploader, so the controller creates a single instance of the manager, which is then used for all operations.

This would allow the backend manager to keep track of instances across all requests.

* Improved reuse after shutdown
2024-12-28 21:25:49 +01:00
Kenneth SkovhedeandGitHub 86f3f6b178 Fixed issues with timeout in msgraph (#5790)
* Fixed issues with timeout in msgraph

Fixed a place where the cancellation token was not applied.

Fixed a problem with msgraph not sending correct DNS information.

Fixed a problem with msgraph downloading the entire file before streaming it, bypassing throttle and tripping the timeout.

Added `ITimeoutExemptBackend` to selectively disable timeouts for backends that do not support it.

Updated the backendmanager to support `ITimeoutExemptBackend`.

Updated the backendmanager to abort as soon as a timeout happens.

* Updated BackendTester to use the `TimeoutObservingStream` and support the `--read-write-timeout` setting.

Updated backend tester code to use `var`

* Fixed a bug reading the `read-write-timeout`

* Updated to using constants with names instead of `-1`.
2024-12-19 09:31:06 +01:00
Kenneth SkovhedeandGitHub ff1b2b7990 Simplified task control (#5753)
* Simplified task control by having all logic in a single class and using async control mechanism
* Fixed the tests
* Made the marking of a backup partial explicitly communicated if the enumeration process is stopped.
* Added support for pausing active transfers
* Introduced transfer token to abort operations.
* Updated the way LiveControl is emitting events.
* Retains the paused state of the server across reboots, this fixes #5760.
* Reworked the way throttle speeds are propagated through LiveControls.
* Added option to pause transfers in the UI.
* Removed throttle settings from the LiveControls class so it is exclusively handled by the ApplicationSettings.
* Removed `thread-priority` as it does not work at all when tasks are used.
* Fixed the stop dialog and API to only support stop and abort.
* Removed unused variables
2024-12-18 08:27:59 +01:00
Kenneth Skovhede 5363bf0d96 Implemented read-write timeout.
This fixes #5725
2024-12-10 22:12:20 +01:00
Kenneth Skovhede 61ea36091a Added DNSNames as async 2024-10-02 09:37:52 +02:00
Kenneth Skovhede 586c08522e Implemented async method for Test, CreateFolder and Quota in all backends.
Still missing the List and DNSHostnames.
The AsyncRequest class is also preventing many backends from fully upgrading.
2024-10-01 09:30:02 +02:00
Kenneth Skovhede ecf2e8b551 Implemented DeleteAsync on all backends 2024-09-29 23:04:54 +02:00
Kenneth Skovhede e323c646f5 Fixed botched line 2024-09-29 22:05:49 +02:00
Kenneth Skovhede e3e3d1305f Implemented async version of Get for all backends. 2024-09-29 22:00:57 +02:00
Kenneth Skovhede b280432476 Changed logic to use HashCalculatingStream instead of CryptoStream, which should fix #4094.
Removed VolumeHashFactory as it dit not support using another hashing algorithm for volumes.
2024-09-28 11:13:06 +02:00
Kenneth Skovhede 627bccbb94 Removed piped download as it was slower than not piping.
It can be re-added later if a better way is found.
2024-09-26 08:43:01 +02:00
Kenneth Skovhede 9729b1442b Made the pre-backup verify method static to reduce possible side-effects 2024-09-10 12:11:05 +02:00
Kenneth Skovhede 473c6cbbb8 Merge commit '4f577c65a4d8806f79637c50c21ca3a683c5a07a' into feature/kestrel-avalonia-upgrade 2024-03-04 12:21:53 +01:00
Kenneth Skovhede 40dd3c6816 Updated all license mentions to MIT 2024-02-28 15:45:30 +01:00
Thomas Suckow d528703fe7 Merge branch 'master' into experiment/net5-split
# Conflicts:
#	.github/workflows/tests.yml
#	Duplicati.sln
#	Duplicati/CommandLine/BackendTester/Duplicati.CommandLine.BackendTester.csproj
#	Duplicati/CommandLine/BackendTool/Duplicati.CommandLine.BackendTool.csproj
#	Duplicati/CommandLine/Duplicati.CommandLine.csproj
#	Duplicati/CommandLine/RecoveryTool/Duplicati.CommandLine.RecoveryTool.csproj
#	Duplicati/GUI/Duplicati.GUI.TrayIcon/Duplicati.GUI.TrayIcon.csproj
#	Duplicati/Library/Backend/AzureBlob/AzureBlobWrapper.cs
#	Duplicati/Library/Backend/Mega/Duplicati.Library.Backend.Mega.csproj
#	Duplicati/Library/Backend/Mega/packages.config
#	Duplicati/Library/Backend/OAuthHelper/Duplicati.Library.OAuthHelper.csproj
#	Duplicati/Library/Backend/Tardigrade/Duplicati.Library.Backend.Tardigrade.csproj
#	Duplicati/Library/Backend/Tardigrade/Properties/AssemblyInfo.cs
#	Duplicati/Library/Backend/Tardigrade/packages.config
#	Duplicati/Library/Common/Duplicati.Library.Common.csproj
#	Duplicati/Library/Common/IO/SystemIOWindows.cs
#	Duplicati/Library/Modules/Builtin/Duplicati.Library.Modules.Builtin.csproj
#	Duplicati/Library/Modules/Builtin/packages.config
#	Duplicati/Library/Utility/Utility.cs
#	Duplicati/License/Duplicati.License.csproj
#	Duplicati/Server/Duplicati.Server.csproj
#	Duplicati/Server/packages.config
#	Duplicati/UnitTest/Duplicati.UnitTest.csproj
2022-02-22 19:04:01 -08:00
Riccardo Paolo Bestetti 528478fda4 Implement exponential backoff for backend errors 2022-01-20 23:22:08 +01:00
Thomas Suckow ecb0249259 dotnet no longer provides a method for a string to hashalgorithm
* Also fix up not diposing of hashers
2021-04-04 11:17:13 -07:00
Thomas Suckow d8ddd6982c Refactor VolumeHash 2021-04-04 08:28:44 -07:00
Thomas Suckow 898193fcef For now, just interrupt since its better than NOP. 2021-04-03 20:54:47 -07:00
Max 3f2eb254c3 buildable again 2020-05-31 14:38:14 +02:00
Max c882953fdd upstream 2020-05-31 13:50:13 +02:00
Kenneth Hsu a42a0a41b6 Remove unnecessary variable assignment. 2020-03-15 09:58:14 -07:00
Kenneth Hsu 7456d12dac Revert "Ensure that IndexVolumeWriter is disposed."
Since a reference to the IndexVolumeWriter is passed to the
FileEntryItem, we should not dispose of it here.

This reverts commit b6123cc156.

This is a potential fix for issue #4130.
2020-03-15 09:53:21 -07:00
Sean Templeton 9739517b0a Fix bandwidth throttling inconsistency
In code the throttling generally works backwards than what one would expect.
The streams that are throttled are the streams that operate on files from the local machine.
This fixes the download throttling not working and the upload throttling applying to
both uploads and downloads.

When doing a Put operation the stream on the local machine is reading from the
local filesystem and hence the upload throttle bytes per second needs to be
applied to the reading of the stream, not the writing.

Similarly when doing a Get operation the download throttle bytes per second needs
to be applied to the writing of the stream.

Fixes #4115
2020-03-04 19:57:48 -06:00
Kenneth Hsu 69005a64f7 Remove unused BackendManager.FlushDbMessages method.
The last reference was removed in revision
8abdf87188.
2020-01-26 10:18:19 -08:00
Kenneth Hsu e241a17af6 Remove unused BackendManager.CreateFolder method.
This was introduced in revision aca4ed95cc
but without any apparent usages.
2020-01-26 10:18:19 -08:00
Kenneth Hsu 7d3d03c06f Remove unused BackendManager.LastException property.
The last reference was removed in revision
8abdf87188.
2020-01-26 10:18:19 -08:00
Kenneth Hsu d847b46716 Remove unused BackendManager.HasDied property.
The last reference was removed in revision
8abdf87188.
2020-01-26 10:18:19 -08:00
Sean Templeton 04662bace1 Write blocklists when compacting files
The list folder was not being written to dindex files when performing a
compact operation. After the data blocks have been moved to a new volume
and the new volume data is finished being written to the new index file,
read the blocklists from the database and write them to the new index
file.
2020-01-19 21:57:08 -06:00
Max 28793ef204 packages update 2019-12-25 18:21:24 +01:00
Max 41a5591532 upstream 2019-12-25 16:34:56 +01:00
Kenneth Hsu 83ac03843d Fix spelling errors in comments.
In doing so, we also normalized some line endings.
2019-12-14 09:52:55 -08:00
drwtsn32x 8f9ec70680 only warn if delete failure is confirmed 2019-11-19 20:28:14 -08:00
Kenneth SkovhedeandGitHub 99d0cb3c57 Merge pull request #3953 from warwickmm/make_fields_readonly
Mark fields that don't need to be reassigned as readonly
2019-10-21 08:36:17 +02:00
Kenneth Hsu 2aa76881a5 Mark fields that don't need to be reassigned as readonly.
This makes it explicit at compile-time that these fields should not be
reassigned outside the constructor.
2019-10-19 13:59:22 -07:00
Kenneth Hsu afab61a05b Use pattern matching to simplify casts. 2019-10-19 10:15:38 -07:00
Kenneth Hsu b6123cc156 Ensure that IndexVolumeWriter is disposed. 2019-09-28 10:33:34 -07:00
Michal Naiman 1bc1e58737 upstream 2019-04-28 13:45:26 +02:00
Sean Templeton b02b19e683 Change Put method name to PutAsync 2019-03-17 18:28:45 -05:00
Sean Templeton 0c122ff550 Change IBackend and IStreamingBackend to return Task for Put() 2019-03-03 21:43:10 -06:00
Kenneth Hsu b7f28aa9f9 Remove empty finally blocks.
Empty finally blocks serve no purpose.  In cases where no exceptions are
caught, we removed the try as well.
2018-12-31 14:00:43 -08:00
Kenneth Hsu 1d57ddd5a6 Flatten AggregateExceptions before referencing InnerException.
The InnerException property of an AggregateException can be another
AggregateException.  By calling Flatten(), we remove inner
AggregateExceptions so that the InnerExceptions property contains all
the original exceptions.  The InnerException property is the first
element of InnerExceptions.
2018-10-13 19:07:49 -07:00
Rune HenriksenandGitHub be7edb580d Merge pull request #3410 from warwickmm/simplify_enumerating_collections
Simplify enumeration of collections
2018-10-07 19:06:28 +02:00
Kenneth Hsu 0c1ac9c489 Replace chained LINQ calls with call to overload with predicate.
This simplifies the code by reducing the number of enumerators created
while also making the code slightly more readable.
2018-10-06 16:20:18 -07:00
Kenneth Hsu d72128ec83 Remove unused method parameters. 2018-10-06 15:09:43 -07:00
Kenneth Hsu 8902da39f2 Undo removal of method parameters.
This should be done on a separate branch for a different pull request.
2018-09-29 10:44:05 -07:00
Kenneth Hsu 9ce1e807c7 Remove unused variables and fields. 2018-09-29 10:41:27 -07:00
Max a22a7a7d24 remove of FasterHashing component 2018-09-09 14:42:40 +02:00
Kenneth Hsu f12858e6a5 Mark fields that shouldn't be reassigned as readonly.
This makes it explicit at compile-time that these fields should not be
reassigned outside the constructor.
2018-05-25 10:04:52 -07:00
Kenneth Skovhede 9e48de8e01 Merge branch 'master' of github.com:duplicati/duplicati 2018-03-22 08:32:48 +01:00