This PR fixes a reported issue with strings that are less than 3 characters.
This is unlikely, but happens if the code tries to get the scheme of a path, like `/a` or `/`.
The updated logic returns the scheme if one exists, and it is less than 15 characters (to avoid leaking sensitive information).
This PR adds checks for free temporary space when starting the server and when restoring.
On systems that have limited space in the temp folder a warning will now be shown.
This is intended to capture issues on some Docker systems where /tmp is mounted in memory instead of being disk backed.
It will also detect the issue on other systems that are space constrained.
- Added Cloudflare to list of known S3 provider domains.
- Removed AWS specific naming on the help texts.
- Fixed an edge case where calculating how many files to test with an empty list would crash
- Fixed a bug with getting a safe display url for very short urls
This fixes an issue with formatting for invariant values that was caused by incrorrect function overload selection.
To avoid future issues, the two similar functions have been renamed to clarify what they are working for.
This fixes pre-calculating the SHA256 hash for AWS SDK requests to S3.
This fixes an issue where a throttled stream would not be throttled correctly, because the AWS library would read the throttled stream twice (once for hashing, and once for transmitting).
To ensure compatibility with Wasabi S3, the hash is also lowercased.
This adds nullability to FilterExpression and Options.
It also updates almost all properties in Options to use a common format that enforces uniform parsing of the values.
A few options have been updates to have the default values specified as a constant, to avoid replicating the value in two places.
This PR adds the option to suppress warnings by their log id. Warning IDs that are supplied to `--suppress-warnings` will be converted to information messages before being logged.
If a specific warning is disabled, such as `CompressionReadErrorFallback`, this will then no longer count as a warning for the job, and the log file will see the warning as an information message.
This PR also adds two simpler filter options that makes it possible to filter log messages by supplying the log IDs. This can already be achieved with log filters, but the ID filter is a bit simpler to apply, as you only need to know the ID.
Finally, this PR also adds common logging for errors in the categories:
- Permission denied, id = `PermissionDenied`
- File locked, id = `FileLocked`
- Path not found, id = `PathNotFound`
- Path too long, id = `PathTooLong`
These new log ids makes it simpler to ignore warnings about locked or inaccesible files.
Some providers, B2, Idrivee2, JottaCloud and S3 (AWS version) needs a checksum for the content.
To calculate this content the libraries will calculate the hash by reading the stream. But since the stream can be throttled, the hash calculation is throttled.
In some cases the contents are buffered, so the transfer is done at full speed, but the hashing at throttled speed. The average speed is the same, but network traffic is bursty, which it should not be with a throttle.
Also added option to disable signing with AWS Sigv4 as that interferes with the throttled stream. It is not currently possible to pass the sha-256 hash to the Sigv4 code, so it will always read the entire (potentially throttled) stream before sending the contents, causing unwanted delays. Use the new option `--s3-disable-payload-signing` to disable v4 signing.
This removes the WorkerThread class and many of the complications around it.
Instead of having a generic worker, there is now the `QueueRunnerService` which takes on all the resposibilities for handling queued tasks.
This is one step towards removing `FIXMEGlobal`
This adds a common way to catch stop exceptions and handles them so error resilient logic does not continue when a stop exception is thrown.
Previously, the logic would only check for `ThreadAbortException` which was used to signal a stop prior to .NET8.
This PR adds a unified way of supplying timeouts to each of the backends.
It also adds an opt-in way of adding commonly used options, so parsing and usage can be standardized across multiple backends.
This adds a retry count and a retry delay for posting backup reports via HTTP requests. This will make the report delivery more reliable if the network is unstable or server is briefly unreachable.
It should be noted that I was unable to completely remove IPv4 on Windows 11 to a state where it would not provide the 127.0.0.1 loopback, even when disabling Ipv4 in various ways, including with registry flags.
On ubuntu/linux, after completely disabling ipv4 the ipv6 loopback was returned.
This enables a fail-early approach, where the remote source loads the initial results.
By storing the `IAsyncEnumerable` it is possible to not perform any additional remote calls, even with the initial probing call.
This makes it possible to detect the absence of sources, so the backup can be aborted if sources are missing.
This fixes the restore issues as all sources will now appear as if they were originally part of the source file system.
The backend implementation has now been simplified so the backends only return names of entries with no path information. Most backends already do this, but the S3 backends did not, so this was fixed as well.
This also clears up the interfaces, so the backend does not know about the `ISourceProviderEntry`. All mapping is done by the `BackendSourceProvider`, making it simpler to support other backends as sources.
With this interface it is possible to wrap the current support for snapshots and USN and simplify the enumeration.
This makes a number of assumptions into fields such as `IsMetaEntry` and `IsRootEntry` that removes code that was checking for various properties.
This first commit has support for VSS and USN as well as "naked" files, keeping feature parity.
There is also an experimental support for loading a backend-as-a-source. The support for this is currently limited as the backends need to be extended to support a folder concept.
* Rewrite the backend manager.
This creates a new more logic backend manager that handles all backend operations.
For each top-level operation, there is now a single backend manager instance that is passed to sub-commands.
The internals of the backend manager are now rewritten to use async logic instead of threading.
The design uses a single task runner that dispatches operations and honors concurrent settings and limits from one place.
The logic for each operation has been moved into a separate files/classes to make it easier to understand each operation.
This fixes#5804
* Fixed a few issues with not awaiting tasks
* Fixed not creating empty databases
* Fixed an issue with hashes not being recorded on download
* Reworked the download logic to avoid attempting to decrypt the file if it has been modified.
* Review fixes
* Renamed db collector to better reflect the purpose.
* Review fixes
* Fixed nullability warnings in `Utility.cs`
* Fixed a number of nullability issues and warnings
* More nullable fixes
* Fixed the overwrite issue
* Fixed additional nullability issues
* Removed additional nullability issues
* More nullability fixes and added some xmldoc
* Added more xmldocs and removed an unused argument
* Removed more warnings
* Simplified task control by having all logic in a single class and using async control mechanism
* Fixed the tests
* Made the marking of a backup partial explicitly communicated if the enumeration process is stopped.
* Added support for pausing active transfers
* Introduced transfer token to abort operations.
* Updated the way LiveControl is emitting events.
* Retains the paused state of the server across reboots, this fixes#5760.
* Reworked the way throttle speeds are propagated through LiveControls.
* Added option to pause transfers in the UI.
* Removed throttle settings from the LiveControls class so it is exclusively handled by the ApplicationSettings.
* Removed `thread-priority` as it does not work at all when tasks are used.
* Fixed the stop dialog and API to only support stop and abort.
* Removed unused variables
* Implemented support for emitting intermediate certificates from pfx files
* Added certificate check to crash earlier
* Fixed loading certificate on Windows
None of these changes have impact on code execution, they are mainly part of the effort to remove all build warnings.
The removal of Thread.ResetAbort leaves a TODO for us to integrate CancelationToken when it gets propagated down here on future changes.
Kenneth Skovhede