Commit Graph
4937 Commits
Author SHA1 Message Date
Kenneth SkovhedeandGitHub e76bbc478f Merge pull request #5950 from duplicati/feature/fix-zip-fallback
Handle automatic fallback when reading zip files
2025-02-10 21:06:39 +01:00
Kenneth SkovhedeandGitHub 11df878673 Merge pull request #5949 from duplicati/feature/protect-datafolder
Add permission protection to data folder and database
2025-02-10 21:06:15 +01:00
Kenneth SkovhedeandGitHub 6693b77d85 Merge pull request #5948 from duplicati/feature/fix-not-storing-updated-timestamps
Fix not storing updated timestamps
2025-02-10 21:05:46 +01:00
Carl Johnsen 043b9230b5 VerifyLocalBlocks opened file in incorrect mode, and the target file may not exist. 2025-02-10 13:44:02 +01:00
Carl Johnsen 82019ea28e Modified VerifyTargetBlocks to not evict block from cache. It is now handled outside, in case of file name retargeting when overwrite is false. 2025-02-10 13:42:27 +01:00
Carl Johnsen 0b3ff33716 Moved retire on exception to final, so regardless of how the FileProcessor exits, it retires the channels. 2025-02-10 12:38:24 +01:00
Carl Johnsen 3b9fc188e1 Retire block request and response on exception in FileProcessor 2025-02-10 12:34:44 +01:00
Carl Johnsen 259322f0d9 CopyOldTargetBlocksToNewTarget was made async 2025-02-10 12:34:16 +01:00
Carl Johnsen 6e2f8156f8 VerifyLocalBlocks should only perform checks when the target differs from the original. 2025-02-10 12:33:45 +01:00
Carl Johnsen dbd3520d9c Formatting 2025-02-10 12:18:16 +01:00
Kenneth Skovhede 3caaa78336 Merge remote-tracking branch 'origin/master' into feature/alternate-source-provider 2025-02-09 16:15:06 +01:00
Kenneth Skovhede 8c935a7344 This creates a new ISourceProvider interface for describing a data source.
With this interface it is possible to wrap the current support for snapshots and USN and simplify the enumeration.

This makes a number of assumptions into fields such as `IsMetaEntry` and `IsRootEntry` that removes code that was checking for various properties.

This first commit has support for VSS and USN as well as "naked" files, keeping feature parity.

There is also an experimental support for loading a backend-as-a-source. The support for this is currently limited as the backends need to be extended to support a folder concept.
2025-02-09 16:14:22 +01:00
Marcelo C. d5cc0ffbb5 Adding 429 observance to backend
This will make listing buckets with large count of files more reliable, if a 429 response is received it will conform to retry-after headers or if not present (as is the case today) will default to a 5 second delay.
2025-02-09 12:13:12 -03:00
Kenneth Skovhede bb3ea2f6cd Fixed a rare startup issue 2025-02-09 11:05:54 +01:00
Kenneth Skovhede f22026cf59 Fixed a compile issue with older SDKs 2025-02-09 10:52:47 +01:00
Kenneth Skovhede da1f6562ea Added option to log to console 2025-02-09 10:29:04 +01:00
Kenneth SkovhedeandGitHub 5645b8ae79 Merge pull request #5945 from duplicati/feature/add-option-to-disable-update-checks
Added option to disable update check
2025-02-07 16:16:03 +01:00
Kenneth SkovhedeandGitHub 35179d9966 Merge pull request #5946 from duplicati/feature/prevent-dataloss-from-outdated-database
Add a check for stale database
2025-02-07 16:15:40 +01:00
Kenneth Skovhede a8dea5c352 Fixed an issue with dropping existing permissions on Windows.
When setting the folder permissions on an existing folder, it would remove the permissions on items inside the folder.

This would cause the files and subfolders to have zero permissions, and reject any interaction.
2025-02-07 16:10:30 +01:00
Kenneth Skovhede b4a5820cf4 Fixed incorrect dispose logic 2025-02-07 14:06:57 +01:00
Kenneth Skovhede 3e72768eec Added warning when using the Mega backend 2025-02-06 21:53:51 +01:00
Kenneth Skovhede f7536df9c2 Added explicit access for the LocalSystem user as suggested. 2025-02-06 21:47:28 +01:00
Kenneth Skovhede adbd6b6a7c Added support for fallback when reading zip files with unsupported compression methods 2025-02-06 21:35:49 +01:00
Kenneth Skovhede 536e62275f Changed the logic in recreate to avoid opening the same zip file multiple times. 2025-02-06 21:34:02 +01:00
Kenneth Skovhede a68dba607e Added support for using the current uid/gid instead of using the file uid/gid 2025-02-06 17:21:17 +01:00
Kenneth Skovhede f0f79418d8 Fixed logic so the permissions are always applied, unless the user specifically opts out by placing a file named insecure-permissions.txt in the data folder. 2025-02-06 17:09:18 +01:00
Kenneth Skovhede 1af6e2b1d6 Experiment with CommonProgramData on Windows in favor of users folder. 2025-02-06 16:57:44 +01:00
Kenneth Skovhede ebdfc774c2 Removed static IO_WIN and IO_SYS as cross-platform calls are not possible. 2025-02-06 16:57:12 +01:00
Kenneth Skovhede aab91eb0df Added protection to newly create data folder and databases 2025-02-06 16:56:06 +01:00
Kenneth Skovhede 1690871d10 Implemented fix that pushes updated timestamps to previous version, in the case that no files were changed, and no new backup is recorded in remote storage. 2025-02-06 15:31:20 +01:00
Marcelo C. 11643c2413 Removing TimeoutObservingStream from backend manager 2025-02-06 11:07:17 -03:00
Kenneth Skovhede bae82feeae Add a check for stale database
This commit adds a check to verify that the remote database is up-to-date before proceeding to repair the remote storage.

Without this check it is possible to start repairs with an old database, and this will cause the remote data to be deleted.

This fixes #4579
2025-02-06 13:20:32 +01:00
Kenneth Skovhede abe79b1a15 Added option to disable update check
This fixes #3983
2025-02-06 11:50:38 +01:00
Marcelo C. a203de90e1 Added support for Topic ID on Telegram
This adds support for TopicID as specified in parameter "send-telegram-topid-id" which allows the messages to be organized in topics on a group.

Finding out your topic ID in a group can be rather tricky the easiest way is to use curl to create it.

```bash
export CHAT_ID="-99999999"
export TOPIC_NAME="Your Topic"
export BOT_TOKEN="7777777777:AAAAAAAAA-4WwwwwwwwwwwwwwwwwwwwwwwQ"

curl -X POST \
    -H "Content-Type: application/json" \
    -d "{\"chat_id\":\"$CHAT_ID\",\"name\":\"$TOPIC_NAME\"}" \
    https://api.telegram.org/bot$BOT_TOKEN/createForumTopic
```
Which will response with a json where message_thread_id is the Topic ID
```bash
{"ok":true,"result":{"message_thread_id":33,"name":.....
```
2025-02-03 11:05:27 -03:00
Kenneth SkovhedeandGitHub 28e3fa1063 Merge pull request #5914 from duplicati/feature/prevent-windows-subfolder-storage
Improve data folder assignment
2025-01-31 14:52:45 +01:00
Kenneth SkovhedeandGitHub 198b6dcd3f Merge pull request #5929 from duplicati/feature/fix-parsing-flags
This adds the option to parse flags, where an enum can have multiple values
2025-01-31 14:52:18 +01:00
Kenneth Skovhede 23e7233691 Do not encode http form extra parameters.
This changes the encoding that broke because the encoding function was changed.

The new logic is to not encode the parameters, but just use them as provided.

Any escaping needs to then be applied to the input by the user/caller.

This fixes #5900
2025-01-30 21:39:41 +01:00
marceloduplicatiandGitHub 03b7fac65c Merge pull request #5889 from marceloduplicati/feature/unified-help-arguments
Help Command unification
2025-01-30 10:23:10 -03:00
Kenneth SkovhedeandGitHub 3db8e03262 Merge pull request #5891 from duplicati/feature/add-support-for-cors
Added support for CORS
2025-01-30 10:53:27 +01:00
Kenneth Skovhede bc8f6dd0c8 This adds the option to parse flags, where an enum can have multiple values.
It is only used by the FTP backend.
2025-01-30 10:52:17 +01:00
Kenneth Skovhede fac866a5cd Merge remote-tracking branch 'origin/master' into feature/add-support-for-cors 2025-01-30 08:39:15 +01:00
Kenneth Skovhede b1e23da40c Implemented correct shutdown of backend manager handler 2025-01-29 11:04:13 +01:00
Kenneth Skovhede de00eee3de Changed logic so overrides to server-datafolder will affect any operation, including CLI, Agent, and ServerUtil 2025-01-28 13:23:57 +01:00
Kenneth Skovhede fac12ba772 Further consolidated the data folder.
Added support for `--portable-mode` and `--server-datafolder` for the CLI, ServerUtil, and Agent.

Fixed a few places where file reads were not cached properly.

Moved some responsibilities of UpdaterManager into DataFolderManager.

This now supports storing `machineid.txt` and `installation.txt` inside the path pointed to by `--server-datafolder` or alternatively with `--portable-mode`.

It is possible that some fringe backwards compatibility is lost with this update.

Fixed an issue with AutoUpdater crashing on check/download.

This fixes #5902
2025-01-28 13:01:35 +01:00
Kenneth SkovhedeandGitHub 039e1a08fb Merge branch 'master' into feature/prevent-windows-subfolder-storage 2025-01-28 08:58:26 +01:00
Kenneth SkovhedeandGitHub 32d4c76c13 Merge pull request #5913 from duplicati/feature/more-visible-warnings
Added UI warning for unencrypted database
2025-01-28 08:57:08 +01:00
Kenneth SkovhedeandGitHub 610f3b47d1 Update reporting module (#5910)
* Updated reporting values.

The reporter module can now emit the following extra values:
- operating-system
- installation-type
- destination-type
- next-scheduled-run

When running from the built-in scheduler, it will automatically set `--next-scheduled-run` so it is applied to the reporting data.

* Implemented an improved machine name reader.
This fixes #5907

* Added a guard against sending too much of the destination url as the destination.

* Fixed reading Windows Machine name

* Removed extra Windows logic as it has no effect
2025-01-28 08:56:42 +01:00
Kenneth SkovhedeandGitHub f7b7555e87 Fixes for tests (#5918)
* Two small test fixes

* Ignore exceptions when comitting an empty transaction

* Attempt to fix FTP connection issue

* Unified test by having a single download function.
Added DNS probing call to attempt fixing sporadic DNS errors.

* Fixed argument order

* Avoid creating temp file during tests, if not needed
2025-01-28 08:55:28 +01:00
Kenneth SkovhedeandGitHub d99f74ed94 Rewrite the backend manager. (#5896)
* Rewrite the backend manager.

This creates a new more logic backend manager that handles all backend operations.

For each top-level operation, there is now a single backend manager instance that is passed to sub-commands.

The internals of the backend manager are now rewritten to use async logic instead of threading.

The design uses a single task runner that dispatches operations and honors concurrent settings and limits from one place.

The logic for each operation has been moved into a separate files/classes to make it easier to understand each operation.

This fixes #5804

* Fixed a few issues with not awaiting tasks

* Fixed not creating empty databases

* Fixed an issue with hashes not being recorded on download

* Reworked the download logic to avoid attempting to decrypt the file if it has been modified.

* Review fixes

* Renamed db collector to better reflect the purpose.

* Review fixes
2025-01-28 08:54:50 +01:00
Kenneth Skovhede 27f0bb4ea0 Unified code to find the data folder.
The `machineid.txt` and `installation.txt` files are now default located in same folder where the database is default located.
2025-01-27 22:51:45 +01:00