Commit Graph
3115 Commits
Author SHA1 Message Date
Kenneth Skovhede 7a979464f1 Merge remote-tracking branch 'origin/HEAD' into feature/add-authentication-region-support-for-s3 2026-01-06 11:37:55 +01:00
Kenneth Skovhede 557fbd12d9 Fixed nullable issues 2026-01-06 10:22:53 +01:00
Kenneth Skovhede 83779a9486 Minor fixes
- Added Cloudflare to list of known S3 provider domains.
- Removed AWS specific naming on the help texts.
- Fixed an edge case where calculating how many files to test with an empty list would crash
- Fixed a bug with getting a safe display url for very short urls
2026-01-06 09:48:23 +01:00
Kenneth Skovhede 625c56b5c1 Updated new column to be not null, for consistency with other columns 2025-12-17 23:04:47 +01:00
Kenneth Skovhede 6e6e993765 Fixed issues with database transaction not being comitted.
Renamed the options to be more descriptive.
Rewrote the dependency query to be more readable and also check for metadata dependency.

Fixed an edge case where there were no modified files, and no new backup is created.

Fixed some issues with local/utc datetime compares.

Fixed logic for deleting filesets to only look at the filesets themselves. This makes the version lock follow the dlist file, and can create "dangling" volumes that have no filesets, but these will be picked up by compaction.

Updated tests to work correctly.

Added lock-info update option for the backup recreate call, so the UI can also re-create lock information on database rebuilds.
2025-12-17 22:27:38 +01:00
Kenneth Skovhede 0f81b42146 Add support for remote locks
This PR adds support for locking files if the backend supports it.

To activate locking, set the option `--file-lock-duration=30D` and the backup will lock the files.

If the database is rebuilt with the intention of continuing the backups, use the option `--repair-refresh-lock-info` which will update lock information in the database after recreating the database.

The locking works by asking the backend to lock files after a backup has completed.

The implementation keeps track of which files are currently assigned a lock and prevents attempting to delete the files that are currently locked.

Note that the bucket should not have a default lock policy as Duplicati needs to finish the backup before the locking is applied.

In this initial version, Azure Blob Storage, B2, S3 and iDrive are supported with locking.

The CLI is updated to allow setting locks on a specific version. The backend tool is updated to allow setting locks on specific files.
2025-12-14 17:17:48 +01:00
Kenneth Skovhede cf42426d1b Store password with TrayIcon
When using the TrayIcon with no hosted server (i.e., connecting to a running server), it is now possible to save the password and url in the secret provider.

There is a checkbox that allows saving the settings, and if checked, will save the settings to the secret provider (using the OS default if none is specifically provided).

This fixes #6379
2025-12-07 22:26:37 +01:00
Kenneth SkovhedeandGitHub 179fd4da47 Merge pull request #6653 from duplicati/feature/secret-provider-write-support
Initial implementation of write-support for SecretProviders
2025-12-05 15:23:32 +01:00
Kenneth Skovhede 7322c46861 Pre-execution check for RemoveRemoteVolumes
This PR adds a check before entering the RemoveRemoteVolumes that checks if there are dangling FilesetEntry records when calling the function.

This makes it possible to figure out if the problem is created by the RemoveRemoteVolumes code, or an already existing problem in the database.

Also added a few tests that check that the condition is correctly detected.
2025-12-03 11:34:30 +01:00
David Kartchner 5113d0e989 added an option to supply a new passphrase for reencryption with RecoveryTool recompress command 2025-12-02 02:24:06 -07:00
Kenneth Skovhede 695d190b91 Updated to allow IsSupported to be async 2025-12-01 15:29:24 +01:00
Kenneth Skovhede 8ac2273f21 Added OS markers to avoid accessing providers from unsupported OS.
Updated Pass provider to probe for pass via path searching.

Changed `IsSupported` to a method because it could take a while to evaluate
2025-12-01 13:21:44 +01:00
Kenneth Skovhede 6d9826df06 Initial implementation of write-support for SecretProviders
This updates the MacOS secret provider to use PInvoke for a more stable API than the secrets tool that was used before.
2025-11-30 23:01:57 +01:00
Kenneth Skovhede ef649f3926 Updated reporting to include more details
This PR updates the reports sent to include a more detailed OS version string, similar to what is used by the usagereporter and systeminfo call.

It also reports parts of the destination hostname, if using S3 and the host is a known public cloud. This is done to avoid leaking internal hostnames or actual bucket names, but still provide some way to identify what storage service is used.
2025-11-28 12:04:47 +01:00
Kenneth Skovhede e1b9015a85 Added support for exclusion of files based on xattrs
This PR adds detection of exclusion attributes on files and folders. If a file or folder has an exclusion extended attribute, the file or folder is excluded from the backup.

The option `--disable-backup-exclusion-xattr` can be used to revert to the previous behavior where the xattrs were not checked.

This has the biggest impact on MacOS where it will not perform like other backup software and avoid files that are marked as excluded from backup.

This fixes #6393
2025-11-21 10:59:28 +01:00
Kenneth Skovhede 3b695cf065 Added support for Photos on MacOS
This PR adds detection of the MacOS Photos folder, and intercepts reads and replaces them with PhotoKit calls.
With this, it is possible to make backups of all MacOS Photos, even if they are not stored locally.

The previous versions would just make a backup of the on-disk structure, which was not guaranteed to contain all photos, but instead has various indexing for finding photos, and may contain some original photos.

The option `--photos-handling` controls how Duplicati now deals with the Photos folder. The options are:
- `LibraryOnly`: Same as before, just treat it as a folder
- `PhotosOnly`: Ignore the folder contents and just back up the actual photos
- `PhotosAndLibrary` (default): Make a backup of the photos and the library on-disk. This may cause images to be stored twice, but de-duplication will usually limit the storage increase.

The option `--photos-library-path` can be used to point to the on-disk Photo library that should be handled, in case the auto-detection does not pick it up. If this does not point to a valid Photoslibrary, or the path is not being backed up, no special handling will be done.

Note that the restore is not restoring into Photos itself, but instead restores into a sub-folder in the Photolibrary that is called `dup_backup`. To get the photos out after a restore, one needs to right-click the Photolibrary folder, and choose "Show package contents" and then the `dup_backup` folder is revealed.

This is done to keep all photos in the same folder, but avoid messing with the structure of the on-disk Photolibrary.

A future update could allow restoring back into Photos, and metadata is captured for each image to eventually allow this.

This fixes #6381
2025-11-17 17:12:10 +01:00
Kenneth SkovhedeandGitHub 2632ba8164 Merge pull request #6625 from duplicati/bugfix/restore-deadlock-timer-better-start
Bugfix/Better starting time for restore deadlock detection.
2025-11-15 11:04:25 +01:00
Kenneth Skovhede 43d38efdd3 Fixed remaining warnings 2025-11-14 16:09:05 +01:00
Kenneth Skovhede 46428936c4 Various updates after the upgrade 2025-11-14 15:45:19 +01:00
Kenneth Skovhede 1ea74a3c63 Updated to .NET10 2025-11-14 15:05:39 +01:00
Carl Johnsen d48afd7496 Check the database for volume sizes reported in remotevolume 2025-11-10 14:56:20 +01:00
Carl Johnsen 77f3000e4f Reduced the timing of a block to the lifetime of the block request as it roundtrips from/to the BlockManager 2025-11-10 13:28:59 +01:00
Carl Johnsen 57e40a6290 Initial deadlock detection threshold is now set to 1 minute per 10 MB of volume size. 2025-11-10 12:54:33 +01:00
Kenneth Skovhede 012aeed7de Add dynamic streaming toggle
This PR adds a dynamic property so a backend can signal if it supports streaming, based on the settings.

This is currently used for the File backend, so that toggling `--use-move-for-put` will disable streaming on the backend instead of relying on the `--disable-streaming-transfers` flag.
2025-11-03 12:48:37 +01:00
Carl Johnsen 96921bc5fb RestoreHandler now sets up the DeadLock timer alongside the restore process network 2025-10-28 14:44:13 +01:00
Carl Johnsen dafb38443c FileProcessor id (used for debugging) started at 1, rather than 0. 2025-10-28 14:43:35 +01:00
Carl Johnsen 91009297c1 Fixed typo 2025-10-28 14:42:58 +01:00
Carl Johnsen 435792c8fa BlockManager now uses times reported by the DeadlockTimer as deadlock detection threshold instead of a hardcoded 5 minutes 2025-10-28 14:42:43 +01:00
Carl Johnsen 4e5d015e0c Added the deadlock timer task 2025-10-28 14:41:58 +01:00
Carl Johnsen ccfcffcc1e Added internal timers to keep track of maximum processing times of the restore processes 2025-10-28 14:41:39 +01:00
Kenneth Skovhede 777bc06842 Report remote filename on hash errors
This PR changes the reported filename to use the remote filename instead of the random temporary filename that has little value to the user as the file is deleted immediately after the error is raised.
2025-10-27 11:37:10 +01:00
Kenneth Skovhede 8224d84f57 Reverted to use the shorter name --asynchronous-upload-limit and then added an alias for --asynchronous-concurrent-upload-limit. 2025-10-16 13:25:34 +02:00
Kenneth Skovhede 55c19c8752 Removed more references to the unused option 2025-10-16 13:21:40 +02:00
Kenneth Skovhede d3b15f136e Removed unused option
The option `asynchronous-upload-limit` was not used, and has been removed.
2025-10-14 21:30:02 +02:00
Kenneth SkovhedeandGitHub 9507b57f1c Merge pull request #6534 from carljohnsen/performance/restore-lock-reduce
Performance/restore lock reduce
2025-10-03 11:46:00 +02:00
Carl Johnsen 91fc698314 The block manager only adds a block to the cache, if it's needed at a later point. 2025-10-03 09:50:08 +02:00
Carl Johnsen 2aa2103f33 Changed BlockManager to only use the cache if the max size is > 0 2025-10-03 09:26:38 +02:00
Carl Johnsen aafe192766 Added docstring 2025-10-03 09:26:08 +02:00
Carl Johnsen a1e1afce28 Changed the default volume cache hint to be volumesize * 100 2025-10-03 08:50:35 +02:00
Carl Johnsen 842cc00979 Added profiling stopwatch for DataBlock instantiation in volume decompressor 2025-10-03 08:32:29 +02:00
Carl Johnsen dff4b52066 Moved instantiation of the block wrapper to the volume decompressor 2025-10-03 08:32:20 +02:00
Carl Johnsen 8f056d5db9 Added profiling stopwatch for volume wrapper in the volume decryptor 2025-10-03 08:25:56 +02:00
Carl Johnsen cd4e23efc4 Moved instantiation of the volume wrapper to the volume decryptor 2025-10-03 08:25:46 +02:00
Carl Johnsen 2ad6e088ac Added another profiling stopwatch to blockmanager 2025-10-03 08:21:42 +02:00
Carl Johnsen 0c1643568e Volume cache now counts the size of the volumes, not just the count of the volumes 2025-10-03 08:11:04 +02:00
Carl Johnsen 48e5a57654 Adapted volume cache eviction to only evict on a download response 2025-10-03 07:51:24 +02:00
Carl Johnsen 42cba24bd9 Removed old decompress in flight code 2025-10-03 07:50:48 +02:00
Carl Johnsen 1f0a4617a6 The reader has to be disposed before disposing the file 2025-10-03 07:50:19 +02:00
Carl Johnsen 34cfd47ceb Added docstrings 2025-10-03 07:50:03 +02:00
Carl Johnsen 1f44b5e01b Removed old ack profiling from the block manager 2025-10-03 07:49:06 +02:00