Add a synchronization point after recreating missing index files to
ensure all uploads are settled and committed to the database before
the backup process continues. This prevents a race condition where
the backup could start while index files are still being uploaded.
In this case, it would be possible for the backup to reference the file that is currently being deleted. This would be very rare in production as the delays have to align and the new data must be changed and processed very quickly to trigger the issue.
In unittests we have observed the issue for a while but only showing up very rarely.
This PR adds guards to prevent creating filesets with multiple files that have the same path.
While it should technically be impossible to have multiple entries that have the same path, it could happend either due to glitches or because the source data (manual lists, remote sources, etc) returns duplicates.
This PR adds a simple check for each folder to ensure that on a folder-level, duplicate paths cannot be introduced.
There is also a post-backup check to evict any duplicates, and the recreate process will reject duplicate paths.
Finally, the repair command will remove duplicates if they somehow manage to get into the database anyway.
A database-level prevention is not currently feasible as it needs a cross-table check for uniqueness, which requires more work from the database to check for each added file. Since this is expected to be a very rare event, the added processing was not justified.
Renamed the options to be more descriptive.
Rewrote the dependency query to be more readable and also check for metadata dependency.
Fixed an edge case where there were no modified files, and no new backup is created.
Fixed some issues with local/utc datetime compares.
Fixed logic for deleting filesets to only look at the filesets themselves. This makes the version lock follow the dlist file, and can create "dangling" volumes that have no filesets, but these will be picked up by compaction.
Updated tests to work correctly.
Added lock-info update option for the backup recreate call, so the UI can also re-create lock information on database rebuilds.
This PR adds support for locking files if the backend supports it.
To activate locking, set the option `--file-lock-duration=30D` and the backup will lock the files.
If the database is rebuilt with the intention of continuing the backups, use the option `--repair-refresh-lock-info` which will update lock information in the database after recreating the database.
The locking works by asking the backend to lock files after a backup has completed.
The implementation keeps track of which files are currently assigned a lock and prevents attempting to delete the files that are currently locked.
Note that the bucket should not have a default lock policy as Duplicati needs to finish the backup before the locking is applied.
In this initial version, Azure Blob Storage, B2, S3 and iDrive are supported with locking.
The CLI is updated to allow setting locks on a specific version. The backend tool is updated to allow setting locks on specific files.
This PR adds detection of exclusion attributes on files and folders. If a file or folder has an exclusion extended attribute, the file or folder is excluded from the backup.
The option `--disable-backup-exclusion-xattr` can be used to revert to the previous behavior where the xattrs were not checked.
This has the biggest impact on MacOS where it will not perform like other backup software and avoid files that are marked as excluded from backup.
This fixes#6393
This PR adds detection of the MacOS Photos folder, and intercepts reads and replaces them with PhotoKit calls.
With this, it is possible to make backups of all MacOS Photos, even if they are not stored locally.
The previous versions would just make a backup of the on-disk structure, which was not guaranteed to contain all photos, but instead has various indexing for finding photos, and may contain some original photos.
The option `--photos-handling` controls how Duplicati now deals with the Photos folder. The options are:
- `LibraryOnly`: Same as before, just treat it as a folder
- `PhotosOnly`: Ignore the folder contents and just back up the actual photos
- `PhotosAndLibrary` (default): Make a backup of the photos and the library on-disk. This may cause images to be stored twice, but de-duplication will usually limit the storage increase.
The option `--photos-library-path` can be used to point to the on-disk Photo library that should be handled, in case the auto-detection does not pick it up. If this does not point to a valid Photoslibrary, or the path is not being backed up, no special handling will be done.
Note that the restore is not restoring into Photos itself, but instead restores into a sub-folder in the Photolibrary that is called `dup_backup`. To get the photos out after a restore, one needs to right-click the Photolibrary folder, and choose "Show package contents" and then the `dup_backup` folder is revealed.
This is done to keep all photos in the same folder, but avoid messing with the structure of the on-disk Photolibrary.
A future update could allow restoring back into Photos, and metadata is captured for each image to eventually allow this.
This fixes#6381
This PR adds the ability to subscribe to messages over a websocket.
To prevent polling data, the server can now push messages through the websocket, so the client can instantly update when new data is available.
The change is backwards compatible, still serving the status updates over the socket. If the client is providing the authentication token, it is auto-subscried to the legacy status message.
If the client is using the new authentication message, it needs to subscribe to get the status updates (and any other services it needs).
The event system has been extended to support new, and more accurate, events. This is the first step towards removing the general status update and the long-poll mechanism.
This also re-introduces the blocking marker, so the client can know if the operation is halted due to too many pending transfers.
Some endpoints have been moved to service implementations, to allow serving the exact same data from both endpoints and websocket.
This PR also updates the way the progress is handled, so that all transfers are returned to the client, and the transfer speeds for each transfer is calculated based on a small sample buffer, so the values are more accurate even if the program is paused during transfers.
This PR adds the option to suppress warnings by their log id. Warning IDs that are supplied to `--suppress-warnings` will be converted to information messages before being logged.
If a specific warning is disabled, such as `CompressionReadErrorFallback`, this will then no longer count as a warning for the job, and the log file will see the warning as an information message.
This PR also adds two simpler filter options that makes it possible to filter log messages by supplying the log IDs. This can already be achieved with log filters, but the ID filter is a bit simpler to apply, as you only need to know the ID.
Finally, this PR also adds common logging for errors in the categories:
- Permission denied, id = `PermissionDenied`
- File locked, id = `FileLocked`
- Path not found, id = `PathNotFound`
- Path too long, id = `PathTooLong`
These new log ids makes it simpler to ignore warnings about locked or inaccesible files.
This ensures that files cannot appear on the remote destination, despite terminating the host program.
This fixes#4485
There is still work needed to handle the repair and compact operations so they use transactions and can commit changes during transfer progress.
This moves the throttle support to the StreamUtil throttled stream, which allows multiple streams to share a throttle value, such that the throttle setting is applied to the entire communication from Duplicati, and not for each connection.
This also cleans up a bit in the progress reporting, where the backend manager now keeps track of the "active" transfer and only emits progress on that. Once it finishes, a new transfer is promoted a "active".
If the user selects the blocksize to be the same as the remote volume size, this will cause each block to be in a separate volume and will trigger repeated compact calls.
This commit adds a check that ensures the volume size is large enough to hold at least two blocks.
Also fixed an issue where the compact could create volumes larger than the set limit (at most one block larger).
This fixes the restore issues as all sources will now appear as if they were originally part of the source file system.
The backend implementation has now been simplified so the backends only return names of entries with no path information. Most backends already do this, but the S3 backends did not, so this was fixed as well.
This also clears up the interfaces, so the backend does not know about the `ISourceProviderEntry`. All mapping is done by the `BackendSourceProvider`, making it simpler to support other backends as sources.
Carl Johnsen