haeaeae/duplicati
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
14,132 Commits 140 Branches 416 Tags
dcaba2fbcd6267e84c1f48cc09bbbe5f20134a7e
Commit Graph
69 Commits
Author SHA1 Message Date
Kenneth SkovhedeandCopilot 3ec07aaee1 Added zstd compression 
This PR adds experimental support for using either ZStandard or GZip compression. Unlike the Zip module, which also supports these methods, the new module does a full-volume compression which is both much faster and compresses better with modern compression algorithms.

Because the compressor can see the full volume (and not just the blocks) it can compress data across blocks.

The downside to this is that it is not possible to take a single entry out of the compressed stream, but instead, Duplicati needs to decompress the whole stream and can then access the contents.

To make the implementation compatible with standard tools, the inner format is Tar (using ustar format). To allow faster reading, a small end-of-file header is added to the Tar file, emulating the concept from Zip files.

This small addition allows Duplicati to have random access to files in a Tar volume without needing to scan the whole thing.

To make sure data is always recoverable, the format is 100% compatible with regular tools, so `untar -xf` will work on the created volumes.

The downside is an extra pass for decompression when reading the volumes. When writing, each entry (a block of data most commonly) is written to a temporary file before being added to the output stream. It is possible to toggle this to use a memory buffer for even more speed up.

The ZStandard compression is from `ZstdSharp.Port` which has excellent performance.

When we move to .NET11 this can easily be changed to the new built-in module.

Since this is the first attempt to add this, it will log a warning on each use, explaining that the feature is currently just for testing.

Co-authored-by: Copilot <copilot@github.com>
 2026-04-30 13:03:24 +02:00
Kenneth SkovhedeandGitHub c54a278050 Merge pull request #6844 from duplicati/feature/copyright-bump-2026 
Updated copyright year to 2026 across the project
 2026-04-17 12:34:06 +02:00
Kenneth Skovhede c0f40e19ca Updated copyright year to 2026 across the project 2026-04-16 15:21:24 +02:00
Kenneth Skovhede 3faa674906 Update SharpCompress 
This PR updates SharpCompress to latest version.
Also adds support for choosing ZStandard as a compression method.

This fixes #4333
 2026-04-06 16:53:05 +02:00
Kenneth Skovhede 1ea74a3c63 Updated to .NET10 2025-11-14 15:05:39 +01:00
Kenneth Skovhede 819af9e66f Update .NET dependencies 2025-06-24 07:51:26 +02:00
Carl Johnsen 04153d2c12 Removed Microsoft.DotNet.Analyzers.Compatibility from csproj's as it's deprecated. 2025-05-16 09:53:26 +02:00
Kenneth Skovhede cd3475791c Updated SharpCompress 2025-03-31 13:28:26 +02:00
Kenneth SkovhedeandGitHub 997891525a Updated license-updater to update csproj files (#5898) 
* Updated license-updater to update csproj files

* License updates
 2025-01-22 21:04:19 +01:00
Kenneth Skovhede 3f3144218f Removed 7z from the source as it has been marked experimental for 7 years. 
This fixes #5630
 2024-11-01 17:40:09 +01:00
Grzegorz Łagocki 9b19bac7da Add Copyright Property to *.csproj files 2024-07-09 00:37:18 +02:00
Kenneth Skovhede 8e25c6e2ab Reduced dynamic scanning with static linking. 
Cleaned up some `proj` files.
Added caching to options.
 2024-04-26 14:32:41 +02:00
Kenneth Skovhede 473c6cbbb8 Merge commit '4f577c65a4d8806f79637c50c21ca3a683c5a07a' into feature/kestrel-avalonia-upgrade 2024-03-04 12:21:53 +01:00
Kenneth Skovhede f395da68d7 Hacky support for 7z 2024-03-01 14:30:28 +01:00
Kenneth Skovhede aa20088c39 Tool based upgrade of all projects with some manual update 2024-03-01 14:29:54 +01:00
Thomas Suckow a3bbf63561 Merge remote-tracking branch 'origin/feature/upgrade-to-sdk-style-projects' into feature/net5-kestrel 
# Conflicts:
#	.appveyor.yml
#	.github/workflows/tests.yml
#	BuildTools/AutoUpdateBuilder/AutoUpdateBuilder.csproj
#	BuildTools/DependencyFinder/DependencyFinder.csproj
#	BuildTools/GnupgSigningTool/GnupgSigningTool.csproj
#	BuildTools/UpdateVersionStamp/UpdateVersionStamp.csproj
#	Duplicati CommandLine Only.sln
#	Duplicati.Library.RestAPI/RESTMethods/RequestInfo.cs
#	Duplicati.Library.RestAPI/Runner.cs
#	Duplicati.sln
#	Duplicati/CommandLine/BackendTester/Duplicati.CommandLine.BackendTester.csproj
#	Duplicati/CommandLine/BackendTester/app.config
#	Duplicati/CommandLine/BackendTool/Duplicati.CommandLine.BackendTool.csproj
#	Duplicati/CommandLine/BackendTool/app.config
#	Duplicati/CommandLine/ConfigurationImporter/Duplicati.CommandLine.ConfigurationImporter.csproj
#	Duplicati/CommandLine/Duplicati.CommandLine.csproj
#	Duplicati/CommandLine/RecoveryTool/Duplicati.CommandLine.RecoveryTool.csproj
#	Duplicati/CommandLine/app.config
#	Duplicati/GUI/Duplicati.GUI.TrayIcon/Assets/icons/normal-warning.png
#	Duplicati/GUI/Duplicati.GUI.TrayIcon/Duplicati.GUI.TrayIcon.csproj
#	Duplicati/GUI/Duplicati.GUI.TrayIcon/Program.cs
#	Duplicati/GUI/Duplicati.GUI.TrayIcon/app.config
#	Duplicati/Library/AutoUpdater/Duplicati.Library.AutoUpdater.csproj
#	Duplicati/Library/Backend/AlternativeFTP/Duplicati.Library.Backend.AlternativeFTP.csproj
#	Duplicati/Library/Backend/AzureBlob/Duplicati.Library.Backend.AzureBlob.csproj
#	Duplicati/Library/Backend/Backblaze/Duplicati.Library.Backend.Backblaze.csproj
#	Duplicati/Library/Backend/Box/Duplicati.Library.Backend.Box.csproj
#	Duplicati/Library/Backend/CloudFiles/Duplicati.Library.Backend.CloudFiles.csproj
#	Duplicati/Library/Backend/Dropbox/Duplicati.Library.Backend.Dropbox.csproj
#	Duplicati/Library/Backend/FTP/Duplicati.Library.Backend.FTP.csproj
#	Duplicati/Library/Backend/File/Duplicati.Library.Backend.File.csproj
#	Duplicati/Library/Backend/GoogleServices/Duplicati.Library.Backend.GoogleServices.csproj
#	Duplicati/Library/Backend/HubiC/Duplicati.Library.Backend.HubiC.csproj
#	Duplicati/Library/Backend/Idrivee2/Duplicati.Library.Backend.Idrivee2.csproj
#	Duplicati/Library/Backend/Jottacloud/Duplicati.Library.Backend.Jottacloud.csproj
#	Duplicati/Library/Backend/Mega/Duplicati.Library.Backend.Mega.csproj
#	Duplicati/Library/Backend/OAuthHelper/Duplicati.Library.OAuthHelper.csproj
#	Duplicati/Library/Backend/OneDrive/Duplicati.Library.Backend.OneDrive.csproj
#	Duplicati/Library/Backend/OpenStack/Duplicati.Library.Backend.OpenStack.csproj
#	Duplicati/Library/Backend/Rclone/Duplicati.Library.Backend.Rclone.csproj
#	Duplicati/Library/Backend/S3/Duplicati.Library.Backend.S3.csproj
#	Duplicati/Library/Backend/SSHv2/Duplicati.Library.Backend.SSHv2.csproj
#	Duplicati/Library/Backend/SharePoint/Duplicati.Library.Backend.SharePoint.csproj
#	Duplicati/Library/Backend/Sia/Duplicati.Library.Backend.Sia.csproj
#	Duplicati/Library/Backend/Storj/Duplicati.Library.Backend.Storj.csproj
#	Duplicati/Library/Backend/TahoeLAFS/Duplicati.Library.Backend.TahoeLAFS.csproj
#	Duplicati/Library/Backend/Tardigrade/Duplicati.Library.Backend.Tardigrade.csproj
#	Duplicati/Library/Backend/TencentCOS/Duplicati.Library.Backend.TencentCOS.csproj
#	Duplicati/Library/Backend/WEBDAV/Duplicati.Library.Backend.WEBDAV.csproj
#	Duplicati/Library/Common/Duplicati.Library.Common.csproj
#	Duplicati/Library/Compression/Duplicati.Library.Compression.csproj
#	Duplicati/Library/DynamicLoader/Duplicati.Library.DynamicLoader.csproj
#	Duplicati/Library/Encryption/Duplicati.Library.Encryption.csproj
#	Duplicati/Library/Interface/Duplicati.Library.Interface.csproj
#	Duplicati/Library/Localization/Duplicati.Library.Localization.csproj
#	Duplicati/Library/Logging/Duplicati.Library.Logging.csproj
#	Duplicati/Library/Main/Duplicati.Library.Main.csproj
#	Duplicati/Library/Modules/Builtin/CheckMonoSSL.cs
#	Duplicati/Library/Modules/Builtin/Duplicati.Library.Modules.Builtin.csproj
#	Duplicati/Library/Modules/Builtin/SendJabberMessage.cs
#	Duplicati/Library/SQLiteHelper/Duplicati.Library.SQLiteHelper.csproj
#	Duplicati/Library/Snapshots/Duplicati.Library.Snapshots.csproj
#	Duplicati/Library/UsageReporter/Duplicati.Library.UsageReporter.csproj
#	Duplicati/Library/Utility/Duplicati.Library.Utility.csproj
#	Duplicati/Library/Utility/HashAlgorithmHelper.cs
#	Duplicati/Library/Utility/Power/WindowsPowerSupplyState.cs
#	Duplicati/License/Duplicati.License.csproj
#	Duplicati/Server/Duplicati.Server.Serialization/Duplicati.Server.Serialization.csproj
#	Duplicati/Server/Duplicati.Server.csproj
#	Duplicati/Server/app.config
#	Duplicati/Service/Duplicati.Service.csproj
#	Duplicati/Service/app.config
#	Duplicati/Tools/Duplicati.Tools.csproj
#	Duplicati/UnitTest/Duplicati.UnitTest.csproj
#	Duplicati/UnitTest/GeneralBlackBoxTesting.cs
#	Duplicati/UnitTest/app.config
#	Duplicati/WindowsService/WindowsService.csproj
#	Installer/debian/docker/debian/control
#	Tools/SQLiteTool/SQLiteTool.csproj
#	Tools/ZipFileDebugger/ZipFileDebugger.csproj
#	guiTests/guiTest.py
#	thirdparty/UnixSupport/UnixSupport.csproj
#	thirdparty/alphavss/Homepage.txt
#	thirdparty/alphavss/License.txt
#	thirdparty/alphavss/licensedata.json
 2024-02-29 21:55:17 -08:00
Kenneth Skovhede 22acb8bdda Fixed compile issues after upgrade 2024-02-26 06:59:30 +01:00
Kenneth Skovhede d7b6dd90be Updated all projects to use SDK-style projects using upgrade-assist 2024-02-25 16:19:25 +01:00
Thomas Suckow 8c518f7d3a .net6 2022-04-12 21:43:50 -07:00
Kenneth Skovhede b550e61d9b Re-added compatibility analyzers 2021-04-03 21:49:51 +02:00
Kenneth Skovhede 557415c9ad Changed implementation to target netstandard2.0. 
Added builds for .Net 4.8
 2021-04-03 14:13:32 +02:00
Kenneth Skovhede 8892ff7df3 Cleaned up all .csproj files. 
Moved things around to make cleaner build files.
Implementing everything with `netstandard2.1` and then thin wrapper executables using .net5
 2021-04-03 13:57:02 +02:00
Kenneth Skovhede 63d8ab7c11 Changed all non-entry methods to use netstandard2.1 2021-04-01 22:37:26 +02:00
Kenneth Hsu e964072690 Target .NET Framework 4.7.1. 
This updates all projects to target .NET Framework 4.7.1.  The
TencentCOS and Tardigrade backends depend on .NET Standard 2.0.  When a
.NET Framework prior to 4.7.1 is targeted, the system cannot be sure
that all the dependencies exist, so it copies all dependent assemblies
to the output directory.  This causes many assemblies from the System
namespace to become bundled in the release.

https://stackoverflow.com/a/48875007

We had previously attempted to make individual projects target 4.7.1
(see pull request #4242), but this can cause compatibility issues when
4.6.2 projects depend on 4.7.1. projects.

This will require Mono 5.10.0 or greater (previously, we required 5.0.0
or greater).

https://www.mono-project.com/docs/about-mono/releases/5.10.0/#class-libraries

This fixes issue #4234.
 2020-07-26 19:46:01 -07:00
Max b008aed1f3 migrate to net5 + runnable 2020-06-03 00:00:38 +02:00
Max 7e2218663e Add Compat Analyzers 2019-12-25 19:08:52 +01:00
Max 28793ef204 packages update 2019-12-25 18:21:24 +01:00
Max dd0b6add42 nuget update 2019-09-05 15:43:59 +02:00
BlueBlock 28074a4d4e update nuget packages 
includes update to FluentFTP
 2019-08-31 16:42:12 -04:00
BlueBlock 815e30b63b update csproj toolverion and nuget packages 2019-07-31 13:20:03 -04:00
BlueBlock 08a1e9998a initial upgrade to framework 4.6.2 
- no code changes except those noted below
- projects upgrade to 4.6.2
- wixinstaller project upgraded automatically by VisualStudio
- wixinstaller updated to require 4.6.2
- Library.Encryption changed to Standard2.0 so accommodate update to SharpAesCrypt
 2019-07-26 09:18:16 -04:00
Michal Naiman ad2105aeb2 NuGet packages update 2019-05-01 09:28:46 +02:00
verhoek 72b1f6b6f8 Updated project files. 2018-11-02 22:45:07 +01:00
verhoek b2cc18426c Renamed Library.IO to Library.Common.IO. 
Moved basic Platform functions to Library.Common.Platform.
Turned IO_OS into property within Library.Common.
 2018-11-02 21:34:07 +01:00
verhoek bc053df294 Moved basic IO helper functions to Library.IO. 2018-10-27 12:17:07 +02:00
Max 30fab22340 unsinged whole solution 2018-09-09 14:21:55 +02:00
Max 795b5c34dc Main 2018-06-30 01:18:59 +02:00
Max f5672f69d9 upgrade .net to 4.6.2 2018-03-25 22:00:45 +02:00
Max d3d494b574 backends ported 2018-03-24 17:48:26 +01:00
Kenneth Skovhede 5f4aef8bbf Update sharpcompress 2017-09-23 00:28:25 +02:00
Kenneth Skovhede a5df2001ec Attempting to fix a build error on Travis 2017-08-04 14:12:18 +02:00
Kenneth Skovhede b4f9d8223e Updated SharpCompress to the latest version 2017-08-04 14:03:37 +02:00
Kenneth Skovhede ced3caa219 Changed back to SharpCompress as it is now updated with support for zip64 2017-06-01 10:49:34 +02:00
Kenneth Skovhede 360a90b2ce Updated SharpCompress to a build from master to include the zip64 fixes 2017-04-04 17:11:01 +02:00
Kenneth Skovhede 680ee82189 Updated SharpCompress to the latest version 2017-02-27 22:21:01 +01:00
Kenneth Skovhede 52eb8e8478 Disabled MSBuild for much faster builds on MacOS 2016-12-29 23:12:41 +01:00
Kenneth Skovhede 3005594b38 Merge branch 'master' into nuget_setup 
# Conflicts:
#	Duplicati/GUI/Duplicati.GUI.TrayIcon/Duplicati.GUI.TrayIcon.csproj
#	Duplicati/Library/Backend/S3/Duplicati.Library.Backend.S3.csproj
#	Duplicati/Library/Backend/S3/S3Wrapper.cs
#	thirdparty/CoCoL/CoCoL.dll
#	thirdparty/SSH.NET/Renci.SshNet.dll
 2016-06-27 15:25:52 +02:00
Kenneth Skovhede 785c4d325f Reverted changes to solution file so it again forces Windows linefeeds. 
This also moved stuff around in some project files.
 2016-04-13 00:36:22 +02:00
Kenneth Skovhede 59cc0005a3 Removed unused references and updated most libraries to come from NuGet sources instead of being provided in the thirdparty library 2016-01-29 10:47:39 +01:00
Kenneth Skovhede df733acc7a Updates to the project files 2015-12-05 11:28:01 +01:00