duplicati/Duplicati/Library/Compression/TarZstdCompression

Tar-Based Compression Format

This compression module provides an ICompression implementation using Tar+Compression with an custom EOF (End-of-File) header entry for fast random access, emulating a Zip-like file format. This format relies more heavily on temporary files as the compression is done on the full tar file for best compression. This approach also means that most generated volumes will be better compressed but file size may be significantly lower that the target volume size.

Two compression variants are available:

• Tar+GZip (.tgz extension) - Uses GZip compression
• Tar+Zstd (.tzstd extension) - Uses Zstd compression (coming soon)

The archives can be decompressed with standard tar tools, and the .eof-header entry will appear as a regular file in the archive for non-Duplicati tools. In case the header is missing, the library will fall back to scanning the entire archive to find the files.

File Extensions

• .tgz - Tar+GZip format
• .tzstd - Tar+Zstd format

File Format Specification

Overall Structure

[GZip Compressed Stream]
  └─ [Tar Archive]
       ├─ [File Entry 1]
       ├─ [File Entry 2]
       ├─ ...
       └─ [.eof-header Entry]  <-- Regular tar entry with special content
            ├─ [Tar Header (512 bytes)]
            ├─ [JSON Dictionary content]
            ├─ [Padding (to make JSON+Trailer fit in 512-byte blocks)]
            ├─ [8 bytes: offset of .eof-header tar header start]
            └─ [6 bytes: "EOFHD1" magic]
            ↑
            All of this is INSIDE the entry content, properly padded

EOF Header Format

The .eof-header entry enables O(1) file lookups without scanning the entire archive. It is stored as a regular tar entry (not external to the tar), which means:

• Standard tar tools can extract the archive normally
• The .eof-header file appears as a regular file in the archive
• Standard compression tools can decompress the archive

The .eof-header entry contains:

1. Tar Header (512 bytes): Standard tar header for .eof-header file
2. Entry Content (padded to 512-byte boundary):
   • JSON Dictionary: Maps filenames to their metadata (offset, size, modification time)
   • Padding: Padding bytes to align the trailer to the 512-byte boundary
   • Header Trailer (14 bytes):
     • Offset (8 bytes): Little-endian long pointing to start of .eof-header tar header
     • Magic (6 bytes): Literal "EOFHD1" marking the end

Important: The trailer is INSIDE the tar entry content, not after it. The total entry content (JSON + padding + trailer) is padded to a multiple of 512 bytes as required by the TAR format.

JSON Dictionary Format

{
    "file1.txt": {
        "offset": 512,
        "size": 1024,
        "lastWriteTime": 1686832200
    },
    "file2.txt": {
        "offset": 2048,
        "size": 2048,
        "lastWriteTime": 1686832260
    }
}

The lastWriteTime field is stored as a Unix timestamp in seconds since epoch (January 1, 1970 UTC).

Reading Process

1. Decompress: Decompress the entire gzip stream to a temporary file
2. Read Trailer: Seek to end of file and read the last 14 bytes (offset + magic)
3. Verify Magic: Check for literal "EOFHD1" magic string at the end
4. Load Dictionary: If valid, use the offset to seek to the start of .eof-header entry and parse the JSON dictionary
5. Fallback to Scanning: If header invalid/missing, scan entire tar file to build dictionary

Note: Since .eof-header is always the last entry and its content is padded to 512 bytes, the trailer (14 bytes) is always at the very end of the file. No searching is needed - just read the last 14 bytes.

Writing Process

1. Collect Entries: Store all file entries in memory with their data
2. Write Tar Entries: Write all file entries to tar archive using System.Formats.Tar
3. Build Dictionary: Create JSON dictionary mapping filenames to their offsets
4. Write EOF Header Entry: Write .eof-header as regular tar entry containing:
   • Tar header (standard 512-byte tar header)
   • JSON dictionary content
   • Padding to 512-byte boundary
   • Trailer with offset (pointing to start of this entry's header) + magic
5. Compress: Compress the entire tar file with the desired method

Tar Format Implementation

The module uses System.Formats.Tar with the Pax format:

• Block Size: 512 bytes
• Format: Pax (POSIX.1-2001 extended format)
• Compatibility: Standard tar tools can read the archive

Compression Options

GZip Compression (.tgz)

• Implementation: System.IO.Compression.GZipStream
• Default Level: 2 (Optimal)
• Level Range: 0-3
• Level Mapping:
  • 0: NoCompression
  • 1: Fastest
  • 2: Optimal
  • 3: SmallestSize
• Option Name: tgz-compression-level

Zstd Compression (.tzstd)

• Implementation: Zstd compression
• Default Level: 10
• Level Range: 1-22
• Option Name: tzstd-compression-level

Performance Characteristics

Write Performance

• Memory Usage: Proportional to total file sizes (stores in memory before writing)
• Compression: Single-pass after all files collected
• I/O: Writes tar to temp file, then compresses to output

Read Performance

• With EOF Header: O(1) file lookup
• Without EOF Header: O(n) scan through entire archive
• Decompression: Full stream decompression required (cannot seek in gzip)

Command-Line Options

tgz-compression-level (Tar+GZip)

Sets the compression level for GZip compression.

• Type: Enumeration
• Default: 2
• Range: 0-3
• Description: Higher values provide better compression but are slower. 0 = no compression, 3 = maximum compression.

duplicati backup ... --tgz-compression-level=2

tzstd-compression-level (Tar+Zstd)

Sets the compression level for Zstd compression.

• Type: Enumeration
• Default: 3
• Range: 1-22
• Description: Higher values provide better compression but are slower

duplicati backup ... --tzstd-compression-level=9

Compatibility

With Standard Tools

The format is designed to be compatible with standard Unix tools:

Tar+GZip (.tgz):

# Decompress and extract with standard tools
gunzip -c backup.tgz | tar -xf -

# List contents
gunzip -c backup.tgz | tar -tf -

# The .eof-header file will be extracted like any other file
# It contains the JSON dictionary with file offsets

Tar+Zstd (.tzstd):

# Decompress and extract with standard tools
zstd -d -c backup.tzstd | tar -xf -

# List contents
zstd -d -c backup.tzstd | tar -tf -

# The .eof-header file will be extracted like any other file
# It contains the JSON dictionary with file offsets

Archive Structure

When extracted with standard tar, you'll see:

• All your files (file1.txt, file2.txt, etc.)
• An extra .eof-header file containing the metadata dictionary

This design ensures:

1. No data loss: Standard tools can fully extract the archive
2. Self-documenting: The .eof-header is human-readable JSON
3. Forward compatibility: Future versions can extend the format

Advantages

1. Fast Random Access: EOF header enables O(1) file lookups
2. Standard Format: Based on standard tar+gzip for universal compatibility
3. Fallback Support: Can read archives even if EOF header is corrupted
4. Self-Documenting: JSON dictionary is human-readable
5. Tool Compatible: Works with standard tar and gzip tools

Limitations

1. Full Decompression Required: Must decompress entire archive to read any file
2. Memory Usage During Write: All file data held in memory until disposal
3. No Streaming Write: Cannot stream large files without memory overhead

File Structure Diagram

┌─────────────────────────────────────────────────────────────┐
│              Compressed Stream (GZip or Zstd)               │
├─────────────────────────────────────────────────────────────┤
│                         Tar Archive                         │
│  ┌───────────────────────────────────────────────────────┐  │
│  │  Entry 1: file1.txt                                   │  │
│  │  ├─ Tar Header (512 bytes)                            │  │
│  │  └─ File Content (padded to 512 boundary)             │  │
│  ├───────────────────────────────────────────────────────┤  │
│  │  Entry 2: file2.txt                                   │  │
│  │  ├─ Tar Header (512 bytes)                            │  │
│  │  └─ File Content (padded to 512 boundary)             │  │
│  ├───────────────────────────────────────────────────────┤  │
│  │  Entry N: .eof-header (REGULAR TAR ENTRY)             │  │
│  │  ├─ Tar Header (512 bytes)                            │  │
│  │  ├─ Content (ALL within 512-byte blocks):             │  │
│  │  │  ├─ JSON Dictionary                                │  │
│  │  │  │   {"file1.txt":{"offset":512,...}               │  │
│  │  │  ├─ Padding (to align trailer)                     │  │
│  │  │  ├─ Header Offset (8 bytes) ───┐                   │  │
│  │  │  └─ "EOFHD1" Magic (6 bytes)   │                   │  │
│  │  │                                │                   │  │
│  │  └────────────────────────────────│───────────────────┘  │
│  │                                   │                      │
│  └───────────────────────────────────┘                      │
│         The offset points back to the start                 │
│         of THIS entry's tar header                          │
└─────────────────────────────────────────────────────────────┘

Implementation Notes

1. The .eof-header is a regular tar entry - it has a tar header just like any other file
2. The trailer (offset + magic) is inside the entry content, at the end, before the 512-byte padding
3. The total entry content size (JSON + trailer) is a multiple of 512 bytes as per TAR spec
4. The offset in the trailer points to the start of the .eof-header tar header
5. When using standard tar tools, the .eof-header file is extracted normally (as a file containing JSON + binary trailer)
6. The JSON dictionary excludes the .eof-header file itself
7. File paths are stored with forward slashes but support both slash types on read

Usage Examples

Creating an Archive

using var stream = File.Create("backup.tzstd");
using var archive = new FileArchiveTarZstd(
    stream,
    ArchiveMode.Write,
    new Dictionary<string, string?> { ["tzstd-compression-level"] = "9" }
);

using (var entry = archive.CreateFile("documents/file1.txt", CompressionHint.Compressible, DateTime.Now))
{
    entry.Write(data, 0, data.Length);
}

Reading an Archive

using var stream = File.OpenRead("backup.tzstd");
using var archive = new FileArchiveTarZstd(stream, ArchiveMode.Read, new Dictionary<string, string?>());

var files = archive.ListFiles(null);
foreach (var file in files)
{
    using var fileStream = archive.OpenRead(file);
    // Read file content
}

Extracting with Standard Tools

Tar+GZip (.tgz):

# The archive is valid tar+gzip
gunzip -c backup.tgz > backup.tar
tar -xf backup.tar

# You'll see:
# - documents/file1.txt
# - .eof-header (contains the JSON offset map)

Tar+Zstd (.tzstd):

# The archive is valid tar+zstd
zstd -d -c backup.tzstd > backup.tar
tar -xf backup.tar

# You'll see:
# - documents/file1.txt
# - .eof-header (contains the JSON offset map)