tuliprox/docs/src/configuration/shared-hls-configuration.md

Shared HLS Configuration

Shared HLS requires one global reverse-proxy cache block and one per-target switch.

The global block tells Tuliprox how to store and manage HLS cache data. The target switch tells Tuliprox which generated live HLS entries should use the shared HLS cache path.

Required switches

config.yml:

reverse_proxy:
  rewrite_secret: "00112233445566778899aabbccddeeff"
  hls_cache:
    cache_path: "/var/lib/tuliprox/cache/hls"

source.yml:

targets:
  - name: xc_m3u
    options:
      share_live_streams:
        hls: true
        mpeg_ts: false

Shared HLS is enabled for a generated target only when both conditions are true:

Condition	File	Required value
HLS cache engine exists	config.yml	reverse_proxy.hls_cache is present.
Target allows HLS sharing	source.yml	target.options.share_live_streams.hls: true.

If reverse_proxy.hls_cache is missing, Tuliprox does not use the shared HLS cache path even when the target option is set. If the target option is missing or false, that target does not use Shared HLS even when the global cache engine is configured.

Full example

reverse_proxy:
  rewrite_secret: "00112233445566778899aabbccddeeff"

  hls_cache:
    cache_path: "/var/lib/tuliprox/cache/hls"
    strip:
      mode: "segments"
      value: 0
    cache_duration: 300
    cache_bytes: "10GB"
    cache_bytes_per_session: "512MB"
    max_segments_prefetch: 6
    max_concurrent_segment_fetches_per_session: 2
    max_concurrent_segment_fetches_global: 64
    origin_manifest_timeout_ms: 3000
    origin_segment_timeout_ms: 10000
    session_idle_timeout: 300
    manifest_recovery_burst:
      level: "off"
    segment_repair:
      max_level: "off"
      apply_to_first_segments: 1
      max_parallel_repairs: 1
      postprocess_timeout_ms: 2000
      size_increase:
        low_percent: 2
        medium_percent: 5
        high_percent: 20
      corrupt_segment_watchdog:
        mode: "off"
        max_parallel_jobs: 1

Target options

targets:
  - name: xc_m3u
    output:
      - type: xtream
      - type: m3u
    options:
      share_live_streams:
        hls: true
        mpeg_ts: false

Option	Default	Meaning
share_live_streams.hls	false	Enables Shared HLS for live HLS entries generated by this target. Requires reverse_proxy.hls_cache.
share_live_streams.mpeg_ts	false	Enables the separate MPEG-TS live stream sharing path. This is independent from Shared HLS.

The object form is recommended. The legacy boolean remains compatible and enables both sharing modes:

share_live_streams: true

The two switches are separate because HLS and MPEG-TS sharing have different runtime behavior and resource costs.

HLS cache parameters

Parameter	Default	Recommended first value	Effect
cache_path	/tmp/tuliprox/cache/hls	Persistent writable path, for example /var/lib/tuliprox/cache/hls	Root directory for HLS segment and MAP cache objects.
strip.mode	segments	segments	Interprets strip.value as either a number of segments or a number of seconds.
strip.value	0	0	Initial holdback for the first rendered startup view. Increase only when players start too close to the live edge.
cache_duration	300	300	Retention baseline, in seconds, for cache objects that are not protected by active use.
cache_bytes	10GB	10GB or larger	Global HLS cache budget.
cache_bytes_per_session	512MB	512MB	Per shared-session HLS cache budget.
max_segments_prefetch	6	6	Maximum prefetch queue depth per shared HLS session.
max_concurrent_segment_fetches_per_session	2	2	Maximum concurrent future segment fetches for one shared HLS session.
max_concurrent_segment_fetches_global	64	64	Global cap for concurrent future segment fetches across all HLS sessions.
origin_manifest_timeout_ms	3000	3000	Timeout for upstream HLS manifest fetches.
origin_segment_timeout_ms	10000	10000	Timeout for upstream HLS segment fetches.
session_idle_timeout	300	300	Access-lease validity window and idle cleanup baseline, in seconds.
manifest_recovery_burst.level	off	off, then friendly or balanced only if needed	Optional extra recovery pressure when manifest refresh needs help.
segment_repair.max_level	off	off, then low only if needed	Maximum codec-aware MPEG-TS repair level.
segment_repair.apply_to_first_segments	1	1	Number of visible TS objects checked per lease activation. Maximum is 6.
segment_repair.max_parallel_repairs	1	1	Maximum concurrent repair jobs. Must not exceed max_segments_prefetch when repair is enabled.
segment_repair.postprocess_timeout_ms	2000	2000	Timeout for segment post-processing, including repair and watchdog work. Minimum is 100.
segment_repair.size_increase.low_percent	2	2	Allowed size increase for low-level repair output.
segment_repair.size_increase.medium_percent	5	5	Allowed size increase for medium-level repair output.
segment_repair.size_increase.high_percent	20	20	Allowed size increase for high-level repair output.
segment_repair.corrupt_segment_watchdog.mode	off	off	Optional watchdog for residual TS packet-corrupt warnings: off, detect_only, sanitize, or diagnostic.
segment_repair.corrupt_segment_watchdog.max_parallel_jobs	1	1	Maximum watchdog jobs. Must not exceed max_segments_prefetch when watchdog mode is enabled.

Cache commits enforce both byte budgets. An individual object may not exceed either budget, and committed usage is reconstructed from disk after restart. Origin manifests are limited to 2 MiB before parsing.

Supported byte-size units for cache_bytes and cache_bytes_per_session are:

• B;
• KB, MB, GB, TB as decimal units;
• KiB, MiB, GiB, TiB as binary units;
• no suffix, meaning bytes.

Choosing cache sizes

Start with the defaults unless you already know your traffic pattern.

A rough sizing approach:

required cache_bytes ≈ active_shared_channels × cache_bytes_per_session

Example: if 20 shared HLS channels can be active at the same time and each session may use up to 512MB, a global budget of 10GB is the minimum starting point. Add extra room when segments are large, when cache_duration is high, or when many users join at slightly different playback positions.

Disk cache does not remove the need for network capacity. Increasing prefetch and concurrency can make playback smoother, but it can also increase upstream and local I/O load.

Recommended profiles

Small or first-time setup

reverse_proxy:
  rewrite_secret: "00112233445566778899aabbccddeeff"
  hls_cache:
    cache_path: "/var/lib/tuliprox/cache/hls"

Use the defaults first. They are conservative and easier to debug.

More concurrent viewers

reverse_proxy:
  rewrite_secret: "00112233445566778899aabbccddeeff"
  hls_cache:
    cache_path: "/var/lib/tuliprox/cache/hls"
    cache_bytes: "50GB"
    cache_bytes_per_session: "1GB"
    max_concurrent_segment_fetches_global: 128

Use this only when the machine has enough disk throughput and upstream bandwidth. Keep the per-session fetch cap modest unless profiling shows it is the bottleneck.

Unstable manifests

reverse_proxy:
  rewrite_secret: "00112233445566778899aabbccddeeff"
  hls_cache:
    cache_path: "/var/lib/tuliprox/cache/hls"
    manifest_recovery_burst:
      level: "friendly"

Use recovery burst levels gradually. Start with friendly. Move to stronger levels only when logs show manifest refresh instability and the provider can tolerate additional requests.

Problematic MPEG-TS segments

reverse_proxy:
  rewrite_secret: "00112233445566778899aabbccddeeff"
  hls_cache:
    cache_path: "/var/lib/tuliprox/cache/hls"
    segment_repair:
      max_level: "low"
      apply_to_first_segments: 1
      max_parallel_repairs: 1

Keep segment repair disabled unless you have real evidence of broken TS segments. Repair costs CPU and can add latency.

Validation rules

Tuliprox validates HLS cache settings during configuration preparation.

Rule	Reason
cache_duration >= 1	Avoids immediate cache expiry.
max_concurrent_segment_fetches_per_session >= 1	At least one per-session fetch worker must be possible.
max_concurrent_segment_fetches_global >= 1	At least one global fetch worker must be possible.
origin_manifest_timeout_ms >= 1	Timeout must be non-zero.
origin_segment_timeout_ms >= 1	Timeout must be non-zero.
session_idle_timeout >= 1	Access leases and sessions need a positive validity window.
segment_repair.apply_to_first_segments <= 6	Limits startup repair work.
segment_repair.postprocess_timeout_ms >= 100	Avoids unrealistic post-processing timeouts.
segment_repair.size_increase.* <= 100	Prevents unbounded repaired-object growth.
repair enabled: max_parallel_repairs >= 1	Repair needs at least one worker.
repair enabled: max_parallel_repairs <= max_segments_prefetch	Repair concurrency must not exceed the session fetch window.
watchdog enabled: max_parallel_jobs <= max_segments_prefetch	Watchdog concurrency must stay within the session fetch window.

Interaction with older HLS settings

reverse_proxy.stream.hls_session_ttl_secs belongs to the older HLS request continuity path. It keeps a virtual provider slot open between legacy HLS segment requests.

Shared HLS uses reverse_proxy.hls_cache.session_idle_timeout for access-lease validity and idle cleanup. Do not tune hls_session_ttl_secs when the problem is Shared HLS lease expiry.

Hot reload and restarts

Configuration reloads can update HLS runtime settings. Changing the HLS cache path is a disruptive change: Tuliprox must clear in-memory HLS sessions, access leases, QoS state, and repair state that point at the old location.

Changing rewrite_secret is also disruptive for public HLS identifiers. Existing player URLs may reference old proxy_session_id or transient resource IDs and should be treated as stale.

Optional startup modes

reverse_proxy.hls_cache.startup.mode defaults to conservative. Omitted startup configuration keeps the existing READY reserve, cache delivery and repair behavior. first_ready lowers initial admission to one processed start segment. Every listed successor must already have a successful HTTP 200 response and a nonempty body prefix.

reverse_proxy:
  hls_cache:
    startup:
      mode: first_ready
      max_progressive_segments: 16
      max_progressive_bytes_per_segment: 32MB
      max_progressive_bytes_total: 128MB
      max_progressive_reader_lifetime_secs: 90
      max_deferred_repairs: 32

progressive lets the first full segment request consume the existing shared origin fill before origin EOF. The complete session delivers immutable raw TS revisions, including subsequent requests, retries and ranges. Repair and the corruption watchdog continue on the processed cache copy. Raw delivery does not correct source timestamp or transport-stream defects; prefer first_ready when a source needs repair.

optimistic behaves like progressive and publishes the initial playlist earlier. The head segment needs a body prefix, while listed successors only need an accepted HTTP 200 origin response. Successor fetches start together with the head, within the configured per-session concurrency. A client request for a successor waits for its complete body. If the origin fails a successor after it was listed, only that segment fails for the client; already published media never changes. Replay or disk fallback switches the session to first_ready exactly like progressive.

Fast startup is limited to live, clear MPEG-TS objects without MAP or origin byte ranges. Archive, encrypted, fragmented-MP4 and transient sessions use their existing path. Before publication an incompatible source can fall back to that path. After a Fast Start manifest is published, a refresh requiring transient delivery is rejected to preserve the bound revisions; it cannot silently change the active session representation. A progressive Range: bytes=0- request may receive a complete HTTP 200 streaming response; other ranges wait for the bound revision to complete.

The replay limits apply across sessions and remain shared during configuration reload. Byte clones stay charged until their last reference is released. An absolute reader lifetime prevents a stalled client from retaining a revision indefinitely. An origin drop or replay overflow after streaming begins produces a body error; it cannot turn into successful EOF or silently change the bytes of an already published URI. Revision and repair peak reservations share the existing disk cache limits. Conservative behavior remains the default; existing sessions keep their startup policy after reload. Restart after changing to conservative when an immediate rollback of running sessions is required.

Existing provider and fetch concurrency limits remain in effect. The initial playlist needs at least three publishable segments; optimistic counts successors the origin has accepted. With fewer than three concurrent fills, a cold session can still wait for an earlier origin EOF before that window is available. Raising the existing per-session concurrency limit to three is an operator choice and must fit the provider's connection allowance; startup mode never raises it automatically.

Measure both first data and decoded-picture time before enabling progressive mode in production. Short successful response runs do not establish long-term player, origin-failure or resource stability.

Reproducible startup checks

The testkit has isolated shared-HLS scenarios for conservative, first_ready, progressive and optimistic, plus missing-head scenarios for every fast mode. Existing scenarios retain their original origin behavior. The new origin.hls_profile publishes six segments, sends a complete synthetic frame immediately, and delays the second frame by prefix_delay_millis. missing_head_segments returns alternating HTTP 404/410 at the head; abort_segments injects a body error after the prefix. unknown_length omits origin Content-Length. These frames test HTTP delivery and cache identity, not MPEG-TS decoding or picture startup.

cargo +stable build --package tuliprox --package tuliprox-testkit
export TULIPROX_TESTKIT_SUT_BINARY="$PWD/target/debug/tuliprox"
cargo +stable test --package tuliprox-testkit --test hls_fast_start_process -- --ignored --test-threads=1

target/debug/tuliprox-testkit controller \
  --scenario test/fixtures/testkit/scenarios/m3u-hls-share-progressive-startup.yml \
  --report-directory /tmp/hls-progressive-report

The process tests check prefix delivery before origin EOF, processed delivery after EOF, 404/410 recovery, terminal errors for both known-length and chunked origin drops, byte-identical retries and HTTP 206 ranges. They also check that successful retries and ranges do not create another origin segment download. The fixed local timing thresholds distinguish the intentionally delayed fixture responses; they are not production SLAs.

For a synthetic HLS playback URL, hls-probe emits first-byte/body-end timing and revision checks as JSON:

target/debug/tuliprox-testkit hls-probe \
  --url "$TESTKIT_HLS_URL" --run-id "$TESTKIT_RUN_ID" --expected-marker 17 \
  --verify-revision --min-prefix-to-eof-millis 500

Use --expect-body-error instead of --verify-revision for an injected segment-body drop. The probe keeps one absolute 30-second deadline and a 1 MiB response bound. It does not print the playback URL in its JSON result.