# 📖 Metadata Update & FFprobe (`metadata_update`) This chapter covers the `metadata_update` block inside `config.yml`, which determines how aggressively or gently Tuliprox manages background tasks for resolving metadata and technical stream properties. Tuliprox utilizes three distinct mechanisms to ensure perfect library quality (especially for Plex/Jellyfin compatibility): 1. **Resolve (Xtream API):** Fetching missing VOD details (Cast, Director, Plot) directly from the provider's API. 2. **TMDB:** Supplementing missing Release Years and high-resolution Covers/Backdrops via The Movie Database. 3. **Probe (FFprobe):** Physically opening the stream to analyze the exact A/V codecs (HEVC, H264) and resolution. ## Top-level entries ```yaml metadata_update: cache_path: metadata retry_delay: 2s worker_idle_timeout: 1m max_queue_size: 100000 no_change_cache_ttl_secs: 3600 probe_fairness_resolve_burst: 200 log: resolve: probe: tmdb: ffprobe: ``` ### Global Options (Flat Keys) | Parameter | Type | Default | Technical Impact & Background | |:-------------------------------|:---------|:-------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `cache_path` | String | `"metadata"` | Directory where TMDB cache files and metadata are stored. Relative paths are resolved against `storage_dir`. Used by all metadata resolution paths (Xtream VOD/Series, local library). | | `retry_delay` | Duration | `"2s"` | General minimum wait time when the worker encounters temporary runtime errors (e.g. socket timeout). Prevents very fast retry loops in case of transient network issues. | | `worker_idle_timeout` | Duration | `"1m"` | Time of inactivity (empty queue) after which the background worker kills itself to free RAM/CPU resources. | | `max_queue_size` | Int | `100000` | RAM Safety Limit: Maximum number of metadata tasks kept in memory per input simultaneously. New tasks are rejected once this limit is reached. | | `no_change_cache_ttl_secs` | Int | `3600` | How long (seconds) a "No Change" status is cached to avoid unnecessary DB checks across subsequent playlist updates. Identical reason sets are skipped while valid. | | `probe_fairness_resolve_burst` | Int | `200` | After 200 consecutive Resolve tasks (API), 1 Probe task (FFprobe) is forcibly prioritized so probes don't starve in large libraries. | > **Note on Durations:** Duration fields support `s`, `m`, `h`, `d` or plain seconds (e.g. `30s`, `10m`, `1h`, `7d`). > Exception: `ffprobe.analyze_duration` and `ffprobe.live_analyze_duration` require explicit unit suffix. > Size fields support `B`, `KB`, `MB`, `GB`, `TB` or plain bytes. --- ## 1. Logging (`log`) Controls background worker logging verbosity to monitor the metadata queue status. ```yaml metadata_update: log: queue_interval: 30s progress_interval: 15s ``` | Parameter | Type | Default | Description | |:--------------------|:---------|:--------|:--------------------------------------------------------------------------------------------| | `queue_interval` | Duration | `"30s"` | Interval to log the current queue size and pending task status of the metadata worker. | | `progress_interval` | Duration | `"15s"` | Interval for progress reports while tasks are being processed (successful/failed resolves). | --- ## 2. API Resolve Limits (`resolve`) Controls API requests for pure metadata (Xtream Info / TMDB). ```yaml metadata_update: resolve: max_retry_backoff: 1h min_retry_base: 5s max_attempts: 3 exhaustion_reset_gap: 1h ``` | Parameter | Type | Default | Description | |:-----------------------|:---------|:--------|:-------------------------------------------------------------------------------------------| | `max_retry_backoff` | Duration | `"1h"` | Upper limit for exponential wait time between repeated API failures. | | `min_retry_base` | Duration | `"5s"` | Initial wait time on the very first failure before exponential backoff kicks in. | | `max_attempts` | Int (u8) | `3` | Max attempts per cycle before a resolve task is marked as "exhausted" for the current run. | | `exhaustion_reset_gap` | Duration | `"1h"` | Time window after a cycle completes before "exhausted" states are reset for the next run. | --- ## 3. FFprobe Retries & Limits (`probe`) Controls technical stream probing retries via FFprobe. ```yaml metadata_update: probe: cooldown: 7d retry_load_retry_delay: 1m retry_backoff_step_1: 10m retry_backoff_step_2: 30m retry_backoff_step_3: 1h max_attempts: 3 backoff_jitter_percent: 20 user_priority: 127 ``` | Parameter | Type | Default | Description | |:-------------------------|:---------|:--------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `cooldown` | Duration | `"7d"` | Hard lock time (cooldown) after probe attempts are exhausted. The stream is ignored for this period to protect the provider. | | `retry_load_retry_delay` | Duration | `"1m"` | Wait time before re-attempting to load the internal `metadata_retry_state.db` after a load failure. | | `retry_backoff_step_1` | Duration | `"10m"` | Wait time after the 1st FFprobe failure. | | `retry_backoff_step_2` | Duration | `"30m"` | Wait time after the 2nd FFprobe failure. | | `retry_backoff_step_3` | Duration | `"1h"` | Wait time from the 3rd FFprobe failure onwards. | | `max_attempts` | Int (u8) | `3` | Max failures to probe a stream before it enters global long-term cooldown. | | `backoff_jitter_percent` | Int (u8) | `20` | Random time deviation in percent (Jitter) so parallel retries don't hit the provider at the same second. | | `user_priority` | Int (i8) | `127` | Priority of the probe task on the Unix Nice-Scale. `127` is the absolute lowest. Probes at 127 are immediately cancelled/preempted if a real user needs the slot. | --- ## 4. TMDB API Integration (`tmdb`) Controls how Tuliprox interacts with The Movie Database API. ```yaml metadata_update: tmdb: enabled: false api_key: "YOUR_KEY" rate_limit_ms: 250 cache_duration_days: 30 language: en-US cooldown: 7d match_threshold: 86 ``` | Parameter | Type | Default | Description | |:----------------------|:----------|:-------------|:-----------------------------------------------------------------------------------------------| | `enabled` | Bool | `false` | Global master switch for TMDB resolution. | | `api_key` | String | *(Internal)* | Your own TMDB API Key. If omitted, Tuliprox uses a built-in default placeholder. | | `rate_limit_ms` | Int (u64) | `250` | Minimum wait between TMDB API calls to prevent IP rate-limiting or bans (milliseconds). | | `cache_duration_days` | Int (u32) | `30` | How long successful TMDB results are kept in the cache. Use `0` for permanent caching. | | `language` | String | `"en-US"` | Preferred metadata language (e.g., `"de-DE"`) for plot and titles. | | `cooldown` | Duration | `"7d"` | Lock time for a movie if the TMDB search was successful but returned "no match" for the title. | | `match_threshold` | Int (u16) | `86` | Minimum similarity score (Jaro-Winkler) for a result to be accepted as a valid "Match". | --- ## 5. FFprobe Process Rules (`ffprobe`) Detailed settings for the technical stream analysis process. ```yaml metadata_update: ffprobe: enabled: true timeout: 60 analyze_duration: 10s probe_size: 10MB live_analyze_duration: 5s live_probe_size: 5MB ``` | Parameter | Type | Default | Description | |:------------------------|:----------|:---------|:-------------------------------------------------------------------------------------------------------------------| | `enabled` | Bool | `false` | Global master switch for ALL stream probing. Must be `true` for input flags like `probe_vod` to work. | | `timeout` | Int (u64) | `60` | Hard timeout (in seconds) for the OS FFprobe process. Prevents zombie processes. | | `analyze_duration` | Duration | `"10s"` | Passes `-analyzeduration` to FFprobe for VODs/Series. *Warning: Requires unit suffix (`s`, `m`)!* | | `probe_size` | Size | `"10MB"` | Passes `-probesize` to FFprobe for VODs/Series (Data Limit). Supports units like KB, MB, GB. | | `live_analyze_duration` | Duration | `"5s"` | Stricter time limit for Live-TV streams to minimize latency and provider traffic. *Warning: Requires unit suffix!* | | `live_probe_size` | Size | `"5MB"` | Stricter data limit for Live-TV streams. | **Why are there 4 FFprobe fields?** `ffprobe.analyze_duration` + `ffprobe.probe_size` are the default pair for non-live probes (VOD/Series). `ffprobe.live_analyze_duration` + `ffprobe.live_probe_size` are the live-specific pair for all live probes. This split is intentional because live probing usually needs lower values (less provider load / lower latency), while VOD/Series can use higher values for better metadata extraction quality. --- ## Global Activation of Video Analysis & TMDB API Fallback While `metadata_update` section in config.yml configures the global behavior, you must explicitly activate analysis per input. **Input Config (`source.yml`):** ```yaml inputs: - name: my-provider type: xtream options: # Attempts to resolve missing TMDB IDs and Release Date via TMDB API based on title resolve_tmdb: true # Probes stream if video/audio info is missing in provider data probe_stream: true ``` **Target Config (`source.yml`):** If `add_quality_to_filename` is set for STRM output, the analyzed quality tags are used in filenames. ```yaml targets: - name: my-library output: - type: strm directory: /media/strm style: jellyfin # Adds tags like [2160p 4K HEVC HDR] to the filename add_quality_to_filename: true # Groups different versions of the same movie into one folder (based on TMDB ID) flat: true ``` > **Note on Probing:** Probing respects the `max_connections` limit of your provider input. > If no connection slot is available, the item is skipped and retried during the next update cycle. ### Glossary & Data Types To ensure precise configuration, the following terms and data formats are used throughout the metadata module: #### Special Data Types (Strings) * **Duration:** Specifies time intervals. Supported units are `s` (seconds), `m` (minutes), `h` (hours), and `d` (days). * *Example:* `3600` (plain seconds) or `1h` (suffixed). * *Note:* `ffprobe.analyze_duration` strictly requires a unit suffix. * **Size:** Specifies data volumes for stream analysis. Supported units are `B`, `KB`, `MB`, `GB`, and `TB`. * *Example:* `10485760` (plain bytes) or `10MB` (suffixed). #### Terminology * `Resolve task`: A metadata job that fetches or enriches item metadata (e.g., VOD/Series details, TMDB IDs, or release dates) via provider APIs or TMDB. * `Probe task`: A technical analysis job that physically inspects stream properties (codecs, resolution, audio tracks) using the FFprobe engine. * `TMDB cooldown`: A per-item lock set when a TMDB lookup completes successfully but returns "no match." This prevents redundant API calls for items not found on TMDB. * `Attempt`: A single execution try of a task. If it fails, the attempt counter for that specific item is incremented. * `Retry`: A subsequent re-attempt of a failed task after a waiting period. * `Backoff delay`: The waiting time before the next retry is allowed after a failure. * `Exponential backoff`: A strategy where the `backoff delay` grows with each failed attempt (e.g., 5s, 10s, 20s) up to a configured `max_retry_backoff`. * `Jitter`: A small random variation added to the backoff delay to prevent "thundering herd" issues, where hundreds of tasks retry at the exact same millisecond. * `Transient error`: A temporary failure (e.g., socket timeout, 502 Bad Gateway) that is likely to resolve itself in a future attempt. * `Exhausted`: The state reached when `max_attempts` for a task type are met. The task will not be retried within the current cycle. * `Cooldown`: A mandatory skip period applied after a task is **Exhausted** (primarily used for Probes). No retries are allowed until this period expires. * `Update cycle`: A full metadata processing run for a specific input, spanning from the first queued item to the point where the queue is idle. * `Resolve exhaustion reset gap`: The time window after a cycle completion after which "Exhausted" states for resolve tasks are cleared for the next run. * `Pending queue`: The in-memory list of metadata tasks currently waiting for an available background worker slot. * `Worker idle timeout`: The duration a metadata worker stays active without work before shutting down to release system resources. > **Persistence:** All retry, exhaustion, and cooldown states are persisted per input in the `metadata_retry_state.db`. > This ensures that Tuliprox remembers the status of broken streams across server restarts.   ## Additional Information Tuliprox is not just a proxy; it is a highly intelligent **Playlist Processing Engine**. A core part of this is the asynchronous update and metadata process that loads information from the provider, updates local databases, and fully automatically supplements missing metadata. ### 1. The Complete Processing Pipeline When a playlist update starts (via Scheduler, API, or Boot), Tuliprox runs this pipeline: 1. **Download & Cache Check:** For each configured `input`, it checks if provider data needs re-downloading (controlled by `cache_duration`). 2. **Input Storage (B+Tree):** The raw data (M3U, Xtream categories) is written to a local, extremely fast B+Tree database (`input_name.db`). This drastically saves RAM. 3. **Target Processing:** For each defined `target` (output playlist), data is loaded from the input and routed through the pipeline (`processing_order`, e.g., Filter âž” Rename âž” Map). 4. **Metadata Resolve & Probe:** Tuliprox analyzes the filtered entries. If data is missing (e.g., TMDB IDs, Video Codecs), these are dispatched as "Jobs" to the `MetadataUpdateManager`. 5. **Target Storage & EPG:** The finished playlist is written to the Target databases. Only then is XMLTV EPG data matched and assigned. --- ### 2. The `MetadataUpdateManager` (Architecture) The `MetadataUpdateManager` is an asynchronous background engine (if `resolve_background: true` is set on the input) that prevents blocking the main playlist update. #### Architecture & Logic * **Per-Input Worker:** A dedicated, isolated *Tokio Task (Worker)* is started for each Provider-Input. This prevents a slow provider from blocking another. * **Task-Merging:** If a stream requires both TMDB info and an FFprobe, they are merged into a single Task. * **Rate-Limiting & Connection-Locks:** The manager strictly respects the `max_connections` of your input. An FFprobe ( Stream Analysis) is *only* initiated if a provider connection is free. A probe task runs at the absolute lowest priority (`user_priority: 127`). If a real user starts streaming, the FFprobe process is **immediately aborted/preempted** to free the slot for the user! * **Smart Retry & Cooldown:** If a fetch or probe fails (e.g., HTTP 502), an exponential backoff with Jitter (random deviation) kicks in. If the max attempts (`max_attempts`) are reached, the task enters a global cooldown (e.g., 7 days) to stop harassing the provider. * **Persistence (Retry State):** The status of failed tasks is stored locally in `metadata_retry_state.db`. A server restart does not cause Tuliprox to immediately bombard the provider with requests for broken streams. * **Cascading Updates:** Once a worker collects a batch of metadata, it saves it in the Input DB and immediately *cascades* (inherits) the updates into all Target DBs, without requiring a full playlist rebuild. --- ### 3. Metadata Collection Mechanisms #### How is a stream queued for analysis? Tuliprox checks every stream for completeness (`has_details()`). A task is queued if the switches in `inputs.options` are active **and** one of these conditions is met: * **Info-Resolve (VOD/Series):** Missing provider info (Cast, Plot, Director) retrievable via Xtream API (`get_vod_info` / `get_series_info`). * **TMDB/Date-Resolve:** Missing `tmdb_id` or `release_date`. * **Probing (FFprobe):** * VOD/Series: Missing technical A/V parameters (`video_codec`, `audio_codec`, `resolution`). * Live-TV: The `last_probed_timestamp` is older than `probe_live_interval_hours`. #### Collection Engines * **Release Year / Date (PTT):** Tuliprox uses a highly optimized internal parser (`PTT` - Parse Torrent Title). It locally analyzes the stream name and extracts the year (e.g., from *"My Movie (2023)"*). If this fails, it queries the TMDB API. * **TMDB Information:** Via TMDB API and a Jaro-Winkler distance comparison (similarity scoring), it fetches IDs, release years, covers, backdrops, genres, directors, and actors. * **Video & Audio (FFprobe):** Tuliprox briefly opens the stream via `ffprobe`. It extracts and normalizes: * *Resolution:* SD, 720p, 1080p, 1440p, 4K, 8K * *Video:* Codec (H264, HEVC, AV1), Bit-Depth (8bit, 10bit), Dynamic Range (HDR10, Dolby Vision, HLG) * *Audio:* Codec (AAC, AC3, EAC3, DTS, TrueHD) and Channels (2.0, 5.1, 7.1) * Tuliprox uses these tags later for the `add_quality_to_filename` target feature (e.g., `My Movie [2160p 4K HEVC HDR].strm`). * **Seasons & Episodes:** For series, the Xtream API delivers a structure of seasons and episodes. Tuliprox "flattens" these into individually playable streams (`PlaylistItemType::Series`). Each episode is treated **individually** during probing, as codecs and resolutions can change from episode to episode.