tuliprox/docs/src/troubleshooting.md

# 🛠️ Troubleshooting & Resilience

Running a reverse proxy for IPTV involves navigating the quirks of various video players and the strict limitations of upstream providers. This page
documents common "real-world" behaviors that can lead to stream interruptions or provider bans, and how to mitigate them using Tuliprox's resilience
features.

The solutions below will help you fine-tune your configuration for a seamless experience.

---

## 1. The VLC "Seek" Problem (Grace Periods)

**The Problem:** A user watches a VOD movie via reverse proxy. They press "Fast forward 10 seconds" in VLC. VLC calculates the new byte offset, kills
the TCP connection, and instantly fires a new HTTP GET request (with a `Range` header) to your Tuliprox server.

Tuliprox opens a new connection to the upstream provider. However, since the old connection takes milliseconds to officially close at the provider
side, the provider sees **two** active streams. If you only paid for 1 connection, the provider throws a 509 Bandwidth Exceeded error or bans your IP!

**The Solution in `config.yml`:**

```yaml
reverse_proxy:
  stream:
    grace_period_hold_stream: true
    grace_period_millis: 2000
    grace_period_timeout_secs: 5
```

**What happens now?**
Tuliprox detects the bottleneck and grants a temporary "grace" state:

* **Hold State:** Because `grace_period_hold_stream: true` is set, Tuliprox keeps the client connection "warm" but
  waits before requesting new bytes from the provider.
* **The Handover:** It waits for `grace_period_millis` (2000ms) to give the provider's server time to register the old connection as closed.
* **Resolution:**
  * **Success:** If the old "ghost" connection dies within the window ➔ The new stream flows instantly.
  * **Timeout:** If the old connection persists beyond `grace_period_timeout_secs` (5s) ➔ Grace is revoked, and the client receives the
    `user_connections_exhausted.ts` video.

---

## 2. Zombie Connections After a Client IP Change (Wi-Fi → Mobile Data)

**The Problem:** A user starts a stream on their mobile phone while connected to home Wi-Fi. They then walk outside, and the phone automatically
switches to 4G/5G. The phone's public IP address changes, and the old stream stalls.

A new connection attempt from the new IP address hits the `max_connections` limit and starts playing the `user_connections_exhausted.ts` video —
even though the user is the only viewer. The original stream slot appears "stuck" and is not released, even though no data is flowing to the client
anymore.

**Root Cause — Why the Slot Gets Stuck:**

When a phone switches networks without explicitly closing its TCP connection (no FIN/RST packet), the server has no way to know the client is gone.
The kernel keeps trying to deliver buffered stream data with exponential retransmission back-off.

TCP Keepalive probes — while configured in Tuliprox — only fire on **idle** connections (no data sent for a period of time). A live IPTV stream is
never idle from the server's perspective, so keepalive probes never trigger. The result is that the kernel can keep retransmitting for **2–15 minutes**
before it finally gives up and closes the connection.

**The Solution — `TCP_USER_TIMEOUT` (Linux only):**

Tuliprox automatically sets the `TCP_USER_TIMEOUT` socket option on every accepted connection on Linux. This option instructs the kernel to forcibly
close a connection once transmitted data has been unacknowledged for a defined period — regardless of whether the connection was idle or actively
sending.

With the default value of **30 seconds**, a dead streaming connection is detected and its slot is freed within 30 seconds of the client disappearing.

```code
t = 0 s    Phone switches from Wi-Fi to 4G (old TCP connection dies silently)
t = 0 s    Server continues sending stream data; kernel buffers it (no ACKs)
t ≤ 30 s   TCP_USER_TIMEOUT exceeded → kernel forcibly closes the connection
t ≤ 30 s   Tuliprox releases the user connection slot
t ≤ 30 s   New connection from the 4G IP can now acquire the slot normally
```

> **Platform Note:** `TCP_USER_TIMEOUT` is a Linux-specific feature (available since kernel 2.6.37). On **Windows** and **macOS**, this option is not
> available. Those platforms handle dead connections through TCP Keepalive probes or platform-specific socket options, which are less effective for
> active streaming connections. In practice this is not an issue since Tuliprox is designed to run on Linux servers and Docker containers.

---

## 3. Database Migration Failures (`LZ4 decompression failed: 0 is not a valid match offset`)

**The Problem:**
During startup migration of legacy V2 databases (e.g., `series.db`, `live.db`), the server or container fails with:

```text
ERROR tuliprox::repository::bplustree::migration] Startup migration failed: Failed to migrate B+Tree /app/data/input_dir/series.db: LZ4 decompression failed: 0 is not a valid match offset
```

**Root Cause:**
In the legacy B+Tree V2 storage format, entries were updated in-place without page-level checksums or write-ahead logging.
If an unexpected power loss, kill signal, or torn disk write occurred during an update, a single value's LZ4 block could
contain zeroed bytes (offset `0` is invalid according to the LZ4 format specification). In older versions, a single unreadable
record caused the entire database migration to abort.

**The Solution & Resilience:**

1. **Automatic Error Recovery:** Tuliprox's migration engine now automatically catches decompression and decoding errors on
   damaged individual records, logs a warning with the path of the skipped record, and continues migrating all healthy records
   into a valid, crash-safe V3 database.
2. **Manual Inspection & Repair:** You can inspect the health of any database or manually trigger the migration using the
   included helper script:

```bash
# Inspect the database (shows healthy vs. corrupt records without changing anything):
./bin/migrate_db.sh --inspect /path/to/series.db

# Migrate and salvage all healthy records (creates a timestamped backup automatically):
./bin/migrate_db.sh /path/to/series.db
```

If Tuliprox runs in Docker:

```bash
docker exec -it tuliprox tuliprox --migrate-db /app/data/input_dir/series.db
```

See [Database Migration & Inspection](./operations-debugging.md#6-database-migration--inspection-migrate_dbsh--cli) for full details.

---

## 4. File Descriptor Exhaustion (`No file descriptors available (os error 24)`)

**The Problem:**

The server logs show cascade errors failing to open files or establish network connections:

```text
[WARN tuliprox_session::qos_aggregation_manager] QoS aggregation run failed: I/O error: No file descriptors available (os error 24)
[INFO tuliprox_processing::processor::playlist] 🌷 Update process started.
[WARN tuliprox_processing::input_cache] Failed to read input status file /app/data/input_PrimeTrial/status.json: No file descriptors available (os error 24)
[WARN tuliprox_iptv::xtream] Failed to login xtream account PrimeTrial repository Network error: can't download input PrimeTrial => Request error: error sending request for url (http://***/player_api.php?...)
[ERROR tuliprox_processing::input_cache] Failed to write input status file /app/data/input_PrimeTrial/status.json: No file descriptors available (os error 24)
```

**Root Cause:**

The operating system error `os error 24` is **`EMFILE`** (*"Too many open files"*). On Unix and Linux systems,
all I/O abstractions consume file descriptors (FDs):

* **Client TCP Sockets:** Every media player connected to a live stream or VOD session (MPEG-TS, HLS chunk streaming).
* **Upstream TCP Sockets:** Every active proxy connection to an IPTV provider.
* **HTTP Client Connection Pools:** Reqwest and Hyper keep-alive connections held open for API requests and playlist downloads.
* **B+Tree Databases & Sidecars:** Every database (`xtream_*.db`, `m3u_*.db`, `epg_*.db`, `target_id_mapping.db`,
  `qos_snapshot.db`), along with WAL files and `.sidecar` lock handles.
* **Async Runtime:** Tokio worker threads, event notification file descriptors (`epoll`, `eventfd`, `timerfd`).

By default, Docker daemons and Linux user sessions often configure a low limit of only **1024** file descriptors
per container/process (`ulimit -n`). In a streaming proxy where background jobs (such as QoS aggregation or scheduled
playlist updates) run concurrently with active media streams, 1024 descriptors can be exhausted quickly. Once exhausted,
any operation requiring a new descriptor (such as opening a file or creating a socket for an HTTP request) fails immediately.

**The Solution:**

1. **Configure `ulimits` in `docker-compose.yml` (Recommended):**
   Increase the maximum open file limit for the Tuliprox container by defining `ulimits.nofile`:

   ```yaml
   services:
     tuliprox:
       image: ghcr.io/euzu/tuliprox:latest
       container_name: tuliprox
       restart: unless-stopped
       ulimits:
         nofile:
           soft: 65535
           hard: 65535
       # ... remaining configuration
   ```

   Apply the updated configuration by recreating the container:

   ```bash
   docker compose up -d
   ```

2. **Host / Systemd Service Limit (Native Deployment):**
   If running Tuliprox natively outside Docker via `systemd`, set `LimitNOFILE` in your unit file (e.g. `/etc/systemd/system/tuliprox.service`):

   ```ini
   [Service]
   LimitNOFILE=65535
   ```

   Then reload systemd and restart the service:

   ```bash
   sudo systemctl daemon-reload
   sudo systemctl restart tuliprox
   ```

3. **Diagnosing Open File Descriptors:**
   To check how many descriptors are currently open and identify what is using them:

   ```bash
   # Find the process ID
   PID=$(pgrep tuliprox)

   # Count active open descriptors
   ls -1 /proc/$PID/fd | wc -l

   # List open files and sockets
   ls -l /proc/$PID/fd
   ```

---

## 5. Server Stops Responding While the Container Keeps Running

**The Problem:** After running normally for a while (often hours), Tuliprox stops processing and logging. The container
still shows `running` with `ExitCode: 0` and low CPU/memory; there is no panic, no shutdown message and no final log
entry. Only a manual restart recovers it.

**Root Cause:** A wedged async runtime. Every Tokio worker is parked on a futex and the I/O driver sits in `epoll_wait`,
which the OS cannot distinguish from an idle process. This is consistent with a task holding a shared lock while waiting
forever on a network operation that has no timeout (for example an upstream that accepts the connection but never sends a
response).

**How to confirm:** Enable the runtime liveness watchdog (off by default):

```yaml
services:
  tuliprox:
    environment:
      - TULIPROX_WATCHDOG=1   # observe only
      # - TULIPROX_WATCHDOG=2 # observe and restart on a confirmed stall
```

When the runtime stops making progress, the watchdog logs a `Runtime liveness stall` line with runtime metrics and a
per-thread `/proc/self/task` inventory, and `GET /healthcheck` reports `runtime.status: stalled`.

In mode `1` the watchdog only reports. In mode `2` it exits the process (code `75`) once the stall has persisted past
the grace period (`TULIPROX_WATCHDOG_RESTART_GRACE_MS`, default 30 s), so a configured `restart: unless-stopped`
brings the container back automatically. Use mode `1` while you are still investigating a cause, and mode `2` once a
restart is the acceptable recovery.

**Immediate mitigation:** restart the container. The watchdog log is what tells you the stall happened on its own and
captures what every thread was doing at that moment.

---