tuliprox/docs/src/troubleshooting.md

🛠️ Troubleshooting & Resilience

Running a reverse proxy for IPTV involves navigating the quirks of various video players and the strict limitations of upstream providers. This page documents common "real-world" behaviors that can lead to stream interruptions or provider bans, and how to mitigate them using Tuliprox's resilience features.

The solutions below will help you fine-tune your configuration for a seamless experience.

---

1. The VLC "Seek" Problem (Grace Periods)

The Problem: A user watches a VOD movie via reverse proxy. They press "Fast forward 10 seconds" in VLC. VLC calculates the new byte offset, kills the TCP connection, and instantly fires a new HTTP GET request (with a Range header) to your Tuliprox server.

Tuliprox opens a new connection to the upstream provider. However, since the old connection takes milliseconds to officially close at the provider side, the provider sees two active streams. If you only paid for 1 connection, the provider throws a 509 Bandwidth Exceeded error or bans your IP!

The Solution in config.yml:

reverse_proxy:
  stream:
    grace_period_hold_stream: true
    grace_period_millis: 2000
    grace_period_timeout_secs: 5

What happens now? Tuliprox detects the bottleneck and grants a temporary "grace" state:

• Hold State: Because grace_period_hold_stream: true is set, Tuliprox keeps the client connection "warm" but waits before requesting new bytes from the provider.
• The Handover: It waits for grace_period_millis (2000ms) to give the provider's server time to register the old connection as closed.
• Resolution:
  • Success: If the old "ghost" connection dies within the window ➔ The new stream flows instantly.
  • Timeout: If the old connection persists beyond grace_period_timeout_secs (5s) ➔ Grace is revoked, and the client receives the user_connections_exhausted.ts video.

---

2. Zombie Connections After a Client IP Change (Wi-Fi → Mobile Data)

The Problem: A user starts a stream on their mobile phone while connected to home Wi-Fi. They then walk outside, and the phone automatically switches to 4G/5G. The phone's public IP address changes, and the old stream stalls.

A new connection attempt from the new IP address hits the max_connections limit and starts playing the user_connections_exhausted.ts video — even though the user is the only viewer. The original stream slot appears "stuck" and is not released, even though no data is flowing to the client anymore.

Root Cause — Why the Slot Gets Stuck:

When a phone switches networks without explicitly closing its TCP connection (no FIN/RST packet), the server has no way to know the client is gone. The kernel keeps trying to deliver buffered stream data with exponential retransmission back-off.

TCP Keepalive probes — while configured in Tuliprox — only fire on idle connections (no data sent for a period of time). A live IPTV stream is never idle from the server's perspective, so keepalive probes never trigger. The result is that the kernel can keep retransmitting for 2–15 minutes before it finally gives up and closes the connection.

The Solution — TCP_USER_TIMEOUT (Linux only):

Tuliprox automatically sets the TCP_USER_TIMEOUT socket option on every accepted connection on Linux. This option instructs the kernel to forcibly close a connection once transmitted data has been unacknowledged for a defined period — regardless of whether the connection was idle or actively sending.

With the default value of 30 seconds, a dead streaming connection is detected and its slot is freed within 30 seconds of the client disappearing.

t = 0 s    Phone switches from Wi-Fi to 4G (old TCP connection dies silently)
t = 0 s    Server continues sending stream data; kernel buffers it (no ACKs)
t ≤ 30 s   TCP_USER_TIMEOUT exceeded → kernel forcibly closes the connection
t ≤ 30 s   Tuliprox releases the user connection slot
t ≤ 30 s   New connection from the 4G IP can now acquire the slot normally

Platform Note: TCP_USER_TIMEOUT is a Linux-specific feature (available since kernel 2.6.37). On Windows and macOS, this option is not available. Those platforms handle dead connections through TCP Keepalive probes or platform-specific socket options, which are less effective for active streaming connections. In practice this is not an issue since Tuliprox is designed to run on Linux servers and Docker containers.

---

3. Database Migration Failures (LZ4 decompression failed: 0 is not a valid match offset)

The Problem: During startup migration of legacy V2 databases (e.g., series.db, live.db), the server or container fails with:

ERROR tuliprox::repository::bplustree::migration] Startup migration failed: Failed to migrate B+Tree /app/data/input_dir/series.db: LZ4 decompression failed: 0 is not a valid match offset

Root Cause: In the legacy B+Tree V2 storage format, entries were updated in-place without page-level checksums or write-ahead logging. If an unexpected power loss, kill signal, or torn disk write occurred during an update, a single value's LZ4 block could contain zeroed bytes (offset 0 is invalid according to the LZ4 format specification). In older versions, a single unreadable record caused the entire database migration to abort.

The Solution & Resilience:

1. Automatic Error Recovery: Tuliprox's migration engine now automatically catches decompression and decoding errors on damaged individual records, logs a warning with the path of the skipped record, and continues migrating all healthy records into a valid, crash-safe V3 database.
2. Manual Inspection & Repair: You can inspect the health of any database or manually trigger the migration using the included helper script:

# Inspect the database (shows healthy vs. corrupt records without changing anything):
./bin/migrate_db.sh --inspect /path/to/series.db

# Migrate and salvage all healthy records (creates a timestamped backup automatically):
./bin/migrate_db.sh /path/to/series.db

If Tuliprox runs in Docker:

docker exec -it tuliprox tuliprox --migrate-db /app/data/input_dir/series.db

See Database Migration & Inspection for full details.

---

4. File Descriptor Exhaustion (No file descriptors available (os error 24))

The Problem:

The server logs show cascade errors failing to open files or establish network connections:

[WARN tuliprox_session::qos_aggregation_manager] QoS aggregation run failed: I/O error: No file descriptors available (os error 24)
[INFO tuliprox_processing::processor::playlist] 🌷 Update process started.
[WARN tuliprox_processing::input_cache] Failed to read input status file /app/data/input_PrimeTrial/status.json: No file descriptors available (os error 24)
[WARN tuliprox_iptv::xtream] Failed to login xtream account PrimeTrial repository Network error: can't download input PrimeTrial => Request error: error sending request for url (http://***/player_api.php?...)
[ERROR tuliprox_processing::input_cache] Failed to write input status file /app/data/input_PrimeTrial/status.json: No file descriptors available (os error 24)

Root Cause:

The operating system error os error 24 is EMFILE ("Too many open files"). On Unix and Linux systems, all I/O abstractions consume file descriptors (FDs):

• Client TCP Sockets: Every media player connected to a live stream or VOD session (MPEG-TS, HLS chunk streaming).
• Upstream TCP Sockets: Every active proxy connection to an IPTV provider.
• HTTP Client Connection Pools: Reqwest and Hyper keep-alive connections held open for API requests and playlist downloads.
• B+Tree Databases & Sidecars: Every database (xtream_*.db, m3u_*.db, epg_*.db, target_id_mapping.db, qos_snapshot.db), along with WAL files and .sidecar lock handles.
• Async Runtime: Tokio worker threads, event notification file descriptors (epoll, eventfd, timerfd).

By default, Docker daemons and Linux user sessions often configure a low limit of only 1024 file descriptors per container/process (ulimit -n). In a streaming proxy where background jobs (such as QoS aggregation or scheduled playlist updates) run concurrently with active media streams, 1024 descriptors can be exhausted quickly. Once exhausted, any operation requiring a new descriptor (such as opening a file or creating a socket for an HTTP request) fails immediately.

The Solution:

1. Configure ulimits in docker-compose.yml (Recommended): Increase the maximum open file limit for the Tuliprox container by defining ulimits.nofile:

   services:
     tuliprox:
       image: ghcr.io/euzu/tuliprox:latest
       container_name: tuliprox
       restart: unless-stopped
       ulimits:
         nofile:
           soft: 65535
           hard: 65535
       # ... remaining configuration
   

   Apply the updated configuration by recreating the container:

   docker compose up -d
   

2. Host / Systemd Service Limit (Native Deployment): If running Tuliprox natively outside Docker via systemd, set LimitNOFILE in your unit file (e.g. /etc/systemd/system/tuliprox.service):

   [Service]
   LimitNOFILE=65535
   

   Then reload systemd and restart the service:

   sudo systemctl daemon-reload
   sudo systemctl restart tuliprox
   

3. Diagnosing Open File Descriptors: To check how many descriptors are currently open and identify what is using them:

   # Find the process ID
   PID=$(pgrep tuliprox)
   
   # Count active open descriptors
   ls -1 /proc/$PID/fd | wc -l
   
   # List open files and sockets
   ls -l /proc/$PID/fd
   

---

5. Server Stops Responding While the Container Keeps Running

The Problem: After running normally for a while (often hours), Tuliprox stops processing and logging. The container still shows running with ExitCode: 0 and low CPU/memory; there is no panic, no shutdown message and no final log entry. Only a manual restart recovers it.

Root Cause: A wedged async runtime. Every Tokio worker is parked on a futex and the I/O driver sits in epoll_wait, which the OS cannot distinguish from an idle process. This is consistent with a task holding a shared lock while waiting forever on a network operation that has no timeout (for example an upstream that accepts the connection but never sends a response).

How to confirm: Enable the runtime liveness watchdog (off by default):

services:
  tuliprox:
    environment:
      - TULIPROX_WATCHDOG=1   # observe only
      # - TULIPROX_WATCHDOG=2 # observe and restart on a confirmed stall

When the runtime stops making progress, the watchdog logs a Runtime liveness stall line with runtime metrics and a per-thread /proc/self/task inventory, and GET /healthcheck reports runtime.status: stalled.

In mode 1 the watchdog only reports. In mode 2 it exits the process (code 75) once the stall has persisted past the grace period (TULIPROX_WATCHDOG_RESTART_GRACE_MS, default 30 s), so a configured restart: unless-stopped brings the container back automatically. Use mode 1 while you are still investigating a cause, and mode 2 once a restart is the acceptable recovery.

Immediate mitigation: restart the container. The watchdog log is what tells you the stall happened on its own and captures what every thread was doing at that moment.

---