Commit Graph
1 Commits
Author SHA1 Message Date
Claude f8f3ca3a82 fix(watchdog): correct bandwidth totals, rps, producers and fatals
SystemInfo::getStats() feeds the watchdog heartbeat (servers.watchdog_data),
servers_stats and the LB /api stats action, and had four defects:

- bytes_sent_total/bytes_received_total were assigned per interface, so
  only the last selected interface was kept, and the received total read
  tx_bytes. aggregateNetwork() now sums both totals over the same
  interfaces as the per-second rates and reads rx_bytes for received.
  network_speed keeps its first-positive rule and the interface selection
  is unchanged, so routing capacity is not affected.
- total_running_streams was `ps ax | grep -c ffmpeg`, which counted
  ffprobe and shell wrappers and missed the fanout daemon's native
  remuxer. ProcessManager::countStreamProducers() now reads /proc and
  counts processes whose program is ffmpeg or `xc_fanout remux`.
- The memory percentage divided by total_mem, which getMemory() reports
  as 0 without MemAvailable (DivisionByZeroError), and nvidia-smi XML
  with a single GPU process decodes process_info as one associative
  array (TypeError). Either aborted the watchdog pass, so the node wrote
  no heartbeat. memUsedPercent() and gpuProcesses() guard both.

The watchdog runs one pass per process and re-execs itself, so the
previous nginx request count held in a variable was always null and
requests_per_second was 0 on every node. CounterRateSampler keeps the
previous reading in a tmp state file (atomic replace, 120 s max age, 0
after a counter reset).

The watchdog_data key set and order are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
2026-09-25 00:10:15 +00:00