cron:servers only checked whether a watchdog process existed. A watchdog
blocked in poll() on a half-open MariaDB socket (CLOSE_WAIT) inside its DB
ping still exists, so cron kept trusting it indefinitely: last_check_ago
went stale and the panel marked the node offline while nginx, php-fpm,
redis and active streams were all healthy. Restarting only the watchdog
restored the heartbeat immediately.
Treat an abnormally old watchdog as stale. A normal generation lives only
a few seconds, so any process alive for more than 90s is killed and a
fresh generation is started; a plain presence check alone would keep
trusting the wedged one forever.
Add ProcessManager::getProcessAge(), which derives age from the mtime of
the /proc/PID directory (time() - filemtime) — the same figure as
`ps -o etimes` without shelling out, consistent with the rest of the class
reading /proc directly and with no HZ/CLK_TCK assumption. Reaping uses the
existing ProcessManager::kill() rather than a raw `kill -9`.
Recovery restarts only the watchdog; nginx, php-fpm, redis, ffmpeg and
running streams are untouched.
Two false positives/negatives in process detection kept the panel's
self-healing loop dead on any real installation:
- isNginxRunning() looked for "nginx: master" only among xc_vm-owned
processes, but the master runs as root on typical installs (workers run
as xc_vm), so cron:servers/cron:streams bailed out with "XC_VM not
running..." before reaching the daemon revival block. Now scans
/proc/*/cmdline user-agnostically, same as RootSignalsCronJob.
- The per-daemon "is it alive" checks piped ps through grep by bare words:
every live-stream ffmpeg carries -thread_queue_size/-max_muxing_queue_size
in its command line, so the "queue" check matched any running stream and
the encode queue daemon was never started while at least one channel was
up — created channels sat at "0% DONE" until console.php queue was run by
hand. All seven checks (signals, cache_handler, network, watchdog, queue,
ondemand, scanner) now use ProcessManager::findProcessPIDs(), a /proc
cmdline scan matching the daemon title (XC_VM[...]) or its exact
console.php invocation. Kill branches use the same PID list — the old
bare "ondemand"/"scanner" greps could kill an innocent ffmpeg whose
source URL contained those words.
Divarion-D