mirror of
https://github.com/Vateron-Media/XC_VM.git
synced 2026-10-18 12:02:57 +02:00
fix(cron): reap wedged watchdog instead of trusting its presence
cron:servers only checked whether a watchdog process existed. A watchdog blocked in poll() on a half-open MariaDB socket (CLOSE_WAIT) inside its DB ping still exists, so cron kept trusting it indefinitely: last_check_ago went stale and the panel marked the node offline while nginx, php-fpm, redis and active streams were all healthy. Restarting only the watchdog restored the heartbeat immediately. Treat an abnormally old watchdog as stale. A normal generation lives only a few seconds, so any process alive for more than 90s is killed and a fresh generation is started; a plain presence check alone would keep trusting the wedged one forever. Add ProcessManager::getProcessAge(), which derives age from the mtime of the /proc/PID directory (time() - filemtime) — the same figure as `ps -o etimes` without shelling out, consistent with the rest of the class reading /proc directly and with no HZ/CLK_TCK assumption. Reaping uses the existing ProcessManager::kill() rather than a raw `kill -9`. Recovery restarts only the watchdog; nginx, php-fpm, redis, ffmpeg and running streams are untouched.
This commit is contained in:
@@ -254,6 +254,40 @@ class ProcessManager {
|
||||
return getmypid();
|
||||
}
|
||||
|
||||
/**
|
||||
* Get the age of a process in seconds (how long it has been running).
|
||||
*
|
||||
* On Linux the mtime of the /proc/PID directory is fixed to the
|
||||
* process start time, so `time() - filemtime()` yields the same figure
|
||||
* as `ps -o etimes` without shelling out to ps — consistent with the
|
||||
* rest of this class reading /proc directly.
|
||||
*
|
||||
* Used to detect daemons that are still present but wedged (e.g. a
|
||||
* watchdog blocked in poll() on a half-open MariaDB socket): a normally
|
||||
* short-lived generation that has been alive far too long is stale.
|
||||
*
|
||||
* @param int $pid Process ID
|
||||
* @return int Age in seconds, or -1 if it cannot be determined
|
||||
*/
|
||||
public static function getProcessAge($pid) {
|
||||
$pid = (int)$pid;
|
||||
|
||||
if ($pid <= 0 || !self::procExists($pid)) {
|
||||
return -1;
|
||||
}
|
||||
|
||||
clearstatcache(true, '/proc/' . $pid);
|
||||
$rStart = @filemtime('/proc/' . $pid);
|
||||
|
||||
if ($rStart === false) {
|
||||
return -1;
|
||||
}
|
||||
|
||||
$rAge = time() - $rStart;
|
||||
|
||||
return $rAge > 0 ? $rAge : 0;
|
||||
}
|
||||
|
||||
// ───────────────────────────────────────────────────────────
|
||||
// Cron Lock Management
|
||||
// ───────────────────────────────────────────────────────────
|
||||
|
||||
Reference in New Issue
Block a user