fix(cron): reap wedged watchdog instead of trusting its presence

cron:servers only checked whether a watchdog process existed. A watchdog
  blocked in poll() on a half-open MariaDB socket (CLOSE_WAIT) inside its DB
  ping still exists, so cron kept trusting it indefinitely: last_check_ago
  went stale and the panel marked the node offline while nginx, php-fpm,
  redis and active streams were all healthy. Restarting only the watchdog
  restored the heartbeat immediately.

  Treat an abnormally old watchdog as stale. A normal generation lives only
  a few seconds, so any process alive for more than 90s is killed and a
  fresh generation is started; a plain presence check alone would keep
  trusting the wedged one forever.

  Add ProcessManager::getProcessAge(), which derives age from the mtime of
  the /proc/PID directory (time() - filemtime) — the same figure as
  `ps -o etimes` without shelling out, consistent with the rest of the class
  reading /proc directly and with no HZ/CLK_TCK assumption. Reaping uses the
  existing ProcessManager::kill() rather than a raw `kill -9`.

  Recovery restarts only the watchdog; nginx, php-fpm, redis, ffmpeg and
  running streams are untouched.
This commit is contained in:
Divarion-D
2026-07-29 18:43:27 +03:00
parent ebdcd6e725
commit 552c4036c9
2 changed files with 50 additions and 1 deletions
+34
View File
@@ -254,6 +254,40 @@ class ProcessManager {
return getmypid();
}
/**
* Get the age of a process in seconds (how long it has been running).
*
* On Linux the mtime of the /proc/PID directory is fixed to the
* process start time, so `time() - filemtime()` yields the same figure
* as `ps -o etimes` without shelling out to ps — consistent with the
* rest of this class reading /proc directly.
*
* Used to detect daemons that are still present but wedged (e.g. a
* watchdog blocked in poll() on a half-open MariaDB socket): a normally
* short-lived generation that has been alive far too long is stale.
*
* @param int $pid Process ID
* @return int Age in seconds, or -1 if it cannot be determined
*/
public static function getProcessAge($pid) {
$pid = (int)$pid;
if ($pid <= 0 || !self::procExists($pid)) {
return -1;
}
clearstatcache(true, '/proc/' . $pid);
$rStart = @filemtime('/proc/' . $pid);
if ($rStart === false) {
return -1;
}
$rAge = time() - $rStart;
return $rAge > 0 ? $rAge : 0;
}
// ───────────────────────────────────────────────────────────
// Cron Lock Management
// ───────────────────────────────────────────────────────────