feat(cluster): report settings reads the replica would not answer

Plan section 9, R1 `settings`: API nodes log unknown-key reads to `audit.settings_misses`. Once a node's replica owns its settings, a key outside lb_settings_keys.php is absent there, and the CI scan behind the allowlist cannot see a key built at run time. The count names such reads before they fail. On the node (Core/Cluster/SettingsAudit): in mode 1 or 2, as the agent's flows.json says (NodeFlows::declared, no database and no servers read), each read through SettingsManager's getters (get, getBool, getInt, getString, getArray, has) of a key outside the allowlist's `keys` and `withheld` is counted. A process keeps its counts in memory and merges them at exit, and every minute while it runs, into a per-day file under its lock (STORAGE_PATH/cluster/settings_misses/YYYYMMDD.json). It then writes config/cluster/audit.json, `{"settings_misses": {key: count}}` over the last seven UTC days, which the agent is to send as the heartbeat's `audit`. Everything is bounded: 64 names, the rest and any key that is not a settings name under "*". cron:cleanup prunes the day files after eight days and rewrites audit.json every hour, so old days drop out; in mode 0 the file is removed. MAIN and legacy nodes have no flows.json and count nothing; a read costs one lookup. On MAIN (Domain/Cluster/NodeAudit): a heartbeat's `audit` is normalised (names and positive integer counts only, at most 16 KiB, the least missed past 64 names folded into "*") and kept in the new cluster_nodes.audit (migration 045, mirrored in database.sql; 044 is taken by work on another branch). It is written only when it differs from what the row holds, which every request reads already, so an unchanged report costs no query, on the cluster bus or not. A heartbeat without `audit`, today's agent's, changes nothing. The Cluster Nodes page shows each mode 1 or 2 node's misses. SettingsAuditTest covers the counting per getter, withheld keys not counted, mode 0 and MAIN counting nothing, the merge across processes, the caps, the seven-day window, the pruning, the empty report and the minute's merge of a long-running process. ClusterApiTest covers the heartbeat storing a report once, an unchanged, missing or malformed one writing nothing, an empty one clearing the page, and the cap; and ReplicaBootTest a child process's miss reaching audit.json at its exit, while booting from the replica reads only allowlisted settings.
2026-10-31 11:03:22 +01:00 · 2026-09-27 02:26:55 +00:00
parent fd5ac539a0
commit 4566ecbd52
16 changed files with 668 additions and 4 deletions
@@ -6,6 +6,7 @@ use XcVm\Cli\CommandInterface;
 use XcVm\Cli\CronTrait;
 use XcVm\Core\Cluster\ConnectAudit;
 use XcVm\Core\Cluster\NodeRole;
+use XcVm\Core\Cluster\SettingsAudit;
 use XcVm\Core\Config\SettingsManager;
 use XcVm\Core\Diagnostics\DiagnosticsService;
 use XcVm\Domain\Server\InstallCredentials;
@@ -213,6 +214,10 @@ class CleanupCronJob implements CommandInterface {

 		// This node's connect audit: the cutover gate reads seven days of it.
 		ConnectAudit::prune(8);
+		// Its settings misses: the days that left the report's window drop out
+		// of the audit.json its agent sends.
+		SettingsAudit::prune(8);
+		SettingsAudit::publish();

 		// Retention of cluster-wide log tables: MAIN's job. Every LB used to
 		// run the same DELETEs against MAIN's database each minute.
@@ -0,0 +1,301 @@
+<?php
+
+namespace XcVm\Core\Cluster;
+
+/**
+ * Settings reads the node replica would not answer (plan, section 9, R1
+ * `settings`: "API nodes log unknown-key reads to audit.settings_misses").
+ *
+ * On a node in mode 1 or 2, each read through SettingsManager of a key
+ * outside `lb_settings_keys.php` (its `keys` and `withheld`) is counted.
+ * Once the replica owns the settings such a key is absent, so the count
+ * names a read the allowlist's scan missed (a key built at run time) before
+ * it fails. Reading a key costs one lookup; nothing reaches a database:
+ *
+ * ```text
+ * a process      counts in memory, merged into the day's file at exit and
+ *                at most every FLUSH_EVERY s while it runs
+ * day file       STORAGE_PATH/cluster/settings_misses/YYYYMMDD.json (UTC),
+ *                {key: count}, at most MAX_KEYS keys, the rest under OTHER
+ * audit.json     config/cluster/audit.json, {"settings_misses": {key: count}}:
+ *                the last WINDOW_DAYS days, most missed first, at most
+ *                MAX_KEYS keys and OTHER; the agent sends it as the
+ *                heartbeat's `audit`, and MAIN keeps it with the node
+ * ```
+ *
+ * audit.json is rewritten when a process merges misses, and by cron:cleanup
+ * every hour, so days that leave the window drop out. Only on a node with an
+ * agent (config/cluster/), and in mode 1 or 2; in mode 0 it is removed.
+ *
+ * Lives in Core: it ships to LBs, where Domain\Cluster does not.
+ */
+final class SettingsAudit {
+	/** Keys a day file, and the report, name at most; the rest count under OTHER. */
+	public const MAX_KEYS = 64;
+
+	/** A key past MAX_KEYS, or not a settings name ([a-z0-9_]{1,64}). */
+	public const OTHER = '*';
+
+	/** Days the report covers, today included. */
+	public const WINDOW_DAYS = 7;
+
+	/** Seconds a long-running process keeps its counts before it merges them. */
+	public const FLUSH_EVERY = 60;
+
+	/** Tests: another directory for the day files; false: off. */
+	private static string|false|null $rDir = null;
+
+	/** Tests: another agent directory (where audit.json goes). */
+	private static ?string $rAgentDir = null;
+
+	/**
+	 * The keys that are no miss (null: not decided yet in this process;
+	 * false: this process counts nothing).
+	 *
+	 * @var array<string, true>|false|null
+	 */
+	private static array|false|null $rKnown = null;
+
+	/** @var array<string, int> */
+	private static array $rCounts = [];
+
+	private static int $rFlushedAt = 0;
+
+	private static bool $rAtExit = false;
+
+	/**
+	 * Tests: another directory for the day files, or false for none (the
+	 * suite's default), and another agent directory; null restores
+	 * STORAGE_PATH's and CONFIG_PATH's. The process decides again whether it
+	 * counts.
+	 */
+	public static function useDir(string|false|null $rDir, ?string $rAgentDir = null): void {
+		self::$rDir = $rDir;
+		self::$rAgentDir = $rAgentDir;
+		self::$rKnown = null;
+		self::$rCounts = [];
+		self::$rAtExit = false;
+	}
+
+	public static function dir(): ?string {
+		if (self::$rDir === false) {
+			return null;
+		}
+		return self::$rDir ?? (defined('STORAGE_PATH') ? STORAGE_PATH . 'cluster/settings_misses/' : null);
+	}
+
+	/**
+	 * A read of $rKey (SettingsManager). Counted when this is a node in mode 1
+	 * or 2 and the key is outside the allowlist. Decided once per process,
+	 * from the agent's flows.json alone (NodeFlows::declared): MAIN has no
+	 * agent, so it counts nothing.
+	 */
+	public static function read(string $rKey): void {
+		if (self::$rKnown === false || (self::$rKnown !== null && isset(self::$rKnown[$rKey]))) {
+			return;
+		}
+		if (self::$rKnown === null) {
+			self::$rKnown = false; // a read while deciding is not counted
+			self::$rKnown = self::known();
+			self::$rFlushedAt = time();
+			if (self::$rKnown === false || isset(self::$rKnown[$rKey])) {
+				return;
+			}
+		}
+		// Bounded in memory too: past MAX_KEYS distinct keys, the rest are OTHER.
+		$rKey = isset(self::$rCounts[$rKey]) || count(self::$rCounts) < self::MAX_KEYS ? $rKey : self::OTHER;
+		self::$rCounts[$rKey] = (self::$rCounts[$rKey] ?? 0) + 1;
+		if (!self::$rAtExit) {
+			self::$rAtExit = true;
+			register_shutdown_function([self::class, 'flush']);
+		} elseif (time() - self::$rFlushedAt >= self::FLUSH_EVERY) {
+			self::flush();
+		}
+	}
+
+	/**
+	 * Merge this process's counts into the day's file (under its lock), then
+	 * rewrite audit.json. Never throws: an audit must not break a read.
+	 */
+	public static function flush(?int $rNow = null): void {
+		$rCounts = self::$rCounts;
+		self::$rCounts = [];
+		self::$rFlushedAt = $rNow ?? time();
+		$rDir = self::dir();
+		if ($rCounts === [] || $rDir === null) {
+			return;
+		}
+		try {
+			if (!is_dir($rDir)) {
+				@mkdir($rDir, 0750, true);
+				self::own($rDir);
+			}
+			$rFile = $rDir . gmdate('Ymd', self::$rFlushedAt) . '.json';
+			$rHandle = @fopen($rFile, 'c+');
+			if ($rHandle === false) {
+				return;
+			}
+			try {
+				flock($rHandle, LOCK_EX);
+				$rDay = json_decode((string) stream_get_contents($rHandle), true);
+				$rJson = (string) json_encode((object) self::merge(is_array($rDay) ? $rDay : [], $rCounts));
+				ftruncate($rHandle, 0);
+				rewind($rHandle);
+				fwrite($rHandle, $rJson);
+				fflush($rHandle);
+			} finally {
+				flock($rHandle, LOCK_UN);
+				fclose($rHandle);
+			}
+			self::own($rFile);
+			self::publish(null, self::$rFlushedAt);
+		} catch (\Throwable) {
+			// Counted, not needed: the next flush tries again.
+		}
+	}
+
+	/**
+	 * The misses of the last WINDOW_DAYS UTC days, today included: most missed
+	 * first (then by name), at most MAX_KEYS names, the rest under OTHER.
+	 *
+	 * @return array<string, int>
+	 */
+	public static function summary(?int $rNow = null): array {
+		$rDir = self::dir();
+		$rNow ??= time();
+		$rSum = [];
+		for ($d = 0; $rDir !== null && $d < self::WINDOW_DAYS; $d++) {
+			$rDay = json_decode((string) @file_get_contents($rDir . gmdate('Ymd', $rNow - $d * 86400) . '.json'), true);
+			foreach (is_array($rDay) ? $rDay : [] as $rKey => $rCount) {
+				if (is_int($rCount) && $rCount > 0) {
+					$rKey = self::name((string) $rKey);
+					$rSum[$rKey] = ($rSum[$rKey] ?? 0) + $rCount;
+				}
+			}
+		}
+		return self::top($rSum);
+	}
+
+	/**
+	 * Write `audit.json` for the agent in $rAgentDir (config/cluster/):
+	 * `{"settings_misses": summary()}` on a node in mode 1 or 2; removed in
+	 * mode 0, where nothing is counted. False when there is no agent (MAIN, a
+	 * legacy node) or the write failed.
+	 */
+	public static function publish(?string $rAgentDir = null, ?int $rNow = null): bool {
+		$rAgentDir ??= self::$rAgentDir ?? (defined('CONFIG_PATH') ? CONFIG_PATH . 'cluster/' : null);
+		if ($rAgentDir === null || self::dir() === null || !is_dir($rAgentDir)) {
+			return false;
+		}
+		if (NodeFlows::declared()['mode'] < 1) {
+			@unlink($rAgentDir . 'audit.json');
+			return false;
+		}
+		$rTmp = $rAgentDir . 'audit.json.' . getmypid() . '.tmp';
+		if (@file_put_contents($rTmp, (string) json_encode(['settings_misses' => (object) self::summary($rNow)])) === false || !@rename($rTmp, $rAgentDir . 'audit.json')) {
+			@unlink($rTmp);
+			return false;
+		}
+		self::own($rAgentDir . 'audit.json');
+		return true;
+	}
+
+	/** Delete day files older than $rKeepDays. Returns how many went. */
+	public static function prune(int $rKeepDays = 8, ?int $rNow = null): int {
+		$rDir = self::dir();
+		if ($rDir === null) {
+			return 0;
+		}
+		$rCut = gmdate('Ymd', ($rNow ?? time()) - $rKeepDays * 86400);
+		$rCount = 0;
+		foreach (glob($rDir . '*.json') ?: [] as $rFile) {
+			$rDay = basename($rFile, '.json');
+			if (preg_match('/^\d{8}$/', $rDay) && $rDay < $rCut && @unlink($rFile)) {
+				$rCount++;
+			}
+		}
+		return $rCount;
+	}
+
+	/**
+	 * The keys that are no miss: the allowlist's, secrets withheld included
+	 * (their reads are known). False when this process counts nothing: no
+	 * directory, or not a node in mode 1 or 2.
+	 *
+	 * @return array<string, true>|false
+	 */
+	private static function known(): array|false {
+		if (self::dir() === null || NodeFlows::declared()['mode'] < 1) {
+			return false;
+		}
+		$rList = @include __DIR__ . '/lb_settings_keys.php';
+		if (!is_array($rList)) {
+			return false;
+		}
+		return array_fill_keys(array_merge((array) ($rList['keys'] ?? []), (array) ($rList['withheld'] ?? [])), true);
+	}
+
+	/**
+	 * $rCounts added to a day's counts, each key a settings name or OTHER, at
+	 * most MAX_KEYS names.
+	 *
+	 * @param array<mixed> $rDay
+	 * @param array<string, int> $rCounts
+	 * @return array<string, int>
+	 */
+	private static function merge(array $rDay, array $rCounts): array {
+		$rOut = [];
+		foreach ($rDay as $rKey => $rCount) {
+			if (is_int($rCount) && $rCount > 0) {
+				$rOut[self::name((string) $rKey)] = $rCount;
+			}
+		}
+		foreach ($rCounts as $rKey => $rCount) {
+			$rKey = self::name((string) $rKey);
+			if (!isset($rOut[$rKey]) && $rKey !== self::OTHER && count(array_diff_key($rOut, [self::OTHER => 0])) >= self::MAX_KEYS) {
+				$rKey = self::OTHER;
+			}
+			$rOut[$rKey] = ($rOut[$rKey] ?? 0) + $rCount;
+		}
+		return $rOut;
+	}
+
+	/**
+	 * Most missed first (then by name), at most MAX_KEYS names; the rest,
+	 * and OTHER, last under OTHER.
+	 *
+	 * @param array<string, int> $rCounts
+	 * @return array<string, int>
+	 */
+	public static function top(array $rCounts): array {
+		$rOther = $rCounts[self::OTHER] ?? 0;
+		unset($rCounts[self::OTHER]);
+		$rKeys = array_map('strval', array_keys($rCounts));
+		usort($rKeys, static fn(string $a, string $b): int => [$rCounts[$b], $a] <=> [$rCounts[$a], $b]);
+		$rOut = [];
+		foreach ($rKeys as $i => $rKey) {
+			if ($i < self::MAX_KEYS) {
+				$rOut[$rKey] = $rCounts[$rKey];
+			} else {
+				$rOther += $rCounts[$rKey];
+			}
+		}
+		if ($rOther > 0) {
+			$rOut[self::OTHER] = $rOther;
+		}
+		return $rOut;
+	}
+
+	/** A settings name, or OTHER. */
+	public static function name(string $rKey): string {
+		return preg_match('/^[a-z0-9_]{1,64}$/', $rKey) ? $rKey : self::OTHER;
+	}
+
+	/** A file or directory root made stays the panel user's, whose processes add to it. */
+	private static function own(string $rPath): void {
+		if (function_exists('posix_geteuid') && posix_geteuid() === 0) {
+			@chown($rPath, 'xc_vm');
+			@chgrp($rPath, 'xc_vm');
+		}
+	}
+}
@@ -2,11 +2,16 @@

 namespace XcVm\Core\Config;

+use XcVm\Core\Cluster\SettingsAudit;
+
 /**
 * SettingsManager — singleton-хранилище настроек приложения.
 *
 * Entry points вызывают set(), потребители — getAll() или get().
 *
+ * On a cluster node in mode 1 or 2, a read through the getters of a key the
+ * node replica does not carry is counted (SettingsAudit).
+ *
 * @package XC_VM_Core_Config
 * @author  Divarion_D <https://github.com/Divarion-D>
 * @copyright 2025-2026 Vateron Media
@@ -38,6 +43,7 @@ class SettingsManager {
 	 * @return mixed
 	 */
 	public static function get(string $key, mixed $default = null) {
+		SettingsAudit::read($key);
 		return self::$settings[$key] ?? $default;
 	}

@@ -52,6 +58,7 @@ class SettingsManager {
 	 * Проверяет наличие ключа в настройках.
 	 */
 	public static function has(string $key): bool {
+		SettingsAudit::read($key);
 		return array_key_exists($key, self::$settings);
 	}

@@ -64,6 +71,7 @@ class SettingsManager {
 	 * @param bool   $default Значение, если ключ отсутствует.
 	 */
 	public static function getBool(string $key, bool $default = false): bool {
+		SettingsAudit::read($key);
 		return array_key_exists($key, self::$settings) ? (bool) self::$settings[$key] : $default;
 	}

@@ -71,6 +79,7 @@ class SettingsManager {
 	 * Возвращает значение как int.
 	 */
 	public static function getInt(string $key, int $default = 0): int {
+		SettingsAudit::read($key);
 		return array_key_exists($key, self::$settings) ? (int) self::$settings[$key] : $default;
 	}

@@ -78,6 +87,7 @@ class SettingsManager {
 	 * Возвращает значение как строку.
 	 */
 	public static function getString(string $key, string $default = ''): string {
+		SettingsAudit::read($key);
 		return array_key_exists($key, self::$settings) ? (string) self::$settings[$key] : $default;
 	}

@@ -88,6 +98,7 @@ class SettingsManager {
 	 * достаточно проверить тип; для скаляров/null возвращается $default.
 	 */
 	public static function getArray(string $key, array $default = []): array {
+		SettingsAudit::read($key);
 		$rValue = self::$settings[$key] ?? null;
 		return is_array($rValue) ? $rValue : $default;
 	}
@@ -1932,6 +1932,8 @@ cluster_panel_fp_help = "SHA-256 of MAIN's panel signing key. On a node enrolled
 cluster_root_pin = "Root pin"
 cluster_root_ready_help = "Root commands: the node's root-owned pin of the panel key matches its agent; reboot, update and other root actions go as signed commands."
 cluster_root_missing_help = "No root pin on this node: root actions keep the signals table. Pinned at SSH install, or on the node: console.php cluster:pin-root <panel fingerprint>"
+cluster_settings_misses = "Settings misses"
+cluster_settings_misses_help = "Settings the node read over the last 7 days that its replica does not carry (outside the LB settings allowlist), as its agent reports them in mode 1 and 2. Once the replica owns the node's settings, such a read finds nothing."
 fanout_debug = "Fanout Debug Log"
 fanout_debug_off = "Off"
 fanout_debug_all = "All categories"
@@ -27,10 +27,12 @@ final class ClusterAdmin {

 	/**
 	 * @param array<int, array<string, mixed>> $rServers ServerRepository::getAll(true)
-	 * @return list<array<string, mixed>> One row per enrolled node, with `server_name` and `health`.
+	 * @return list<array<string, mixed>> One row per enrolled node, with `server_name`, `health`
+	 *                                    and `settings_misses` (NodeAudit; null when not reported).
 	 */
 	public static function nodes(array $rServers, int $rOfflineAfterSec): array {
 		$rReady = ClusterMeta::readyAtMs(); // its own query: before ours, not between query() and get_rows()
+		$rMisses = NodeAudit::settingsMisses(); // likewise
 		$rHeard = HeartbeatService::lastSeen(); // MySQL's copy may be a flush behind
 		$rNow = ClusterClock::nowMs();
 		self::db()->query('SELECT `server_id`, `node_uuid`, `state`, `mode`, `flows`, `root_ready`, `gen`, `epoch`, `token_exp`, `last_seen_at`, `agent_version`, `quarantine_reason` FROM `cluster_nodes` ORDER BY `server_id`;');
@@ -40,6 +42,7 @@ final class ClusterAdmin {
 			$rRow['last_seen_at'] = $rLastSeen;
 			$rRow['server_name'] = (string) ($rServers[(int) $rRow['server_id']]['server_name'] ?? ('#' . $rRow['server_id']));
 			$rRow['health'] = $rRow['state'] === 'active' ? NodeHealth::state($rLastSeen, $rReady, $rNow, $rOfflineAfterSec) : (string) $rRow['state'];
+			$rRow['settings_misses'] = $rMisses[(int) $rRow['server_id']] ?? null;
 			$rOut[] = $rRow;
 		}
 		return $rOut;
@@ -553,6 +553,8 @@ final class ClusterApi {
 		if ($rUses !== []) {
 			NodeRegistry::update((int) $rNode['server_id'], $rUses);
 		}
+		// Its own audit (settings misses), kept only when it changed.
+		NodeAudit::record($rNode, $rP['audit'] ?? null);
 		// A node that holds its viewers sends its registry's digest; a drift
 		// that outlives the events in flight gets its snapshot asked for.
 		$rWant = false;
@@ -0,0 +1,104 @@
+<?php
+
+namespace XcVm\Domain\Cluster;
+
+use XcVm\Core\Cluster\SettingsAudit;
+use XcVm\Infrastructure\Database\DatabaseAware;
+
+/**
+ * What a node's heartbeat reports of its own audit (`audit`), kept with the
+ * node in `cluster_nodes.audit` and shown on the Cluster Nodes page. Today
+ * one member: `settings_misses`, the settings reads outside the replica's
+ * allowlist over the node's last seven days (Core\Cluster\SettingsAudit;
+ * plan, section 9, R1 `settings`).
+ *
+ * ```text
+ * audit            {"settings_misses": {key: count}}, at most MAX_BYTES as sent
+ * settings_misses  key: [a-z0-9_]{1,64} or "*" (the rest), count: an integer >= 1;
+ *                  at most SettingsAudit::MAX_KEYS keys and "*"
+ * ```
+ *
+ * A heartbeat without `audit` (today's agent), or with one that is not the
+ * above, changes nothing. A new report is written only when it differs from
+ * the stored one: the row is read for every request already, so an
+ * unchanged report costs no query, on the cluster bus or not.
+ */
+final class NodeAudit {
+	use DatabaseAware;
+
+	/** Largest `audit` taken, as MAIN encodes it. */
+	public const MAX_BYTES = 16384;
+
+	/**
+	 * The heartbeat's `audit` as MAIN keeps it, or null when it is not one
+	 * (the stored report stays). Entries that are not a settings name and a
+	 * count are dropped; past SettingsAudit::MAX_KEYS names, the least missed
+	 * count under "*".
+	 *
+	 * @return array{settings_misses: array<string, int>}|null
+	 */
+	public static function normalise(mixed $rAudit): ?array {
+		if (!is_array($rAudit) || !is_array($rAudit['settings_misses'] ?? null) || strlen((string) json_encode($rAudit, JSON_PARTIAL_OUTPUT_ON_ERROR)) > self::MAX_BYTES) {
+			return null;
+		}
+		$rMisses = [];
+		foreach ($rAudit['settings_misses'] as $rKey => $rCount) {
+			$rKey = (string) $rKey;
+			if (is_int($rCount) && $rCount > 0 && ($rKey === SettingsAudit::OTHER || SettingsAudit::name($rKey) === $rKey)) {
+				$rMisses[$rKey] = $rCount;
+			}
+		}
+		return ['settings_misses' => SettingsAudit::top($rMisses)];
+	}
+
+	/** The stored form: `{"settings_misses": {…}}`, an object even when empty. */
+	public static function encode(array $rAudit): string {
+		return (string) json_encode(['settings_misses' => (object) $rAudit['settings_misses']]);
+	}
+
+	/**
+	 * Keep a heartbeat's `audit` with its node when it is one and differs
+	 * from what the row holds. Before migration 045 the row has no `audit`,
+	 * and nothing is written. A failed write is tried again at the next
+	 * heartbeat, which still differs.
+	 *
+	 * @param array<string, mixed> $rNode cluster_nodes row, as the request read it
+	 */
+	public static function record(array $rNode, mixed $rAudit): void {
+		$rDoc = self::normalise($rAudit);
+		if ($rDoc === null || !array_key_exists('audit', $rNode)) {
+			return;
+		}
+		$rJson = self::encode($rDoc);
+		if ($rJson === $rNode['audit']) {
+			return;
+		}
+		try {
+			NodeRegistry::update((int) $rNode['server_id'], ['audit' => $rJson]);
+		} catch (\Throwable) {
+			// The next heartbeat tries again.
+		}
+	}
+
+	/**
+	 * Each node's last reported settings misses, for the Cluster Nodes page
+	 * (server id => {key: count}); a node that reported none is absent.
+	 *
+	 * @return array<int, array<string, int>>
+	 */
+	public static function settingsMisses(): array {
+		try {
+			$rRows = self::db()->query('SELECT `server_id`, `audit` FROM `cluster_nodes` WHERE `audit` IS NOT NULL;') ? self::db()->get_rows() : [];
+		} catch (\Throwable) {
+			$rRows = []; // before migration 045
+		}
+		$rOut = [];
+		foreach ($rRows ?: [] as $rRow) {
+			$rDoc = json_decode((string) $rRow['audit'], true);
+			if (is_array($rDoc) && is_array($rDoc['settings_misses'] ?? null)) {
+				$rOut[(int) $rRow['server_id']] = $rDoc['settings_misses'];
+			}
+		}
+		return $rOut;
+	}
+}
@@ -96,12 +96,13 @@ $rWhen = static fn(?int $rTs): string => $rTs ? gmdate('Y-m-d H:i:s', $rTs) . '
                    <th><?= $language::get('cluster_token_expires'); ?></th>
                    <th><?= $language::get('cluster_last_seen'); ?></th>
                    <th><?= $language::get('cluster_agent'); ?></th>
+                    <th title="<?= $language::get('cluster_settings_misses_help'); ?>"><?= $language::get('cluster_settings_misses'); ?></th>
                    <th><?= $language::get('actions'); ?></th>
                </tr>
            </thead>
            <tbody>
                <?php if (empty($clusterNodes)): ?>
-                    <tr><td colspan="15" class="text-center text-body-secondary"><?= $language::get('cluster_no_nodes'); ?></td></tr>
+                    <tr><td colspan="16" class="text-center text-body-secondary"><?= $language::get('cluster_no_nodes'); ?></td></tr>
                <?php endif; ?>
                <?php foreach ($clusterNodes as $rNode): ?>
                    <tr>
@@ -140,6 +141,22 @@ $rWhen = static fn(?int $rTs): string => $rTs ? gmdate('Y-m-d H:i:s', $rTs) . '
                        <td><?= $rWhen($rNode['token_exp'] === null ? null : (int) $rNode['token_exp']); ?></td>
                        <td><?= $rWhen($rNode['last_seen_at'] === null ? null : intdiv((int) $rNode['last_seen_at'], 1000)); ?></td>
                        <td><?= htmlspecialchars((string) ($rNode['agent_version'] ?? '—'), ENT_QUOTES); ?></td>
+                        <td>
+                            <?php if ((int) $rNode['mode'] < 1 || !is_array($rNode['settings_misses'] ?? null)): ?>
+                                <span class="text-body-secondary">—</span>
+                            <?php elseif ($rNode['settings_misses'] === []): ?>
+                                <span class="badge bg-label-success">0</span>
+                            <?php else: ?>
+                                <details>
+                                    <summary><span class="badge bg-label-warning"><?= count($rNode['settings_misses']); ?></span></summary>
+                                    <ul class="list-unstyled small font-monospace mb-0">
+                                        <?php foreach ($rNode['settings_misses'] as $rKey => $rCount): ?>
+                                            <li><?= htmlspecialchars((string) $rKey, ENT_QUOTES); ?> × <?= (int) $rCount; ?></li>
+                                        <?php endforeach; ?>
+                                    </ul>
+                                </details>
+                            <?php endif; ?>
+                        </td>
                        <td class="text-nowrap">
                            <?php if ($rNode['state'] !== 'revoked'): ?>
                                <form method="POST" class="d-inline js-cluster-revoke">
@@ -152,6 +152,7 @@ CREATE TABLE IF NOT EXISTS `cluster_nodes` (
  `flows` int(10) unsigned NOT NULL DEFAULT '0',
  `root_ready` tinyint(1) NOT NULL DEFAULT '0',
  `features` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
+  `audit` text COLLATE utf8_unicode_ci DEFAULT NULL,
  `gen` int(10) unsigned NOT NULL DEFAULT '1',
  `node_sign_pub` binary(32) DEFAULT NULL,
  `node_box_pub` binary(32) DEFAULT NULL,
@@ -0,0 +1,2 @@
+-- Reverse 045_add_cluster_node_audit.sql.
+ALTER TABLE `cluster_nodes` DROP COLUMN IF EXISTS `audit`;
@@ -0,0 +1,5 @@
+-- What a node's heartbeat last reported of its own audit, as MAIN keeps it
+-- (JSON, Domain\Cluster\NodeAudit): today {"settings_misses": {key: count}},
+-- the settings reads outside the replica's allowlist over the node's last
+-- seven days. Shown on the Cluster Nodes page.
+ALTER TABLE `cluster_nodes` ADD COLUMN IF NOT EXISTS `audit` text COLLATE utf8_unicode_ci DEFAULT NULL AFTER `features`;
@@ -12,6 +12,8 @@ use XcVm\Core\Cluster\Crypto\PanelSig;
 use XcVm\Core\Cluster\Crypto\Seal;
 use XcVm\Core\Config\OpensslExtra;
 use XcVm\Core\Config\SettingsManager;
+use XcVm\Core\Cluster\SettingsAudit;
+use XcVm\Domain\Cluster\ClusterAdmin;
 use XcVm\Domain\Cluster\ClusterApi;
 use XcVm\Domain\Cluster\ClusterBus;
 use XcVm\Domain\Cluster\ClusterClock;
@@ -20,6 +22,7 @@ use XcVm\Domain\Cluster\ClusterPolicy;
 use XcVm\Domain\Cluster\ClusterSemaphore;
 use XcVm\Domain\Cluster\EnrolmentService;
 use XcVm\Domain\Cluster\HeartbeatService;
+use XcVm\Domain\Cluster\NodeAudit;
 use XcVm\Domain\Cluster\NodeHealth;
 use XcVm\Domain\Cluster\NodeRegistry;
 use XcVm\Domain\Cluster\NonceStore;
@@ -73,6 +76,7 @@ final class ClusterApiTest extends TestCase {
 		$this->rDb->exec('ALTER TABLE `cluster_node_epochs` ADD COLUMN `agent_eph_pub` binary(32) DEFAULT NULL');
 		$this->rDb->exec('ALTER TABLE `cluster_nodes` ADD COLUMN `root_ready` tinyint(1) NOT NULL DEFAULT 0');
 		$this->rDb->exec('ALTER TABLE `cluster_nodes` ADD COLUMN `features` varchar(255) DEFAULT NULL');
+		$this->rDb->exec('ALTER TABLE `cluster_nodes` ADD COLUMN `audit` text DEFAULT NULL');
 		$this->rDb->exec('CREATE TABLE `servers` (`id` INTEGER PRIMARY KEY, `status` int NOT NULL DEFAULT 0)');
 		$this->rDb->exec('INSERT INTO `servers` (`id`, `status`) VALUES (5, 0)');
 		DatabaseFactory::set($this->rDb);
@@ -470,6 +474,50 @@ final class ClusterApiTest extends TestCase {
 		$this->assertCount(4, $rRecorded());
 	}

+	public function testAHeartbeatKeepsTheNodesAuditWhenItChanges(): void {
+		$rKeys = $this->active();
+		$rAudit = ['settings_misses' => ['rare_key' => 1, 'hot_key' => 40, 'Bad Key' => 3, 'neg_key' => -1, 'float_key' => 1.5, '*' => 2]];
+		[$rRes, $rCtx] = $this->call('heartbeat', ['audit' => $rAudit], 1, $rKeys);
+		$this->reply($rRes, $rCtx, $rKeys);
+		$this->assertSame('{"settings_misses":{"hot_key":40,"rare_key":1,"*":2}}', NodeRegistry::byServer(self::SID)['audit'], 'only names and counts, most missed first');
+		$this->assertSame([self::SID => ['hot_key' => 40, 'rare_key' => 1, '*' => 2]], NodeAudit::settingsMisses());
+		$this->assertSame(['hot_key' => 40, 'rare_key' => 1, '*' => 2], ClusterAdmin::nodes([], 30)[0]['settings_misses'], 'on the Cluster Nodes page');
+
+		// The same report again, today's agent (no audit) and a malformed one write nothing.
+		$rLog = new QueryLogDb($this->rDb);
+		DatabaseFactory::set($rLog);
+		foreach ([['audit' => $rAudit], [], ['audit' => 'misses'], ['audit' => ['settings_misses' => 'x']], ['audit' => ['settings_misses' => ['k' => 1], 'pad' => str_repeat('x', NodeAudit::MAX_BYTES)]]] as $rPayload) {
+			[$rRes, $rCtx] = $this->call('heartbeat', $rPayload, 1, $rKeys);
+			$this->reply($rRes, $rCtx, $rKeys);
+		}
+		$this->assertSame([], array_values(array_filter($rLog->writes(), static fn(string $rSql): bool => str_contains($rSql, '`audit`'))), 'the audit is not written again');
+		$this->assertSame(['hot_key' => 40, 'rare_key' => 1, '*' => 2], NodeAudit::settingsMisses()[self::SID]);
+
+		// Nothing missed any more: an empty report clears the list.
+		[$rRes, $rCtx] = $this->call('heartbeat', ['audit' => ['settings_misses' => new \stdClass()]], 1, $rKeys);
+		$this->reply($rRes, $rCtx, $rKeys);
+		$this->assertSame('{"settings_misses":{}}', NodeRegistry::byServer(self::SID)['audit']);
+		$this->assertSame([], ClusterAdmin::nodes([], 30)[0]['settings_misses']);
+	}
+
+	public function testAnAuditOverTheCapKeepsTheMostMissed(): void {
+		$rMisses = [];
+		for ($i = 0; $i < SettingsAudit::MAX_KEYS + 3; $i++) {
+			$rMisses[sprintf('key_%03d', $i)] = $i + 1;
+		}
+		$rDoc = NodeAudit::normalise(['settings_misses' => $rMisses + ['*' => 5], 'sql_connects' => 9]);
+		$this->assertCount(SettingsAudit::MAX_KEYS + 1, $rDoc['settings_misses']);
+		$this->assertSame(sprintf('key_%03d', SettingsAudit::MAX_KEYS + 2), array_key_first($rDoc['settings_misses']));
+		$this->assertSame(5 + 1 + 2 + 3, $rDoc['settings_misses']['*'], 'the three least missed, with the node\'s own rest');
+		$this->assertSame(['settings_misses'], array_keys($rDoc), 'members MAIN does not know yet are not kept');
+		$this->assertNull(NodeAudit::normalise(null));
+		$this->assertNull(NodeAudit::normalise(['sql_connects' => 1]));
+		// Before migration 045 the row has no `audit`: nothing to write, nothing to show.
+		NodeAudit::record(['server_id' => self::SID], ['settings_misses' => ['k' => 1]]);
+		$this->rDb->exec('CREATE TABLE `bare` (`x` int)');
+		$this->assertSame([], NodeAudit::settingsMisses());
+	}
+
 	public function testHelloFromAnotherInstanceQuarantines(): void {
 		$rKeys = $this->active();
 		$this->peer(6, ReplicaBuilder::FEATURE_CONFIG_CHANGED);
@@ -4,13 +4,13 @@ use PHPUnit\Framework\TestCase;

 /**
 * The cluster schema exists twice: as the core migrations from 028 on for
- * upgrades (028–043, then later ones that ALTER a cluster table) and in
+ * upgrades (028â045, then later ones that ALTER a cluster table) and in
 * database.sql for fresh installs. Both were loaded into MariaDB 10.11 and
 * compared column by column when written; this test keeps them from drifting
 * where CI has no database.
 */
 final class ClusterSchemaTest extends TestCase {
-	private const MIGRATIONS = ['028_add_cluster_settings', '029_create_cluster_nodes', '030_create_cluster_commands', '031_create_cluster_enrolment', '032_create_cluster_audit', '033_add_crontab_role', '034_create_cluster_changes', '035_add_cluster_epoch_eph', '036_add_cluster_enrol_request_eph', '037_enable_cluster_cron', '038_add_cluster_endpoint_settings', '039_add_cluster_node_root_ready', '040_add_cluster_db_allowlist', '041_add_cluster_node_features', '042_crontab_cleanup_role_all', '043_crontab_main_roles'];
+	private const MIGRATIONS = ['028_add_cluster_settings', '029_create_cluster_nodes', '030_create_cluster_commands', '031_create_cluster_enrolment', '032_create_cluster_audit', '033_add_crontab_role', '034_create_cluster_changes', '035_add_cluster_epoch_eph', '036_add_cluster_enrol_request_eph', '037_enable_cluster_cron', '038_add_cluster_endpoint_settings', '039_add_cluster_node_root_ready', '040_add_cluster_db_allowlist', '041_add_cluster_node_features', '042_crontab_cleanup_role_all', '043_crontab_main_roles', '045_add_cluster_node_audit'];

 	private function src(string $rPath): string {
 		return (string) file_get_contents(dirname(__DIR__, 2) . '/src/' . $rPath);
@@ -205,6 +205,7 @@ final class ReplicaBootTest extends TestCase {
 			}
 			$rC = ServiceContainer::getInstance();
 			$rSettings = SettingsManager::getAll();
+			SettingsManager::get('zz_not_carried'); // a read the replica would not answer (SettingsAudit)
 			echo json_encode([
 				'replica' => ReplicaBoot::active(),
 				'db' => get_class($GLOBALS['db']),
@@ -315,6 +316,9 @@ final class ReplicaBootTest extends TestCase {
 			if (!$rWebApi) {
 				$this->assertSame([true, true, [3 => ['id' => 3]]], [$rDump['container_db'], $rDump['container_settings'], $rDump['container_bouquets']]);
 			}
+			// The boot itself read only allowlisted settings; the process's one miss
+			// reached the audit.json the agent sends, at its exit.
+			$this->assertSame(['settings_misses' => ['zz_not_carried' => $rWebApi ? 2 : 1]], json_decode((string) @file_get_contents($this->rHome . 'config/cluster/audit.json'), true));
 		}
 	}
 }
@@ -0,0 +1,154 @@
+<?php
+
+use PHPUnit\Framework\TestCase;
+use XcVm\Core\Cluster\NodeFlows;
+use XcVm\Core\Cluster\SettingsAudit;
+use XcVm\Core\Config\SettingsManager;
+
+/**
+ * `audit.settings_misses` on the node (cluster plan, section 9, R1
+ * `settings`): on a node in mode 1 or 2, a read through SettingsManager of a
+ * key outside lb_settings_keys.php is counted, without a database: per
+ * process, merged into a per-day file, reported for the last seven days in
+ * the audit.json the agent sends with its heartbeats. Bounded everywhere;
+ * MAIN, legacy nodes and mode 0 count nothing.
+ */
+final class SettingsAuditTest extends TestCase {
+	private string $rDir;
+
+	private const NOW = 1800000000; // 2027-01-15 08:00 UTC
+
+	protected function setUp(): void {
+		$this->rDir = sys_get_temp_dir() . '/xcvm-misses-' . bin2hex(random_bytes(4)) . '/';
+		mkdir($this->rDir . 'cluster', 0777, true);
+		$this->mode(1);
+		SettingsManager::set(['server_name' => 'Panel', 'api_pass' => 'x']);
+	}
+
+	protected function tearDown(): void {
+		SettingsAudit::useDir(false);
+		NodeFlows::usePath(null);
+		SettingsManager::set([]);
+		exec('rm -rf ' . escapeshellarg($this->rDir));
+	}
+
+	/** This node's mode as its agent wrote it (null: no agent file), and a process that decides afresh. */
+	private function mode(?int $rMode): void {
+		@unlink($this->rDir . 'flows.json');
+		if ($rMode !== null) {
+			file_put_contents($this->rDir . 'flows.json', json_encode(['mode' => $rMode, 'flows' => 63, 'state' => 'active']));
+		}
+		NodeFlows::usePath($this->rDir . 'flows.json');
+		SettingsAudit::useDir($this->rDir . 'misses/', $this->rDir . 'cluster/');
+	}
+
+	/** @return array<string, int>|null */
+	private function day(int $rAt = self::NOW): ?array {
+		$rDay = json_decode((string) @file_get_contents($this->rDir . 'misses/' . gmdate('Ymd', $rAt) . '.json'), true);
+		return is_array($rDay) ? $rDay : null;
+	}
+
+	/** @return array<string, mixed>|null */
+	private function published(): ?array {
+		$rDoc = json_decode((string) @file_get_contents($this->rDir . 'cluster/audit.json'), true);
+		return is_array($rDoc) ? $rDoc : null;
+	}
+
+	public function testAnApiNodeCountsReadsTheReplicaWouldNotAnswer(): void {
+		SettingsManager::get('server_name');
+		SettingsManager::get('api_pass'); // withheld: a known read, the secret travels apart or not at all
+		SettingsManager::get('not_a_setting');
+		SettingsManager::getBool('not_a_setting');
+		SettingsManager::getInt('another_key', 3);
+		SettingsManager::getString('another_key');
+		SettingsManager::getArray('third_key');
+		SettingsManager::has('third_key');
+		SettingsAudit::flush(self::NOW);
+		$this->assertSame(['not_a_setting' => 2, 'another_key' => 2, 'third_key' => 2], $this->day());
+		$this->assertSame(['settings_misses' => ['another_key' => 2, 'not_a_setting' => 2, 'third_key' => 2]], $this->published(), 'most missed first, then by name');
+
+		// Another process the same day adds to the file.
+		SettingsAudit::useDir($this->rDir . 'misses/', $this->rDir . 'cluster/');
+		SettingsManager::get('third_key');
+		SettingsAudit::flush(self::NOW + 60);
+		$this->assertSame(['not_a_setting' => 2, 'another_key' => 2, 'third_key' => 3], $this->day());
+		$this->assertSame(['third_key' => 3, 'another_key' => 2, 'not_a_setting' => 2], SettingsAudit::summary(self::NOW + 60));
+	}
+
+	public function testMainALegacyNodeAndModeZeroCountNothing(): void {
+		foreach ([null, 0] as $rMode) {
+			file_put_contents($this->rDir . 'cluster/audit.json', '{"settings_misses":{"old":1}}');
+			$this->mode($rMode);
+			SettingsManager::get('not_a_setting');
+			SettingsAudit::flush(self::NOW);
+			$this->assertNull($this->day());
+			$this->assertFalse(SettingsAudit::publish(null, self::NOW));
+			$this->assertFileDoesNotExist($this->rDir . 'cluster/audit.json', 'nothing counted: nothing for the agent to send');
+		}
+		// Off (the suite's default): nothing at all, not even a decision.
+		SettingsAudit::useDir(false);
+		$this->mode(2);
+		SettingsAudit::useDir(false, $this->rDir . 'cluster/');
+		SettingsManager::get('not_a_setting');
+		SettingsAudit::flush(self::NOW);
+		$this->assertNull($this->day());
+		$this->assertFalse(SettingsAudit::publish(null, self::NOW));
+	}
+
+	public function testEveryCountIsBounded(): void {
+		for ($i = 0; $i < SettingsAudit::MAX_KEYS + 6; $i++) {
+			SettingsManager::get(sprintf('key_%03d', $i));
+		}
+		SettingsManager::get('Not A Name!');
+		SettingsAudit::flush(self::NOW);
+		$rDay = $this->day();
+		$this->assertCount(SettingsAudit::MAX_KEYS + 1, $rDay);
+		$this->assertSame(7, $rDay[SettingsAudit::OTHER], 'past the cap, and a key that is not a name');
+
+		// A day at the cap keeps its names; a new key counts under OTHER.
+		SettingsAudit::useDir($this->rDir . 'misses/', $this->rDir . 'cluster/');
+		SettingsManager::get('key_000');
+		SettingsManager::get('brand_new');
+		SettingsAudit::flush(self::NOW);
+		$rDay = $this->day();
+		$this->assertSame([2, 8], [$rDay['key_000'], $rDay[SettingsAudit::OTHER]]);
+		$this->assertArrayNotHasKey('brand_new', $rDay);
+
+		// The report: at most MAX_KEYS names over the days, the rest under OTHER, last.
+		file_put_contents($this->rDir . 'misses/' . gmdate('Ymd', self::NOW - 86400) . '.json', json_encode(['yesterday_key' => 50]));
+		$rSummary = SettingsAudit::summary(self::NOW);
+		$this->assertCount(SettingsAudit::MAX_KEYS + 1, $rSummary);
+		$this->assertSame(['yesterday_key', 'key_000'], array_slice(array_keys($rSummary), 0, 2));
+		$this->assertSame(SettingsAudit::OTHER, array_key_last($rSummary));
+		$this->assertSame(8 + 1, $rSummary[SettingsAudit::OTHER], 'OTHER, and the one name that no longer fits');
+		$this->assertLessThan(16384, strlen((string) file_get_contents($this->rDir . 'cluster/audit.json')));
+	}
+
+	public function testTheReportCoversSevenDaysAndOlderDaysArePruned(): void {
+		foreach ([0 => 'today', 6 => 'six_days', 7 => 'seven_days', 9 => 'nine_days'] as $rDays => $rKey) {
+			@mkdir($this->rDir . 'misses/', 0777, true);
+			file_put_contents($this->rDir . 'misses/' . gmdate('Ymd', self::NOW - $rDays * 86400) . '.json', json_encode([$rKey => 1]));
+		}
+		$this->assertSame(['six_days' => 1, 'today' => 1], SettingsAudit::summary(self::NOW));
+		$this->assertSame(1, SettingsAudit::prune(8, self::NOW));
+		$this->assertFileDoesNotExist($this->rDir . 'misses/' . gmdate('Ymd', self::NOW - 9 * 86400) . '.json');
+		$this->assertTrue(SettingsAudit::publish(null, self::NOW));
+		$this->assertSame(['settings_misses' => ['six_days' => 1, 'today' => 1]], $this->published());
+
+		// Nothing missed: an empty object, which clears what MAIN shows.
+		exec('rm -rf ' . escapeshellarg($this->rDir . 'misses'));
+		$this->assertTrue(SettingsAudit::publish(null, self::NOW));
+		$this->assertSame('{"settings_misses":{}}', file_get_contents($this->rDir . 'cluster/audit.json'));
+		// No agent here (MAIN, a legacy node): nothing written.
+		$this->assertFalse(SettingsAudit::publish($this->rDir . 'no-agent/', self::NOW));
+	}
+
+	public function testALongRunningProcessMergesEveryMinute(): void {
+		SettingsManager::get('first_key');
+		$this->assertNull($this->day(time()), 'held in memory');
+		(new \ReflectionProperty(SettingsAudit::class, 'rFlushedAt'))->setValue(null, time() - SettingsAudit::FLUSH_EVERY);
+		SettingsManager::get('first_key');
+		$this->assertSame(['first_key' => 2], $this->day(time()));
+		$this->assertSame(['settings_misses' => ['first_key' => 2]], $this->published());
+	}
+}
@@ -114,3 +114,8 @@ register_shutdown_function(static function () use ($ingestLockDir): void {
 // the same instant, so a cached section would outlive the test that built it:
 // off for the suite; a test that checks the caching gives it a directory.
 \XcVm\Core\Cluster\ReplicaEtagCache::useDir(false);
+
+// The settings-miss audit (SettingsAudit) counts reads on nodes in mode 1 or
+// 2, and many tests point NodeFlows at such a file: off for the suite; a
+// test that checks it gives it its own directories.
+\XcVm\Core\Cluster\SettingsAudit::useDir(false);