diff --git a/docs/adr/0004-cluster-api.md b/docs/adr/0004-cluster-api.md index d24f20eb..b4d2627a 100644 --- a/docs/adr/0004-cluster-api.md +++ b/docs/adr/0004-cluster-api.md @@ -1237,11 +1237,11 @@ It grants like every whole section: without a licence the extension refuses to s | `core.bouquets`, `core.categories`, `bouquets`, `categories` | those caches as they are, or `[]`: no section carries them yet (R2) | | the xc_vm crontab (once per boot) | only the jobs the replica owns (`ReplicaApply::crontabText(null)`); otherwise left as it is | -Once a process booted this way (`ReplicaBoot::active`), `SettingsRepository::getAll` and `ServerRepository::getAll` never read MAIN's database, forced or not. Any other query opens it lazily, on first use: that is the connect `ConnectAudit` counts and mode 2's refusal (not built) will refuse. +Once a process booted this way (`ReplicaBoot::active`), `SettingsRepository::getAll` and `ServerRepository::getAll` never read MAIN's database, forced or not. Any other query opens it lazily, on first use: that is the connect `ConnectAudit` counts, and since the eighth Phase 7 increment mode 2's refusal refuses it. **Who boots from the replica.** `BootKernel::resolve` decides for the CLI profile, `WebApiBootstrap::coreStages` for the web API endpoints: -- **A node in mode 2**, active or quarantined, by the agent's `flows.json` (`NodeFlows::declared`: the file alone, read before the settings, the servers or a database handle exist). Only once an apply built the replica's settings and servers caches since the reboot (`ReplicaBoot::ready`: both in `replica_owned`). Until then `ReplicaStage` runs `DatabaseStage` and `LegacyCoreStage` itself, so a mode 2 node without a replica boots as before, and the refusal will make that boot the fail-closed answer. +- **A node in mode 2**, active or quarantined, by the agent's `flows.json` (`NodeFlows::declared`: the file alone, read before the settings, the servers or a database handle exist). Only once an apply built the replica's settings and servers caches since the reboot (`ReplicaBoot::ready`: both in `replica_owned`). Until then `ReplicaStage` runs `DatabaseStage` and `LegacyCoreStage` itself, so a mode 2 node without a replica boots as before; since the eighth Phase 7 increment the refusal makes that boot fail closed. - **`cluster:apply`**, in every mode (`console.php` passes `ReplicaBoot::forArgv`, which answers `always` for it): its work is to build those caches. With CONFIG on it needs no database at all, so the agent's applies go on with MAIN's MariaDB stopped. In shadow, its comparison with MAIN's crontab and RTMP publishers reads MAIN's database on first use, as before. - **Everything else boots exactly as before:** nodes in mode 0 and 1, MAIN, the admin UI (`BootContext::Admin`), the streaming entry points (`BootContext::Stream`, `StreamingRequestBootstrap`, `initStreaming`), and a caller that passes `replica => false`. A stale `flows.json` on MAIN cannot move it off its database: no apply built replica caches there. @@ -1261,7 +1261,7 @@ Without `--from-disk` nothing changes: the agent runs `cluster:apply` right afte - It counts on a node in mode 1 or 2, by `flows.json` alone (decided once per process; MAIN and legacy nodes have no file and count nothing). A read of a key outside `lb_settings_keys.php` is a miss; the allowlist's `withheld` keys are known reads, not misses. A read of a known key costs one array lookup. - A process keeps its counts in memory, at most 64 names and the rest under `*`. It merges them at exit, and at most every 60 s while it runs, into `STORAGE_PATH/cluster/settings_misses/YYYYMMDD.json` (UTC day, `{key: count}`, 64 names and `*`) under the file's lock. A miss thus costs one locked merge per process: per request under PHP-FPM, which keeps no state between requests. The merge rewrites `config/cluster/audit.json` only when it added a key to the day or the file is at least 60 s old, so requests do not queue on it; the counts in it may lag until the next such merge or `cron:cleanup`. - `audit.json` is `{"settings_misses": {key: count}}` over the last seven UTC days, today included: most missed first, then by name, at most 64 names, the rest (and every key that is not `[a-z0-9_]{1,64}`) under `*`, last. `{}` when nothing was missed. It is written only where the agent's directory exists, atomically, and removed in mode 0. `cron:cleanup` prunes day files older than eight days and rewrites it every hour, so days that leave the window drop out. -- Nothing reaches a database; a failed write is dropped (the next merge tries again). A root process (`cron:root_signals` runs root-only commands that read settings) makes the day directory one level at a time, 0750, and hands each level and file it makes, `audit.json` included, to the owner of the agent's directory (xc_vm). `service` also makes `storage/cluster/` xc_vm's at boot, since an LB build ships no `storage/`. A process that cannot read the days (a level it may not search, a day file it may not read) writes no report and leaves `audit.json` as it is. +- Nothing reaches a database; a failed write is dropped (the next merge tries again). A root process (`cron:root_signals` runs root-only commands that read settings) makes the day directory one level at a time, 0750, and hands each level and file it makes, `audit.json` included, to the owner of the agent's directory (xc_vm). Since the eighth Phase 7 increment root does this work as that user instead (`SettingsAudit::asAgentUser`), and no longer writes or chowns with its own rights. `service` also makes `storage/cluster/` xc_vm's at boot, since an LB build ships no `storage/`. A process that cannot read the days (a level it may not search, a day file it may not read) writes no report and leaves `audit.json` as it is. - The allowlist's CI scan (`tools/ci/lb_settings_keys.php`) now reads every `ADD` of an `ALTER TABLE` on `settings`, not only the first. Migration 016 adds three columns in one statement, so `update_channel_bin` and `update_channel_fanout`, which `UpdateChannels` reads on a node, were missing: they are in the settings section now, and no miss. **`audit.settings_misses` on MAIN.** `Domain/Cluster/NodeAudit`: the heartbeat's `audit` is kept in `cluster_nodes.audit` (migration 045, mirrored in `database.sql`) as `{"settings_misses": {…}}`: @@ -1285,7 +1285,7 @@ Without `--from-disk` nothing changes: the agent runs `cluster:apply` right afte - Mode 2 is not switched on yet, and its refusal is not built: in a process booted from the replica, a query outside the settings and servers still opens MAIN's database. Since the eighth Phase 7 increment such a query is refused. - No section carries the bouquets, categories, proxies or allowed-IPs caches (R2). A mode 2 node's `cron:cache` still builds them from MAIN's database, and during a MAIN outage it stops at the first such read. The agent's `cluster:apply` still applies the replica then. - After a reboot the streaming endpoints have no stream definitions until MAIN answers: they live in `tmp/`, a tmpfs, and no section carries them (R2 `streams`, not built). The same holds in mode 2. In mode 1 the daemons and crons also still boot through MAIN's database (the watchdog waits for it), as the plan has it for hybrid mode. -- A mode 2 process that boots before an apply built the caches after a reboot (a cron in the first minute, before `service`'s apply) boots through MAIN's database. +- A mode 2 process that boots before an apply built the caches after a reboot (a cron in the first minute, before `service`'s apply) boots through MAIN's database. Since the eighth Phase 7 increment that boot is refused: the process fails closed. - After a re-enrolment (new node keys) or a new panel root, the stored records no longer verify, and `--from-disk` applies none of them until the agent fetches them again. Today's agent keeps its ETags and never does while the data is unchanged (contract below). - The misses reach MAIN only once the agent sends `audit.json`; today's agent does not, and the page shows `—`. The heartbeat's other audit counters (`audit.sql_connects`, `audit.redis_connects`, `audit.sites`) are not reported yet; the same `audit` object is meant to carry them. It does since the eighth Phase 7 increment. @@ -1323,10 +1323,10 @@ Without `--from-disk` nothing changes: the agent runs `cluster:apply` right afte - **The boot.** A mode 2 node rebooted before its first apply falls back to `DatabaseStage` (seventh increment), whose connect is now refused: the process ends at its boot, the refusal in the panel's error log and in the audit. `cluster:apply` boots from the replica and is not affected, so `service`'s `cluster:apply --from-disk` builds the caches and the next processes boot from them. - **A manual trace** (`XCVM_CONNECT_AUDIT=1`, or `STORAGE_PATH/cluster/sql_audit/enabled`) still counts in mode 0; nothing wrote that file, so until now nothing was counted. -**Direct connects.** Sixteen places built `new DatabaseHandler()`, each an eager connect. The eight a load balancer runs now take `DatabaseFactory::open()`, which builds the same handle and keeps it as the process's (`DatabaseFactory::set` was called at each): `LegacyCoreStage`'s reconnect, `Public/admin/{live,thumb,timeshift,vod}.php` and the enigma2, xplugin and playlist API controllers. `DatabaseFactory` and `DatabaseStage` keep theirs; the other five (`Public/admin/api.php`, `proxy_api.php`, `ActiveCodeApiController`, the setup view and `migration_logic.php`) are MAIN-only. `ArchitectureTest` enforces it, reading the Makefile's `LB_DIRS_TO_REMOVE` and `LB_FILES_TO_REMOVE` for what is MAIN-only and skipping comments: +**Direct connects.** Fifteen places built `new DatabaseHandler()`, each an eager connect. The eight a load balancer runs now take `DatabaseFactory::open()`, which builds the same handle and keeps it as the process's (`DatabaseFactory::set` was called at each): `LegacyCoreStage`'s reconnect, `Public/admin/{live,thumb,timeshift,vod}.php` and the enigma2, xplugin and playlist API controllers. `DatabaseFactory` and `DatabaseStage` keep theirs; the other five (`Public/admin/api.php`, `proxy_api.php`, `ActiveCodeApiController`, the setup view and `migration_logic.php`) are MAIN-only. `ArchitectureTest` enforces it, reading the Makefile's `LB_DIRS_TO_REMOVE` and `LB_FILES_TO_REMOVE` for what is MAIN-only and skipping comments (and, for the second rule, string literals): - no `new Database(`/`new DatabaseHandler(` outside `DatabaseFactory`, `DatabaseStage`, migrations and MAIN-only files; -- in load balancer code, `\XC_VM::db_connect(` and `new \PDO(` only in `Core/Database/Database.php`, `\XC_VM::redis_connect(` only in `RedisManager`, `new \Redis(` only in `RedisCache`, and each of those three calls `ConnectAudit::guard(`. +- in load balancer code, `XC_VM::db_connect(` and `new PDO` (or `PDO::connect(`) only in `Core/Database/Database.php`, `XC_VM::redis_connect(` only in `RedisManager`, `new Redis` only in `RedisCache`, and each of those three calls `ConnectAudit::guard(`; `new mysqli`, `mysqli_connect(`, `mysqli_real_connect(` and `mysqli_init(` nowhere. PHP's names are case-insensitive and a global-namespace file needs no leading backslash, so each pattern matches any case, with or without it. **The audit.** `STORAGE_PATH/cluster/sql_audit/` (the plan's `var/cluster/sql_audit/`; `STORAGE_PATH` survives reboots): @@ -1339,8 +1339,9 @@ since unix seconds: when this node's audit began ``` - **A site** is the first caller outside the connect machinery (`Database`, `DatabaseHandler`, `LazyDatabaseHandler`, `DatabaseFactory`, `RedisManager`, `RedisCache`): `:`, relative to `MAIN_HOME`. Its key is ` :`, every byte outside printable ASCII replaced by `?`, at most 160 bytes: a longer path keeps its end after `...`. A boot's connect names `Core/Bootstrap/Stage/DatabaseStage.php:`. -- **Bounded.** At most 32 sites a day, the rest under `*`. Eight days are kept (`cron:cleanup`, both files), so the directory stays under 9 MiB. A broken day file starts over. The counts never throw: an audit must not break the connect it audits. -- **Ownership.** As `SettingsAudit` (seventh increment), whose helpers it shares: a root process (`cron:root_signals` boots through MAIN's database in mode 1) makes each level 0750 and hands each level and file it makes to the owner of `config/cluster/`. +- **Bounded.** At most 32 sites a day, the rest under `*`. Eight days are kept (`cron:cleanup`, both files, before its first query, so in mode 2 too), so the directory stays under 9 MiB. A broken day file starts over. The counts never throw: an audit must not break the connect it audits. +- **Reads.** A count rewrites its day file in place under the file's exclusive lock (truncated, then written). The report reads each day file under its shared lock (`SettingsAudit::readDay`, the settings misses' too), so a report built while another process counts never sees a day emptied, and its counts never go down. +- **Root.** A root process (`cron:root_signals`, `startup` and `cluster:root` boot through MAIN's database in mode 1) counts as the owner of `config/cluster/` (xc_vm), and merges its settings misses and writes `audit.json` the same way: `SettingsAudit::asAgentUser` switches its effective gid, its groups (`initgroups`) and its uid to that user's around the file work, and back after. The kernel then applies xc_vm's permissions, so a link xc_vm planted where it can write leads root nowhere xc_vm could not go, and what root makes (levels 0750, files) is xc_vm's without a `chown`. Root writes nothing when `config/cluster/` is missing, a link or root's, when its owner has no passwd entry or root's group, or when a switch fails. Until this increment's review, root wrote as root and handed what it made over with `chown`, following links in directories xc_vm can write. **The report.** The last seven UTC days, today included, go into the `audit.json` the agent sends as the heartbeat's `audit` (seventh increment), beside `settings_misses`: @@ -1351,8 +1352,8 @@ since unix seconds: when this node's audit began - `sql_connects`, `redis_connects`: integers ≥ 0, attempts, refused ones included. - `sites`: most first, then by name, at most 32 names, the rest (and `*`) last under `*`. `{}` when there were none. -- `connects_since`: when the node's audit began. It is written at the first report in mode 1 or 2 and removed with `audit.json` in mode 0, so a node back in mode 1 starts a new window. The seven-day gate needs both: zero connects, and a `connects_since` at least seven days old. -- **When.** A connect rewrites `audit.json` when it adds a site to its day or the file is at least 60 s old; `cron:cleanup` rewrites it every hour, so a node without connects reports zeros. A process that cannot read the days (a level it may not search, a day file it may not read) leaves the report as it is. The file is written with unescaped slashes. Its largest form (64 settings keys and 32 sites of 160 bytes, with ten-digit counts) stays under 11 KiB, within the 16 KiB the agent sends and MAIN takes. +- `connects_since`: when the node's audit began. It is written at the first report in mode 1 or 2. Mode 0 removes it with `audit.json` and deletes the days it counted (a manual trace's days, counted without it, stay), so a node back in mode 1 starts a new window from nothing. The counts start at its UTC day: an older day file is left out of the report. The seven-day gate needs both: zero connects, and a `connects_since` at least seven days old. +- **When.** A connect rewrites `audit.json` when it adds a site to its day or the file is at least 60 s old; `cron:cleanup` rewrites it every hour, before its first query (so in mode 2 too), and a node without connects reports zeros. A process that cannot read the days (a level it may not search, a day file it may not read) leaves the report as it is. The file is written with unescaped slashes. Its largest form (64 settings keys and 32 sites of 160 bytes, with ten-digit counts) stays under 11 KiB, within the 16 KiB the agent sends and MAIN takes. **On MAIN.** `Domain/Cluster/NodeAudit` keeps the connect members in `cluster_nodes.audit` with the misses, as the seventh increment keeps those: only with an object `settings_misses`, and only when `sql_connects` and `redis_connects` are integers ≥ 0 and `sites` an object, all three or none. Sites that are not `(sql|redis) ` of at most 160 bytes (or `*`) with an integer count ≥ 1 are dropped; past 32 the least counted fold into `*`. A `connects_since` that is not an integer ≥ 1 is dropped alone. The row is written only when the report changed. The Cluster Nodes page adds a column for nodes in mode 1 or 2: `—` without a report, else `SQL n · Redis n` (green at zero), when the count began, and the sites. @@ -1365,15 +1366,17 @@ since unix seconds: when this node's audit began **How it differs from the plan.** - The plan names a log; this keeps exact per-day counts beside it and bounds both, since a mode 1 node logs every boot and stream request. The report's window and `connects_since` are not in the plan: the gate needs to know a count covers seven days. -- Refusing graceful callers, the MAIN build check, and `DatabaseFactory::open()` for the direct sites are not in the plan. It counts 14 direct sites; there were 16, five of them MAIN-only. -- `ArchitectureTest` also covers the lower-level connects (`\XC_VM`, PDO, `\Redis`), which the plan leaves out. +- Refusing graceful callers, the MAIN build check, and `DatabaseFactory::open()` for the direct sites are not in the plan. It counts 14 direct sites; there were 15, five of them MAIN-only. +- `ArchitectureTest` also covers the lower-level connects (`\XC_VM`, PDO, `\Redis`, mysqli), which the plan leaves out. +- The plan does not say who writes the audit. Root counts as xc_vm, so it never writes where xc_vm can plant a link. - The plan's `blocklist_sync` root action is not built. The replica's minute sync already applies the blocklist; the flush, a `node.root` action since Phase 4, now runs. - Moving the flush off the row is only feasible where MAIN sends root commands (COMMANDS and the pin). Elsewhere the row stays. **Known limits.** - Mode 2 cannot be switched on yet (`lb_new_node_mode = api` waits for the cutover phase), and a node in mode 2 still has paths that need MAIN's database. The refusal stops them, counted: - - `cron:root_signals` reads the `signals` table for its loop (the crontab and sysctl checks are inside it), so it ends there, before the fanout and agent keepalives. + - `cron:root_signals` reads MAIN's `signals` table. Without COMMANDS or root's pin its first read is the flush-row poll at the top, which stops it before the iptables sync and the fanout and agent keepalives. With both (the intended mode 2 setup) it runs the iptables sync (from the replica's cache with CONFIG on) and the keepalives, and stops at the signals loop: the signal actions, the ramdisk and ports reconciliation, the crontab and sysctl checks and `close_mysql` are lost. A server IP that differs from the replica's, which it writes to `servers`, stops it before the keepalives too. + - `cron:cleanup`'s stream, archive and VOD checks read `streams` from MAIN's database whenever `cleanup` (on by default) or `check_vod` is on. Its audit pruning and hourly `audit.json` run before them. - Root actions log to `mysql_syslog` through MAIN's database, several before acting (reboot, restarting or stopping the services), so `cluster:root` reports them refused and those never act. - `cron:cache` builds the bouquets, categories, proxies and allowed-IPs caches from MAIN's database (R2). - The watchdog waits for MAIN's database. @@ -1385,11 +1388,11 @@ since unix seconds: when this node's audit began **Tests.** -- `DbConnectRefusalTest`: who refuses and who counts, per mode and state, and a switch taking effect at the next connect; MAIN's build never refusing a stray `flows.json`. In this process every path is refused before `\XC_VM` is asked, the first three naming their caller: `new DatabaseHandler()`, `DatabaseFactory::open()` (keeping no handle), a lazy handle's first query, a graceful `db_connect`, `reconnect`, `db_explicit_connect`, `RedisManager::connect` and `instance`, and `RedisCache`. Each is counted and logged with `r`, even with the audit forced off. In a child PHP, the real bootstrap in a throwaway deploy root with an `\XC_VM` that logs each connect: a mode 2 node's boot and Redis are refused without one connect, the boot's site counted; MAIN, mode 0 and mode 1 connect as before, mode 1 counted. -- `ConnectAuditTest`: MAIN and mode 0 write nothing, a manual trace still counts; the caller as the site through the lazy handle; the caps (sites, site length and bytes, the log); a broken day file; the seven-day window and the eight-day pruning; `audit.json` with both audits, rewritten for a new site or once a minute, `connects_since` kept and reset through mode 0; zeros in a report without connects; the ranking across days; the largest report within 16 KiB; the hooks in both connect paths. As root: every level and file handed to the agent's user, a publish as that user reading them, and a day file it cannot read leaving the report as it is. -- `ArchitectureTest`: the two rules above. `ReplicaBootTest`: a mode 2 node before its first apply is refused at boot, the CLI and the web API alike. -- `ClusterApiTest`: a heartbeat's connect counters stored with the misses and shown on the page; malformed counters dropping the connect report and keeping the misses; a bad `connects_since` dropped alone; the sites cap. -- `ClusterRootCommandTest`: `rootCommandsFromMain()` per flows and pin; the `flush` case, and the row read only where MAIN still sends it. `NodeRpcActionsTest`: every root action, `flush` included, is a case of `executeAction()`. +- `DbConnectRefusalTest`: who refuses and who counts, per mode and state, and a switch taking effect at the next connect; MAIN's build never refusing a stray `flows.json`. In this process, on a fixed clock, every path is refused before `\XC_VM` is asked, the first three and `RedisCache` naming their caller: `new DatabaseHandler()`, `DatabaseFactory::open()` (keeping no handle), a lazy handle's first query, a graceful `db_connect`, `reconnect`, `db_explicit_connect`, `RedisManager::connect` and `instance`, and `RedisCache`. Each is counted and logged with `r`, even with the audit forced off. In a child PHP, the real bootstrap in a throwaway deploy root with an `\XC_VM` that logs each connect: a mode 2 node's boot and Redis are refused without one connect, the boot's site counted; MAIN, mode 0 and mode 1 connect as before, mode 1 counted; `DatabaseFactory::open()` keeps the handle it built (an SQLite one there) as the process's. +- `ConnectAuditTest`: MAIN and mode 0 write nothing, a manual trace still counts; the caller as the site through the lazy handle; the caps (sites, site length and bytes, the log); a broken day file; the seven-day window and the eight-day pruning; `audit.json` with both audits, rewritten at once for a new site however fresh, else once a minute, `connects_since` kept and reset through mode 0; the counts starting at `connects_since`'s day (mode 1, then 0, then 1 again reports none of the first); zeros in a report without connects; the ranking across days; the largest report within 16 KiB; the hooks in both connect paths; a report read while another process counts 2000 times never going down (`SettingsAuditTest` the same for the misses). As root: every level and file made as the agent's user, root's rights back after; links planted in the day directories and in `config/cluster/` (a dangling day file, the log, the report's temporary file, a misses day file) leaving a root-only directory as it was; nothing written when `config/cluster/` is root's; a publish as that user reading them, and a day file it cannot read leaving the report as it is. A suite run as root hands each test's tree to nobody (`Tests\Support\AgentUser`), as a node's is xc_vm's. +- `ArchitectureTest`: the two rules above, and the connect patterns fed each form they must catch (any case, with or without the backslash, `new Redis;`, `Pdo\Mysql`, mysqli) and a few they must not (`new RedisCache(`, `RedisManager::connect()`). `ReplicaBootTest`: a mode 2 node before its first apply is refused at boot, the CLI and the web API alike. `NodeRoleTest`: `cron:cleanup` prunes the audits and publishes `audit.json` before its first query. +- `ClusterApiTest`: a heartbeat's connect counters stored with the misses and shown on the page; malformed counters dropping the connect report and keeping the misses; a bad `connects_since` (not an integer, or 0) dropped alone; the sites cap. +- `ClusterRootCommandTest`: `rootCommandsFromMain()` per flows, and with COMMANDS on, with and without a pin root trusts; the `flush` action run through `executeAction()` with iptables stubbed and a database that refuses every other statement: flushed and saved once each, one `FLUSH` line, nothing of the `reboot` case below it; the row read only where MAIN still sends it. `NodeRpcActionsTest`: every root action, `flush` included, is a case of `executeAction()`. **The agent's contract (XC_VM_Fanout).** No change is needed: diff --git a/docs/en/development/cluster-seams.md b/docs/en/development/cluster-seams.md index 4bac016e..c24a14f2 100644 --- a/docs/en/development/cluster-seams.md +++ b/docs/en/development/cluster-seams.md @@ -59,8 +59,12 @@ action. `connectLazy()` or `open()`. Every connect to MAIN's MySQL or Redis passes `ConnectAudit::guard()`: on a node in mode 1 or 2 it is counted with its caller and shown on *Servers → Cluster Nodes*, and in mode 2 it throws - `LbDatabaseAccessException`. Do not open PDO or `\Redis` connections of your - own in code a load balancer runs. + `LbDatabaseAccessException`. Do not open PDO, `\Redis` or mysqli connections + of your own in code a load balancer runs. +- The node's audit files (`storage/cluster/`, `config/cluster/audit.json`) sit + where xc_vm can write. A root process writes them only inside + `SettingsAudit::asAgentUser()`, which does the work as xc_vm, never with + root's own rights. Tests that pin these rules: `SignalDispatcherParityTest`, `StreamStateWriterTest`, `StreamRowMergeTest`, `StreamCacheBuilderSourceTest`, `LogSinkTest`, diff --git a/docs/superpowers/specs/2026-09-21-main-lb-api-communication-design.md b/docs/superpowers/specs/2026-09-21-main-lb-api-communication-design.md index 95564eb8..18e2c38c 100644 --- a/docs/superpowers/specs/2026-09-21-main-lb-api-communication-design.md +++ b/docs/superpowers/specs/2026-09-21-main-lb-api-communication-design.md @@ -1003,8 +1003,8 @@ Names below are the ones in the code (panel `src/`, agent in XC_VM_Fanout); ADR - The `secrets` section: `live_streaming_pass` and OPENSSL_EXTRA, each as `{kid, current, previous, previous_valid_until}`. The kid is a keyed fingerprint of the value, and OPENSSL_EXTRA's previous value is MAIN's `openssl_extra.prev` while it is open. It is served whole by ETag like the others and licence-gated as they are, but only to an active node in mode 1 or 2, and it is read for each request, never cached on MAIN. With CONFIG on, `cluster:apply` writes OPENSSL_EXTRA to `config/openssl_extra` (0600), keeping MAIN's previous value until `previous_valid_until`. It also makes the `settings` section authoritative with it: the settings cache is built from both, in the settings loader's shape, and `SettingsRepository::getAll` stops reading MAIN's database. In shadow the report names what differs, never a value (`ReplicaBuilderSecretsTest`, `ReplicaApplyTest`, `ClusterApiTest`, `OpensslExtraTest`; ADR 0004, sixth Phase 7 increment, with the agent's contract). MAIN never builds a section, the blocklist included, from a failed read, a missing settings row or an unset secret: the `config` op answers `503 DB` and the node keeps what it holds. A node refuses a settings section that is not a whole row, and `cluster:apply` exits 3 when writing OPENSSL_EXTRA fails. The other secrets the allowlist withholds are then absent from a CONFIG node's settings, so its `cron:maxmind` updates from the free GeoLite2 release. - `ReplicaStage` and the boot from disk. A node in mode 2 (active or quarantined, by its agent's `flows.json`) boots its CLI and web API through `ReplicaStage` once an apply built the replica's settings and servers caches since the reboot; until then, and on mode 0 and 1 nodes and MAIN, the boot is unchanged. It leaves the globals and container entries `DatabaseStage` and `LegacyCoreStage` leave, from the caches, with a lazy database handle, and its settings and servers never come from MAIN's database. `cluster:apply` boots that way in every mode, so with CONFIG on it needs no database. With `--from-disk`, which `service` runs at boot, it takes each section from the record the agent stored, once that opens with the node's key and verifies under the pinned panel key (`ReplicaRecords`). A node rebooted while MAIN is unreachable thus rebuilds its R1 caches (settings, servers, crontab, blocklist, secrets) from disk, the whole sections before the blocklist and without resolving RTMP publishers' names. Serving streams after such a reboot waits for R2: the stream definitions live in `tmp/`, a tmpfs, and no section carries them yet (`ReplicaBootTest`, `ReplicaRecordsTest`; ADR 0004, seventh Phase 7 increment, with the agent's contract). - `audit.settings_misses`. On a node in mode 1 or 2, a read through `SettingsManager` of a key outside `lb_settings_keys.php` is counted without a database, per UTC day. The last seven days go to `config/cluster/audit.json` (at most 64 keys and `*`) for the agent to send as the heartbeat's `audit`. MAIN keeps it in `cluster_nodes.audit` (migration 045) when it changes and shows it on the Cluster Nodes page (`SettingsAuditTest`, `ClusterApiTest`). The allowlist's scan reads every `ADD` of an `ALTER TABLE` on `settings`, so `update_channel_bin` and `update_channel_fanout` (migration 016) travel in the settings section. -- The mode-2 refusal and the connect audit. Every connect to MAIN's MySQL or Redis passes `ConnectAudit::guard()` (`db_connect`, `db_explicit_connect`, `RedisManager::connect`, `RedisCache`). On a node in mode 1 or 2 it is counted per UTC day with its site (`sql|redis path:line`, at most 32 a day and `*`), and logged up to 1 MiB a day under `STORAGE_PATH/cluster/sql_audit/`, eight days kept. In mode 2 (active or quarantined, by `flows.json`) it is then refused with `LbDatabaseAccessException` before anything opens, graceful callers included; never on MAIN's build, which ships `Public/cluster/index.php`. The LB-shipped `new DatabaseHandler()` sites take `DatabaseFactory::open()`, and `ArchitectureTest` forbids new ones outside `DatabaseFactory`, `DatabaseStage`, migrations and MAIN-only files, and any other MySQL or Redis connect in LB code. The heartbeat's `audit` carries `sql_connects`, `redis_connects`, `sites` and `connects_since` for the last seven days, which MAIN keeps and shows on the Cluster Nodes page. The blocklist flush runs as the `node.root` command MAIN already sent (it was acked and ignored), and a node that takes root commands stops polling `signals` for it (`DbConnectRefusalTest`, `ConnectAuditTest`, `ArchitectureTest`, `ClusterApiTest`, `ClusterRootCommandTest`; ADR 0004, eighth Phase 7 increment, with the agent's contract). -- **Not built yet:** the R2 `streams` section (the stream bundles, `cluster_stream_ver` and the section-hash resync), the `secrets` section's ticket kids (with Phase 8's tickets) and a previous `live_streaming_pass` (with the Phase 9 rotation), and the paths a mode 2 node still takes to MAIN's database, which the refusal now stops: `cron:root_signals`' signals loop (before its keepalives), root actions' `mysql_syslog` lines, the watchdog's wait and `cluster:apply`'s shadow comparison. Mode 1 still boots through MAIN's database, so its `sql_connects` is never zero and the seven-day gate cannot pass until mode 1 boots from its replica. No section carries the bouquets, categories, proxies or allowed-IPs caches, which `cron:cache` builds from MAIN's database (refused in mode 2); the stream endpoints keep their boot, and mode 2 refuses its queries. The agent's half of the new whole sections, `secrets` included, of `config.changed`, of the heartbeat's `audit`, and of refetching a stored record that no longer verifies after new keys (XC\_VM\_Fanout). No PHP on the node reads the `cluster` section, the `node` section's nginx settings (`cloudflare`, `mag_legacy_redirect`) are not applied yet, and the `servers` section's node list waits for Phase 8's ticket checks. The root flush still arrives as a `signals` row on a node without the COMMANDS flow or root's pin, and the plan's `blocklist_sync` root action is not built. +- The mode-2 refusal and the connect audit. Every connect to MAIN's MySQL or Redis passes `ConnectAudit::guard()` (`db_connect`, `db_explicit_connect`, `RedisManager::connect`, `RedisCache`). On a node in mode 1 or 2 it is counted per UTC day with its site (`sql|redis path:line`, at most 32 a day and `*`), and logged up to 1 MiB a day under `STORAGE_PATH/cluster/sql_audit/`, eight days kept; a root process does that file work as the agent's user (xc\_vm), and a report reads each day under its lock. In mode 2 (active or quarantined, by `flows.json`) it is then refused with `LbDatabaseAccessException` before anything opens, graceful callers included; never on MAIN's build, which ships `Public/cluster/index.php`. The LB-shipped `new DatabaseHandler()` sites take `DatabaseFactory::open()`, and `ArchitectureTest` forbids new ones outside `DatabaseFactory`, `DatabaseStage`, migrations and MAIN-only files, and any other `\XC_VM`, PDO, `\Redis` or mysqli connect in LB code, in any case, with or without the leading backslash. The heartbeat's `audit` carries `sql_connects`, `redis_connects`, `sites` and `connects_since` for the last seven days, which MAIN keeps and shows on the Cluster Nodes page. The blocklist flush runs as the `node.root` command MAIN already sent (it was acked and ignored), and a node that takes root commands stops polling `signals` for it (`DbConnectRefusalTest`, `ConnectAuditTest`, `ArchitectureTest`, `ClusterApiTest`, `ClusterRootCommandTest`; ADR 0004, eighth Phase 7 increment, with the agent's contract). +- **Not built yet:** the R2 `streams` section (the stream bundles, `cluster_stream_ver` and the section-hash resync), the `secrets` section's ticket kids (with Phase 8's tickets) and a previous `live_streaming_pass` (with the Phase 9 rotation), and the paths a mode 2 node still takes to MAIN's database, which the refusal now stops: `cron:root_signals`' signals loop (after its iptables sync and keepalives with COMMANDS and root's pin; without them its flush-row poll stops it before those), `cron:cleanup`'s stream and archive checks (after its audit pruning), root actions' `mysql_syslog` lines, the watchdog's wait and `cluster:apply`'s shadow comparison. Mode 1 still boots through MAIN's database, so its `sql_connects` is never zero and the seven-day gate cannot pass until mode 1 boots from its replica. No section carries the bouquets, categories, proxies or allowed-IPs caches, which `cron:cache` builds from MAIN's database (refused in mode 2); the stream endpoints keep their boot, and mode 2 refuses its queries. The agent's half of the new whole sections, `secrets` included, of `config.changed`, of the heartbeat's `audit`, and of refetching a stored record that no longer verifies after new keys (XC\_VM\_Fanout). No PHP on the node reads the `cluster` section, the `node` section's nginx settings (`cloudflare`, `mag_legacy_redirect`) are not applied yet, and the `servers` section's node list waits for Phase 8's ticket checks. The root flush still arrives as a `signals` row on a node without the COMMANDS flow or root's pin, and the plan's `blocklist_sync` root action is not built. **Phase 8: Data plane (\~4.5 pw).** diff --git a/src/Cli/CronJobs/CleanupCronJob.php b/src/Cli/CronJobs/CleanupCronJob.php index 345d65ae..01df2743 100644 --- a/src/Cli/CronJobs/CleanupCronJob.php +++ b/src/Cli/CronJobs/CleanupCronJob.php @@ -57,6 +57,15 @@ class CleanupCronJob implements CommandInterface { private function loadCron(): void { global $db; + // First, and without a database: on a node in mode 2 the queries below + // are refused (ConnectAudit::guard), and this cron ends at the first. + // This node's connect audit: the cutover gate reads seven days of it. + ConnectAudit::prune(8); + // Its settings misses: the days that left the report's window drop out + // of the audit.json its agent sends. + SettingsAudit::prune(8); + SettingsAudit::publish(); + if (intval(SettingsManager::get('cleanup')) == 1) { $rStreams = []; $db->query('SELECT `id` FROM `streams` LEFT JOIN `streams_servers` ON `streams_servers`.`stream_id` = `streams`.`id` WHERE `streams`.`type` IN (1,3,4) AND `streams_servers`.`server_id` = ?;', SERVER_ID); @@ -212,13 +221,6 @@ class CleanupCronJob implements CommandInterface { } } - // This node's connect audit: the cutover gate reads seven days of it. - ConnectAudit::prune(8); - // Its settings misses: the days that left the report's window drop out - // of the audit.json its agent sends. - SettingsAudit::prune(8); - SettingsAudit::publish(); - // Retention of cluster-wide log tables: MAIN's job. Every LB used to // run the same DELETEs against MAIN's database each minute. if (!NodeRole::isMain()) { diff --git a/src/Cli/CronJobs/RootSignalsCronJob.php b/src/Cli/CronJobs/RootSignalsCronJob.php index 09ba9431..9076b3ea 100644 --- a/src/Cli/CronJobs/RootSignalsCronJob.php +++ b/src/Cli/CronJobs/RootSignalsCronJob.php @@ -119,12 +119,13 @@ class RootSignalsCronJob implements CommandInterface { } } - private function flushIPs(): void { + /** Flush iptables and the flood guard's block files (protected: a test records the call). */ + protected function flushIPs(): void { exec('sudo iptables -F && sudo ip6tables -F'); shell_exec('sudo rm ' . FLOOD_TMP_PATH . 'block_*'); } - private function saveiptables(): void { + protected function saveiptables(): void { exec('sudo iptables-save && sudo ip6tables-save'); } diff --git a/src/Core/Cluster/ConnectAudit.php b/src/Core/Cluster/ConnectAudit.php index 5fbeca6a..180ea667 100644 --- a/src/Core/Cluster/ConnectAudit.php +++ b/src/Core/Cluster/ConnectAudit.php @@ -30,9 +30,10 @@ namespace XcVm\Core\Cluster; * which the seven days need. The last seven days go into the audit.json the * agent sends as the heartbeat's `audit` (SettingsAudit::publish, report()), * rewritten when a connect adds a site to its day or the file is a minute - * old, and by cron:cleanup every hour. A root process hands every level and - * file it makes to the agent's user, as SettingsAudit does. Nothing here - * throws but the refusal. + * old, and by cron:cleanup every hour. A root process counts as the owner of + * the agent's directory (xc_vm), as SettingsAudit does + * (SettingsAudit::asAgentUser), and a day file is read under its lock. + * Nothing here throws but the refusal. * * Lives in Core: it ships to LBs, where Domain\Cluster does not. * @@ -127,23 +128,27 @@ final class ConnectAudit { if ($rDir === null || !in_array($rKind, [self::SQL, self::REDIS], true)) { return; } + $rNow = self::$rNow ?? time(); try { - if (!is_dir($rDir) && !SettingsAudit::makeDir($rDir)) { - return; - } - $rNow = self::$rNow ?? time(); - $rDay = $rDir . gmdate('Ymd', $rNow); - $rKey = self::siteKey($rKind, $rSite); - $rNew = self::count($rDay . '.json', $rKind, $rKey); - self::log($rDay . '.ndjson', ['t' => $rNow, 'k' => $rKind, 's' => substr($rKey, strlen($rKind) + 1), 'p' => getmypid()] + ($rRefused ? ['r' => 1] : [])); - $rAgentDir = SettingsAudit::agentDir(); - if ($rAgentDir === null) { - return; - } - clearstatcache(true, $rAgentDir . 'audit.json'); - if ($rNew || (int) @filemtime($rAgentDir . 'audit.json') <= $rNow - self::PUBLISH_EVERY) { - SettingsAudit::publish($rAgentDir, $rNow); - } + // Root counts as xc_vm: these directories are xc_vm's to write. + SettingsAudit::asAgentUser(static function () use ($rDir, $rKind, $rSite, $rRefused, $rNow): bool { + if (!SettingsAudit::makeDir($rDir)) { + return false; + } + $rDay = $rDir . gmdate('Ymd', $rNow); + $rKey = self::siteKey($rKind, $rSite); + $rNew = self::count($rDay . '.json', $rKind, $rKey); + self::log($rDay . '.ndjson', ['t' => $rNow, 'k' => $rKind, 's' => substr($rKey, strlen($rKind) + 1), 'p' => getmypid()] + ($rRefused ? ['r' => 1] : [])); + $rAgentDir = SettingsAudit::agentDir(); + if ($rAgentDir === null) { + return true; + } + clearstatcache(true, $rAgentDir . 'audit.json'); + if ($rNew || (int) @filemtime($rAgentDir . 'audit.json') <= $rNow - self::PUBLISH_EVERY) { + SettingsAudit::publish($rAgentDir, $rNow); + } + return true; + }); } catch (\Throwable) { // Counted, not needed: the connect goes on. } @@ -202,9 +207,10 @@ final class ConnectAudit { /** * This node's connects for audit.json: `sql_connects`, `redis_connects` * and `sites` over the last WINDOW_DAYS days, and `connects_since`, when - * the audit began (the window starts no earlier). [] when there is no - * directory (the members are left out); null when this process cannot - * read the days (the report there stays as it is). + * the audit began. The counts start at its UTC day: an earlier day in the + * window is left out. [] when there is no directory (the members are left + * out); null when this process cannot read the days (the report there + * stays as it is). * * @return array|null */ @@ -212,20 +218,27 @@ final class ConnectAudit { if (self::dir() === null) { return []; } - $rSum = self::days(self::WINDOW_DAYS, $rNow); + $rSince = self::since($rNow); + $rSum = self::days(self::WINDOW_DAYS, $rNow, $rSince ?? 0); if ($rSum === null) { return null; } $rOut = ['sql_connects' => $rSum['sql'], 'redis_connects' => $rSum['redis'], 'sites' => (object) $rSum['sites']]; - $rSince = self::since($rNow); return $rSince === null ? $rOut : $rOut + ['connects_since' => $rSince]; } - /** Mode 0: the audit is over, and a later one starts its window afresh. */ + /** + * Mode 0: the audit is over. Its start goes, and the days it counted with + * it, so a later audit starts from nothing. Days a manual trace counted + * without an audit (no `since`) stay. + */ public static function forget(): void { $rDir = self::dir(); - if ($rDir !== null) { - @unlink($rDir . 'since'); + if ($rDir === null || !@unlink($rDir . 'since')) { + return; + } + foreach (array_keys(self::dayFiles($rDir)) as $rFile) { + @unlink($rFile); } } @@ -237,20 +250,34 @@ final class ConnectAudit { } $rCut = gmdate('Ymd', ($rNow ?? self::$rNow ?? time()) - $rKeepDays * 86400); $rCount = 0; - foreach (array_merge(glob($rDir . '*.json') ?: [], glob($rDir . '*.ndjson') ?: []) as $rFile) { - if (preg_match('/^(\d{8})\.(?:json|ndjson)$/', basename($rFile), $rMatch) && $rMatch[1] < $rCut && @unlink($rFile)) { + foreach (self::dayFiles($rDir) as $rFile => $rDay) { + if ($rDay < $rCut && @unlink($rFile)) { $rCount++; } } return $rCount; } + /** + * The day files in $rDir, counts and logs: path => YYYYMMDD. + * + * @return array + */ + private static function dayFiles(string $rDir): array { + $rOut = []; + foreach (array_merge(glob($rDir . '*.json') ?: [], glob($rDir . '*.ndjson') ?: []) as $rFile) { + if (preg_match('/^(\d{8})\.(?:json|ndjson)$/', basename($rFile), $rMatch)) { + $rOut[$rFile] = $rMatch[1]; + } + } + return $rOut; + } + /** * Add one connect at $rKey to a day's counts, under the file's lock. * True when the site (or OTHER) is new to the day. */ private static function count(string $rFile, string $rKind, string $rKey): bool { - $rMade = !is_file($rFile); $rHandle = @fopen($rFile, 'c+'); if ($rHandle === false) { return false; @@ -272,9 +299,6 @@ final class ConnectAudit { flock($rHandle, LOCK_UN); fclose($rHandle); } - if ($rMade) { - SettingsAudit::own($rFile); - } return $rNew; } @@ -290,9 +314,6 @@ final class ConnectAudit { return; } @file_put_contents($rFile, json_encode($rLine, JSON_UNESCAPED_SLASHES) . "\n", FILE_APPEND | LOCK_EX); - if ($rSize === false) { - SettingsAudit::own($rFile); - } } /** @@ -321,14 +342,16 @@ final class ConnectAudit { } /** - * The last $rDays days' counts, summed, the sites ranked. Null when this - * process cannot tell: the directory (or the nearest level above it) is - * not one it may search, or a day file is not one it may read. An absent - * day is a day without a connect. + * The last $rDays days' counts, summed, the sites ranked; a day before + * $rFrom's is left out. Null when this process cannot tell: the directory + * (or the nearest level above it) is not one it may search, or a day file + * is not one it may read. An absent day is a day without a connect. Each + * day file is read under its lock (SettingsAudit::readDay), so a count + * rewriting it is never seen half done. * * @return array{sql: int, redis: int, sites: array}|null */ - private static function days(int $rDays, int $rNow): ?array { + private static function days(int $rDays, int $rNow, int $rFrom = 0): ?array { $rOut = ['sql' => 0, 'redis' => 0, 'sites' => []]; $rDir = self::dir(); if ($rDir === null) { @@ -337,12 +360,17 @@ final class ConnectAudit { if (!SettingsAudit::searchable($rDir)) { return null; } + $rFirst = gmdate('Ymd', $rFrom); for ($d = 0; $d < $rDays; $d++) { - $rFile = $rDir . gmdate('Ymd', $rNow - $d * 86400) . '.json'; + $rDate = gmdate('Ymd', $rNow - $d * 86400); + if ($rDate < $rFirst) { + break; + } + $rFile = $rDir . $rDate . '.json'; if (!is_file($rFile)) { continue; } - $rRaw = @file_get_contents($rFile); + $rRaw = SettingsAudit::readDay($rFile); if ($rRaw === false) { return null; } @@ -369,10 +397,9 @@ final class ConnectAudit { if ($rSince > 0) { return $rSince; } - if ((!is_dir($rDir) && !SettingsAudit::makeDir($rDir)) || @file_put_contents($rFile, (string) $rNow) === false) { + if (!SettingsAudit::makeDir($rDir) || @file_put_contents($rFile, (string) $rNow) === false) { return null; } - SettingsAudit::own($rFile); return $rNow; } } diff --git a/src/Core/Cluster/ReplicaBoot.php b/src/Core/Cluster/ReplicaBoot.php index e73119b8..406d2f6c 100644 --- a/src/Core/Cluster/ReplicaBoot.php +++ b/src/Core/Cluster/ReplicaBoot.php @@ -12,7 +12,8 @@ use XcVm\Core\Cache\FileCache; * - A node in mode 2 (api) boots its CLI processes and web API endpoints * from the caches its replica built (ReplicaApply), once an apply has * built the settings and servers caches since the last reboot. Until then - * it boots as before, through MAIN's database. + * it falls back to the boot through MAIN's database, whose connect mode 2 + * refuses: the process ends at its boot (fails closed). * - `cluster:apply` always boots from the replica, whatever the mode: its * work is to build those caches, at boot too (`service` runs it with * `--from-disk` before the daemons), when MAIN may be unreachable. @@ -21,9 +22,9 @@ use XcVm\Core\Cache\FileCache; * Once a process has booted from the replica (start()), its settings and * servers never come from MAIN's database: SettingsRepository and * ServerRepository answer the caches however old, or nothing. Any other - * query still opens MAIN's database, lazily, on first use: that is the - * connect ConnectAudit counts, and that mode 2 refuses once the refusal - * exists (plan, section 10, step 1). + * query opens MAIN's database, lazily, on first use: that is the connect + * ConnectAudit counts, and on a node in mode 2 refuses (plan, section 10, + * step 1). */ final class ReplicaBoot { /** Commands that boot from the replica in every mode. */ diff --git a/src/Core/Cluster/SettingsAudit.php b/src/Core/Cluster/SettingsAudit.php index f44e967e..b2e07971 100644 --- a/src/Core/Cluster/SettingsAudit.php +++ b/src/Core/Cluster/SettingsAudit.php @@ -31,9 +31,11 @@ namespace XcVm\Core\Cluster; * day file per process: per request under PHP-FPM, which keeps no state * between requests. * - * A root process hands every directory level and file it makes to the user - * of the agent's directory (xc_vm), whose processes add to them; a report is - * written only from days this process can read. + * A root process does all of this as the owner of the agent's directory + * (xc_vm), whose processes add to the same files (asAgentUser): root never + * writes with its own rights where xc_vm can plant a link. A report is + * written only from days this process can read, and a day file is read under + * its lock, never half rewritten. * * Lives in Core: it ships to LBs, where Domain\Cluster does not. */ @@ -140,39 +142,41 @@ final class SettingsAudit { if ($rCounts === [] || $rDir === null) { return; } + $rAt = self::$rFlushedAt; try { - if (!is_dir($rDir) && !self::makeDir($rDir)) { - return; - } - $rFile = $rDir . gmdate('Ymd', self::$rFlushedAt) . '.json'; - $rHandle = @fopen($rFile, 'c+'); - if ($rHandle === false) { - return; - } - $rNewKey = false; - try { - flock($rHandle, LOCK_EX); - $rDay = json_decode((string) stream_get_contents($rHandle), true); - $rDay = is_array($rDay) ? $rDay : []; - $rMerged = self::merge($rDay, $rCounts); - $rNewKey = array_diff_key($rMerged, $rDay) !== []; - ftruncate($rHandle, 0); - rewind($rHandle); - fwrite($rHandle, (string) json_encode((object) $rMerged)); - fflush($rHandle); - } finally { - flock($rHandle, LOCK_UN); - fclose($rHandle); - } - self::own($rFile); - $rAgentDir = self::agentDir(); - if ($rAgentDir === null) { - return; - } - clearstatcache(true, $rAgentDir . 'audit.json'); - if ($rNewKey || (int) @filemtime($rAgentDir . 'audit.json') <= self::$rFlushedAt - self::FLUSH_EVERY) { - self::publish($rAgentDir, self::$rFlushedAt); - } + self::asAgentUser(static function () use ($rDir, $rCounts, $rAt): bool { + if (!self::makeDir($rDir)) { + return false; + } + $rHandle = @fopen($rDir . gmdate('Ymd', $rAt) . '.json', 'c+'); + if ($rHandle === false) { + return false; + } + $rNewKey = false; + try { + flock($rHandle, LOCK_EX); + $rDay = json_decode((string) stream_get_contents($rHandle), true); + $rDay = is_array($rDay) ? $rDay : []; + $rMerged = self::merge($rDay, $rCounts); + $rNewKey = array_diff_key($rMerged, $rDay) !== []; + ftruncate($rHandle, 0); + rewind($rHandle); + fwrite($rHandle, (string) json_encode((object) $rMerged)); + fflush($rHandle); + } finally { + flock($rHandle, LOCK_UN); + fclose($rHandle); + } + $rAgentDir = self::agentDir(); + if ($rAgentDir === null) { + return true; + } + clearstatcache(true, $rAgentDir . 'audit.json'); + if ($rNewKey || (int) @filemtime($rAgentDir . 'audit.json') <= $rAt - self::FLUSH_EVERY) { + self::publish($rAgentDir, $rAt); + } + return true; + }); } catch (\Throwable) { // Counted, not needed: the next flush tries again. } @@ -196,31 +200,33 @@ final class SettingsAudit { * nothing is counted, and the connect audit's window with it. False when * there is no agent (MAIN, a legacy node), the write failed, or this * process cannot read the days (a directory or day file another user's): - * the report there stays. + * the report there stays. Root writes it as $rAgentDir's owner + * (asAgentUser), and returns false when it cannot. */ public static function publish(?string $rAgentDir = null, ?int $rNow = null): bool { $rAgentDir ??= self::agentDir(); if ($rAgentDir === null || self::dir() === null || !is_dir($rAgentDir)) { return false; } - if (NodeFlows::declared()['mode'] < 1) { - @unlink($rAgentDir . 'audit.json'); - ConnectAudit::forget(); - return false; - } - $rNow ??= time(); - $rSum = self::days($rNow); - $rConnects = $rSum === null ? null : ConnectAudit::report($rNow); - if ($rSum === null || $rConnects === null) { - return false; - } - $rTmp = $rAgentDir . 'audit.json.' . getmypid() . '.tmp'; - if (@file_put_contents($rTmp, (string) json_encode(['settings_misses' => (object) self::top($rSum)] + $rConnects, JSON_UNESCAPED_SLASHES)) === false || !@rename($rTmp, $rAgentDir . 'audit.json')) { - @unlink($rTmp); - return false; - } - self::own($rAgentDir . 'audit.json', $rAgentDir); - return true; + return self::asAgentUser(static function () use ($rAgentDir, $rNow): bool { + if (NodeFlows::declared()['mode'] < 1) { + @unlink($rAgentDir . 'audit.json'); + ConnectAudit::forget(); + return false; + } + $rNow ??= time(); + $rSum = self::days($rNow); + $rConnects = $rSum === null ? null : ConnectAudit::report($rNow); + if ($rSum === null || $rConnects === null) { + return false; + } + $rTmp = $rAgentDir . 'audit.json.' . getmypid() . '.tmp'; + if (@file_put_contents($rTmp, (string) json_encode(['settings_misses' => (object) self::top($rSum)] + $rConnects, JSON_UNESCAPED_SLASHES)) === false || !@rename($rTmp, $rAgentDir . 'audit.json')) { + @unlink($rTmp); + return false; + } + return true; + }, $rAgentDir); } /** Delete day files older than $rKeepDays. Returns how many went. */ @@ -336,7 +342,7 @@ final class SettingsAudit { if (!is_file($rFile)) { continue; } - $rRaw = @file_get_contents($rFile); + $rRaw = self::readDay($rFile); if ($rRaw === false) { return null; } @@ -362,42 +368,92 @@ final class SettingsAudit { } /** - * Make $rDir level by level (0750), each level made handed to the agent's - * user: a level a root process left its own would shut xc_vm's processes - * out of every day file below it. (ConnectAudit's directory too.) + * A day file's contents, read under its shared lock: a merge rewrites the + * file in place under its exclusive lock (truncated, then written), and a + * read between the two would find a day without counts. False when it + * cannot be read. (ConnectAudit's day files too.) */ - public static function makeDir(string $rDir): bool { - $rMissing = []; - for ($rPath = rtrim($rDir, '/'); !is_dir($rPath); $rPath = dirname($rPath)) { - if (dirname($rPath) === $rPath) { - return false; - } - $rMissing[] = $rPath; + public static function readDay(string $rFile): string|false { + $rHandle = @fopen($rFile, 'r'); + if ($rHandle === false) { + return false; } - foreach (array_reverse($rMissing) as $rPath) { - if (@mkdir($rPath, 0750)) { - self::own($rPath); - } elseif (!is_dir($rPath)) { - return false; - } + try { + flock($rHandle, LOCK_SH); + return stream_get_contents($rHandle); + } finally { + flock($rHandle, LOCK_UN); + fclose($rHandle); } - return true; } /** - * What a root process made goes to the owner of the agent's directory - * (xc_vm on a node), whose processes add to it. (ConnectAudit's files too.) + * Make $rDir and each missing level above it, 0750. (ConnectAudit's + * directory too.) Root makes them under asAgentUser, as xc_vm: a level + * root's own would shut xc_vm's processes out of every day file below it. */ - public static function own(string $rPath, ?string $rAgentDir = null): void { + public static function makeDir(string $rDir): bool { + return is_dir($rDir) || @mkdir($rDir, 0750, true) || is_dir($rDir); + } + + /** + * Run $rWork with the rights of the owner of $rAgentDir (the agent's + * directory, config/cluster/: xc_vm on a node) and return what it returns. + * Anyone but root runs it as itself. Root switches its effective gid, its + * groups and its uid to that user's around it, and back after. The + * audits' files sit in directories xc_vm can write, where root must not + * create, truncate, append to or chown a file, nor follow a link xc_vm + * planted, with its own rights: root trusts nothing xc_vm can write, as + * RootPin has it. The kernel then applies xc_vm's permissions, and what + * root makes is xc_vm's, as xc_vm's processes need. (ConnectAudit's files + * too.) + * + * False, with nothing done, when root cannot become another user that + * way: no agent directory, one that is a link or root's, an owner without + * a passwd entry or whose group is root's, or a switch that fails. + * + * @param \Closure(): bool $rWork + */ + public static function asAgentUser(\Closure $rWork, ?string $rAgentDir = null): bool { if (!function_exists('posix_geteuid') || posix_geteuid() !== 0) { - return; + return $rWork(); } $rOf = $rAgentDir ?? self::agentDir(); - $rOwner = $rOf === null ? false : @fileowner($rOf); - $rGroup = $rOf === null ? false : @filegroup($rOf); - if ($rOwner !== false && $rGroup !== false) { - @chown($rPath, $rOwner); - @chgrp($rPath, $rGroup); + if ($rOf === null || !function_exists('posix_initgroups')) { + return false; + } + $rOf = rtrim($rOf, '/'); + clearstatcache(true, $rOf); + $rStat = @lstat($rOf); + $rUser = is_array($rStat) && ($rStat['mode'] & 0170000) === 0040000 && $rStat['uid'] !== 0 ? posix_getpwuid($rStat['uid']) : false; + if (!is_array($rUser) || (int) $rUser['gid'] === 0) { + return false; + } + $rUid = (int) $rStat['uid']; + $rGid = (int) $rUser['gid']; + $rRootGid = posix_getegid(); + $rRoot = posix_getpwuid(0); + $rRootName = is_array($rRoot) ? (string) $rRoot['name'] : 'root'; + // Groups and gid while still root, the uid last; back in the reverse order. + if (!posix_initgroups((string) $rUser['name'], $rGid) || !posix_setegid($rGid) || !posix_seteuid($rUid)) { + self::backToRoot($rRootGid, $rRootName); + return false; + } + try { + return $rWork(); + } finally { + self::backToRoot($rRootGid, $rRootName); + } + } + + /** + * Root's own ids back after asAgentUser: the uid first (the saved uid is + * root's), then the gid and the groups. Logged when that fails: the rest + * of the process would run without root's rights. + */ + private static function backToRoot(int $rGid, string $rName): void { + if (!posix_seteuid(0) || !posix_setegid($rGid) || !posix_initgroups($rName, $rGid)) { + error_log('XC_VM: the node audit could not restore root\'s user and groups'); } } } diff --git a/tests/Support/AgentUser.php b/tests/Support/AgentUser.php new file mode 100644 index 00000000..ce99312e --- /dev/null +++ b/tests/Support/AgentUser.php @@ -0,0 +1,28 @@ + $mainOnly */ private function isMainOnly(string $relative, array $mainOnly): bool { foreach ($mainOnly as $path) { @@ -284,38 +297,72 @@ final class ArchitectureTest extends TestCase { $this->assertSame([], $offenders, 'use DatabaseFactory::open() (or connect()/connectLazy()) instead of `new DatabaseHandler()`'); } + /** + * The lower-level MySQL and Redis connects, each with the one file that + * may open it, behind ConnectAudit::guard() (null: none may). PHP's class + * and function names are case-insensitive, and a global-namespace file + * (Public/stream/*.php) needs no leading backslash, so the patterns take + * both forms. + */ + private const LOW_LEVEL_CONNECTS = [ + '/(? 'Core/Database/Database.php', + '/\bnew\s+\\\\?PDO\b|(? 'Core/Database/Database.php', + '/(? 'Infrastructure/Redis/RedisManager.php', + '/\bnew\s+\\\\?Redis\b/i' => 'Core/Cache/RedisCache.php', + '/\bnew\s+\\\\?mysqli\b|(? null, + ]; + /** * The lower-level connects, in the code a load balancer runs: \XC_VM's - * database and Redis connects, and PDO and \Redis clients, only where - * ConnectAudit::guard() stands in front of them (a node in mode 2 is - * refused there, and modes 1 and 2 count every attempt). + * database and Redis connects, and PDO, \Redis and mysqli clients, only + * where ConnectAudit::guard() stands in front of them (a node in mode 2 + * is refused there, and modes 1 and 2 count every attempt). */ public function testLbShippedCodeConnectsOnlyBehindTheGuard(): void { if (!is_file(self::SRC_DIR . '/../Makefile')) { $this->markTestSkipped('needs the repo\'s Makefile (the LB build lists)'); } $mainOnly = $this->mainOnlyPaths(); - $guarded = [ - '/\\\\XC_VM::db_connect\s*\(/' => 'Core/Database/Database.php', - '/\bnew\s+\\\\?PDO\s*\(/' => 'Core/Database/Database.php', - '/\\\\XC_VM::redis_connect\s*\(/' => 'Infrastructure/Redis/RedisManager.php', - '/\bnew\s+\\\\?Redis\s*\(/' => 'Core/Cache/RedisCache.php', - ]; foreach ($this->srcFiles() as $relative => $content) { if ($this->isMainOnly($relative, $mainOnly)) { continue; } - foreach ($guarded as $pattern => $home) { + $code = $this->withoutStrings($content); + foreach (self::LOW_LEVEL_CONNECTS as $pattern => $home) { if ($relative !== $home) { - $this->assertDoesNotMatchRegularExpression($pattern, $content, $relative . ': connect through ' . $home); + $this->assertDoesNotMatchRegularExpression($pattern, $code, $relative . ': ' . ($home === null ? 'no such connect on a load balancer' : 'connect through ' . $home)); } } } - foreach (array_unique($guarded) as $home) { + foreach (array_unique(array_filter(self::LOW_LEVEL_CONNECTS)) as $home) { $this->assertStringContainsString('ConnectAudit::guard(', (string) file_get_contents(self::SRC_DIR . '/' . $home), $home); } } + /** The patterns above catch each way PHP can spell those connects, and nothing else. */ + public function testTheConnectPatternsCatchEveryForm(): void { + $caught = [ + '\\XC_VM::db_connect();', 'XC_VM::db_connect();', '\\xc_vm::DB_CONNECT ($a);', + '\\XC_VM::redis_connect();', 'xc_vm::redis_connect();', + 'new \\PDO($dsn);', 'new pdo($dsn);', 'new \\Pdo\\Mysql($dsn);', '\\PDO::connect($dsn);', + 'new \\Redis();', 'new redis;', 'new \\REDIS ;', + 'new mysqli(\'h\');', 'new \\MySQLi;', 'mysqli_connect(\'h\');', '\\mysqli_real_connect($m, \'h\');', 'MYSQLI_INIT();', + ]; + $clean = [ + 'new RedisCache($host);', 'RedisManager::connect();', 'new PDOStatement();', '$rDb->db_connect();', + 'Database::db_connect();', 'Other\\XC_VM::db_connect();', 'MyXC_VM::redis_connect();', '$m->real_connect();', + ]; + foreach ($caught as $code) { + $hits = array_filter(array_keys(self::LOW_LEVEL_CONNECTS), static fn(string $pattern): bool => preg_match($pattern, 'assertCount(1, $hits, $code); + } + foreach ($clean as $code) { + foreach (array_keys(self::LOW_LEVEL_CONNECTS) as $pattern) { + $this->assertDoesNotMatchRegularExpression($pattern, 'assertSame([], $rNode['settings_misses']); // Counters that are not counts: the settings misses are kept, the connect report is not. - foreach ([['sql_connects' => -1], ['redis_connects' => '0'], ['sites' => 'x'], ['connects_since' => 'yesterday']] as $rBad) { + foreach ([['sql_connects' => -1], ['redis_connects' => '0'], ['sites' => 'x'], ['connects_since' => 'yesterday'], ['connects_since' => 0]] as $rBad) { $rDoc = NodeAudit::normalise(array_merge($rAudit, $rBad)); $this->assertSame(isset($rBad['connects_since']) ? ['settings_misses', 'sql_connects', 'redis_connects', 'sites'] : ['settings_misses'], array_keys((array) $rDoc), json_encode($rBad)); } diff --git a/tests/Unit/ClusterRootCommandTest.php b/tests/Unit/ClusterRootCommandTest.php index 851c92d5..2ba85a8e 100644 --- a/tests/Unit/ClusterRootCommandTest.php +++ b/tests/Unit/ClusterRootCommandTest.php @@ -62,21 +62,73 @@ final class ClusterRootCommandTest extends TestCase { NodeFlows::usePath($rFlows); $this->assertSame($rExpected, RootSignalsCronJob::rootCommandsFromMain(), json_encode($rDoc)); } - // Without root's pin MAIN keeps the signals table for them. + // The same flows without a pin root trusts: MAIN keeps the signals table for them. + file_put_contents($rFlows, json_encode(['mode' => 1, 'flows' => NodeFlows::COMMANDS, 'state' => 'active'])); + NodeFlows::usePath($rFlows); + $this->assertTrue(RootSignalsCronJob::rootCommandsFromMain(), 'COMMANDS on, the pin in place'); chmod($this->rBase . 'etc', 0775); - $this->assertFalse(RootSignalsCronJob::rootCommandsFromMain()); + $this->assertFalse(RootSignalsCronJob::rootCommandsFromMain(), 'COMMANDS on, no pin'); } finally { NodeFlows::usePath(null); } $rSource = (string) file_get_contents(dirname(__DIR__, 2) . '/src/Cli/CronJobs/RootSignalsCronJob.php'); - $rAction = substr($rSource, (int) strpos($rSource, 'public function executeAction(')); - $this->assertStringContainsString("case 'flush':", $rAction, 'a node.root flush runs'); $rGate = strpos($rSource, 'if (!self::rootCommandsFromMain()) {'); $rRead = strpos($rSource, "SELECT `signal_id` FROM `signals` WHERE `server_id` = ? AND `custom_data` = '{"); $this->assertNotFalse($rGate); $this->assertTrue($rRead > $rGate && $rRead - $rGate < 120, 'the legacy flush row is read only where MAIN still sends it'); } + /** + * What the flush does, run as cluster:root runs it: iptables flushed and + * saved, one FLUSH line in mysql_syslog, and nothing of the next case + * (reboot: its REBOOT line, close_mysql, `sudo reboot`). The database + * throws on any statement but the FLUSH line, so a fall-through stops + * before the reboot's shell_exec. + */ + public function testTheFlushActionFlushesIptablesAndLogsOnce(): void { + if (!defined('SERVER_ID')) { + define('SERVER_ID', 5); + } + $rJob = new class extends RootSignalsCronJob { + /** @var list */ + public array $rCalls = []; + + protected function flushIPs(): void { + $this->rCalls[] = 'flushIPs'; + } + + protected function saveiptables(): void { + $this->rCalls[] = 'saveiptables'; + } + }; + $rDb = new class { + /** @var list */ + public array $rCalls = []; + + public function query(string $rQuery, mixed ...$rArgs): bool { + $this->rCalls[] = $rQuery; + if (!str_contains($rQuery, "VALUES(?, 'FLUSH',")) { + throw new \RuntimeException('not the flush\'s line: ' . $rQuery); + } + return true; + } + + public function close_mysql(): void { + $this->rCalls[] = 'close_mysql'; + } + }; + ob_start(); + try { + $rJob->executeAction(['action' => 'flush'], [], $rDb); + } finally { + $rOut = (string) ob_get_clean(); + } + $this->assertSame(['flushIPs', 'saveiptables'], $rJob->rCalls, 'flushed, then saved, once each'); + $this->assertCount(1, $rDb->rCalls, 'one statement, and no close_mysql'); + $this->assertStringStartsWith('INSERT INTO `mysql_syslog`', $rDb->rCalls[0]); + $this->assertStringNotContainsString('Rebooting', $rOut); + } + public function testThePinIsReadOnlyFromASafeDirectory(): void { $rPin = RootPin::read(); $this->assertSame([$this->rCrypto->info()['panel_sign_pub'], self::NODE], [$rPin['pub'], $rPin['node']]); diff --git a/tests/Unit/ConnectAuditTest.php b/tests/Unit/ConnectAuditTest.php index d7c4d9ff..360672ac 100644 --- a/tests/Unit/ConnectAuditTest.php +++ b/tests/Unit/ConnectAuditTest.php @@ -5,6 +5,7 @@ use XcVm\Core\Cluster\ConnectAudit; use XcVm\Core\Cluster\NodeFlows; use XcVm\Core\Cluster\NodeRole; use XcVm\Core\Cluster\SettingsAudit; +use XcVm\Tests\Support\AgentUser; /** * ConnectAudit — the trace behind the cluster plan's cutover gate (seven days @@ -22,6 +23,7 @@ final class ConnectAuditTest extends TestCase { protected function setUp(): void { $this->rDir = sys_get_temp_dir() . '/xcvm-connects-' . bin2hex(random_bytes(4)) . '/'; mkdir($this->rDir . 'cluster', 0777, true); + AgentUser::own($this->rDir); // as root, the audit writes as this tree's owner $this->mode(1); ConnectAudit::useDir($this->rDir . 'sql_audit/'); ConnectAudit::useClock(self::NOW); @@ -181,7 +183,13 @@ final class ConnectAuditTest extends TestCase { ConnectAudit::useClock(self::NOW + 10); $this->connect(ConnectAudit::SQL); $this->assertSame(1, $this->published()['sql_connects']); - ConnectAudit::useClock(self::NOW + ConnectAudit::PUBLISH_EVERY); + // A new site rewrites it at once, however fresh. + touch($this->rDir . 'cluster/audit.json', self::NOW + 10); + ConnectAudit::guard(ConnectAudit::REDIS); $rLine = __LINE__; + $this->assertSame([2, 1], [$this->published()['sql_connects'], $this->published()['redis_connects']], 'a new site: at once'); + $this->assertSame(1, $this->published()['sites']['redis ' . __FILE__ . ':' . $rLine]); + touch($this->rDir . 'cluster/audit.json', self::NOW + 10); + ConnectAudit::useClock(self::NOW + 10 + ConnectAudit::PUBLISH_EVERY); $this->connect(ConnectAudit::SQL); $this->assertSame(3, $this->published()['sql_connects'], 'once a minute'); $this->assertTrue(SettingsAudit::publish(null, self::NOW + 100)); @@ -198,8 +206,8 @@ final class ConnectAuditTest extends TestCase { } public function testANodeWithoutConnectsReportsZerosSinceItsAuditBegan(): void { - $this->assertTrue(SettingsAudit::publish(null, self::NOW)); // cron:cleanup, every hour - $this->assertSame('{"settings_misses":{},"sql_connects":0,"redis_connects":0,"sites":{},"connects_since":' . self::NOW . '}', file_get_contents($this->rDir . 'cluster/audit.json'), 'objects, as MAIN reads them'); + $this->assertTrue(SettingsAudit::publish(null, self::NOW - 86400)); // cron:cleanup, every hour + $this->assertSame('{"settings_misses":{},"sql_connects":0,"redis_connects":0,"sites":{},"connects_since":' . (self::NOW - 86400) . '}', file_get_contents($this->rDir . 'cluster/audit.json'), 'objects, as MAIN reads them'); // Most connects first, at most MAX_SITES sites over the week, the rest under "*". for ($i = 0; $i < ConnectAudit::MAX_SITES; $i++) { @@ -219,6 +227,66 @@ final class ConnectAuditTest extends TestCase { $this->assertSame([ConnectAudit::OTHER => 1], array_slice($rDoc['sites'], -1, 1, true), 'the least, last'); } + /** + * `connects_since` marks where the counts start: a node back in mode 1 + * reports none of the connects it counted before mode 0, and a day before + * the audit began is left out of the report. + */ + public function testTheCountsStartWhenTheAuditBegan(): void { + ConnectAudit::useClock(self::NOW - 3 * 86400); + ConnectAudit::record(ConnectAudit::SQL, 'Old.php:1'); + $this->assertSame([1, self::NOW - 3 * 86400], [$this->published()['sql_connects'], $this->published()['connects_since']]); + $this->mode(0); + $this->assertFalse(SettingsAudit::publish(null, self::NOW - 2 * 86400)); + $this->assertSame([], glob($this->rDir . 'sql_audit/*'), 'the window and its days went'); + $this->mode(1); + $this->assertTrue(SettingsAudit::publish(null, self::NOW)); + $this->assertSame(['settings_misses' => [], 'sql_connects' => 0, 'redis_connects' => 0, 'sites' => [], 'connects_since' => self::NOW], $this->published()); + + // A day file older than the window's start (a manual trace's, say) is not counted. + file_put_contents($this->rDir . 'sql_audit/' . gmdate('Ymd', self::NOW - 86400) . '.json', '{"sql":5,"redis":0,"sites":{"sql Old.php:1":5}}'); + $this->assertTrue(SettingsAudit::publish(null, self::NOW + 60)); + $this->assertSame(0, $this->published()['sql_connects']); + $this->assertSame(5, ConnectAudit::summary(7, self::NOW)['sql'], 'the file is still there'); + } + + /** + * A count rewrites its day file in place, under its lock: a report read + * at the same time (another process's publish) waits for it rather than + * seeing a day emptied, so the counts it reports never go down. + */ + public function testAReportNeverReadsADayHalfWritten(): void { + $rScript = $this->rDir . 'count.php'; + file_put_contents($rScript, <<<'PHP' + rDir . 'flows.json', $this->rDir . 'misses/', $this->rDir . 'cluster/', $this->rDir . 'sql_audit/', (string) self::NOW]; + $rProc = proc_open($rCommand, [0 => ['file', '/dev/null', 'r'], 1 => ['file', '/dev/null', 'w'], 2 => ['file', '/dev/null', 'w']], $rPipes); + $this->assertIsResource($rProc); + [$rLast, $rReads, $rDrops] = [0, 0, 0]; + do { + $rRunning = proc_get_status($rProc)['running']; + $rSql = ConnectAudit::summary(7, self::NOW)['sql']; + $rDrops += (int) ($rSql < $rLast); + [$rLast, $rReads] = [max($rLast, $rSql), $rReads + 1]; + } while ($rRunning); + $this->assertSame(0, proc_close($rProc)); + $this->assertSame(0, $rDrops, $rReads . ' reads while the writer counted'); + $this->assertSame(2000, ConnectAudit::summary(7, self::NOW)['sql']); + } + public function testAuditJsonStaysWithinWhatMainTakes(): void { foreach (range(1, SettingsAudit::MAX_KEYS + 5) as $i) { SettingsAudit::read(str_repeat('k', 60) . sprintf('%04d', $i)); @@ -265,21 +333,23 @@ final class ConnectAuditTest extends TestCase { return json_decode($rOut) === true; } - public function testARootProcessHandsWhatItMakesToTheAgentsUser(): void { - if (!function_exists('posix_geteuid') || posix_geteuid() !== 0) { + public function testARootProcessCountsAsTheAgentsUser(): void { + if (!AgentUser::root()) { $this->markTestSkipped('needs root, as the node\'s root crons run'); } - // A node: storage/ is not in the LB build, and config/cluster/ is xc_vm's (nobody here). - chown($this->rDir . 'cluster', 65534); - chgrp($this->rDir . 'cluster', 65534); + // A node: storage/ is not in the LB build, and the deploy root and + // config/cluster/ are xc_vm's (nobody here, setUp): root makes each + // level and file as that user. $rDir = $this->rDir . 'storage/cluster/sql_audit/'; ConnectAudit::useDir($rDir); ConnectAudit::guard(ConnectAudit::SQL); // cron:root_signals boots through MAIN's database in mode 1 + $this->assertSame(0, posix_geteuid(), 'root again after the count'); $rDay = $rDir . gmdate('Ymd', self::NOW) . '.json'; foreach ([$this->rDir . 'storage', $this->rDir . 'storage/cluster', $rDir, $rDay, $rDir . gmdate('Ymd', self::NOW) . '.ndjson', $rDir . 'since', $this->rDir . 'cluster/audit.json'] as $rPath) { clearstatcache(true, $rPath); - $this->assertSame([65534, 65534], [fileowner($rPath), filegroup($rPath)], $rPath); + $this->assertSame([AgentUser::UID, AgentUser::UID], [fileowner($rPath), filegroup($rPath)], $rPath); } + $this->assertSame(0750, fileperms($this->rDir . 'storage') & 0777); // xc_vm's hourly publish (cron:cleanup) reads root's counts. $this->assertTrue($this->publishAsNobody($rDir)); @@ -291,4 +361,53 @@ final class ConnectAuditTest extends TestCase { $this->assertFalse($this->publishAsNobody($rDir)); $this->assertSame(7, $this->published()['sql_connects']); } + + /** + * Root (cron:root_signals, `startup` and cluster:root boot through MAIN's + * database in mode 1) counts in directories xc_vm can write. A link xc_vm + * planted there leads root nowhere xc_vm could not go itself: nothing + * outside them is made, written or handed over. Where the agent's + * directory is root's, root has no user to count as, and writes nothing. + */ + public function testRootFollowsNoLinkTheAgentsUserPlanted(): void { + if (!AgentUser::root()) { + $this->markTestSkipped('needs root, as the node\'s root crons run'); + } + $rSafe = $this->rDir . 'root-only/'; + mkdir($rSafe, 0700); + chown($rSafe, 0); + file_put_contents($rSafe . 'log', "root's\n"); + file_put_contents($rSafe . 'report', "root's\n"); + $rDay = gmdate('Ymd', self::NOW); + $rLinks = [ + $this->rDir . 'sql_audit/' . $rDay . '.json' => $rSafe . 'counts', // none there: root would make it + $this->rDir . 'sql_audit/' . $rDay . '.ndjson' => $rSafe . 'log', // root would append to it + $this->rDir . 'misses/' . $rDay . '.json' => $rSafe . 'misses', + $this->rDir . 'cluster/audit.json.' . getmypid() . '.tmp' => $rSafe . 'report', // root would overwrite and hand it over + ]; + foreach ($rLinks as $rLink => $rTarget) { + @mkdir(dirname($rLink), 0750); + AgentUser::own(dirname($rLink)); + symlink($rTarget, $rLink); + lchown($rLink, AgentUser::UID); + } + ConnectAudit::guard(ConnectAudit::SQL); + SettingsAudit::read('zz_missed_by_root'); + SettingsAudit::flush(self::NOW); + $this->assertSame(0, posix_geteuid()); + clearstatcache(); + $this->assertSame(['log', 'report'], array_values(array_diff(scandir($rSafe), ['.', '..'])), 'nothing made there'); + foreach (['log', 'report'] as $rFile) { + $this->assertSame(["root's\n", 0, 0], [file_get_contents($rSafe . $rFile), fileowner($rSafe . $rFile), filegroup($rSafe . $rFile)], $rFile); + } + + // The agent's directory root's: nothing written at all. + exec('rm -rf ' . escapeshellarg($this->rDir . 'sql_audit') . ' ' . escapeshellarg($this->rDir . 'misses')); + chown($this->rDir . 'cluster', 0); + ConnectAudit::guard(ConnectAudit::SQL); + SettingsAudit::read('zz_missed_by_root'); + SettingsAudit::flush(self::NOW); + $this->assertDirectoryDoesNotExist($this->rDir . 'sql_audit'); + $this->assertDirectoryDoesNotExist($this->rDir . 'misses'); + } } diff --git a/tests/Unit/DbConnectRefusalTest.php b/tests/Unit/DbConnectRefusalTest.php index 10d719f5..4240b125 100644 --- a/tests/Unit/DbConnectRefusalTest.php +++ b/tests/Unit/DbConnectRefusalTest.php @@ -11,6 +11,7 @@ use XcVm\Core\Database\DatabaseHandler; use XcVm\Core\Database\LazyDatabaseHandler; use XcVm\Infrastructure\Database\DatabaseFactory; use XcVm\Infrastructure\Redis\RedisManager; +use XcVm\Tests\Support\AgentUser; /** * The mode-2 refusal (cluster plan, section 10, step 1): on a node in API @@ -25,15 +26,19 @@ use XcVm\Infrastructure\Redis\RedisManager; * xcvm_core stand-in that logs each connect it is asked for. */ final class DbConnectRefusalTest extends TestCase { + private const NOW = 1800000000; // 2027-01-15 08:00 UTC: the in-process counts' day + private string $rDir; protected function setUp(): void { $this->rDir = sys_get_temp_dir() . '/xcvm-refusal-' . bin2hex(random_bytes(4)) . '/'; mkdir($this->rDir . 'config/cluster', 0777, true); + AgentUser::own($this->rDir); // as root, the audit writes as this tree's owner NodeFlows::usePath($this->rDir . 'config/cluster/flows.json'); NodeRole::useMainBuild(false); ConnectAudit::useDir($this->rDir . 'storage/cluster/sql_audit/'); SettingsAudit::useDir($this->rDir . 'storage/cluster/settings_misses/', $this->rDir . 'config/cluster/'); + ConnectAudit::useClock(self::NOW); DatabaseFactory::reset(); } @@ -42,6 +47,7 @@ final class DbConnectRefusalTest extends TestCase { NodeRole::useMainBuild(null); NodeRole::resetAudit(); ConnectAudit::useDir(false); + ConnectAudit::useClock(null); SettingsAudit::useDir(false); DatabaseFactory::reset(); exec('rm -rf ' . escapeshellarg($this->rDir)); @@ -111,8 +117,8 @@ final class DbConnectRefusalTest extends TestCase { $this->assertNull(DatabaseFactory::get(), 'nothing kept as the process\'s handle'); $rLazy = new LazyDatabaseHandler(); // opens nothing yet - $rLine = __LINE__ + 1; - $this->assertStringEndsWith(':' . $rLine, $this->refused(static fn() => $rLazy->query('SELECT 1'))->rSite, 'a lazy handle: its first query'); + $rLazyLine = __LINE__ + 1; + $this->assertStringEndsWith(':' . $rLazyLine, $this->refused(static fn() => $rLazy->query('SELECT 1'))->rSite, 'a lazy handle: its first query'); $this->refused(static fn() => $rLazy->db_connect(false, true)); // graceful or not: a refusal is no outage to wait out $this->refused(static fn() => $rLazy->reconnect()); $this->refused(static fn() => $rLazy->db_explicit_connect('10.0.0.1', 3306, 'xc_vm', 'u', 'p')); @@ -122,33 +128,36 @@ final class DbConnectRefusalTest extends TestCase { $this->assertStringContainsString('Redis', $e->getMessage()); RedisManager::closeInstance(); // whatever an earlier test left: the singleton connects afresh $this->refused(static fn() => RedisManager::instance()); - $this->refused(static fn() => (new RedisCache('10.0.0.1'))->connect()); + $rCache = new RedisCache('10.0.0.1'); + $rLine = __LINE__ + 1; + $this->assertStringEndsWith('DbConnectRefusalTest.php:' . $rLine, $this->refused(static fn() => $rCache->connect())->rSite, 'RedisCache is connect machinery, not the caller'); // Each attempt is counted, marked refused in the log. - $rDay = json_decode((string) file_get_contents($this->rDir . 'storage/cluster/sql_audit/' . gmdate('Ymd') . '.json'), true); + $rDay = json_decode((string) file_get_contents($this->rDir . 'storage/cluster/sql_audit/' . gmdate('Ymd', self::NOW) . '.json'), true); $this->assertSame([6, 3], [$rDay['sql'], $rDay['redis']]); - $rLog = array_map(static fn(string $rLine): array => json_decode($rLine, true), file($this->rDir . 'storage/cluster/sql_audit/' . gmdate('Ymd') . '.ndjson', FILE_IGNORE_NEW_LINES)); + $rLog = array_map(static fn(string $rLine): array => json_decode($rLine, true), file($this->rDir . 'storage/cluster/sql_audit/' . gmdate('Ymd', self::NOW) . '.ndjson', FILE_IGNORE_NEW_LINES)); $this->assertCount(9, $rLog); $this->assertSame([1], array_values(array_unique(array_column($rLog, 'r')))); - $this->assertSame(1, array_sum(array_map(static fn(string $rSite): int => (int) str_starts_with($rSite, 'sql ') * (int) str_ends_with($rSite, 'DbConnectRefusalTest.php:' . $rLine), array_keys($rDay['sites']))), 'the lazy handle\'s query, as its own site'); + $this->assertSame(1, array_sum(array_map(static fn(string $rSite): int => (int) str_starts_with($rSite, 'sql ') * (int) str_ends_with($rSite, 'DbConnectRefusalTest.php:' . $rLazyLine), array_keys($rDay['sites']))), 'the lazy handle\'s query, as its own site'); } public function testTheRefusalIsCountedEvenWithTheAuditForcedOff(): void { $this->flows(2); NodeRole::resetAudit(false); $this->refused(static fn() => new DatabaseHandler()); - $this->assertFileExists($this->rDir . 'storage/cluster/sql_audit/' . gmdate('Ymd') . '.json', 'a refusal is what the page must show'); + $this->assertFileExists($this->rDir . 'storage/cluster/sql_audit/' . gmdate('Ymd', self::NOW) . '.json', 'a refusal is what the page must show'); } // ── The real boot, in a child PHP ──────────────────────────────── /** * Run $rCode after the real bootstrap's autoloader, in a throwaway deploy - * root whose xcvm_core logs each connect it is asked for. + * root whose xcvm_core logs each connect it is asked for (and answers it + * with an SQLite handle when $rEnv sets XCVM_TEST_PDO=1). * * @return array{0: int, 1: string, 2: list} exit code, output, the connects xcvm_core saw */ - private function child(string $rCode, array $rArgs = []): array { + private function child(string $rCode, array $rArgs = [], array $rEnv = []): array { $rPrepend = $this->rDir . 'prepend.php'; file_put_contents($rPrepend, <<<'PHP' rDir . 'child.php'; file_put_contents($rScript, $rCode); @unlink($this->rDir . 'connects.log'); - $rProc = proc_open(array_merge([PHP_BINARY, '-d', 'auto_prepend_file=' . $rPrepend, $rScript], $rArgs), [0 => ['file', '/dev/null', 'r'], 1 => ['pipe', 'w'], 2 => ['pipe', 'w']], $rPipes, $this->rDir, ['XCVM_TEST_HOME' => $this->rDir, 'XCVM_TEST_SRC' => dirname(__DIR__, 2) . '/src/', 'PATH' => (string) getenv('PATH')]); + $rProc = proc_open(array_merge([PHP_BINARY, '-d', 'auto_prepend_file=' . $rPrepend, $rScript], $rArgs), [0 => ['file', '/dev/null', 'r'], 1 => ['pipe', 'w'], 2 => ['pipe', 'w']], $rPipes, $this->rDir, $rEnv + ['XCVM_TEST_HOME' => $this->rDir, 'XCVM_TEST_SRC' => dirname(__DIR__, 2) . '/src/', 'PATH' => (string) getenv('PATH')]); $this->assertIsResource($rProc); $rOut = (string) stream_get_contents($rPipes[1]) . (string) stream_get_contents($rPipes[2]); fclose($rPipes[1]); @@ -204,9 +213,16 @@ final class DbConnectRefusalTest extends TestCase { PHP; } - /** @return array|null the child's counts for today */ + /** + * The child's counts: its one day file, whichever UTC day it counted on + * (the child runs on the real clock). + * + * @return array|null + */ private function counted(): ?array { - $rDay = json_decode((string) @file_get_contents($this->rDir . 'storage/cluster/sql_audit/' . gmdate('Ymd') . '.json'), true); + $rFiles = glob($this->rDir . 'storage/cluster/sql_audit/*.json') ?: []; + $this->assertLessThanOrEqual(1, count($rFiles), 'one boot, one day file'); + $rDay = $rFiles === [] ? null : json_decode((string) file_get_contents($rFiles[0]), true); return is_array($rDay) ? $rDay : null; } @@ -246,4 +262,23 @@ final class DbConnectRefusalTest extends TestCase { $this->assertSame(['redis'], $rConnects); $this->assertSame('NULL', trim($rOut)); } + + /** + * DatabaseFactory::open(), which the load balancer's direct connects + * take, keeps the handle it builds as the process's, as each of them did + * with DatabaseFactory::set(). + */ + public function testOpenKeepsItsHandleAsTheProcesss(): void { + [$rCode, $rOut, $rConnects] = $this->child(<<<'PHP' + query('SELECT 1')]); + PHP, [], ['XCVM_TEST_PDO' => '1']); + $this->assertSame(0, $rCode, $rOut); + $this->assertSame(['sql'], $rConnects); + $this->assertSame([true, true], json_decode($rOut, true)); + } } diff --git a/tests/Unit/NodeRoleTest.php b/tests/Unit/NodeRoleTest.php index 63d31fa0..a2cde3ad 100644 --- a/tests/Unit/NodeRoleTest.php +++ b/tests/Unit/NodeRoleTest.php @@ -54,16 +54,20 @@ final class NodeRoleTest extends TestCase { * cron:cleanup's node-side part runs on every node, before its MAIN-only * part: it is the only path that drops old days out of the settings-miss * report (a node in mode 0 counts nothing, so never merges one) and - * removes it in mode 0. + * removes it in mode 0. It also runs before the cron's first query, which + * a node in mode 2 refuses: the connect audit stays bounded there too. */ public function testCleanupKeepsEachNodesAuditsBeforeItsMainGate(): void { $rSource = (string) file_get_contents(dirname(__DIR__, 2) . '/src/Cli/CronJobs/CleanupCronJob.php'); $rGate = strpos($rSource, 'if (!NodeRole::isMain())'); + $rQuery = strpos($rSource, '$db->query('); $this->assertNotFalse($rGate); + $this->assertNotFalse($rQuery); foreach (['ConnectAudit::prune(8);', 'SettingsAudit::prune(8);', 'SettingsAudit::publish();'] as $rCall) { $rAt = strpos($rSource, $rCall); $this->assertNotFalse($rAt, "CleanupCronJob.php must call $rCall"); $this->assertLessThan($rGate, $rAt, "$rCall runs on every node, before the MAIN-only part"); + $this->assertLessThan($rQuery, $rAt, "$rCall runs before the first query, which mode 2 refuses"); } } diff --git a/tests/Unit/ReplicaBootTest.php b/tests/Unit/ReplicaBootTest.php index 3a5bd0ec..23a9012b 100644 --- a/tests/Unit/ReplicaBootTest.php +++ b/tests/Unit/ReplicaBootTest.php @@ -21,6 +21,7 @@ use XcVm\Core\Cluster\ReplicaBoot; use XcVm\Core\Container\ServiceContainer; use XcVm\Core\Enum\BootContext; use XcVm\Infrastructure\Bootstrap\WebApiBootstrap; +use XcVm\Tests\Support\AgentUser; use XcVm\Tests\Support\ReplicaFixture; /** @@ -50,6 +51,8 @@ final class ReplicaBootTest extends TestCase { $this->rHome = sys_get_temp_dir() . '/xcvm-boot-' . bin2hex(random_bytes(4)) . '/'; mkdir($this->rHome . 'config/cluster', 0777, true); mkdir($this->rHome . 'tmp/cache', 0777, true); + // As root, a node's audits write as the owner of config/cluster/. + AgentUser::own($this->rHome); // LegacyInitializer regenerates the xc_vm crontab once per boot: a // `crontab` first on the child's PATH logs it, never the test user's. mkdir($this->rHome . 'stub'); diff --git a/tests/Unit/SettingsAuditTest.php b/tests/Unit/SettingsAuditTest.php index 94e5bb35..f2a0e976 100644 --- a/tests/Unit/SettingsAuditTest.php +++ b/tests/Unit/SettingsAuditTest.php @@ -4,6 +4,7 @@ use PHPUnit\Framework\TestCase; use XcVm\Core\Cluster\NodeFlows; use XcVm\Core\Cluster\SettingsAudit; use XcVm\Core\Config\SettingsManager; +use XcVm\Tests\Support\AgentUser; /** * `audit.settings_misses` on the node (cluster plan, section 9, R1 @@ -21,6 +22,7 @@ final class SettingsAuditTest extends TestCase { protected function setUp(): void { $this->rDir = sys_get_temp_dir() . '/xcvm-misses-' . bin2hex(random_bytes(4)) . '/'; mkdir($this->rDir . 'cluster', 0777, true); + AgentUser::own($this->rDir); // as root, the audit writes as this tree's owner $this->mode(1); SettingsManager::set(['server_name' => 'Panel', 'api_pass' => 'x']); } @@ -200,21 +202,22 @@ final class SettingsAuditTest extends TestCase { return json_decode($rOut) === true; } - public function testARootProcessHandsEveryLevelItMakesToTheAgentsUser(): void { - if (!function_exists('posix_geteuid') || posix_geteuid() !== 0) { + public function testARootProcessMergesAsTheAgentsUser(): void { + if (!AgentUser::root()) { $this->markTestSkipped('needs root, as the node\'s root crons run'); } - // A node: storage/ is not in the LB build, and config/cluster/ is xc_vm's (nobody here). - chown($this->rDir . 'cluster', 65534); - chgrp($this->rDir . 'cluster', 65534); + // A node: storage/ is not in the LB build, and the deploy root and + // config/cluster/ are xc_vm's (nobody here, setUp): root makes each + // level and file as that user. $rDir = $this->rDir . 'storage/cluster/settings_misses/'; SettingsAudit::useDir($rDir, $this->rDir . 'cluster/'); SettingsManager::get('update_channel_x'); // as a root-only command reads it SettingsAudit::flush(self::NOW); + $this->assertSame(0, posix_geteuid(), 'root again after the merge'); $rDay = $rDir . gmdate('Ymd', self::NOW) . '.json'; foreach ([$this->rDir . 'storage', $this->rDir . 'storage/cluster', $rDir, $rDay, $this->rDir . 'cluster/audit.json'] as $rPath) { clearstatcache(true, $rPath); - $this->assertSame([65534, 65534], [fileowner($rPath), filegroup($rPath)], $rPath); + $this->assertSame([AgentUser::UID, AgentUser::UID], [fileowner($rPath), filegroup($rPath)], $rPath); } $this->assertSame(0750, fileperms($this->rDir . 'storage') & 0777); @@ -235,6 +238,41 @@ final class SettingsAuditTest extends TestCase { $this->assertSame(['settings_misses' => ['update_channel_x' => 1]], $this->published()); } + /** + * A merge rewrites its day file in place, under its lock: a report read + * at the same time waits for it rather than seeing a day emptied, so the + * counts it reports never go down. + */ + public function testAReportNeverReadsADayHalfWritten(): void { + $rScript = $this->rDir . 'merge.php'; + file_put_contents($rScript, <<<'PHP' + rDir . 'flows.json', $this->rDir . 'misses/', $this->rDir . 'cluster/', (string) self::NOW]; + $rProc = proc_open($rCommand, [0 => ['file', '/dev/null', 'r'], 1 => ['file', '/dev/null', 'w'], 2 => ['file', '/dev/null', 'w']], $rPipes); + $this->assertIsResource($rProc); + [$rLast, $rReads, $rDrops] = [0, 0, 0]; + do { + $rRunning = proc_get_status($rProc)['running']; + $rCount = SettingsAudit::summary(self::NOW)['zz_writer'] ?? 0; + $rDrops += (int) ($rCount < $rLast); + [$rLast, $rReads] = [max($rLast, $rCount), $rReads + 1]; + } while ($rRunning); + $this->assertSame(0, proc_close($rProc)); + $this->assertSame(0, $rDrops, $rReads . ' reads while the writer merged'); + $this->assertSame(['zz_writer' => 2000], SettingsAudit::summary(self::NOW)); + } + public function testALongRunningProcessMergesEveryMinute(): void { SettingsManager::get('first_key'); $this->assertNull($this->day(time()), 'held in memory'); diff --git a/tests/bootstrap.php b/tests/bootstrap.php index a51b4395..b06ee9e8 100644 --- a/tests/bootstrap.php +++ b/tests/bootstrap.php @@ -95,6 +95,7 @@ require_once __DIR__ . '/Support/BusServer.php'; require_once __DIR__ . '/Support/QueryLogDb.php'; require_once __DIR__ . '/Support/InstallSchema.php'; require_once __DIR__ . '/Support/ReplicaFixture.php'; +require_once __DIR__ . '/Support/AgentUser.php'; // EventIngest's lane locks are real flock files. Without this they would land // in the shared TMP_PATH or system temp dir, where suite runs from other