Plan section 9, R1 `settings`: API nodes log unknown-key reads to
`audit.settings_misses`. Once a node's replica owns its settings, a key
outside lb_settings_keys.php is absent there, and the CI scan behind the
allowlist cannot see a key built at run time. The count names such reads
before they fail.
On the node (Core/Cluster/SettingsAudit): in mode 1 or 2, as the agent's
flows.json says (NodeFlows::declared, no database and no servers read),
each read through SettingsManager's getters (get, getBool, getInt,
getString, getArray, has) of a key outside the allowlist's `keys` and
`withheld` is counted. A process keeps its counts in memory and merges
them at exit, and every minute while it runs, into a per-day file under
its lock (STORAGE_PATH/cluster/settings_misses/YYYYMMDD.json). It then
writes config/cluster/audit.json, `{"settings_misses": {key: count}}`
over the last seven UTC days, which the agent is to send as the
heartbeat's `audit`. Everything is bounded: 64 names, the rest and any
key that is not a settings name under "*". cron:cleanup prunes the day
files after eight days and rewrites audit.json every hour, so old days
drop out; in mode 0 the file is removed. MAIN and legacy nodes have no
flows.json and count nothing; a read costs one lookup.
On MAIN (Domain/Cluster/NodeAudit): a heartbeat's `audit` is normalised
(names and positive integer counts only, at most 16 KiB, the least
missed past 64 names folded into "*") and kept in the new
cluster_nodes.audit (migration 045, mirrored in database.sql; 044 is taken
by work on another branch). It is written only when it differs from what
the row holds, which every request reads already, so an unchanged report
costs no query, on the cluster bus or not. A heartbeat without `audit`,
today's agent's, changes nothing. The Cluster Nodes page shows each mode 1
or 2 node's misses.
SettingsAuditTest covers the counting per getter, withheld keys not
counted, mode 0 and MAIN counting nothing, the merge across processes,
the caps, the seven-day window, the pruning, the empty report and the
minute's merge of a long-running process. ClusterApiTest covers the
heartbeat storing a report once, an unchanged, missing or malformed one
writing nothing, an empty one clearing the page, and the cap; and
ReplicaBootTest a child process's miss reaching audit.json at its exit,
while booting from the replica reads only allowlisted settings.
Plan section 10, step 2: ReplicaStage replaces DatabaseStage and
LegacyCoreStage in the CLI profile and in WebApiBootstrap for a node in
mode 2. The code ships now; the switch to mode 2 comes later. Section 9
also has src/service run `cluster:apply --from-disk` at boot so that a
node rebooted while MAIN is unreachable serves from its replica, but
cluster:apply booted through DatabaseStage, which exits when MAIN's
database does not answer.
ReplicaStage leaves what the two stages leave, from the replica's
caches: a global $db that is a LazyDatabaseHandler (nothing opened until
a query needs it), DatabaseFactory and the domain wiring, SERVER_ID, the
settings (SettingsManager, $rSettings, core.settings), the servers
($rServers, core.servers), the request, the FFmpeg paths, core.config,
and the bouquets and categories caches as they are, since no section
carries them yet. The crontab is regenerated only from jobs the replica
owns. Once a process booted this way (ReplicaBoot::active),
SettingsRepository and ServerRepository never read MAIN's database: the
caches however old, or nothing.
Who boots this way (BootKernel::resolve, ReplicaBoot):
- a node in mode 2, active or quarantined by the agent's flows.json
(NodeFlows::declared, read before the settings, the servers or a
database handle exist; MAIN runs no agent, so has no such file), once
an apply built the replica's settings and servers caches since the
reboot. Until then it boots through the database as before, which
mode 2's refusal will turn into the fail-closed answer;
- cluster:apply in every mode (console.php passes ReplicaBoot::forArgv),
since its work is to build those caches. With CONFIG on it now needs no
database at all; in shadow its comparison with MAIN's crontab and RTMP
publishers still reads it, on first use.
Nodes in mode 0 or 1, and MAIN, boot exactly as before: the same stages,
and WebApiBootstrap::coreStages() the same two.
ReplicaBootTest checks the profiles per mode and state, the fallback
until an apply, and runs the real console.php and bootstrap in a child
PHP with a throwaway deploy root and an xcvm_core stand-in whose database
never answers: cluster:apply --from-disk and the agent's cluster:apply
build every cache without one connect, a mode 1 node still connects at
boot, and a mode 2 node boots its CLI and a cached web API endpoint with
the globals and container entries listed above, without one connect.
The plan has src/service run `cluster:apply --from-disk` at boot, before
the daemons, so a node rebooted while MAIN is unreachable serves from its
replica. Until now the flag changed nothing: cluster:apply always applied
the `<name>.json` files the agent writes for PHP, which carry no
signature. At boot the agent may not run yet, and those files may predate
a key change or be torn.
With --from-disk, each section now comes from the sealed, panel-signed
record the agent stored beside its .json (Core/Cluster/ReplicaRecords):
the record must open with the node's box key from the agent's
agent.json (XCVM-SEAL-v1, purpose `replica`, context the node uuid), its
signature must verify under the panel key the agent pinned there (tag
`rep`, `blk` for the blocklist's deltas), and it must name this node and
the section. The blocklist is materialised again from blocklist.rep and
its deltas in seq order, as the agent does. A section is read only where
the agent stored its .json, which is what ReplicaApply::owns() and the
readers go by.
A record that does not verify is treated as an unreadable section: it
writes no cache and hands back one the replica owned, so its readers go
back to MAIN's database. The report lists the sections under `from_disk`
(`verified`, `unverified`), by name only, never their content. Without
--from-disk nothing changes: the agent runs cluster:apply right after it
verified what it stored.
ReplicaRecordsTest covers the records winning over a planted or stale
.json, a corrupt record, one signed by another panel key, one for another
node, another section's record, a missing record, a missing or broken
agent.json, the blocklist's deltas in order and out of order, and a
report with no secret in it. The test fixture (tests/Support/ReplicaFixture)
builds the records as MAIN does, with a panel key of its own.
Review findings on the early release of old MAIN ports:
- An enrolment code carries one MAIN URL, the policy's first when it was
issued, and the agent dials only that URL for enrol_code and
enrol_code_status. release() now releases nothing while a code has not
expired, or a request made with one waits for approval and is less than
30 minutes old (the agent's wait). Once approved, the enrolling row holds
the ports as before.
- Under https_required no node reports a plain-HTTP port, yet the
plain-HTTP URLs a node has known carry the signed challenge it recovers
through when HTTPS fails. A kept plain-HTTP port or http:// URL now keeps
its 7 days there; a kept https:// URL still goes early. The transport is
read from the database with the kept lists.
- A cluster_nodes row whose server was deleted no longer holds every port
forever: portsInUse() reads only rows whose server exists.
- An agent heartbeats MAIN in every mode. A node in mode 0 is still not
waited for, but while it is heard the port it last reached MAIN on stays.
Tests cover each of these, the kept https:// URL on a port a node still
uses surviving a release, a quarantined node, the port read from nginx's
SERVER_PORT, the last second of an enrolment, the largest policy_ver and
the bounds of the offline window.
The diagnostics page says that early release needs an agent that reports
its policy version, which none does yet, and what delays it. ADR 0004 and
the plan record the new rules and limits; the agent contract is unchanged.
ADR 0004 gets the sixth Phase 2 increment: what hello and heartbeat now
record of the URL a node uses (the policy_ver the agent says it dials,
the MAIN port nginx took the request on, migration 046), when
cron:cluster releases a kept port before its 7 days, why the current
policy version stands in for the announcing one, and the agent's
contract for the policy_ver field its Go half sends next. The earlier
endpoint and nginx increments point to it.
The plan's Phase 2 status lists the early release as built, and the
agent's half as not built yet. The diagnostics page says when an old
port closes sooner and what keeps it open for the full 7 days.
A port MAIN kept after an endpoint change (cluster_legacy_ports, or the
old HTTPS port of a kept URL) stayed served for its full 7 days, though
plan section 3 keeps it only "until every node uses the new URL, or 7
days".
cron:cluster now runs ClusterEndpoint::release() every minute, after
prune() and before nginx is rendered. It reads every cluster_nodes row
in mode >= 1 that is not revoked, and releases nothing while any of
them is offline or never heard (cluster_offline_after_sec, MAIN's own
downtime included), dials a policy other than the current
cluster_policy_ver (behind, 0 for an agent that does not say, or above
MAIN's own), or has no recorded port. An enrolment past its deadline is
skipped: it can no longer complete. Otherwise every kept port no such
node last reached MAIN on goes, with the kept URLs on it, the policy
version goes up and nginx closes the port in the same pass. A port a
node still reaches MAIN on stays: its new URL fails and it dials the
kept one.
The current version is at or above the one that announced any kept
entry, so no version per entry is stored, and the check is never less
strict. The write applies only over the two lists as read, so a change
stored meanwhile keeps what it kept. Old agents, which send no
policy_ver, keep the 7 days.
Plan section 3 keeps an old MAIN port "until every node uses the new
URL, or 7 days", but MAIN recorded neither the policy a node dials nor
the URL it used: cluster_nodes.policy_ver existed and nothing wrote it,
and the agent sends neither in its hello or heartbeat.
ClusterEndpoint::nodeUses() now gives the cluster_nodes fields a hello
or heartbeat changes: policy_ver, the version the payload's policy_ver
says the agent dials (0, unknown, when it is absent or not an integer
from 0 to 2^32-1, as for today's agent or after a downgrade), and
main_port, the MAIN port nginx took the request on. Public/cluster
passes that port from SERVER_PORT ($server_port in fastcgi_params); it
tells a kept old port from a current one, since each port is a single
listener in MAIN's nginx, where the address would not behind NAT.
The hello writes both with its other fields. The heartbeat writes them
only when one changed, so a heartbeat on the cluster bus still writes
nothing to MySQL. Migration 046 adds the main_port column (045 is taken
by the Phase 7 work on a parallel branch); before it runs, only
policy_ver is written. ClusterSchemaTest now scans every core migration
from 028 on for columns added to cluster tables.
Review findings on the second endpoint increment:
- An old HTTPS URL is kept only on the HTTPS broadcast port the old row
stored. A row with no HTTPS port (NULL) still gets https://<name>:443 in
the policy from its default. Keeping that URL made nginx bind 443
unchecked, a port it never held and another program (a reverse proxy)
may hold: the reload failed, and the next nginx start would too.
- The policy lists a kept https:// URL only while it lists HTTPS (not
under http, nor under auto without a verified certificate), and a kept
http:// URL never under https_required, as before.
- A cluster_api_port save reads the kept ports and URLs as stored:
stageApiPort() renders from them and recordApiPortChange() merges into
them, so settings loaded before a port was kept no longer drop it.
- The backup settings form (SettingsService::editBackup()) can no longer
set cluster_policy_ver, cluster_legacy_ports or cluster_legacy_urls,
as the settings form cannot.
- RootSignalsCronJob::rewriteServerIP() returns the updated row, which
the cron assigns, so it cannot be handed a row that already carries the
new address and announce nothing.
Tests cover a row without an HTTPS port, the transport rules when
listing, a kept URL that is current again, an IPv6 address, the API on
its own port, the fail-safe paths (a failed node check, no
cluster_legacy_urls column), the call site and a port kept while an API
port is staged.
ADR 0004: why an old HTTPS port is not checked for being free, a
rollback dropping cluster_legacy_urls through down/044, the old port
nginx holds for 7 days as a limit, and a precise agent contract for the
known-good URL sets: what an authenticated answer is, only the current
set recorded (not on HTTPS_REQUIRED), set equality, writes only on
change, emptied at Install, no http:// fallback under https_required,
the dial order, and the challenge over HTTP left on http_urls.
A settings save that changes cluster_api_port renders nginx before the
value is stored (stageApiPort), from the settings the panel process
loaded. Those may predate a URL ClusterEndpoint kept a moment ago, so
the staged render dropped an old HTTPS port's TLS server until
commitApiPort() rendered again from what is stored, and nodes on that
port lost MAIN meanwhile. The staged render now reads the kept URLs as
stored.
ADR 0004 gets the second MAIN endpoint increment: how a change of MAIN's
HTTPS port, server_ip or private_ip is announced and its old URL kept,
when it is not (no node in mode >= 1, the same URLs, an HTTPS port no
node uses), the old HTTPS port nginx keeps serving over TLS, what
today's agent does, and the contract for the agent's last 3 known-good
URL sets, which the Go half builds next. It also records where this
differs from the plan: announced through policy_ver rather than a
pushed policy.update, after the change rather than before, the HTTP
port now following the mode >= 1 rule, migration 044.
The plan's Phase 3 status now lists this as built, and the agent's
known-good sets and the settings-side changes as not built yet. The
diagnostics page says what the old HTTPS port and an old address do.
Only changes of MAIN's HTTP broadcast port and cluster_api_port were
announced to the nodes. A new HTTPS port, server_ip or private_ip
reached a node only at its next hello, and the old URL left the policy
at once, so a node that could not reach the new one lost MAIN.
ClusterEndpoint::recordMainChange() now handles every change of MAIN's
server row: the admin's save (ServerService::announceMainEndpoints(),
once the row is stored and before the ports are applied) and
cron:root_signals' automatic server_ip rewrite
(RootSignalsCronJob::rewriteServerIP()). When the policy's URLs change
while the API is on and a node is in mode >= 1 and not revoked, it keeps
an old HTTP broadcast port as before, keeps every URL the policy no
longer lists for 7 days in the new settings column cluster_legacy_urls
(migration 044, at most 8, the latest first), and raises
cluster_policy_ver. Nothing happens when the URLs stay the same or no
node may use them; the HTTP port change follows that rule too now.
The policy lists the kept URLs last, each while MAIN serves its port
with its scheme. ClusterNginxConfig serves an old HTTPS port over TLS
with the public server's ssl.conf, for the cluster API alone, until
cron:cluster prunes it. The kept lists are read again from the database
before they are merged or rendered, and a settings form cannot set them.
Migration 044 rather than 042: 042 and 043 are the Phase 7 crontab
migrations on a parallel branch, and the runner goes by file name.
No wire change: today's agent refetches the policy on the higher
policy_ver and tries the extra URLs last.
Database::query answers a failed statement with false, and get_row()
then answers false or the previous statement's row. The section
builders used `get_row() ?: []`, so a settings read failing mid-request
signed an empty `settings` section and a `secrets` entry with an empty
`current`. A CONFIG node took the empty row as its whole settings, with
secure_stream_tokens, verify_host, the disable_* and flood settings
unset, until a later poll. A failed crontab read likewise signed a
crontab with no job, and a failed blocklist read a blocklist with
nothing blocked.
Every read of the blocklist (BlocklistDelta) and of the settings,
secrets, servers, node and crontab sections now throws when it fails,
as do a missing settings row and an unset secret (an empty
live_streaming_pass, which cron:root_signals sets within the minute, or
OPENSSL_EXTRA). The config op answers 503 DB and the node keeps what it
holds. On the node, ReplicaApply refuses a settings section that is
not a whole raw row (no string server_name, so an empty section or a
list), handing the settings back to MAIN's database.
cluster:apply exits 3 after printing its report when a part of it
failed (today a write of config/openssl_extra), so the agent logs it
like any failed run instead of never hearing of it.
New tests cover what the suite let through: the settings handed back
when a section goes bad after an apply, and the refused mode; the boot
order (no servers read without an apply record, a servers cache rebuilt
before the settings are loaded); a settings read racing an apply; a
failed OPENSSL_EXTRA write reported with .prev written first; a
previous value from MAIN that is already closed, and the last open
second. The secrets tests read MAIN's previous OPENSSL_EXTRA from their
own temporary directory instead of the deploy root's config/.
ADR 0004's sixth Phase 7 note and the agent's contract gain the 503 DB
reply and cluster:apply's exit codes; the plan's Phase 7 status says so.
ADR 0004 gains the sixth Phase 7 increment: what the sealed secrets
section holds and how its kids are made, why it goes only to active
nodes in mode 1 or 2 and is never cached on MAIN, how cluster:apply
writes OPENSSL_EXTRA and makes the settings section authoritative with
it, how the settings loader avoids reading the servers before the
settings are loaded, where it differs from the plan, its known limits
(the other withheld secrets, among them cron:maxmind's licence key), and
the agent's contract for storing the section without ever logging it.
The plan's Phase 7 status lines now list the secrets section as built,
and its ticket kids and a previous stream secret as not built yet.
Plan section 13 makes the replica's settings section authoritative
together with the secrets section: the settings section withholds
live_streaming_pass, which the node reads, so on its own it could only
be compared.
With the CONFIG flow on and both sections usable, cluster:apply now
writes the settings cache from them: the settings section's raw row
with the secrets' live_streaming_pass, decoded by
SettingsRepository::decode as the settings loader decodes MAIN's row.
The apply records that it owns the cache, and SettingsRepository::getAll
then returns that cache however old, even when forced, rebuilds it from
disk when it is gone, and never writes MAIN's row over it; cron:cache
goes through getAll. The servers cache built in the same apply takes its
URLs from those settings. A missing or malformed secrets section leaves
the settings `incomplete`, handed back to MAIN's database.
OPENSSL_EXTRA goes where the node reads it, config/openssl_extra (0600,
owned like its directory), through OpensslExtra::adopt: MAIN's previous
value stays accepted until MAIN's previous_valid_until, else the value
replaced here for the usual window, and nothing is rewritten once the
node holds both, so the minute's re-apply never extends a window.
With CONFIG off nothing is written. The report names only the settings
keys and the secrets whose value differs, never a value, kid or hash:
cluster:apply prints it and the agent may log that output.
SettingsRepository::getAll asks ReplicaApply::owns only once an apply
recorded it, since owns() asks NodeFlows, which reads the servers before
the settings are loaded; a servers read that comes first takes the
settings cache (SettingsRepository::loaded) instead of empty settings.
The replica's settings section withholds every secret, yet a node needs
the viewer-token secret and OPENSSL_EXTRA to open the tokens MAIN mints.
Plan section 9 gives them a sealed R1 `secrets` section, and section 10
makes it the licence-gated, panel-signed part of the gate that replaces
db_grant.
ReplicaBuilder now builds `secrets` as a whole section: for
live_streaming_pass and openssl_extra, {kid, current, previous,
previous_valid_until}. The kid is an HMAC keyed by the value (for
OPENSSL_EXTRA, the fingerprint every node already publishes); previous
is MAIN's openssl_extra.prev while it is still open on MAIN's clock,
through the new OpensslExtra::previousEntry. The config op serves it
only to an agent that names it in `have`, and only to an active node in
mode 1 or 2 (ReplicaBuilder::serves); a legacy node is left out. It is
signed and sealed like the other whole sections, so without a licence
a changed section is left out of the reply.
It is read for every request and never cached: ReplicaEtagCache keeps
plain JSON files, and now refuses the `secrets` key outright.
ReplicaBuilderSecretsTest proves the section carries only its two keys
and that no secret leaves in any other section.
Review of the ingest permits (Phase 2, fourth bus increment).
Bulk had no permit of its own: P0 could take every permit up to the
total, and while it held them config, snapshots, logs and recordings
were all refused. One permit is now kept for bulk (BULK_RESERVE), as
ceil(n / 2) are for P0. The acquire script takes both reserves: a lane
under its reserve always gets a permit, and past it a lane is refused
when the permits held, plus the other lane's reserve it does not hold
yet, reach the total. At n = 1 that is one P0 and one bulk; at the
default, 3 for P0, 1 for bulk and 2 shared. Either lane keeps its
reserve after the concurrency is lowered.
RecordingFinalizer::create was check-then-insert, so a node's retry of
recording_complete that overlapped a request MAIN was still running
(the agent gives up after 10 s) made a second VOD. The new row is now
recorded as created_id only while none is, and read back; a call that
lost deletes its row, before any bouquet has it, and returns the
winner's.
RecordCommand::vodFor() holds the CONTENT-flow call, so the retrying
call site is tested. New tests: the epoch is marked used only once an
ingest permit is taken; mainRetrying makes one try after each wait;
the concurrency's lower clamp; ClusterApiTest never reaches a bus at
the checkout's default socket.
ADR 0004 and the plan: bulk's reserve, today's agent waits (P0 about
1 s, P1 10 s, P2 20 s), the retry bounds, and one rule for the agent's
bulk events interval, the recording_complete socket deadline and the
ingest ops still to come.
ADR 0004 gains the fourth cluster bus increment: the ingest permits,
ceil(n / 2) kept for P0 events, when they are taken (once the BOX is
open), their expiry, the busy refusal's wire fields (503 RATE_LIMITED
with retry_after_ms, op and lane), what today's agent does with it, the
agent's contract for the Go half (P0's shorter wait on a connection of
its own, stretched bulk intervals), and where it differs from the plan.
The check order under "MAIN's API" and the pools increment point to it.
The plan's Phase 2 status lists what was built and what the agent still
owes.
Plan section 8 caps MAIN's ingest at cluster_ingest_concurrency
concurrent requests and keeps ceil(n / 2) of those permits for P0
events, so bulk transfers never delay P0; without a free permit MAIN
answers 503 with retry_after_ms at once.
Every ingest op the API serves (ClusterPool::INGEST_OPS: events,
config, conn_snapshot, recording_complete) now holds an ingest permit
on the cluster bus while its handler runs. A P0 events batch takes any
free permit, and always its reserve; every other ingest request (P1/P2
events and the rest) only the permits not kept for P0. Bulk keeps at
least one, so a concurrency of 1 gives one P0 and one bulk permit, as
the ingest pool's floor does.
The permits reuse ClusterSemaphore's machinery: sorted sets
sem:ingest:p0 and sem:ingest:bulk without a TTL, scored by expiry on
the bus's own clock, dropped after the ingest pool's 90 s timeout or
when taken before a clock step, given back in finally. The acquire
script now prunes and counts a second key when given one. A batch's
lane is inside its BOX, so the permit is taken once the BOX is open
(no database is touched before it), after the nonce claim, the node
state and any per-op permit, and before the epoch is marked used.
None free: the per-op semaphores' signed 503 RATE_LIMITED, with
retry_after_ms (250-750 ms for P0, 1-3 s for bulk), op and lane
("p0" or "bulk"). Today's agent already waits retry_after_ms (at
least 1 s) and resends the same batch, snapshot chunk or config poll.
Without the bus no permit is taken, as before.
On a node whose CONTENT flow is on, RecordCommand asks MAIN for the
recording's VOD through the agent (recording_complete). Any failure
marked the recording failed and deleted its .ts file. MAIN is about to
refuse bulk ingest while its ingest permits are held (503 RATE_LIMITED),
and today's agent hands every refusal back to PHP as a bare 409, so a
busy spell of MAIN would lose recordings.
AgentClient::mainRetrying() asks again while no answer comes, after
1, 2, 4, 8, 15, 30, 30 and 30 s (two minutes in all). recording_complete
is applied once on MAIN (a retry gets the same VOD), so a retry is safe
whatever the failure was.
Review of the cluster_ctl listen-queue guard.
- The pass that saw the queue end judged nodes by heard times read before
it, and the queue's time counted as node silence. A lone TELEMETRY node
silenced only by the queue was therefore marked offline at the drain.
The queue reason now keeps a hold in health.json (`ctl_queue_hold:
{from, until}`). When the reason clears, whether drained or unknown, the
hold lasts one more cluster_offline_after_sec, so each node gets a full
offline window after the queue, as ready_at gives it after MAIN's own
downtime. The pass also reads the probe before HeartbeatService::flush().
- The hold spares only the nodes the queue may have silenced: those whose
silence counts from the run's start minus 10 s or later. It lasts at
most 4 offline windows from the run's start (QUEUE_HOLD_WINDOWS), so a
chronic mild queue no longer keeps a dead node suspect for as long as
it lasts.
- A pass now joins a run whose `at` is within QUEUE_GAP_MS on either side.
Before, cron:cluster broke the daemon's run when the daemon wrote
health.json between cron's clock read and its read of the file. A
joined run keeps the later `at`. A clock stepped back further still
starts a new run.
- Tests: a lone node silenced by the queue is never marked offline (drained
and unknown, read back from health.json); nodes silent before the queue
are not held; ready_at counts; the cap, with and without a drain near
it; the join edges (a few ms behind, exactly 5 s, a clock step back);
the run persisted through health.json; the probe read before the bus
flush; the Cluster Nodes page's alert per reason. ClusterPoolTest adds a
late answer that keeps the queue's start and a drain that forgets it, a
probe dropped for another pool, and an answer past MAX_ANSWER (responder
modes sleep:<ms> and huge). The tests that tick liveness without faking
the probe now fake one, so none probes a socket under MAIN_HOME.
- ADR 0004 and the plan describe the hold, its limits and its tests. The
wire is unchanged, so the agent's contract stays empty.
ADR 0004 gains the fifth Phase 2 increment: why FPM's own listen-queue
count cannot be used on the pools' unix sockets, how the probe times its
own status request instead, how the liveness loop turns that into a run
and a guard reason shared by the signals daemon and cron:cluster, the
audits and alerts per reason, and where this differs from the plan. The
wire does not change, so the agent's contract is empty. The pools'
increment and the Phase 3 liveness notes point to it.
The plan's section 13 moves the listen-queue input from Phase 2's "Not
built yet" to what is built, and names it among ClusterLivenessTest's
cases.
Plan section 8: a cluster_ctl pool listen queue lasting over 5 s triggers
the same guard as the fleet silence. That pool takes every heartbeat, so a
queue there makes live nodes look silent.
LivenessService::tick() reads ClusterPool::listenQueueMs('cluster_ctl')
once a pass (the reader is injectable for tests; the clock is
ClusterClock). A run lasts from now minus the age the probe reports, and
the guard's `ctl_queue` reason is up while it is over 5 s old. A pass that
sees no queue, or cannot tell (pool down, no status page, a reader that
throws), ends the run, so a missing pool never raises the guard. A pass
within 5 s of the last one that saw the run joins it, so cron:cluster's
own short-lived request does not restart the signals daemon's run; the
run is kept in health.json (`ctl_queue: {since, at}`) for both.
ClusterHealth keeps `guard` (what routing and the orphan purge read) plus
its `reasons` (silence, ctl_queue); a file without reasons reads as the
silence. Either reason holds nodes instead of marking them offline and
suspends the orphan purge. Each reason is audited on its own:
cluster.fleet_silence(_clear) as before, now per reason, and
cluster.ctl_queue {queued_ms} / cluster.ctl_queue_clear {lasted_ms,
queue: drained|unknown}. The Cluster Nodes page shows one alert per
reason, with a new en.ini string for the queue.
Tests: a 5 s queue raises nothing; over 5 s raises the guard with its
reason and holds a lone silent node until the queue drains; a pool that
cannot tell never raises it; both reasons together, each clearing on its
own; another pass joining a run, and a gap starting a new one; a legacy
health.json.
fcgiSend() took its connect errno as a nullable by-ref parameter and
passed it straight to stream_socket_client(), whose out-parameter PHPStan
types as int|null; PHPStan flagged the by-ref type as never assigned
null. The errno is now read into a local and handed back as an int, 0
when the connect was not the cause, and the probe initialises it before
the call.
Plan section 8 has a cluster_ctl listen queue lasting over 5 s raise the
fleet guard. FPM's status page counts `listen queue` only on TCP sockets:
on the pools' unix sockets it always reports 0 (checked with php-fpm 8.3).
FPM also serves that page from a worker, so a status request to a pool
without a free worker waits in the very queue it asks about.
ClusterPool::listenQueueMs() therefore times its own `GET /status?json`
over the pool's socket and says how long the queue has lasted, in ms:
- answered within 250 ms with FPM counting no queue: 0;
- not answered: the request is kept waiting and read without blocking
on later calls, and the queue has lasted since the first one that
waited; a late answer is followed at once by a new request, so the
queue ends only when one is answered in time;
- a connect refused with EAGAIN (the backlog is full) and FPM's own
count above 0 are queues too;
- no socket, nobody listening, a cut answer, no status page or another
pool's status: null, the pool cannot tell.
The ping and the probe now share one non-blocking FastCGI client.
Tests: the probe against a FastCGI responder, a full backlog, the ping
on the shared client, and opt-in against a real php-fpm (XCVM_TEST_FPM).
A node's replica now owns the servers cache and the crontab's jobs only
once an authoritative apply has built them (a `replica_owned` record in
tmp/cache), not as soon as the agent has stored the sections. A refused
section, and CONFIG turned off, hand the caches back to MAIN's database.
cron:cache applies the replica from disk every minute while CONFIG is on,
and getAll() rebuilds a missing owned cache from disk and never writes
the database's rows over one. A database copy with frozen liveness can no
longer pass for the replica's, and an old crontab is not reused after
CONFIG was off.
- Both crontab readers write ReplicaApply::crontabText, which is tested.
- The crontab section leaves out rows a node would refuse and audits them
once. cronJob() anchors with \z, so a trailing newline is refused.
- Migration 043 gives the plan's MAIN-only crons the role main (maxmind
stays all), so the section sends no job the LB build strips.
- config.changed goes only to agents that say config_changed at hello,
each node on its own; a clone quarantine announces it too.
- The ETag cache's generation is a random token, so racing bumps differ.
- cron:certbot reads its own certbot_ssl from its row; service runs the
boot apply only on a CONFIG node, with a 15 s timeout.
- Tests for the review's surviving mutations, test isolation fixes, and
the ADR, the plan and the agent contract brought in line.
Migration 033 marked cron:cleanup 'main', but only its table rotation
is MAIN's: on every node it prunes the node's own stream files, TV
archive and created channels. The role changed nothing while nodes
copied the crontab whole; the replica's crontab section filters rows by
role, so a node with CONFIG on would have stopped that pruning and let
its archive fill the disk.
Migration 042 (and database.sql) gives cleanup the role 'all', as the
plan's cron table has it. A test builds the crontab section from the
install's own rows and checks that every job a node runs is kept and
MAIN's own ones are not.
A config request that read the old rows could write its cached section
after a revoke had bumped the cache. For the next 10 s that section was
served again, so a node pushed by config.changed could be told its list
was unchanged and wait for its next poll.
The cache now has a generation: each entry records the generation read
before the database was, a bump raises it, and an entry from an older
generation is never served. The ADR also records that cluster:apply at
boot still needs MAIN's database until ReplicaStage exists.
A whole section is a granting rep record, so without a licence MAIN
cannot sign it, and the whole config call was refused with
LICENCE_INVALID. As long as any section the node names had changed,
every poll failed, and the blocklist's bans, which sign without a
licence, never reached the node either. The more sections an agent
names, the likelier that is.
A whole section MAIN cannot sign is now left out of the reply, which
agents already read as "keep what you hold" (today's agent skips an
absent settings field), and the blocklist part of the same reply is
served as before. Other refusals still deny the call.
ADR 0004 gets the fifth Phase 7 increment: what each section holds and
what it never carries, how it differs from the plan (whitelist_ips,
xc_vm_version and time_offset carried, one counter for urls_ver and
policy_ver, the legacy crontab role, config.changed on activation too),
the 10 s ETag cache and its bumps, config.changed, how the node applies
each section and when the replica owns a cache, the known limits, and
the agent's contract for XC_VM_Fanout. The plan's Phase 7 status lines
move the four sections to built and name what is left.
The node applies the replica's whole sections servers, node, crontab and
cluster (ReplicaApply), from the replica/<name>.json files the agent
writes after verifying each rep record.
- servers + node: with CONFIG off, a shadow report of the servers and
fields that differ from the cache cron:cache built from MAIN's
database; with CONFIG on, the servers cache in getAll's exact shape,
built through ServerRepository::decorate (split out of getAll), the
node's own row from the node section, api_url from the node's own
settings, and liveness unknown (null; server_online is enabled).
- crontab: shadow lists MAIN's rows the node leaves out; with CONFIG on
the cron_jobs cache, read by generateCron and cron:root_signals. A
job must be a cron: name and five schedule fields.
- cluster: compared with the agent's policy; no PHP on the node reads it.
Once the agent has stored a section and CONFIG is on, the replica owns
its cache: getAll returns it however old, even forced, cron:cache stops
writing it, and the crontab readers leave the crontab alone until it
is applied. A missing, foreign or malformed section never writes or
clears a cache, and a node whose agent predates the sections keeps
MAIN's database. service runs cluster:apply --from-disk at boot, so a
node rebooted without MAIN rebuilds its caches from disk.
NodeFlows no longer recurses when its MAIN check reads servers that the
replica owns: the inner call gets what the agent's file says.
MAIN now builds the rest of the replica's R1 whole sections (plan,
section 9) and serves them through the config op exactly as the
settings section: a sealed, panel-signed rep record per node, sent only
to an agent that names the section in have, and unchanged while the
node holds its ETag.
- servers: every server's routing and relay fields and the node list
{sid, gen, state, ed_pub}; node: the node's own row (ports, limits,
services, disk, HTTPS, interface, governor, sysctl, clock offset) and
its nginx settings; crontab: the enabled rows whose role fits the
node's mode; cluster: the transport policy, panel keys, min_proto and
off-air file names. Core/Cluster/ReplicaSections classifies every
servers column, so liveness, telemetry, watchdog_data, php_pids and
api_url never ship, and a new column fails the test until classified.
- The ETag (SHA-256 of the canonical data) and its section are reused
for 10 s (ReplicaEtagCache), and dropped by SettingsChangedEvent, the
new ServerSavedEvent and CrontabChangedEvent, which the admin's saves
now dispatch to a core listener.
- A revoked, re-enrolled or newly active node drops the cache and sends
every other node taking commands a restrictive config.changed at once;
cluster:exec acks it as deferred for an agent that does not run it.
Old agents name only blocklist and settings, so nothing changes for
them.
Bus heartbeats now belong to the enrolment that sent them. cl:hb
carries the node's gen, RECORD_LUA keeps root_ready and the pending
document only within the same gen, and a flush writes cluster_nodes
and servers.status only while the row's gen matches. Re-enrolment and
revocation drop the node's cl:hb, cl:hb_flushed and cl:tel:<sid>
(HeartbeatService::forget). A stale beat can no longer set root_ready,
last_seen_at or status 1 on a re-enrolled row.
After a root_ready change the flush reads the column back, so a beat
that hello overtook is not recorded as written, and the next heartbeat
is flushed at once instead of 5 s later. A node that is flushed and
forgotten in the same pass no longer leaves its cl:hb_flushed record
behind.
When a pass reads nothing from the bus (bus lost, or restarted empty),
LivenessService uses the heard times it last read from that bus for up
to FLUSH_EVERY_MS. MySQL alone could then be up to 12 s behind a live
node and turn it suspect.
Tests cover each guard the review found untested: the authority gate,
a refused servers write, telemetry judged on heard time, the forget
guards and cleanup, a future flusher stamp, the lock token, carried
root_ready and telemetry, re-enrolment and revocation, and bus loss.
Beats no longer depend on wall-clock time. QueryLogDb is loaded from
the test bootstrap.
ADR 0004 now states the agent contract precisely: a cadence of 3 s at
most, root_ready timing, and the size rule for MAIN's own encoding. It
also covers cl:tel under volatile-ttl and the bus-loss liveness bound.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
The previous commit lets markUsed() skip epochs no newer than the
node's current one, on the invariant that the current epoch was marked
used. In production a failed UPDATE returns false instead of throwing,
so the node could have moved to an epoch whose used flag was never
written, and nothing would mark it later: a refresh authenticated with
the epoch before could then replace the node's current epoch.
markUsed() now advances cluster_nodes.epoch only after the used flag
was written; otherwise the next request with that epoch marks it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
ADR 0004 gains the third cluster bus increment: the cl:hb and
cl:tel:<sid> keys, the flusher in the liveness loop and its rules,
heartbeats on the bus only while a flusher finishes clean passes,
liveness on the freshest of the bus and MySQL, the fallbacks, the
orphan purge's bound, how it differs from the plan, and the agent's
contract (no wire change; the cadence, root_ready and size MAIN
relies on). The plan's Phase 2 line no longer says each heartbeat
writes cluster_nodes, and Phase 3 lists the bus path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
TokenService::markUsed() ran an UPDATE on cluster_node_epochs for every
authenticated request, a no-op after an epoch's first use, so even with
heartbeats on the cluster bus each heartbeat still wrote MySQL.
An epoch no newer than the node's current one was marked used when it
became current: every newer epoch is minted above cluster_nodes.epoch
(refresh, re-key), and enrolment starts from epoch 0. markUsed() now
returns at once for such an epoch and writes only on an epoch's first
use, as before.
ClusterApiTest checks a heartbeat end to end on the bus: MySQL sees
only the two reads of authentication (the node and its epoch), and the
flush then brings last_seen_at, the clock offset, root_ready and the
server's status.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
Plan section 8 has the health loop copy heartbeats from cl:tel:<sid>
into MySQL every 5 s, and heartbeats hold no DB connection. Until now
every heartbeat (every 2 s per node) wrote cluster_nodes, the servers
status and, for TELEMETRY nodes, read and maybe wrote servers.
While the bus runs and a flusher has finished a clean pass within 5 s,
HeartbeatService::record() now asks MySQL nothing: the time MAIN heard
the node, its clock offset and root_ready go to cl:hb, and the
telemetry document to cl:tel:<sid> (10 min TTL), in one Lua script.
HeartbeatService::flush(), run by LivenessService::tick every second
(and by cron:cluster each minute), copies them under a bus lock:
- cluster_nodes at most every 5 s per node, at once for a new node or
a changed root_ready, never taking last_seen_at back (hello may be
newer) unless MySQL's is ahead of MAIN's clock;
- servers.status = 1 with each of those writes, as each heartbeat did;
- each authoritative telemetry document into servers and servers_stats
by the direct path's own 5 s / 60 s rules, on the time MAIN heard it,
so the rows are the ones each heartbeat wrote.
Liveness and the Cluster Nodes page judge the later of the bus and
MySQL, so a MySQL copy one flush behind never makes a node look silent.
Without the bus, when a script fails, for a document over 128 KiB, or
when no flusher has finished a clean pass for 5 s (signals daemon down,
MySQL refusing the flush), each heartbeat writes MySQL itself exactly
as before. HeartbeatService::telemetry() reads the newer of cl:tel:<sid>
and the tel_<sid>.json shadow file, which only the direct path writes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
EventIngest's class docblock said every batch commits with its cursor under
the lane lock; since the P2 lane (conn.touch) that holds for P0 and P1 only,
as the ADR says. The lane locks are real flock files, and every test class
that ingests P0/P1 events but ConnectionIngestIdempotencyTest left them in
the shared TMP_PATH or system temp dir, where suite runs from other
checkouts contend on the same files. The test bootstrap now points
EventIngest::useLockDir() at a directory of the process's own under
tests/.tmp, removed at exit; useLockDir() returns the directory it replaces
so ConnectionIngestIdempotencyTest puts the suite's back. The ADR's sixth
increment notes that touches on the bus and conn.divergence were built in
the tenth.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
Review of the Phase 2, 4 and 6 acceptance-test fixes.
The fleet silence guard has no time limit: a lasting loss of most of the
fleet (both LBs of two) never clears it, so those nodes' viewers were
never purged and kept counting against their lines. HlsReaping now holds
the orphan watch for at most 4 x cluster_orphan_conn_ttl_sec from the
pass that first saw the guard, and never holds a node that was offline
before it came up. A purge during a longer cut of MAIN's own network is
undone by the nodes' snapshots when they come back.
A hard-mode LICENCE_INVALID answers a request whose MAC cannot be
checked, so it handed a node's pending kills, with viewer uuids and
worker pids, to anyone who named the node. The list is now SEALed to the
node's box key (commands_sealed, purpose "commands"). The agent contract
says it runs them without raising its long-poll high-water and keeps
their cmd_ids, so a granting command queued before them is still
delivered once the licence is back.
ConnectionSnapshot applied 20k records as separate autocommitted writes
after deleting the staged chunks, answered ok when the connection was
lost part-way, and removed a viewer whose write had failed. In MySQL mode
the apply is now one transaction whose failed commit answers 503 DB with
the chunks still staged, and the removal pass spares every uuid the
snapshot names.
Tests: MainOutageNoPurgeTest (lasting majority loss, offline before the
guard, never-heard node, a five-minute outage); HardModeKillChannelTest
(forged request, other node's, expired and acked kills, the licence back,
the fake's restrictive list); LargeSnapshotChunkingTest (connection lost
mid-apply, a failed write); ConnectionIngestIdempotencyTest (the cursor
read and moved under the lane's lock, the cursor failure on Redis too,
EventIngest::useLockDir); HttpsRequiredRecoveryTest (every op refused,
the nginx HTTPS wiring, cluster_legacy_ports kept). ADR 0004 and the
plan record each choice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
LargeSnapshotChunkingTest held all twenty chunks at once and ran in the
suite's own process, so the suite's peak memory went from about 102 MB
to 127 MB, just under a 128 MB memory_limit. It now builds each chunk
when it sends it, computes the registry's digest from uuids and owners
only, and runs its tests in separate processes, as ClusterApiTest does
for its tests with global state. The suite's peak is back to 102 MB,
and each 20k run peaks at about 89 MB on its own.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
- Plan section 13: HttpsRequiredRecoveryTest, HardModeKillChannelTest,
ConnectionIngestIdempotencyTest, MainOutageNoPurgeTest and
LargeSnapshotChunkingTest leave the "Not built yet" lines. Phases 2, 4
and 6 gain a line each for what the tests made MAIN do: plain HTTP
refused under https_required and policy versions raised on a transport
change, kills in the hard mode's LICENCE_INVALID, one batch per node
and lane with its cursor, and the orphan purge counting from
cluster_ready_at and holding under the fleet silence guard. The
agent's side of HTTPS_REQUIRED and of the denial's commands stays
listed as not built (XC_VM_Fanout).
- ADR 0004: a section on the tests that found gaps and how each was
closed, where they differ from the plan (health stays on plain HTTP;
the guard also restarts the orphan watch) and the known gap left (a
close's activity row is written outside the batch's transaction). The
orphan rule and the list of settings that bump cluster_policy_ver are
brought up to date.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
Under cluster_transport = https_required the plan serves only GET
challenge over plain HTTP, and lets agents adopt a policy only if its
policy_ver is not lower than the one they hold. Neither held on MAIN:
- MAIN did not know which transport a request came over, so HTTP was
served for every op whatever the transport said;
- saving a new transport (or cluster_main_host) left cluster_policy_ver
as it was, so nodes never noticed the new policy in their heartbeats,
and a policy recorded before the switch, genuinely signed and of the
same version, could be replayed to them;
- a settings POST could set cluster_policy_ver itself, back to 1.
The cluster front controller now passes nginx's HTTPS flag, and under
https_required every op but challenge (health is served before the
settings are read) gets a panel-signed 403 HTTPS_REQUIRED over plain
HTTP. A settings save that changes the transport policy raises
cluster_policy_ver in the same UPDATE, and a POST can no longer set
cluster_policy_ver or cluster_legacy_ports. ClusterSettings gets a test
seam for the HTTPS self-probe.
HttpsRequiredRecoveryTest (plan section 13, Phase 2) runs the drill
through the admin's save path: a node enrolled under https_required
loses HTTPS, is refused on plain HTTP, polls the signed challenge,
rejects a replayed older policy, and is back on plain HTTP with no SSH
once the admin switches to auto.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
With lb_revocation_mode=hard and no licence, the extension refuses the
node's session (CLUSTER_SESSION_HARD), so neither the commands long-poll
nor a MAC'd reply reaches the node. Kills, drops and closes were still
signed, being restrictive, but sat in cluster_commands until they
expired: the plan's "next heartbeat gets a signed denial with pending
kills and stops" was not built.
A LICENCE_INVALID from the session check now carries the node's pending
restrictive commands (CommandBus::restrictive), each under its own cmd
signature, for a node that takes commands. The request is not
authenticated, so the read changes nothing: no command is marked
delivered. The agent checks them as it checks the long-poll's; taking
them from the denial is the Go agent's side.
FakeClusterCrypto now does what the extension does: it refuses a hard
session without a licence, and classes a cmd record by its type, so a
restrictive command signs without one. HardModeKillChannelTest (plan
section 13, Phase 4) covers both modes: graceful delivers kills on the
long-poll, hard in the denial, never the RPC queued before the lapse,
and only to a node whose COMMANDS flow is on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
EventIngest trusted the cursor in the node row the request had read
when it authenticated. A node whose events request timed out sends the
batch again while MAIN may still be applying the first copy; both
requests then passed the P0 gap check with the same cursor and applied
the batch twice, so a closed viewer was re-opened and its close wrote a
second activity row.
A failed cursor write was also ignored: when the database connection
dropped mid-batch, DatabaseHandler::query() answered false, the
transaction's writes were gone, and the node was still told the batch
was applied, so it moved on past events MAIN never kept.
MAIN now applies one batch per node and lane at a time, under a file
lock in TMP_PATH/cluster_ingest/ (never the node's database row, which
every heartbeat writes), and reads the cursor under that lock. A cursor
that is not written fails the batch; ClusterApi answers 503 DB and the
node sends it again.
ConnectionIngestIdempotencyTest (plan section 13, Phase 6) replays and
duplicates conn.upsert, conn.remove and conn.close on both stores: a
batch resent after a lost reply, one resent while the first copy was
being applied, the same event under new numbers, a batch waiting for
the one on its lane, and a batch whose cursor could not be written.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
The plan's Phase 6 acceptance asks that a 20 000-connection conn_snapshot
apply atomically, and that a bad chunk leave MAIN's store unchanged.
ConnectionSnapshotTest covered the rules with a handful of records;
LargeSnapshotChunkingTest runs them at the acceptance size against
lines_live as install/database.sql creates it.
Twenty chunks of 1000 are staged without touching the store, which then
changes at once with the last chunk: every record upserted in place, the
node's stale viewers removed, another node's untouched, the digests in
agreement and nothing left staged. Oversized, malformed and past-the-cap
chunks are refused, chunks out of order or of another snapshot get the
expected number, and a staged chunk that no longer reads makes the last
chunk apply nothing; in each case the store is byte-for-byte as before,
and the snapshot sent again applies.
No code change: the staging already behaves as the plan asks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
The orphan purge measured a node's silence from last_seen_at alone and
kept its own watch across reaper gaps of up to three minutes. After a MAIN
restart shorter than that, a watch that began while MAIN's nginx was
already stopping survived, and the first users cron pass purged every
CONNECTIONS node's viewers before the nodes could reconnect. It also
ignored the fleet silence guard, so a cut in MAIN's own network purged
the whole fleet once the TTL was up.
HlsReaping::begin() now counts silence from max(last_seen_at,
cluster_ready_at), as NodeHealth already does, and orphans nobody while
the guard is up; the watch starts over under the guard, so the time MAIN
suspected itself never counts.
MainOutageNoPurgeTest (plan section 13, Phase 6) runs the orphan watch
and ConnectionIngest::purgeNode across a MAIN restart and under the
guard, and pins that liveness right after a restart marks nobody
offline. HlsReapingTest now keeps its health file private.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
obscuremind