phpcs (the repo ruleset, run locally without the Slevomat sniffs, which
could not be fetched here) flagged 22 auto-fixable issues in code from
this series: multi-line IF closing parentheses, one argument per line in
SessionKeys, short array syntax in the restored SegmentReader, and blank
lines before functions. Fixed with phpcbf; the whole tree is clean again
under that ruleset. No behaviour change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Es8UhyPS8MNjuYTagqfGWi
- `console.php xcvm_core status` reports the loaded extension and its
cluster API: the version against the range the panel speaks, whether
the root is initialised (with its fingerprint), the licence binding kid
and the clock state.
- The installer rolls back an update whose cluster API falls outside
ClusterCryptoFactory::API_MIN..API_MAX when the installed one was
inside it, so an update cannot silently strand enrolled nodes.
Installing an exact pinned version still needs versioned paths in the
binaries repo.
- ClusterExtensionIntegrationTest runs the panel facade against a real
test-hooks build, and skips without one or without a throwaway
XCVM_CONFIG_DIR. It covers init, token issue, the agent opening its
token, MAIN's session, BOX both ways, per-node binding of epoch records,
and signing with record classes. It passes against the 2.2.2 build.
- ADR 0004 records the panel's side of the contract:
- who owns which part of the wire;
- the canonical request/response form;
- fail-closed rules;
- migration renumbering (the plan's 026–032 ship as 028–034);
- where the settings live;
- the measured crypto budget.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Es8UhyPS8MNjuYTagqfGWi
Core\Cluster\ClusterSettings owns the defaults and bounds of the 19
cluster settings. normalize() clamps numbers, falls back to the default
for an unknown enum, and refuses rather than clamps where clamping would
change what the admin meant:
- a cluster_api_port outside 1024–65535, or one MAIN already uses
(broadcast, extra and RTMP ports) or that is reserved (31210, 31290,
3306, 6379);
- a cluster_main_host that is not a DNS name;
- scan roots that are not absolute paths free of '..';
- https_required, unless MAIN's own HTTPS verifies (httpsSelfProbe: a
non-IP domain and a certificate with more than 7 days left) and no
active node blocks it (every active node blocks until Phase 3
telemetry reports HTTPS);
- enabling the LB API without xcvm_core's cluster API
(ClusterCryptoFactory);
- API mode for new nodes before the cutover phase.
SettingsService::edit() runs these checks. A refusal fails the whole
save, and its translated message is shown on the settings page. It lives
in Core/Cluster, which ships to LBs, so nodes apply the same bounds.
The settings page gets a Cluster tab after Streaming. It shows every
field, and a status line: the extension and its API range, whether HTTPS
is available, and the derived token grace. The strings come in all 8
languages.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Es8UhyPS8MNjuYTagqfGWi
Migrations 028–034 (the plan's 026–032; 026 and 027 are taken by the SSH
host key and the fanout switch), each with a down file, mirrored in
database.sql:
- 028: the 19 cluster settings with the plan's defaults. They stay inert
until cluster_api_enabled and cluster:init.
- 029: cluster_nodes (mode, flow bits, gen, node keys, state, sequence
high-waters, row MAC), cluster_node_epochs (the extension's sealed epoch
records; deleting a row erases z) and cluster_meta.
- 030: cluster_commands, FIFO per node with UNIQUE(server_id, seq),
UNIQUE(server_id, dedupe_key) and UNIQUE(cmd_id).
- 031: cluster_enrol_codes (sealed K_req/K_res under a row MAC, looked up
by SHA-256 of the secret) and cluster_enrol_requests (UNIQUE(server_id):
no bulk approve).
- 032: cluster_audit, cluster_nonces (replay cache) and
cluster_reservations (MySQL-mode admission).
- 033: crontab.role (all/main/legacy). The four crons Phase 0 gated to
MAIN become 'main', and a disabled 'cluster' row is added for
cron:cluster (Phase 2).
- 034: cluster_changes (blocklist delta log) and cluster_stream_ver (R2
versions).
Checked on MariaDB 10.11:
- a fresh database.sql and the previous database.sql plus these
migrations produce identical columns, indexes, cron rows and defaults;
- the migrations are idempotent;
- the down files remove everything and up re-applies.
ClusterSchemaTest keeps the two copies in step in CI.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Es8UhyPS8MNjuYTagqfGWi
src/Core/Cluster/Crypto implements the wire formats of the extension's
cluster API v1 (xcvm_core ADR-002), in plain PHP with sodium and OpenSSL:
- Enc: the lp()/u64 canonical encoding.
- Box: XCVM-BOX-v1 session bodies.
- Seal: XCVM-SEAL-v1 to an X25519 key; refuses the all-zero shared secret
and binds purpose and context.
- PanelSig: verifies panel signatures against the closed tag registry.
- NodeSig: the node key's own signatures, under a separate domain string
so they can never pass as panel records.
- Canonical: the panel-owned request/response serialisation. It is the
BOX context and the MAC input, with a ±90 s window and header parsing.
- Ticket, RelayAuth, FileDigest: relay and file tickets, the child's
per-connect relay proof, and owner-signed file digests.
- ClusterCrypto: a facade over XC_VM::cluster_*. Refusals throw
ClusterRefusedException with the extension's reason code.
- ClusterCryptoFactory: fails closed (ClusterUnavailableException) without
the extension, outside API 1–1, or without sodium/AES-GCM. No PHP
fallback ships.
Tests:
- The panel matches the extension's shared vectors byte for byte
(tests/Support/cluster_vectors.json, copied from xcvm_core): SEAL, BOX,
panel signatures, the licence binding and CK_B, and the whole token
chain an agent runs.
- tests/Support/cluster_canonical_vectors.json freezes the canonical
request/response bytes for the Go agent.
- Also added: SignTagSeparationTest, ClusterTicketsTest,
ClusterCryptoFailClosedTest (including a check that nothing under
Core/Cluster signs as the panel or computes tokens) and
ClusterCryptoBenchTest.
- Timing: 64 KB is ~0.25 ms p99 against the 1 ms budget; 8 MB is ~41 ms
here against a 40 ms target, guarded at 2x.
The reference crypto that only xcvm_core or the Go agent computes
(binding, token MAC, session keys, panel signing with a seed) lives in
tests/Support and never ships.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Es8UhyPS8MNjuYTagqfGWi
The last two Phase 0 seams:
- Core\Cluster\NodeRpc wraps every ApiClient::systemRequest() and
asyncRequest() call. It covers 37 call sites across the admin
controllers, the admin API wrapper, ServerRepository/ServerService,
StreamService, DiagnosticsService and AuthRepository. The HTTP transport
is unchanged. In Phase 4 these become node.rpc{action} commands.
- Core\Cluster\NodeActions wraps every SignalDispatcher::rootAction() call
(26 sites) with typed methods: reboot, restartServices, update,
rollback, updateBinaries, setRamdisk, setPorts, setServices,
setGovernor, setSysctl, flushBlocklist, certbot. send() takes payloads
the caller builds (modules, OPENSSL_EXTRA). The queued JSON is
byte-for-byte what the callers wrote. In Phase 4 these become
node.root{action} commands.
Each seam refuses an action outside its catalogue (NodeRpc::ACTIONS,
NodeActions::ROOT_ACTIONS). NodeRpcActionsTest checks that:
- every call site uses a catalogued RPC action;
- RootSignalsCronJob handles every catalogued root action;
- nothing outside Core\Cluster calls the raw functions.
docs/en/development/cluster-seams.md lists all the Phase 0 seams, their
legacy backends and the rules for new code.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Es8UhyPS8MNjuYTagqfGWi
Three Phase 0 seams, each with the current database behaviour as its only
backend:
- StreamSource: the stream row, this node's streams_servers row and the
stream options a node reads before running a stream. The launcher
(live, VOD and LLOD starts), MonitorCommand, ProxyCommand, ScannerCommand
and live.php's fanout proxy hand-off ran these queries themselves.
- StreamRowMerge: MAIN's side of a node's runtime state. StreamStateWriter's
SQL backend now calls it; later it merges `stream.state` events keyed by
the reporting node. eventFields() keeps only runtime columns and redacts
the source URL.
- StreamCacheBuilder: the stream_<id> cache entry (columns, server rows,
the direct-source rule), extracted from cron:cache_engine so the R2
stream delta can build the same shape.
OndemandCommand's stop path also still wrote streams_servers and signals
with raw SQL. It now uses StreamStateWriter and SignalDispatcher, like the
rest.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Es8UhyPS8MNjuYTagqfGWi
The node-side log writers now hand their rows to Core\Cluster\LogSink by
type, instead of each building its own INSERT: client request logs
(lines_logs), stream logs (streams_logs), stream errors (streams_errors),
panel errors (panel_logs, INSERT IGNORE) and restream detections
(detect_restream_logs). The legacy backend writes the same multi-row
INSERTs, chunked at 1000 rows, with bound values instead of hand-escaped
SQL (streams_logs used addslashes). A useSink() hook is the seam the
cluster API's log.* events plug into (spec Phase 0/5).
Core\Cluster\Redactor strips credentials before anything is journaled:
password/token/username values, the /user/pass/ segments of Xtream URLs,
and user:pass@ in URLs. The API backend will use it; the SQL backend
writes what it wrote before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Es8UhyPS8MNjuYTagqfGWi
With fanout switched off (FanoutMode::legacyDelivery(), which is also true
when the licence denies fanout), live delivery goes back to the paths that
ADR 0003 Phase E removed. Each is restored from git history:
- HLS (E3): live.php serves HLSGenerator::generateHLS() over the on-disk
playlist. segment.php serves on-disk LIVE segments again through
X-Accel /xc_hls/, AES-128-encrypting on first read, and burns a pending
send-message signal with SignalSender. generateHLS now mints tokens with
Encryption::mintToken (so secure_stream_tokens applies), rewrites one
segment line at a time (a str_replace of 1_1.ts also hit 11_1.ts), and
tokenizes an fMP4 init map.
- TS (E2): live.php chase-reads the on-disk segments again (SegmentReader
and its test restored). $rSegmentSettings, which the old loop read but
never defined, is now set there. A signalled segment is no longer sent
twice.
- Proxy streams (E1): StreamProcess::startProxy() starts ProxyCommand
(XC_VMProxy[<id>]), and live.php relays its datagrams from the viewer's
unix socket. Fixed while restoring:
- the read loop ended after its first read (a stray break);
- it spun on an idle source;
- it never exited when the last viewer left;
- it passed the proxy argument's whole row to ffmpeg;
- it ran ffmpeg with no input when no source opened.
Runtime state goes through StreamStateWriter.
With fanout on, nothing changes: a daemon that is merely down still gives
not-on-air until the keepalive restarts it, and segment.php still 404s
non-daemon live segments.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Es8UhyPS8MNjuYTagqfGWi
settings.fanout_enabled (migration 027, default on) is a new "Fanout
Delivery" toggle beside the other fanout settings. Streaming\Fanout\FanoutMode
reads it and applies it to a node:
- Off: FanoutMode::applyToNode() writes bin/xc_fanout/disabled, then stops
run.sh (so it cannot respawn the daemon) and then xc_fanout. It runs on
MAIN when the settings are saved, and on every node from RootSignalsCronJob
within a minute. The RootSignals keepalive and the hourly fanout_binary
self-heal skip the daemon while the flag is set; `service boot` and run.sh,
which cannot read settings, check the flag file.
- Every daemon-facing gate honours the switch. LicenseGate::fanoutUsable()
and each FanoutClient control-socket call behave as if the daemon were
down. IngestFeeder::forStream() returns a no-op feeder, and DelayCommand
stops queueing segments for it. fanout_sync stops writing the daemon
config and closes the pid-0 connection rows the stopped daemon left.
- The dashboard no longer reports "xc_fanout down" when it was switched off.
The pre-fanout delivery paths for fanout off follow in the next commit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Es8UhyPS8MNjuYTagqfGWi
A node records the runtime state of the streams it runs (pids, status,
probe results, progress, current source) by updating its streams_servers
row; ~30 places did that with their own UPDATE. They now call
StreamStateWriter::update(stream, server, fields) or updateRow(id, fields),
which accept only the runtime-state columns (STATE_FIELDS) and write
through a sink. The SQL sink issues the same UPDATE as before
(StreamStateWriterTest); the cluster API plan (Phase 5) adds one that sends
a stream.state event for MAIN to merge instead.
Desired state (what runs where, parents, on-demand, the panel's edits)
stays MAIN's and is untouched. Phase 0 seam.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Es8UhyPS8MNjuYTagqfGWi
DatabaseHandler connects in its constructor, so every
DatabaseFactory::connect() in live/vod/timeshift and the shutdown handler
opened a MySQL connection to MAIN whether or not a query followed. The
cluster API plan counts every such connect before a node may leave the
legacy link.
LazyDatabaseHandler connects on the first query, simple_query, escape or
transaction and otherwise behaves like the eager handler, failure
included; DatabaseFactory::connectLazy() hands it out and keeps an
unopened one instead of replacing it. The streaming endpoints use it
(LazyDatabaseHandlerTest). Phase 0 seam.
Also replace call_user_func_array('parent::query', ...) in
DatabaseHandler::query() with parent::query(...func_get_args()): callable
strings using "parent" are deprecated since PHP 8.2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Es8UhyPS8MNjuYTagqfGWi
Once the signals daemon had run its first settings check, it called
ServerRepository::getAll(true) - a SELECT * FROM servers against MAIN - on
every pass of a loop that sleeps 250 ms: up to 240 reads a minute from every
node. The watchdog and the cache handler tied the reload to the settings
interval instead (10-60 s), so their view of the cluster could be a minute
stale.
DaemonTrait::serversRefreshDue()/refreshServers() reload at most every five
seconds, and the three daemons use it: 12 reads a minute, never more than
five seconds stale (SignalsLoopQueryRateTest). Phase 0 of the MAIN <-> LB
API plan.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Es8UhyPS8MNjuYTagqfGWi
The cluster API plan switches a node off the legacy link only after seven
days with no connect to MAIN's MySQL or Redis, counted at
Database::db_connect() and RedisManager::connect(). Both now call
ConnectAudit::record(), which appends {t, kind, call site, pid} to
STORAGE_PATH/cluster/sql_audit/YYYYMMDD.ndjson (persistent, unlike tmp/)
and summary() turns into per-site counts for the heartbeat later.
Trace only and off by default: NodeRole::auditConnects() is on when
sql_audit/enabled exists (the cluster API sets it for hybrid nodes) or
under XCVM_CONNECT_AUDIT=1, checked once per process; nothing is refused.
cron:cleanup prunes day files older than eight days. Phase 0 seam.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Es8UhyPS8MNjuYTagqfGWi
47 call sites wrote `signals` rows directly, in three shapes: a viewer
kill (pid, rtmp), a root action for cron:root_signals (custom_data.action)
and a cache job (cache = 1, custom_data.type). They now name the intent -
SignalDispatcher::kill / rootAction / cache / cacheBatch - and the
dispatcher writes through a SignalSink. The only sink is LegacySqlSignalSink,
which writes the same rows; the cluster API plan (Phase 4) adds the sink
that turns them into signed commands for API-mode nodes. Phase 0 seam.
Behaviour kept column for column (SignalDispatcherParityTest), with two
deliberate differences:
- Actions the panel wrote as literal JSON with a space ('{"action":
"update_binaries"}', restart_services, reboot) are json_encode()d like
every other one. StatusCommand deletes update_binaries rows by their
json_encode() form, so the panel-queued ones were never cleaned up.
- kill() writes nothing for a pid <= 0: the signals daemon skipped those
kills, but for RTMP it sent a drop for client 0.
The "insert unless an identical cache signal is pending" check that
LineService and StreamProcess repeated is cache(..., once: true).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Es8UhyPS8MNjuYTagqfGWi
server:install received the root username and password as positional
arguments, visible in ps and /proc/<pid>/cmdline for the whole install, and
LbInstallFlow/ProxyInstallFlow saved them in a plaintext
bin/install/<id>.json for reinstalls. Any SSH host key was accepted.
- The panel writes a 0600 bin/install/<id>.cred and runs
`server:install <type> <id> <port> - - … --cred-file=<path>`; the command
reads and deletes the file before connecting (InstallCredentials). Only
<id>.cred files in bin/install/ are accepted.
- <id>.json keeps the non-secret parameters only; reinstall_server now needs
root_password in the request. cron:cleanup (MAIN) scrubs root_password from
files written by earlier releases.
- The node's SSH host key (SHA-1) must match the optional Expected SSH Host
Key (hex or `ssh-keygen -l -E sha1` output), else the key stored by the
first install (new servers.ssh_hostkey_sha1, migration 026), else it is
trusted on first use and stored.
- New install-form field and strings in all eight languages; cli-tools docs.
Phase 0 stop-gap of the MAIN <-> LB API plan.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Es8UhyPS8MNjuYTagqfGWi
proxy_api.php trusted the posted server_id. Any proxy could report stats
for any server, collect that server's signals (which the proxy runs as
root), and write its servers.whitelist_ips, which feeds allowed_ips, the
source allowlist of every LB's /api.
The server is now the proxy whose server_ip or private_ip is the request's
source address (ProxyIdentity::resolve); a posted server_id naming another
server gets the 404. Posted addresses are kept as telemetry
(watchdog_data.ips_seen) and no longer written to whitelist_ips, which stays
admin-maintained. Phase 0 stop-gap of the MAIN <-> LB API plan, until
proxies move onto the token scheme (D8).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Es8UhyPS8MNjuYTagqfGWi
Cron locks were CRONS_TMP_PATH . md5(generateUniqueCode(live_streaming_pass)
. class), so rotating the viewer-token secret (the cluster plan rotates it
at cutover and again at lockdown) renamed every lock and let a second
instance of each cron start beside one still running.
ProcessManager::cronLockPath() names the lock from the class alone; CronTrait,
RootSignalsCronJob and server:diagnose use it. For the upgrade itself,
exitIfCronLockHeld() honours a live, fresh lock under the previous name, so
an instance started by the old release is not doubled.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Es8UhyPS8MNjuYTagqfGWi
The crontab is copied verbatim to every load balancer, so each LB ran the
same cluster-wide work against MAIN's database: the retention DELETEs on
lines_activity/lines_logs/login_logs/streams_errors/streams_logs/
ondemand_check, the TMDb and TMDb-popular crawls, the purge of every node's
signals, the GitHub update check that rewrites settings.update_data, and
the marketplace check for module updates (modules are MAIN-only).
Add XcVm\Core\Cluster\NodeRole::isMain() (Phase 0 of the MAIN <-> LB API
plan; the node mode and flow bits land there later) and gate each of those
on it. An unknown answer reads as "not MAIN", so a node that cannot tell
skips one run instead of doing cluster-wide work. Per-node work (local
stream and archive cleanup, this node's certbot, this node's own signals)
is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Es8UhyPS8MNjuYTagqfGWi
The LB strip lists still named files by their pre-PSR-4 www/ paths, so
those entries matched nothing and the files kept shipping under Public/:
Public/admin/api.php, Public/admin/proxy_api.php, Public/stream/auth.php
and Public/stream/probe.php reached every LB. auth.php and probe.php are
routed by the LB nginx but fatal there on the stripped Domain/User.
LB_DIRS still listed the removed resources/ and www/ trees, and
LB_DIRS_TO_REMOVE listed the non-existent Domain/Auth. The gate never
noticed, because it only checked the SENSITIVE prefixes.
Makefile:
- Drop the dead entries (resources, www, Domain/Auth, GeoLite2-City.mmdb,
the nine www/* files).
- Strip the MAIN-only files that no LB route or LB code needs: the four
endpoints above, PlayerV2, AdminAPIWrapper, ResellerAPIWrapper,
ActiveCodeApiController, ResellerApiDispatcher, ResellerTableRenderer,
Cli/migration_logic.php and DbMigrateCommand (on an LB it would run
MAIN's schema migrations). The viewer-API controllers still ship
because the LB nginx still routes them; they go later with the routes.
- lb_delete_files_list now always writes the LB deleted_files.txt as the
LB-scoped git deletions plus every stripped code file (Cli, Core,
Domain, Infrastructure, Public, Streaming; never bin/config/content).
An update only deletes listed files, so without this the files newly
stripped here would stay on LBs that are already installed. The
retired resources/ and www/ trees stay in the git-deletion scope
(LB_RETIRED_DIRS), so their old entries are still applied.
lb_configs/nginx.conf: stop routing /stream/auth, /stream/probe and
/probe.php. They now get nginx's 404 instead of a PHP fatal. MAIN nginx
is unchanged.
verify-lb-archive.sh now also fails when:
- an LB_* entry matches no tracked path (STALE);
- a script the LB nginx executes is stripped (MISSING): every
SCRIPT_FILENAME, and Public/<scope>/<handler>.php for each handler the
/stream/ and /admin/ gateway locations accept.
SENSITIVE covers the new strips, CacheHandlerCommand and
CacheEngineCronJob. The manifest lookups now use here-strings, because
under pipefail an early `grep -q` exit could SIGPIPE the writer and hide
a leak. LbArchiveGateTest runs the real script with each failure mode
injected through MAKEFLAGS.
Behaviour: the LB archive loses 26 files that no LB route reaches (or
that fatal there today), and the next LB update deletes them. It also
deletes any lingering Domain/User or Domain/Device files. Fresh LBs
never had those, so an older LB that still carries them loses rtmp and
viewer-API paths that only worked because of them, just like a fresh LB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
server:sync-openssl-extra now refuses to send when the main's own
published fingerprint differs from the value it read. Run as root, it
could read a config/openssl_extra that xc_vm cannot read, while the
main's php-fpm runs on the default, and push a value that breaks every
LB. The command and cron:root_signals now share one contract:
OpensslExtra::SIGNAL_ACTION, signal() and applySignal(), with
publish() for cron:servers.
Encryption::readToken now tries the sealed format with both values
before the legacy one. A wrong-value legacy decrypt() could return
garbage (about 1 in 256), so a sealed token minted with the replaced
value was refused. Legacy tokens keep that inherent miss rate.
proxy_api trusts the posted server_id and handed any registered proxy
an LB's signal rows, including the main's OPENSSL_EXTRA. It now skips
set_openssl_extra rows until the source-IP binding lands.
Tests cover sync() with TestDb, the signal contract and the case's own
break. The docs cover the check and the repair.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
Load balancers added before provisioning shipped config/openssl_extra
run on the built-in OPENSSL_EXTRA while their main holds a random one,
so they reject every token the main mints for them. server:diagnose now
shows this; nothing could fix it short of a manual copy.
server:sync-openssl-extra <server_id>|--all [--force], run on the main,
queues one root signal carrying the main's value for each streaming LB
whose published fingerprint differs. Nodes that publish none (not
updated), offline nodes and nodes already in sync need --force; proxies
and the main are never sent it. A main without config/openssl_extra
runs on the default like its LBs, and the command sends nothing.
The LB's cron:root_signals handles 'set_openssl_extra' in its own case
(ignored on the main, whose value keys hmac_keys and image names) with
OpensslExtra::install: it writes the value it replaces to
openssl_extra.prev, accepted for 600 s by Encryption::readToken, then
swaps openssl_extra in through a 0600 temporary file owned like
config/, so php-fpm never sees a file it cannot read. php-fpm picks the
new value up on its next request. Installing the value already in use
keeps an earlier .prev. server:install now also clears a stale .prev,
and server:diagnose names the command in its mismatch report.
The value sits in the signals table until the LB's root cron deletes the
row, about a minute; a legacy LB reads that database in full anyway.
The command is MAIN-only: it is stripped from the LB archive and listed
in verify-lb-archive.sh.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
A load balancer that runs on another OPENSSL_EXTRA than its main has to
be moved onto the main's value. Every stream link the node minted with
its old value (HLS segment and key tokens, VOD links a player reuses to
seek, tokens from direct-to-LB auth) would stop opening at that moment.
OpensslExtra::previous() reads config/openssl_extra.prev, the value a
node replaced and when it stops being accepted, and returns it while it
is still valid and differs from the value in use. Encryption::readToken
tries a token that fails with OPENSSL_EXTRA once more with that value.
Only the OPENSSL_EXTRA context falls back and only on the failure path,
so a valid token costs nothing extra. The file is read once per process;
until something writes it (the repair signal that follows), nothing
changes. encrypt/decrypt/seal/open/mintToken are untouched: new tokens,
hmac_keys and image names use only the current value.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
LbInstallFlow::provisionConfig shipped only config.enc. A MAIN set up by
the current installer holds a random config/openssl_extra, so every LB
added with server:install ran on the built-in OPENSSL_EXTRA and rejected
the stream tokens MAIN mints for the redirects it serves (/auth, /vauth,
/tsauth, /thauth, subtitles), and could not decrypt MAIN's hmac_keys.
The new provisionOpensslExtra step removes any config/openssl_extra a
previous MAIN left on the host, then sends the MAIN's file when it holds
a value and makes it 0600. It runs after the config.enc upload and
before config/ is chowned to xc_vm, so php-fpm can read it; a failed
upload marks the server errored (status 4) like a failed config.enc.
When the MAIN has no file it runs on the built-in default, and so does
the LB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
OPENSSL_EXTRA keys the stream tokens MAIN mints for the redirects an LB
serves, and hmac_keys. It is config/openssl_extra, or the built-in
default when that file is absent. The current installer writes a random
value on a fresh MAIN, but nothing ships it to an LB, so an LB added
with server:install runs on the default and rejects every token MAIN
mints for it. Nothing showed which clusters are affected.
Each node's cron:servers now publishes a keyed 64-bit fingerprint of
its value (never the value) as server_hardware.openssl_extra_fp, in the
UPDATE it already runs every minute. server:diagnose compares the
node's fingerprint with the main's, from the main and on the node
itself, and reports "OPENSSL_EXTRA mismatch: tokens minted on MAIN are
rejected by this node". A node or main that does not publish one reads
as "unknown (... not updated)"; proxies are skipped. No behaviour
changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
Review follow-up to the bandwidth/rps/producers fix. The nginx stub_status
parse moves into SystemInfo::nginxRequestCount(), so the requests field the
watchdog samples is tested against a real status body and an unreachable
one (no count, rps 0). The getStats() test now pins the whole array_replace
statement (a `+` would leave the bandwidth keys at 0), and the sampler test
checks that 120 s is in range by default. No behaviour change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
SystemInfo::getStats() feeds the watchdog heartbeat (servers.watchdog_data),
servers_stats and the LB /api stats action, and had four defects:
- bytes_sent_total/bytes_received_total were assigned per interface, so
only the last selected interface was kept, and the received total read
tx_bytes. aggregateNetwork() now sums both totals over the same
interfaces as the per-second rates and reads rx_bytes for received.
network_speed keeps its first-positive rule and the interface selection
is unchanged, so routing capacity is not affected.
- total_running_streams was `ps ax | grep -c ffmpeg`, which counted
ffprobe and shell wrappers and missed the fanout daemon's native
remuxer. ProcessManager::countStreamProducers() now reads /proc and
counts processes whose program is ffmpeg or `xc_fanout remux`.
- The memory percentage divided by total_mem, which getMemory() reports
as 0 without MemAvailable (DivisionByZeroError), and nvidia-smi XML
with a single GPU process decodes process_info as one associative
array (TypeError). Either aborted the watchdog pass, so the node wrote
no heartbeat. memUsedPercent() and gpuProcesses() guard both.
The watchdog runs one pass per process and re-execs itself, so the
previous nginx request count held in a variable was always null and
requests_per_second was 0 on every node. CounterRateSampler keeps the
previous reading in a tmp state file (atomic replace, 120 s max age, 0
after a counter reset).
The watchdog_data key set and order are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
Review fixes for the php_pids change.
- cron:users judged a remote PHP-served connection by date_start, but a
reused connection (VOD range/seek, TS or timeshift reconnect) gets a new
worker pid and keeps its original date_start. A pid assigned after the
node's last heartbeat was then looked up in a list that could not hold
it, and the live viewer was killed. The pid write also stamps
hls_last_read (MAIN's clock), so the later of the two now dates the pid.
The check moves into isRemoteWorkerRunning().
- phpFpmWorkerPIDs() returns null when no FPM master is visible, so an
idle ondemand pool ("[]") now reaps dead workers' rows while an unknown
list still keeps them. A server missing from either map no longer hits
count(null).
- getAllSimple/getStreamingSimple/getProxySimple and the admin API
get_server leave out php_pids; getById keeps the full row for REPLACE.
- Tests pin the watchdog and reaper wiring, exercise sort(), and reset
the ServerRepository db seam; array_unique() is dropped (unreachable).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
A connection row records the PHP-FPM worker serving it (getmypid() in
live.php, vod.php and timeshift.php), but the watchdog published as
servers.php_pids the contents of bin/php/sockets/*.pid: the FPM master
pids. In Redis mode MAIN's cron:users reaps every node's connections
and, for a remote PHP-served connection older than the node's last
heartbeat, checks the worker pid against that list. It never matched,
so the connection was closed and the LB told to kill its worker: every
VOD, timeshift and PHP-served live viewer on an LB was dropped about a
minute after connecting. MySQL mode was unaffected (each node probes its
own pids locally).
- ProcessManager::phpFpmWorkerPIDs() lists the processes titled
"php-fpm: pool xc_vm" (sorted, unique); findProcessPIDs() takes an
optional procfs root so it can run against a fixture tree.
- WatchdogCommand publishes that list instead of the pid files.
- ServerRepository::getAll() drops php_pids from the rows it returns and
caches, so the per-request servers cache does not carry every node's
worker list. CacheCronJob's unset($rServers['php_pids']) meant to do
that but hit a top-level key of an id-keyed map; it is removed.
- UsersCronJob reads php_pids with its own SELECT `id`, `php_pids`.
The `- 1` heartbeat margin is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
live.php's Redis branch now calls ConnectionTracker::acceptedLineIP(),
which skips an empty (HMAC) line id without asking Redis and otherwise
returns the oldest active connection's IP. The HMAC guard and the
oldest-row choice are now covered by tests: a null id never queries
Redis, and unordered LINE# keys yield the earliest date_start's IP.
getLineConnectionRows() takes an int line id; its untested HMAC-identity
form is dropped. The source-grep bans are replaced by a check that
live.php calls the seam, the misnamed key-shape test is renamed, and the
redundant igbinary skip is removed (tests/bootstrap.php polyfills it).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
In Redis mode live.php took ConnectionTracker::getLineConnections() as
connection rows, but it answers the LINE# set's members: connection
keys (uuid strings). array_column() over them gave [], and
array_multisort([], SORT_ASC, $rConnections) threw "ValueError: Array
sizes are inconsistent" on PHP 8 as soon as the line had one active
connection, including the same viewer's next playlist refresh. The
request died with a 500 instead of being limited. For an HMAC token the
line id is null, so passing it to getLineConnections(int) was a
TypeError before any connection was counted.
Add ConnectionTracker::getLineConnectionRows(), which reads and
unserializes the rows behind a line's LINE# keys (an unreachable Redis,
a failed call or an unreadable payload reads as no connection), and
ConnectionTracker::oldestConnectionIP(), which picks the user_ip with the
earliest date_start. live.php now accepts that IP, as the MySQL branch
does with the first lines_live row. HMAC identities skip the check, as
they already do in MySQL mode where `user_id = NULL` matches nothing.
getLineConnections() and ConnectionLimiter are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
Review follow-up to the log-import fix (71108b20):
- Record last activity with one UPDATE per batch instead of an
INSERT ... ON DUPLICATE KEY UPDATE, so activity for a deleted line no
longer recreates it as an empty `lines` row. Each line is set once,
from its latest row, in id order so nodes lock rows alike, and
`updated` is kept so cache_engine does not rebuild the line.
- Flush a batch at 4 MiB of spool as well as at 1000 rows, so oversized
client rows cannot push an INSERT past max_allowed_packet.
- Skip a claimed spool that cannot be opened instead of failing on fgets.
- Document that a claim left by a crashed run is re-imported from its
start (at-least-once).
- Tests: writer appending mid-import, missing keys, the server_id check,
last_activity_array contents, duplicate lines in a batch, byte-split
batches, the lines_logs failed-INSERT path; reset the injected DB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
ActivityCronJob and LinesLogsCronJob drain a base64(JSON)-per-line spool
(LOGS_TMP_PATH/activity, client_request.log) into MySQL once a minute on
MAIN and on every LB. Both parsers broke out of the read loop after the
first row, and ActivityCronJob also stopped at the first invalid row
instead of skipping it. The caller then deleted the whole spool, so each
node kept at most one lines_activity row and one lines_logs row per
minute. The rest were lost, and lines.last_activity was updated for one
closed connection only. In LinesLogsCronJob a line that did not decode
to an array raised a TypeError in array_map() before the unlink, so a
bad line wedged the spool.
Read every line, skip empty, undecodable and (for activity) incomplete
rows, and insert in batches of 1000 so a busy minute stays well under
max_allowed_packet. Each activity batch upserts `lines` with its own
consecutive ids from last_insert_id(). Values come from a fixed key list
with a '' default, so a record with a missing key cannot break the SQL
for its whole batch. The SQL statements and column lists are unchanged.
The spool is now claimed by renaming it to <spool>.import before it is
read. The writers append by path, so rows written during the import go
to a fresh spool and are not deleted with the claimed file. A claimed
file left by a crashed run is imported first. Rows are still dropped
when their INSERT fails; loss-free delivery is planned for Phase 5.
StreamsLogsCronJob and ErrorsCronJob already read every line and are
unchanged. ErrorsCronJob keeps error_log.log when every line is
filtered or its INSERT fails, and StreamsLogsCronJob unlinks even when
its INSERT fails. Both are left for a separate change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
Condensed plan (revision 3 + 28 consistency corrections): LB nodes stop
reaching MAIN's MySQL and Redis and move every exchange to a signed,
encrypted cluster API with licence-derived, rotating node tokens.
This is the source design for the task-level implementation plan.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
Rollback previously swapped in older code but never touched schema
changes newer migrations applied — the rollback's own comment admitted
"migrations are forward-only, so a downgrade cannot undo them". Of the
25 core migrations, two actually DROP a column older code still reads
(013_drop_asn_allocated, 016_per_repo_update_channel); rolling back
past either without restoring it would crash the restored old code.
Add MigrationRunner::rollback(): diffs the currently-applied migration
set (the `migrations` tracking table) against the TARGET version's own
migrations/database/up/ file list — read straight out of the
already-downloaded rollback tarball via the new
ReleaseArchiveInspector (PharData/tar, no extra network call, no full
extraction) — and reverses, in descending order, every migration the
target version doesn't carry. Only migrations with a down.sql
counterpart are reversed; migrations lacking one are left applied and
logged as skipped, which is safe by design since every migration
without a down file only ever ADDs schema or business data that older
code simply never references.
Restructured src/migrations/ into database/up/ (the 25 existing
forward migrations, unchanged content) and database/down/ (18 new
reverse migrations: the two critical drops plus every mechanically-safe
ADD-column/cron-row inverse; excluded are three migrations touching
real business data — activation_codes, category_templates, lines'
custom_data — and four data UPDATEs with unrecoverable prior values).
Kept as two directories rather than a `.down.sql`-suffixed sibling of
the up file: glob('*.sql') in run() would otherwise pick up the down
file too (it still ends in .sql) and try to run it forward.
Wired into UpdateCommand's rollback case right after the tarball
download+MD5 verification (MAIN only, aborts the rollback on failure —
the pre-rollback DB backup remains the recovery path either way).
Also extends the shared TestDb harness (tests/Support/TestDb.php) to
strip MySQL-only ENGINE=/CHARSET=/COLLATE= table-option clauses under
its default SQLite backend, so MigrationRunner's real production DDL
can be exercised in tests without a live MariaDB.
Public/Views/layouts/footer.php defined one global function
(renderUnifiedLayoutFooter()), a thin delegate to
XcVm\Core\Util\LayoutRenderer::renderFooter(). 138 Public/Views/*.php
page templates called it directly, each with its own local
require_once __DIR__ . '/../layouts/footer.php' right before the call.
Converted every call site to `use XcVm\Core\Util\LayoutRenderer;` +
LayoutRenderer::renderFooter(...), dropping the require. Applied via a
scripted transform (regex-based require removal + call-site rewrite,
inserting/merging the `use` import without reordering any existing
line, so an inline comment on an existing `use` statement — e.g.
settings.php's FfmpegBinaries import — is never at risk of being
dropped), then verified: every file syntax-checked individually,
`make gates` confirmed every LayoutRenderer reference has its import,
`make cs`/`make phpstan`/the full PHPUnit suite stayed green.
With zero remaining callers of the global function anywhere in the
codebase, layouts/footer.php itself is deleted. Updated the two
remaining comment references to it (admin/footer.php,
layouts/reseller/footer.php) and LayoutRenderer's own docblock, which
now documents that neither renderHeader() nor renderFooter() has a
global-function wrapper left.
Public/Views/layouts/admin.php defined one global function
(renderUnifiedLayoutHeader()), itself a thin delegate to
XcVm\Core\Util\LayoutRenderer::renderHeader(). Its only real callers were
BaseAdminController, BasePlayerController, BasePlayerV2Controller (each
render()) and admin/setup.php — all converted to call LayoutRenderer
directly (use import + LayoutRenderer::renderHeader($scope)), dropping
the now-unnecessary require_once of layouts/admin.php in each.
With zero remaining callers of the global function anywhere in the
codebase (verified by full-tree grep), layouts/admin.php itself is
deleted. Updated the two remaining comment references to it
(admin/header.php, layouts/reseller/header.php) and a stale doc claim
in BaseAdminController's own docblock (it never actually called the
footer function — that's left to each page view template).
@see core/Http/Router.php and @see public/routes/admin.php did not
match the actual directory casing (Core/, Public/) — broken on any
case-sensitive filesystem. bin/nginx/conf/codes/template was already
correct.
renderUnifiedLayoutHeader()/renderUnifiedLayoutFooter() (layouts/admin.php,
footer.php) dated from the pre-2.0.0 "Stage 6" migration (afa55322,
2026-02-24) and still carried its now-stale "temporary compat wrapper"
docblock plus a dead $rootPath variable in both files and a dead
'rThemes' entry (replaced by the Theme enum) in both global pull-lists.
Move the actual rendering logic into XcVm\Core\Util\LayoutRenderer;
the two global functions become one-line delegates so the ~150 existing
call sites are untouched.
Core/Http/RequestGuard.php was a procedural prelude, called only from
WebApiBootstrap::init(), that duplicated logic already living as proper
BootStageInterface classes used by every other BootContext:
FloodProtectionStage and HostVerificationStage. Reuse those directly;
add WebApiLoggerStage for the one real behavioral difference (WebApi's
PHP_ERRORS also honours the runtime `debug_show_errors` setting, not
just DEV_MODE like the shared ConstantsStage). RequestGuard.php is
deleted — it had exactly one caller.
Also removes two redundant manual `require_once` of already-autoloadable
classes in DatabaseLogger/FileLogger (same namespace as LoggerInterface,
loaded automatically on the `implements` reference).
Adds a BootStageTest case for WebApiLoggerStage, restoring the previous
error/exception handlers afterward since Logger::init() registers global
ones as a side effect.
The class docblock described a pending admin/session.php +
reseller/session.php backward-compat plan that never materialized —
those files don't exist anywhere in the repo. Replace it with the
class's actual current usage: start()/requireAuth() from the
$noBootstrapPages view scripts, clearContext() from the scope
bootstraps and player controllers, adminSessionValid() shared between
AdminScopeBootstrap and Admin\TableController. Also documents the
previously-undocumented 'player' session-key context.
Enable SlevomatCodingStandard.Namespaces.UnusedUses so a dangling `use`
(e.g. left behind by a rename or refactor, like ModuleLoader's stale
RouteProviderInterface import) gets caught by `make cs` instead of only
an IDE hint. searchAnnotations=true is required, not optional: several
core interfaces (StreamMiddlewareProviderInterface, MigratableInterface)
reference their imported type only inside a @return/@param docblock, and
without that property the sniff's fixer removes the import and silently
breaks PHPStan's docblock type resolution.
Also removes 10 genuinely dead imports across 7 files surfaced once the
rule was correctly configured.
Resellers' sidebar was a hardcoded PHP array in
layouts/reseller/header.php with no extension point, unlike admin's
NavbarRegistry/NavbarItem/CoreNavbarProvider. Add a sibling
ResellerNavbarRegistry + ResellerNavbarProviderInterface (mirroring the
existing PermissionRegistry precedent for reseller-specific module
extensions), migrate the hardcoded menu into CoreResellerNavbarProvider,
and wire both into ModuleLoader::bootAll() next to the other
registries. Modules now add a reseller sidebar entry the same way they
already do for admin.
Design: docs/superpowers/specs/2026-09-24-reseller-navbar-registry-design.md
Claude