CommandBus queues typed, panel-signed (`cmd`) commands per node in
cluster_commands: FIFO by seq, dedupe_key for desired state, expiry,
pruned by cron:cluster. New ops: `commands` (long-poll after the node's
high-water, held up to 20 s; frozen while quarantined) and `ack` (own
commands only; raises cluster_nodes.cmd_seq).
For nodes with the COMMANDS flow on (toggled on Cluster Nodes),
ClusterRoute sends NodeRpc::request() as a signed node.rpc and waits for
its ack, and NodeRpc::broadcast() / SignalDispatcher::kill() queue
commands; other nodes and every LB keep the legacy transport.
On the node, `console.php cluster:exec` checks a command again against
the panel key the agent pinned and runs node.rpc through the legacy /api
handlers (InternalApiController::runCommand, NodeRpc::ACTIONS only) or
conn.kill_worker. Root actions stay on the signals table for now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DcNdxHieZuUh3EnMUKt7zf
LivenessService::tick() runs every second in MAIN's signals daemon
(cron:cluster is the minute fallback) and judges the active nodes whose
TELEMETRY flow is on by their silence: ok, suspect after 10 s, offline
after cluster_offline_after_sec (30 s). The result, in
tmp/cluster/health.json, is read through Core\Cluster\ClusterHealth:
ServerRepository::getAll() takes server_online from it for those nodes
(instead of the 90 s last_check_ago rule) and getCapacity() doubles a
suspect node's weight. Each transition rewrites the servers cache and is
audited.
When over half of the judged nodes (at least two) fall silent together,
the fleet silence guard holds every node at its last state instead of
marking any offline, audits it and alerts on the Cluster Nodes page.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DcNdxHieZuUh3EnMUKt7zf
For a node with the TELEMETRY flow on, MAIN writes the server's
watchdog_data, last_check_ago, requests_per_second, php_pids (and,
without the Redis handler, connections/users) from the agent's heartbeat
every 5 s, and its servers_stats row every minute.
HeartbeatService::toWatchdogData keeps the legacy getStats() shape,
pinned by ClusterTelemetryTest against getStats()'s source.
On the LB, Core\Cluster\NodeFlows reads the mode and flows the agent
writes to config/cluster/flows.json. With TELEMETRY on, the watchdog
stops writing its row and only refreshes local.json (nginx rps, fanout
status) for the agent; cron:servers skips its stats row and network.py
is stopped. The Cluster Nodes page toggles the flow per node.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DcNdxHieZuUh3EnMUKt7zf
Servers → Cluster Nodes (permission `servers`) is the admin side of the
cluster CLI, via Domain\Cluster\ClusterAdmin: enrolled nodes with state,
liveness, epoch, token expiry, last heartbeat and agent version; code
enrolments waiting for a SAS (approve / reject); issuing an enrolment
code (shown once); and revocation. The deciding admin is recorded.
Strings in all eight languages.
cron:cluster (enabled by migration 037) deletes expired epochs (erasing
their z), the replay cache and used challenges, unused expired codes and
decided requests after a day. It returns at once on load balancers and
while the API is disabled.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DcNdxHieZuUh3EnMUKt7zf
Core\Cluster\ClusterSettings owns the defaults and bounds of the 19
cluster settings. normalize() clamps numbers, falls back to the default
for an unknown enum, and refuses rather than clamps where clamping would
change what the admin meant:
- a cluster_api_port outside 1024–65535, or one MAIN already uses
(broadcast, extra and RTMP ports) or that is reserved (31210, 31290,
3306, 6379);
- a cluster_main_host that is not a DNS name;
- scan roots that are not absolute paths free of '..';
- https_required, unless MAIN's own HTTPS verifies (httpsSelfProbe: a
non-IP domain and a certificate with more than 7 days left) and no
active node blocks it (every active node blocks until Phase 3
telemetry reports HTTPS);
- enabling the LB API without xcvm_core's cluster API
(ClusterCryptoFactory);
- API mode for new nodes before the cutover phase.
SettingsService::edit() runs these checks. A refusal fails the whole
save, and its translated message is shown on the settings page. It lives
in Core/Cluster, which ships to LBs, so nodes apply the same bounds.
The settings page gets a Cluster tab after Streaming. It shows every
field, and a status line: the extension and its API range, whether HTTPS
is available, and the derived token grace. The strings come in all 8
languages.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Es8UhyPS8MNjuYTagqfGWi
settings.fanout_enabled (migration 027, default on) is a new "Fanout
Delivery" toggle beside the other fanout settings. Streaming\Fanout\FanoutMode
reads it and applies it to a node:
- Off: FanoutMode::applyToNode() writes bin/xc_fanout/disabled, then stops
run.sh (so it cannot respawn the daemon) and then xc_fanout. It runs on
MAIN when the settings are saved, and on every node from RootSignalsCronJob
within a minute. The RootSignals keepalive and the hourly fanout_binary
self-heal skip the daemon while the flag is set; `service boot` and run.sh,
which cannot read settings, check the flag file.
- Every daemon-facing gate honours the switch. LicenseGate::fanoutUsable()
and each FanoutClient control-socket call behave as if the daemon were
down. IngestFeeder::forStream() returns a no-op feeder, and DelayCommand
stops queueing segments for it. fanout_sync stops writing the daemon
config and closes the pid-0 connection rows the stopped daemon left.
- The dashboard no longer reports "xc_fanout down" when it was switched off.
The pre-fanout delivery paths for fanout off follow in the next commit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Es8UhyPS8MNjuYTagqfGWi
server:install received the root username and password as positional
arguments, visible in ps and /proc/<pid>/cmdline for the whole install, and
LbInstallFlow/ProxyInstallFlow saved them in a plaintext
bin/install/<id>.json for reinstalls. Any SSH host key was accepted.
- The panel writes a 0600 bin/install/<id>.cred and runs
`server:install <type> <id> <port> - - … --cred-file=<path>`; the command
reads and deletes the file before connecting (InstallCredentials). Only
<id>.cred files in bin/install/ are accepted.
- <id>.json keeps the non-secret parameters only; reinstall_server now needs
root_password in the request. cron:cleanup (MAIN) scrubs root_password from
files written by earlier releases.
- The node's SSH host key (SHA-1) must match the optional Expected SSH Host
Key (hex or `ssh-keygen -l -E sha1` output), else the key stored by the
first install (new servers.ssh_hostkey_sha1, migration 026), else it is
trusted on first use and stored.
- New install-form field and strings in all eight languages; cli-tools docs.
Phase 0 stop-gap of the MAIN <-> LB API plan.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Es8UhyPS8MNjuYTagqfGWi
The fanout debug field was free text (empty, "all", or a comma list of
category names typed from the tooltip), so a typo silently meant off.
Make it a dropdown of every value the daemon takes: Off, All categories,
or one category. FanoutConfig::DEBUG_CATEGORIES is now the single list
both the dropdown and debugCats() use; a combination saved before stays
selectable as it is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4P9p5BXijMoGXd31tN6Zp
The panel mapped fanout_debug -> the daemon's debug_cats, but nothing could
set it: there was no `fanout_debug` column, so QueryHelper::verifyPostTable
(which filters a settings POST against the table's real columns) dropped it,
and there was no field in the settings form to post it with. The toggle was
inert -- FanoutConfig always wrote debug_cats "" (off).
Add the missing column (migration 025 + the install schema), a text field in
the admin settings form beside the other fanout knobs, and the fanout_debug
label in all seven locales. Empty = off (default), "all", or a comma list of
the daemon's categories; FanoutConfig normalises it as before.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The integration had dropped all 208 ac_* (Active Codes) keys from every
language file while adding new feature keys, so the Active Codes UI rendered
raw key names. Restore the language files to their committed state (ac_*
intact) and append only the 57 new keys that are actually referenced in the
panel (59 unused keys from the re-send are skipped). Additive-only; the files
parse under the app's INI_SCANNER_RAW.
Co-Authored-By: AbdoAhmedElbanaa <115952879+AbdoAhmedElbanaa@users.noreply.github.com>
A new "Resources" column between Stream Info and Actions: which process is
producing the channel (the fanout daemon's native remuxer, ffmpeg, or PHP for
the LLOD segmenter / loopback relay), the CPU it is burning and the memory it
holds. With the native remuxer now an option per stream, "what does this
channel actually cost" and "which backend is it on" are the two questions the
list could not answer.
ProcessManager reads both from /proc/PID/stat (fields 14/15 and 24, with the
page size derived rather than assumed — 64K pages are normal on arm64). CPU
there is cumulative, so a percentage needs two readings: cron:streams samples
each producer once a pass and folds cpu/mem/producer into the stream's
progress_info, carrying the previous reading in the same JSON to subtract from.
cpuPercent() returns null rather than a wild figure when the pair says nothing
— no previous sample, same instant, or a counter that went backwards because
the producer restarted. Only the node running a stream can read its own /proc,
so the sampling happens there and reaches the panel in the row the cron already
writes; MAIN just renders it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UZP7fc2Hz36wbPmuLd9F9o
Whether live streams on each server are handed to the xc_fanout daemon's
encoder supervisor instead of getting a per-stream PHP watchdog
(console.php monitor). On by default: the supervisor does everything the
PHP monitor did, from one process per node instead of one per channel,
and the PHP monitor stays the fallback for a daemon that cannot be
reached.
Touches the places every fanout_* setting lives: migration 018 for
upgrades, database.sql for fresh installs (277/277 columns and values
still aligned), the FanoutConfig mapping that writes `supervise` into the
daemon's config.json (applied live by the daemon, no restart), the
settings UI, the checkbox whitelist in SettingsService, all seven language
files, and the unit test. The source-backend tooltip now says what the
backend means for the panel's own streams as well.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012QL93N6dzGkmKoQgA4oh16
xc_fanout 0.12.0 can convert a non-mp2t source to MPEG-TS in-process instead of
spawning an ffmpeg child per stream. For the common IPTV case — HLS whose
segments are already MPEG-TS — that child was doing little more than
concatenating bytes the daemon can concatenate itself, at ~27 MB RSS apiece,
plus a process spawn and a probe window on every on-demand join and reconnect.
The daemon reads which path to take from `source_backend` in its config.json;
this adds the panel setting that writes it.
auto (default) — convert natively where the daemon's reader can, ffmpeg for
everything it declines
ffmpeg — always spawn ffmpeg; the pre-0.12 behaviour and the
kill-switch if a native pull ever misbehaves
native — native only, no fallback. Diagnostic: it answers "what is
actually eligible on this node", and a declined source there
is a dead channel rather than a slightly more expensive one.
auto is the default because the fallback makes it strictly safer than
ffmpeg-always: anything the native reader will not take (fMP4/CMAF HLS, RTMP,
SRT, RTSP, AES-128 encrypted sources, anything it cannot positively identify)
runs exactly the pipeline it ran before.
A `<select>`, not a numeric input, so it is deliberately kept out of the
numeric-coercion list in settings.php that would otherwise mangle the string.
An unrecognised value maps to auto rather than being written through — the
daemon clamps unknown values itself, but a config file we write should not carry
a backend that does not exist, and a typo must never take channels off air.
Touches the six places every fanout_* setting lives: migration 017 for upgrades,
database.sql for fresh installs (column, INSERT list and VALUES — 276/276 still
aligned), the FanoutConfig mapping, the settings UI, all seven language files,
and the unit test. SettingsService needs no change: its whitelist is only for
checkboxes, and verifyPostTable already maps a posted field to its column.
Verified against the real daemon end to end: the panel writing "ffmpeg" boots
xc_fanout with backend=ffmpeg, and changing it to "auto" is picked up live on
the next config poll. Read-modify-write still preserves daemon keys the panel
does not manage.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PGCQcWGMg1Wn8RdCxYoioN
Rebuild the panel's largest management table on the Bootstrap 5 stack. handleLines
now emits structured rows (status code, online/trial/restreamer flags, live
connection count, current stream, owner, formatted expiration + last-connection,
notes, ban/enable state, contact) instead of HTML; the view renders every cell
and the per-row action dropdown client-side via datatables-bs5.
Features: reseller (select2-ajax) + status + search filters, bulk-select
toolbar (enable/disable/ban/unban/kill/delete via action=multi&type=line),
per-row edit + fingerprint iframe modals, native Download-Playlist modal
(output_devices) and WhatsApp-renewal modal, kill/ban/unban/enable/disable/
delete via action=line, and CSV export. $rOrder gains the control + checkbox
offset. Add lines to the allowlist + E2E suite (ajax wait widened to 40s for
this heavy handler) and the `all` / `download_playlist` i18n keys.
Refs: #148#178
Move the .ini language files from src/resources/langs/ to
src/Core/Localization/lang/, next to the Translator subsystem that owns
them — which already defaulted its $langsDir to __DIR__ . '/lang/'. This
dissolves the now-vestigial src/resources/ bucket (its data/ tree went
with admin_constants, libs/ was empty).
- bootstrap.php calls Translator::init() with no argument, relying on the
class's own __DIR__-relative default instead of MAIN_HOME . 'resources/langs/'.
- Makefile LB removal list points at Core/Localization/lang (and drops the
gone resources/langs, resources/libs); LB still ships no UI translations.
- Drop the stale src/resources entries from phpstan scanDirectories and the
phpunit coverage excludes.
- Update the English docs (translations guide, build-system table); the ru
tree is regenerated before release.
Claude