fanout_binary path verified on canary (0.7.3 -> pulls 0.8.0 from the new GitHub
release -> installs -> restart). Panel P4 (connectionRates + writeDivergence) still
needs to land in the panel release for the telemetry to be consumed fleet-wide.
Daemon GET /rates + fanout_sync writeDivergence proven end-to-end on the canary
(redis mode): viewer on stream 568 -> /rates 857 KB/s -> lines_divergence written
(0.0, healthy). Takes effect fleet-wide once the 0.8.0 daemon is released as
latest.
- 0002 §2.5 rewritten: light-P4 reconciliation shipped; per-viewer divergence is the
remaining P4 gap (daemon-served viewers never enter the chase-read loop)
- 0002 S6 marked done (scale validation: O(N) worker-per-viewer ceiling removed)
- 0002 Risks/§2.4/S1-sketch: auth_request references reframed as superseded by X-Accel;
removed live_fanout flag operational claims
- 0003 Phase C + Risks + snapshot: new P4-in-progress bullet for divergence/bitrate
telemetry, with slice-1 plan (daemon /rates -> fanout_sync writes lines_live.divergence)
Status → Accepted/in-progress. Record what the code actually shows: phases 0/A/B
done, Phase C in progress (C1a cold-start done, C1b moot, C2 adaptive verified,
C3 restreamer-prebuffer fixed), E1 done, Phase F cancelled (on-disk HLS stays).
Add a Phase C work breakdown (C1–C5 + C-ops) including the fanout_sync
self-respawn fix and the daemon-restart re-feed gap surfaced in production.
The daemon moves to its own repo (XC_VM_Fanout, GIT_REPO_FANOUT) that ships
per-arch static binaries as GitHub Release assets. Since we're fully
migrating onto the daemon, the panel needs to fetch and update it.
- FanoutBinaryCommand (`console.php fanout_binary`): reads the installed
version straight from the binary (`xc_fanout -version`), compares to the
latest release, and when they differ downloads the arch-matched asset,
verifies its SHA-256 against the release SHA256SUMS, installs it
atomically and restarts the daemon (the service keepalive respawns it).
`force` reinstalls the current version. Modelled on BinariesCommand.
- GIT_REPO_FANOUT constant.
- docs/xc_fanout.md: separate-repo + install/update section.
The daemon repo itself (source + release.sh + a v* release GitHub Action,
binaries gitignored) is prepared under XC_VM_Fanout/ for the user to push.
Plain-language companion to the ADRs: the worker-per-viewer problem it
solves, what it does (and doesn't), and the client/control surfaces PHP
uses (FanoutClient, X-Accel routes, fanout_sync).
Close Phase B's unencrypted-only limitation: encrypted HLS is now served from
the daemon's in-RAM segmenter too, so encrypt_hls streams no longer need the
legacy tmpfs .enc path — one of the two prerequisites for dropping the
streaming tmpfs (Phase F).
- StreamProcess: generate the stream's HLS key/iv BEFORE registering the daemon
ingest (moved writeStreamKeyIv up) and, when encrypt_hls is on, hand the hex
key/iv to the daemon at FanoutClient::registerIngest so it encrypts the HLS
segments it serves.
- FanoutClient::registerIngest: optional key/iv passed in the ingest PUT body.
- HLSGenerator::tokenizeDaemonPlaylist: prepend #EXT-X-KEY (URI /key/<token>,
IV from <id>_.iv) for encrypted streams, matching generateHLS.
- live.php m3u8: drop the !encrypt_hls gate — daemon HLS now covers both.
- segment.php unchanged: the daemon-segment X-Accel already returns the
daemon's (now-encrypted) bytes.
Box-validated: encrypt_hls=1 → 569.m3u8 has #EXT-X-KEY + tokenized daemon
segments; a segment decrypts (stream key + playlist IV) to a valid TS. Daemon
crypto is byte-identical to openssl. PHPStan clean, 443 tests. Daemon 0.6.0.
Refs ADR 0003 (Phase B encrypted). Daemon: XC_VM_Binaries 0.6.0.
Phase C (ADR 0003): fix the connection-limit leak for daemon-served live TS.
Under X-Accel the auth PHP-FPM worker returns immediately, so UsersCronJob's
reaper — which closes a TS row when its pid stops being a running php-fpm
worker — can neither track nor time out a daemon viewer (its recorded pid is
a reused, still-alive worker). Those rows leaked, filling the line's slots
until a false "max connections reached".
Fix (light P4):
- live.php marks daemon-served TS rows pid=0 (decided once per request via
$rTSDaemon: proxy $rFanout, or non-proxy isStreamFed), and passes the
connection uuid to the daemon in the X-Accel URL (?c=<uuid>).
- daemon (0.5.0) tracks connected uuids per stream and serves them at
control GET /connections.
- new fanout_sync daemon reconciles pid=0 open rows against that set every
~10s, closing any whose uuid is no longer connected (20s connect-grace);
skips reconcile when the daemon is unreachable (added to service boot()).
- UsersCronJob leaves pid=0 TS rows to fanout_sync (reaper would leak or,
on worker recycle, close them early).
HLS was already self-heartbeating (playlist re-fetch → updateLive), so this
targets long-lived TS. Box-validated: viewer → pid=0 row + uuid in
/connections; disconnect → daemon drops the uuid and fanout_sync closes the
row (slot freed). PHPStan clean, 443 tests. Daemon: XC_VM_Binaries 0.5.0.
Refs ADR 0003 (Phase C).
Phase C parity (ADR 0003): restore the legacy startProxy off-air behaviour
for daemon-served proxy streams. After registering a proxy source, live.php
calls FanoutClient::probe($id, on_demand_wait_time*1000) → daemon
POST /probe/<id>, which prewarms the puller and waits for the source to
produce data. If it stays dead, live.php shows the not-on-air page instead
of X-Accel'ing the viewer onto an empty/hanging daemon stream. A warm stream
(already producing) returns immediately, so only a cold/dead source pays the
wait — same as the legacy path.
Non-proxy off-air was already covered (the process block waits for the
playlist, and A3 gates on isStreamFed/has_data).
Box-validated: dead-source proxy → not-on-air video after on_demand_wait_time
(20s here); live-source proxy → fast play via the daemon (4.4 MB valid TS).
PHPStan clean, 11 FanoutClient tests.
Refs ADR 0003 (Phase C). Daemon: XC_VM_Binaries 0.4.0.
Phase B (ADR 0003): HLS is now served from the xc_fanout daemon's in-RAM
HLS end-to-end for unencrypted streams, PHP doing only auth — the stepping
stone toward dropping the streaming tmpfs.
- live.php m3u8 branch: when the daemon is reachable, the stream is fed
(isStreamFed) and encrypt_hls is off, fetch the daemon playlist
(FanoutClient::hlsPlaylist → client socket) and tokenize its <seq>.ts
entries (HLSGenerator::tokenizeDaemonPlaylist) into the same /hls/<token>
auth'd URLs as the legacy path, with the segment marked <id>_d<seq>.ts.
Else the legacy on-disk generateHLS runs (encrypted / daemon down).
- segment.php: a daemon-segment token runs the same uuid + IP checks, then
X-Accel-Redirect: /xc_fanout_hls/<id>_<seq> → daemon /hls/<id>/<seq>.ts.
- nginx: internal location ^~ /xc_fanout_hls/ → proxy_pass daemon (two path
segments, id_seq, to avoid the server-level rewrite hijack — same reason
as /xc_fanout/ and /xc_hls/).
The daemon's HLS on-demand lifecycle was already closed (0.2.0 touch+reaper).
Box-validated: 569.m3u8 → daemon MEDIA-SEQUENCE + tokenized segments; a
segment fetch → 200 valid TS (953 KB) from the daemon's RAM. PHPStan clean.
Limitation: daemon HLS is plain mpegts → unencrypted streams only; encrypted
streams stay on legacy tmpfs HLS. Daemon-side encryption + removing the tee's
on-disk HLS slave are what Phase F needs to drop the tmpfs.
Refs ADR 0003 (Phase B).
The last live stream type not on the daemon. LLOD v3 is a PHP MPEG-TS
segmenter (not ffmpeg), so it can't tee — instead it now mirrors the raw
bytes it reads into the daemon's push-fed ingest socket
(FanoutClient::registerIngest + a non-blocking stream_socket_client write).
The same stream then fans out via /live/<id> and in-RAM /hls, and live.php's
A3 isStreamFed() routes viewers to the daemon.
- Non-blocking, best-effort write: a daemon stall never slows LLOD's own
segmenting (its primary job); on a write error (daemon gone) it stops
mirroring and live.php falls back to legacy. Null/failed connect ⇒
legacy-only, no behaviour change.
- LLOD keeps writing its on-disk segments (legacy fallback intact).
- Cleanup: closes the conn and unregisters on clean exit; stopStream() is
the backstop when LLOD is killed mid-loop.
Box-validated (stream 599, llod=2): daemon fed (has_data) with a routed
viewer (refs:1), valid /live TS + /hls m3u8, LLOD segments still produced,
playback OK (3.98 MB). PHPStan clean.
Now every live stream type runs through the daemon when reachable: proxy
(S3), non-proxy llod=0/1 (A2 tee + A3), and LLOD v3 (push + A3).
Refs ADR 0003 (Phase A).
Completes Phase A (ADR 0003): non-proxy live streams now both feed the
daemon (A2 tee) and are SERVED by it. At live.php's non-proxy delivery
point, when the daemon is reachable and FanoutClient::isStreamFed($id)
(control GET /streams/<id> → has_data), hand the byte path to nginx→daemon
via X-Accel-Redirect: /xc_fanout/<id> — like proxy mode — instead of
pinning the worker in the per-viewer .ts chase-read. Daemon down / not fed
⇒ the legacy chase-read runs (the reachability rollback).
The tee feeds the daemon from the same ffmpeg that writes the on-disk HLS,
so by delivery time (playlist ready) the daemon has data. Ingest-fed
streams report running:false (no puller), so the gate is has_data.
Box-validated: 6 concurrent non-proxy viewers on stream 569 all served by
the daemon (daemon_conns=6) with php-fpm workers flat (not 1-per-viewer),
valid TS. PHPStan clean, 10 FanoutClient tests.
ADR 0003: A3 done; added a ⚠️ TODO that LLOD v3 (LlodCommand, streams.llod=2)
has no daemon path (separate PHP segmenter) and must be integrated before
Phase D — llod=0/1 already covered via buildLive+tee.
Refs ADR 0003 (Phase A).
Record the daemon ingest listener (0.3.0) as done and box-validated, with
the validated production tee command for A2, and the A2/A3 coordination
(register ingest before ffmpeg; reachability switch).
For a permanent cutover a runtime settings flag is throwaway scaffolding
(and a settings column that ends up always-1 is dead weight). Remove the
`live_fanout` dependency from live.php entirely; the switch is now the
daemon itself.
live.php decides proxy → daemon vs legacy on REGISTRATION SUCCESS, moved
ahead of the legacy process block: if the control socket is present and
FanoutClient::register succeeds, skip legacy startProxy and X-Accel to the
daemon; otherwise run the FULL legacy path (producer included). This also
fixes a latent bug in the old flag path — it skipped startProxy but could
fall through to a producerless legacy relay when registration failed.
Rollback is now "stop the daemon": its socket disappears / register fails,
every stream falls back automatically, per node, no DB write, no cache
rebuild, self-healing via the keepalive. Which stream types go to the
daemon stays a code decision (proxy now; +non-proxy after Phase A).
ADR 0003 Phase 0 dropped (no schema column / admin toggle); ADR 0002
rollback-flag note marked superseded. Test-box live_fanout column to be
dropped.
Refs ADR 0003.
Wrap the xc_fanout launch in service boot() in a keepalive loop (2s
backoff) so a daemon crash is auto-restarted — the Restart=always
equivalent without adding a second systemd unit. Both the loop and the
daemon run as xc_vm, so stop()'s `pkill -u xc_vm` ends both together and
nothing respawns during shutdown; the daemon still logs to tmp/xc_fanout.log.
ADR 0003: mark the Phase C foundations landed (HLS on-demand lifecycle,
off-air status endpoint, auto-restart — daemon 0.2.0 in XC_VM_Binaries).
Refs ADR 0003.
Plan the full transition of all LIVE streaming (proxy + non-proxy, TS +
HLS) onto the xc_fanout daemon, then retire the legacy byte path. Scopes
VOD/timeshift OUT (random-access, not fan-out); keeps per-stream ffmpeg
(transcode/logo) and PHP auth, moving only delivery to the daemon.
Phases: 0 schema/flag column + admin toggle → A non-proxy via tee'd
ingest (S4) → B HLS end-to-end through the daemon (+ HLS on-demand
lifecycle) → C parity/robustness (off-air, cold start, adaptive,
restreamer, limits, disconnect accounting) → D make daemon default → E
delete legacy (ProxyCommand, live.php byte loops, tmpfs HLS serving,
CONS_TMP_PATH sockets) → F drop streaming tmpfs → G LB rollout.
Records the box A/B result and the two daemon gaps surfaced (HLS must join
the on-demand lifecycle; daemon needs a health/last-data signal for
off-air detection).
The fanout hand-off used `X-Accel-Redirect: /xc_fanout/live/<id>`, but nginx
server-level `rewrite` directives run during the rewrite phase on internal
redirects too — before location selection — and the 3-segment catch-all
`^/(user)/(pass)/(stream)` rewrote `/xc_fanout/live/569` into
`/stream/auth?username=xc_fanout&password=live&stream=569`, yielding
INVALID_CREDENTIALS instead of reaching the internal proxy location.
Fix: make the target two path segments (`/xc_fanout/<id>`, location
`^~ /xc_fanout/`), which matches no server rewrite — the same reason P1's
`/xc_hls/<file>` target is two segments. Added a NOTE in nginx.conf so it
isn't "simplified" back to a 3-segment form.
Validated end-to-end on the test node with a real player: flag-off vs
flag-on A/B at 8 concurrent viewers on a real proxy stream — legacy pinned
~1 php-fpm worker + ~24 MB/viewer (9 workers / 242 MB); fanout served all 8
through the daemon (daemon_conns=8) with php-fpm flat at baseline (1 worker
/ 45 MB). PHP fully out of the byte path.
Refs ADR 0002.
Move the daemon's control/client sockets from /run/xc_fanout/ to
bin/xc_fanout/sockets/{control,http}.sock, next to the daemon binary and
mirroring the existing php-fpm sockets layout (bin/php/sockets/) that nginx
already reaches over unix:. Keeps runtime sockets in the app tree the panel
owns rather than a separate FHS runtime dir; a unix socket stores no stream
bytes (pure IPC), so its location is orthogonal to the tmpfs-free byte-path
goal.
- Paths: FANOUT_RUN_PATH now BIN_PATH.'xc_fanout/sockets/' (was /run/...).
- nginx: proxy_pass unix:/home/xc_vm/bin/xc_fanout/sockets/http.sock.
- ADR 0002: §3 placement + §2.4/S3/S4/diagram paths updated; corrected the
imprecise "tmpfs-free runtime dir" wording.
Daemon must launch with -sock .../sockets/http.sock -ctl .../sockets/control.sock
(wired into supervision in S5).
Refs ADR 0002.
Design for the P2 live fan-out daemon: one Go process pulls each live
source once and fans it out to many viewers, keeping PHP out of the byte
path. Records the locked decisions (Go; daemon owns the puller; both TS
sub-modes), the ring-buffer/clean-join model, on-demand lifecycle, and the
build slices S1–S6.
S3 hand-off refined to X-Accel-Redirect (flag stays in PHP, no nginx
reload, mirrors the shipped P1 pattern) instead of auth_request; §2.4/§3
and the S3 slice updated to match the implementation.
Records the decision to take PHP out of the streaming byte path and remove
the tmpfs mounts: native xc_fanout daemon for live fan-out + in-RAM HLS,
nginx X-Accel/proxy_pass with auth_request, connection telemetry in Redis.
Root cause from P0 load test: the ~400-concurrent ceiling is a linear RAM
law (~25 MB per viewer) because each proxy viewer pins a PHP-FPM worker in a
socket_read loop — not tmpfs pressure and not pm.max_children.
Drop the two ministra restoration working-notes now that the work is
done: ministra-unused-modules.md (dead-module audit complete) and
ministra-5.6.10-lost-customizations.md (all items resolved). Also drop
the now-dangling doc reference from the get_modules debug comment.
The ministra-browser-emulation.md integration guide is kept.
Review of the remaining 5.6.10 lost-customization items:
- player.js: multi-audio (titles/infoCurtitle) kept and improved by
5.6.10; xc_vm PVR branch intact; parental flow lives in tv.js, not
here. No changes needed.
- account.js: keep the full 5.6.10 account screen (decision) instead of
restoring the minimal Phone+message fork screen.
- xpcom.common.js: allowed_stb_types/aurahd richer in 5.6.10;
outdated_firmware (player<1382) only gates non-whitelisted types and
is more permissive, not a regression; check_image_version is a no-op
(backend sends no autoupdate); load_channels parental left to 5.6.10 +
tv.js guards to avoid a double password prompt.
Re-apply two xc_vm customizations that the 5.6.10 client drop wiped out:
- Channel logos (tv.js handling_block + player.js preview): on xc_vm the
backend sends a ready-to-use logo URL, but 5.6.10 rebuilt a classic
stalker misc/logos/<size>/ path, breaking the image. Use the URL
directly; keep the 5.6.10 timeshift_mode class on the player logo.
- TV color buttons (tv.js): in the xc_vm theme the 4th button is channel
search (search_menu_switcher) instead of the fav-manage "move" mode.
Hybrid with the new 5.6.10 quality-filter button — quality wins when
the profile enables tv_quality_filter, otherwise search.
Also drop the obsolete snumber/sortIDs note: the backend now numbers
channels sequentially, so 5.6.10 sortBy("number") is correct.
- Add ministra-5.6.10-lost-customizations.md: what our fork's clean
5.6.10 drop wiped out and how to restore it (get_types_list, tv.js
parental control, channel logos, account screen, DVR .ts format, ...).
- Update ministra-browser-emulation.md for the ?debug_key gate.
Drop portal JS modules that the loader never pulls in (not in
base_modules/all_modules, no dynamic loader, no live references) and
their theme assets:
- infoportal branch: infoportal, anecdote, cityinfo, horoscope,
weather.current, weather.day, course.cbr, course.nbu, game.mastermind
(root module dead + crashes main menu when force-enabled)
- magiccast (leaked proxy creds to an external Infomir service)
- youtube (web wrapper dir absent from repo)
- demo, service_management (stub features, no server handlers)
- karaoke (no backend; "radio only")
- pvr.js (legacy Pvr, never instantiated; live local PVR is
pvr_local/records)
- JsHttpRequest-debug.js (core loads JsHttpRequest.js)
Removes 15 per-theme CSS variants + menu icons per module. The dead-code
audit is tracked in docs/ru/guides/ministra-unused-modules.md.
Add an entry (RU + EN) explaining why a MAG/STB box gets its IP blocked
after a factory reset / firmware change: the portal's MAGSCAN anti-clone
check compares the posted serial against the stored mag_devices.sn (and
device_id/device_id2/hw_version when lock_device is on) and bans the IP on
mismatch (blocked_ips -> iptables). Documents the fix — reset the device's
stored sn/device_id in the panel — and how to unblock: the web panel
(Tools -> IP Management, /<admin-code>/ips), console.php tools flush, or
manual iptables + flood-marker removal.
Ministra stops being a module — the whole Stalker portal (portal.php,
MinistraBootstrap, PortalHandler/PortalHelpers and the STB front-end) now
lives in src/Ministra/ under the XcVm\Ministra namespace, served at
/home/xc_vm/Ministra via the nginx alias.
- src/ministra/* and Modules/ministra_85a7d/{PortalHandler,PortalHelpers}
→ src/Ministra/; MinistraModule.php + module.json removed. Ministra was
the only committed module, so src/Modules/ keeps a .gitkeep.
- portal.php resolves PortalHandler as a sibling and derives MAIN_HOME from
its new location (glob crutch gone).
- nginx alias + AuthRepository $rAlias switched to /home/xc_vm/Ministra
(PascalCase); ministra entry dropped from bundled_modules.php.
- Makefile: Modules/ removed from LB_DIRS — all modules are MAIN-only, so
the ~50 MB of portal assets no longer ship to LB nodes.
- ArchitectureTest: zero committed modules is now a valid state.
- PHPStan: analyse src/Ministra, exclude the procedural portal.php entry,
repath the ministra baseline entries.
- Docs (architecture, ministra-browser-emulation, extraction plan) updated
to the new layout; the "extract to a separate repo" plan is cancelled.
Verified: php -l, make gates, make phpstan (No errors), full unit suite
(432 tests). On-server smoke: handshake + get_profile work end-to-end with
a registered MAC after deploy.
BoundaryInterface was a marker interface with no runtime consumer —
nothing read getEntryPoint()/isIsolated() and, being static-less
metadata, it enforced nothing. Its only implementor was MinistraModule.
Removes the interface, drops `implements BoundaryInterface` plus the two
orphaned methods from MinistraModule (getName/getVersion stay — they come
from BaseModule/ModuleInterface), and deletes the now-empty Core/Boundary/.
Test contract (InterfaceContractTest) loses the three BoundaryInterface
assertions; docs (en/ru architecture + modules, .github instructions) now
describe isolated subsystems like Ministra as a convention — own entry
point + bootstrap — rather than a marker interface.
Verified: php -l, make gates, make phpstan (No errors);
InterfaceContractTest + ArchitectureTest green (33 tests, 96 assertions).
The script was never wired up (no `make module-hashes` target, no CI/Makefile
caller). Module hash_id generation now lives in the standalone Module_Template
kit; for existing modules, generate inline with
`php -r 'echo bin2hex(random_bytes(16));'`.
Updated the module-dev docs (en + ru) that referenced the removed script /
non-existent `make module-hashes` target to use the one-liner.
Standalone Python (stdlib-only) tool that verifies a stream delivers
segments correctly and its delivery queue does not break:
- HLS: EXT-X-MEDIA-SEQUENCE contiguity, no dropped/rewound segments, no
EXT-X-DISCONTINUITY, every newly appearing segment downloadable.
- MPEG-TS: per-PID continuity_counter, sync-byte loss, TEI, delivery stalls,
with a --tolerance for rare source glitches relayed by -c copy.
- --live: colored TUI dashboard modelling a virtual player — received
timeline from PCR (TS) / EXTINF (HLS), playhead, and buffered cache
seconds graphed over time.
Documented in docs/{en,ru}/development/streaming-subsystem.md.
Per-distro distribution plan updated with what the three deployed
binaries actually are (XUI 2018 / mardock2009 no-GPU / our glibc-2.35
build), the code-driven target codec set (+nvenc/cuvid/librtmp, -AV1),
the 8.1 label decision, and the -nofix_dts custom-flag replacement plan
validated via tools/dts-audio-test.
The heartbeat is written by the watchdog daemon (which now waits out DB
outages), not by cron:servers; the babysitter section documents the
crontab/cron-service/hung-lock sub-checks, and the cheat-sheet covers
the 'all nodes drop at the same moment' scenario. cli-tools blurbs
updated to match (en/ru).
update-system.md and faq.md (ru+en) linked to <lang>/development/cli-tools.md,
but the file lives at <lang>/guides/cli-tools.md (as the sidebar already points) —
the old path rendered an empty docsify page. Also repoint two non-existent anchors
(#миграции-базы-данных, #database-migrations) to the real section
(#обновление-бд-после-обновления-версии / #database-updates-after-version-upgrade).
The plan is done: proxy.tar.gz is fetched from XC_VM_Proxy releases at install and
kept fresh by cron:proxy, and the LFS object is gone. Rationale lives in the commit
history and XC_VM_Proxy/RELEASE.md.
Divarion_D