Replace the server-side HTML builder in the `search` admin-ajax action with a
structured JSON contract: each result item now carries `{id, url, text, entity,
data}` and the client renders the card. Permission checks, status resolution and
category/server lookups stay server-side; only markup moves to the browser. This
unblocks the upcoming redesign, where the embedded HTML builder was in the way.
- SearchAjaxController::search() now emits structured items (per-entity `data`
payloads for stream/movie/channel/radio/episode, series, user, line, mag,
enigma) plus a `no_results` item for empty responses. Data-gathering is
unchanged (byte-identical to the previous fuzzy full-text gather).
- New client renderer `assets/admin/js/search.js` (`renderSearchItem`), loaded
before common.js; the Select2 quick-search `templateResult` now calls it
(with a loading-state guard) instead of consuming a server `html` field.
- ADR docs/adr/search-json-contract.md documents the envelope, item shape,
self-describing actions and per-entity payloads.
Note: this changes only the search render path, not search matching. Live
streams missing from results are a stale streams FULLTEXT index on the DB
(rebuild with `ALTER TABLE streams ENGINE=InnoDB`), unrelated to this change.
Replace the hand-maintained Docsify site (parallel docs/en + docs/ru trees
that had already drifted) with a MkDocs Material build where English is the
single source of truth and Russian is generated at build time.
Engine & structure
- mkdocs.yml: Material theme, site_url for the /XC_VM/ Pages subpath, and a
two-tab information architecture — User Guide (administration, API/Swagger,
UI translations, diagnostics, info/FAQ) vs Developer Guide (architecture,
workflow, security, integrations, build). Files are NOT moved — the split is
nav-only, so URLs and cross-links stay stable.
- mkdocs-static-i18n (folder mode): English at root, Russian under /ru/, with a
language switcher. `mkdocs build --strict` validates every link/anchor.
Translation pipeline (tools/docs/translate.py)
- Engine-agnostic via DOCS_TRANSLATE_PROVIDER: translators (free, no API key —
default), anthropic, deepl, or noop. Per-file sha256 cache so only changed
English files are re-translated. Markdown-safe: code, URLs, HTML tags and
glossary terms (XC_VM, FFmpeg, HLS, ...) are masked and never translated.
A file whose translation fails falls back to English so the build never breaks.
- docs/ru is generated and gitignored — never committed. It is produced locally
(`make docs-serve` / `docs-build`) and in CI.
CI & tooling
- pages.yml: build-then-upload (setup-python -> install -> restore .docs-cache
with restore-keys -> translate -> mkdocs build --strict -> deploy), replacing
the verbatim docs/ upload.
- Makefile: docs-venv / docs-translate / docs-build / docs-serve.
- docs/requirements.txt; .gitignore for docs/ru, site/, .docs-cache.
Migration details
- Removed Docsify control files (index.html, _navbar.md, _sidebar.md, .nojekyll)
and de-Docsify-ed body links in 6 English files (en-us/ aliases -> relative,
swagger _media paths, stripped ':ignore' link syntax).
- Preserved the two Russian-only planning docs (no English source) by moving
them into docs/adr/ as *.ru.md (repo-internal, excluded from the site).
The PHP SignalSender byte-path overlay class had 0 callers after E3 moved
the admin "send message" feature into xc_fanout (drawtext on the viewer's
HLS segment / TS window via FanoutClient::sendSignal -> POST /signal/<uuid>).
Delete the class and scrub its now-dangling mentions:
- src/Streaming/Delivery/SignalSender.php: removed (git rm)
- FanoutClient.php: comment reworded (legacy PHP byte-path, not the class)
- docs/{en,ru}/development/streaming-subsystem.md: dropped the tree line
- docs/adr/0003: overlay is e2e-proven on the LB; note the drawtext-ffmpeg
selection gotcha (bundled 8.0/7.1 lack the filter)
E2 re-applied on the current feature work; live.php 698->467, chase-read +
orphaned SegmentReader removed; daemon-only non-proxy TS. Gating satisfied (daemon
box-proven across all types on LB this session). E3 stays not-done by decision.
xcvm_core self-update ships config_set_redis to LB nodes -> configureRedisLb points
Redis at MAIN -> RedisManager connects -> viewer served video/mp2t by the daemon on
the LB (LINE_CREATE_FAIL gone, /connections+/rates populated, php-fpm flat). Item 1
(LB non-proxy source-mode e2e) closed; edge-types remain for coverage.
Deployed current code to the canary LB (was pre-daemon): ffmpeg tee -> daemon
has_data, /live + /hls served on the LB, viewer redirect lands on the LB. Blocked
at LINE_CREATE_FAIL: redis_connect reads creds from config.enc only, the sole
setter config_set_redis is absent in all xcvm_core builds, and the LB config.enc
has no Redis creds. socat fixes the host (refused->NOAUTH) but not the password.
Needs a new xcvm_core with config_set_redis (StatusCommand::configureRedisLb is
the committed forward-looking fix).
LoopbackCommand mirrors to the daemon ingest socket (like LlodCommand), so an LB
restreaming from MAIN via php_loopback serves viewers through the daemon, not the
PHP byte path. Only a live-node e2e remains (canary LB currently unreachable).
Root cause of the LB 'no fanout locations' defect: the LB build overrides
nginx.conf with the stale lb_configs/nginx.conf (missing all X-Accel locations).
Fixed. libogg already in getPackages; daemon 0.8.0 released. Only a real-source
stream-to-LB e2e remains (external source/assignment).
fanout_binary path verified on canary (0.7.3 -> pulls 0.8.0 from the new GitHub
release -> installs -> restart). Panel P4 (connectionRates + writeDivergence) still
needs to land in the panel release for the telemetry to be consumed fleet-wide.
Daemon GET /rates + fanout_sync writeDivergence proven end-to-end on the canary
(redis mode): viewer on stream 568 -> /rates 857 KB/s -> lines_divergence written
(0.0, healthy). Takes effect fleet-wide once the 0.8.0 daemon is released as
latest.
- 0002 §2.5 rewritten: light-P4 reconciliation shipped; per-viewer divergence is the
remaining P4 gap (daemon-served viewers never enter the chase-read loop)
- 0002 S6 marked done (scale validation: O(N) worker-per-viewer ceiling removed)
- 0002 Risks/§2.4/S1-sketch: auth_request references reframed as superseded by X-Accel;
removed live_fanout flag operational claims
- 0003 Phase C + Risks + snapshot: new P4-in-progress bullet for divergence/bitrate
telemetry, with slice-1 plan (daemon /rates -> fanout_sync writes lines_live.divergence)
Status → Accepted/in-progress. Record what the code actually shows: phases 0/A/B
done, Phase C in progress (C1a cold-start done, C1b moot, C2 adaptive verified,
C3 restreamer-prebuffer fixed), E1 done, Phase F cancelled (on-disk HLS stays).
Add a Phase C work breakdown (C1–C5 + C-ops) including the fanout_sync
self-respawn fix and the daemon-restart re-feed gap surfaced in production.
Close Phase B's unencrypted-only limitation: encrypted HLS is now served from
the daemon's in-RAM segmenter too, so encrypt_hls streams no longer need the
legacy tmpfs .enc path — one of the two prerequisites for dropping the
streaming tmpfs (Phase F).
- StreamProcess: generate the stream's HLS key/iv BEFORE registering the daemon
ingest (moved writeStreamKeyIv up) and, when encrypt_hls is on, hand the hex
key/iv to the daemon at FanoutClient::registerIngest so it encrypts the HLS
segments it serves.
- FanoutClient::registerIngest: optional key/iv passed in the ingest PUT body.
- HLSGenerator::tokenizeDaemonPlaylist: prepend #EXT-X-KEY (URI /key/<token>,
IV from <id>_.iv) for encrypted streams, matching generateHLS.
- live.php m3u8: drop the !encrypt_hls gate — daemon HLS now covers both.
- segment.php unchanged: the daemon-segment X-Accel already returns the
daemon's (now-encrypted) bytes.
Box-validated: encrypt_hls=1 → 569.m3u8 has #EXT-X-KEY + tokenized daemon
segments; a segment decrypts (stream key + playlist IV) to a valid TS. Daemon
crypto is byte-identical to openssl. PHPStan clean, 443 tests. Daemon 0.6.0.
Refs ADR 0003 (Phase B encrypted). Daemon: XC_VM_Binaries 0.6.0.
Phase C (ADR 0003): fix the connection-limit leak for daemon-served live TS.
Under X-Accel the auth PHP-FPM worker returns immediately, so UsersCronJob's
reaper — which closes a TS row when its pid stops being a running php-fpm
worker — can neither track nor time out a daemon viewer (its recorded pid is
a reused, still-alive worker). Those rows leaked, filling the line's slots
until a false "max connections reached".
Fix (light P4):
- live.php marks daemon-served TS rows pid=0 (decided once per request via
$rTSDaemon: proxy $rFanout, or non-proxy isStreamFed), and passes the
connection uuid to the daemon in the X-Accel URL (?c=<uuid>).
- daemon (0.5.0) tracks connected uuids per stream and serves them at
control GET /connections.
- new fanout_sync daemon reconciles pid=0 open rows against that set every
~10s, closing any whose uuid is no longer connected (20s connect-grace);
skips reconcile when the daemon is unreachable (added to service boot()).
- UsersCronJob leaves pid=0 TS rows to fanout_sync (reaper would leak or,
on worker recycle, close them early).
HLS was already self-heartbeating (playlist re-fetch → updateLive), so this
targets long-lived TS. Box-validated: viewer → pid=0 row + uuid in
/connections; disconnect → daemon drops the uuid and fanout_sync closes the
row (slot freed). PHPStan clean, 443 tests. Daemon: XC_VM_Binaries 0.5.0.
Refs ADR 0003 (Phase C).
Phase C parity (ADR 0003): restore the legacy startProxy off-air behaviour
for daemon-served proxy streams. After registering a proxy source, live.php
calls FanoutClient::probe($id, on_demand_wait_time*1000) → daemon
POST /probe/<id>, which prewarms the puller and waits for the source to
produce data. If it stays dead, live.php shows the not-on-air page instead
of X-Accel'ing the viewer onto an empty/hanging daemon stream. A warm stream
(already producing) returns immediately, so only a cold/dead source pays the
wait — same as the legacy path.
Non-proxy off-air was already covered (the process block waits for the
playlist, and A3 gates on isStreamFed/has_data).
Box-validated: dead-source proxy → not-on-air video after on_demand_wait_time
(20s here); live-source proxy → fast play via the daemon (4.4 MB valid TS).
PHPStan clean, 11 FanoutClient tests.
Refs ADR 0003 (Phase C). Daemon: XC_VM_Binaries 0.4.0.
Phase B (ADR 0003): HLS is now served from the xc_fanout daemon's in-RAM
HLS end-to-end for unencrypted streams, PHP doing only auth — the stepping
stone toward dropping the streaming tmpfs.
- live.php m3u8 branch: when the daemon is reachable, the stream is fed
(isStreamFed) and encrypt_hls is off, fetch the daemon playlist
(FanoutClient::hlsPlaylist → client socket) and tokenize its <seq>.ts
entries (HLSGenerator::tokenizeDaemonPlaylist) into the same /hls/<token>
auth'd URLs as the legacy path, with the segment marked <id>_d<seq>.ts.
Else the legacy on-disk generateHLS runs (encrypted / daemon down).
- segment.php: a daemon-segment token runs the same uuid + IP checks, then
X-Accel-Redirect: /xc_fanout_hls/<id>_<seq> → daemon /hls/<id>/<seq>.ts.
- nginx: internal location ^~ /xc_fanout_hls/ → proxy_pass daemon (two path
segments, id_seq, to avoid the server-level rewrite hijack — same reason
as /xc_fanout/ and /xc_hls/).
The daemon's HLS on-demand lifecycle was already closed (0.2.0 touch+reaper).
Box-validated: 569.m3u8 → daemon MEDIA-SEQUENCE + tokenized segments; a
segment fetch → 200 valid TS (953 KB) from the daemon's RAM. PHPStan clean.
Limitation: daemon HLS is plain mpegts → unencrypted streams only; encrypted
streams stay on legacy tmpfs HLS. Daemon-side encryption + removing the tee's
on-disk HLS slave are what Phase F needs to drop the tmpfs.
Refs ADR 0003 (Phase B).
The last live stream type not on the daemon. LLOD v3 is a PHP MPEG-TS
segmenter (not ffmpeg), so it can't tee — instead it now mirrors the raw
bytes it reads into the daemon's push-fed ingest socket
(FanoutClient::registerIngest + a non-blocking stream_socket_client write).
The same stream then fans out via /live/<id> and in-RAM /hls, and live.php's
A3 isStreamFed() routes viewers to the daemon.
- Non-blocking, best-effort write: a daemon stall never slows LLOD's own
segmenting (its primary job); on a write error (daemon gone) it stops
mirroring and live.php falls back to legacy. Null/failed connect ⇒
legacy-only, no behaviour change.
- LLOD keeps writing its on-disk segments (legacy fallback intact).
- Cleanup: closes the conn and unregisters on clean exit; stopStream() is
the backstop when LLOD is killed mid-loop.
Box-validated (stream 599, llod=2): daemon fed (has_data) with a routed
viewer (refs:1), valid /live TS + /hls m3u8, LLOD segments still produced,
playback OK (3.98 MB). PHPStan clean.
Now every live stream type runs through the daemon when reachable: proxy
(S3), non-proxy llod=0/1 (A2 tee + A3), and LLOD v3 (push + A3).
Refs ADR 0003 (Phase A).
Completes Phase A (ADR 0003): non-proxy live streams now both feed the
daemon (A2 tee) and are SERVED by it. At live.php's non-proxy delivery
point, when the daemon is reachable and FanoutClient::isStreamFed($id)
(control GET /streams/<id> → has_data), hand the byte path to nginx→daemon
via X-Accel-Redirect: /xc_fanout/<id> — like proxy mode — instead of
pinning the worker in the per-viewer .ts chase-read. Daemon down / not fed
⇒ the legacy chase-read runs (the reachability rollback).
The tee feeds the daemon from the same ffmpeg that writes the on-disk HLS,
so by delivery time (playlist ready) the daemon has data. Ingest-fed
streams report running:false (no puller), so the gate is has_data.
Box-validated: 6 concurrent non-proxy viewers on stream 569 all served by
the daemon (daemon_conns=6) with php-fpm workers flat (not 1-per-viewer),
valid TS. PHPStan clean, 10 FanoutClient tests.
ADR 0003: A3 done; added a ⚠️ TODO that LLOD v3 (LlodCommand, streams.llod=2)
has no daemon path (separate PHP segmenter) and must be integrated before
Phase D — llod=0/1 already covered via buildLive+tee.
Refs ADR 0003 (Phase A).
Record the daemon ingest listener (0.3.0) as done and box-validated, with
the validated production tee command for A2, and the A2/A3 coordination
(register ingest before ffmpeg; reachability switch).
For a permanent cutover a runtime settings flag is throwaway scaffolding
(and a settings column that ends up always-1 is dead weight). Remove the
`live_fanout` dependency from live.php entirely; the switch is now the
daemon itself.
live.php decides proxy → daemon vs legacy on REGISTRATION SUCCESS, moved
ahead of the legacy process block: if the control socket is present and
FanoutClient::register succeeds, skip legacy startProxy and X-Accel to the
daemon; otherwise run the FULL legacy path (producer included). This also
fixes a latent bug in the old flag path — it skipped startProxy but could
fall through to a producerless legacy relay when registration failed.
Rollback is now "stop the daemon": its socket disappears / register fails,
every stream falls back automatically, per node, no DB write, no cache
rebuild, self-healing via the keepalive. Which stream types go to the
daemon stays a code decision (proxy now; +non-proxy after Phase A).
ADR 0003 Phase 0 dropped (no schema column / admin toggle); ADR 0002
rollback-flag note marked superseded. Test-box live_fanout column to be
dropped.
Refs ADR 0003.
Wrap the xc_fanout launch in service boot() in a keepalive loop (2s
backoff) so a daemon crash is auto-restarted — the Restart=always
equivalent without adding a second systemd unit. Both the loop and the
daemon run as xc_vm, so stop()'s `pkill -u xc_vm` ends both together and
nothing respawns during shutdown; the daemon still logs to tmp/xc_fanout.log.
ADR 0003: mark the Phase C foundations landed (HLS on-demand lifecycle,
off-air status endpoint, auto-restart — daemon 0.2.0 in XC_VM_Binaries).
Refs ADR 0003.
Plan the full transition of all LIVE streaming (proxy + non-proxy, TS +
HLS) onto the xc_fanout daemon, then retire the legacy byte path. Scopes
VOD/timeshift OUT (random-access, not fan-out); keeps per-stream ffmpeg
(transcode/logo) and PHP auth, moving only delivery to the daemon.
Phases: 0 schema/flag column + admin toggle → A non-proxy via tee'd
ingest (S4) → B HLS end-to-end through the daemon (+ HLS on-demand
lifecycle) → C parity/robustness (off-air, cold start, adaptive,
restreamer, limits, disconnect accounting) → D make daemon default → E
delete legacy (ProxyCommand, live.php byte loops, tmpfs HLS serving,
CONS_TMP_PATH sockets) → F drop streaming tmpfs → G LB rollout.
Records the box A/B result and the two daemon gaps surfaced (HLS must join
the on-demand lifecycle; daemon needs a health/last-data signal for
off-air detection).
The fanout hand-off used `X-Accel-Redirect: /xc_fanout/live/<id>`, but nginx
server-level `rewrite` directives run during the rewrite phase on internal
redirects too — before location selection — and the 3-segment catch-all
`^/(user)/(pass)/(stream)` rewrote `/xc_fanout/live/569` into
`/stream/auth?username=xc_fanout&password=live&stream=569`, yielding
INVALID_CREDENTIALS instead of reaching the internal proxy location.
Fix: make the target two path segments (`/xc_fanout/<id>`, location
`^~ /xc_fanout/`), which matches no server rewrite — the same reason P1's
`/xc_hls/<file>` target is two segments. Added a NOTE in nginx.conf so it
isn't "simplified" back to a 3-segment form.
Validated end-to-end on the test node with a real player: flag-off vs
flag-on A/B at 8 concurrent viewers on a real proxy stream — legacy pinned
~1 php-fpm worker + ~24 MB/viewer (9 workers / 242 MB); fanout served all 8
through the daemon (daemon_conns=8) with php-fpm flat at baseline (1 worker
/ 45 MB). PHP fully out of the byte path.
Refs ADR 0002.
Move the daemon's control/client sockets from /run/xc_fanout/ to
bin/xc_fanout/sockets/{control,http}.sock, next to the daemon binary and
mirroring the existing php-fpm sockets layout (bin/php/sockets/) that nginx
already reaches over unix:. Keeps runtime sockets in the app tree the panel
owns rather than a separate FHS runtime dir; a unix socket stores no stream
bytes (pure IPC), so its location is orthogonal to the tmpfs-free byte-path
goal.
- Paths: FANOUT_RUN_PATH now BIN_PATH.'xc_fanout/sockets/' (was /run/...).
- nginx: proxy_pass unix:/home/xc_vm/bin/xc_fanout/sockets/http.sock.
- ADR 0002: §3 placement + §2.4/S3/S4/diagram paths updated; corrected the
imprecise "tmpfs-free runtime dir" wording.
Daemon must launch with -sock .../sockets/http.sock -ctl .../sockets/control.sock
(wired into supervision in S5).
Refs ADR 0002.
Design for the P2 live fan-out daemon: one Go process pulls each live
source once and fans it out to many viewers, keeping PHP out of the byte
path. Records the locked decisions (Go; daemon owns the puller; both TS
sub-modes), the ring-buffer/clean-join model, on-demand lifecycle, and the
build slices S1–S6.
S3 hand-off refined to X-Accel-Redirect (flag stays in PHP, no nginx
reload, mirrors the shipped P1 pattern) instead of auth_request; §2.4/§3
and the S3 slice updated to match the implementation.
Records the decision to take PHP out of the streaming byte path and remove
the tmpfs mounts: native xc_fanout daemon for live fan-out + in-RAM HLS,
nginx X-Accel/proxy_pass with auth_request, connection telemetry in Redis.
Root cause from P0 load test: the ~400-concurrent ceiling is a linear RAM
law (~25 MB per viewer) because each proxy viewer pins a PHP-FPM worker in a
socket_read loop — not tmpfs pressure and not pm.max_children.
Divarion_D