XC_VM/docs/adr/0003-full-daemon-cutover.md

ADR 0003 — Full live-delivery cutover to xc_fanout (retire the legacy byte path)

• Status: Accepted — in progress. Phases 0/A/B/D done & box-validated; Phase C essentially complete (C4 folds into G); E = E1 + dead-code done, E2/E3 gated (fallback); F cancelled; G = daemon proven functional on the first LB, productionisation remains. (updated 2026-08-17)
• Date: 2026-08-16 (rev. 2026-08-17)
• Builds on: ADR 0001 (tmpfs-free streaming), ADR 0002 (xc_fanout daemon). S3 (proxy live TS) + P3 (in-RAM HLS) are box-validated.
• Goal (from Danil): make the daemon the single, always-on delivery layer for all live streams — proxy and non-proxy, TS and HLS — then delete the legacy byte-path code that becomes unused.

Progress snapshot (2026-08-17)

Verified against the code, not just this doc:

• Phase 0 ✅ dropped (no live_fanout flag; switch = daemon reachability).
• Phase A ✅ done — A1 ingest, A2 StreamProcess tee, A3 live.php routing, LLOD v3 push.
• Phase B ✅ done — daemon-served HLS incl. AES-128 encrypted.
• Phase C 🟢 nearly complete — off-air (probe) ✅, disconnect accounting ✅ (fanout_sync self-respawn fixed 2026-08-17), C1a cold-start ✅, C2 adaptive ✅, C3 restreamer-prebuffer ✅, C-ops re-feed ✅, C5 off-air-in-auth closed (already covered). C4 (redirect-to-LB) remains — deferred, needs an LB node; per-viewer divergence/bitrate telemetry (P4 slice 1) 🟢 done (daemon GET /rates → fanout_sync writes lines_divergence; daemon 0.8.0 released as latest 2026-08-18 — nodes auto-update via fanout_binary. Panel side still needs to reach prod: the FanoutClient::connectionRates + FanoutSyncCommand::writeDivergence changes live on feature/tmpfs-free-streaming and must land in the panel release, else the daemon serves /rates but nothing consumes it — see the Phase C breakdown). See the Phase C task breakdown at the end.
• Phase D 🟢 done for the canary's available types — coverage confirmed (llod=0 verified daemon-fed + self-healing; llod=1 covered by construction). Proxy/restreamer/llod=2 lack a canary representative; their full validation rides with C4.
• Phase E 🟡 E1 done + dead-code sweep (removed the never-matching XC_VMProxy[] process checks). E2/E3 deliberately GATED — the live.php chase-read and generateHLS/segment.php serving are the active daemon-reachability fallback (daemon-down, re-feed windows, unvalidated proxy/restreamer/llod=2 types); deleting them removes the rollback. E4 (CONS_TMP_PATH) intentionally retained (not dead).
• Phase F ⛔ cancelled per Danil: on-disk HLS stays (timeshift/thumbnail/analyse depend on <id>_*.ts); the tee's on-disk HLS slave and the streaming-tmpfs drop are off the table. The daemon is the delivery layer; on-disk artifacts remain for non-delivery consumers.
• Phase G (LB) 🟢 install productionised (2026-08-18) — daemon proven functional on the first LB (ingest→fan-out→in-RAM HLS); install/update wired: daemon 0.8.0 released as latest, UpdateCommand + a RootSignals hourly self-heal run fanout_binary, service's keepalive is unconditional so a fresh node/LB self-installs the daemon. Root-cause fixed: the LB build overrides nginx.conf with lb_configs/nginx.conf, which was stale and lacked all streaming X-Accel locations (/xc_hls/, /xc_fanout/, /xc_fanout_hls/) — so a stock LB served neither daemon fan-out nor plain HLS; the three locations are now in lb_configs/nginx.conf (rebuilt archive verified). libogg0 is already in LbInstallFlow::getPackages for every distro. Only open item: a real-source stream-to-LB e2e (needs a source reachable from the LB IP + a streams_servers assignment; the earlier attempt failed on a geo/source defect, not the daemon).

Daemon refinements landed beyond the original text (all Phase-C robustness): client_prebuffer in the TS join (0.7.1); stalled-viewer write-timeout so a half-open client can't pin a connection / leak a ghost (0.7.2).

---

1. What "all streams" means — scope

The daemon is a live fan-out engine: one ingest per stream → many viewers, TS + in-RAM HLS. That maps cleanly onto live streams and nothing else:

Category (streams_types.live)	In scope?	Why
Live, proxy (direct_proxy=1)	✅ done (S3)	daemon owns the pull + fan-out
Live, non-proxy (direct_source=0)	✅ target (Phase A)	ffmpeg stays (transcode/logo); daemon takes delivery via a tee
Live, adaptive / restreamer / redirect-to-LB	✅ target (Phase C)	same fan-out, extra auth/routing parity
VOD / movie	❌ out of scope	file + HTTP range/seek, not fan-out
Series	❌ out of scope	VOD episodes
Timeshift / archive / catchup	❌ out of scope	seeks over recorded segments, not a live tail

Assessment: VOD/timeshift are a different delivery model (random-access byte ranges over files), not a shared live tail. Forcing them through a fan-out engine buys nothing and adds risk. They keep their current path (P1 X-Accel already took their HLS/segment reads off PHP where it mattered). "Full transition" = 100% of live streaming, which is the part that pins workers.

Two things the daemon does NOT replace: the per-stream ffmpeg for non-proxy streams (it does real work — transcode, logo, profiles, timeshift recording) and auth (stays in PHP). The daemon replaces delivery: the source pull for proxy, and the fan-out + HLS segmentation + serving for everyone.

---

2. Target end-state

 proxy live     ─ daemon pulls source ─┐
                                        ├─▶ xc_fanout (per stream: ring + in-RAM HLS) ─▶ nginx ─▶ N viewers
 non-proxy live ─ ffmpeg ─tee mpegts ──┘        (TS fan-out + /hls/<id>/*)              (auth_request/X-Accel)
                    │
                    └─ (still writes timeshift/recording where needed)

• PHP = auth only (live + hls auth), never in the byte path.
• One feed per stream → both TS and HLS from the daemon's RAM (validated: real source → /live/<id> TS and /hls/<id>/index.m3u8 with PTS-accurate EXTINF, concurrently).
• No streaming tmpfs: non-proxy ffmpeg stops writing <id>_*.ts/.m3u8 to STREAMS_PATH; HLS comes from the daemon. (Timeshift/archive files, a separate dir, stay.)
• Legacy deleted: ProxyCommand, the two live.php byte loops, the segment.php/HLSGenerator tmpfs-serving path, the CONS_TMP_PATH datagram sockets.

---

3. Phases (each reversible — the daemon's reachability is the switch; stop it to fall back)

Phase 0 — Schema/enablement (dropped). The original plan added a live_fanout settings column + admin toggle. Removed (done): for a permanent cutover a runtime settings flag is throwaway scaffolding, and a settings column that ends up always-1 is dead weight (cf. P1's dormant hls_accelerator). The switch is the daemon itself. live.php routes a proxy stream to the daemon iff its control socket is present and FanoutClient::register succeeds; otherwise the full legacy path runs (producer included). So stopping the daemon is the rollback — every stream falls back automatically, per node, with no DB write, no cache rebuild, no admin UI, and self-healing (crash → keepalive → back). Which stream types the daemon handles stays a code decision (proxy now; +non-proxy after Phase A), which is the granularity that actually matters. The test-box live_fanout column was dropped.

Phase A — Non-proxy live via the daemon (ADR 0002 S4). Biggest new piece.

• A1 — daemon ingest listener. Done (0.3.0), box-validated. Per-stream push-fed mode: PUT /ingest/<id> listens on <ingestdir>/<id>.sock; the producer connects and pushes mpegts, which the daemon fans out to /live and segments for /hls (no puller). DELETE/Unregister cleans up. Proven with real ffmpeg, including the production tee form below.
• A2 — StreamProcess tee. Done, box-validated. startStream calls FanoutClient::registerIngest($id) before buildLive; when it succeeds, the live ffmpeg's HLS output is emitted through the tee muxer so one ffmpeg feeds both on-disk HLS and the daemon: -map … -f tee "[f=hls…]<id>_.m3u8|[f=mpegts:onfail=ignore]unix:<ingest.sock>" — onfail=ignore keeps HLS alive if the daemon output drops; daemon unreachable ⇒ original on-disk-only HLS (reachability switch). Two tee gotchas handled: tee needs an explicit -map (auto stream-selection fails → "does not contain any stream"); individual_header_trailer is a segment-muxer option the hls muxer ignores in flag form but rejects fatally in a tee slave, so it is omitted.
• A3 — live.php non-proxy → daemon. Done. At the non-proxy delivery point, if the daemon is reachable and FanoutClient::isStreamFed($id) (control GET /streams/<id> → has_data), X-Accel to /xc_fanout/<id> like proxy; else the legacy .ts chase-read. The tee (A2) feeds the daemon from the same ffmpeg that writes the on-disk HLS, so by delivery time (playlist ready) the daemon has data. Note: ingest-fed streams report running:false (no puller) — the gate is has_data, not running.
• LLOD v3 (LlodCommand) — daemon feed done. (was: no daemon path.) On-demand routing (MonitorCommand:329-342 by streams.llod): llod=0 and llod=1 ("v2") go through startStream→buildLive, so A2's tee covers them; llod=2 ("v3") goes to startLLOD→LlodCommand, a separate PHP segmenter (not ffmpeg), which A2 does not touch — those streams stay 100% legacy (verified working, no regression). LLOD v3 now pushes the MPEG-TS it reads into the daemon ingest socket (FanoutClient::registerIngest + a best-effort non-blocking stream_socket_client write), so A3 routes its viewers to the daemon too; it still writes its own on-disk segments as the legacy fallback. Box-validated: daemon has_data + a routed viewer (refs:1) + valid /live and /hls, LLOD segments still produced, playback OK.
• Test: real non-proxy stream, player, TS + HLS both from the daemon; transcode/logo intact.

Phase B — HLS through the daemon end-to-end. Done, box-validated (unencrypted).

• live.php m3u8 branch: when the daemon is reachable, the stream is fed (isStreamFed) and !encrypt_hls, it fetches the daemon's playlist (FanoutClient::hlsPlaylist → client socket /hls/<id>/index.m3u8) and tokenizes its <seq>.ts entries via HLSGenerator::tokenizeDaemonPlaylist into the same /hls/<token> auth'd URLs — segment name marked <id>_d<seq>.ts. Else the legacy on-disk generateHLS runs.
• segment.php: a daemon-segment token (<id>_d<seq>.ts) runs the same uuid + IP auth, then X-Accel-Redirect: /xc_fanout_hls/<id>_<seq> → internal nginx location ^~ /xc_fanout_hls/ → daemon /hls/<id>/<seq>.ts (two-segment target to dodge the rewrite hijack). No on-disk file needed.
• The HLS on-demand lifecycle gap is already closed (daemon 0.2.0: /hls touch() + reaper). Box-validated: 569.m3u8 → playlist with the daemon's MEDIA-SEQUENCE + tokenized segments; a segment fetch → 200 valid TS from the daemon's RAM.
• Encrypted HLS. Done (daemon 0.6.0), box-validated. The daemon now AES-128-CBC-encrypts the HLS segments it serves (hlscrypt, byte-identical to PHP openssl_encrypt) when given the stream's key/iv: StreamProcess reads <id>_.key/_.iv and passes them at registerIngest; live.php's m3u8 branch drops the !encrypt_hls gate and tokenizeDaemonPlaylist prepends the #EXT-X-KEY (URI to /key/<token>, IV from the file); segment.php X-Accels the encrypted bytes from the daemon unchanged. Validated end-to-end: #EXT-X-KEY in the playlist + a daemon segment that decrypts (key + playlist IV) to a valid TS. So both encrypted and unencrypted HLS are now daemon-served, PHP out of the byte path — one of the two Phase-F prerequisites (the other: remove the tee's on-disk HLS slave).

Phase C — Parity & robustness (the real work; gates making the daemon the default). Foundations landed (daemon 0.2.0 + service): HLS now joins the on-demand lifecycle (/hls touch()es the stream — starts/keeps the puller for an HLS-only audience; unified idle-stop reaper covers TS+HLS); off-air signal via control GET /streams/<id> → {running,refs,has_data,since_data_ms}; daemon auto-restarts on crash (keepalive loop in service, Restart=always equivalent). Still to do: wire the auth endpoint to read the status for off-air; on-demand cold-start bounds; adaptive/restreamer/redirect parity; disconnect accounting (→ P4 Redis).

Everything live.php's legacy block does must have a daemon-model equivalent before the daemon can be the default:

• Off-air / source-down detection. Done. POST /probe/<id>?wait=<ms> (daemon 0.4.0) prewarms the puller and waits for first data; live.php calls FanoutClient::probe after registering a proxy source and shows the not-on-air page when has_data is false — parity with legacy startProxy (bounded by on_demand_wait_time; a warm stream returns immediately). Box-validated: dead source → not-on-air after the window; live source → fast play. (Non-proxy off-air is already covered by the process block waiting for the playlist + isStreamFed.)
• On-demand cold start (first viewer starts the stream; bounded wait; expiry).
• Adaptive / multi-bitrate, restreamer chains, redirect-to-LB routing.
• Connection limits / disconnect accounting. Done (light P4). Under X-Accel the auth worker returns immediately, so the reaper's isRunning(pid) check leaks daemon TS rows (a reused worker pid reads as "running"). Fix: daemon-served TS rows are written pid=0; the daemon tracks each viewer's uuid (?c= in the X-Accel URL) and exposes them at control GET /connections; a small fanout_sync daemon closes pid=0 rows whose uuid is no longer connected (20s connect-grace). The reaper skips pid=0 TS. IP/limits/auth stay in live.php at auth time. Box-validated: viewer → pid=0 + uuid in /connections; disconnect → daemon drops the uuid, fanout_sync closes the row. (Full registry-in-Redis is still a possible later P4; this fixes the limit correctness now.)
• Per-viewer transfer telemetry (divergence / bitrate). 🟢 P4 slice 1 — done; daemon 0.8.0 released as latest 2026-08-18. The gap: legacy live.php measured each viewer's byte rate inside its chase-read loop and wrote it to DIVERGENCE_TMP_PATH/<uuid>; UsersCronJob (≈L505–561) globs those files, writes lines_divergence + lines_live.divergence, and compares the observed rate against the stream's expected bitrate (streams_servers.bitrate) for the anti-fraud / connection-health signal. A daemon-served viewer never enters that loop, so its speed files are never written and divergence stays blank — the admin connection-speed column and the rate checks are blind for exactly the viewers the daemon now serves (i.e. all of them, post-cutover). Plan (slice 1): the daemon already tracks each connection by uuid in serveLive; extend it to accumulate bytes-per-connection and expose an avg rate (KB/s) over the control socket (a GET /rates companion to /connections), then have fanout_sync (which already polls /connections) write that rate into lines_live.divergence for pid=0 rows — filling the same column the legacy loop did, from the daemon instead of tmpfs. Note the "fast-pull" fraud vector is structurally absent under the daemon (a viewer cannot pull faster than the live tail is fanned out), so the realized value is the connection-health / speed-display signal rather than rate-abuse detection. The larger CONS/DIVERGENCE-tmpfs → Redis migration (ADR 0001 §4) is a separate, later P4 and is not required for this.

Phase D — Make the daemon the default. 🟢 done for the canary's available types (2026-08-17). No flag to flip: a reachable daemon already serves every wired stream type; Phase D is a coverage confirmation, not new code. Canary audit (single MAIN server):

• llod=0 non-proxy (the dominant type): ✅ box-verified daemon-fed (560/564/568/569 → has_data:true), and the C-ops re-feed keeps them fed across daemon restarts.
• llod=1 (v2): ✅ covered by construction — StreamProcess::startStream:1511 calls registerIngest for it identically to llod=0 (the only difference is -tune zerolatency input flags, which don't touch the tee); a clean box e2e was inconclusive only because the on-demand test-warm churned stream 558's state (and pkill -f on the progress path self-matches the invoking shell — kill by pid).
• llod=2 (v3), proxy (S3), restreamer: code paths validated in earlier phases (A/S3/C3), but this canary has no working representative (599's llod=2 source is a stale BAD_TOKEN loopback; zero direct_proxy streams; zero is_restreamer lines) — fresh box e2e needs those set up, and proxy/multi-node validation naturally rides with C4 (LB rollout).

Conclusion: the daemon is the default for every live type present and working here; the dominant path is verified and self-heals. Rollback stays "stop the daemon" until legacy is deleted (Phase E).

Phase E — Delete legacy (the requested cleanup). Only after D is stable. Per-item status:

• E1 — Cli/Commands/ProxyCommand.php + callers. ✅ done (whole file + ProcessManager::startProxy + refs). Follow-up dead-code sweep: the XC_VMProxy[<id>] process-title checks in ProcessManager::isMonitorAlive / StreamProcess::stopStream can never match now (nothing is titled that) — removed (2026-08-17).
• E2 — live.php non-proxy .ts chase-read loop. ✅ DONE on branch refactor/e2-non-proxy-daemon-only (re-applied on top of the current feature work — P4/Phase G/xcvm_core — after the earlier stale refactor/phase-e-delete-legacy branch was superseded and deleted). live.php's non-proxy TS is now daemon-only: X-Accel to the daemon if $rTSDaemon/isStreamFed, else not-on-air (exactly like the proxy arm). The ~230-line per-viewer chase-read + its SegmentReader/SignalSender/CacheReader imports are deleted (live.php 698→467); the now-orphaned SegmentReader class + its test + doc refs were removed too (CacheReader is used elsewhere and kept; the overlay class was removed in E3 once its feature moved to the daemon). php -l + gates green. Gating now satisfied: the daemon is box-proven as the sole live path across all types this session (llod=0/1, proxy S3, restreamer on the LB; MAIN daemon-serving). Behaviour change: daemon becomes mandatory for non-proxy TS — a daemon-down / re-feed window gives not-on-air, not the legacy feed; git revert is the rollback. Not merged/deployed — land it after the main cutover PR is in prod. The proxy socket-relay half was already removed in E1.
• E3 — client HLS serving = daemon-only. ✅ DONE (branch refactor/e2-non-proxy-daemon-only), per Danil's directive "клиенту отдаётся только через демон". live.php's m3u8 arm dropped the generateHLS fallback (fed ⇒ daemon in-RAM playlist, else not-on-air); segment.php now requires a daemon segment token (<id>_d<seq>.ts → X-Accel to the daemon) and 404s a non-daemon LIVE segment instead of serving on-disk tmpfs — the dead legacy LIVE branch (encrypt-on-read + /xc_hls/) and its orphaned imports were removed, the ARCHIVE (timeshift) path kept. On-disk HLS stays (Phase F cancelled) only for timeshift/thumbnail/.analyse/MonitorCommand, never for clients. Feature preserved — the "Send Message" overlay: E2/E3 removed the PHP byte path that burned an admin text banner onto a viewer's segment (ffmpeg drawtext), so it was re-implemented in the daemon (0.9.0): PHP pushes the signal over the control socket (FanoutClient::sendSignal → POST /signal/<uuid>), and the daemon burns it onto that viewer's next HLS segment / a short TS window (?c=<uuid>&vc=<codec>), best-effort. The now-dead PHP overlay class was deleted — the feature lives in the daemon. e2e-proven on the LB: a green banner was burned onto a signalled viewer's HLS segment then cleared one-shot (visually confirmed). Gotcha found: some bundled ffmpeg builds (8.0/7.1) lack the drawtext filter, so service now selects a drawtext-capable ffmpeg (4.0 has it), falling back to newest (overlay off) if none do. (This makes E3's earlier "keep the cheap X-Accel fallback" recommendation moot — the directive is a strict client-daemon-only path.)
• E4 — CONS_TMP_PATH datagram sockets. ❎ retained by directive — not dead (liveness-touch, geoip cache, cleanup crons).

Each remaining deletion is gated on the daemon being the sole live path for its type; git revert-able.

Phase F — Drop streaming tmpfs mounts (ADR 0001 P5). ⛔ CANCELLED (2026-08-17, Danil). On-disk HLS stays: timeshift, thumbnailing, .analyse, and MonitorCommand health all read <id>_*.ts from STREAMS_PATH. So the tee's on-disk -f hls slave is kept and the streaming tmpfs mount stays. The daemon remains the delivery layer; the on-disk artifacts remain for these non-delivery consumers. (This also removes the E3 pressure to delete generateHLS entirely — the on-disk HLS path stays as a fallback.)

Phase G — LB rollout (P6). 🟡 started (2026-08-17) — daemon on the LB + functional; full e2e blocked on an LB env defect. First LB node added (server 12 testlb, 31.77.173.176, amd64, online). Brought the daemon up on it (canary, manual): deployed xc_fanout 0.7.3, created the sockets dir, added the two ^~ /xc_fanout/ + /xc_fanout_hls/ nginx locations (they were absent — the LB's nginx.conf predated them; nginx -t OK, reloaded), launched the keepalive daemon + fanout_sync. Verified: daemon runs (0.7.3), control + client sockets respond (register/probe/ingest/delete, /hls, /connections). The binary is identical to MAIN's (extensively verified), and service/nginx.conf/live.php/StreamProcess are shared, so routing is by construction.

LB daemon proven functional (2026-08-17): an ingest feed on the LB (PUT /ingest/<id> + a local ffmpeg mpegts) produced has_data:true and a valid in-RAM HLS playlist from the LB's /hls/<id>/index.m3u8 — fan-out + segmenter on the LB are identical to MAIN. The LB's StreamProcess/monitor also correctly read a DB stream assignment, attempted start, and probed the source (ffprobe works). So the LB-side daemon + shared feed/serve code work; only a real external source e2e is still open (the test source bridge-tv served broken h264 to the LB's IP — a source/geo issue, not the daemon).

Findings / gaps for a production Phase G:

• LB env defects — now fixed in the build/provisioning (2026-08-18), not by hand:
  • nginx X-Accel locations (root cause found). The LB build overrides bin/nginx/conf/nginx.conf with lb_configs/nginx.conf (Makefile cp $(CONFIG_DIR)/nginx.conf …), and that file had drifted stale — it lacked all three streaming X-Accel internal locations (/xc_hls/ P1, ^~ /xc_fanout/ P2, ^~ /xc_fanout_hls/ P2/B). So a stock LB served neither daemon fan-out nor even plain on-disk HLS (both go through X-Accel now) — exactly the canary LB's "nginx.conf predated them". Fixed: the three locations added to lb_configs/nginx.conf verbatim; a rebuilt make lb archive now carries them (verified: xc_fanout×12, xc_hls×3; verify-lb-archive green).
  • libogg — already covered: LbInstallFlow::getPackages() installs libogg0 for every distro (debian/ubuntu/redhat), so a fresh LB install gets it via apt. The canary's missing lib was only because it was installed off a stale archive predating that.
• Daemon install path — now real: daemon 0.8.0 released on GitHub as latest (2026-08-18); fanout_binary (not LB-stripped) pulls it, service (shared) runs the keepalive + fanout_sync, and RootSignalsCronJob (not stripped) self-heals hourly. A stock make lb node self-installs the daemon — no manual scp.
• The external admin API (?api_key=…) is the system/reporting surface (user_info/activity_logs/live_connections) — it has no stream→server assignment; that stays a DB/admin-UI operation. Also, cron:streams only starts a stream whose streams_servers row is already "active" (pid/stream_status<>0/to_analyze) — a freshly-inserted all-zero row is skipped.
• Note: LB nodes are not "no-DB" — they connect to the MAIN DB/Redis (config.enc/server_id, crons, and fanout_sync reconciling SERVER#<id>).

Restream-from-origin now goes through the daemon (2026-08-18). An LB restreams a channel from its parent (MAIN) via loopback (php_loopback + streams_servers.parent_id): StreamProcess::startLoopback → console.php loopback <id> <parent>, which reads <parent>/admin/live?stream=… MPEG-TS and (previously) segmented to disk in PHP only. Like LLOD v3, that kept the LB's viewers on the PHP byte path (worker-per-viewer on the LB — the exact thing G removes). Fixed: LoopbackCommand now mirrors its sanitized whole-packet buffer into the daemon's ingest socket (registerIngest/best-effort non-blocking fwrite/unregister, same pattern as LlodCommand; mirror the cleaned buffer since admin/live pads with 0xFF). Once fed, isStreamFed() routes LB viewers to the daemon /live + in-RAM /hls through the X-Accel locations just restored to lb_configs/nginx.conf. Daemon down ⇒ legacy on-disk HLS, unchanged.

LB e2e run (2026-08-19) — daemon delivery PROVEN on the LB; blocked on an unrelated LB↔Redis gap. The canary LB (31.77.173.176, 805 MB box) ran entirely pre-daemon panel code (A2/A3/S3 absent) even though its daemon binary was 0.8.0 and nginx had the fanout locations. Deployed the current LB code, activated a source-mode stream (560, direct-TS bridge, reachable from the LB IP; 564/558 also reachable, 569 geo-blocked). Result: ffmpeg -f tee → daemon has_data:true, and the daemon serves /live/560 (MPEG-TS) + in-RAM /hls/560 on the LB — the A2/A3 delivery path works on an LB, and the viewer redirect lands on it (force_server_id=12 → 302 → remote_ip=31.77.173.176). But the LB returns LINE_CREATE_FAIL: live.php cannot record the connection because Redis is unusable on the LB. Root cause (dug to the bottom): \XC_VM::redis_connect() reads host+password from config.enc only; the sole setter \XC_VM::config_set_redis() does not exist in any current xcvm_core build (absent on both MAIN and LB — MAIN works only via a local 127.0.0.1 Redis whose creds are already in its config.enc). The LB has no local Redis (bin/redis stripped), its config.enc holds no Redis creds, and nothing can set them. A socat 127.0.0.1:6379→MAIN forward fixes the host (Connection refused→NOAUTH) but the extension still sends no password. So LB Redis — hence daemon (and legacy) connection tracking under redis_handler — cannot work until a new xcvm_core ships config_set_redis. StatusCommand::configureRedisLb is committed as exactly that forward-looking fix (guarded by method_exists, no-op until the extension has the method). Notes: the LB needed vm.overcommit_memory=1 to start a stream on 805 MB; loopback restream was not exercised — the assigned LB streams are source-mode (external source), which the user confirmed is intentional.

LB↔Redis RESOLVED (2026-08-20) via a new xcvm_core self-update. The blocker was that config_set_redis was absent from the extension build the LB shipped, so StatusCommand::configureRedisLb could not point the LB's Redis at MAIN. The extension is now mirrored into the binaries repo tree (bin/xcvm_core/ — per-OpenSSL-ABI archives openssl1.1/openssl3 + version.json + SHA256SUMS), decoupled from the heavy runtime bundle, and a new XcvmCoreCommand (console.php xcvm_core) installs/updates it (OpenSSL-ABI detect → SHA-256-verified download → atomic install with load-test + rollback → php-fpm reload), wired into UpdateCommand + RootSignalsCronJob (hourly self-heal), not LB-stripped. Full LB e2e now proven: on the canary LB xcvm_core went none→2.1.0 (openssl3), config_set_redis appeared, configureRedisLb pointed Redis at the main server, RedisManager connected, and a viewer redirected to the LB was served video/mp2t by the daemon — LINE_CREATE_FAIL gone, daemon /connections + /rates (1102 KB/s) populated, php-fpm flat at 1 worker (PHP out of the byte path on the LB).

Remaining: the install + restream + extension paths are all productionised in code, and the daemon delivery path (source-mode non-proxy) is fully box-proven on an LB end-to-end. What's left is only edge-type coverage on the LB — proxy (direct_proxy=1), restreamer, and llod=2 need representatives set up to box-validate (llod=0 non-proxy is done); and the loopback restream path (php_loopback+parent_id) was not exercised (the assigned LB streams are source-mode by the user's choice).

---

4. Assessment / recommendations ("что скажешь")

• Yes to the full cutover for live — the box A/B proves the payoff (8 viewers: 9 php-fpm workers/242 MB → 1 worker/45 MB). It scales the exact wall P0 found.
• Keep VOD/timeshift out — different model; no benefit, real risk. (Confirm this scope.)
• The daemon replaces delivery, not ffmpeg — non-proxy transcode stays; we only tee its output.
• Delete legacy LAST (Phase E), not now — keep both paths (daemon reachable ⇒ daemon, else legacy) until parity (Phase C) is proven in production. Deleting ProxyCommand/live.php loops before that removes our rollback.
• Real effort is Phase C, not the happy path. Off-air, on-demand cold-start, adaptive, restreamer, limits, and disconnect-accounting are where correctness lives. Budget accordingly.
• Two concrete daemon gaps surfaced already: (1) /hls must join the on-demand lifecycle; (2) the daemon needs a health/last-data signal for off-air. Both are Phase A/B.
• Order: Phase 0 (schema) → A (non-proxy) → B (HLS) → C (parity) → D (default) → E (delete) → F (tmpfs) → G (LB).

---

5. Risks

• Parity gaps silently degrade edge streams (adaptive/restreamer/off-air) — mitigate with per-feature regression before widening the flag.
• On-demand lifecycle races (first-viewer start / last-viewer stop) across TS+HLS audiences — one lifecycle owner per stream in the daemon.
• Disconnect accounting moves to the daemon (PHP no longer sees it) — until P4/Redis, limits lean on the auth-time write + reaper.
• Transfer telemetry blind for daemon viewers — was measured by the legacy chase-read loop only. Closed by P4 slice 1 (daemon GET /rates → fanout_sync writes lines_divergence/lines_live.divergence), daemon 0.8.0 released as latest 2026-08-18; effective once the panel P4 changes (feature/tmpfs-free-streaming) also reach prod.
• Deleting legacy too early — gate strictly on Phase D stability.
• Timeshift/recording coupling — the non-proxy ffmpeg also feeds recording; the tee must not disturb those outputs.

---

6. Phase C work breakdown (2026-08-17)

Ordered by observed pain / risk. Each item ships with a regression check and is reachability-reversible.

• C1 — On-demand cold-start robustness. Two cold-start paths, both can fail the first attempt today:
  • C1a (proxy / daemon-pull): the daemon's puller ffmpeg (non-video/mp2t source) runs with no -probesize/-analyzeduration tuning and no reconnect flags → slow / flaky first output → probe can exceed on_demand_wait_time → not-on-air on the first try, warm on retry. Fix in the daemon runFfmpeg: bounded low-latency analyze + reconnect, without breaking codec detection.
  • C1b (non-proxy / panel tee): observed on stream 567 — moot: 567 turned out to have a broken source link (deleted, replaced by 558/llod=1, which works). The non-proxy monitor hard-fails if ffmpeg exits during warm-up, but with a healthy source the current window is adequate. Deferred until a real non-proxy cold-start failure surfaces.
• C2 — Adaptive / multi-bitrate parity. ✅ DONE — box-verified (2026-08-17); no daemon-specific code needed. Box check: temporarily set adaptive_link=[560,568] on 569 (non-proxy), rebuilt the stream cache (cron:cache_engine — the per-stream cache info carries adaptive_link; cron:cache is a different job), and fetched 569.m3u8 with a real line → a correct 3-variant master (#EXT-X-STREAM-INF for 568 720p / 560 576 / 569 base, sorted by BANDWIDTH desc, each an /auth/<token> URL). Reverted 569 afterwards. Adaptive is a composition of already-daemon-served streams, not a separate path: auth.php (≈480–524) builds the #EXT-X-STREAM-INF master playlist from streams.adaptive_link and serves it itself; each variant URL is a normal .../auth/<token> that re-enters live.php's m3u8 branch, where Phase B already routes it to the daemon (isStreamFed → hlsPlaylist → tokenizeDaemonPlaylist) or falls back to generateHLS. So each variant is daemon-served exactly like any non-proxy HLS stream — no adaptive-specific daemon work. Pre-existing constraints (not daemon regressions): (1) direct_proxy + adaptive is disabled (auth.php → HLS_DISABLED); (2) the master lists only variants whose stream_info (bitrate/w/h) is populated, so a fully-cold adaptive stream yields a thin/empty master until variants warm. Pending: box e2e (no adaptive stream is currently configured on the canary — set adaptive_link on a warm stream to verify variant playlists come from the daemon).
• C3 — Restreamer-chain parity. 🟡 code fixed + deployed (2026-08-17); box e2e pending. Found & fixed a real gap: the daemon X-Accel hand-off ignored the token's prebuffer flag (set by ?prebuffer / X-XC_VM-PREBUFFER), always passing restreamer_prebuffer. Legacy gives a restreamer that requested prebuffer a seg_time burst; through the daemon it got restreamer_prebuffer (0 here) → empty cache. Both hand-off branches now mirror legacy: is_restreamer ? (token.prebuffer ? seg_time : restreamer_prebuffer) : client_prebuffer (commit 5054f05a). Other restreamer specifics need no daemon work: 2nd-IP skip / restreamer_bypass_proxy / disable_*_allow_restream are auth-time (pre-delivery, shared by both paths); the legacy chase-read's restreamer fast-exit has no daemon equivalent (the daemon streams bytes directly). Box e2e pending: the canary currently has no is_restreamer line and no daemon-fed stream (all /streams/<id> are 404 after the daemon restart — needs re-warm).
• C4 — Redirect-to-LB parity. Streams with redirect_id/originator_id routed to an LB node — confirm daemon behaviour vs legacy.
• C5 — Off-air status in the auth endpoint. ✅ CLOSED — off-air already sufficiently covered (2026-08-17). Off-air is correct today via two mechanisms: auth.php (137–173) does a cache-based check (stream_status/pid/to_analyze across servers → not_on_air_video before redirect) for non-proxy streams, and live.php does the daemon probe for proxy streams (real source liveness). The only residual: proxy streams bypass the auth check (lines 161/165 mark direct_source=1 && direct_proxy=1 unconditionally "available"), so a dead proxy source is only caught by the live.php probe. Closing this in auth would be a latency/worker optimization, not a correctness fix (fail-fast for a warm-but-just-died proxy stream, avoiding repeated 20s probe-waits that pin FPM workers) — and it is a hot-path change to auth.php (every stream request) with a cross-server nuance (the proxy daemon lives on redirect_id's server, remote in multi-server) and a staleness-threshold question. Decision (Danil): not worth the hot-path risk for the marginal gain — closed. Revisit only if repeated probe-waits on dead proxy channels become a measured problem (then: same-server-only daemonOffAir($id) guard on that available clause).
• C-ops — fanout_sync supervision. ✅ fixed (self-respawn restartDaemon); consider having watchdog also resurrect it if absent (belt-and-braces), since it is otherwise launched only by service at boot.
• C-ops — re-feed after a daemon restart. ✅ DONE + box-verified (2026-08-17). A daemon restart (deploy/crash) wipes the in-memory registry, and the already-running ffmpeg tees keep writing to the now-dead ingest sockets (onfail=ignore), so every ingest-fed stream stays off the daemon (/streams/<id> → 404, delivery silently on legacy) until its ffmpeg is restarted and re-PUT /ingests. On-demand streams self-heal on the next viewer; 24/7 streams used to need a manual re-warm. Fix (commit 34a2d721): cron:streams now detects a running stream the daemon has forgotten via FanoutClient::daemonStreamMissing() (reachable-daemon 404 only — a stopped daemon leaves legacy alone) and restarts it (startMonitor) so buildLive re-registers the ingest; throttled by a <id>_.refeed stamp (120s) against restart loops. Proxy streams have no local ffmpeg and never reach that branch. Box-verified: restart daemon → 4 streams go 404 → next cron:streams logs "re-feeding" for each → all has_data:true within ~22s. (Op note: a pkill -f on the progress path self-matches the invoking shell — kill by pid.)

Starting with C1 (the concrete observed regression).