The probe endpoint and the source scanner injected the prebuffer/detect
markers with a stray backslash in the header field-name
(`X-\XC_VM-Prebuffer` / `X-\XC_VM-Detect`), a leftover from the namespace
migration. A backslash is not a valid HTTP token character, so strict
origins/CDNs (e.g. Akamai) answer 400 Bad Request and the probe fails
silently (ffprobe runs with `-v quiet`), breaking Load Maps and stream
checks for otherwise-valid sources.
Use the correct `X-XC_VM-Prebuffer` / `X-XC_VM-Detect` names, matching what
StreamProcess sends and what auth.php/live.php read back.
Bundled binaries are stored in plain Git now, so the release/pre-release builds no longer need (and must not depend on) a Git LFS fetch — which was failing once the repo hit its LFS budget. The Makefile verify_no_lfs_pointers guard still fails the build if an LFS pointer stub ever slips into the archive.
Convert the 12 bundled binaries at HEAD from Git LFS pointers to regular Git objects (ffmpeg/ffprobe x3, redis-server, yt-dlp, MaxMind GeoIP2-ISP + cidr, guess, login-bg.mp4) and remove the filter=lfs rules from .gitattributes. Plain-git clone bandwidth is not metered, so this stops consuming the repo's Git LFS budget. HEAD-only: pre-existing commits/tags still reference LFS objects (no history rewrite).
Run tools/docs/translate.py with python -u and print per-file progress so 'make docs-translate' shows live output instead of going silent. Relocate the translation cache and venv under build/ (build/docs-cache, build/docs-venv).
The /stream.ts endpoint spawned a fresh per-client ffmpeg reading the file from the start, so the panel re-opening a channel always rewound to the beginning. Each client's ffmpeg now seeks to the current live position of the loop (offset = (now - start) %% duration); -ss lands on the nearest keyframe for a fast LLOD start, and clients connecting at the same wall-clock get the same offset (in sync).
New streamtest-gen / -gen-stop / -backup / streamtest commands: configure the panel once by hand, snapshot the DB, then restore + run repeatably. The container runner rebuilds caches, launches the test-stream-generator, fetches the panel's output m3u for the test line, and runs the checker (aggregate JSON + per-stream logs copied out). STREAMTEST.md documents the flow; docker-compose mounts tools/ and exposes port 8088; ignore out/, fixtures/, graphs/.
stream_queue_check.py: batch --playlist mode, per-stream JSON via --out-dir, TTY-aware summary vs JSON, HLS health judged by rebuffers (TS by stall), 120s default duration, --tolerance/--stall-timeout. New stream_graph.py renders the checker JSON as dependency-free SVG charts (per-stream + --combined comparison, unique colour per stream). Both moved under tools/stream-check/ with a README; tools/README.md and docs/en updated.
yt-dlp resolves media URLs (StreamUtils) but is a static bundled binary that
only refreshes on a panel update — the shipped 2025.07.21 is ~13 months stale,
which breaks extraction. Add self-updating, reusing the existing binaries
self-heal mechanism (no new cron/crontab row):
- New `console.php ytdlp` (YtDlpCommand), modelled on FanoutBinaryCommand:
resolves the latest upstream `yt-dlp/yt-dlp` release, and on a version mismatch
downloads the `yt-dlp` asset, verifies it against upstream `SHA2-256SUMS`,
run-tests `--version`, then atomically swaps + chowns xc_vm. A broken download
never replaces a working binary.
- RootSignalsCronJob: poll `console.php ytdlp` daily (stamp `ytdlp_check`, 86400s),
next to the fanout_binary / xcvm_core self-heal — runs on main + LB, first pass
immediately.
- Settings → Info: show the live yt-dlp `--version`; drop the (now auto-updated,
always-stale) yt-dlp version row from the README tech stack.
Validated against upstream: asset `yt-dlp`, `SHA2-256SUMS`, checksum match,
`--version` run-test (latest 2026.08.19).
Apply the new phpcs + Slevomat ruleset across src/ (phpcbf): 85 files. The bulk
are unused `use` imports that PHP-CS-Fixer's no_unused_imports missed (class
name only present in a PHPDoc description), plus blank-line normalization around
the use/namespace blocks. Verified safe: `make phpstan` stays green (0 errors) —
no import used in code or a real docblock type was removed.
PHP-CS-Fixer's `no_unused_imports` is conservative — it treats a class name that
merely appears in a PHPDoc *description* as "used", so genuinely-dead imports
(e.g. `use ...Request;` next to a "Request IP" doc description) were never
flagged. Slevomat's UnusedUses is precise: it parses annotation *types*
(@param/@return/@var), so it keeps docblock-typed imports but removes truly
unused ones — matching what Intelephense (P1003) reports.
- Swap require-dev: friendsofphp/php-cs-fixer -> squizlabs/php_codesniffer +
slevomat/coding-standard (+ phpcodesniffer-composer-installer, allow-listed).
- New narrow ruleset build/phpcs.xml.dist (import/namespace hygiene only, NOT
full PSR-12): UnusedUses (searchAnnotations=true), UseFromSameNamespace,
UseDoesNotStartWithBackslash, AlphabeticallySortedUses, UseSpacing,
NamespaceSpacing. View templates stay excluded.
- Makefile: `make cs` -> phpcs, `make cs-fix` -> phpcbf (same target names).
- CI code-style job, CLAUDE.md, CONTRIBUTING.md, docs, .gitignore updated;
build/.php-cs-fixer.dist.php removed. Committed vendor stays production-only.
Move the generated site out of the repo root: site_dir -> build/site, with the
Pages upload path and .gitignore updated to match. build/ already holds tooling
config, so the build output no longer clutters the top-level listing.
The Docsify hash routes (`/XC_VM/#/en-us/…`, `#/ru-ru/…`) no longer exist after
the MkDocs Material migration. Point the README at the new URLs: English at the
site root, Russian under /ru/, and pages as directory URLs.
Add a "Documentation" section to CLAUDE.md so every contributor/agent knows the
MkDocs workflow this branch introduces: English (docs/en) is the single source;
docs/ru is generated by tools/docs/translate.py and regenerated locally before a
release — never hand-edit it.
Sidebar/tab labels live in mkdocs.yml nav, not in the translated Markdown, so
the ru site was showing English menu titles over Russian content. Add a
nav_translations map for the ru locale (mkdocs-static-i18n) covering every tab,
section and page title — "User Guide" → "Руководство пользователя", etc. English
nav is unchanged; strict build clean.
Fix Markdown-mangling artifacts the free web engine (yandex) produced in the
committed docs/ru, and regenerate the whole tree cleanly (0 fallbacks):
- Sentinel format @@N@@ -> {N}. MT engines are trained to preserve curly
format-string placeholders, so {N} survives code-heavy lines where @@N@@ (and
ZZZ…ZZZ, which also duplicated its Z) were split/moved — e.g. the stray
"@0@@" in the FAQ and "load balancerZ" in the README are gone.
- Possessive: a trailing English `'s` is consumed INTO the masked span and
dropped on restore. Every sentinel format breaks when a bare `'s` sits right
after it, and Russian has no possessive `'s`.
- Validate + retry: after restore, any leftover brace fragment triggers a retry
(the engine is non-deterministic); after a few failures the line stays English
so a broken token is never emitted.
- Glossary += KeyDB, yt-dlp, Ubuntu, iptables, MAGSCAN.
Regenerated docs/ru (37 files, translators/yandex): no residual sentinels,
mkdocs build --strict clean.
CI translation was slow, so move it out of GitHub Actions: docs/ru is now a
committed, generated tree refreshed LOCALLY before a release; CI only builds it.
- pages.yml: drop the setup-python/cache/translate steps — the workflow now just
installs the build toolchain and runs `mkdocs build --strict` on the committed
en+ru trees.
- .gitignore: stop ignoring docs/ru (now committed); keep site/ + .docs-cache/.
- Split deps: docs/requirements.txt = build only (mkdocs-material, static-i18n,
used by CI); tools/docs/requirements.txt = translators (local-only).
- Makefile: docs-venv installs both; docs-build/docs-serve no longer translate
(translation is the deliberate `make docs-translate` release step).
- updates_checklist.md: new "Regenerate translated documentation" step
(make docs-translate + docs-build, commit docs/ru with the release commit).
- Commit the generated docs/ru (37 files, translators/yandex).
Rule: edit ONLY docs/en; docs/ru is generated — never hand-edit it.
Enrich the English source (ru regenerates automatically) with subsystem
behaviour that was missing or stale:
- streaming-subsystem: live client delivery is now daemon-only via xc_fanout
(X-Accel handoff, fan-out over a unix socket, control-socket off-air/telemetry,
fanout_sync reconciliation); on-disk HLS kept only for timeshift/thumbnail/
analyse. Documented the admin "Send Message" drawtext overlay via
POST /signal/<uuid>. Replaced the stale generateHLS/chase-read delivery step.
- caching-and-redis: how long-lived daemon connections survive a server idle
`timeout` close (phpredis silent reconnect without AUTH → non-PONG guard +
re-authenticated reconnect; getCapacity multi() guard), and cold-cache
fail-closed defaults in LegacyInitializer::initStreaming().
Replace the hand-maintained Docsify site (parallel docs/en + docs/ru trees
that had already drifted) with a MkDocs Material build where English is the
single source of truth and Russian is generated at build time.
Engine & structure
- mkdocs.yml: Material theme, site_url for the /XC_VM/ Pages subpath, and a
two-tab information architecture — User Guide (administration, API/Swagger,
UI translations, diagnostics, info/FAQ) vs Developer Guide (architecture,
workflow, security, integrations, build). Files are NOT moved — the split is
nav-only, so URLs and cross-links stay stable.
- mkdocs-static-i18n (folder mode): English at root, Russian under /ru/, with a
language switcher. `mkdocs build --strict` validates every link/anchor.
Translation pipeline (tools/docs/translate.py)
- Engine-agnostic via DOCS_TRANSLATE_PROVIDER: translators (free, no API key —
default), anthropic, deepl, or noop. Per-file sha256 cache so only changed
English files are re-translated. Markdown-safe: code, URLs, HTML tags and
glossary terms (XC_VM, FFmpeg, HLS, ...) are masked and never translated.
A file whose translation fails falls back to English so the build never breaks.
- docs/ru is generated and gitignored — never committed. It is produced locally
(`make docs-serve` / `docs-build`) and in CI.
CI & tooling
- pages.yml: build-then-upload (setup-python -> install -> restore .docs-cache
with restore-keys -> translate -> mkdocs build --strict -> deploy), replacing
the verbatim docs/ upload.
- Makefile: docs-venv / docs-translate / docs-build / docs-serve.
- docs/requirements.txt; .gitignore for docs/ru, site/, .docs-cache.
Migration details
- Removed Docsify control files (index.html, _navbar.md, _sidebar.md, .nojekyll)
and de-Docsify-ed body links in 6 English files (en-us/ aliases -> relative,
swagger _media paths, stripped ':ignore' link syntax).
- Preserved the two Russian-only planning docs (no English source) by moving
them into docs/adr/ as *.ru.md (repo-internal, excluded from the site).
A long-lived daemon (the watchdog) holds one Redis connection through the
RedisManager singleton. When it sits idle past the server's `timeout`, the
server closes the socket; phpredis then transparently reopens it WITHOUT
replaying AUTH, so the next command answers NOAUTH / returns false — which is
how getCapacity() hit "zCard() on bool".
instance()'s debounced health-check only nulled the connection when ping()
*threw*; a non-true ping reply (the silent-reconnect/NOAUTH state) slipped
through and was trusted for another 30s. Treat any non-PONG reply as dead and
fall through to the existing re-authenticated reconnect (ensureConnected →
XC_VM::redis_connect). Verified live on an LB node: after a server-side
CLIENT KILL, reconnect() restores a working, authenticated connection.
This is the client-side fix; the MAIN redis.conf `timeout` is left unchanged.
Two distinct production faults from the panel logs:
Root — cold/absent file cache (MAIN warnings). The streaming bootstrap read
the servers/blocklist/proxy caches with no default, so when a cache file is
not yet built (fresh boot, cleared tmp, before the first cache build) they
came back null. Every downstream consumer that iterates or in_array()'s them
then warns — or fatals on PHP 8 (`in_array($x, null)`, `array_key_exists(...,
null)`). The two logged sites (StreamRedirector foreach, auth.php enable_proxy
offset) were just the ones that happened to hit it.
- LegacyInitializer::initStreaming(): default every cache-read global to an
array (matching the existing rBouquets guard). A cold cache now fails closed
(no servers → not-on-air for that request) instead of warning/fatalling.
- Defence-in-depth at the two use sites: StreamRedirector normalises $rServers;
auth.php null-coalesces $rServers[SERVER_ID]['enable_proxy'].
LB Redis fatal — ConnectionTracker::getCapacity(). A dropped LB→MAIN socket
makes phpredis multi() return false (its 30s-debounced ping misses the drop),
and zCard() on that bool fatals outside the RedisException catch, taking down
the watchdog. Guard the multi() result and throw RedisException so the existing
reconnect+retry handles it; a still-dead Redis degrades to a 0-count cycle.
- Footer (getFooter): show "Vateron Media" as a link to the project + an
AGPL-3.0 license link and version, en-dash year range; rendered on
admin/reseller/player footers.
- Login page: add a short brand line (project + AGPL-3.0 links).
- Settings → Info: two new version badges — the xc_fanout daemon
(`-version`) and the xcvm_core extension (version marker), N/A when
unavailable.
- AdminHelpers: flatten the empty-if/else artifacts (validateCIDR,
overwriteData, sortArrayByArray, convertToCSV) and translate the class
docblock to English; behaviour unchanged.
The PHP SignalSender byte-path overlay class had 0 callers after E3 moved
the admin "send message" feature into xc_fanout (drawtext on the viewer's
HLS segment / TS window via FanoutClient::sendSignal -> POST /signal/<uuid>).
Delete the class and scrub its now-dangling mentions:
- src/Streaming/Delivery/SignalSender.php: removed (git rm)
- FanoutClient.php: comment reworded (legacy PHP byte-path, not the class)
- docs/{en,ru}/development/streaming-subsystem.md: dropped the tree line
- docs/adr/0003: overlay is e2e-proven on the LB; note the drawtext-ffmpeg
selection gotcha (bundled 8.0/7.1 lack the filter)
The send-message overlay needs ffmpeg's drawtext filter, but some bundled builds
(8.0/7.1) lack it despite libfreetype — so the newest-ffmpeg pick silently no-op'd
the overlay (verified on the LB: 8.0/7.1 no drawtext, 4.0 yes). The service now
scans newest→oldest for a drawtext-capable ffmpeg and uses that, falling back to
the newest (overlay off) if none have it. Proven e2e: green 'HELLO XC_VM OVERLAY'
burned onto a signalled viewer's HLS segment; one-shot cleared after.
Complete the client cutover: HLS is now served to clients ONLY by the daemon, and
the admin "send message" text overlay (previously burned in by the PHP byte path)
moves to the daemon so the cutover doesn't regress it.
E3 (client HLS = daemon-only):
- live.php m3u8 arm: drop the legacy generateHLS fallback — a fed stream serves the
daemon's in-RAM playlist (plain or AES-128), else not-on-air (like the TS arm).
- segment.php: LIVE segments must be daemon tokens ("<id>_d<seq>.ts") → X-Accel to
the daemon; a non-daemon LIVE segment now 404s instead of serving on-disk tmpfs.
Removed the dead legacy LIVE branch (encrypt-on-read + /xc_hls/) and its orphaned
SignalSender/LegacyInitializer/AsyncFileOperations imports; kept the ARCHIVE
(timeshift) path untouched. Cleaned empty-if/else guard artifacts along the way.
- On-disk HLS stays (Phase F cancelled) only for timeshift/thumb/.analyse/Monitor —
never served to clients.
Send-message overlay via the daemon (feature parity — see the daemon's 0.9.0):
- FanoutClient::sendSignal → control POST /signal/<uuid>; InternalApiController's
signal_send pushes there (legacy tmpfs signal file kept as a harmless no-op).
- segment.php passes ?c=<uuid>&vc=<codec> and live.php passes &vc=<codec> so the
daemon can burn the banner onto that viewer's HLS segment / TS window.
- service launches the daemon with -ffmpeg (newest bundled) + -font free-sans.ttf.
php -l + make gates green.
E2 re-applied on the current feature work; live.php 698->467, chase-read +
orphaned SegmentReader removed; daemon-only non-proxy TS. Gating satisfied (daemon
box-proven across all types on LB this session). E3 stays not-done by decision.
E2 deleted live.php's non-proxy chase-read — the only user of SegmentReader
(playlist segment extraction). Remove the now-dead class, its unit test, and the
doc references. SignalSender (still used by segment.php) and CacheReader (used
widely) are kept.
Phase E (ADR 0003), re-applied on top of the current feature work (P4/Phase G/
xcvm_core). live.php's non-proxy TS delivery is now daemon-only: if the daemon
serves the stream ($rTSDaemon / isStreamFed) it X-Accels to the daemon; otherwise
it shows not-on-air, exactly like the proxy arm. The ~230-line legacy per-viewer
chase-read (playlist read → prebuffer wait → segment loop → connection monitoring)
and its now-unused SegmentReader/SignalSender/CacheReader imports are deleted;
live.php shrinks 698→467 lines. php -l + make gates green.
Behaviour change: the daemon becomes mandatory for non-proxy TS — during a
daemon-down / re-feed window a viewer gets not-on-air, not the legacy feed. This
removes PHP from the TS byte path entirely. git revert is the rollback.
xcvm_core self-update ships config_set_redis to LB nodes -> configureRedisLb points
Redis at MAIN -> RedisManager connects -> viewer served video/mp2t by the daemon on
the LB (LINE_CREATE_FAIL gone, /connections+/rates populated, php-fpm flat). Item 1
(LB non-proxy source-mode e2e) closed; edge-types remain for coverage.
The xcvm_core PHP extension is mirrored into the binaries repo tree — decoupled
from the heavy per-distro runtime bundle — under bin/xcvm_core/ as per-OpenSSL-ABI
archives (openssl1.1 / openssl3) + version.json + SHA256SUMS. Nothing panel-side
kept it current, so nodes drifted onto whatever build their install archive shipped
— and critically the LB's build lacked XC_VM::config_set_redis, so an LB could not
point its Redis at the main server and every redis_handler connection failed with
LINE_CREATE_FAIL.
Add XcvmCoreCommand (console.php xcvm_core), modelled on FanoutBinaryCommand:
- resolves the latest version from version.json (raw, main branch);
- detects this host's OpenSSL ABI (libcrypto.so.3 -> openssl3, else openssl1.1;
both present -> prefer openssl3 with a load-test fallback to the other);
- downloads the matched archive, verifies SHA-256;
- installs the .so ATOMICALLY with a backup + fresh-php load-test + rollback, so a
wrong-ABI/broken extension never takes php-fpm down, then USR2-reloads php-fpm;
- version-compared via a sidecar marker next to the .so (idempotent).
Wired into UpdateCommand (post-update) and RootSignalsCronJob (hourly self-heal),
not LB-stripped, so every node converges. Proven on the canary LB: none -> 2.1.0
(openssl3), config_set_redis appears, StatusCommand::configureRedisLb points Redis
at main, RedisManager connects, and a viewer redirected to the LB is served
video/mp2t by the daemon (/connections + /rates populated, php-fpm flat at 1).
Deployed current code to the canary LB (was pre-daemon): ffmpeg tee -> daemon
has_data, /live + /hls served on the LB, viewer redirect lands on the LB. Blocked
at LINE_CREATE_FAIL: redis_connect reads creds from config.enc only, the sole
setter config_set_redis is absent in all xcvm_core builds, and the LB config.enc
has no Redis creds. socat fixes the host (refused->NOAUTH) but not the password.
Needs a new xcvm_core with config_set_redis (StatusCommand::configureRedisLb is
the committed forward-looking fix).
configureRedis (which calls XC_VM::config_set_redis to sync the extension's Redis
target) runs on MAIN only, so an LB never configured its Redis host — the xcvm_core
extension kept its 127.0.0.1 default, XC_VM::redis_connect() got connection-refused
(LBs have no local Redis; bin/redis is stripped from the LB build), and under
redis_handler live.php failed every connection with LINE_CREATE_FAIL.
Add configureRedisLb: on a non-main node, resolve the MAIN server's IP (private
link preferred, else server_ip) and point the extension's Redis at <main>:6379 with
the shared password READ from settings (never rotated — that stays MAIN's job).
Guarded by method_exists(config_set_redis): a node whose xcvm_core predates that
API no-ops safely and needs its binaries/extension updated to pick this up.
panel_logs entries could not be attributed to their origin node: FileLogger wrote
each record WITHOUT server_id, and ErrorsCronJob::parseLog always tagged rows with
the SERVER_ID of whichever node parsed the file — so a log written on an LB but
collected/parsed as MAIN showed up as MAIN (server_id=1), making 'is this error
from the LB or MAIN?' unanswerable.
- FileLogger::log() now stamps server_id into the record at write time (guarded by
defined('SERVER_ID') for very-early failures), so each log line is self-describing.
- ErrorsCronJob::parseLog() reads that server_id and inserts it, falling back to the
local SERVER_ID for legacy files written before the field existed.
- server_id is now part of the dedup hash / `unique` key, so the same error from
LB and MAIN stays two distinct rows instead of INSERT IGNORE collapsing them into
one and losing the attribution.
The panel_logs table already has the server_id column; nothing schema-side to add.
LoopbackCommand mirrors to the daemon ingest socket (like LlodCommand), so an LB
restreaming from MAIN via php_loopback serves viewers through the daemon, not the
PHP byte path. Only a live-node e2e remains (canary LB currently unreachable).
Loopback is how an LB restreams a channel from its parent (MAIN) — it reads
MPEG-TS from <parent>/admin/live itself (no ffmpeg) and segments to disk in PHP.
Like LLOD v3, that left it OFF the daemon: an LB restreaming via loopback served
every viewer through the PHP byte path (worker-per-viewer on the LB), defeating
the fan-out on exactly the nodes P6/G targets.
Mirror the SANITIZED whole-packet buffer into the daemon's push-fed ingest socket
(registerIngest before the loop, best-effort non-blocking fwrite, unregister
after) — the same pattern as LlodCommand. We mirror the cleaned buffer, not the
raw read, because admin/live interleaves 0xFF padding that would break the
daemon's packet parsing. Once fed, isStreamFed() routes LB viewers to the daemon
(/live + in-RAM /hls) via the X-Accel locations just restored to lb_configs.
Daemon unreachable ⇒ null connect ⇒ legacy on-disk HLS only, no behaviour change.
Root cause of the LB 'no fanout locations' defect: the LB build overrides
nginx.conf with the stale lb_configs/nginx.conf (missing all X-Accel locations).
Fixed. libogg already in getPackages; daemon 0.8.0 released. Only a real-source
stream-to-LB e2e remains (external source/assignment).
The LB build overrides bin/nginx/conf/nginx.conf with lb_configs/nginx.conf
(Makefile: cp $(CONFIG_DIR)/nginx.conf ...), and that file had drifted stale —
it lacked ALL three streaming X-Accel internal locations that the main nginx.conf
carries. On a freshly-installed LB node this breaks:
- P1 HLS: segment.php always emits X-Accel-Redirect: /xc_hls/<file> -> 404
- P2 live fan-out: live.php emits /xc_fanout/<id> -> 404
- P2/B daemon HLS segments: /xc_fanout_hls/<id>_<seq> -> 404
i.e. daemon delivery (and even plain on-disk HLS) never worked on a stock LB —
matching the canary LB, whose nginx.conf 'predated' these and was patched by hand.
Add the three internal locations verbatim from the main nginx.conf so a stock
'make lb' node serves HLS + daemon fan-out out of the box. (libogg0 is already in
LbInstallFlow::getPackages for every distro; service starts the daemon keepalive +
fanout_sync on any node; fanout_binary/FanoutSync/RootSignals are not LB-stripped.)
Divarion_D