mirror of
https://github.com/Vateron-Media/XC_VM.git
synced 2026-10-27 11:03:28 +01:00
Claude
6c391f25d9
server:sync-openssl-extra now refuses to send when the main's own published fingerprint differs from the value it read. Run as root, it could read a config/openssl_extra that xc_vm cannot read, while the main's php-fpm runs on the default, and push a value that breaks every LB. The command and cron:root_signals now share one contract: OpensslExtra::SIGNAL_ACTION, signal() and applySignal(), with publish() for cron:servers. Encryption::readToken now tries the sealed format with both values before the legacy one. A wrong-value legacy decrypt() could return garbage (about 1 in 256), so a sealed token minted with the replaced value was refused. Legacy tokens keep that inherent miss rate. proxy_api trusts the posted server_id and handed any registered proxy an LB's signal rows, including the main's OPENSSL_EXTRA. It now skips set_openssl_extra rows until the source-IP binding lands. Tests cover sync() with TestDb, the signal contract and the case's own break. The docs cover the check and the repair. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
152 lines
11 KiB
Markdown
152 lines
11 KiB
Markdown
# Server Diagnostics (`server:diagnose`)
|
|
|
|
The panel marks a proxy/LB node **offline** purely from a stale heartbeat (`enabled` + `status = 1` + a recent `last_check_ago`), which never tells you *why* the node stopped reporting. The `server:diagnose` command answers that question.
|
|
|
|
```bash
|
|
/home/xc_vm/console.php server:diagnose [server_id]
|
|
```
|
|
|
|
The command is **read-only**: it only runs ping/curl/`fsockopen` probes, `SELECT` queries, and `sudo -n iptables -nL`. It never restarts or reconfigures anything.
|
|
|
|
Implemented by `src/Cli/Commands/ServerDiagnoseCommand.php`. The command ships in **both** MAIN and LB builds — the local mode is the whole point of having it on the node.
|
|
|
|
---
|
|
|
|
## Two Modes
|
|
|
|
The mode is auto-selected from where the command runs (`server_id` in `config.ini` → `is_main` in the `servers` table):
|
|
|
|
### Mode A — remote probe from the MAIN
|
|
|
|
```bash
|
|
sudo /home/xc_vm/console.php server:diagnose <server_id>
|
|
```
|
|
|
|
Probes the target node from the outside and reads its panel-side state:
|
|
|
|
| Check | What it tells you |
|
|
| --- | --- |
|
|
| Enabled / Status / Heartbeat | How the panel currently sees the node (`status = 4` means a failed install/provision) |
|
|
| ICMP ping | Is the host alive at all |
|
|
| TCP to `http_broadcast_port` | Is nginx reachable, or is the port dropped |
|
|
| HTTP `GET /api` | Does PHP actually answer behind nginx |
|
|
| Clock offset | `time_offset` vs the panel (skew > 30 s can flap the node online/offline) |
|
|
| Signal queue | Unconsumed rows in `signals` for this node (backlog > 120 s = the node's callback loop is stuck) |
|
|
| OPENSSL_EXTRA | Whether the node holds the main's `OPENSSL_EXTRA` (see [below](#repairing-an-openssl_extra-mismatch)). `unknown (node not updated)` / `unknown (main not updated)`: that side runs a build that does not publish it yet. Not checked for proxies |
|
|
|
|
The probe combinations map to causes:
|
|
|
|
- **No ICMP + port closed** — node powered off, network-partitioned, or fully firewalled.
|
|
- **ICMP replies but the port is dropped** — the classic: the node's own iptables blocked the main's IP (RootSignals flood/block false-positive), or nginx/the service is down. Run the local mode on the node for the exact cause.
|
|
- **Port open but `/api` silent** — nginx is up, PHP is not: check php-fpm on the node.
|
|
- **`/api` answers but the heartbeat is stale** — the node's watchdog daemon (the heartbeat writer) is not running, or it cannot write to the panel DB. Run the local mode on the node.
|
|
|
|
### Mode B — local self-diagnosis ON the node
|
|
|
|
```bash
|
|
sudo /home/xc_vm/console.php server:diagnose
|
|
```
|
|
|
|
Run this **on the silent LB/proxy node itself** — the causes usually live there. No argument needed; the node identifies itself from `config.ini`. Checks:
|
|
|
|
1. **My own panel row** — enabled/status/heartbeat as the main sees them.
|
|
2. **Can I reach the MAIN** — DB connectivity (implicitly proven), ICMP, and TCP to the main's broadcast port.
|
|
3. **Did I firewall the main?** — scans this node's `iptables INPUT` chain for a `DROP` of the main's IP, plus the flood-block marker file. This is the classic "node went silent for no reason" cause: the flood protection drops public IPs silently, and the main's callbacks stop landing.
|
|
4. **Service / nginx** — `systemctl is-active xc_vm` and a local TCP check on the node's own broadcast port.
|
|
5. **Watchdog daemon** — the actual heartbeat writer: the `watchdog` daemon updates `last_check_ago` every few seconds. When its MySQL connection to the main drops it **waits for the database to come back** (retrying every 5 s) and resumes the heartbeat immediately. Older builds exited instead — which made **all nodes go offline at the same moment** on any MySQL restart/blip on the main, until `cron:servers` revived them.
|
|
6. **Babysitter cron** — three sub-checks, because a dead watchdog stays dead only when the babysitter chain is broken:
|
|
- is `cron:servers` present in the **`xc_vm` user's crontab** (if missing, regenerate with `rm -f /home/xc_vm/tmp/crontab` and restart the service);
|
|
- is the system **cron service** active (no cron → the crontab never fires);
|
|
- is a previous `cron:servers` instance **hung on its cron lock** — a hung instance blocks every subsequent run for up to 30 minutes (`acquireCronLock` stale timeout), which is exactly how one DB blip keeps a node offline for half an hour. The command prints the holding PID and the kill command.
|
|
7. **Clock skew** — `time_offset` vs the panel.
|
|
8. **OPENSSL_EXTRA** — whether this node holds the main's value, as in Mode A.
|
|
|
|
> **Note:** the iptables check requires passwordless sudo (`sudo -n`). Without it the check reports `cannot check (need sudo iptables)` instead of failing — run the command as `root` for a full picture.
|
|
|
|
---
|
|
|
|
## Repairing an OPENSSL_EXTRA mismatch
|
|
|
|
`OPENSSL_EXTRA` (`config/openssl_extra`, or a built-in default when that file is absent) keys the stream tokens the main mints for the redirects an LB serves (`/auth`, `/vauth`, `/tsauth`, `/thauth`, subtitles). A main and its LBs must hold the same value; an LB that holds another one rejects all of those tokens, so playback redirected from the main fails there. The current installer gives a new main a random value, and LBs added by `server:install` before it shipped that file to them run on the default.
|
|
|
|
Each node's `cron:servers` publishes a fingerprint of its value (never the value itself) in `servers.server_hardware`, and the OPENSSL_EXTRA check compares the node's with the main's. To repair a mismatch, run on the **MAIN**:
|
|
|
|
```bash
|
|
sudo /home/xc_vm/console.php server:sync-openssl-extra <server_id>
|
|
sudo /home/xc_vm/console.php server:sync-openssl-extra --all
|
|
```
|
|
|
|
- The command queues one root signal with the main's value for each streaming LB that reports another fingerprint. The LB's `cron:root_signals` applies it within a minute: it writes `config/openssl_extra` (0600, owned by `xc_vm`) and php-fpm uses it from the next request, with no restart. The value stays in the `signals` table until then.
|
|
- For **10 minutes** afterwards the LB still accepts tokens it minted with its old value (kept in `config/openssl_extra.prev`), so links it has just handed out keep playing. Legacy-format tokens (`secure_stream_tokens` off) can still be refused in that window, about 1 in 256.
|
|
- LBs already in sync, LBs that publish no fingerprint (not updated yet) and offline LBs are skipped. `--force` queues them anyway; an offline LB applies the signal if it comes back within a day (queued signals expire after 24 hours). Proxies and the main are never sent the value.
|
|
- When the main has **no** `config/openssl_extra`, it runs on the built-in default and there is nothing to send. An LB that still reports a mismatch has a stale `config/openssl_extra` of its own: delete it on that LB (`sudo rm -f /home/xc_vm/config/openssl_extra`); the change applies at the next request.
|
|
- The command sends nothing when the main publishes another fingerprint than its `config/openssl_extra` holds: either `xc_vm` cannot read that file, so the main's php-fpm runs on the default (`sudo chown xc_vm:xc_vm /home/xc_vm/config/openssl_extra && sudo chmod 600 /home/xc_vm/config/openssl_extra`), or the file changed within the last minute (wait for `cron:servers`). It also sends nothing when the main has not published a fingerprint yet.
|
|
|
|
Check the result with `server:diagnose <server_id>` a minute or two later, after the LB's next `cron:servers` run.
|
|
|
|
---
|
|
|
|
## Output & Exit Codes
|
|
|
|
Each check prints one aligned `[OK]`/`[WARN]` line, followed by a numbered **Probable cause(s)** summary with the exact fix command where one exists (e.g. the `iptables -D INPUT ... -j DROP` unblock line).
|
|
|
|
| Exit code | Meaning |
|
|
| --- | --- |
|
|
| `0` | No obvious cause found (or the target is the MAIN itself — nothing to diagnose) |
|
|
| `1` | Usage error: missing/unknown `server_id` |
|
|
| `2` | One or more probable causes were found and printed |
|
|
|
|
Example (local mode, self-blocked main):
|
|
|
|
```text
|
|
Self-diagnosis on node #3 — LB-Frankfurt (type 1)
|
|
----------------------------------------------------------------
|
|
[OK] Enabled yes
|
|
[OK] Status 1 (online)
|
|
[WARN] Heartbeat last check-in 641s ago (limit 180s)
|
|
[OK] DB → main reachable (this query ran)
|
|
[OK] Ping main reply (203.0.113.10)
|
|
[WARN] Main :8080 closed/timeout
|
|
[WARN] Main in iptables DROP present (+flood marker)
|
|
[OK] Service xc_vm active
|
|
[OK] nginx :8080 listening
|
|
[OK] watchdog daemon running
|
|
[OK] cron:servers in xc_vm crontab
|
|
[OK] Clock offset 2s vs panel
|
|
----------------------------------------------------------------
|
|
Probable cause(s):
|
|
1. Heartbeat is stale (641s > 180s): the node stopped reporting — the checks below narrow down why.
|
|
2. This node has DROPPED the main's IP 203.0.113.10 in its own iptables (flood/block false-positive). Unblock: `sudo iptables -D INPUT -s 203.0.113.10 -j DROP && sudo rm -f /home/xc_vm/tmp/flood/block_203.0.113.10`.
|
|
```
|
|
|
|
---
|
|
|
|
## Quick Reference — Common Causes
|
|
|
|
| Symptom | Likely cause | Fix |
|
|
| --- | --- | --- |
|
|
| Pings, port dropped | Node's iptables blocked the main's IP | `sudo iptables -D INPUT -s <main_ip> -j DROP` + remove the `block_<ip>` marker (the command prints the exact line) |
|
|
| No ping, no port | Host down / network partition / external firewall | Check hosting console, routes, provider firewall |
|
|
| Port open, `/api` dead | php-fpm down | Restart the `xc_vm` service on the node |
|
|
| `/api` fine, heartbeat stale | Watchdog daemon dead (exits on DB connection loss) | `sudo -u xc_vm console.php watchdog` on the node; check DB grants (`tools mysql` on the main) |
|
|
| **All nodes drop at the same moment** | A MySQL restart/blip on the main hit every node's watchdog at once (fatal for pre-fix builds; current builds wait it out) | Check the main's MySQL error log around the drop time; update the nodes so the watchdog survives outages |
|
|
| Node flaps online/offline | Clock skew > 30 s (or recurring MySQL blips) | Sync NTP on the node; check MySQL stability on the main |
|
|
| Status = 4 | Install/provision errored | Re-run `server:install` from the main |
|
|
| Playback redirected from the main fails on one LB | `OPENSSL_EXTRA` mismatch (the check reports it) | `sudo /home/xc_vm/console.php server:sync-openssl-extra <server_id>` on the main (see [above](#repairing-an-openssl_extra-mismatch)) |
|
|
|
|
---
|
|
|
|
## Related files
|
|
|
|
| File | Role |
|
|
| --- | --- |
|
|
| `src/Cli/Commands/ServerDiagnoseCommand.php` | The diagnostic command (both modes) |
|
|
| `src/Cli/Commands/WatchdogCommand.php` | The watchdog daemon — writes the heartbeat (`last_check_ago`) |
|
|
| `src/Cli/CronJobs/ServersCronJob.php` | Babysitter cron — relaunches a dead watchdog |
|
|
| `src/Cli/CronJobs/RootSignalsCronJob.php` | Applies iptables blocks (the false-positive source) and the `OPENSSL_EXTRA` repair signal |
|
|
| `src/Cli/Commands/ServerSyncOpensslExtraCommand.php` | `server:sync-openssl-extra` — queues the main's `OPENSSL_EXTRA` for mismatched LBs (MAIN only) |
|
|
| `src/Core/Config/OpensslExtra.php` | `OPENSSL_EXTRA` fingerprint, repair signal and the 10-minute fallback |
|
|
| `src/Domain/Server/ServerRepository.php` | `servers` table access |
|
|
|
|
See also: [CLI Tools](../guides/cli-tools.md), [Updating a Server](../administration/server-update.md).
|