Files
XC_VM/tests/Unit/CounterRateSamplerTest.php
T
Claude f8f3ca3a82 fix(watchdog): correct bandwidth totals, rps, producers and fatals
SystemInfo::getStats() feeds the watchdog heartbeat (servers.watchdog_data),
servers_stats and the LB /api stats action, and had four defects:

- bytes_sent_total/bytes_received_total were assigned per interface, so
  only the last selected interface was kept, and the received total read
  tx_bytes. aggregateNetwork() now sums both totals over the same
  interfaces as the per-second rates and reads rx_bytes for received.
  network_speed keeps its first-positive rule and the interface selection
  is unchanged, so routing capacity is not affected.
- total_running_streams was `ps ax | grep -c ffmpeg`, which counted
  ffprobe and shell wrappers and missed the fanout daemon's native
  remuxer. ProcessManager::countStreamProducers() now reads /proc and
  counts processes whose program is ffmpeg or `xc_fanout remux`.
- The memory percentage divided by total_mem, which getMemory() reports
  as 0 without MemAvailable (DivisionByZeroError), and nvidia-smi XML
  with a single GPU process decodes process_info as one associative
  array (TypeError). Either aborted the watchdog pass, so the node wrote
  no heartbeat. memUsedPercent() and gpuProcesses() guard both.

The watchdog runs one pass per process and re-execs itself, so the
previous nginx request count held in a variable was always null and
requests_per_second was 0 on every node. CounterRateSampler keeps the
previous reading in a tmp state file (atomic replace, 120 s max age, 0
after a counter reset).

The watchdog_data key set and order are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
2026-09-25 00:10:15 +00:00

99 lines
3.6 KiB
PHP

<?php
use PHPUnit\Framework\TestCase;
use XcVm\Core\Util\CounterRateSampler;
/**
* CounterRateSampler — the per-second rate of a monotonic counter (nginx's
* total request count) across processes. The watchdog runs one pass per
* process and re-execs itself, so a previous reading held in a local variable
* was always null and servers.requests_per_second was 0 on every node. The
* sampler keeps the previous reading in a state file instead.
*/
final class CounterRateSamplerTest extends TestCase {
private string $rFile;
protected function setUp(): void {
$this->rFile = sys_get_temp_dir() . '/xcvm_rate_' . uniqid('', true) . '.json';
}
protected function tearDown(): void {
foreach (glob($this->rFile . '*') ?: [] as $rPath) {
@unlink($rPath);
}
}
public function testTheFirstSampleIsZeroAndStoresTheReading(): void {
$this->assertSame(0, (new CounterRateSampler($this->rFile))->sample(1000, 100));
$this->assertFileExists($this->rFile);
}
public function testRateIsTheCounterDeltaPerSecondAcrossInstances(): void {
(new CounterRateSampler($this->rFile))->sample(1000, 100);
// A fresh instance, as a re-exec'd watchdog pass would build.
$this->assertSame(100, (new CounterRateSampler($this->rFile))->sample(1500, 105));
$this->assertSame(33, (new CounterRateSampler($this->rFile))->sample(1600, 108));
}
public function testACounterThatWentDownIsZeroAndStillStored(): void {
$rSampler = new CounterRateSampler($this->rFile);
$rSampler->sample(5000, 100);
// nginx restarted: its counter starts over.
$this->assertSame(0, $rSampler->sample(40, 104));
// The next reading is measured from the stored post-restart value.
$this->assertSame(10, $rSampler->sample(80, 108));
}
public function testTheSameTimestampIsZero(): void {
$rSampler = new CounterRateSampler($this->rFile);
$rSampler->sample(1000, 100);
$this->assertSame(0, $rSampler->sample(1500, 100));
$this->assertSame(0, $rSampler->sample(2000, 99), 'a clock that stepped back');
}
public function testAReadingOlderThanMaxAgeIsZero(): void {
$rSampler = new CounterRateSampler($this->rFile, 120);
$rSampler->sample(1000, 100);
$this->assertSame(0, $rSampler->sample(100000, 221));
$this->assertSame(10, $rSampler->sample(101200, 341), '120 s is still in range');
}
public function testTheDefaultMaxAgeIs120Seconds(): void {
$rSampler = new CounterRateSampler($this->rFile);
$rSampler->sample(1000, 100);
$this->assertSame(0, $rSampler->sample(2210, 221));
}
public function testACorruptStateFileIsZeroAndOverwritten(): void {
file_put_contents($this->rFile, 'not json {');
$rSampler = new CounterRateSampler($this->rFile);
$this->assertSame(0, $rSampler->sample(1000, 100));
$this->assertSame(100, $rSampler->sample(1500, 105));
file_put_contents($this->rFile, '{"v":"x","t":null}');
$this->assertSame(0, $rSampler->sample(2000, 110));
}
public function testAnUnwritableStateFileIsZeroWithoutAWarning(): void {
$rSampler = new CounterRateSampler(sys_get_temp_dir() . '/xcvm_missing_' . uniqid('', true) . '/state.json');
$this->assertSame(0, $rSampler->sample(1000, 100));
$this->assertSame(0, $rSampler->sample(1500, 105));
}
public function testWatchdogSamplesNginxRequestsThroughTheStateFile(): void {
$rPath = MAIN_HOME . 'Cli/Commands/WatchdogCommand.php';
$this->assertFileExists($rPath);
$rSource = (string) file_get_contents($rPath);
$this->assertStringContainsString('new CounterRateSampler(TMP_PATH . ', $rSource);
$this->assertStringNotContainsString('$rLastRequests', $rSource, 'a per-process previous reading is always null');
}
}