* fix(playback): route v3 direct play and remux through proxy nodes
Protocol v3 consulted the node planner only for the HLS deliveries, so
`original_http` and `server_remux_progressive` sessions returned an
API-local `/stream/{session_id}` URL and the API node served the bytes —
ServeDirectPlay for direct play, a locally spawned ffmpeg for the remux.
An operator running dedicated proxy nodes still saw all of that egress on
the API node.
The capability already existed: the proxy implements /stream/direct and
/stream/remux, and the Jellyfin-compat transport already plans a proxy for
exactly these two methods. Native v3 was the only surface skipping it, so
Jellyfin clients routed correctly on a deployment where Silo's own clients
did not. This wires the same shape into the v3 identity transport rather
than inventing a second selection path.
The proxy serves from the stream token alone, so the token now carries the
media path, the file's Dolby Vision profile (a P7 remux must strip the
dangling RPU) and the audio-only flag (which picks audio/mp4 over
video/mp4, the MIME the plan promised). RecipeCard models none of the
three; a missing claim would not fail loudly, it would serve a subtly
different stream than the plan promised.
Two related fixes:
- Proxy direct play served via http.ServeFile, which sets no strong ETag.
direct_stream_resume_v1 depends on the ETag ServeDirectPlay sets before
ServeContent, so routing direct play to a proxy without this would have
silently broken resumable direct streams: If-Range never validates and a
resumed range restarts at 200. The proxy now uses the same serve path.
- playback.local_transcode_fallback was only checked in the HLS branch, so
a progressive remux that converts audio still spawned ffmpeg locally on
an API-only node with the setting disabled. Identity deliveries now
honor the gate too — direct play still falls back locally, since moving
bytes is not transcode work and single-node deployments must keep
working.
Falling back to the API-local path when no proxy is eligible preserves
single-node behavior, and a planner reservation is released whenever the
session does not actually reach a proxy.
Closes #619
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(playback): validate proxy recipes and keep proxy sessions alive
Addresses three P1 findings on the proxy-transport change.
Proxies do run ffmpeg — /stream/remux converts audio and strips Dolby
Vision RPUs — but they exposed no capability endpoint, so unlike the HLS
offload path nothing checked that the selected proxy could execute the
transformations a plan froze. A pool whose proxies carry a different
ffmpeg build (rolling upgrade, custom image) would fail at stream time: a
missing aac encoder 500s, a missing dovi_rpu filter is refused outright by
the remux itself. Proxies now serve /hw-capabilities in the same shape and
at the same path as a transcode node, and identity planning validates the
frozen recipe against the selected proxy, falling back to a node that can
do the work. A proxy that does not answer is treated as incapable rather
than assumed good: an older proxy predating the endpoint is exactly the
mismatched build the check exists to catch. Direct play copies bytes and
needs no recipe, so it skips the probe entirely.
meteredResponseWriter implemented neither Unwrap nor SetWriteDeadline, so
RollingDeadlineWriter could not install its stall deadline on any proxy
stream. With the standalone proxy running WriteTimeout 0 there was no
server-level guard behind it, so a client that stopped reading without
closing its connection would block a write forever, holding the session,
the file, the goroutine and the connection.
A proxy-served session never produces a transport request on the API node,
so activeTransportCount — what protects a local stream from the idle
reaper — stays zero and a heartbeat gap longer than the active grace would
reap a healthy stream, after which progress, stop and replan all fail with
session-not-found while bytes still flow. Sessions are now marked as
remotely transported, which widens their idle windows rather than granting
immunity: this manager has no absolute session lifetime, so unconditional
immunity would leak a session forever when a client disappears without
stopping. The mark is always set on commit, so a re-plan that moves a
session back onto the API clears a stale one.
Also adopts the exported transformation constants in the tests and covers
the effective-recipe bitrate branch, per review.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(playback): pick capable sibling proxies and refresh transport locality
Narrow proxy selection by capability *before* selection rather than
rejecting a single round-robin pick afterwards. Abandoning the pool on one
mismatch meant a capable proxy with free capacity sat unused while
playback either ran ffmpeg on the API node or, with
playback.local_transcode_fallback disabled, was refused outright — the
exact api/proxy split this branch targets, during exactly the rolling
ffmpeg upgrade the capability check exists for. PlanSessionWith now
applies its eligibility predicate to the proxy on proxy-only plans (the
proxy is the executor there), mirroring how HLS filters transcode nodes,
and the planner grows ProxyNodeURLs to match TranscodeNodeURLs. Direct
play still skips the probe: it copies bytes and needs no recipe.
Every committed route now records transport locality, not just the
identity-proxy one. A session replanned from a proxy onto the integrated
transcoder previously kept a stale remote-transport mark, and the widened
idle grace it grants would hold that session's stream and transcode slots
for five minutes after the local stream disconnected without an explicit
stop. The remote HLS route sets it too — it also hands the client an
absolute proxy URL that never reaches this server.
The proxy's CORS config exposed no response headers, so cross-origin
JavaScript could send the If-Range/Range request headers it already allows
but never read the ETag, Accept-Ranges or Content-Range needed to build
them. direct_stream_resume_v1 silently degraded to a full restart whenever
the proxy was on a different origin than the web app, which is the normal
deployment.
Also regenerates internal/playback/testdata/protocol_v3 and the schema
fixtures, which were stale for output_change_v1 since #613/#617 and failed
CI on every branch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(playback): restore the alternate-version fallback for burn-in refusals
#617 renamed the terminal a burn-in-forced adaptation reports: when the
subtitle burn requirement is the sole trigger, an HDR source that cannot
be re-encoded now returns subtitle_conversion_unsupported instead of
hdr_transcode_unsupported, so the refusal names the thing the viewer can
actually act on.
terminalAllowsAlternateFileV3 was not updated to match, and it gates the
alternate-version retry on the old reason strings. That silently retired
the fallback for exactly the case its own comment describes — a bitmap
subtitle needing burn-in that an HDR source cannot support while an SDR
alternate can. Playback was refused outright instead of switching to the
version that can serve it.
Adds the new reason to the gate and covers it directly, so a future
rename of a refusal reason fails on the gate rather than only on the
end-to-end replan test.
Also drops debug instrumentation that was committed by mistake in
TestHandleReplanPlaybackV3BitmapSubtitleFallsBackFromHDRToSDRVersion; the
assertion is back to its original form and now passes on the merits.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
674 lines
24 KiB
Go
674 lines
24 KiB
Go
package nodepool
|
|
|
|
import (
|
|
"strconv"
|
|
"testing"
|
|
"time"
|
|
)
|
|
|
|
func strPtr(s string) *string { return &s }
|
|
func intPtr(i int) *int { return &i }
|
|
|
|
type plannerFixture struct {
|
|
planner *Planner
|
|
proxies *ProxyPool
|
|
transcodes *TranscodePool
|
|
now time.Time
|
|
}
|
|
|
|
func newFixture(proxies, transcodes []*Node) *plannerFixture {
|
|
pp := NewProxyPool()
|
|
pp.SetNodes(proxies)
|
|
tp := NewTranscodePool()
|
|
tp.SetNodes(transcodes)
|
|
f := &plannerFixture{
|
|
planner: NewPlanner(pp, tp),
|
|
proxies: pp,
|
|
transcodes: tp,
|
|
now: time.Date(2026, 6, 10, 12, 0, 0, 0, time.UTC),
|
|
}
|
|
f.planner.now = func() time.Time { return f.now }
|
|
return f
|
|
}
|
|
|
|
func proxyNode(id int, url string, group *string) *Node {
|
|
return &Node{ID: id, Name: url, Type: NodeTypeProxy, URL: url, Enabled: true, Healthy: true, Group: group}
|
|
}
|
|
|
|
func transcodeNode(id int, url string, group *string, activeJobs int) *Node {
|
|
return &Node{ID: id, Name: url, Type: NodeTypeTranscode, URL: url, Enabled: true, Healthy: true, Group: group, ActiveJobs: activeJobs}
|
|
}
|
|
|
|
func TestPlanTranscodePairsProxyFromSameGroup(t *testing.T) {
|
|
f := newFixture(
|
|
[]*Node{
|
|
proxyNode(1, "http://proxy-a", strPtr("rack-a")),
|
|
proxyNode(2, "http://proxy-b", strPtr("rack-b")),
|
|
},
|
|
[]*Node{
|
|
transcodeNode(3, "http://tc-a", strPtr("rack-a"), 5),
|
|
transcodeNode(4, "http://tc-b", strPtr("rack-b"), 0),
|
|
},
|
|
)
|
|
|
|
plan := f.planner.PlanSession("s1", "", true, 0)
|
|
if plan.TranscodeNode == nil || plan.TranscodeNode.URL != "http://tc-b" {
|
|
t.Fatalf("expected least-loaded tc-b, got %+v", plan.TranscodeNode)
|
|
}
|
|
if plan.ProxyNode == nil || plan.ProxyNode.URL != "http://proxy-b" {
|
|
t.Fatalf("expected same-group proxy-b, got %+v", plan.ProxyNode)
|
|
}
|
|
}
|
|
|
|
func TestPlanSessionWithRestrictsEligibleTranscodeNodes(t *testing.T) {
|
|
f := newFixture(nil, []*Node{
|
|
transcodeNode(1, "http://tc-a", nil, 0),
|
|
transcodeNode(2, "http://tc-b", nil, 5),
|
|
})
|
|
eligible := func(n *Node) bool { return n != nil && n.URL == "http://tc-b" }
|
|
|
|
plan := f.planner.PlanSessionWith("s1", "", true, 0, eligible)
|
|
if plan.TranscodeNode == nil || plan.TranscodeNode.URL != "http://tc-b" {
|
|
t.Fatalf("expected the eligible node despite its higher load, got %+v", plan.TranscodeNode)
|
|
}
|
|
if none := f.planner.PlanSessionWith("s2", "", true, 0, func(*Node) bool { return false }); none.TranscodeNode != nil {
|
|
t.Fatalf("no eligible node must select nothing, got %+v", none.TranscodeNode)
|
|
}
|
|
// Soft affinity to the session's current node must not survive the
|
|
// current node becoming ineligible.
|
|
if sticky := f.planner.PlanSessionWith("s3", "http://tc-a", true, 0, eligible); sticky.TranscodeNode == nil || sticky.TranscodeNode.URL != "http://tc-b" {
|
|
t.Fatalf("affinity to an ineligible node must yield to an eligible one, got %+v", sticky.TranscodeNode)
|
|
}
|
|
if unrestricted := f.planner.PlanSessionWith("s4", "", true, 0, nil); unrestricted.TranscodeNode == nil || unrestricted.TranscodeNode.URL != "http://tc-a" {
|
|
t.Fatalf("nil predicate must behave like PlanSession, got %+v", unrestricted.TranscodeNode)
|
|
}
|
|
}
|
|
|
|
func TestReleaseSessionDropsProvisionalReservation(t *testing.T) {
|
|
node := transcodeNode(1, "http://tc-1", nil, 0)
|
|
node.MaxJobs = intPtr(1)
|
|
f := newFixture(nil, []*Node{node})
|
|
if got := f.planner.PlanSession("s1", "", true, 0).TranscodeNode; got == nil {
|
|
t.Fatal("first session was not reserved")
|
|
}
|
|
if got := f.planner.PlanSession("s2", "", true, 0).TranscodeNode; got != nil {
|
|
t.Fatalf("second session bypassed reservation: %+v", got)
|
|
}
|
|
f.planner.ReleaseSession("s1")
|
|
if got := f.planner.PlanSession("s2", "", true, 0).TranscodeNode; got == nil {
|
|
t.Fatal("released reservation still blocked the node")
|
|
}
|
|
}
|
|
|
|
func TestDegradedGroupExcludesItsTranscodeNodes(t *testing.T) {
|
|
unhealthyProxy := proxyNode(1, "http://proxy-a", strPtr("rack-a"))
|
|
unhealthyProxy.Healthy = false
|
|
f := newFixture(
|
|
[]*Node{
|
|
unhealthyProxy,
|
|
proxyNode(2, "http://proxy-b", strPtr("rack-b")),
|
|
},
|
|
[]*Node{
|
|
transcodeNode(3, "http://tc-a", strPtr("rack-a"), 0), // idle but group degraded
|
|
transcodeNode(4, "http://tc-b", strPtr("rack-b"), 9),
|
|
},
|
|
)
|
|
|
|
plan := f.planner.PlanSession("s1", "", true, 0)
|
|
if plan.TranscodeNode == nil || plan.TranscodeNode.URL != "http://tc-b" {
|
|
t.Fatalf("expected tc-b (rack-a degraded), got %+v", plan.TranscodeNode)
|
|
}
|
|
if plan.ProxyNode == nil || plan.ProxyNode.URL != "http://proxy-b" {
|
|
t.Fatalf("expected proxy-b, got %+v", plan.ProxyNode)
|
|
}
|
|
}
|
|
|
|
func TestUnhealthyTranscodeMemberDegradesGroup(t *testing.T) {
|
|
deadTC := transcodeNode(5, "http://tc-a2", strPtr("rack-a"), 0)
|
|
deadTC.Healthy = false
|
|
f := newFixture(
|
|
[]*Node{proxyNode(1, "http://proxy-a", strPtr("rack-a"))},
|
|
[]*Node{
|
|
transcodeNode(3, "http://tc-a1", strPtr("rack-a"), 0),
|
|
deadTC,
|
|
},
|
|
)
|
|
|
|
// All enabled members of a group must be healthy for the group to be
|
|
// eligible — even the healthy sibling is excluded.
|
|
plan := f.planner.PlanSession("s1", "", true, 0)
|
|
if plan.TranscodeNode != nil {
|
|
t.Fatalf("expected no transcode node, got %+v", plan.TranscodeNode)
|
|
}
|
|
}
|
|
|
|
func TestUngroupedNodesKeepLegacyBehavior(t *testing.T) {
|
|
f := newFixture(
|
|
[]*Node{
|
|
proxyNode(1, "http://proxy-1", nil),
|
|
proxyNode(2, "http://proxy-2", nil),
|
|
},
|
|
[]*Node{
|
|
transcodeNode(3, "http://tc-1", nil, 2),
|
|
transcodeNode(4, "http://tc-2", nil, 1),
|
|
},
|
|
)
|
|
|
|
plan := f.planner.PlanSession("s1", "", true, 0)
|
|
if plan.TranscodeNode == nil || plan.TranscodeNode.URL != "http://tc-2" {
|
|
t.Fatalf("expected least-connections tc-2, got %+v", plan.TranscodeNode)
|
|
}
|
|
if plan.ProxyNode == nil {
|
|
t.Fatal("expected a proxy node")
|
|
}
|
|
|
|
// Round-robin across both proxies for subsequent sessions.
|
|
first := plan.ProxyNode.URL
|
|
second := f.planner.PlanSession("s2", "", true, 0).ProxyNode.URL
|
|
if first == second {
|
|
t.Fatalf("expected round-robin to alternate proxies, got %s twice", first)
|
|
}
|
|
}
|
|
|
|
func TestGroupWithoutProxiesFallsBackToGlobalProxy(t *testing.T) {
|
|
f := newFixture(
|
|
[]*Node{proxyNode(1, "http://proxy-1", nil)},
|
|
[]*Node{transcodeNode(2, "http://tc-a", strPtr("rack-a"), 0)},
|
|
)
|
|
|
|
plan := f.planner.PlanSession("s1", "", true, 0)
|
|
if plan.TranscodeNode == nil || plan.TranscodeNode.URL != "http://tc-a" {
|
|
t.Fatalf("expected tc-a, got %+v", plan.TranscodeNode)
|
|
}
|
|
if plan.ProxyNode == nil || plan.ProxyNode.URL != "http://proxy-1" {
|
|
t.Fatalf("expected global proxy fallback, got %+v", plan.ProxyNode)
|
|
}
|
|
}
|
|
|
|
func TestSoftAffinityKeepsCurrentNode(t *testing.T) {
|
|
f := newFixture(
|
|
[]*Node{proxyNode(1, "http://proxy-1", nil)},
|
|
[]*Node{
|
|
transcodeNode(2, "http://tc-1", nil, 2),
|
|
transcodeNode(3, "http://tc-2", nil, 1),
|
|
},
|
|
)
|
|
|
|
// Difference of 1 job: stay on current.
|
|
plan := f.planner.PlanSession("s1", "http://tc-1", true, 0)
|
|
if plan.TranscodeNode == nil || plan.TranscodeNode.URL != "http://tc-1" {
|
|
t.Fatalf("expected soft affinity to keep tc-1, got %+v", plan.TranscodeNode)
|
|
}
|
|
|
|
// Difference of 2+: switch to the less-loaded node.
|
|
f.transcodes.Nodes()[0].ActiveJobs = 4
|
|
plan = f.planner.PlanSession("s1", "http://tc-1", true, 0)
|
|
if plan.TranscodeNode == nil || plan.TranscodeNode.URL != "http://tc-2" {
|
|
t.Fatalf("expected switch to tc-2, got %+v", plan.TranscodeNode)
|
|
}
|
|
}
|
|
|
|
func TestTranscodeCapSkipsFullNode(t *testing.T) {
|
|
capped := transcodeNode(2, "http://tc-1", nil, 3)
|
|
capped.MaxJobs = intPtr(3)
|
|
f := newFixture(
|
|
[]*Node{proxyNode(1, "http://proxy-1", nil)},
|
|
[]*Node{
|
|
capped,
|
|
transcodeNode(3, "http://tc-2", nil, 5),
|
|
},
|
|
)
|
|
|
|
plan := f.planner.PlanSession("s1", "", true, 0)
|
|
if plan.TranscodeNode == nil || plan.TranscodeNode.URL != "http://tc-2" {
|
|
t.Fatalf("expected at-cap tc-1 to be skipped, got %+v", plan.TranscodeNode)
|
|
}
|
|
|
|
// All nodes at cap: no transcode node.
|
|
f.transcodes.Nodes()[1].MaxJobs = intPtr(5)
|
|
plan = f.planner.PlanSession("s2", "", true, 0)
|
|
if plan.TranscodeNode != nil {
|
|
t.Fatalf("expected no eligible node, got %+v", plan.TranscodeNode)
|
|
}
|
|
}
|
|
|
|
func TestProxyCapSkipsFullProxy(t *testing.T) {
|
|
capped := proxyNode(1, "http://proxy-1", nil)
|
|
capped.MaxJobs = intPtr(2)
|
|
capped.ActiveJobs = 2
|
|
f := newFixture(
|
|
[]*Node{capped, proxyNode(2, "http://proxy-2", nil)},
|
|
[]*Node{},
|
|
)
|
|
|
|
for i := 0; i < 3; i++ {
|
|
plan := f.planner.PlanSession("s", "", false, 0)
|
|
if plan.ProxyNode == nil || plan.ProxyNode.URL != "http://proxy-2" {
|
|
t.Fatalf("expected proxy-2 (proxy-1 at cap), got %+v", plan.ProxyNode)
|
|
}
|
|
}
|
|
}
|
|
|
|
func TestGroupAtProxyCapacityExcludesGroupTranscode(t *testing.T) {
|
|
groupProxy := proxyNode(1, "http://proxy-a", strPtr("rack-a"))
|
|
groupProxy.MaxJobs = intPtr(1)
|
|
groupProxy.ActiveJobs = 1
|
|
f := newFixture(
|
|
[]*Node{groupProxy, proxyNode(2, "http://proxy-1", nil)},
|
|
[]*Node{
|
|
transcodeNode(3, "http://tc-a", strPtr("rack-a"), 0),
|
|
transcodeNode(4, "http://tc-1", nil, 7),
|
|
},
|
|
)
|
|
|
|
// rack-a's only proxy is full, so its transcode node must not be used —
|
|
// streams pinned to rack-a would have nowhere to go.
|
|
plan := f.planner.PlanSession("s1", "", true, 0)
|
|
if plan.TranscodeNode == nil || plan.TranscodeNode.URL != "http://tc-1" {
|
|
t.Fatalf("expected ungrouped tc-1, got %+v", plan.TranscodeNode)
|
|
}
|
|
if plan.ProxyNode == nil || plan.ProxyNode.URL != "http://proxy-1" {
|
|
t.Fatalf("expected ungrouped proxy-1, got %+v", plan.ProxyNode)
|
|
}
|
|
}
|
|
|
|
func TestGroupProxyReservationsGateGroupCapacity(t *testing.T) {
|
|
groupProxy := proxyNode(1, "http://proxy-a", strPtr("rack-a"))
|
|
groupProxy.MaxJobs = intPtr(1)
|
|
f := newFixture(
|
|
[]*Node{groupProxy},
|
|
[]*Node{transcodeNode(2, "http://tc-a", strPtr("rack-a"), 0)},
|
|
)
|
|
|
|
// The first session reserves the group's only proxy slot.
|
|
plan := f.planner.PlanSession("s1", "", true, 0)
|
|
if plan.TranscodeNode == nil || plan.ProxyNode == nil {
|
|
t.Fatalf("first session should get both nodes, got %+v", plan)
|
|
}
|
|
|
|
// With the group's proxy fully reserved, its transcode node is
|
|
// ineligible too — streams pinned to the group would have nowhere to go.
|
|
plan = f.planner.PlanSession("s2", "", true, 0)
|
|
if plan.TranscodeNode != nil || plan.ProxyNode != nil {
|
|
t.Fatalf("second session should be rejected, got %+v", plan)
|
|
}
|
|
}
|
|
|
|
func TestReservationsCountTowardCaps(t *testing.T) {
|
|
capped := transcodeNode(2, "http://tc-1", nil, 0)
|
|
capped.MaxJobs = intPtr(2)
|
|
lastCheck := time.Date(2026, 6, 10, 11, 59, 0, 0, time.UTC)
|
|
capped.LastHealthCheck = &lastCheck
|
|
f := newFixture(
|
|
[]*Node{proxyNode(1, "http://proxy-1", nil)},
|
|
[]*Node{capped},
|
|
)
|
|
|
|
// Two sessions fill the cap via reservations before any health refresh.
|
|
if f.planner.PlanSession("s1", "", true, 0).TranscodeNode == nil {
|
|
t.Fatal("first session should be admitted")
|
|
}
|
|
if f.planner.PlanSession("s2", "", true, 0).TranscodeNode == nil {
|
|
t.Fatal("second session should be admitted")
|
|
}
|
|
if got := f.planner.PlanSession("s3", "", true, 0).TranscodeNode; got != nil {
|
|
t.Fatalf("third session should be rejected, got %+v", got)
|
|
}
|
|
|
|
// Re-planning an admitted session must not double-count it.
|
|
if f.planner.PlanSession("s2", "http://tc-1", true, 0).TranscodeNode == nil {
|
|
t.Fatal("re-plan of s2 should be admitted")
|
|
}
|
|
|
|
// A health report newer than the reservations becomes authoritative:
|
|
// the node now says 1 job, so one slot is free again.
|
|
newer := f.now.Add(10 * time.Second)
|
|
capped.LastHealthCheck = &newer
|
|
capped.ActiveJobs = 1
|
|
f.now = f.now.Add(20 * time.Second)
|
|
if f.planner.PlanSession("s4", "", true, 0).TranscodeNode == nil {
|
|
t.Fatal("session should be admitted after fresh health report")
|
|
}
|
|
}
|
|
|
|
func TestReservationsExpire(t *testing.T) {
|
|
capped := transcodeNode(2, "http://tc-1", nil, 0)
|
|
capped.MaxJobs = intPtr(1)
|
|
f := newFixture(
|
|
[]*Node{proxyNode(1, "http://proxy-1", nil)},
|
|
[]*Node{capped},
|
|
)
|
|
|
|
if f.planner.PlanSession("s1", "", true, 0).TranscodeNode == nil {
|
|
t.Fatal("first session should be admitted")
|
|
}
|
|
if got := f.planner.PlanSession("s2", "", true, 0).TranscodeNode; got != nil {
|
|
t.Fatalf("second session should be rejected, got %+v", got)
|
|
}
|
|
|
|
// Without health reports (LastHealthCheck nil) reservations still expire
|
|
// after maxReservationAge so a stalled health checker can't wedge admission.
|
|
f.now = f.now.Add(maxReservationAge + time.Second)
|
|
if f.planner.PlanSession("s3", "", true, 0).TranscodeNode == nil {
|
|
t.Fatal("session should be admitted after reservation expiry")
|
|
}
|
|
}
|
|
|
|
func TestDirectPlayIgnoresGroups(t *testing.T) {
|
|
f := newFixture(
|
|
[]*Node{proxyNode(1, "http://proxy-a", strPtr("rack-a"))},
|
|
[]*Node{},
|
|
)
|
|
|
|
plan := f.planner.PlanSession("s1", "", false, 0)
|
|
if plan.ProxyNode == nil || plan.ProxyNode.URL != "http://proxy-a" {
|
|
t.Fatalf("expected grouped proxy to serve direct play, got %+v", plan.ProxyNode)
|
|
}
|
|
if plan.TranscodeNode != nil {
|
|
t.Fatalf("direct play must not pick a transcode node, got %+v", plan.TranscodeNode)
|
|
}
|
|
}
|
|
|
|
func TestGroupRoundRobinAcrossGroupProxies(t *testing.T) {
|
|
f := newFixture(
|
|
[]*Node{
|
|
proxyNode(1, "http://proxy-a1", strPtr("rack-a")),
|
|
proxyNode(2, "http://proxy-a2", strPtr("rack-a")),
|
|
},
|
|
[]*Node{transcodeNode(3, "http://tc-a", strPtr("rack-a"), 0)},
|
|
)
|
|
|
|
seen := map[string]bool{}
|
|
for i, id := range []string{"s1", "s2"} {
|
|
plan := f.planner.PlanSession(id, "", true, 0)
|
|
if plan.ProxyNode == nil {
|
|
t.Fatalf("plan %d: expected a proxy", i)
|
|
}
|
|
seen[plan.ProxyNode.URL] = true
|
|
}
|
|
if len(seen) != 2 {
|
|
t.Fatalf("expected round-robin across both group proxies, saw %v", seen)
|
|
}
|
|
}
|
|
|
|
func TestNilPlannerReturnsEmptyPlan(t *testing.T) {
|
|
var p *Planner
|
|
plan := p.PlanSession("s1", "", true, 0)
|
|
if plan.TranscodeNode != nil || plan.ProxyNode != nil {
|
|
t.Fatalf("expected empty plan from nil planner, got %+v", plan)
|
|
}
|
|
}
|
|
|
|
func TestBandwidthCapSkipsSaturatedProxy(t *testing.T) {
|
|
saturated := proxyNode(1, "http://proxy-1", nil)
|
|
saturated.MaxBandwidthKbps = intPtr(100_000) // 100 Mbps
|
|
saturated.EgressKbps = 97_000
|
|
f := newFixture(
|
|
[]*Node{saturated, proxyNode(2, "http://proxy-2", nil)},
|
|
[]*Node{},
|
|
)
|
|
|
|
// A 6 Mbps stream doesn't fit in proxy-1's 3 Mbps of headroom.
|
|
for i := 0; i < 3; i++ {
|
|
plan := f.planner.PlanSession("s", "", false, 6_000)
|
|
if plan.ProxyNode == nil || plan.ProxyNode.URL != "http://proxy-2" {
|
|
t.Fatalf("expected proxy-2 (proxy-1 saturated), got %+v", plan.ProxyNode)
|
|
}
|
|
}
|
|
|
|
// A 2 Mbps stream still fits.
|
|
plan := f.planner.PlanSession("s2", "", false, 2_000)
|
|
if plan.ProxyNode == nil {
|
|
t.Fatal("expected a proxy for a stream that fits")
|
|
}
|
|
}
|
|
|
|
func TestBandwidthReservationsCountDuringBridge(t *testing.T) {
|
|
capped := proxyNode(1, "http://proxy-1", nil)
|
|
capped.MaxBandwidthKbps = intPtr(10_000)
|
|
f := newFixture([]*Node{capped}, []*Node{})
|
|
|
|
// Two 4 Mbps admissions fit; the third would exceed the 10 Mbps cap
|
|
// because the first two are still bridged as reservations.
|
|
if f.planner.PlanSession("s1", "", false, 4_000).ProxyNode == nil {
|
|
t.Fatal("first stream should be admitted")
|
|
}
|
|
if f.planner.PlanSession("s2", "", false, 4_000).ProxyNode == nil {
|
|
t.Fatal("second stream should be admitted")
|
|
}
|
|
if got := f.planner.PlanSession("s3", "", false, 4_000).ProxyNode; got != nil {
|
|
t.Fatalf("third stream should be rejected, got %+v", got)
|
|
}
|
|
|
|
// Unlike job reservations, bandwidth bridges ignore health freshness —
|
|
// a report right after admission would not reflect the streams yet.
|
|
newer := f.now.Add(5 * time.Second)
|
|
f.proxies.ApplyHealth(1, true, 0, 0, newer)
|
|
f.now = f.now.Add(10 * time.Second)
|
|
if got := f.planner.PlanSession("s4", "", false, 4_000).ProxyNode; got != nil {
|
|
t.Fatalf("stream should still be rejected during bridge window, got %+v", got)
|
|
}
|
|
|
|
// After the bridge window the measured egress is authoritative. The
|
|
// meter now reports 8 Mbps, so one more 4 Mbps stream still won't fit,
|
|
// but a 2 Mbps one will.
|
|
f.now = f.now.Add(bandwidthBridgeAge)
|
|
f.proxies.ApplyHealth(1, true, 0, 8_000, f.now)
|
|
if got := f.planner.PlanSession("s5", "", false, 4_000).ProxyNode; got != nil {
|
|
t.Fatalf("4 Mbps stream should not fit at 8/10 Mbps, got %+v", got)
|
|
}
|
|
if f.planner.PlanSession("s6", "", false, 2_000).ProxyNode == nil {
|
|
t.Fatal("2 Mbps stream should fit at 8/10 Mbps")
|
|
}
|
|
}
|
|
|
|
func TestGroupBandwidthGatesGroupTranscode(t *testing.T) {
|
|
groupProxy := proxyNode(1, "http://proxy-a", strPtr("rack-a"))
|
|
groupProxy.MaxBandwidthKbps = intPtr(10_000)
|
|
groupProxy.EgressKbps = 9_000
|
|
f := newFixture(
|
|
[]*Node{groupProxy, proxyNode(2, "http://proxy-1", nil)},
|
|
[]*Node{
|
|
transcodeNode(3, "http://tc-a", strPtr("rack-a"), 0),
|
|
transcodeNode(4, "http://tc-1", nil, 7),
|
|
},
|
|
)
|
|
|
|
// rack-a's proxy has no bandwidth headroom for a 4 Mbps stream, so the
|
|
// group's idle transcode node must be skipped.
|
|
plan := f.planner.PlanSession("s1", "", true, 4_000)
|
|
if plan.TranscodeNode == nil || plan.TranscodeNode.URL != "http://tc-1" {
|
|
t.Fatalf("expected ungrouped tc-1, got %+v", plan.TranscodeNode)
|
|
}
|
|
if plan.ProxyNode == nil || plan.ProxyNode.URL != "http://proxy-1" {
|
|
t.Fatalf("expected ungrouped proxy-1, got %+v", plan.ProxyNode)
|
|
}
|
|
|
|
// A 500 kbps stream fits and stays pinned to the group.
|
|
plan = f.planner.PlanSession("s2", "", true, 500)
|
|
if plan.TranscodeNode == nil || plan.TranscodeNode.URL != "http://tc-a" {
|
|
t.Fatalf("expected rack-a tc-a for small stream, got %+v", plan.TranscodeNode)
|
|
}
|
|
if plan.ProxyNode == nil || plan.ProxyNode.URL != "http://proxy-a" {
|
|
t.Fatalf("expected rack-a proxy, got %+v", plan.ProxyNode)
|
|
}
|
|
}
|
|
|
|
func TestUnknownBitrateAdmittedBelowCap(t *testing.T) {
|
|
p := proxyNode(1, "http://proxy-1", nil)
|
|
p.MaxBandwidthKbps = intPtr(10_000)
|
|
p.EgressKbps = 9_999
|
|
f := newFixture([]*Node{p}, []*Node{})
|
|
|
|
// Unknown bitrate (0): admitted while measured egress is below the cap.
|
|
if f.planner.PlanSession("s1", "", false, 0).ProxyNode == nil {
|
|
t.Fatal("unknown-bitrate stream should be admitted below cap")
|
|
}
|
|
|
|
f.proxies.ApplyHealth(1, true, 0, 10_000, f.now)
|
|
if got := f.planner.PlanSession("s2", "", false, 0).ProxyNode; got != nil {
|
|
t.Fatalf("unknown-bitrate stream should be rejected at cap, got %+v", got)
|
|
}
|
|
}
|
|
|
|
func TestReserveTranscodeWorkSharesCapacityReservations(t *testing.T) {
|
|
capOne := 1
|
|
transcodes := NewTranscodePool()
|
|
transcodes.SetNodes([]*Node{
|
|
{URL: "http://transcode-a", Enabled: true, Healthy: true, MaxJobs: &capOne},
|
|
{URL: "http://transcode-b", Enabled: true, Healthy: true, MaxJobs: &capOne},
|
|
})
|
|
planner := NewPlanner(NewProxyPool(), transcodes)
|
|
|
|
first, releaseFirst := planner.ReserveTranscodeWork("download-1")
|
|
if first == nil || first.URL != "http://transcode-a" {
|
|
t.Fatalf("first node = %+v", first)
|
|
}
|
|
second, releaseSecond := planner.ReserveTranscodeWork("download-2")
|
|
if second == nil || second.URL != "http://transcode-b" {
|
|
t.Fatalf("second node = %+v", second)
|
|
}
|
|
if third, _ := planner.ReserveTranscodeWork("download-3"); third != nil {
|
|
t.Fatalf("third node = %+v, want no capacity", third)
|
|
}
|
|
|
|
releaseFirst()
|
|
third, releaseThird := planner.ReserveTranscodeWork("download-3")
|
|
if third == nil || third.URL != "http://transcode-a" {
|
|
t.Fatalf("third node after release = %+v", third)
|
|
}
|
|
releaseSecond()
|
|
releaseThird()
|
|
}
|
|
|
|
func TestReserveTranscodeWorkOverlappingAttemptsReleaseOnlyTheirOwnReservation(t *testing.T) {
|
|
capTwo := 2
|
|
transcodes := NewTranscodePool()
|
|
transcodes.SetNodes([]*Node{{URL: "http://transcode-a", Enabled: true, Healthy: true, MaxJobs: &capTwo}})
|
|
planner := NewPlanner(NewProxyPool(), transcodes)
|
|
|
|
first, releaseFirst := planner.ReserveTranscodeWork("same-artifact")
|
|
second, releaseSecond := planner.ReserveTranscodeWork("same-artifact")
|
|
if first == nil || second == nil {
|
|
t.Fatalf("overlapping reservations = first %+v second %+v", first, second)
|
|
}
|
|
if third, _ := planner.ReserveTranscodeWork("third-attempt"); third != nil {
|
|
t.Fatalf("third reservation = %+v, want full node", third)
|
|
}
|
|
|
|
releaseFirst()
|
|
third, releaseThird := planner.ReserveTranscodeWork("third-attempt")
|
|
if third == nil {
|
|
t.Fatal("first release did not free its own reservation")
|
|
}
|
|
if fourth, _ := planner.ReserveTranscodeWork("fourth-attempt"); fourth != nil {
|
|
t.Fatalf("second overlapping reservation was lost: fourth = %+v", fourth)
|
|
}
|
|
|
|
releaseSecond()
|
|
releaseThird()
|
|
}
|
|
|
|
func TestPlanDownloadSkipsBandwidthCappedProxiesAndReservesJobCapacity(t *testing.T) {
|
|
bandwidthCap := 100_000
|
|
jobCap := 1
|
|
proxies := NewProxyPool()
|
|
proxies.SetNodes([]*Node{
|
|
{URL: "http://bandwidth-capped", Enabled: true, Healthy: true, MaxBandwidthKbps: &bandwidthCap},
|
|
{URL: "http://uncapped", Enabled: true, Healthy: true, MaxJobs: &jobCap},
|
|
})
|
|
planner := NewPlanner(proxies, NewTranscodePool())
|
|
|
|
first := planner.PlanDownload("download-1")
|
|
if first.ProxyNode == nil || first.ProxyNode.URL != "http://uncapped" {
|
|
t.Fatalf("first plan = %+v", first)
|
|
}
|
|
if second := planner.PlanDownload("download-2"); second.ProxyNode != nil {
|
|
t.Fatalf("second plan = %+v, want job-cap rejection", second)
|
|
}
|
|
planner.ReleaseSession("download-1")
|
|
if second := planner.PlanDownload("download-2"); second.ProxyNode == nil {
|
|
t.Fatal("download was not admitted after reservation release")
|
|
}
|
|
}
|
|
|
|
func TestPlanDownloadPrefersArtifactOriginGroup(t *testing.T) {
|
|
groupA, groupB := "host-a", "host-b"
|
|
proxies := NewProxyPool()
|
|
proxies.SetNodes([]*Node{
|
|
{URL: "http://proxy-a", Group: &groupA, Enabled: true, Healthy: true},
|
|
{URL: "http://proxy-b", Group: &groupB, Enabled: true, Healthy: true},
|
|
})
|
|
planner := NewPlanner(proxies, NewTranscodePool())
|
|
plan := planner.PlanDownload("download-grouped", groupB)
|
|
if plan.ProxyNode == nil || plan.ProxyNode.URL != "http://proxy-b" {
|
|
t.Fatalf("plan = %+v, want proxy-b", plan)
|
|
}
|
|
}
|
|
|
|
func TestPlanDownloadFallsBackWhenOriginGroupHasNoProxy(t *testing.T) {
|
|
group := "host-a"
|
|
proxies := NewProxyPool()
|
|
proxies.SetNodes([]*Node{{URL: "http://proxy-a", Group: &group, Enabled: true, Healthy: true}})
|
|
planner := NewPlanner(proxies, NewTranscodePool())
|
|
plan := planner.PlanDownload("download-fallback", "host-missing")
|
|
if plan.ProxyNode == nil || plan.ProxyNode.URL != "http://proxy-a" {
|
|
t.Fatalf("plan = %+v, want proxy-a fallback", plan)
|
|
}
|
|
}
|
|
|
|
// A proxy-only plan applies the eligibility predicate to the proxy: it is the
|
|
// node that executes the recipe. Without this, one incapable round-robin pick
|
|
// would abandon a pool that still holds a capable sibling.
|
|
func TestPlanSessionWithFiltersProxiesForProxyOnlyPlans(t *testing.T) {
|
|
proxies := NewProxyPool()
|
|
proxies.SetNodes([]*Node{
|
|
{ID: 1, URL: "http://proxy-old", Enabled: true, Healthy: true},
|
|
{ID: 2, URL: "http://proxy-new", Enabled: true, Healthy: true},
|
|
})
|
|
planner := NewPlanner(proxies, NewTranscodePool())
|
|
|
|
// Only the upgraded proxy can run the recipe; every selection must land
|
|
// there regardless of where the round-robin cursor happens to be.
|
|
for i := 0; i < 4; i++ {
|
|
plan := planner.PlanSessionWith("session-"+strconv.Itoa(i), "", false, 0, func(n *Node) bool {
|
|
return n.URL == "http://proxy-new"
|
|
})
|
|
if plan.ProxyNode == nil || plan.ProxyNode.URL != "http://proxy-new" {
|
|
t.Fatalf("selection %d = %#v, want the capable proxy", i, plan.ProxyNode)
|
|
}
|
|
}
|
|
}
|
|
|
|
func TestPlanSessionWithReturnsNoProxyWhenNoneAreCapable(t *testing.T) {
|
|
proxies := NewProxyPool()
|
|
proxies.SetNodes([]*Node{{ID: 1, URL: "http://proxy-old", Enabled: true, Healthy: true}})
|
|
planner := NewPlanner(proxies, NewTranscodePool())
|
|
|
|
plan := planner.PlanSessionWith("session-none", "", false, 0, func(*Node) bool { return false })
|
|
if plan.ProxyNode != nil {
|
|
t.Fatalf("proxy = %#v, want none when the pool cannot execute the recipe", plan.ProxyNode)
|
|
}
|
|
// An empty plan must not leave a reservation pinning capacity.
|
|
if _, reserved := planner.reserved["session-none"]; reserved {
|
|
t.Fatal("an unsatisfiable plan left a reservation behind")
|
|
}
|
|
}
|
|
|
|
func TestProxyNodeURLsListsEnabledProxies(t *testing.T) {
|
|
proxies := NewProxyPool()
|
|
proxies.SetNodes([]*Node{
|
|
{ID: 1, URL: "http://proxy-1", Enabled: true, Healthy: true},
|
|
{ID: 2, URL: "http://proxy-2", Enabled: true, Healthy: false},
|
|
})
|
|
planner := NewPlanner(proxies, NewTranscodePool())
|
|
|
|
// Unhealthy nodes are still listed: capability planning wants the
|
|
// deployment's toolchain, and an unreachable node excludes itself when its
|
|
// capability fetch fails.
|
|
urls := planner.ProxyNodeURLs()
|
|
if len(urls) != 2 {
|
|
t.Fatalf("proxy urls = %v, want both pooled proxies", urls)
|
|
}
|
|
}
|