Files
silo-server/internal/transcodenode/server_test.go
T
854d07cf8f feat(playback): add protocol v3 planning and recovery (#398)
* docs(playback): plan protocol v3 server implementation

* docs(playback): incorporate protocol v3 review

* feat(playback): implement protocol v3 server

* fix(playback): persist empty route diagnostics

* feat(playback): harden protocol v3 HDR routing

* feat(playback): complete protocol v3 client contract

* fix(playback): harden protocol v3 recovery

* fix(playback): restore dovi_rpu strip filter for DV remuxes

The v3 work renamed the Dolby Vision strip recipe to a dovi_split=mode=bl
bitstream filter that does not exist in stock FFmpeg or jellyfin-ffmpeg;
the probe failed closed on every deployment, disabling the new validated
DV7-to-HDR10 route and regressing the previously working dovi_rpu=strip=1
remux path from main. Restore dovi_rpu across the probe, remux and HLS
copy arguments, and the recipe-card constant.

Also from review: validate the remux DV mode for every profile (garbage
modes on non-P7 sources silently no-opped), reject preserve mode for P7
outright (a base-layer-only remux cannot preserve dual-layer DV), tag
dvhe sample entries only for the explicit v3 preserve recipe so legacy
web/jellycompat remuxes keep their pre-v3 hev1 labeling, and honor the
token-frozen DV mode in the proxy remux path instead of legacy-auto.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(playback): correct v3 planner policy and contract validation

Review fixes to the v3 planner and wire contracts:

- Bar Profile 7 sources from the non-strip progressive remux route: a
  base-layer-only remux can never deliver native dual-layer DV, so the
  planner no longer emits plans claiming validated Dolby Vision while
  the executed remux drops the enhancement layer.
- Accept the device-quirks feature flag from either capability location,
  matching every other dual-location feature check.
- Treat legacy hdr_unknown rows as HDR10 for HDR10-capable clients with
  a degradation warning instead of leaving them unplayable under v3.
- Honor bandwidth_cap_kbps as a hard ceiling in every quality mode and
  wire the previously dead Metered signal into conservative auto rungs.
- Degrade to the validated source-quality route instead of a terminal
  when only an implicit quality reduction demanded an unsupported
  transcode; explicit user-selected rungs keep terminal behavior.
- Bound inner capability lists and strings; compare attempt keys exactly
  instead of case-folded; make ParseTrackIDV3 strict about canonical
  numerics; accept dvdsub/pgssub/dvbsub aliases and stop promising
  burn-in for unknown subtitle codecs; probe every h264 encoder rather
  than requiring libx264; normalize the file-level bitrate fallback.
- Evaluate subtitle renderability against the engine each candidate
  route executes on, not always media3_direct.
- Pin the with-quirks attempt-key preimage arity in the cross-language
  fixture so the Kotlin client stays in lockstep.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(playback): harden v3 control-plane reliability

Review fixes to the v3 session, store, and handler layer:

- Bound concurrent replans with a slot semaphore: each replan pins a
  pooled connection for its advisory lock while issuing further store
  queries from the same pool, so an unbounded recovery storm could turn
  every connection into a lock holder and deadlock the server.
- Make CompleteReplan a real compare-and-swap (base-revision predicate,
  ErrReplanSupersededV3) and map BeginReplan insert races to a replay
  instead of a raw unique violation.
- Fingerprint start requests (request_digest column): an attempt ID
  reused with different input is now a 409-style conflict rather than a
  silent replay, and both replay paths check session liveness so dead
  sessions surface as retryable terminals.
- Pre-delete expired attempt rows on SaveAttempt so a retry during the
  cleanup window cannot wedge on an unreachable conflict.
- Align the in-memory store's semantics with Postgres and add DB-backed
  planstore tests (SILO_TEST_DATABASE_URL), including a regression test
  inserting every route-event name against the real CHECK constraint.
- Session manager: v3 route-set updates own RemuxDVMode outright so a
  replan onto an SDR source clears a stale strip mode; replacement
  reservations survive unrelated legacy stream updates; replacement
  admission excludes the replaced session explicitly instead of
  decrementing totals it may no longer be part of; the admission CAS
  loop is bounded and decider errors are logged.
- Map transient store failures to 500s instead of terminal 404/403s;
  authorize route events via identity-only projections after the rate
  limiter; keep sanitized diagnostics deterministic.
- Merge the server-computed durable plan key into replan exclusions so
  unreproducible client history cannot re-select the failed route.
- Remap tracks only when the effective edition changes (a same-file
  replan no longer switches audio to a lookalike track) and remap
  ID-only subtitle selections on edition fallback.
- Cache the v3/shadow feature flags for five seconds instead of one
  settings SELECT per playback request; stop remote transports
  best-effort when the start call times out; carry dvm/tid claims and
  the transport-scoped job identity through the legacy audio-change
  re-mint; index playback_route_events(received_at) for the retention
  delete; run store maintenance for DB-less deployments too.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(transcode): reap idle node jobs and gate WebVTT conversion

- Add an idle reaper to the transcode node: a job untouched by manifest
  or segment requests for ten minutes is closed and unregistered. After
  a v3 replan retires a transport ID, a stale in-flight stream token
  could resurrect the old job via reconstruct and encode to end-of-file
  for nobody; jobs waiting on readiness count registration as access
  and are never reaped mid-wait, and reaping keeps the recipe so a
  still-valid token reconstructs on the next hit.
- Reject bitmap subtitle tracks (PGS) on the .vtt conversion path with
  415 before headers are written instead of spawning an ffmpeg command
  that always fails mid-response, and make the extract-format override
  fall back to source-driven mapping for bitmap codecs.
- Drain error bodies on non-202 node responses so the HTTP transport
  can reuse connections.
- Pin the transcode-dir cleanup separator-boundary semantics with a
  regression test (a session ID sharing another's prefix must not
  retain foreign directories).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(playback): close v3 planner policy gaps from review

- Clamp the final transcode bitrate to bandwidth_cap_kbps: the ladder has
  no rung below 480p/1500kbps, so lower caps were silently exceeded even
  though the cap is documented as a hard delivery ceiling.
- Treat video-only media as audio-compatible instead of forcing an AAC
  conversion (or an audio_conversion_unsupported terminal) onto a file
  with no audio stream. Tracks whose codec failed to probe keep the gate.
- Only promise a bitmap subtitle sidecar for embedded PGS with an engine
  that renders embedded bitmap: external/downloaded bitmap and embedded
  DVD/DVB published artifact URLs that always failed at fetch. They now
  fall through to burn-in or its terminal.
- Accept client_video_transformations_v1 from either client_features or
  the nested context when validating client-executor transformations,
  matching the planner's dual-source reads.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(playback): probe and execute DV remuxes with one ffmpeg binary

The v3 transformation registry probed the configured playback.ffmpeg_path
while progressive remux execution resolved the process-global discovery
path, so a deployment where only one binary carries dovi_rpu could plan a
server_dv7_to_hdr10 route and then fail it at stream time. Resolution now
goes through a shared ResolveFFmpegPath (configured path first, discovery
fallback — the same rule the transcode pipeline already used), the
dovi_rpu probe is cached per binary path, and the stream handler and proxy
worker pass their configured path into ServeRemuxWithDVMode.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(playback): harden v3 replan identity and control-plane limits

- Seed failure-replan track selections from the durable current plan
  before overlaying the request: after an alternate-version fallback the
  normalized request still carries requested-edition track IDs, so a
  replan omitting unchanged tracks was rejected as a track/file mismatch.
- Remap ID-only audio selections across edition changes (parse the ID to
  an index like the subtitle remap already does) instead of leaving a
  stale file-bound ID to fail validation.
- Release the node planner reservation when a prepared remote transport
  rolls back after the node accepted the job; repeated failed starts
  could otherwise pin max-job/bandwidth budgets for the full reservation
  age.
- Size the replan semaphore below the PostgreSQL pool via a store
  capacity advisor: with max_connections at or below the fixed bound,
  advisory-lock holders could starve the inner store queries they need
  to finish.
- Contain shadow-planner panics with a recover boundary; it runs on a
  bare goroutine where an escaped panic kills the process for what is
  telemetry-only work. Document why the memory store's session lock is
  deliberately a no-op.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(transcode): serialize node job teardown against reconstructs

- Look up and touch manifest/segment sessions in one critical section so
  the idle reaper cannot unregister a job between the lookup and its
  liveness refresh.
- Re-validate each reap candidate under the per-session lifecycle lock
  before closing it: Close removes the output directory, and without the
  lock it could race a token reconstruct and wipe the segments the fresh
  ffmpeg is writing.
- Take the lifecycle lock in handleStop so a stop racing a RequireReady
  start's readiness wait blocks until registration and tears the job
  down, instead of 404ing and orphaning the ffmpeg until the reaper.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 11:51:27 -04:00

362 lines
12 KiB
Go
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
package transcodenode
import (
"bytes"
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/go-chi/chi/v5"
"github.com/Silo-Server/silo-server/internal/config"
"github.com/Silo-Server/silo-server/internal/nodeconfig"
"github.com/Silo-Server/silo-server/internal/nodesessions"
"github.com/Silo-Server/silo-server/internal/playback"
"github.com/Silo-Server/silo-server/internal/streamtoken"
)
const testSecret = "node-reconstruct-test-secret"
// newTestServer builds a transcode Server whose config carries a known JWT secret
// so reconstructFromToken can verify forwarded stream tokens. The tracker is left
// nil: the guard-rejection cases never reach the spawn/track path.
func newTestServer(t *testing.T) *Server {
t.Helper()
w := nodeconfig.NewWatcher(nil, nil, nil, nodeconfig.BootstrapOverrides{})
cfg := &config.Config{}
cfg.Auth.JWTSecret = testSecret
cfg.Playback.TranscodeDir = t.TempDir()
w.SetConfigForTest(cfg)
return &Server{
watcher: w,
sessions: make(map[string]*playback.TranscodeSession),
}
}
func TestHandleStartRequireReadyRejectsExitedFFmpeg(t *testing.T) {
server := newTestServer(t)
ffmpegPath := filepath.Join(t.TempDir(), "failing-ffmpeg.sh")
if err := os.WriteFile(ffmpegPath, []byte("#!/bin/sh\nexit 1\n"), 0o755); err != nil {
t.Fatal(err)
}
server.watcher.Config().Playback.FFmpegPath = ffmpegPath
requestBody, err := json.Marshal(TranscodeStartRequest{
SessionID: "ready-failure-1",
InputPath: "/media/movie.mkv",
TargetCodecVideo: "h264",
TargetCodecAudio: "aac",
SegmentDuration: 2,
RequireReady: true,
})
if err != nil {
t.Fatal(err)
}
req := httptest.NewRequest(http.MethodPost, "/transcode/start", bytes.NewReader(requestBody))
rr := httptest.NewRecorder()
server.handleStart(rr, req)
if rr.Code != http.StatusInternalServerError {
t.Fatalf("status = %d, body = %s", rr.Code, rr.Body.String())
}
server.mu.RLock()
_, registered := server.sessions["ready-failure-1"]
server.mu.RUnlock()
if registered {
t.Fatal("failed readiness session was registered")
}
}
func signCard(t *testing.T, card playback.RecipeCard) string {
t.Helper()
tok, err := streamtoken.Sign(card.ToClaims(), testSecret, time.Hour)
if err != nil {
t.Fatalf("sign card: %v", err)
}
return tok
}
func requestWithToken(sessionID, token string) *http.Request {
r := httptest.NewRequest(http.MethodGet, "/transcode/"+sessionID+"/master.m3u8", nil)
if token != "" {
r.Header.Set("X-Silo-Stream-Token", token)
}
return r
}
func transcodeCard(sessionID string) playback.RecipeCard {
return playback.NewRecipeCard(7, "profile-1", 42, "", playback.TranscodeOpts{
SessionID: sessionID,
InputPath: "/media/movie.mkv",
TargetCodecVideo: "h264",
SegmentDuration: 6,
})
}
// reconstructFromToken must refuse — without spawning ffmpeg — every request that
// does not carry a valid, matching transcode token. These guards run before any
// StartTranscode, so they are safe to assert without ffmpeg or a media file.
func TestReconstructFromToken_RejectsUnusableTokens(t *testing.T) {
const sid = "sess-123"
s := newTestServer(t)
t.Run("missing token header", func(t *testing.T) {
if got := s.reconstructFromToken(requestWithToken(sid, ""), sid, -1); got != nil {
t.Fatalf("expected nil for missing token, got %v", got)
}
})
t.Run("invalid signature", func(t *testing.T) {
bad, err := streamtoken.Sign(transcodeCard(sid).ToClaims(), "wrong-secret", time.Hour)
if err != nil {
t.Fatalf("sign: %v", err)
}
if got := s.reconstructFromToken(requestWithToken(sid, bad), sid, -1); got != nil {
t.Fatalf("expected nil for bad signature, got %v", got)
}
})
t.Run("session id mismatch", func(t *testing.T) {
tok := signCard(t, transcodeCard("other-session"))
if got := s.reconstructFromToken(requestWithToken(sid, tok), sid, -1); got != nil {
t.Fatalf("expected nil for session id mismatch, got %v", got)
}
})
t.Run("non-transcode card", func(t *testing.T) {
tok := signCard(t, playback.NewDirectRecipeCard(sid, 7, "profile-1", 42))
if got := s.reconstructFromToken(requestWithToken(sid, tok), sid, -1); got != nil {
t.Fatalf("expected nil for direct-play card, got %v", got)
}
})
// The jellycompat node hop signs an identity-only transcode token (the recipe
// lives in the central compat store). Its card decodes as PlayTranscode for the
// right session id but with no encode parameters; with no recipe store wired the
// node must refuse it rather than spawn a malformed ffmpeg.
t.Run("recipe-less transcode token, no recipe store", func(t *testing.T) {
tok := signCard(t, playback.RecipeCard{
SessionID: sid,
UserID: 7,
PlayMethod: playback.PlayTranscode,
InputPath: "/media/movie.mkv",
})
if got := s.reconstructFromToken(requestWithToken(sid, tok), sid, 5); got != nil {
t.Fatalf("expected nil for recipe-less transcode token, got %v", got)
}
})
}
// stubRecipeStore is a recipeStore for the jellycompat node-restart fetch path.
type stubRecipeStore struct {
card *playback.RecipeCard
ok bool
hits int
deletes []string
delErr error
}
func (s *stubRecipeStore) Get(context.Context, string) (*playback.RecipeCard, bool) {
s.hits++
return s.card, s.ok
}
func (s *stubRecipeStore) Delete(_ context.Context, sessionID string) error {
s.deletes = append(s.deletes, sessionID)
return s.delErr
}
// When the forwarded token is recipe-less (jellycompat), the node consults the
// recipe store. A miss or an incomplete recipe must yield a clean nil (404) with
// no ffmpeg spawn — these assert the resolve guards without needing ffmpeg.
func TestReconstructFromToken_JellycompatRecipeFetch(t *testing.T) {
const sid = "compat-sess-1"
recipeLessToken := func(t *testing.T) string {
return signCard(t, playback.RecipeCard{
SessionID: sid,
UserID: 7,
PlayMethod: playback.PlayTranscode,
InputPath: "/media/movie.mkv",
})
}
t.Run("store miss -> nil", func(t *testing.T) {
s := newTestServer(t)
store := &stubRecipeStore{ok: false}
s.SetRecipeStore(store)
if got := s.reconstructFromToken(requestWithToken(sid, recipeLessToken(t)), sid, 5); got != nil {
t.Fatalf("expected nil on store miss, got %v", got)
}
if store.hits != 1 {
t.Fatalf("recipe store consulted %d times, want 1", store.hits)
}
})
t.Run("incomplete fetched recipe -> nil", func(t *testing.T) {
s := newTestServer(t)
// Right session id but missing encode params: must not spawn.
s.SetRecipeStore(&stubRecipeStore{ok: true, card: &playback.RecipeCard{SessionID: sid, PlayMethod: playback.PlayTranscode}})
if got := s.reconstructFromToken(requestWithToken(sid, recipeLessToken(t)), sid, 5); got != nil {
t.Fatalf("expected nil for incomplete fetched recipe, got %v", got)
}
})
t.Run("fetched recipe for wrong session -> nil", func(t *testing.T) {
s := newTestServer(t)
s.SetRecipeStore(&stubRecipeStore{ok: true, card: &playback.RecipeCard{
SessionID: "other", PlayMethod: playback.PlayTranscode, SegmentDuration: 6, TargetCodecVideo: "h264",
}})
if got := s.reconstructFromToken(requestWithToken(sid, recipeLessToken(t)), sid, 5); got != nil {
t.Fatalf("expected nil for wrong-session recipe, got %v", got)
}
})
}
// handleStop is a deliberate teardown, so it must drop the session's recipe to
// stop a buffered/retrying post-restart request from reconstructing a brand-new
// ffmpeg for an already-stopped session. A zero-value TranscodeSession needs no
// ffmpeg or media file to Close, so this asserts the wiring without a real spawn.
func TestHandleStop_DeletesRecipe(t *testing.T) {
const sid = "stop-sess-1"
s := newTestServer(t)
s.tracker = nodesessions.NewTracker(nil, "node-url", "node-name", "transcode")
store := &stubRecipeStore{}
s.SetRecipeStore(store)
s.sessions[sid] = &playback.TranscodeSession{}
s.activeJobs.Store(1)
r := httptest.NewRequest(http.MethodDelete, "/transcode/"+sid, nil)
rctx := chi.NewRouteContext()
rctx.URLParams.Add("session_id", sid)
r = r.WithContext(context.WithValue(r.Context(), chi.RouteCtxKey, rctx))
rec := httptest.NewRecorder()
s.handleStop(rec, r)
if rec.Code != http.StatusNoContent {
t.Fatalf("handleStop status = %d, want %d", rec.Code, http.StatusNoContent)
}
if len(store.deletes) != 1 || store.deletes[0] != sid {
t.Fatalf("recipe deletes = %v, want [%q]", store.deletes, sid)
}
if _, ok := s.sessions[sid]; ok {
t.Fatalf("session %q still registered after stop", sid)
}
}
// The idle reaper must close only jobs whose last access predates the TTL;
// registration counts as an access, so a just-started job (including one still
// waiting on its manifest in the RequireReady flow) is spared. Zero-value
// TranscodeSessions Close without ffmpeg, so this runs without a real spawn.
func TestReapIdleSessions_ClosesOnlyIdleJobs(t *testing.T) {
s := newTestServer(t)
s.tracker = nodesessions.NewTracker(nil, "node-url", "node-name", "transcode")
s.sessions["fresh-1"] = &playback.TranscodeSession{}
s.sessions["stale-1"] = &playback.TranscodeSession{}
s.lastAccess = map[string]time.Time{
"fresh-1": time.Now(),
"stale-1": time.Now().Add(-sessionIdleTTL - time.Minute),
}
s.activeJobs.Store(2)
s.reapIdleSessions(sessionIdleTTL)
s.mu.RLock()
_, freshAlive := s.sessions["fresh-1"]
_, staleAlive := s.sessions["stale-1"]
_, staleTracked := s.lastAccess["stale-1"]
s.mu.RUnlock()
if !freshAlive {
t.Fatal("recently accessed session was reaped")
}
if staleAlive {
t.Fatal("idle session survived the reaper")
}
if staleTracked {
t.Fatal("reaped session's idle clock was not dropped")
}
if got := s.activeJobs.Load(); got != 1 {
t.Fatalf("activeJobs = %d, want 1", got)
}
}
// A registered job with no recorded access (untracked registration) must not
// be closed; the sweep starts its idle clock instead of reaping a job that may
// be actively serving.
func TestReapIdleSessions_StartsClockForUntrackedJob(t *testing.T) {
s := newTestServer(t)
s.sessions["untracked-1"] = &playback.TranscodeSession{}
s.activeJobs.Store(1)
s.reapIdleSessions(sessionIdleTTL)
s.mu.RLock()
_, alive := s.sessions["untracked-1"]
last, tracked := s.lastAccess["untracked-1"]
s.mu.RUnlock()
if !alive {
t.Fatal("untracked session was reaped")
}
if !tracked || last.IsZero() {
t.Fatal("sweep did not start the untracked session's idle clock")
}
if got := s.activeJobs.Load(); got != 1 {
t.Fatalf("activeJobs = %d, want 1", got)
}
}
// touchSession must refresh a registered job's idle clock and ignore ids with
// no live session (a reconstruct records its own first access on register).
func TestTouchSession_RefreshesIdleClock(t *testing.T) {
s := newTestServer(t)
s.sessions["live-1"] = &playback.TranscodeSession{}
stale := time.Now().Add(-sessionIdleTTL - time.Minute)
s.lastAccess = map[string]time.Time{"live-1": stale}
s.touchSession("live-1")
s.touchSession("ghost-1")
s.mu.RLock()
defer s.mu.RUnlock()
if !s.lastAccess["live-1"].After(stale) {
t.Fatal("touch did not refresh the live session's idle clock")
}
if _, ok := s.lastAccess["ghost-1"]; ok {
t.Fatal("touch recorded access for an unregistered session")
}
}
// spawnReconstruct must NOT apply the fast seg×dur resume seek for copy-mode
// cards: copy-mode segments have variable durations, so seg×dur points at the
// wrong source time. The card's original start must stand. Asserting opts off a
// real spawn would need ffmpeg, so this checks the gating condition directly.
func TestCopyModeReconstruct_SkipsFastSeek(t *testing.T) {
const dur = 6
card := playback.RecipeCard{
SessionID: "copy-sess-1",
PlayMethod: playback.PlayTranscode,
TargetCodecVideo: "copy",
SegmentDuration: dur,
StartSegmentNumber: 0,
}
const requestedSegment = 10
applyFastSeek := requestedSegment > card.StartSegmentNumber && card.SegmentDuration > 0 &&
!strings.EqualFold(card.TargetCodecVideo, "copy")
if applyFastSeek {
t.Fatalf("copy-mode card must not apply the seg×dur fast seek")
}
// Same shape but ENCODED: the fast seek must apply.
card.TargetCodecVideo = "h264"
applyFastSeek = requestedSegment > card.StartSegmentNumber && card.SegmentDuration > 0 &&
!strings.EqualFold(card.TargetCodecVideo, "copy")
if !applyFastSeek {
t.Fatalf("encoded card must apply the seg×dur fast seek")
}
}