* feat(playback): unified restart-resilient playback via shared TranscodeManager Make direct, remux, and native HLS transcode sessions survive a server restart through one shared flow instead of per-method paths. A missing in-memory session becomes a reconstruct trigger, not a 404: the server rebuilds the session from a tiny durable recipe card plus the position the client re-supplies on its next request. - internal/playback/transcode_manager.go: shared TranscodeManager owning the transcodes map, recipe-card lifecycle, reconstruct single-flight + concurrency cap, LoadOrReconstructSession front door, ReconstructSession / ReconstructTranscode, and orphan cleanup. ~90% is logic moved out of the native handler (no behavior change), not new surface. - internal/playback/recipecard.go + recipecard_postgres.go: RecipeCard with a PlayMethod discriminator (direct/remux/transcode; empty decodes as transcode for back-compat) behind a swappable, nil-safe RecipeStore interface backed by transcode_recipes. - internal/playback/session.go: RegisterReconstructed inserts a rebuilt Session under its existing id (no UUID mint, no limit double-count, race-yielding). - internal/playback/transcode.go: CloseProcess keeps the output dir so a reconstruct winner keeps serving; Close removes it. - internal/api/handlers: drain the transcode lifecycle into the manager; wire reconstruct into the stream/segment serve paths; re-bind ownership to the live caller (refuse userID==0/mismatch); card-aware orphan cleanup. - migrations: add transcode_recipes (expires_at TTL, filter-on-read, indexed). Ownership stays two-factor: an authenticated caller AND a session.UserID that matches; the card stores no secrets and identity is re-resolved per request. Tests: recipe-card round-trip/legacy-decode/disabled-noop, RegisterReconstructed insert/race/concurrency, close-vs-close-process dir semantics, the LoadOrReconstructSession status matrix, and the reconstruct concurrency cap. AI-use: implemented with AI assistance (design, implementation, adversarial review). * feat(jellycompat): reconstruct transcodes across restart via shared manager Bring Jellyfin (jellycompat) HLS playback onto the same restart-resilient flow as the native path. Previously jellycompat owned a separate PlaybackHandler with a private transcodes map and a duplicated transcode lifecycle that never grew the reconstruct half, so an in-flight Jellyfin transcode died on restart and the next segment request 404'd. - Embed the shared playback.TranscodeManager and delete the duplicate lifecycle, so jellycompat gets reconstruct, the concurrency cap, the node-affinity rule, and the card lifecycle for free. - internal/jellycompat/playback_sessions_postgres.go: DurableCompatPlaybackStore, a write-through cache over jellycompat_playback_sessions behind the new CompatPlaybackStore interface (nil pool degrades to cache-only). This persists the load-bearing PlaySessionId -> UpstreamSessionID mapping (plus media sources, route item id, seek) so it survives a restart instead of vanishing with the map. - Write a recipe card on compat transcode start keyed by the upstream session id, using the native StreamAppUserID so the ownership re-bind matches; reconstruct the upstream session and the transcode seeked to the requested seg_NNNNN. - migrations: add jellycompat_playback_sessions (expires_at TTL + compat_token index, full PlaybackSession in data JSONB). Auth is mapped to the native user id before reconstruct so the same two-factor ownership check and userID==0/mismatch refusal apply unchanged. Tests: DB-gated (SILO_TEST_DATABASE_URL) durable-store round-trip proving a session written by one instance reloads in a fresh one (the restart case), plus a nil-pool cache-only path; existing handler tests updated to the manager. AI-use: implemented with AI assistance (design, implementation, adversarial review). * docs(playback): consolidate unified playback reconstruction design Replace the three overlapping playback docs (the native Postgres restart-resilience spec, the jellycompat plan, and the unification spec) with a single self-contained design at docs/superpowers/specs/unified-playback-reconstruct.md. The doc leads with the unified design — the one-idea reconstruct model, a strong visual flow of a restart mid-playback, the shared TranscodeManager + recipe card, the two swappable durable stores, security, the concurrency cap and node-affinity constraint, preconditions, and verification. The design history and rationale (reconstruct-not-rehydrate, phased delivery, Redis-vs-Postgres, token-as- descriptor, failure analysis) move to an appendix. It references no other md file. AI-use: written with AI assistance. * fix(playback): address review on restart-resilient playback Four fixes from PR review of the unified reconstruction work: - Rewrite the recipe card on audio-track change. HandleChangeAudioTrack only updated the in-memory session/transcode, so after a restart reconstruct resumed with the stale AudioTrackIndex/TranscodeAudio (and stale play method) from the start-time card. Re-save the card (direct/remux/transcode) with the switched state, mirroring the start-card pattern. - Guard nil TranscodeManager in LoadOrReconstructSession and ReconstructSession. StreamHandler.TM is documented optional (tests/minimal setups); a missing session previously panicked in recipeEnabled instead of returning SessionMissing. ReconstructTranscode already guarded nil; make the two siblings consistent. - Reject direct/remux cards in doReconstructTranscode before spawning ffmpeg, so a non-transcode card id can never enter the HLS reconstruction path. - Log a non-success status from the remote transcode-node DELETE in CloseTranscodeSession; a 401/404/500 was previously silent. AI-use: implemented with AI assistance. * fix(playback): harden restart-resilient compat sessions * feat(playback): token-carried reconstruction across restarts Build on the shared TranscodeManager (introduced earlier in this branch) so a playback session survives an API-server or transcode-node restart without the client re-negotiating, and retire the Postgres transcode_recipes store in favor of a recipe carried inside the signed stream token. - RecipeCard encodes the byte-affecting encode parameters and rides inside the stream token; LoadOrReconstructSession rebuilds the in-memory Session (and, for integrated transcodes, the ffmpeg process) on a cold miss, single-flighted per session and paced by a spawn semaphore. Removes recipecard_postgres.go and the 20260617233705_add_transcode_recipes migration. - transcodenode reconstructs a lost ffmpeg node-side from the forwarded token. - TR-lease: proxy/streamauth enforce a revocation deny-marker on every served segment, with a 500ms Redis timeout, a bounded per-session "allowed" cache (3s TTL, expiry-first graceful eviction), and a degraded-fail-open counter. Review hardening folded in: - Manifest/segment handlers do the in-memory session lookup first and only verify the stream token on a reconstruct miss (token HMAC was per-segment). - Copy-mode reconstruct never applies the encoded-only seg*dur seek, at spawn time or via the recovery path: RestartSeekTarget reports "unresolved" for a copy session whose manifest cannot yet map the segment, so the client retries instead of seeking to a fabricated source time. - Crash teardown is a compare-and-delete (CloseTranscodeSessionIf returns whether it matched); the crash closure tears down the playback session only when it matched, so a session reconstructed under the same id is not killed. - Reconstruct enforces the same per-user stream/transcode caps as a fresh start (RegisterReconstructedWithLimits), closing a token-replay slot bypass. AI-use disclosure: implemented with AI assistance (Claude Code), including a two-round multi-agent adversarial review whose findings drove the hardening. * feat(jellycompat): node-side transcode reconstruct via shared recipe store Make Jellyfin-compat playback sessions survive a server or transcode-node restart by reusing the shared TranscodeManager reconstruct path and a durable recipe store, on top of the durable compat session store added earlier in this branch. - Node-side transcode reconstruct goes through the shared recipe store; the recipe is persisted to the control-plane store (Redis) when a dedicated transcode node is used so the node can rebuild ffmpeg after its own restart. - Adopt the shared manager's API (3-arg OnFFmpegCrash carrying the dead session, guarded CloseTranscodeSessionIf, RegisterReconstructedWithLimits). Review hardening folded in: - Recipe lifecycle: noderecipe.Store gains Delete, called on deliberate teardown (stop, method-switch discard, node stop/force-reload) so a stopped session cannot be resurrected by a buffered request after a node restart; crash paths intentionally keep the recipe so a resume can reconstruct. - Crash closure tears down the upstream session only when the guarded transcode close matched, so a reconstructed successor is never left orphaned. - Copy-mode segment recovery surfaces a retryable not-found instead of a wrong-position restart, matching the native and node paths. - Durable Update is now a SELECT ... FOR UPDATE transaction, removing the lost-update clobber that could silently drop a transcode recipe. - Empty-token route resolution no longer falls back to an unbounded full-table scan; DB expiry filters bind the injected clock; the redundant re-Get is gone. AI-use disclosure: implemented with AI assistance (Claude Code), including a two-round multi-agent adversarial review whose findings drove the hardening. * docs(playback): consolidate restart-resilient playback design Replace the superpowers spec with a single architecture record describing the token-carried recipe card, the shared TranscodeManager reconstruct path for direct/remux/transcode, the jellycompat durable session + node recipe store, and the revocation-lease model with its fail-open tradeoff. AI-use disclosure: written with AI assistance (Claude Code). * docs(playback): correct jellycompat node-recipe rationale in comments The noderecipe / transcode-node / jellycompat comments justified the Redis recipe store with "a Jellyfin client cannot round-trip a token". The real reason: the node-hop token is server-minted and could carry the recipe, but the recipe is mutated in place under a stable session id (a /Sessions/Playing/Progress audio switch restarts ffmpeg without re-minting the client's token) and a third-party Jellyfin client cannot be driven to refresh a stale token, so the node must reconstruct from a server-authoritative, node-reachable store. Aligns the comments with docs/architecture/restart-resilient-playback.md §10. Comment-only; no behavior change. * refactor(playback): remove deny-lease revocation, defer to future PR The deny-lease stream-revocation mechanism (the internal/streamauth package, its silo:streamauth:<sid> Redis markers, the proxy Allowed() enforcement, and the admin Stop/Terminate deny write) only ever enforced on the offload-proxy topology and was a silent no-op on the integrated single box and the dedicated transcode node. Rather than ship a partial revocation feature that looks complete but isn't, remove it wholesale and defer a uniform cross-topology revocation design to a dedicated follow-up. Removed: internal/streamauth (package + tests); the LeaseDenier field, StreamLeaseDenier interface, and denyStreamLease helper in playback.go; the admin deny write; the router/main wiring; and the proxy verifyToken Allowed() gate. The unified-reconstruct core (recipe-token, LoadOrReconstructSession) is orthogonal and untouched. Known limitation (now on every topology): admin Terminate and user Stop tear down the live in-memory session and ffmpeg producer, but a still-valid stream token can reconstruct the session until its 24h TTL expires. No node-side byte-withholding ships in this PR. docs/architecture/restart-resilient-playback.md is updated to mark the revocation/deny-lease sections as deferred and to drop the overstated "instant revocation on admin kill" claim. * fix(playback): allow zero-caller bearer on transcode reconstruct The authless HLS transcode delivery routes (master.m3u8 / segment) treat the session UUID as the bearer credential, so a real request carries requestUserID == 0. The live serve path already allows this, but ReconstructSession hard-rejected a zero caller, so a request that worked before a restart became SessionMissing -> 404 after the in-memory session was gone, breaking the restart resilience these routes advertise. Match the live-path contract in LoadOrReconstructSession: allow a zero caller (UUID-as-bearer) and refuse only a non-zero caller that mismatches the card owner. The reconstructed session is bound to card.UserID either way. Adds TestReconstructSession_Ownership covering both cases. * fix(jellycompat): re-persist recipe on local audio switch A Jellyfin client switching audio on an integrated/local compat transcode restarted live ffmpeg with the new track but did not re-persist PlaybackSession.Recipe. The remote branch already re-persists via startRemoteTranscode -> persistTranscodeRecipe. After a central restart, reconstruct rebuilt ffmpeg from the stale Recipe.AudioTrackIndex, so the integrated session resumed on the original audio track. Persist the updated recipe (best-effort) after a successful Restart in the local branch, mirroring the remote branch, so the durable Recipe.AudioTrackIndex tracks live ffmpeg. Adds a regression test. * fix(playback): strip stream token from proxied transcode-node URL proxyToTranscodeNode appended the client's raw query string to the internal transcode-node URL and logged that URL on transport failure. When a remote transcode runs without a separate proxy node, that query carries ?st=<signed JWT> — a 24h bearer reconstruction descriptor exposing the media path and recipe claims — placing the token into internal requests and error logs. Strip the "st" param before building targetURL, preserving any other query params. The token is neither forwarded to the node nor present in the logged URL. Header-forwarding of the token (so the node can reconstruct) is a separate follow-up (#6). * fix(playback): fail open on transient limit-provider error in reconstruct During the reconstruct wave right after a restart (Postgres under peak load), a transient limit-provider DB error was collapsed into a hard 404, permanently stopping playback for a user within their limits. limitsForUser wrapped any provider error, RegisterReconstructedWithLimits propagated it, and ReconstructSession mapped every error to SessionMissing -> 404 - indistinguishable from a genuine over-cap rejection. Distinguish the two: tag provider errors with a new ErrLimitProviderUnavailable sentinel and, during reconstruct, fail OPEN on a provider error (admit via RegisterReconstructed + log a degraded warning) rather than refuse - mirroring the reliability-first fail-open-on-dependency-error philosophy. A genuine ErrTooManyStreams / ErrTooManyTranscodes over-cap still refuses. Adds tests for both the fail-open and still-refused paths. * fix(playback): forward stream token to transcode node as header The dedicated transcode node's reconstruct path reads the stream token only from the X-Silo-Stream-Token header, but proxyToTranscodeNode forwarded only the node-API bearer token (and #5 now strips st from the URL). So when the central API proxied to the node and the node self-restarted, it could not reconstruct from the recipe-complete native token -> 404. Capture st before stripping it from the URL, verify it at the API boundary (streamtoken.Verify + SessionID match, mirroring the node's own check), and forward it as X-Silo-Stream-Token. Best-effort: a missing/invalid token never blocks the live proxy, and the token is still kept out of the forwarded URL and logs. * fix(playback): restart node ffmpeg on native remote audio switch A native audio-track switch on an offloaded/remote transcode was a no-op at the node yet returned 200 with a fresh URL: HandleChangeAudioTrack restarted ffmpeg only when the API owned a LOCAL TranscodeSession, so for an offloaded transcode the node kept serving the OLD audio (the node consults the token only on a session miss). The replacement URL was also minted from identity- only claims, so a later node restart 404'd. For the offloaded transcode case (detected via session.TranscodeNodeURL), POST a fresh /transcode/start to the node with the new AudioTrackIndex (handleStart tears down and restarts ffmpeg) and mint the replacement proxy URL from a full RecipeCard so reconstruct survives a node restart. The encode recipe is derived from the durable session target fields plus the file, mirroring HandleStartTranscode. A concrete SegmentDuration (playback.DefaultSegmentDuration) is embedded rather than 0: the node's token completeness gate treats SegmentDuration<=0 as incomplete and falls back to a recipe store the native path never populates, which would 404 on a node restart - the exact resilience this path provides. A failed node POST now surfaces 502 rather than a false 200. Remux and non-offloaded (local) transcode paths keep their prior identity-claim URLs unchanged. Known limitation: Session does not persist the original SegmentDuration or SubtitleTrackIndex/SubtitleBurnIn, so a remote audio switch resets subtitle selection to none and assumes the default segment length; a client that started with a non-default segment length will resegment on switch. Making that state durable on the session is a follow-up. * docs(playback): scrub stale deny-lease/revalidator comments The deny-lease revocation mechanism and its "central revalidator" were removed earlier in this branch, but four comments still described them as live (transcode_manager.go, noderecipe/store.go, streamtoken/token.go, proxy/server.go). Reword them to match the shipped behavior: ownership claims are re-resolved at reconstruct, the noderecipe store shares Redis only with the node-session tracker, and a sub-TTL hard cut depends on a node-side revocation mechanism that is deferred to a future PR. * fix(jellycompat): surface durable playback-session write failures DurableCompatPlaybackStore.Update applied the in-memory mutation and then swallowed every Postgres commit-failure path, returning nil. Callers that promise restart resilience (persistTranscodeRecipe's recipe write, the upstream-session binds in streams.go) were told the session was durably persisted when only the cache held it, so a transient DB hiccup could leave the next restart reloading a stale row (wrong audio track) or 404ing. updateDB now returns the genuine DB round-trip error (begin/query/unmarshal/ marshal/exec/commit); Update propagates it while still applying the in-memory mutation so live state stays correct. A nil pool and a genuinely absent/expired row remain best-effort (return nil) — only real infrastructure failures propagate, so existing rollback paths fire exactly when durability is lost. Part of #174 * fix(playback): re-inject stream token into proxied transcode manifests API-proxied remote transcode manifests dropped the reconstruct token from their segment URLs, so playback died after a node or API restart. When a remote transcode has no separate proxy node, the client loads its manifest via the API-local path; proxyToTranscodeNode strips the signed token ("st") from the forwarded URL (keeping it off node URLs and logs, forwarded only as the X-Silo-Stream-Token header), and the node builds relative segment URIs from that token-less query. The segment URLs the client received carried no token, and the proxy only re-attached the header when an incoming segment request already had "st" — which it never did — so a restart made those segments non-reconstructable and they 404'd. proxyToTranscodeNode now rewrites the manifest body at the boundary: every segment and #EXT-X-MAP init URI gets the client-facing, API-verified token re-appended (new playback.AppendManifestQueryParam helper), so the client's later segment fetches carry "st" again and reconstruct after a restart. The token still never reaches the node URL or its logs. Only 200 .m3u8 responses are rewritten (Content-Length corrected); segments stream through untouched. Part of #174 * fix(playback): preserve subtitle/cadence recipe across offloaded audio switch Switching audio on a remote (offloaded) transcode with burned-in subtitles silently dropped them, and reset a non-default segment cadence. The offloaded audio-switch restart rebuilt the node start request from Session state, but Session/SessionStreamState retained no subtitle or segment-duration state (only the live local ts.Opts() and the RecipeCard did), so the branch hard-coded SubtitleTrackIndex:-1, SubtitleBurnIn:false and SegmentDuration:Default — signing that altered recipe into the replacement stream token. An audio switch then changed bytes beyond audio selection, and any later reconstruct kept the wrong no-subtitle/wrong-cadence recipe. Persist the byte-affecting recipe on the session: SubtitleTrackIndex, SubtitleBurnIn and SegmentDuration are added to Session/SessionStreamState, populated at start (finalizeTranscodeStart) and on post-restart reconstruct (ReconstructSession from the card), carried forward on every audio-switch state update, and read back when rebuilding the offloaded node request and its recipe card. The restart now reproduces the exact live stream. Also resolves the M-4b non-default segment_duration reset. Part of #174 * fix(playback): serialize transcode spawn paths with a per-session lock Reconstruct was single-flighted only against other reconstructs, so a restart-driven segment reconstruct racing a quality/seek/audio fresh start could spawn two ffmpeg processes writing the same output directory at once — segment corruption, partial-write closes, orphaned processes, and skewed active-job accounting. The atomic register-after-spawn (GetOrRegister / the reconstruct compare-on-register) prevented a map leak but not the concurrent disk writers, because the losing path had already spawned. The dedicated transcode node had the same split between handleStart and spawnReconstruct. Add a refcounted per-session lifecycle lock to both TranscodeManager and the node Server, held across "check existing -> spawn -> register": - reconstruct (doReconstructTranscode / spawnReconstruct) re-checks under the lock and yields to any live session instead of spawning a duplicate; - the native and jellycompat fresh-start paths take the lock around their spawn+register (the native path also closes any session a reconstruct rebuilt in the meantime so its fresh ffmpeg is the sole writer); - the node handleStart holds it across teardown+spawn+register. The refcount drops the map entry once no path holds/waits, keeping it bounded. GetOrRegisterTranscodeSession is removed — the lock supersedes it and keeping a register-after-spawn primitive would invite reintroducing the race. Part of #174 * fix(playback): serialize restart re-spawn under the session lifecycle lock TranscodeSession.Restart() releases s.mu across cancel -> wait-for-done -> re-exec and spawns ffmpeg into opts.OutputDir without holding the per-session lifecycle lock. LockSessionLifecycle's contract (fresh start, restart, reconstruct) requires restart to hold it too, but all five callers invoked Restart unlocked: native audio-switch and segment-recovery, compat audio-switch and segment-recovery, and the transcode-node segment-recovery. A restart racing another restart (audio-switch vs segment-recovery) or a fresh-start/reconstruct could land two ffmpeg processes writing the same segment directory -- mixed timelines, init.mp4/segment mismatch, and an orphaned-but-still-writing ffmpeg -- the exact concurrent-writer corruption the lifecycle lock exists to prevent. Add RestartSessionLocked (TranscodeManager) and restartSessionLocked (node Server) that hold LockSessionLifecycle only across the cancel->respawn transition, re-check that the handle is still the live mapped session under the lock, and return ErrSessionSuperseded rather than re-spawning a stale handle. Route all five call sites through them. The lock is released before callers wait on segments so recovery latency is unchanged. Tests: gating (restart blocks until the lifecycle lock frees, then spawns), concurrent-restart serialization, and superseded re-check on both the manager (covers native + compat) and node lock owners. --------- Co-authored-by: Quick <31828688+Quick104@users.noreply.github.com>
752 lines
33 KiB
Go
752 lines
33 KiB
Go
package playback
|
||
|
||
import (
|
||
"context"
|
||
"errors"
|
||
"log/slog"
|
||
"net/http"
|
||
"os"
|
||
"path/filepath"
|
||
"runtime"
|
||
"strings"
|
||
"sync"
|
||
"time"
|
||
|
||
"golang.org/x/sync/singleflight"
|
||
)
|
||
|
||
// TranscodeRuntimeConfig is the subset of playback configuration the transcode
|
||
// manager needs to (re)start ffmpeg. It is a small, config-package-free struct so
|
||
// internal/playback does not import internal/config (avoiding an import cycle);
|
||
// each embedding handler adapts its own config snapshot into this shape.
|
||
type TranscodeRuntimeConfig struct {
|
||
TranscodeDir string
|
||
FFmpegPath string
|
||
HWAccel string
|
||
HWDevice string
|
||
}
|
||
|
||
// sessionReconstructor is the SessionManager capability used to re-register a
|
||
// session under an existing ID during reconstruct. *SessionManager implements it.
|
||
// RegisterReconstructedWithLimits additionally enforces the per-user admission
|
||
// caps so replaying a token cannot reconstruct past the concurrent stream /
|
||
// transcode limits a fresh StartSession would reject.
|
||
type sessionReconstructor interface {
|
||
RegisterReconstructed(s *Session) *Session
|
||
RegisterReconstructedWithLimits(ctx context.Context, s *Session) (*Session, error)
|
||
}
|
||
|
||
// TranscodeManager owns the transcode-session lifecycle shared by every playback
|
||
// front end (native API and jellycompat): the live in-memory transcode map, the
|
||
// recipe-card persistence used to reconstruct a session after a server restart,
|
||
// and the reconstruct machinery (single-flight + concurrency cap) that rebuilds a
|
||
// lost ffmpeg from a card. Both PlaybackHandlers embed one and delegate to it so
|
||
// the card lifetime rules, the reconstruct cap, and the node-affinity constraint
|
||
// live in exactly one place.
|
||
//
|
||
// Dependencies are injected as function fields so an embedding handler can wire
|
||
// them lazily from its own (often late-set) fields without an ordering hazard.
|
||
type TranscodeManager struct {
|
||
// Sessions re-registers a reconstructed session under its existing id.
|
||
Sessions sessionReconstructor
|
||
// Config returns the current transcode runtime config (ffmpeg path, dir,
|
||
// hwaccel) so operator changes apply to newly (re)started transcodes.
|
||
Config func() TranscodeRuntimeConfig
|
||
// LogSinkFn returns the ffmpeg log sink for reconstructed processes.
|
||
LogSinkFn func() FFmpegLogSink
|
||
// JWTSecretFn returns the bearer used for remote transcode-node DELETEs.
|
||
JWTSecretFn func() string
|
||
// OnFFmpegCrash is invoked when a reconstructed/local ffmpeg exits with an
|
||
// error so the embedding handler can tear down the playback session (keeping
|
||
// the card, so a resume can respawn). dead is the exact session that crashed;
|
||
// the handler passes it back through CloseTranscodeSessionIf so a successor
|
||
// reconstructed under the same id between the exit and teardown is not killed.
|
||
// No-op when nil.
|
||
OnFFmpegCrash func(ctx context.Context, sessionID string, dead *TranscodeSession)
|
||
// StartThrottler optionally starts the segment throttler for a (re)started
|
||
// transcode, reading the embedding handler's settings. No-op when nil.
|
||
StartThrottler func(ctx context.Context, ts *TranscodeSession)
|
||
|
||
transcodeMu sync.RWMutex
|
||
transcodes map[string]*TranscodeSession
|
||
|
||
// inFlightMu guards reconstructInFlight, the set of session ids whose ffmpeg
|
||
// is mid-reconstruct. Cleanup unions it with the live map so a dir being
|
||
// rebuilt right now is never reaped (token-carried reconstruction has no
|
||
// durable card index to consult instead).
|
||
inFlightMu sync.Mutex
|
||
reconstructInFlight map[string]struct{}
|
||
|
||
// reconstructGroup single-flights transcode reconstruction per session id so
|
||
// concurrent manifest/segment requests for a lost session spawn exactly one
|
||
// ffmpeg writing to the shared output directory, never a racing duplicate.
|
||
reconstructGroup singleflight.Group
|
||
// reconstructSem bounds how many transcodes may be reconstructed (ffmpeg
|
||
// re-spawned) at once. After a restart, every buffered client re-requests at
|
||
// once; without a cap that is a thundering herd of simultaneous cold-start
|
||
// ffmpeg launches. The semaphore paces the burst — sessions still all
|
||
// reconstruct, just not all in the same instant. Lazily sized on first use.
|
||
reconstructSemOnce sync.Once
|
||
reconstructSem chan struct{}
|
||
|
||
// lifecycleMu guards lifecycleLocks, the per-session mutexes that serialize
|
||
// every path which spawns ffmpeg into a session's output directory (fresh
|
||
// start, quality/audio restart, and reconstruct). reconstructGroup only
|
||
// single-flights reconstructs against each other; without this a reconstruct
|
||
// racing a fresh start could run two ffmpeg writers against the same dir.
|
||
lifecycleMu sync.Mutex
|
||
lifecycleLocks map[string]*lifecycleLock
|
||
}
|
||
|
||
// lifecycleLock is a refcounted per-session mutex. The refcount lets the manager
|
||
// drop the map entry once no path holds or waits on it, so the map does not grow
|
||
// unbounded across the lifetime of a long-running server.
|
||
type lifecycleLock struct {
|
||
mu sync.Mutex
|
||
refs int
|
||
}
|
||
|
||
// NewTranscodeManager returns a manager with its internal maps initialized. The
|
||
// caller wires the dependency function fields before use.
|
||
func NewTranscodeManager() *TranscodeManager {
|
||
return &TranscodeManager{
|
||
transcodes: make(map[string]*TranscodeSession),
|
||
reconstructInFlight: make(map[string]struct{}),
|
||
}
|
||
}
|
||
|
||
func (m *TranscodeManager) jwtSecret() string {
|
||
if m.JWTSecretFn == nil {
|
||
return ""
|
||
}
|
||
return m.JWTSecretFn()
|
||
}
|
||
|
||
func (m *TranscodeManager) logSink() FFmpegLogSink {
|
||
if m.LogSinkFn == nil {
|
||
return nil
|
||
}
|
||
return m.LogSinkFn()
|
||
}
|
||
|
||
func (m *TranscodeManager) runtimeConfig() TranscodeRuntimeConfig {
|
||
if m.Config == nil {
|
||
return TranscodeRuntimeConfig{TranscodeDir: filepath.Join(os.TempDir(), "silo-transcode")}
|
||
}
|
||
return m.Config()
|
||
}
|
||
|
||
// defaultReconstructConcurrency caps simultaneous transcode reconstructs when no
|
||
// explicit limit is configured. One in-flight ffmpeg launch per CPU paces the
|
||
// post-restart spawn burst without starving a host that genuinely ran many
|
||
// concurrent transcodes before the restart.
|
||
func defaultReconstructConcurrency() int {
|
||
if n := runtime.NumCPU(); n > 0 {
|
||
return n
|
||
}
|
||
return 4
|
||
}
|
||
|
||
// acquireReconstructSlot blocks until a reconstruct slot is free or the request
|
||
// context is canceled. It returns a release func and true on success, or a nil
|
||
// func and false if the caller gave up (so the burst does not queue work no one
|
||
// is waiting for). The semaphore is lazily initialized so struct-literal-built
|
||
// managers (tests) work without a constructor.
|
||
func (m *TranscodeManager) acquireReconstructSlot(ctx context.Context) (func(), bool) {
|
||
m.reconstructSemOnce.Do(func() {
|
||
if m.reconstructSem == nil {
|
||
m.reconstructSem = make(chan struct{}, defaultReconstructConcurrency())
|
||
}
|
||
})
|
||
select {
|
||
case m.reconstructSem <- struct{}{}:
|
||
return func() { <-m.reconstructSem }, true
|
||
case <-ctx.Done():
|
||
return nil, false
|
||
}
|
||
}
|
||
|
||
// GetTranscodeSession returns the live in-memory transcode session for sessionID,
|
||
// or nil if none is registered.
|
||
func (m *TranscodeManager) GetTranscodeSession(sessionID string) *TranscodeSession {
|
||
if m == nil {
|
||
return nil
|
||
}
|
||
m.transcodeMu.RLock()
|
||
defer m.transcodeMu.RUnlock()
|
||
return m.transcodes[sessionID]
|
||
}
|
||
|
||
// RegisterTranscodeSession inserts a freshly started transcode session into the
|
||
// live map. Used by the normal (non-reconstruct) start paths.
|
||
func (m *TranscodeManager) RegisterTranscodeSession(sessionID string, ts *TranscodeSession) {
|
||
m.transcodeMu.Lock()
|
||
m.transcodes[sessionID] = ts
|
||
m.transcodeMu.Unlock()
|
||
}
|
||
|
||
// LockSessionLifecycle acquires the per-session lifecycle mutex and returns a
|
||
// release func. Every path that spawns ffmpeg into a session's output directory
|
||
// (fresh start, restart, reconstruct) must hold it across "check existing → spawn
|
||
// → register" so two paths never run concurrent writers against the same dir. The
|
||
// lock is refcounted: the map entry is dropped once the last holder/waiter
|
||
// releases, so the map stays bounded.
|
||
func (m *TranscodeManager) LockSessionLifecycle(sessionID string) func() {
|
||
m.lifecycleMu.Lock()
|
||
if m.lifecycleLocks == nil {
|
||
m.lifecycleLocks = make(map[string]*lifecycleLock)
|
||
}
|
||
lk := m.lifecycleLocks[sessionID]
|
||
if lk == nil {
|
||
lk = &lifecycleLock{}
|
||
m.lifecycleLocks[sessionID] = lk
|
||
}
|
||
lk.refs++
|
||
m.lifecycleMu.Unlock()
|
||
|
||
lk.mu.Lock()
|
||
return func() {
|
||
lk.mu.Unlock()
|
||
m.lifecycleMu.Lock()
|
||
lk.refs--
|
||
if lk.refs == 0 {
|
||
delete(m.lifecycleLocks, sessionID)
|
||
}
|
||
m.lifecycleMu.Unlock()
|
||
}
|
||
}
|
||
|
||
// RestartSessionLocked re-spawns ts under the per-session lifecycle lock so a
|
||
// restart (audio-switch or segment-recovery) can never race a fresh start,
|
||
// reconstruct, or another restart into the same output directory — the
|
||
// concurrent-writer corruption the lifecycle lock exists to prevent. It holds
|
||
// the lock only across the cancel→respawn transition inside Restart and
|
||
// releases it before the caller waits on segments. Under the lock it confirms
|
||
// ts is still the live mapped session; if a concurrent teardown or reconstruct
|
||
// replaced it, the stale handle is not re-spawned and ErrSessionSuperseded is
|
||
// returned.
|
||
func (m *TranscodeManager) RestartSessionLocked(ctx context.Context, sessionID string, ts *TranscodeSession, seekSeconds float64, startSegment int) error {
|
||
unlock := m.LockSessionLifecycle(sessionID)
|
||
defer unlock()
|
||
if live := m.GetTranscodeSession(sessionID); live != ts {
|
||
return ErrSessionSuperseded
|
||
}
|
||
return ts.Restart(ctx, seekSeconds, startSegment)
|
||
}
|
||
|
||
// markReconstructing records that sessionID's ffmpeg is mid-reconstruct and
|
||
// returns a release func to clear it. Cleanup unions this set with the live map
|
||
// so a dir being rebuilt is never reaped before it registers.
|
||
func (m *TranscodeManager) markReconstructing(sessionID string) func() {
|
||
if m == nil || sessionID == "" {
|
||
return func() {}
|
||
}
|
||
m.inFlightMu.Lock()
|
||
if m.reconstructInFlight == nil {
|
||
m.reconstructInFlight = make(map[string]struct{})
|
||
}
|
||
m.reconstructInFlight[sessionID] = struct{}{}
|
||
m.inFlightMu.Unlock()
|
||
return func() {
|
||
m.inFlightMu.Lock()
|
||
delete(m.reconstructInFlight, sessionID)
|
||
m.inFlightMu.Unlock()
|
||
}
|
||
}
|
||
|
||
// SessionLoadStatus is the outcome of LoadOrReconstructSession, letting each
|
||
// handler render its own error shape (native vs jellycompat) without the manager
|
||
// touching the http response.
|
||
type SessionLoadStatus int
|
||
|
||
const (
|
||
// SessionLoaded: a live or reconstructed session is returned, ownership ok.
|
||
SessionLoaded SessionLoadStatus = iota
|
||
// SessionMissing: no live session and no usable card (genuine not-found).
|
||
SessionMissing
|
||
// SessionLoadFailed: the session backend errored (not a clean miss).
|
||
SessionLoadFailed
|
||
// SessionForbidden: a live session exists but belongs to another user.
|
||
SessionForbidden
|
||
)
|
||
|
||
// LoadOrReconstructSession is the single front door every serve handler uses to
|
||
// obtain a playback Session: it looks the session up via getSession and, on a
|
||
// not-found miss (e.g. after a restart), reconstructs it from the recipe card,
|
||
// re-binding ownership to the live caller. The two-factor ownership rule is
|
||
// preserved exactly — a live session with a non-zero, mismatched caller is
|
||
// refused; reconstruct itself refuses a zero/mismatched caller — so this widens
|
||
// no access. getSession is supplied by the caller (its SessionManager.GetSession)
|
||
// so the manager needs no direct handle on the manager type.
|
||
//
|
||
// card is the reconstruction recipe the caller decoded from the verified stream
|
||
// token the client presented (nil when the request carried no usable token).
|
||
// Under token-carried reconstruction it is the sole descriptor source — there is
|
||
// no shared per-session store to fall back on — so a not-found session with a nil
|
||
// card is a genuine miss.
|
||
func (m *TranscodeManager) LoadOrReconstructSession(ctx context.Context, getSession func(string) (*Session, error), sessionID string, requestUserID int, card *RecipeCard) (*Session, SessionLoadStatus) {
|
||
session, err := getSession(sessionID)
|
||
if err != nil {
|
||
if !errors.Is(err, ErrSessionNotFound) {
|
||
return nil, SessionLoadFailed
|
||
}
|
||
// A nil manager (documented optional on StreamHandler) cannot reconstruct,
|
||
// so a missing session is simply not-found rather than a panic.
|
||
if m == nil || card == nil {
|
||
return nil, SessionMissing
|
||
}
|
||
// Lost the in-memory session (e.g. restart): rebuild it from the token's
|
||
// recipe. ReconstructSession re-binds the session to the card owner and
|
||
// refuses a non-zero caller that mismatches it (a zero caller is allowed for
|
||
// the authless bearer routes), so a nil result here is a genuine not-found.
|
||
session = m.ReconstructSession(ctx, sessionID, requestUserID, *card)
|
||
if session == nil {
|
||
return nil, SessionMissing
|
||
}
|
||
return session, SessionLoaded
|
||
}
|
||
// Live session: enforce the existing ownership check. A zero caller is
|
||
// allowed (these routes treat the session UUID as a bearer when auth is
|
||
// optional); a non-zero mismatch is refused.
|
||
if requestUserID != 0 && session.UserID != requestUserID {
|
||
return nil, SessionForbidden
|
||
}
|
||
return session, SessionLoaded
|
||
}
|
||
|
||
// ReconstructSession rebuilds the in-memory playback Session from a persisted
|
||
// recipe card after the server lost its state (restart). It re-binds the session
|
||
// to the live authenticated caller and refuses if ownership cannot be confirmed.
|
||
// Returns the (re)registered session, or nil if reconstruct is not possible (no
|
||
// card, ownership mismatch, or unsupported session manager).
|
||
func (m *TranscodeManager) ReconstructSession(ctx context.Context, sessionID string, requestUserID int, card RecipeCard) *Session {
|
||
if m == nil || m.Sessions == nil {
|
||
return nil
|
||
}
|
||
if card.SessionID == "" || card.SessionID != sessionID {
|
||
// The token's recipe must be for the session id in the URL; a mismatch is
|
||
// a forged or stale request.
|
||
return nil
|
||
}
|
||
// Re-bind ownership to the card owner. A zero caller is allowed (the authless
|
||
// transcode delivery routes — HLS master.m3u8 / segment — treat the session
|
||
// UUID as the bearer credential when auth is optional); a non-zero caller that
|
||
// mismatches the card owner is refused. Either way the reconstructed session is
|
||
// bound to card.UserID, never to the request's user.
|
||
if requestUserID != 0 && requestUserID != card.UserID {
|
||
slog.Warn("transcode reconstruct ownership rejected",
|
||
"session", sessionID, "playback_session_id", sessionID,
|
||
"request_user", requestUserID, "card_user", card.UserID)
|
||
return nil
|
||
}
|
||
|
||
// An empty PlayMethod is a card written before direct/remux were
|
||
// reconstructable; treat it as a transcode (the only kind then persisted).
|
||
method := card.PlayMethod
|
||
if method == "" {
|
||
method = PlayTranscode
|
||
}
|
||
|
||
s := &Session{
|
||
ID: card.SessionID,
|
||
UserID: card.UserID,
|
||
ProfileID: card.ProfileID,
|
||
MediaFileID: card.MediaFileID,
|
||
PlayMethod: method,
|
||
BasePlayMethod: method,
|
||
TranscodeNodeURL: card.TranscodeNodeURL,
|
||
AudioTrackIndex: card.AudioTrackIndex,
|
||
TranscodeAudio: card.TranscodeAudio,
|
||
TargetResolution: card.TargetResolution,
|
||
TargetVideoCodec: card.TargetCodecVideo,
|
||
TargetAudioCodec: card.TargetCodecAudio,
|
||
TargetBitrateKbps: card.TargetBitrateKbps,
|
||
TranscodeHWAccel: card.HWAccel,
|
||
// Preserve the byte-affecting recipe so an audio switch after a restart
|
||
// rebuilds the same stream (subtitles/cadence) instead of dropping them.
|
||
SubtitleTrackIndex: card.SubtitleTrackIndex,
|
||
SubtitleBurnIn: card.SubtitleBurnIn,
|
||
SegmentDuration: card.SegmentDuration,
|
||
}
|
||
// Enforce the same per-user concurrency caps a fresh StartSession would, so a
|
||
// replayed token cannot reconstruct past the user's limit. Reconstructing the
|
||
// user's own surviving sessions still succeeds up to the cap; only the over-cap
|
||
// replay is rejected.
|
||
session, err := m.Sessions.RegisterReconstructedWithLimits(ctx, s)
|
||
if err != nil {
|
||
// A genuine over-cap rejection (the user is at their concurrent stream /
|
||
// transcode limit) must still refuse: a replayed token cannot reconstruct
|
||
// past the cap a fresh StartSession would enforce.
|
||
if errors.Is(err, ErrTooManyStreams) || errors.Is(err, ErrTooManyTranscodes) {
|
||
slog.Warn("playback session reconstruct refused by admission cap",
|
||
"session", sessionID, "playback_session_id", sessionID,
|
||
"user", card.UserID, "method", method, "error", err)
|
||
return nil
|
||
}
|
||
// Otherwise the limit provider itself could not be evaluated (e.g. a
|
||
// transient Postgres error during a post-restart reconstruct wave). Fail
|
||
// open and admit the session WITHOUT the limit gate: denying here would
|
||
// collapse a recoverable dependency error into a permanent 404 and stop
|
||
// playback for a user who is within their limits. The cap will re-apply on
|
||
// the next fresh StartSession once the provider recovers.
|
||
slog.Warn("playback session reconstruct admitting despite unevaluated limits (degraded; limit provider unavailable)",
|
||
"session", sessionID, "playback_session_id", sessionID,
|
||
"user", card.UserID, "method", method, "error", err)
|
||
session = m.Sessions.RegisterReconstructed(s)
|
||
}
|
||
slog.Info("playback session reconstructed from recipe card",
|
||
"session", sessionID, "playback_session_id", sessionID, "user", card.UserID, "method", method)
|
||
return session
|
||
}
|
||
|
||
// ReconstructTranscode rebuilds the in-memory TranscodeSession (and, if
|
||
// necessary, the ffmpeg process) for a session whose card survived a restart. It
|
||
// is only used for local/integrated transcodes (no transcode node URL).
|
||
//
|
||
// requestedSegment is the segment number the caller is fetching, or a negative
|
||
// value when there is no segment context (manifest path). When the client has
|
||
// advanced past the card's original start position, the rebuilt ffmpeg is spawned
|
||
// at that position so playback resumes near the requested segment instead of
|
||
// restarting from the original seek point and stalling while the segment-recovery
|
||
// machinery seeks forward.
|
||
//
|
||
// Reconstruction is single-flighted per session id: concurrent manifest and
|
||
// segment requests for the same lost session share one ffmpeg process rather than
|
||
// racing to spawn duplicates against the shared output directory. Spawns are
|
||
// additionally bounded by reconstructSem so a post-restart wave of buffered
|
||
// clients paces its ffmpeg launches instead of stampeding the host.
|
||
//
|
||
// NODE AFFINITY CONSTRAINT: this re-spawns ffmpeg on the LOCAL host. The playback
|
||
// SessionManager is per-process and not shared across API front-ends, but recipe
|
||
// cards are shared (Postgres). For an integrated transcode (empty
|
||
// TranscodeNodeURL) the card carries no owning-node identity, so if requests for
|
||
// one session are spread across multiple API front-ends WITHOUT sticky session
|
||
// affinity, each front-end that misses the in-memory session will reconstruct its
|
||
// OWN local ffmpeg — a split-brain with divergent segment dirs. Integrated
|
||
// transcode is therefore only safe single-front-end or with session affinity at
|
||
// the load balancer. Remote transcode-node sessions are unaffected: their
|
||
// non-empty TranscodeNodeURL routes every front-end to the same ffmpeg via the
|
||
// proxy path, so ReconstructTranscode is never reached for them.
|
||
//
|
||
// This constraint is currently documented, not enforced: a robust fix needs a
|
||
// per-session owning-instance claim in a store shared across front-ends (e.g.
|
||
// a shared Redis or the recipe store), so a front-end refuses to
|
||
// reconstruct an integrated session it does not own. The TranscodeManager has no
|
||
// such shared handle wired today — only per-process config/secret closures and
|
||
// in-memory maps — so the claim cannot be made cheaply here. Until a topology
|
||
// signal reaches the manager, deploy integrated transcode single-front-end or
|
||
// behind sticky session affinity. See M8.
|
||
// card is the reconstruction recipe decoded from the client's verified stream
|
||
// token; it carries the encode parameters formerly read from the Postgres store.
|
||
// Returns the live session, or nil if reconstruct was not possible.
|
||
func (m *TranscodeManager) ReconstructTranscode(ctx context.Context, sessionID string, requestedSegment int, card RecipeCard) *TranscodeSession {
|
||
if m == nil {
|
||
return nil
|
||
}
|
||
if card.SessionID == "" || card.SessionID != sessionID {
|
||
return nil
|
||
}
|
||
|
||
// A concurrent reconstruct may already have registered the session; serve it
|
||
// directly so we never enter single-flight only to discard a duplicate.
|
||
if existing := m.GetTranscodeSession(sessionID); existing != nil {
|
||
return existing
|
||
}
|
||
|
||
v, err, _ := m.reconstructGroup.Do(sessionID, func() (interface{}, error) {
|
||
return m.doReconstructTranscode(ctx, sessionID, requestedSegment, card), nil
|
||
})
|
||
if err != nil || v == nil {
|
||
return nil
|
||
}
|
||
session, _ := v.(*TranscodeSession)
|
||
return session
|
||
}
|
||
|
||
// fastResumeSeek decides whether a reconstructed ffmpeg should be spawned at the
|
||
// segment the client is actually requesting instead of the card's original
|
||
// start. Resuming near requestedSegment avoids a wait-then-seek-restart stall
|
||
// when the client has already played past the card position.
|
||
//
|
||
// The returned (segment, seekSeconds) maps via seg×SegmentDuration, which is
|
||
// ONLY valid for ENCODED transcodes: their forced keyframes make every segment
|
||
// exactly SegmentDuration long. COPY-mode segments inherit the source's variable
|
||
// GOP boundaries, so seg×dur lands on the wrong source time and desyncs A/V after
|
||
// a restart — so for copy-mode cards this returns ok=false and the caller keeps
|
||
// the card's original start, letting the manifest-driven segment recovery
|
||
// (RestartSeekTarget) seek forward once the rebuilt manifest exposes the real
|
||
// per-segment timing. A negative requestedSegment (manifest path, no segment
|
||
// context) and a non-advanced client also return ok=false.
|
||
func fastResumeSeek(card RecipeCard, requestedSegment int) (segment int, seekSeconds float64, ok bool) {
|
||
if strings.EqualFold(card.TargetCodecVideo, "copy") {
|
||
return 0, 0, false
|
||
}
|
||
if requestedSegment > card.StartSegmentNumber && card.SegmentDuration > 0 {
|
||
return requestedSegment, float64(requestedSegment * card.SegmentDuration), true
|
||
}
|
||
return 0, 0, false
|
||
}
|
||
|
||
// doReconstructTranscode performs the actual rebuild for a single reconstruct
|
||
// leader. It is only ever invoked inside reconstructGroup.Do, so it is the sole
|
||
// writer racing to register sessionID for this session.
|
||
func (m *TranscodeManager) doReconstructTranscode(ctx context.Context, sessionID string, requestedSegment int, card RecipeCard) *TranscodeSession {
|
||
// Only transcode cards drive ffmpeg reconstruction. Direct/remux sessions
|
||
// reconstruct without a runtime and must never reach here; guard so a
|
||
// direct/remux card ID cannot accidentally spawn an encode. An empty
|
||
// PlayMethod is back-compat for a token minted before the discriminator
|
||
// (transcode).
|
||
if card.PlayMethod != "" && card.PlayMethod != PlayTranscode {
|
||
return nil
|
||
}
|
||
|
||
// Mark in-flight for the whole rebuild so a concurrent cleanup never reaps the
|
||
// output dir between spawn and map registration.
|
||
release := m.markReconstructing(sessionID)
|
||
defer release()
|
||
|
||
cfg := m.runtimeConfig()
|
||
outputDir := filepath.Join(cfg.TranscodeDir, sessionID)
|
||
opts := card.TranscodeOpts(outputDir, cfg.FFmpegPath, m.logSink())
|
||
// Re-resolve environment-specific encode knobs from current config so an
|
||
// operator config change applies to reconstructed sessions too.
|
||
opts.HWAccel = cfg.HWAccel
|
||
opts.HWDevice = cfg.HWDevice
|
||
|
||
// Resume near the segment the client is actually requesting. The card records
|
||
// the original start; if the client has played past it, spawning ffmpeg at the
|
||
// old position forces a wait-then-seek-restart cycle (a visible stall). Seeking
|
||
// straight to requestedSegment avoids it. A negative requestedSegment (manifest
|
||
// path) carries no segment context, so the card position stands.
|
||
//
|
||
if seg, seek, ok := fastResumeSeek(card, requestedSegment); ok {
|
||
opts.StartSegmentNumber = seg
|
||
opts.SeekSeconds = seek
|
||
}
|
||
|
||
// Pace the spawn so a post-restart wave of reconstructs does not launch a
|
||
// thousand cold-start ffmpeg processes at once. A client that disconnects while
|
||
// waiting releases its place rather than queueing dead work.
|
||
slotRelease, ok := m.acquireReconstructSlot(ctx)
|
||
if !ok {
|
||
return nil
|
||
}
|
||
|
||
// Serialize against every other spawn path (fresh start, restart) for this
|
||
// session so a reconstruct and a fresh start never run two ffmpeg writers
|
||
// against the same output dir. reconstructGroup only single-flights reconstructs
|
||
// against each other, not against starts.
|
||
unlock := m.LockSessionLifecycle(sessionID)
|
||
defer unlock()
|
||
|
||
// Re-check under the lifecycle lock: a fresh start (or a reconstruct that ran
|
||
// just before us) may already have a live session. Yield to it instead of
|
||
// spawning a duplicate writer.
|
||
if existing := m.GetTranscodeSession(sessionID); existing != nil {
|
||
slotRelease()
|
||
return existing
|
||
}
|
||
|
||
transcodeSession, err := StartTranscode(context.WithoutCancel(ctx), opts)
|
||
slotRelease()
|
||
if err != nil {
|
||
slog.Error("reconstruct transcode start failed", "error", err, "session", sessionID, "playback_session_id", sessionID)
|
||
return nil
|
||
}
|
||
|
||
// Register under the map lock. The lifecycle lock guarantees no other path
|
||
// registered since the re-check above; the existing-check is kept as defensive
|
||
// belt-and-braces, closing only the duplicate ffmpeg process (never the shared
|
||
// output dir the winner serves) on the should-be-impossible race.
|
||
m.transcodeMu.Lock()
|
||
if existing := m.transcodes[sessionID]; existing != nil {
|
||
m.transcodeMu.Unlock()
|
||
_ = transcodeSession.CloseProcess()
|
||
return existing
|
||
}
|
||
m.transcodes[sessionID] = transcodeSession
|
||
m.transcodeMu.Unlock()
|
||
|
||
// Mirror the handler's start path: re-arm the throttler and exit monitor
|
||
// after every Restart of this reconstructed session, so seek/audio-switch
|
||
// restarts keep the same wiring as a freshly started transcode.
|
||
transcodeSession.SetRestartHook(func(ctx context.Context) {
|
||
if m.StartThrottler != nil {
|
||
m.StartThrottler(ctx, transcodeSession)
|
||
}
|
||
m.MonitorLocalTranscodeExit(sessionID, transcodeSession)
|
||
})
|
||
|
||
if m.StartThrottler != nil {
|
||
m.StartThrottler(ctx, transcodeSession)
|
||
}
|
||
m.MonitorLocalTranscodeExit(sessionID, transcodeSession)
|
||
slog.Info("transcode process reconstructed from recipe card",
|
||
"session", sessionID, "playback_session_id", sessionID,
|
||
"requested_segment", requestedSegment, "start_segment_number", opts.StartSegmentNumber)
|
||
return transcodeSession
|
||
}
|
||
|
||
// MonitorLocalTranscodeExit watches a local ffmpeg process and, on an error exit,
|
||
// invokes OnFFmpegCrash so the embedding handler tears down the playback session.
|
||
// A clean exit (no error) leaves the segments servable until the client stops.
|
||
func (m *TranscodeManager) MonitorLocalTranscodeExit(sessionID string, session *TranscodeSession) {
|
||
if m == nil || sessionID == "" || session == nil {
|
||
return
|
||
}
|
||
|
||
done := session.Done()
|
||
if done == nil {
|
||
return
|
||
}
|
||
|
||
go func() {
|
||
<-done
|
||
time.Sleep(2 * time.Second)
|
||
|
||
m.transcodeMu.RLock()
|
||
current := m.transcodes[sessionID]
|
||
m.transcodeMu.RUnlock()
|
||
if current != session {
|
||
return
|
||
}
|
||
if session.IsRunning() {
|
||
return
|
||
}
|
||
|
||
// When ffmpeg exits cleanly (no error), the segments are fully written and
|
||
// should remain servable until the client stops the session. This is
|
||
// critical for copy-mode where ffmpeg finishes writing all content much
|
||
// faster than real-time playback. Only tear down the session on error exits.
|
||
if session.WaitError() == nil {
|
||
return
|
||
}
|
||
|
||
// ffmpeg crash — tear the session down; a client holding a valid token can
|
||
// reconstruct it on the next request. Pass the dead session so teardown is a
|
||
// compare-and-delete: a reconstruct that registered a successor under this id
|
||
// between the current!=session check above and teardown must not be killed.
|
||
if m.OnFFmpegCrash != nil {
|
||
m.OnFFmpegCrash(context.Background(), sessionID, session)
|
||
}
|
||
}()
|
||
}
|
||
|
||
// CloseTranscodeSession stops a transcode session. If transcodeNodeURL is
|
||
// non-empty, sends DELETE to the remote transcode node. Otherwise closes the
|
||
// local session.
|
||
//
|
||
// Under token-carried reconstruction there is no durable card to drop: a stopped
|
||
// session simply stops being served, and its segment dir is reaped by the
|
||
// in-memory-liveness + age cleanup once no live token could still reconstruct it
|
||
// (see CleanupOrphanedTranscodes). A sub-TTL hard cut of an abusive stream
|
||
// before the token expires depends on a node-side revocation mechanism that is
|
||
// deferred to a future PR; today a stopped session can be reconstructed by a
|
||
// still-valid token until it expires.
|
||
func (m *TranscodeManager) CloseTranscodeSession(sessionID, transcodeNodeURL string) {
|
||
// Clean up local session if one exists (defensive).
|
||
m.transcodeMu.Lock()
|
||
session := m.transcodes[sessionID]
|
||
delete(m.transcodes, sessionID)
|
||
m.transcodeMu.Unlock()
|
||
if session != nil {
|
||
_ = session.Close()
|
||
}
|
||
|
||
m.deleteRemoteTranscode(sessionID, transcodeNodeURL)
|
||
}
|
||
|
||
// CloseTranscodeSessionIf tears down a transcode session only when the live map
|
||
// still holds the exact session the caller observed dying (expected). This is
|
||
// the crash path: between a local ffmpeg's error exit and this teardown, a
|
||
// concurrent reconstruct can register a fresh successor under the same id. An
|
||
// unconditional close would delete+Close() that live successor — and Close()
|
||
// removes the shared output dir out from under it. Comparing under the same lock
|
||
// that reconstruct registers through makes the swap atomic: a non-matching entry
|
||
// is left untouched. The remote-DELETE still fires for the matched case (and is
|
||
// skipped entirely when the local successor already won, since there is nothing
|
||
// of ours to stop).
|
||
//
|
||
// Returns true iff the live entry still matched expected and was torn down;
|
||
// false iff a different (successor) or nil session held the slot and was left
|
||
// untouched. Callers MUST treat this return as the authoritative gate for any
|
||
// further teardown (e.g. stopping the upstream playback session): when it is
|
||
// false, a successor owns the id and must not be disturbed.
|
||
func (m *TranscodeManager) CloseTranscodeSessionIf(sessionID string, expected *TranscodeSession, transcodeNodeURL string) bool {
|
||
m.transcodeMu.Lock()
|
||
current := m.transcodes[sessionID]
|
||
if current != expected {
|
||
// A successor (or an already-completed close) holds the slot; leave it.
|
||
m.transcodeMu.Unlock()
|
||
return false
|
||
}
|
||
delete(m.transcodes, sessionID)
|
||
m.transcodeMu.Unlock()
|
||
if expected != nil {
|
||
_ = expected.Close()
|
||
}
|
||
|
||
m.deleteRemoteTranscode(sessionID, transcodeNodeURL)
|
||
return true
|
||
}
|
||
|
||
// deleteRemoteTranscode sends DELETE to the assigned transcode node if any
|
||
// (synchronous with timeout). A no-op for local/integrated sessions.
|
||
func (m *TranscodeManager) deleteRemoteTranscode(sessionID, transcodeNodeURL string) {
|
||
if transcodeNodeURL != "" {
|
||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||
defer cancel()
|
||
|
||
deleteURL := transcodeNodeURL + "/transcode/" + sessionID
|
||
req, err := http.NewRequestWithContext(ctx, http.MethodDelete, deleteURL, nil)
|
||
if err != nil {
|
||
slog.Error("remote transcode delete: build request", "error", err, "session", sessionID, "playback_session_id", sessionID)
|
||
return
|
||
}
|
||
req.Header.Set("Authorization", "Bearer "+m.jwtSecret())
|
||
|
||
resp, err := http.DefaultClient.Do(req)
|
||
if err != nil {
|
||
slog.Warn("remote transcode delete failed", "error", err, "session", sessionID, "node", transcodeNodeURL, "playback_session_id", sessionID)
|
||
return
|
||
}
|
||
_ = resp.Body.Close()
|
||
if resp.StatusCode >= http.StatusMultipleChoices {
|
||
slog.Warn("remote transcode delete returned non-success status",
|
||
"status", resp.StatusCode, "session", sessionID, "node", transcodeNodeURL, "playback_session_id", sessionID)
|
||
}
|
||
}
|
||
}
|
||
|
||
// CleanupOrphanedTranscodes removes stale per-session temp directories for
|
||
// transcodes that are no longer reconstructable. Under token-carried
|
||
// reconstruction there is no durable card index to consult, so the liveness
|
||
// signal is: the in-process live transcode map, the set of sessions currently
|
||
// mid-reconstruct, and directory age. A dir is reaped only when it is absent from
|
||
// both sets AND older than the maximum token lifetime — past which no surviving
|
||
// token could reconstruct it. Each process owns its own TranscodeDir, so there is
|
||
// no cross-process enumeration-failure mode to fail safe against.
|
||
func (m *TranscodeManager) CleanupOrphanedTranscodes() (int, error) {
|
||
// Snapshot the live map and the in-flight set under both locks held at once.
|
||
// A reconstruct registers into m.transcodes and clears m.reconstructInFlight
|
||
// at different moments; snapshotting the two sets separately could miss a
|
||
// session that migrated between them, leaving its live dir absent from active
|
||
// and exposed to reaping. inFlightMu is taken first to match the only other
|
||
// site that holds both (none nests the reverse order).
|
||
m.inFlightMu.Lock()
|
||
m.transcodeMu.RLock()
|
||
active := make(map[string]struct{}, len(m.transcodes)+len(m.reconstructInFlight))
|
||
for sessionID := range m.transcodes {
|
||
active[sessionID] = struct{}{}
|
||
}
|
||
// Spare sessions mid-reconstruct: their dir is being written right now but is
|
||
// not yet registered in the live map.
|
||
for sessionID := range m.reconstructInFlight {
|
||
active[sessionID] = struct{}{}
|
||
}
|
||
m.transcodeMu.RUnlock()
|
||
m.inFlightMu.Unlock()
|
||
|
||
return CleanupOrphanedTranscodeDirs(m.runtimeConfig().TranscodeDir, active, MaxTokenTTL)
|
||
}
|