* docs(playback): add v3 neutral-contract finalization plan Supersedes the wire-contract sections of the 2026-07-12 v3 plan: server-owned attempt keys, delivery-keyed negotiation without Media3 engine names, tiered capability evidence, neutral device/output context, track/quality replan operations, audio-only planning, and coordinated no-back-compat rollout across server, Android, Apple, and web. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(playback): make v3 attempt keys server-owned and replace engines with deliveries Contract core of the platform-neutral v3 finalization (plan sections 3.1 and 3.2), breaking on purpose — v3 is dark and all clients move together: - Every PlanV3 now carries plan_attempt_key, an opaque server-computed token clients store and echo in attempted_plan_keys; ReplanRequestV3 gains bounded local_mutations that the replan handler folds into the failed plan's key. Clients never hash anything. - KotlinName() is deleted from DeliveryV3, StreamProtocolV3 and SubtitleModeV3; the attempt-key canonical string now uses lowercase wire tokens, and PlanRecipeVersionV3 bumps to v3.3 so no key or plan ID computed under the old canonicalization can collide. - EngineV3 leaves the wire: ClientPlaybackContextV3.Engines (media3_*) becomes Deliveries keyed original_http|progressive|hls, with EngineCapabilityV3 renamed DeliveryCapabilityV3. PlanV3.Engine is removed; the planner, subtitle policy and quirk registry re-key on delivery class, and the media3_only feature token is deleted. - Validated-claim strings drop the prefix: media3_h264_decode -> h264_decode, media3_audio_decode -> audio_decode. - Golden fixtures in testdata/protocol_v3 are regenerated by Go and are now the cross-repo source of truth. Part of the playback protocol v3 neutral-contract train (steps 2-3 of docs/superpowers/plans/2026-07-30-playback-protocol-v3-neutral-contract.md). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(playback): add v3 evidence tiers and neutral device/output context Implement plan sections 3.3 and 3.4 of the v3 neutral-contract pass: - ClientCodecCapabilitiesV3 gains required video_evidence and audio_evidence closed enums (exact | platform_attested | declared). Planner strictness follows the tier: exact keeps the strict decode-entry validation, platform_attested validates codec/resolution/bit-depth/ frame-rate but skips profile/level matching, declared grants copy routes from the flat codec lists. Only exact audio evidence earns passthrough claims. The detailed_decode_capabilities feature token is deleted (subsumed by video_evidence=exact), and evidence-blocked direct routes carry the new evidence_insufficient_for_direct reason/warning. - DeviceContextV3 is now platform/os_version/manufacturer/model plus a bounded platform_details map (<=16 entries, <=128 chars); the Android Build dump fields are gone. Fire TV quirks keep matching on manufacturer/model (brand fallback removed with the field). - output_route_generation (int64, dual-location) becomes an optional opaque output_context_id string on the output context; the dual-location consistency validation is deleted. Attempt keys, plan invalidation, route events, and the planstore column follow (new Goose migration). - Feature advertisement collapses to the top-level client_features list only; ClientPlaybackContextV3.Features is deleted and ReplanRequestV3 gains an optional client_features refresh. - PlanRecipeVersionV3 bumped v3.3 -> v3.4; fixtures re-keyed. Part of the playback protocol v3 neutral-contract finalization plan (docs/superpowers/plans/2026-07-30-playback-protocol-v3-neutral-contract.md). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(playback): add v3 intent replans, quality menu, and audio-only routes Protocol v3 could only replan after a failure, so changing the audio track or the quality still required the legacy audio PATCH and the client-recipe transcode start — the two endpoints v3 is meant to replace. Clients also had to own a resolution ladder to render a quality menu, and a source with no video track was terminaled by the video/HDR gates, keeping audiobooks on the legacy path. Add track_change and quality_change replan operations. They carry no failure classification and route through the existing replan transaction, so they inherit its idempotency, capacity reservation, and staged-successor commit for free. Because nothing failed, the previous route stays eligible: neither the attempted-key history nor the failed-plan exclusion applies to them. Publish the server ladder on the plan as available_qualities so the quality menu is server-owned; the rungs come from the same resolutionLabelV3 and ladderBitrateKbpsV3 helpers the planner itself uses, not a parallel table. Plan audio-only sources through their own reduced route family: original_http when the client decodes the codec, otherwise a progressive AAC conversion. The plan advertises audio/mp4 for that remux and the transport now serves the same value, because a declared-tier client probes the advertised MIME with isTypeSupported before attaching a source buffer, and "video/mp4" on a stream with no video track is exactly the mismatch that makes the probe lie. Name the protocol's string vocabulary (dynamic ranges, transformations, executors, validated claims, terminal reasons) as constants while touching these lines, so the wire values have one definition. Part of #135 * docs(playback): publish the v3 protocol contract and fix subtitle ordinals Protocol v3 exists only as Go code today, so the Android and Apple ports have no authority to implement against other than reading this repository. Publish the contract as a normative document, machine-checkable schemas, and generated golden fixtures, and fix the one place where the server's own wire output disagreed with the ordinal space it publishes. - docs/architecture/playback-protocol-v3.md is self-contained enough for a third-party client: endpoints and status codes, evidence tiers and their bound-matching rules, delivery classes, the timeline model, replan semantics, registries, track identity, plan identity, quality, and transformations. - docs/design/schemas/playback-v3/ carries JSON Schemas for the five wire shapes plus valid and invalid fixtures, following the client-diagnostics layout. internal/playback/contract validates every fixture against its schema, so a schema that drifts from the Go types fails the Go suite. - cmd/playbackfixtures generates internal/playback/testdata/protocol_v3 from the production planner. `make playback-fixtures` writes them and `make verify-playback-fixtures` (wired into CI) fails when they are stale. These files are what the client ports consume, so drift would otherwise surface as a playback bug on three platforms at once. The subtitle fix: combined ordinals are one dense space over externals, then embedded tracks, then downloaded ones, but the legacy URL builder skipped burn-in-only tracks while assigning indices, so every track after a DVD/DVB track was numbered one too low and resolved to its neighbour. Ordinal assignment now lives in playback.BuildSubtitleInventoryV3 and both the plan inventory and the legacy `subtitle_urls` shape project from it; the legacy shape still filters burn-in-only entries but keeps each track's real index. Part of #135 * feat(web): migrate the players to the neutral playback v3 contract The web player was the last client still speaking the legacy start protocol: it picked its own file version from a codec probe, posted an ffmpeg recipe to start a transcode, PATCHed an endpoint to change audio tracks, and derived its own quality ladder. None of that survives a server-owned plan, and none of it produced telemetry the apps could be compared against. Video player: starts with a v3 request that advertises `declared` evidence from `isTypeSupported` probes and the three delivery classes, then consumes the returned plan for its URL, timeline, tracks and warnings. Quality and track changes become replans (`quality_change`, `track_change`), the quality menu renders `available_qualities` instead of computing rungs, and playback failures emit `route-events` so web failures land in the same diagnostics as Android and Apple. The duration comes from `source.duration_seconds` rather than the playback engine, and the "how was this delivered" overlay reads the plan's delivery and server transformations instead of comparing codec strings. Audiobook player: starts against the audio-only planner path with a single `original` rung, and takes its seek anchor from `timeline.player_start_seconds` so the progressive-remux route (which anchors the stream and restarts the player clock at zero) does not seek twice. Server side, `disable_progress_persistence` left the wire, so the rule it encoded is now derived. Resume state is keyed on the item, but every part of a multipart presentation shares that key while carrying its own file-local clock — persisting part 4's position would store "12 minutes in" as the book's resume point. `PresentationPartTotal > 1` expresses that directly and generalizes to multipart movies and split episodes, and a client can no longer forget to ask or lie about it. `useTranscodeQuality` and the legacy response types are deleted, and `WEBTEST_KNOWN_FAILURES` loses the audiobook entry along with its fix. Part of #135 * feat(playback)!: make v3 the only playback protocol Protocol v3 shipped behind a flag, alongside the legacy start path it was designed to replace. Running both meant every planner change had to be made twice, in two shapes that disagree about who decides the route: the legacy body carried a decision the client had already made, while v3 asks the server to make it. This deletes the legacy half. Removed: - `handleStartPlaybackLegacy` and its request/response bodies. The `POST /playback/start` route stays, but the protocol-version dispatch envelope is now a strict v3 decode — a body that does not declare `protocol_version: 3` gets `426 client_upgrade_required` so an outdated app can render a clear "update required" state instead of misreading a plan. Deliberately not a `400`: the request may be well-formed for the protocol it was written against. - `POST /playback/transcode/start`, superseded by the `quality_change` replan operation, and `PATCH /playback/{session_id}/audio`, superseded by `track_change`. Both mutated a session without re-planning. - The shadow planner and both rollout settings rows. With v3 the only protocol, `playback.protocol_v3_enabled` would mean "no playback at all"; `playback.protocol_v3_shadow_enabled` gated a comparison against a path that no longer exists. `409 protocol_disabled` on route-events goes with them, and capability `enabled` is now constant `true` (the field stays — clients feature-detect against it). - Version-selection helpers in `internal/playback/resolver.go` that only legacy start reached. `Resolve`/`ClientCapabilities`/`PlayDecision` stay: downloads consumes them. `internal/jellycompat` has its own resolution surface and is untouched. Behaviour the legacy handlers owned and v3 now owns explicitly: series version and audio-track preferences are persisted on start and on a `track_change` replan (not on failure recovery, whose forced route is not a user choice); an omitted `start_position` resolves to the profile's saved resume point; and an omitted audio track resolves through the series preference, the profile audio language, then the library override. Both are settled before planning, because the plan's timeline is cut at the start position. Spec §2.2 documents this as "omission is a request, not a default". The encode-target clamp that lived in the deleted transcode handler is already enforced in the planner, twice — `availableQualitiesV3` omits rungs at or above the source height, and the encode path clamps `targetHeight` to it. Unchanged: progress, stop, HLS manifest and segment delivery, the realtime control socket, stream tokens and restart reconstruction, watch together, downloads, jellycompat. Every removal is recorded in the pre-lock removals table in docs/architecture/v1-scope.md. Part of #135 * fix(scanner): stop recording embedded cover art as a video track ffprobe reports embedded cover art as a video stream carrying disposition.attached_pic. convertProbeData appended every "video" stream to VideoTracks without consulting isMainVideoStream, the predicate that already existed for duration decisions, so the picture was persisted as a playable track. That misreports the file twice: - An audio file with a cover picks up a video track, so it no longer satisfies MediaFile.IsAudioOnly and the v3 planner routes an audiobook through the video path instead of planAudioOnlyV3. - When the picture is ordered ahead of the real stream, the flat codec_video/resolution/hdr columns describe the poster: a 954x720 h264 episode was stored as mjpeg 480x480. Filter attached_pic streams out of the track loop. The guard is the disposition flag, not the codec name, so a genuine MJPEG video is still probed as video — the library has one. Already-probed rows self-heal on the next playback: NeedsCriticalProbeRepair already reprobes tracks missing color_range, which covers 21 of the 23 affected rows, and applyProbeData overwrites VideoTracks wholesale. The remaining two need a rescan; nothing persisted records attached_pic, and keying repair off still-image codec names would reprobe the genuine MJPEG file on every playback forever. Part of the playback v3 neutral-contract work: it is what lets Android drop AUDIOBOOK_COVER_ART_CODECS, which fabricated decode support the client cannot honestly claim under video_evidence: "exact". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(playback): publish subtitle URLs even when playback starts with subtitles off The v3 plan's subtitle inventory is the authoritative track list a client builds its subtitle menu from, but the handler only rewrote it with session-scoped URLs when a track was actually selected. A start or replan that resolved to `subtitle.mode: "off"` therefore returned the planner's URL-less inventory, so a client whose picker reads the inventory had a menu it could not fetch anything from. The Cast path hits this every time: it starts with subtitles off and needs the receiver's text tracks up front. attachSubtitleArtifactV3 now scopes and publishes the inventory unconditionally and gates only the artifact stamping on the selection. Spec §8 records that the `url` on a sidecar entry does not depend on the current selection. Part of the v3 neutral-contract finalization. * chore(playback): reconcile neutral v3 with main * fix(playback): preserve subtitle intent across replans * fix(playback): retain subtitle inventory on adapted routes * fix(playback): software-decode High10 AVC for QSV * fix(playback): scale High10 frames before QSV upload * fix(playback): preserve empty subtitle inventories * fix(playback): freeze terminal attempt contract * chore(playback): name fixture contract tokens * fix(playback): close v3 conformance review gaps * chore(playback): name conformance category * fix(playback): complete v3 conformance contract * fix(playback): keep schema fixtures generated * fix(playback): emit schema-valid conformance arrays * fix(playback): omit empty replan failures * fix(web): omit empty replan failures * fix(playback): close neutral v3 contract gaps * fix(playback): harden v3 replan, transcode, and quality-ladder edge cases Review remediation for the neutral v3 cutover, server side: - A failed replan no longer overwrites the durable StartResponse with a terminal or advances the replan request ID; an idempotent start replay of a still-healthy session returns the original plan. - SoftwareVideoDecode is now derived inside the transcode layer from source facts (codec/profile/bit depth) carried on TranscodeOpts, so jellycompat, downloads, recipe-card reconstruction, and transcode nodes get the High10 software-decode fix, not just the v3 handler. video_to_h264 recipe version bumps to 2 so mixed-version node pools that would silently drop the flag fail validation instead. - Local transport startup shares the 30s ManifestStartupTimeout; a timeout with the process still running stays retryable and is no longer persisted as a durable terminal against the attempt. - Sparse replan bodies (failure_recovery et al) no longer reset a user-selected quality preference to auto; the empty-value guard now covers every operation. - availableQualitiesV3 publishes no fixed rungs when the source height is unknown, keeping the no-upscaling ladder contract. - The proxy remux path serves audio-only fMP4 as audio/mp4 via a new additive AudioOnly token claim, matching the integrated path. - Plain text subtitle sidecars accept any requested extension again (served as VTT), restoring the permissive v1 behavior; ASS and bitmap handling is unchanged. - The 4K-disallowed terminal message discloses when a lower-resolution alternate exists but was pinned away by quality "original". Part of #135. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(web): keep playback alive through failed replans and honest audio claims Review remediation for the neutral v3 cutover, web player: - A failed or refused replan no longer unmounts the player: the fatal error screen is reserved for loads with no adopted plan, and replan failures surface through the existing non-fatal replanError path. - changeQuality rolls its optimistic preference back when the replan is refused or errors, so a failed switch is not silently applied by the next unrelated replan and the menu shows the real active rung. - The capability probe now tests mp3/vorbis codecs and mp3/flac/ogg containers (MediaSource with a canPlayType fallback), restoring direct play for mp3 audiobooks instead of per-part AAC re-encodes. - Reanchor seeks issued while a replan is in flight coalesce and run when it settles instead of being silently dropped with the scrubber pinned to a phantom position. - Subtitle refresh/translation replans use the resume anchor while the media element has no metadata, so a subtitle_ready broadcast during startup no longer restarts a resumed stream at 0:00. - An exhausted failure-recovery chain sets a visible error instead of returning silently. Part of #135. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(playback): accept video-only and VP9 probe metadata Treat audio and video probe completeness independently so legitimate video-only assets converge without repeated ffprobe repair. Allow unknown codec profile/level metadata to fall through to server adaptation while preserving exact direct-decode constraints. Fixes #574 * fix(playback): address protocol v3 review findings * fix(playback): harden lease and probe repair decisions * fix(playback): close remaining v3 review gaps * fix(playback): recover failed transcode starts * fix(playback): address remaining review-bot findings on v3 replan and audio planning Server: - The deferred replan lease release is bounded by a 3s timeout so a saturated pool or DB outage cannot wedge a handler goroutine that holds the per-session store lock on an uncancellable context. - planAudioOnlyV3 honors the request bandwidth cap: an over-cap source skips the original_http direct route and converts to AAC with the same bandwidth_cap_applied warning and decision reason the video ladder uses. Unknown source bitrate never triggers the cap. - A copy-audio progressive plan rejected only by a per-delivery audio_decode_codecs subset retries as an AAC conversion instead of returning adaptation_unavailable, and the AAC recipe respects the delivery's max_channels. Web: - failure_recovery replans issued while another replan is in flight queue (superseding a pending seek reanchor) instead of being silently dropped with the fatal overlay already suppressed. - A terminal response to a fresh non-preserving start clears the previous plan and stops its session, so episode navigation cannot keep rendering the prior item under the new title. - A refused recovery replan for a transport-dead plan surfaces the error and re-arms the plan failure key, so transient recovery failures no longer strand an endless spinner; the audiobook player gets the same guard reset. - A track-less subtitle_translation_completed hands off to the refreshed persisted track once the inventory settles, clearing the live overlay, instead of pinning the synthetic live track forever. Part of #135. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(playback): reuse HLS transport for sidecar replans * fix(playback): stabilize copy HLS remount timeline * fix(playback): address v3 review findings * fix(playback): satisfy player contract types --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
830 lines
36 KiB
Go
830 lines
36 KiB
Go
package playback
|
||
|
||
import (
|
||
"context"
|
||
"errors"
|
||
"log/slog"
|
||
"net/http"
|
||
"os"
|
||
"path/filepath"
|
||
"runtime"
|
||
"strings"
|
||
"sync"
|
||
"time"
|
||
|
||
"golang.org/x/sync/singleflight"
|
||
)
|
||
|
||
// TranscodeRuntimeConfig is the subset of playback configuration the transcode
|
||
// manager needs to (re)start ffmpeg. It is a small, config-package-free struct so
|
||
// internal/playback does not import internal/config (avoiding an import cycle);
|
||
// each embedding handler adapts its own config snapshot into this shape.
|
||
type TranscodeRuntimeConfig struct {
|
||
TranscodeDir string
|
||
FFmpegPath string
|
||
HWAccel string
|
||
HWDevice string
|
||
}
|
||
|
||
// sessionReconstructor is the SessionManager capability used to re-register a
|
||
// session under an existing ID during reconstruct. *SessionManager implements it.
|
||
// RegisterReconstructedWithLimits additionally enforces the per-user admission
|
||
// caps so replaying a token cannot reconstruct past the concurrent stream /
|
||
// transcode limits a fresh StartSession would reject.
|
||
type sessionReconstructor interface {
|
||
RegisterReconstructed(s *Session) *Session
|
||
RegisterReconstructedWithLimits(ctx context.Context, s *Session) (*Session, error)
|
||
}
|
||
|
||
// TranscodeManager owns the transcode-session lifecycle shared by every playback
|
||
// front end (native API and jellycompat): the live in-memory transcode map, the
|
||
// recipe-card persistence used to reconstruct a session after a server restart,
|
||
// and the reconstruct machinery (single-flight + concurrency cap) that rebuilds a
|
||
// lost ffmpeg from a card. Both PlaybackHandlers embed one and delegate to it so
|
||
// the card lifetime rules, the reconstruct cap, and the node-affinity constraint
|
||
// live in exactly one place.
|
||
//
|
||
// Dependencies are injected as function fields so an embedding handler can wire
|
||
// them lazily from its own (often late-set) fields without an ordering hazard.
|
||
type TranscodeManager struct {
|
||
// Sessions re-registers a reconstructed session under its existing id.
|
||
Sessions sessionReconstructor
|
||
// Config returns the current transcode runtime config (ffmpeg path, dir,
|
||
// hwaccel) so operator changes apply to newly (re)started transcodes.
|
||
Config func() TranscodeRuntimeConfig
|
||
// LogSinkFn returns the ffmpeg log sink for reconstructed processes.
|
||
LogSinkFn func() FFmpegLogSink
|
||
// JWTSecretFn returns the bearer used for remote transcode-node DELETEs.
|
||
JWTSecretFn func() string
|
||
// OnFFmpegCrash is invoked when a reconstructed/local ffmpeg exits with an
|
||
// error so the embedding handler can tear down the playback session (keeping
|
||
// the card, so a resume can respawn). dead is the exact session that crashed;
|
||
// the handler passes it back through CloseTranscodeSessionIf so a successor
|
||
// reconstructed under the same id between the exit and teardown is not killed.
|
||
// No-op when nil.
|
||
OnFFmpegCrash func(ctx context.Context, sessionID string, dead *TranscodeSession)
|
||
// StartThrottler optionally starts the segment throttler for a (re)started
|
||
// transcode, reading the embedding handler's settings. No-op when nil.
|
||
StartThrottler func(ctx context.Context, ts *TranscodeSession)
|
||
|
||
transcodeMu sync.RWMutex
|
||
transcodes map[string]*TranscodeSession
|
||
|
||
// inFlightMu guards reconstructInFlight, the set of session ids whose ffmpeg
|
||
// is mid-reconstruct. Cleanup unions it with the live map so a dir being
|
||
// rebuilt right now is never reaped (token-carried reconstruction has no
|
||
// durable card index to consult instead).
|
||
inFlightMu sync.Mutex
|
||
reconstructInFlight map[string]struct{}
|
||
|
||
// reconstructGroup single-flights transcode reconstruction per session id so
|
||
// concurrent manifest/segment requests for a lost session spawn exactly one
|
||
// ffmpeg writing to the shared output directory, never a racing duplicate.
|
||
reconstructGroup singleflight.Group
|
||
// reconstructSem bounds how many transcodes may be reconstructed (ffmpeg
|
||
// re-spawned) at once. After a restart, every buffered client re-requests at
|
||
// once; without a cap that is a thundering herd of simultaneous cold-start
|
||
// ffmpeg launches. The semaphore paces the burst — sessions still all
|
||
// reconstruct, just not all in the same instant. Lazily sized on first use.
|
||
reconstructSemOnce sync.Once
|
||
reconstructSem chan struct{}
|
||
|
||
// lifecycleMu guards lifecycleLocks, the per-session mutexes that serialize
|
||
// every path which spawns ffmpeg into a session's output directory (fresh
|
||
// start, quality/audio restart, and reconstruct). reconstructGroup only
|
||
// single-flights reconstructs against each other; without this a reconstruct
|
||
// racing a fresh start could run two ffmpeg writers against the same dir.
|
||
lifecycleMu sync.Mutex
|
||
lifecycleLocks map[string]*lifecycleLock
|
||
}
|
||
|
||
// lifecycleLock is a refcounted per-session mutex. The refcount lets the manager
|
||
// drop the map entry once no path holds or waits on it, so the map does not grow
|
||
// unbounded across the lifetime of a long-running server.
|
||
type lifecycleLock struct {
|
||
mu sync.Mutex
|
||
refs int
|
||
}
|
||
|
||
// NewTranscodeManager returns a manager with its internal maps initialized. The
|
||
// caller wires the dependency function fields before use.
|
||
func NewTranscodeManager() *TranscodeManager {
|
||
return &TranscodeManager{
|
||
transcodes: make(map[string]*TranscodeSession),
|
||
reconstructInFlight: make(map[string]struct{}),
|
||
}
|
||
}
|
||
|
||
func (m *TranscodeManager) jwtSecret() string {
|
||
if m.JWTSecretFn == nil {
|
||
return ""
|
||
}
|
||
return m.JWTSecretFn()
|
||
}
|
||
|
||
func (m *TranscodeManager) logSink() FFmpegLogSink {
|
||
if m.LogSinkFn == nil {
|
||
return nil
|
||
}
|
||
return m.LogSinkFn()
|
||
}
|
||
|
||
func (m *TranscodeManager) runtimeConfig() TranscodeRuntimeConfig {
|
||
if m.Config == nil {
|
||
return TranscodeRuntimeConfig{TranscodeDir: filepath.Join(os.TempDir(), "silo-transcode")}
|
||
}
|
||
return m.Config()
|
||
}
|
||
|
||
// defaultReconstructConcurrency caps simultaneous transcode reconstructs when no
|
||
// explicit limit is configured. One in-flight ffmpeg launch per CPU paces the
|
||
// post-restart spawn burst without starving a host that genuinely ran many
|
||
// concurrent transcodes before the restart.
|
||
func defaultReconstructConcurrency() int {
|
||
if n := runtime.NumCPU(); n > 0 {
|
||
return n
|
||
}
|
||
return 4
|
||
}
|
||
|
||
// acquireReconstructSlot blocks until a reconstruct slot is free or the request
|
||
// context is canceled. It returns a release func and true on success, or a nil
|
||
// func and false if the caller gave up (so the burst does not queue work no one
|
||
// is waiting for). The semaphore is lazily initialized so struct-literal-built
|
||
// managers (tests) work without a constructor.
|
||
func (m *TranscodeManager) acquireReconstructSlot(ctx context.Context) (func(), bool) {
|
||
m.reconstructSemOnce.Do(func() {
|
||
if m.reconstructSem == nil {
|
||
m.reconstructSem = make(chan struct{}, defaultReconstructConcurrency())
|
||
}
|
||
})
|
||
select {
|
||
case m.reconstructSem <- struct{}{}:
|
||
return func() { <-m.reconstructSem }, true
|
||
case <-ctx.Done():
|
||
return nil, false
|
||
}
|
||
}
|
||
|
||
// GetTranscodeSession returns the live in-memory transcode session for sessionID,
|
||
// or nil if none is registered.
|
||
func (m *TranscodeManager) GetTranscodeSession(sessionID string) *TranscodeSession {
|
||
if m == nil {
|
||
return nil
|
||
}
|
||
m.transcodeMu.RLock()
|
||
defer m.transcodeMu.RUnlock()
|
||
return m.transcodes[sessionID]
|
||
}
|
||
|
||
// RegisterTranscodeSession inserts a freshly started transcode session into the
|
||
// live map. Used by the normal (non-reconstruct) start paths.
|
||
func (m *TranscodeManager) RegisterTranscodeSession(sessionID string, ts *TranscodeSession) {
|
||
m.transcodeMu.Lock()
|
||
m.transcodes[sessionID] = ts
|
||
m.transcodeMu.Unlock()
|
||
}
|
||
|
||
// SwapTranscodeSession atomically publishes a prepared successor and returns
|
||
// the predecessor without closing it. Protocol-v3 replans use plan-scoped
|
||
// output directories, so the caller can commit state first, publish the new
|
||
// process, and only then reap the old process without either process writing to
|
||
// the other's directory.
|
||
func (m *TranscodeManager) SwapTranscodeSession(sessionID string, successor *TranscodeSession) *TranscodeSession {
|
||
m.transcodeMu.Lock()
|
||
predecessor := m.transcodes[sessionID]
|
||
m.transcodes[sessionID] = successor
|
||
m.transcodeMu.Unlock()
|
||
return predecessor
|
||
}
|
||
|
||
// SwapTranscodeSessionIf publishes successor only while the live map still
|
||
// contains expected. Callers use it after staging a process in a distinct
|
||
// output directory so a stale replacement cannot overwrite a newer process.
|
||
func (m *TranscodeManager) SwapTranscodeSessionIf(
|
||
sessionID string,
|
||
expected *TranscodeSession,
|
||
successor *TranscodeSession,
|
||
) bool {
|
||
m.transcodeMu.Lock()
|
||
defer m.transcodeMu.Unlock()
|
||
if m.transcodes[sessionID] != expected {
|
||
return false
|
||
}
|
||
m.transcodes[sessionID] = successor
|
||
return true
|
||
}
|
||
|
||
// StopRemoteTranscode removes only the remote node process. It deliberately
|
||
// leaves the local live-map entry untouched for an atomic remote-to-local v3
|
||
// replacement.
|
||
func (m *TranscodeManager) StopRemoteTranscode(sessionID, transcodeNodeURL string) {
|
||
m.deleteRemoteTranscode(sessionID, transcodeNodeURL)
|
||
}
|
||
|
||
// LockSessionLifecycle acquires the per-session lifecycle mutex and returns a
|
||
// release func. Every path that spawns ffmpeg into a session's output directory
|
||
// (fresh start, restart, reconstruct) must hold it across "check existing → spawn
|
||
// → register" so two paths never run concurrent writers against the same dir. The
|
||
// lock is refcounted: the map entry is dropped once the last holder/waiter
|
||
// releases, so the map stays bounded.
|
||
func (m *TranscodeManager) LockSessionLifecycle(sessionID string) func() {
|
||
m.lifecycleMu.Lock()
|
||
if m.lifecycleLocks == nil {
|
||
m.lifecycleLocks = make(map[string]*lifecycleLock)
|
||
}
|
||
lk := m.lifecycleLocks[sessionID]
|
||
if lk == nil {
|
||
lk = &lifecycleLock{}
|
||
m.lifecycleLocks[sessionID] = lk
|
||
}
|
||
lk.refs++
|
||
m.lifecycleMu.Unlock()
|
||
|
||
lk.mu.Lock()
|
||
return func() {
|
||
lk.mu.Unlock()
|
||
m.lifecycleMu.Lock()
|
||
lk.refs--
|
||
if lk.refs == 0 {
|
||
delete(m.lifecycleLocks, sessionID)
|
||
}
|
||
m.lifecycleMu.Unlock()
|
||
}
|
||
}
|
||
|
||
// RestartSessionLocked re-spawns ts under the per-session lifecycle lock so a
|
||
// restart (audio-switch or segment-recovery) can never race a fresh start,
|
||
// reconstruct, or another restart into the same output directory — the
|
||
// concurrent-writer corruption the lifecycle lock exists to prevent. It holds
|
||
// the lock only across the cancel→respawn transition inside Restart and
|
||
// releases it before the caller waits on segments. Under the lock it confirms
|
||
// ts is still the live mapped session; if a concurrent teardown or reconstruct
|
||
// replaced it, the stale handle is not re-spawned and ErrSessionSuperseded is
|
||
// returned.
|
||
func (m *TranscodeManager) RestartSessionLocked(ctx context.Context, sessionID string, ts *TranscodeSession, seekSeconds float64, startSegment int) error {
|
||
return m.restartSessionLocked(sessionID, ts, func() error {
|
||
return ts.Restart(ctx, seekSeconds, startSegment)
|
||
})
|
||
}
|
||
|
||
func (m *TranscodeManager) restartSessionLocked(sessionID string, ts *TranscodeSession, restart func() error) error {
|
||
unlock := m.LockSessionLifecycle(sessionID)
|
||
defer unlock()
|
||
if live := m.GetTranscodeSession(sessionID); live != ts {
|
||
return ErrSessionSuperseded
|
||
}
|
||
return restart()
|
||
}
|
||
|
||
// RestartSessionLockedWithCopySeekAnchor is RestartSessionLocked with the
|
||
// resolved keyframe origin for a legacy copy-video seek restart.
|
||
func (m *TranscodeManager) RestartSessionLockedWithCopySeekAnchor(
|
||
ctx context.Context,
|
||
sessionID string,
|
||
ts *TranscodeSession,
|
||
seekSeconds float64,
|
||
startSegment int,
|
||
streamOriginSeconds float64,
|
||
) error {
|
||
return m.restartSessionLocked(sessionID, ts, func() error {
|
||
return ts.RestartWithCopySeekAnchor(ctx, seekSeconds, startSegment, streamOriginSeconds)
|
||
})
|
||
}
|
||
|
||
// markReconstructing records that sessionID's ffmpeg is mid-reconstruct and
|
||
// returns a release func to clear it. Cleanup unions this set with the live map
|
||
// so a dir being rebuilt is never reaped before it registers.
|
||
func (m *TranscodeManager) markReconstructing(sessionID string) func() {
|
||
if m == nil || sessionID == "" {
|
||
return func() {}
|
||
}
|
||
m.inFlightMu.Lock()
|
||
if m.reconstructInFlight == nil {
|
||
m.reconstructInFlight = make(map[string]struct{})
|
||
}
|
||
m.reconstructInFlight[sessionID] = struct{}{}
|
||
m.inFlightMu.Unlock()
|
||
return func() {
|
||
m.inFlightMu.Lock()
|
||
delete(m.reconstructInFlight, sessionID)
|
||
m.inFlightMu.Unlock()
|
||
}
|
||
}
|
||
|
||
// SessionLoadStatus is the outcome of LoadOrReconstructSession, letting each
|
||
// handler render its own error shape (native vs jellycompat) without the manager
|
||
// touching the http response.
|
||
type SessionLoadStatus int
|
||
|
||
const (
|
||
// SessionLoaded: a live or reconstructed session is returned, ownership ok.
|
||
SessionLoaded SessionLoadStatus = iota
|
||
// SessionMissing: no live session and no usable card (genuine not-found).
|
||
SessionMissing
|
||
// SessionLoadFailed: the session backend errored (not a clean miss).
|
||
SessionLoadFailed
|
||
// SessionForbidden: a live session exists but belongs to another user.
|
||
SessionForbidden
|
||
)
|
||
|
||
// LoadOrReconstructSession is the single front door every serve handler uses to
|
||
// obtain a playback Session: it looks the session up via getSession and, on a
|
||
// not-found miss (e.g. after a restart), reconstructs it from the recipe card,
|
||
// re-binding ownership to the live caller. The two-factor ownership rule is
|
||
// preserved exactly — a live session with a non-zero, mismatched caller is
|
||
// refused; reconstruct itself refuses a zero/mismatched caller — so this widens
|
||
// no access. getSession is supplied by the caller (its SessionManager.GetSession)
|
||
// so the manager needs no direct handle on the manager type.
|
||
//
|
||
// card is the reconstruction recipe the caller decoded from the verified stream
|
||
// token the client presented (nil when the request carried no usable token).
|
||
// Under token-carried reconstruction it is the sole descriptor source — there is
|
||
// no shared per-session store to fall back on — so a not-found session with a nil
|
||
// card is a genuine miss.
|
||
func (m *TranscodeManager) LoadOrReconstructSession(ctx context.Context, getSession func(string) (*Session, error), sessionID string, requestUserID int, card *RecipeCard) (*Session, SessionLoadStatus) {
|
||
session, err := getSession(sessionID)
|
||
if err != nil {
|
||
if !errors.Is(err, ErrSessionNotFound) {
|
||
return nil, SessionLoadFailed
|
||
}
|
||
// A nil manager (documented optional on StreamHandler) cannot reconstruct,
|
||
// so a missing session is simply not-found rather than a panic.
|
||
if m == nil || card == nil {
|
||
return nil, SessionMissing
|
||
}
|
||
// Lost the in-memory session (e.g. restart): rebuild it from the token's
|
||
// recipe. ReconstructSession re-binds the session to the card owner and
|
||
// refuses a non-zero caller that mismatches it (a zero caller is allowed for
|
||
// the authless bearer routes), so a nil result here is a genuine not-found.
|
||
session = m.ReconstructSession(ctx, sessionID, requestUserID, *card)
|
||
if session == nil {
|
||
return nil, SessionMissing
|
||
}
|
||
return session, SessionLoaded
|
||
}
|
||
// Live session: enforce the existing ownership check. A zero caller is
|
||
// allowed (these routes treat the session UUID as a bearer when auth is
|
||
// optional); a non-zero mismatch is refused.
|
||
if requestUserID != 0 && session.UserID != requestUserID {
|
||
return nil, SessionForbidden
|
||
}
|
||
return session, SessionLoaded
|
||
}
|
||
|
||
// ReconstructSession rebuilds the in-memory playback Session from a persisted
|
||
// recipe card after the server lost its state (restart). It re-binds the session
|
||
// to the live authenticated caller and refuses if ownership cannot be confirmed.
|
||
// Returns the (re)registered session, or nil if reconstruct is not possible (no
|
||
// card, ownership mismatch, or unsupported session manager).
|
||
func (m *TranscodeManager) ReconstructSession(ctx context.Context, sessionID string, requestUserID int, card RecipeCard) *Session {
|
||
if m == nil || m.Sessions == nil {
|
||
return nil
|
||
}
|
||
if card.SessionID == "" || card.SessionID != sessionID {
|
||
// The token's recipe must be for the session id in the URL; a mismatch is
|
||
// a forged or stale request.
|
||
return nil
|
||
}
|
||
// Re-bind ownership to the card owner. A zero caller is allowed (the authless
|
||
// transcode delivery routes — HLS master.m3u8 / segment — treat the session
|
||
// UUID as the bearer credential when auth is optional); a non-zero caller that
|
||
// mismatches the card owner is refused. Either way the reconstructed session is
|
||
// bound to card.UserID, never to the request's user.
|
||
if requestUserID != 0 && requestUserID != card.UserID {
|
||
slog.WarnContext(ctx, "transcode reconstruct ownership rejected", "component", "playback",
|
||
"session", sessionID, "playback_session_id", sessionID,
|
||
"request_user", requestUserID, "card_user", card.UserID)
|
||
return nil
|
||
}
|
||
|
||
// An empty PlayMethod is a card written before direct/remux were
|
||
// reconstructable; treat it as a transcode (the only kind then persisted).
|
||
method := card.PlayMethod
|
||
if method == "" {
|
||
method = PlayTranscode
|
||
}
|
||
|
||
s := &Session{
|
||
ID: card.SessionID,
|
||
UserID: card.UserID,
|
||
ProfileID: card.ProfileID,
|
||
MediaFileID: card.MediaFileID,
|
||
PlayMethod: method,
|
||
BasePlayMethod: method,
|
||
TranscodeNodeURL: card.TranscodeNodeURL,
|
||
TranscodeTransportID: card.TranscodeTransportID,
|
||
AudioTrackIndex: card.AudioTrackIndex,
|
||
TranscodeAudio: card.TranscodeAudio,
|
||
RemuxDVMode: card.RemuxDVMode,
|
||
TargetResolution: card.TargetResolution,
|
||
TargetVideoCodec: card.TargetCodecVideo,
|
||
TargetAudioCodec: card.TargetCodecAudio,
|
||
TargetAudioChannels: card.TargetAudioChannels,
|
||
TargetAudioBitrateKbps: card.TargetAudioBitrateKbps,
|
||
TargetBitrateKbps: card.TargetBitrateKbps,
|
||
TranscodeHWAccel: card.HWAccel,
|
||
// Client metadata survives the restart so the admin views keep the
|
||
// client label and Jellyfin identification for the session's lifetime.
|
||
ClientName: normalizeClientMetadataValue(card.ClientName, 128),
|
||
ClientVersion: normalizeClientMetadataValue(card.ClientVersion, 64),
|
||
ClientUserAgent: normalizeClientMetadataValue(card.ClientUserAgent, 512),
|
||
IsJellyfinCompat: card.IsJellyfinCompat,
|
||
// Preserve the byte-affecting recipe so an audio switch after a restart
|
||
// rebuilds the same stream (subtitles/cadence) instead of dropping them.
|
||
SubtitleTrackIndex: card.SubtitleTrackIndex,
|
||
SubtitleBurnIn: card.SubtitleBurnIn,
|
||
SegmentDuration: card.SegmentDuration,
|
||
}
|
||
// Enforce the same per-user concurrency caps a fresh StartSession would, so a
|
||
// replayed token cannot reconstruct past the user's limit. Reconstructing the
|
||
// user's own surviving sessions still succeeds up to the cap; only the over-cap
|
||
// replay is rejected.
|
||
session, err := m.Sessions.RegisterReconstructedWithLimits(ctx, s)
|
||
if err != nil {
|
||
// Admission denials must still refuse: a replayed token cannot reconstruct
|
||
// past a current cap or after transcoding has been disabled for the user.
|
||
if errors.Is(err, ErrTooManyStreams) || errors.Is(err, ErrTooManyTranscodes) || errors.Is(err, ErrTranscodingDisabled) || errors.Is(err, ErrAudioTranscodingDisabled) {
|
||
slog.WarnContext(ctx, "playback session reconstruct refused by admission policy", "component", "playback",
|
||
"session", sessionID, "playback_session_id", sessionID,
|
||
"user", card.UserID, "method", method, "error", err)
|
||
return nil
|
||
}
|
||
// Otherwise the limit provider itself could not be evaluated (e.g. a
|
||
// transient Postgres error during a post-restart reconstruct wave). Fail
|
||
// open and admit the session WITHOUT the limit gate: denying here would
|
||
// collapse a recoverable dependency error into a permanent 404 and stop
|
||
// playback for a user who is within their limits. The cap will re-apply on
|
||
// the next fresh StartSession once the provider recovers.
|
||
slog.WarnContext(ctx, "playback session reconstruct admitting despite unevaluated limits (degraded; limit provider unavailable)", "component", "playback",
|
||
"session", sessionID, "playback_session_id", sessionID,
|
||
"user", card.UserID, "method", method, "error", err)
|
||
session = m.Sessions.RegisterReconstructed(s)
|
||
}
|
||
slog.InfoContext(ctx, "playback session reconstructed from recipe card", "component", "playback",
|
||
"session", sessionID, "playback_session_id", sessionID, "user", card.UserID, "method", method)
|
||
return session
|
||
}
|
||
|
||
// ReconstructTranscode rebuilds the in-memory TranscodeSession (and, if
|
||
// necessary, the ffmpeg process) for a session whose card survived a restart. It
|
||
// is only used for local/integrated transcodes (no transcode node URL).
|
||
//
|
||
// requestedSegment is the segment number the caller is fetching, or a negative
|
||
// value when there is no segment context (manifest path). When the client has
|
||
// advanced past the card's original start position, the rebuilt ffmpeg is spawned
|
||
// at that position so playback resumes near the requested segment instead of
|
||
// restarting from the original seek point and stalling while the segment-recovery
|
||
// machinery seeks forward.
|
||
//
|
||
// Reconstruction is single-flighted per session id: concurrent manifest and
|
||
// segment requests for the same lost session share one ffmpeg process rather than
|
||
// racing to spawn duplicates against the shared output directory. Spawns are
|
||
// additionally bounded by reconstructSem so a post-restart wave of buffered
|
||
// clients paces its ffmpeg launches instead of stampeding the host.
|
||
//
|
||
// NODE AFFINITY CONSTRAINT: this re-spawns ffmpeg on the LOCAL host. The playback
|
||
// SessionManager is per-process and not shared across API front-ends, but recipe
|
||
// cards are shared (Postgres). For an integrated transcode (empty
|
||
// TranscodeNodeURL) the card carries no owning-node identity, so if requests for
|
||
// one session are spread across multiple API front-ends WITHOUT sticky session
|
||
// affinity, each front-end that misses the in-memory session will reconstruct its
|
||
// OWN local ffmpeg — a split-brain with divergent segment dirs. Integrated
|
||
// transcode is therefore only safe single-front-end or with session affinity at
|
||
// the load balancer. Remote transcode-node sessions are unaffected: their
|
||
// non-empty TranscodeNodeURL routes every front-end to the same ffmpeg via the
|
||
// proxy path, so ReconstructTranscode is never reached for them.
|
||
//
|
||
// This constraint is currently documented, not enforced: a robust fix needs a
|
||
// per-session owning-instance claim in a store shared across front-ends (e.g.
|
||
// a shared Redis or the recipe store), so a front-end refuses to
|
||
// reconstruct an integrated session it does not own. The TranscodeManager has no
|
||
// such shared handle wired today — only per-process config/secret closures and
|
||
// in-memory maps — so the claim cannot be made cheaply here. Until a topology
|
||
// signal reaches the manager, deploy integrated transcode single-front-end or
|
||
// behind sticky session affinity. See M8.
|
||
// card is the reconstruction recipe decoded from the client's verified stream
|
||
// token; it carries the encode parameters formerly read from the Postgres store.
|
||
// Returns the live session, or nil if reconstruct was not possible.
|
||
func (m *TranscodeManager) ReconstructTranscode(ctx context.Context, sessionID string, requestedSegment int, card RecipeCard) *TranscodeSession {
|
||
if m == nil {
|
||
return nil
|
||
}
|
||
if card.SessionID == "" || card.SessionID != sessionID {
|
||
return nil
|
||
}
|
||
|
||
// A concurrent reconstruct may already have registered the session; serve it
|
||
// directly so we never enter single-flight only to discard a duplicate.
|
||
if existing := m.GetTranscodeSession(sessionID); existing != nil {
|
||
return existing
|
||
}
|
||
|
||
v, err, _ := m.reconstructGroup.Do(sessionID, func() (interface{}, error) {
|
||
return m.doReconstructTranscode(ctx, sessionID, requestedSegment, card), nil
|
||
})
|
||
if err != nil || v == nil {
|
||
return nil
|
||
}
|
||
session, _ := v.(*TranscodeSession)
|
||
return session
|
||
}
|
||
|
||
// fastResumeSeek decides whether a reconstructed ffmpeg should be spawned at the
|
||
// segment the client is actually requesting instead of the card's original
|
||
// start. Resuming near requestedSegment avoids a wait-then-seek-restart stall
|
||
// when the client has already played past the card position.
|
||
//
|
||
// The returned (segment, seekSeconds) maps via seg×SegmentDuration, which is
|
||
// ONLY valid for ENCODED transcodes: their forced keyframes make every segment
|
||
// exactly SegmentDuration long. COPY-mode segments inherit the source's variable
|
||
// GOP boundaries, so seg×dur lands on the wrong source time and desyncs A/V after
|
||
// a restart — so for copy-mode cards this returns ok=false and the caller keeps
|
||
// the card's original start, letting the manifest-driven segment recovery
|
||
// (RestartSeekTarget) seek forward once the rebuilt manifest exposes the real
|
||
// per-segment timing. A negative requestedSegment (manifest path, no segment
|
||
// context) and a non-advanced client also return ok=false.
|
||
func fastResumeSeek(card RecipeCard, requestedSegment int) (segment int, seekSeconds float64, ok bool) {
|
||
if strings.EqualFold(card.TargetCodecVideo, "copy") {
|
||
return 0, 0, false
|
||
}
|
||
if requestedSegment > card.StartSegmentNumber && card.SegmentDuration > 0 {
|
||
return requestedSegment, float64(requestedSegment * card.SegmentDuration), true
|
||
}
|
||
return 0, 0, false
|
||
}
|
||
|
||
// doReconstructTranscode performs the actual rebuild for a single reconstruct
|
||
// leader. It is only ever invoked inside reconstructGroup.Do, so it is the sole
|
||
// writer racing to register sessionID for this session.
|
||
func (m *TranscodeManager) doReconstructTranscode(ctx context.Context, sessionID string, requestedSegment int, card RecipeCard) *TranscodeSession {
|
||
// Only transcode cards drive ffmpeg reconstruction. Direct/remux sessions
|
||
// reconstruct without a runtime and must never reach here; guard so a
|
||
// direct/remux card ID cannot accidentally spawn an encode. An empty
|
||
// PlayMethod is back-compat for a token minted before the discriminator
|
||
// (transcode).
|
||
if card.PlayMethod != "" && card.PlayMethod != PlayTranscode {
|
||
return nil
|
||
}
|
||
|
||
// Mark in-flight for the whole rebuild so a concurrent cleanup never reaps the
|
||
// output dir between spawn and map registration.
|
||
release := m.markReconstructing(sessionID)
|
||
defer release()
|
||
|
||
cfg := m.runtimeConfig()
|
||
outputDir := reconstructionOutputDir(cfg.TranscodeDir, sessionID, card.OutputSubdir)
|
||
opts := card.TranscodeOpts(outputDir, cfg.FFmpegPath, m.logSink())
|
||
// Re-resolve environment-specific encode knobs from current config so an
|
||
// operator config change applies to reconstructed sessions too.
|
||
opts.HWAccel = cfg.HWAccel
|
||
opts.HWDevice = cfg.HWDevice
|
||
|
||
// Resume near the segment the client is actually requesting. The card records
|
||
// the original start; if the client has played past it, spawning ffmpeg at the
|
||
// old position forces a wait-then-seek-restart cycle (a visible stall). Seeking
|
||
// straight to requestedSegment avoids it. A negative requestedSegment (manifest
|
||
// path) carries no segment context, so the card position stands.
|
||
//
|
||
if seg, seek, ok := fastResumeSeek(card, requestedSegment); ok {
|
||
opts.StartSegmentNumber = seg
|
||
opts.SeekSeconds = seek
|
||
}
|
||
|
||
// Pace the spawn so a post-restart wave of reconstructs does not launch a
|
||
// thousand cold-start ffmpeg processes at once. A client that disconnects while
|
||
// waiting releases its place rather than queueing dead work.
|
||
slotRelease, ok := m.acquireReconstructSlot(ctx)
|
||
if !ok {
|
||
return nil
|
||
}
|
||
|
||
// Serialize against every other spawn path (fresh start, restart) for this
|
||
// session so a reconstruct and a fresh start never run two ffmpeg writers
|
||
// against the same output dir. reconstructGroup only single-flights reconstructs
|
||
// against each other, not against starts.
|
||
unlock := m.LockSessionLifecycle(sessionID)
|
||
defer unlock()
|
||
|
||
// Re-check under the lifecycle lock: a fresh start (or a reconstruct that ran
|
||
// just before us) may already have a live session. Yield to it instead of
|
||
// spawning a duplicate writer.
|
||
if existing := m.GetTranscodeSession(sessionID); existing != nil {
|
||
slotRelease()
|
||
return existing
|
||
}
|
||
|
||
transcodeSession, err := StartTranscode(context.WithoutCancel(ctx), opts)
|
||
slotRelease()
|
||
if err != nil {
|
||
slog.ErrorContext(ctx, "reconstruct transcode start failed", "component", "playback", "error", err, "session", sessionID, "playback_session_id", sessionID)
|
||
return nil
|
||
}
|
||
|
||
// Register under the map lock. The lifecycle lock guarantees no other path
|
||
// registered since the re-check above; the existing-check is kept as defensive
|
||
// belt-and-braces, closing only the duplicate ffmpeg process (never the shared
|
||
// output dir the winner serves) on the should-be-impossible race.
|
||
m.transcodeMu.Lock()
|
||
if existing := m.transcodes[sessionID]; existing != nil {
|
||
m.transcodeMu.Unlock()
|
||
_ = transcodeSession.CloseProcess()
|
||
return existing
|
||
}
|
||
m.transcodes[sessionID] = transcodeSession
|
||
m.transcodeMu.Unlock()
|
||
|
||
// Mirror the handler's start path: re-arm the throttler and exit monitor
|
||
// after every Restart of this reconstructed session, so seek/audio-switch
|
||
// restarts keep the same wiring as a freshly started transcode.
|
||
transcodeSession.SetRestartHook(func(ctx context.Context) {
|
||
if m.StartThrottler != nil {
|
||
m.StartThrottler(ctx, transcodeSession)
|
||
}
|
||
m.MonitorLocalTranscodeExit(sessionID, transcodeSession)
|
||
})
|
||
|
||
if m.StartThrottler != nil {
|
||
m.StartThrottler(ctx, transcodeSession)
|
||
}
|
||
m.MonitorLocalTranscodeExit(sessionID, transcodeSession)
|
||
slog.InfoContext(ctx, "transcode process reconstructed from recipe card", "component", "playback",
|
||
"session", sessionID, "playback_session_id", sessionID,
|
||
"requested_segment", requestedSegment, "start_segment_number", opts.StartSegmentNumber)
|
||
return transcodeSession
|
||
}
|
||
|
||
func reconstructionOutputDir(root, sessionID, signedSubdir string) string {
|
||
outputSubdir := sessionID
|
||
if candidate := filepath.Clean(signedSubdir); candidate != "." && !filepath.IsAbs(candidate) && candidate != ".." && !strings.HasPrefix(candidate, ".."+string(filepath.Separator)) {
|
||
first, _, _ := strings.Cut(candidate, string(filepath.Separator))
|
||
if first == sessionID || strings.HasPrefix(first, sessionID+"-") {
|
||
outputSubdir = candidate
|
||
}
|
||
}
|
||
return filepath.Join(root, outputSubdir)
|
||
}
|
||
|
||
// MonitorLocalTranscodeExit watches a local ffmpeg process and, on an error exit,
|
||
// invokes OnFFmpegCrash so the embedding handler tears down the playback session.
|
||
// A clean exit (no error) leaves the segments servable until the client stops.
|
||
func (m *TranscodeManager) MonitorLocalTranscodeExit(sessionID string, session *TranscodeSession) {
|
||
if m == nil || sessionID == "" || session == nil {
|
||
return
|
||
}
|
||
|
||
done := session.Done()
|
||
if done == nil {
|
||
return
|
||
}
|
||
|
||
go func() {
|
||
<-done
|
||
time.Sleep(2 * time.Second)
|
||
|
||
m.transcodeMu.RLock()
|
||
current := m.transcodes[sessionID]
|
||
m.transcodeMu.RUnlock()
|
||
if current != session {
|
||
return
|
||
}
|
||
if session.IsRunning() {
|
||
return
|
||
}
|
||
|
||
// When ffmpeg exits cleanly (no error), the segments are fully written and
|
||
// should remain servable until the client stops the session. This is
|
||
// critical for copy-mode where ffmpeg finishes writing all content much
|
||
// faster than real-time playback. Only tear down the session on error exits.
|
||
if session.WaitError() == nil {
|
||
return
|
||
}
|
||
|
||
// ffmpeg crash — tear the session down; a client holding a valid token can
|
||
// reconstruct it on the next request. Pass the dead session so teardown is a
|
||
// compare-and-delete: a reconstruct that registered a successor under this id
|
||
// between the current!=session check above and teardown must not be killed.
|
||
if m.OnFFmpegCrash != nil {
|
||
m.OnFFmpegCrash(context.Background(), sessionID, session)
|
||
}
|
||
}()
|
||
}
|
||
|
||
// CloseTranscodeSession stops a transcode session. If transcodeNodeURL is
|
||
// non-empty, sends DELETE to the remote transcode node. Otherwise closes the
|
||
// local session.
|
||
//
|
||
// Under token-carried reconstruction there is no durable card to drop: a stopped
|
||
// session simply stops being served, and its segment dir is reaped by the
|
||
// in-memory-liveness + age cleanup once no live token could still reconstruct it
|
||
// (see CleanupOrphanedTranscodes). A sub-TTL hard cut of an abusive stream
|
||
// before the token expires depends on a node-side revocation mechanism that is
|
||
// deferred to a future PR; today a stopped session can be reconstructed by a
|
||
// still-valid token until it expires.
|
||
func (m *TranscodeManager) CloseTranscodeSession(sessionID, transcodeNodeURL string) {
|
||
// Clean up local session if one exists (defensive).
|
||
m.transcodeMu.Lock()
|
||
session := m.transcodes[sessionID]
|
||
delete(m.transcodes, sessionID)
|
||
m.transcodeMu.Unlock()
|
||
if session != nil {
|
||
_ = session.Close()
|
||
}
|
||
|
||
m.deleteRemoteTranscode(sessionID, transcodeNodeURL)
|
||
}
|
||
|
||
// CloseTranscodeSessionIf tears down a transcode session only when the live map
|
||
// still holds the exact session the caller observed dying (expected). This is
|
||
// the crash path: between a local ffmpeg's error exit and this teardown, a
|
||
// concurrent reconstruct can register a fresh successor under the same id. An
|
||
// unconditional close would delete+Close() that live successor — and Close()
|
||
// removes the shared output dir out from under it. Comparing under the same lock
|
||
// that reconstruct registers through makes the swap atomic: a non-matching entry
|
||
// is left untouched. The remote-DELETE still fires for the matched case (and is
|
||
// skipped entirely when the local successor already won, since there is nothing
|
||
// of ours to stop).
|
||
//
|
||
// Returns true iff the live entry still matched expected and was torn down;
|
||
// false iff a different (successor) or nil session held the slot and was left
|
||
// untouched. Callers MUST treat this return as the authoritative gate for any
|
||
// further teardown (e.g. stopping the upstream playback session): when it is
|
||
// false, a successor owns the id and must not be disturbed.
|
||
func (m *TranscodeManager) CloseTranscodeSessionIf(sessionID string, expected *TranscodeSession, transcodeNodeURL string) bool {
|
||
m.transcodeMu.Lock()
|
||
current := m.transcodes[sessionID]
|
||
if current != expected {
|
||
// A successor (or an already-completed close) holds the slot; leave it.
|
||
m.transcodeMu.Unlock()
|
||
return false
|
||
}
|
||
delete(m.transcodes, sessionID)
|
||
m.transcodeMu.Unlock()
|
||
if expected != nil {
|
||
_ = expected.Close()
|
||
}
|
||
|
||
m.deleteRemoteTranscode(sessionID, transcodeNodeURL)
|
||
return true
|
||
}
|
||
|
||
// deleteRemoteTranscode sends DELETE to the assigned transcode node if any
|
||
// (synchronous with timeout). A no-op for local/integrated sessions.
|
||
func (m *TranscodeManager) deleteRemoteTranscode(sessionID, transcodeNodeURL string) {
|
||
if transcodeNodeURL != "" {
|
||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||
defer cancel()
|
||
|
||
deleteURL := transcodeNodeURL + "/transcode/" + sessionID
|
||
req, err := http.NewRequestWithContext(ctx, http.MethodDelete, deleteURL, nil)
|
||
if err != nil {
|
||
slog.Error("remote transcode delete: build request", "error", err, "session", sessionID, "playback_session_id", sessionID)
|
||
return
|
||
}
|
||
req.Header.Set("Authorization", "Bearer "+m.jwtSecret())
|
||
|
||
resp, err := http.DefaultClient.Do(req)
|
||
if err != nil {
|
||
slog.Warn("remote transcode delete failed", "error", err, "session", sessionID, "node", transcodeNodeURL, "playback_session_id", sessionID)
|
||
return
|
||
}
|
||
_ = resp.Body.Close()
|
||
if resp.StatusCode >= http.StatusMultipleChoices {
|
||
slog.Warn("remote transcode delete returned non-success status",
|
||
"status", resp.StatusCode, "session", sessionID, "node", transcodeNodeURL, "playback_session_id", sessionID)
|
||
}
|
||
}
|
||
}
|
||
|
||
// CleanupOrphanedTranscodes removes stale per-session temp directories for
|
||
// transcodes that are no longer reconstructable. Under token-carried
|
||
// reconstruction there is no durable card index to consult, so the liveness
|
||
// signal is: the in-process live transcode map, the set of sessions currently
|
||
// mid-reconstruct, and directory age. A dir is reaped only when it is absent from
|
||
// both sets AND older than the maximum token lifetime — past which no surviving
|
||
// token could reconstruct it. Each process owns its own TranscodeDir, so there is
|
||
// no cross-process enumeration-failure mode to fail safe against.
|
||
func (m *TranscodeManager) CleanupOrphanedTranscodes() (int, error) {
|
||
// Snapshot the live map and the in-flight set under both locks held at once.
|
||
// A reconstruct registers into m.transcodes and clears m.reconstructInFlight
|
||
// at different moments; snapshotting the two sets separately could miss a
|
||
// session that migrated between them, leaving its live dir absent from active
|
||
// and exposed to reaping. inFlightMu is taken first to match the only other
|
||
// site that holds both (none nests the reverse order).
|
||
m.inFlightMu.Lock()
|
||
m.transcodeMu.RLock()
|
||
active := make(map[string]struct{}, len(m.transcodes)+len(m.reconstructInFlight))
|
||
for sessionID := range m.transcodes {
|
||
active[sessionID] = struct{}{}
|
||
}
|
||
// Spare sessions mid-reconstruct: their dir is being written right now but is
|
||
// not yet registered in the live map.
|
||
for sessionID := range m.reconstructInFlight {
|
||
active[sessionID] = struct{}{}
|
||
}
|
||
m.transcodeMu.RUnlock()
|
||
m.inFlightMu.Unlock()
|
||
|
||
return CleanupOrphanedTranscodeDirs(m.runtimeConfig().TranscodeDir, active, MaxTokenTTL)
|
||
}
|