Files
silo-server/internal/scanner/probe.go
881c96864b feat(playback): finalize platform-neutral protocol v3 (#567)
* docs(playback): add v3 neutral-contract finalization plan

Supersedes the wire-contract sections of the 2026-07-12 v3 plan: server-owned
attempt keys, delivery-keyed negotiation without Media3 engine names, tiered
capability evidence, neutral device/output context, track/quality replan
operations, audio-only planning, and coordinated no-back-compat rollout
across server, Android, Apple, and web.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(playback): make v3 attempt keys server-owned and replace engines with deliveries

Contract core of the platform-neutral v3 finalization (plan sections 3.1
and 3.2), breaking on purpose — v3 is dark and all clients move together:

- Every PlanV3 now carries plan_attempt_key, an opaque server-computed
  token clients store and echo in attempted_plan_keys; ReplanRequestV3
  gains bounded local_mutations that the replan handler folds into the
  failed plan's key. Clients never hash anything.
- KotlinName() is deleted from DeliveryV3, StreamProtocolV3 and
  SubtitleModeV3; the attempt-key canonical string now uses lowercase
  wire tokens, and PlanRecipeVersionV3 bumps to v3.3 so no key or plan
  ID computed under the old canonicalization can collide.
- EngineV3 leaves the wire: ClientPlaybackContextV3.Engines (media3_*)
  becomes Deliveries keyed original_http|progressive|hls, with
  EngineCapabilityV3 renamed DeliveryCapabilityV3. PlanV3.Engine is
  removed; the planner, subtitle policy and quirk registry re-key on
  delivery class, and the media3_only feature token is deleted.
- Validated-claim strings drop the prefix: media3_h264_decode ->
  h264_decode, media3_audio_decode -> audio_decode.
- Golden fixtures in testdata/protocol_v3 are regenerated by Go and are
  now the cross-repo source of truth.

Part of the playback protocol v3 neutral-contract train (steps 2-3 of
docs/superpowers/plans/2026-07-30-playback-protocol-v3-neutral-contract.md).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(playback): add v3 evidence tiers and neutral device/output context

Implement plan sections 3.3 and 3.4 of the v3 neutral-contract pass:

- ClientCodecCapabilitiesV3 gains required video_evidence and
  audio_evidence closed enums (exact | platform_attested | declared).
  Planner strictness follows the tier: exact keeps the strict decode-entry
  validation, platform_attested validates codec/resolution/bit-depth/
  frame-rate but skips profile/level matching, declared grants copy routes
  from the flat codec lists. Only exact audio evidence earns passthrough
  claims. The detailed_decode_capabilities feature token is deleted
  (subsumed by video_evidence=exact), and evidence-blocked direct routes
  carry the new evidence_insufficient_for_direct reason/warning.

- DeviceContextV3 is now platform/os_version/manufacturer/model plus a
  bounded platform_details map (<=16 entries, <=128 chars); the Android
  Build dump fields are gone. Fire TV quirks keep matching on
  manufacturer/model (brand fallback removed with the field).

- output_route_generation (int64, dual-location) becomes an optional
  opaque output_context_id string on the output context; the dual-location
  consistency validation is deleted. Attempt keys, plan invalidation,
  route events, and the planstore column follow (new Goose migration).

- Feature advertisement collapses to the top-level client_features list
  only; ClientPlaybackContextV3.Features is deleted and ReplanRequestV3
  gains an optional client_features refresh.

- PlanRecipeVersionV3 bumped v3.3 -> v3.4; fixtures re-keyed.

Part of the playback protocol v3 neutral-contract finalization plan
(docs/superpowers/plans/2026-07-30-playback-protocol-v3-neutral-contract.md).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(playback): add v3 intent replans, quality menu, and audio-only routes

Protocol v3 could only replan after a failure, so changing the audio track
or the quality still required the legacy audio PATCH and the client-recipe
transcode start — the two endpoints v3 is meant to replace. Clients also had
to own a resolution ladder to render a quality menu, and a source with no
video track was terminaled by the video/HDR gates, keeping audiobooks on the
legacy path.

Add track_change and quality_change replan operations. They carry no failure
classification and route through the existing replan transaction, so they
inherit its idempotency, capacity reservation, and staged-successor commit
for free. Because nothing failed, the previous route stays eligible: neither
the attempted-key history nor the failed-plan exclusion applies to them.

Publish the server ladder on the plan as available_qualities so the quality
menu is server-owned; the rungs come from the same resolutionLabelV3 and
ladderBitrateKbpsV3 helpers the planner itself uses, not a parallel table.

Plan audio-only sources through their own reduced route family: original_http
when the client decodes the codec, otherwise a progressive AAC conversion.
The plan advertises audio/mp4 for that remux and the transport now serves the
same value, because a declared-tier client probes the advertised MIME with
isTypeSupported before attaching a source buffer, and "video/mp4" on a stream
with no video track is exactly the mismatch that makes the probe lie.

Name the protocol's string vocabulary (dynamic ranges, transformations,
executors, validated claims, terminal reasons) as constants while touching
these lines, so the wire values have one definition.

Part of #135

* docs(playback): publish the v3 protocol contract and fix subtitle ordinals

Protocol v3 exists only as Go code today, so the Android and Apple ports have
no authority to implement against other than reading this repository. Publish
the contract as a normative document, machine-checkable schemas, and generated
golden fixtures, and fix the one place where the server's own wire output
disagreed with the ordinal space it publishes.

- docs/architecture/playback-protocol-v3.md is self-contained enough for a
  third-party client: endpoints and status codes, evidence tiers and their
  bound-matching rules, delivery classes, the timeline model, replan
  semantics, registries, track identity, plan identity, quality, and
  transformations.
- docs/design/schemas/playback-v3/ carries JSON Schemas for the five wire
  shapes plus valid and invalid fixtures, following the client-diagnostics
  layout. internal/playback/contract validates every fixture against its
  schema, so a schema that drifts from the Go types fails the Go suite.
- cmd/playbackfixtures generates internal/playback/testdata/protocol_v3 from
  the production planner. `make playback-fixtures` writes them and
  `make verify-playback-fixtures` (wired into CI) fails when they are stale.
  These files are what the client ports consume, so drift would otherwise
  surface as a playback bug on three platforms at once.

The subtitle fix: combined ordinals are one dense space over externals, then
embedded tracks, then downloaded ones, but the legacy URL builder skipped
burn-in-only tracks while assigning indices, so every track after a DVD/DVB
track was numbered one too low and resolved to its neighbour. Ordinal
assignment now lives in playback.BuildSubtitleInventoryV3 and both the plan
inventory and the legacy `subtitle_urls` shape project from it; the legacy
shape still filters burn-in-only entries but keeps each track's real index.

Part of #135

* feat(web): migrate the players to the neutral playback v3 contract

The web player was the last client still speaking the legacy start
protocol: it picked its own file version from a codec probe, posted an
ffmpeg recipe to start a transcode, PATCHed an endpoint to change audio
tracks, and derived its own quality ladder. None of that survives a
server-owned plan, and none of it produced telemetry the apps could be
compared against.

Video player: starts with a v3 request that advertises `declared`
evidence from `isTypeSupported` probes and the three delivery classes,
then consumes the returned plan for its URL, timeline, tracks and
warnings. Quality and track changes become replans (`quality_change`,
`track_change`), the quality menu renders `available_qualities` instead
of computing rungs, and playback failures emit `route-events` so web
failures land in the same diagnostics as Android and Apple. The
duration comes from `source.duration_seconds` rather than the playback
engine, and the "how was this delivered" overlay reads the plan's
delivery and server transformations instead of comparing codec strings.

Audiobook player: starts against the audio-only planner path with a
single `original` rung, and takes its seek anchor from
`timeline.player_start_seconds` so the progressive-remux route (which
anchors the stream and restarts the player clock at zero) does not seek
twice.

Server side, `disable_progress_persistence` left the wire, so the rule
it encoded is now derived. Resume state is keyed on the item, but every
part of a multipart presentation shares that key while carrying its own
file-local clock — persisting part 4's position would store "12 minutes
in" as the book's resume point. `PresentationPartTotal > 1` expresses
that directly and generalizes to multipart movies and split episodes,
and a client can no longer forget to ask or lie about it.

`useTranscodeQuality` and the legacy response types are deleted, and
`WEBTEST_KNOWN_FAILURES` loses the audiobook entry along with its fix.

Part of #135

* feat(playback)!: make v3 the only playback protocol

Protocol v3 shipped behind a flag, alongside the legacy start path it was
designed to replace. Running both meant every planner change had to be made
twice, in two shapes that disagree about who decides the route: the legacy
body carried a decision the client had already made, while v3 asks the server
to make it. This deletes the legacy half.

Removed:

- `handleStartPlaybackLegacy` and its request/response bodies. The
  `POST /playback/start` route stays, but the protocol-version dispatch
  envelope is now a strict v3 decode — a body that does not declare
  `protocol_version: 3` gets `426 client_upgrade_required` so an outdated app
  can render a clear "update required" state instead of misreading a plan.
  Deliberately not a `400`: the request may be well-formed for the protocol it
  was written against.
- `POST /playback/transcode/start`, superseded by the `quality_change` replan
  operation, and `PATCH /playback/{session_id}/audio`, superseded by
  `track_change`. Both mutated a session without re-planning.
- The shadow planner and both rollout settings rows. With v3 the only
  protocol, `playback.protocol_v3_enabled` would mean "no playback at all";
  `playback.protocol_v3_shadow_enabled` gated a comparison against a path that
  no longer exists. `409 protocol_disabled` on route-events goes with them, and
  capability `enabled` is now constant `true` (the field stays — clients
  feature-detect against it).
- Version-selection helpers in `internal/playback/resolver.go` that only legacy
  start reached. `Resolve`/`ClientCapabilities`/`PlayDecision` stay: downloads
  consumes them. `internal/jellycompat` has its own resolution surface and is
  untouched.

Behaviour the legacy handlers owned and v3 now owns explicitly: series version
and audio-track preferences are persisted on start and on a `track_change`
replan (not on failure recovery, whose forced route is not a user choice); an
omitted `start_position` resolves to the profile's saved resume point; and an
omitted audio track resolves through the series preference, the profile audio
language, then the library override. Both are settled before planning, because
the plan's timeline is cut at the start position. Spec §2.2 documents this as
"omission is a request, not a default".

The encode-target clamp that lived in the deleted transcode handler is already
enforced in the planner, twice — `availableQualitiesV3` omits rungs at or above
the source height, and the encode path clamps `targetHeight` to it.

Unchanged: progress, stop, HLS manifest and segment delivery, the realtime
control socket, stream tokens and restart reconstruction, watch together,
downloads, jellycompat.

Every removal is recorded in the pre-lock removals table in
docs/architecture/v1-scope.md.

Part of #135

* fix(scanner): stop recording embedded cover art as a video track

ffprobe reports embedded cover art as a video stream carrying
disposition.attached_pic. convertProbeData appended every "video" stream
to VideoTracks without consulting isMainVideoStream, the predicate that
already existed for duration decisions, so the picture was persisted as a
playable track. That misreports the file twice:

  - An audio file with a cover picks up a video track, so it no longer
    satisfies MediaFile.IsAudioOnly and the v3 planner routes an
    audiobook through the video path instead of planAudioOnlyV3.
  - When the picture is ordered ahead of the real stream, the flat
    codec_video/resolution/hdr columns describe the poster: a 954x720
    h264 episode was stored as mjpeg 480x480.

Filter attached_pic streams out of the track loop. The guard is the
disposition flag, not the codec name, so a genuine MJPEG video is still
probed as video — the library has one.

Already-probed rows self-heal on the next playback: NeedsCriticalProbeRepair
already reprobes tracks missing color_range, which covers 21 of the 23
affected rows, and applyProbeData overwrites VideoTracks wholesale. The
remaining two need a rescan; nothing persisted records attached_pic, and
keying repair off still-image codec names would reprobe the genuine MJPEG
file on every playback forever.

Part of the playback v3 neutral-contract work: it is what lets Android
drop AUDIOBOOK_COVER_ART_CODECS, which fabricated decode support the
client cannot honestly claim under video_evidence: "exact".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(playback): publish subtitle URLs even when playback starts with subtitles off

The v3 plan's subtitle inventory is the authoritative track list a client builds
its subtitle menu from, but the handler only rewrote it with session-scoped URLs
when a track was actually selected. A start or replan that resolved to
`subtitle.mode: "off"` therefore returned the planner's URL-less inventory, so a
client whose picker reads the inventory had a menu it could not fetch anything
from. The Cast path hits this every time: it starts with subtitles off and needs
the receiver's text tracks up front.

attachSubtitleArtifactV3 now scopes and publishes the inventory unconditionally
and gates only the artifact stamping on the selection. Spec §8 records that the
`url` on a sidecar entry does not depend on the current selection.

Part of the v3 neutral-contract finalization.

* chore(playback): reconcile neutral v3 with main

* fix(playback): preserve subtitle intent across replans

* fix(playback): retain subtitle inventory on adapted routes

* fix(playback): software-decode High10 AVC for QSV

* fix(playback): scale High10 frames before QSV upload

* fix(playback): preserve empty subtitle inventories

* fix(playback): freeze terminal attempt contract

* chore(playback): name fixture contract tokens

* fix(playback): close v3 conformance review gaps

* chore(playback): name conformance category

* fix(playback): complete v3 conformance contract

* fix(playback): keep schema fixtures generated

* fix(playback): emit schema-valid conformance arrays

* fix(playback): omit empty replan failures

* fix(web): omit empty replan failures

* fix(playback): close neutral v3 contract gaps

* fix(playback): harden v3 replan, transcode, and quality-ladder edge cases

Review remediation for the neutral v3 cutover, server side:

- A failed replan no longer overwrites the durable StartResponse with a
  terminal or advances the replan request ID; an idempotent start replay
  of a still-healthy session returns the original plan.
- SoftwareVideoDecode is now derived inside the transcode layer from
  source facts (codec/profile/bit depth) carried on TranscodeOpts, so
  jellycompat, downloads, recipe-card reconstruction, and transcode
  nodes get the High10 software-decode fix, not just the v3 handler.
  video_to_h264 recipe version bumps to 2 so mixed-version node pools
  that would silently drop the flag fail validation instead.
- Local transport startup shares the 30s ManifestStartupTimeout; a
  timeout with the process still running stays retryable and is no
  longer persisted as a durable terminal against the attempt.
- Sparse replan bodies (failure_recovery et al) no longer reset a
  user-selected quality preference to auto; the empty-value guard now
  covers every operation.
- availableQualitiesV3 publishes no fixed rungs when the source height
  is unknown, keeping the no-upscaling ladder contract.
- The proxy remux path serves audio-only fMP4 as audio/mp4 via a new
  additive AudioOnly token claim, matching the integrated path.
- Plain text subtitle sidecars accept any requested extension again
  (served as VTT), restoring the permissive v1 behavior; ASS and bitmap
  handling is unchanged.
- The 4K-disallowed terminal message discloses when a lower-resolution
  alternate exists but was pinned away by quality "original".

Part of #135.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(web): keep playback alive through failed replans and honest audio claims

Review remediation for the neutral v3 cutover, web player:

- A failed or refused replan no longer unmounts the player: the fatal
  error screen is reserved for loads with no adopted plan, and replan
  failures surface through the existing non-fatal replanError path.
- changeQuality rolls its optimistic preference back when the replan is
  refused or errors, so a failed switch is not silently applied by the
  next unrelated replan and the menu shows the real active rung.
- The capability probe now tests mp3/vorbis codecs and mp3/flac/ogg
  containers (MediaSource with a canPlayType fallback), restoring
  direct play for mp3 audiobooks instead of per-part AAC re-encodes.
- Reanchor seeks issued while a replan is in flight coalesce and run
  when it settles instead of being silently dropped with the scrubber
  pinned to a phantom position.
- Subtitle refresh/translation replans use the resume anchor while the
  media element has no metadata, so a subtitle_ready broadcast during
  startup no longer restarts a resumed stream at 0:00.
- An exhausted failure-recovery chain sets a visible error instead of
  returning silently.

Part of #135.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(playback): accept video-only and VP9 probe metadata

Treat audio and video probe completeness independently so legitimate video-only assets converge without repeated ffprobe repair. Allow unknown codec profile/level metadata to fall through to server adaptation while preserving exact direct-decode constraints.

Fixes #574

* fix(playback): address protocol v3 review findings

* fix(playback): harden lease and probe repair decisions

* fix(playback): close remaining v3 review gaps

* fix(playback): recover failed transcode starts

* fix(playback): address remaining review-bot findings on v3 replan and audio planning

Server:
- The deferred replan lease release is bounded by a 3s timeout so a
  saturated pool or DB outage cannot wedge a handler goroutine that
  holds the per-session store lock on an uncancellable context.
- planAudioOnlyV3 honors the request bandwidth cap: an over-cap source
  skips the original_http direct route and converts to AAC with the
  same bandwidth_cap_applied warning and decision reason the video
  ladder uses. Unknown source bitrate never triggers the cap.
- A copy-audio progressive plan rejected only by a per-delivery
  audio_decode_codecs subset retries as an AAC conversion instead of
  returning adaptation_unavailable, and the AAC recipe respects the
  delivery's max_channels.

Web:
- failure_recovery replans issued while another replan is in flight
  queue (superseding a pending seek reanchor) instead of being
  silently dropped with the fatal overlay already suppressed.
- A terminal response to a fresh non-preserving start clears the
  previous plan and stops its session, so episode navigation cannot
  keep rendering the prior item under the new title.
- A refused recovery replan for a transport-dead plan surfaces the
  error and re-arms the plan failure key, so transient recovery
  failures no longer strand an endless spinner; the audiobook player
  gets the same guard reset.
- A track-less subtitle_translation_completed hands off to the
  refreshed persisted track once the inventory settles, clearing the
  live overlay, instead of pinning the synthetic live track forever.

Part of #135.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(playback): reuse HLS transport for sidecar replans

* fix(playback): stabilize copy HLS remount timeline

* fix(playback): address v3 review findings

* fix(playback): satisfy player contract types

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 18:14:49 -04:00

982 lines
29 KiB
Go

package scanner
import (
"bufio"
"context"
"encoding/json"
"fmt"
"io"
"math"
"os/exec"
"slices"
"strconv"
"strings"
"github.com/Silo-Server/silo-server/internal/lang"
"github.com/Silo-Server/silo-server/internal/models"
)
// ffprobeOutput represents the top-level JSON output from ffprobe.
type ffprobeOutput struct {
Format ffprobeFormat `json:"format"`
Streams []ffprobeStream `json:"streams"`
Chapters []ffprobeChapter `json:"chapters"`
}
// ffprobeScalarString accepts ffprobe fields that may be emitted as either
// JSON strings or numbers depending on codec/container details.
type ffprobeScalarString string
func (s *ffprobeScalarString) UnmarshalJSON(data []byte) error {
if string(data) == "null" {
*s = ""
return nil
}
var str string
if err := json.Unmarshal(data, &str); err == nil {
*s = ffprobeScalarString(str)
return nil
}
var num json.Number
if err := json.Unmarshal(data, &num); err == nil {
*s = ffprobeScalarString(num.String())
return nil
}
return fmt.Errorf("unsupported ffprobe scalar %s", string(data))
}
// ffprobeFormat represents the "format" section of ffprobe JSON output.
type ffprobeFormat struct {
Filename string `json:"filename"`
FormatName string `json:"format_name"`
FormatLongName string `json:"format_long_name"`
StartTime string `json:"start_time"`
Duration string `json:"duration"`
Size string `json:"size"`
BitRate string `json:"bit_rate"`
Tags map[string]string `json:"tags"`
}
// ffprobeStream represents a single stream entry in ffprobe JSON output.
type ffprobeStream struct {
Index int `json:"index"`
CodecName string `json:"codec_name"`
CodecLongName string `json:"codec_long_name"`
CodecType string `json:"codec_type"`
Profile string `json:"profile"`
Level int `json:"level"`
Width int `json:"width"`
Height int `json:"height"`
DisplayAspectRatio string `json:"display_aspect_ratio"`
FieldOrder string `json:"field_order"`
AvgFrameRate string `json:"avg_frame_rate"`
StartTime string `json:"start_time"`
Duration string `json:"duration"`
BitRate string `json:"bit_rate"`
ColorRange string `json:"color_range"`
ColorTransfer string `json:"color_transfer"`
ColorPrimaries string `json:"color_primaries"`
ColorSpace string `json:"color_space"`
PixFmt string `json:"pix_fmt"`
Refs int `json:"refs"`
BitsPerRawSample ffprobeScalarString `json:"bits_per_raw_sample"`
BitsPerSample ffprobeScalarString `json:"bits_per_sample"`
Channels int `json:"channels"`
ChannelLayout string `json:"channel_layout"`
SampleRate string `json:"sample_rate"`
Disposition ffprobeDisp `json:"disposition"`
Tags map[string]string `json:"tags"`
SideDataList []ffprobeSideData `json:"side_data_list"`
}
type ffprobeChapter struct {
ID int `json:"id"`
Start ffprobeScalarString `json:"start"`
End ffprobeScalarString `json:"end"`
TimeBase string `json:"time_base"`
StartTime ffprobeScalarString `json:"start_time"`
EndTime ffprobeScalarString `json:"end_time"`
Tags map[string]string `json:"tags"`
}
type ffprobeSideData struct {
SideDataType string `json:"side_data_type"`
DVProfile int `json:"dv_profile"`
DVBlPresent int `json:"dv_bl_present"`
DVElPresent int `json:"dv_el_present"`
DVBLCompatID int `json:"dv_bl_signal_compatibility_id"`
}
// ffprobeDisp represents the disposition flags on a stream.
type ffprobeDisp struct {
Default int `json:"default"`
Forced int `json:"forced"`
AttachedPic int `json:"attached_pic"`
}
// ProbeFile runs ffprobe on the given file and returns parsed ProbeData.
// ffprobePath is the path to the ffprobe binary. filePath is the media file to probe.
func ProbeFile(ctx context.Context, ffprobePath string, filePath string) (*ProbeData, error) {
cmd := exec.CommandContext(ctx, ffprobePath,
"-v", "quiet",
"-print_format", "json",
"-show_format",
"-show_streams",
"-show_chapters",
filePath,
)
output, err := cmd.Output()
if err != nil {
return nil, fmt.Errorf("ffprobe failed for %s: %w", filePath, err)
}
var raw ffprobeOutput
if err := json.Unmarshal(output, &raw); err != nil {
return nil, fmt.Errorf("ffprobe JSON parse failed for %s: %w", filePath, err)
}
probe := convertProbeData(&raw)
if probe.Duration == 0 {
if frameRate, hasVideo := primaryVideoFrameRate(raw.Streams); hasVideo {
// A failed or empty packet scan must not discard the codec and
// track metadata that already parsed successfully: callers persist
// the partial probe, and the repair layer retries rows whose
// duration is still unknown.
if duration, packetErr := probeVideoPacketDuration(ctx, ffprobePath, filePath, frameRate); packetErr == nil && duration > 0 {
probe.Duration = duration
}
}
}
return probe, nil
}
// FFprobePathFromFFmpeg derives the sibling ffprobe binary path from a configured ffmpeg path.
func FFprobePathFromFFmpeg(ffmpegPath string) string {
if i := strings.LastIndex(ffmpegPath, "ffmpeg"); i >= 0 {
ffprobePath := ffmpegPath[:i] + "ffprobe" + ffmpegPath[i+len("ffmpeg"):]
if ffprobePath != "" && ffprobePath != ffmpegPath {
return ffprobePath
}
}
return "ffprobe"
}
// convertProbeData transforms raw ffprobe JSON output into ProbeData.
func convertProbeData(raw *ffprobeOutput) *ProbeData {
pd := &ProbeData{
Container: detectContainer(raw.Format.FormatName),
}
if duration, ok := durationFromProbeMetadata(raw); ok {
pd.Duration = duration
}
// Parse bitrate from format (bps to kbps).
if raw.Format.BitRate != "" {
if br, err := strconv.Atoi(raw.Format.BitRate); err == nil {
pd.Bitrate = br / 1000
}
}
for _, s := range raw.Streams {
switch s.CodecType {
case "video":
// Embedded cover art is a "video" stream to ffprobe, but it is a
// still image, not a playable video track. Recording it as one
// misreports the file twice: an audio file with a cover picks up a
// video track and stops satisfying MediaFile.IsAudioOnly, so the
// planner routes an audiobook through the video path; and when the
// picture is ordered ahead of the real stream, the flat
// codec_video/resolution/hdr columns describe the poster instead of
// the movie.
if !isMainVideoStream(s) {
continue
}
dvProfile := dolbyVisionProfileNumber(s.SideDataList)
// ffprobe omits unspecified optional fields by default; "unknown" is
// FFmpeg's canonical name for AVCOL_RANGE_UNSPECIFIED.
colorRange := firstNonEmpty(s.ColorRange, "unknown")
track := VideoTrackInfo{
Title: firstNonEmpty(s.Tags["title"], s.CodecLongName, strings.ToUpper(s.CodecName)),
Codec: s.CodecName,
DolbyVision: dolbyVisionProfile(s.SideDataList),
DVProfile: dvProfile,
DVBLCompatID: dolbyVisionBLCompatID(s.SideDataList),
DVELPresent: dolbyVisionELPresent(s.SideDataList),
DVEnhancementLayer: dolbyVisionEnhancementLayer(dolbyVisionELPresent(s.SideDataList)),
HDR10Plus: hasHDR10Plus(s.SideDataList),
Profile: s.Profile,
Level: s.Level,
Width: s.Width,
Height: s.Height,
AspectRatio: s.DisplayAspectRatio,
Interlaced: isInterlaced(s.FieldOrder),
FrameRate: normalizeFrameRate(s.AvgFrameRate),
Bitrate: parseNumeric(s.BitRate) / 1000,
VideoRange: videoRangeLabel(s),
VideoRangeType: videoRangeType(s),
ColorRange: colorRange,
ColorPrimaries: s.ColorPrimaries,
ColorSpace: s.ColorSpace,
ColorTransfer: s.ColorTransfer,
BitDepth: models.NormalizeVideoBitDepth(parseBitDepth(s), s.PixFmt, s.Profile),
PixelFormat: s.PixFmt,
ReferenceFrames: s.Refs,
}
pd.VideoTracks = append(pd.VideoTracks, track)
if pd.CodecVideo == "" {
pd.CodecVideo = s.CodecName
pd.Resolution = mapResolution(s.Width, s.Height)
pd.HDR = isHDR(s.ColorTransfer) || dvProfile > 0 || track.HDR10Plus
}
case "audio":
track := AudioTrackInfo{
Title: firstNonEmpty(s.Tags["title"], s.CodecLongName, strings.ToUpper(s.CodecName)),
EmbeddedTitle: s.Tags["title"],
Language: lang.Canonical(s.Tags["language"]),
Codec: s.CodecName,
Profile: s.Profile,
Layout: s.ChannelLayout,
Channels: s.Channels,
Bitrate: parseNumeric(s.BitRate) / 1000,
SampleRate: parseNumeric(s.SampleRate),
BitDepth: parseBitDepth(s),
Default: s.Disposition.Default == 1,
}
pd.AudioTracks = append(pd.AudioTracks, track)
if pd.CodecAudio == "" {
pd.CodecAudio = s.CodecName
pd.AudioChannels = s.Channels
}
case "subtitle":
track := SubtitleTrackInfo{
Index: s.Index,
Codec: s.CodecName,
Language: lang.Canonical(s.Tags["language"]),
Title: firstNonEmpty(s.Tags["title"], strings.ToUpper(s.CodecName)),
EmbeddedTitle: s.Tags["title"],
Resolution: subtitleResolutionLabel(s),
Forced: s.Disposition.Forced == 1,
Default: s.Disposition.Default == 1,
HearingImpaired: dispositionFlag(s.Tags, "hearing_impaired"),
}
pd.SubtitleTracks = append(pd.SubtitleTracks, track)
}
}
pd.Chapters = normalizeChapters(raw.Chapters, pd.Duration)
pd.FormatTags = normalizeFormatTags(raw.Format.Tags)
return pd
}
const (
maxReasonableMediaDurationSeconds = 100_000
// Corroborated metadata and packet-derived durations have stronger evidence
// than a lone container timestamp, so they may use the same bounded ceiling
// as long-form audio. This supports multi-day video without accepting the
// multi-year timelines seen in malformed containers.
maxValidatedMediaDurationSeconds = 1_000_000
// Audio-only files (audiobooks, podcasts) legitimately exceed the video
// ceiling, but still need a cap so malformed containers cannot persist
// multi-year durations.
maxReasonableAudioDurationSeconds = maxValidatedMediaDurationSeconds
longVideoDurationAbsoluteToleranceSeconds = 1
longVideoDurationRelativeTolerance = 0.001
)
// A video duration is implausible when it is either far too short in absolute
// terms, or when it implies a bitrate no real medium reaches. Both are
// signatures of malformed container timestamps (and of the legacy probe that
// divided large durations by one million).
//
// The absolute rule alone cannot catch a feature film that probed as, say, 61
// seconds — well past the floor, yet still wrong by two orders of magnitude.
// Size and duration together pin an implied bitrate, which separates the two
// cases the absolute rule conflates: a genuine short clip has an ordinary
// bitrate, while a 100 GB file claiming 61 seconds implies ~13 Gbps.
//
// The ceiling sits far above any real medium — UHD Blu-ray peaks near
// 150 Mbps and ProRes 4444 XQ at 4K near 500 Mbps — so legitimate content
// cannot trip it. This also makes the rule safer than the absolute floor
// alone, which false-positives on a genuine high-bitrate short.
//
// The shape is shared with the repair triggers in probe_repair.go and
// scanner.go so the probe parser and the repair layers cannot drift apart.
const (
implausiblyShortVideoMaxSeconds = 10
implausiblyShortVideoMinBytes = 100 * 1024 * 1024
implausibleVideoBitrateBps = 1_000_000_000
)
func videoDurationImplausible(durationSeconds float64, sizeBytes int64, hasVideo bool) bool {
if !hasVideo || durationSeconds <= 0 || sizeBytes <= 0 {
return false
}
if durationSeconds <= implausiblyShortVideoMaxSeconds && sizeBytes >= implausiblyShortVideoMinBytes {
return true
}
return impliedBitrateBps(sizeBytes, durationSeconds) > implausibleVideoBitrateBps
}
// impliedBitrateBps is the bitrate a file's size and duration imply. Callers
// use it as a duration-sanity signal, not as a real bitrate estimate: it
// counts container overhead and every stream, which is precisely what makes it
// a conservative upper bound.
func impliedBitrateBps(sizeBytes int64, durationSeconds float64) float64 {
return float64(sizeBytes) * 8 / durationSeconds
}
func durationFromProbeMetadata(raw *ffprobeOutput) (int, bool) {
if raw == nil {
return 0, false
}
formatDuration := parseFloat(raw.Format.Duration)
if !hasVideoStream(raw.Streams) &&
durationIsPositiveFinite(formatDuration) && formatDuration <= maxReasonableAudioDurationSeconds {
return truncatedDuration(formatDuration), true
}
if durationIsReasonable(formatDuration) && !durationLooksImplausible(raw, formatDuration) {
return truncatedDuration(formatDuration), true
}
for _, stream := range raw.Streams {
if !isMainVideoStream(stream) {
continue
}
streamDuration := parseFloat(stream.Duration)
if durationIsReasonable(streamDuration) && !durationLooksImplausible(raw, streamDuration) {
return truncatedDuration(streamDuration), true
}
duration := durationAfterStart(streamDuration, parseFloat(stream.StartTime))
if duration > 0 && !durationLooksImplausible(raw, duration) {
return truncatedDuration(duration), true
}
}
duration := durationAfterStart(formatDuration, parseFloat(raw.Format.StartTime))
if duration > 0 && !durationLooksImplausible(raw, duration) {
return truncatedDuration(duration), true
}
if duration, ok := corroboratedLongVideoDuration(raw, formatDuration); ok {
return truncatedDuration(duration), true
}
return 0, false
}
// corroboratedLongVideoDuration accepts an extended-range video duration only
// when the container and the primary video stream independently report nearly the
// same value. A small absolute/relative tolerance covers container rounding
// and stream-boundary differences without trusting a lone malformed timeline.
func corroboratedLongVideoDuration(raw *ffprobeOutput, formatDuration float64) (float64, bool) {
if raw == nil {
return 0, false
}
for _, stream := range raw.Streams {
if !isMainVideoStream(stream) {
continue
}
streamDuration := parseFloat(stream.Duration)
formatStart := parseFloat(raw.Format.StartTime)
streamStart := parseFloat(stream.StartTime)
// Some MPEG-TS/HLS timelines report duration as an absolute end
// timestamp. Use normalized spans to reconcile raw end timestamps that
// disagree, or when matching timestamps have a dominant start offset that
// strongly indicates the absolute-end shape. Ordinary non-zero starts remain
// part of an already corroborated raw duration.
normalizedFormatDuration := durationAfterStartWithinValidatedLimit(
formatDuration,
formatStart,
)
normalizedStreamDuration := durationAfterStartWithinValidatedLimit(
streamDuration,
streamStart,
)
rawDurationsAgree := longVideoDurationsAgree(formatDuration, streamDuration)
normalizedDurationsAgree := longVideoDurationsAgree(normalizedFormatDuration, normalizedStreamDuration)
matchingAbsoluteEnds := rawDurationsAgree &&
durationHasDominantStartOffset(formatDuration, formatStart) &&
durationHasDominantStartOffset(streamDuration, streamStart)
if normalizedDurationsAgree && (!rawDurationsAgree || matchingAbsoluteEnds) &&
!durationLooksImplausible(raw, normalizedFormatDuration) {
return normalizedFormatDuration, true
}
if matchingAbsoluteEnds {
// Dominant starts identify the raw values as absolute end
// timestamps. If their normalized spans do not corroborate, reject
// the metadata for packet repair instead of persisting an inflated
// raw end timestamp.
return 0, false
}
if rawDurationsAgree &&
!durationLooksImplausible(raw, formatDuration) {
return formatDuration, true
}
return 0, false
}
return 0, false
}
func longVideoDurationsAgree(first, second float64) bool {
if first <= maxReasonableMediaDurationSeconds ||
!durationIsWithinValidatedLimit(first) ||
!durationIsWithinValidatedLimit(second) {
return false
}
tolerance := max(
longVideoDurationAbsoluteToleranceSeconds,
max(first, second)*longVideoDurationRelativeTolerance,
)
return math.Abs(first-second) <= tolerance
}
// durationHasDominantStartOffset identifies the conservative absolute-end
// shape where the start timestamp occupies at least half of the reported end.
// Smaller starts are common media offsets and cannot disambiguate a duration
// field from an absolute end timestamp.
func durationHasDominantStartOffset(end, start float64) bool {
return start > 0 && end > start && start >= end-start
}
func durationLooksImplausible(raw *ffprobeOutput, duration float64) bool {
if raw == nil {
return false
}
size := int64(parseFloat(raw.Format.Size))
return videoDurationImplausible(duration, size, hasVideoStream(raw.Streams))
}
func durationAfterStart(end, start float64) float64 {
if start <= 0 || end <= start {
return 0
}
duration := end - start
if !durationIsReasonable(duration) {
return 0
}
return duration
}
func durationAfterStartWithinValidatedLimit(end, start float64) float64 {
if start <= 0 || end <= start {
return 0
}
duration := end - start
if !durationIsWithinValidatedLimit(duration) {
return 0
}
return duration
}
func durationIsReasonable(duration float64) bool {
return durationIsPositiveFinite(duration) && duration <= maxReasonableMediaDurationSeconds
}
func durationIsWithinValidatedLimit(duration float64) bool {
return durationIsPositiveFinite(duration) && duration <= maxValidatedMediaDurationSeconds
}
func durationIsPositiveFinite(duration float64) bool {
return duration > 0 && !math.IsNaN(duration) && !math.IsInf(duration, 0)
}
func roundedDuration(duration float64) int {
return max(1, int(math.Round(duration)))
}
func truncatedDuration(duration float64) int {
return max(1, int(duration))
}
// isMainVideoStream reports whether the stream is a real video stream.
// Embedded cover art (attached_pic) is reported by ffprobe as a video stream
// but must not drive duration decisions: it would route audiobooks and music
// through the video duration gauntlet and packet-scan a single still image.
func isMainVideoStream(stream ffprobeStream) bool {
return stream.CodecType == "video" && stream.Disposition.AttachedPic == 0
}
func hasVideoStream(streams []ffprobeStream) bool {
return slices.ContainsFunc(streams, isMainVideoStream)
}
func primaryVideoFrameRate(streams []ffprobeStream) (string, bool) {
for _, stream := range streams {
if isMainVideoStream(stream) {
return stream.AvgFrameRate, true
}
}
return "", false
}
// probeVideoPacketDuration derives a duration for files whose duration
// metadata is unusable by scanning video packet timestamps. It intentionally
// demuxes the whole file: the timestamps being repaired are the same ones
// ffprobe would need for reliable interval seeking, so sampling cannot be
// trusted here. The repair layer keeps this one-shot per file.
func probeVideoPacketDuration(
ctx context.Context,
ffprobePath string,
filePath string,
frameRate string,
) (int, error) {
cmd := exec.CommandContext(ctx, ffprobePath,
"-v", "error",
"-select_streams", "v:0",
"-show_entries", "packet=pts_time",
"-of", "csv=p=0",
filePath,
)
stdout, err := cmd.StdoutPipe()
if err != nil {
return 0, fmt.Errorf("opening ffprobe packet output: %w", err)
}
if err := cmd.Start(); err != nil {
return 0, fmt.Errorf("starting ffprobe packet scan: %w", err)
}
duration := estimateVideoPacketDuration(stdout, frameRate)
if err := cmd.Wait(); err != nil {
return 0, fmt.Errorf("ffprobe packet scan failed for %s: %w", filePath, err)
}
return duration, nil
}
func estimateVideoPacketDuration(reader io.Reader, frameRate string) int {
scanner := bufio.NewScanner(reader)
packetCount := 0
minTimestamp := math.Inf(1)
maxTimestamp := math.Inf(-1)
for scanner.Scan() {
value := strings.TrimSpace(scanner.Text())
if value == "" {
continue
}
packetCount++
timestamp, err := strconv.ParseFloat(value, 64)
if err != nil {
continue
}
minTimestamp = min(minTimestamp, timestamp)
maxTimestamp = max(maxTimestamp, timestamp)
}
packetSpan := 0.0
if !math.IsInf(minTimestamp, 1) && !math.IsInf(maxTimestamp, -1) {
span := maxTimestamp - minTimestamp
if durationIsWithinValidatedLimit(span) {
packetSpan = span
}
}
frameDuration := 0.0
if fps := parseFrameRate(frameRate); fps > 0 && packetCount > 0 {
frameDuration = float64(packetCount) / fps
}
best := packetSpan
if packetSpan > maxReasonableMediaDurationSeconds &&
durationIsReasonable(frameDuration) &&
!longVideoDurationsAgree(packetSpan, frameDuration) {
// A long PTS span is strong evidence only when a sane frame-count
// estimate contradicts it. Malformed frame rates can produce finite but
// unusable estimates and must not veto an otherwise valid packet span.
best = 0
}
if durationIsReasonable(frameDuration) && frameDuration > best {
best = frameDuration
}
if best <= 0 {
return 0
}
return roundedDuration(best)
}
// parseFrameRate parses ffprobe's rational frame-rate shape ("30000/1001")
// or a plain float, returning 0 when unparsable. normalizeFrameRate formats
// the same parse for persistence; keep the parsing logic here only.
func parseFrameRate(raw string) float64 {
raw = strings.TrimSpace(raw)
parts := strings.SplitN(raw, "/", 2)
if len(parts) != 2 {
fps, _ := strconv.ParseFloat(raw, 64)
return fps
}
numerator, err := strconv.ParseFloat(parts[0], 64)
if err != nil {
return 0
}
denominator, err := strconv.ParseFloat(parts[1], 64)
if err != nil || denominator == 0 {
return 0
}
return numerator / denominator
}
func parseNumeric(raw string) int {
if raw == "" {
return 0
}
v, err := strconv.Atoi(raw)
if err != nil {
return 0
}
return v
}
func parseFloat(raw string) float64 {
if raw == "" {
return 0
}
value, err := strconv.ParseFloat(raw, 64)
if err != nil {
return 0
}
return value
}
func normalizeChapters(raw []ffprobeChapter, durationSeconds int) []ChapterInfo {
if len(raw) == 0 {
return []ChapterInfo{}
}
limit := float64(durationSeconds)
type chapterRange struct {
title string
start float64
end float64
}
ranges := make([]chapterRange, 0, len(raw))
for _, chapter := range raw {
start := parseFloat(string(chapter.StartTime))
end := parseFloat(string(chapter.EndTime))
if end <= 0 {
end = parseFloat(string(chapter.End))
}
if start <= 0 {
start = parseFloat(string(chapter.Start))
}
if limit > 0 {
if start < 0 {
start = 0
}
if end > limit {
end = limit
}
}
if end <= start {
continue
}
title := strings.TrimSpace(firstNonEmpty(
chapter.Tags["title"],
chapter.Tags["TITLE"],
))
ranges = append(ranges, chapterRange{
title: title,
start: start,
end: end,
})
}
if len(ranges) == 0 {
return []ChapterInfo{}
}
slices.SortStableFunc(ranges, func(a, b chapterRange) int {
switch {
case a.start < b.start:
return -1
case a.start > b.start:
return 1
case a.end < b.end:
return -1
case a.end > b.end:
return 1
default:
return 0
}
})
chapters := make([]ChapterInfo, 0, len(ranges))
for i, chapter := range ranges {
end := chapter.end
if i+1 < len(ranges) && ranges[i+1].start < end {
end = ranges[i+1].start
}
if end <= chapter.start {
continue
}
title := chapter.title
if title == "" {
title = fmt.Sprintf("Chapter %02d", len(chapters)+1)
}
chapters = append(chapters, ChapterInfo{
Index: len(chapters),
Title: title,
StartSeconds: chapter.start,
EndSeconds: end,
Source: "embedded",
})
}
return chapters
}
func parseBitDepth(s ffprobeStream) int {
if v := parseNumeric(string(s.BitsPerRawSample)); v > 0 {
return v
}
return parseNumeric(string(s.BitsPerSample))
}
func normalizeFrameRate(raw string) string {
if raw == "" || raw == "0/0" {
return ""
}
if !strings.Contains(raw, "/") {
return raw
}
fps := parseFrameRate(raw)
if fps == 0 {
return raw
}
return strconv.FormatFloat(fps, 'f', 3, 64)
}
func isInterlaced(fieldOrder string) bool {
switch strings.ToLower(fieldOrder) {
case "tt", "bb", "tb", "bt":
return true
default:
return false
}
}
func videoRangeLabel(s ffprobeStream) string {
if dv := dolbyVisionProfile(s.SideDataList); dv != "" {
return "DolbyVision"
}
if isHDR(s.ColorTransfer) {
return "HDR"
}
return ""
}
func dolbyVisionProfile(sideData []ffprobeSideData) string {
if profile := dolbyVisionProfileNumber(sideData); profile > 0 {
return fmt.Sprintf("Profile %d", profile)
}
return ""
}
func dolbyVisionProfileNumber(sideData []ffprobeSideData) int {
for _, data := range sideData {
if strings.EqualFold(data.SideDataType, "DOVI configuration record") && data.DVProfile > 0 {
return data.DVProfile
}
}
return 0
}
func dolbyVisionBLCompatID(sideData []ffprobeSideData) int {
for _, data := range sideData {
if strings.EqualFold(data.SideDataType, "DOVI configuration record") && data.DVBLCompatID > 0 {
return data.DVBLCompatID
}
}
return 0
}
func dolbyVisionELPresent(sideData []ffprobeSideData) bool {
for _, data := range sideData {
if strings.EqualFold(data.SideDataType, "DOVI configuration record") {
return data.DVElPresent > 0
}
}
return false
}
// dolbyVisionEnhancementLayer remains conservative until a libdovi-backed
// analyzer has inspected the RPU mapping. ffprobe can prove that an enhancement
// layer exists, but it cannot distinguish MEL from FEL.
func dolbyVisionEnhancementLayer(present bool) string {
if !present {
return "none"
}
return "unknown"
}
func hasHDR10Plus(sideData []ffprobeSideData) bool {
for _, data := range sideData {
typ := strings.ToLower(data.SideDataType)
if strings.Contains(typ, "hdr10+") || strings.Contains(typ, "smpte2094-40") {
return true
}
}
return false
}
func videoRangeType(s ffprobeStream) string {
profile := dolbyVisionProfileNumber(s.SideDataList)
hdr10Plus := hasHDR10Plus(s.SideDataList)
if profile > 0 {
switch profile {
case 5:
return "DOVI"
case 7:
if hdr10Plus {
return "DOVIWithELHDR10Plus"
}
return "DOVIWithEL"
case 8:
if hdr10Plus {
return "DOVIWithHDR10Plus"
}
switch dolbyVisionBLCompatID(s.SideDataList) {
case 1:
return "DOVIWithHDR10"
case 2:
return "DOVIWithSDR"
case 4:
return "DOVIWithHLG"
default:
if isHLG(s.ColorTransfer) {
return "DOVIWithHLG"
}
if isHDR(s.ColorTransfer) {
return "DOVIWithHDR10"
}
return "DOVIWithSDR"
}
default:
return "DOVI"
}
}
if hdr10Plus {
return "HDR10Plus"
}
if isHLG(s.ColorTransfer) {
return "HLG"
}
if isHDR(s.ColorTransfer) {
return "HDR10"
}
return "SDR"
}
func subtitleResolutionLabel(s ffprobeStream) string {
if s.Width <= 0 || s.Height <= 0 {
return ""
}
return fmt.Sprintf("%dx%d", s.Width, s.Height)
}
func dispositionFlag(tags map[string]string, key string) bool {
if tags == nil {
return false
}
value := strings.TrimSpace(strings.ToLower(tags[key]))
return value == "1" || value == "true" || value == "yes"
}
func firstNonEmpty(values ...string) string {
for _, value := range values {
if strings.TrimSpace(value) != "" {
return value
}
}
return ""
}
// mapResolution converts video dimensions to a standard resolution string.
// Uses upper-bound bucketing (similar to Jellyfin) checking both width and
// height, which correctly handles ultra-wide and non-standard aspect ratios.
func mapResolution(width, height int) string {
switch {
case width <= 0 && height <= 0:
return ""
case width <= 854 && height <= 480:
return "480p"
case width <= 1280 && height <= 962:
return "720p"
case width <= 2560 && height <= 1440:
return "1080p"
case width <= 4096 && height <= 3072:
return "2160p"
case width <= 8192 && height <= 6144:
return "4320p"
default:
return "2160p"
}
}
// isHDR checks whether the color transfer characteristic indicates HDR content.
func isHDR(colorTransfer string) bool {
ct := strings.ToLower(colorTransfer)
return strings.Contains(ct, "smpte2084") || strings.Contains(ct, "arib-std-b67")
}
func isHLG(colorTransfer string) bool {
return strings.Contains(strings.ToLower(colorTransfer), "arib-std-b67")
}
// normalizeFormatTags lowercases tag keys so callers can look up
// "title", "artist", "album" without worrying about ffprobe's mixed-case
// output. Trims whitespace from values.
func normalizeFormatTags(raw map[string]string) map[string]string {
if len(raw) == 0 {
return nil
}
out := make(map[string]string, len(raw))
for k, v := range raw {
out[strings.ToLower(strings.TrimSpace(k))] = strings.TrimSpace(v)
}
return out
}
// detectContainer maps ffprobe format names to common container names.
func detectContainer(formatName string) string {
// ffprobe format_name can contain multiple names separated by commas
// e.g. "mov,mp4,m4a,3gp,3g2,mj2"
parts := strings.Split(formatName, ",")
for _, p := range parts {
p = strings.TrimSpace(p)
switch p {
case "matroska", "webm":
return "mkv"
case "mov", "mp4", "m4a":
return "mp4"
case "avi":
return "avi"
case "mpegts":
return "ts"
case "flv":
return "flv"
case "ogg":
return "ogg"
case "wmv", "asf":
return "wmv"
}
}
// Fallback: return first part
if len(parts) > 0 && parts[0] != "" {
return strings.TrimSpace(parts[0])
}
return formatName
}