Files
silo-server/web/src/player/components/PlayerControls.tsx
T
39ba284c9d feat(ai): shared AI core — metadata translation, Whisper ASR, per-profile language, on-view translation (#127)
* docs: design + plan for shared AI core, metadata translation, Whisper ASR

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(ai): shared LLM client, segment translator, and job runner packages

internal/ai/llm: OpenAI-compatible chat client moved out of subtitles/ai,
plus /v1/audio/transcriptions (verbose_json) for the ASR work; one shared
retry/backoff loop for both. internal/ai/translate: the batched indexed-JSON
translation protocol generalized to text segments. internal/ai/jobrunner:
dispatch/heartbeat/reaper/cancel lifecycle extracted behind a minimal store
interface, with a semaphore shareable across job services.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(subtitles): consume shared AI core

LLMTranslator becomes a thin cue<->segment adapter over aitranslate; the
service delegates dispatch/heartbeat/reaper/cancel to jobrunner; the local
OpenAI client is gone in favor of internal/ai/llm. Behavior (prompts, wire
protocol, job rows, recovery semantics) is unchanged. NewService now takes
the dispatch semaphore so all AI job services can share one bound.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(config): shared ai.* settings, metadata translation job table, localization provenance columns

ai.* connection keys (chat + optional separate ASR endpoint) load with a
fallback to the legacy subtitle_ai.* rows — those are never renamed in SQL
because encrypted values are GCM-bound to their setting key. New toggles:
subtitle_ai.transcribe_enabled, metadata_ai.enabled. Migration adds
metadata_translation_jobs, per-field provenance (provider|ai|manual) on the
localization tables, and media_folders.auto_translate_metadata.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(catalog): localization field provenance with provider/ai/manual precedence

Provider upserts keep manual values and never blank a field with an empty
incoming value; new UpsertAITranslation/UpsertAIOverview methods write AI
fields only over empty or ai-sourced values (force adds provider, never
manual) — all enforced in single-statement SQL. Serving now merges only
non-empty localized fields onto the base item, since localization rows are
legitimately partial (AI rows carry no titles/artwork).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(metadata): AI translation service, refresh auto-fallback, and admin API

internal/metadata/translation: job service over the shared AI core that
expands an item to its season/episode overviews, skips already-localized
fields (zero model calls on repeat runs), batches paragraphs through the
generic translator, and persists per batch with provenance-aware upserts.
MetadataService gains an AutoTranslator seam invoked after each refresh for
libraries with auto_translate_metadata. Admin endpoints under the metadata
curation guard: enqueue, list (poll), cancel; plus a status probe.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(subtitles): Whisper ASR transcribe and transcribe_translate jobs

New WhisperTranscriber: one ffmpeg pass extracts the audio track to 10-min
16kHz mono WAV chunks (temp dir cleaned on every exit path), each chunk goes
to the OpenAI-compatible /v1/audio/transcriptions endpoint (verbose_json,
per-request timeout sized to 3x chunk duration), segment timestamps are
offset and built into wrapped cues. Chunks process playhead-first and stream
live to the requesting session. The transcript is stored as an ordinary
downloaded subtitle (provider 'transcribed'); transcribe_translate chains
the existing translator and stores the translated track as the job result.
Enqueue accepts an optional kind; status reports transcribe_enabled.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(web): AI services settings, metadata translate action, library auto-translate, generate-from-audio

New AI Services admin page hosts the shared endpoint config (reads fall back
to legacy subtitle_ai.* values, writes target ai.*) and the three feature
toggles; the AI card moves out of Subtitles settings. The metadata editor
gains a Translate-with-AI panel with job polling and force/re-translate. The
library form gains the auto-translate toggle (threaded through the libraries
API). The player translate modal gains a From-audio mode that lists audio
tracks and submits transcribe / transcribe_translate jobs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* style: gofmt import grouping in router and translation tests

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(catalog): per-profile metadata language and viewer-triggered description translation

user_profiles.preferred_metadata_language threads through the access scope
into catalog serving: presentation language now resolves explicit param ->
profile preference -> library metadata language (native API and jellycompat).
ItemDetail gains pending_translation_language when the viewer's language is
missing a localized overview. New metadata_ai.on_view setting (off|button|
auto) gates POST /items/{id}/translate-description: any profile with item
access may request its language, with in-flight dedup and a 15-minute
failure cooldown so page views never hammer a broken endpoint.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(web): on-view description translation with per-profile metadata language

Profile playback settings gain a Metadata language picker (library default
inherit). Detail pages: when the server reports pending_translation_language
and metadata_ai.on_view is 'auto', the description translates on view with a
pulse animation until the refetched detail comes back localized (45s
timeout); in 'button' mode a small Translate chip triggers the same flow.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(web): expose metadata_ai.on_view in AI Services settings

The on-view translation mode had no UI control, so it could only ever be
'off' — viewers got neither the auto translation nor the fallback button.
Adds the off/button/auto selector to the Features card, and the config
loader now warns and falls back to 'off' on a bad row instead of refusing
to start.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai): clear configuration hint when the transcription endpoint is chat-only

A blank Transcription base URL falls back to the chat endpoint; chat-only
gateways reject the multipart upload with an opaque 400 that reads like a
pipeline bug. 400/404/405 transcription failures now carry a hint to set a
Whisper-compatible endpoint in AI Services.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(subtitles): wrap ASR cue text by rune count, not bytes

Arabic/Cyrillic/Greek text is 2+ bytes per character in UTF-8, so byte-based
wrapping broke lines at roughly half the intended visual width.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(web): steer transcription base URL hint away from chat-only gateways

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(ai): block chat-only gateways for transcription, add endpoint presets

llm.IsChatOnlyGateway (OpenRouter et al — no timestamped transcription API)
is enforced in three layers: the settings API rejects ai.asr_base_url values
pointing at one, the router disables ASR with a warning when the blank-URL
fallback would land on one, and llm.Transcribe refuses outright. The AI
Services page gains one-click transcription presets (Groq turbo/accurate,
OpenAI, self-hosted speaches) plus the mirrored client-side check, and the
settings API now also validates metadata_ai.on_view.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(subtitles): tighten ASR subtitle sync

Three systematic timing-error sources addressed: cue offsets now use the
segment muxer's exact per-chunk start times (segment_list CSV) instead of
assuming index*chunk_seconds; the audio stream's start delay relative to the
container timeline (common in TS remuxes) is probed via ffprobe and added to
every cue; and the chunk length is now operator-tunable via
subtitle_ai.asr_chunk_seconds (60-600s, default 600) since shorter chunks
bound Whisper's within-chunk timestamp drift. Playhead-first ordering now
pivots on real chunk starts, and a beyond-end playhead starts at the final
chunk instead of restarting from zero.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai): tolerate base URLs that already include the /v1 segment

Providers like DeepInfra expose their OpenAI-compatible API under a base
that contains the version segment (api.deepinfra.com/v1/openai); always
appending /v1/... mangled those. endpointURL now appends bare paths when
the base already carries /v1.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(web): prefer self-hosted transcription in presets and hints

Preset order becomes self-hosted (recommended) -> Groq turbo -> Groq
large-v3 -> OpenAI, and the settings hint plus the job-error hint lead with
the self-hosted option. The self-hosted preset now fills the turbo CT2 model
to match the recommended speaches setup.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(subtitles): request VAD and word timestamps for ASR cue accuracy

Without vad_filter, faster-whisper servers report wall-to-wall segment
times: cues linger on screen through silence (verified up to 91s) and
paragraph-length segments become single 400+ char cues. Request
vad_filter=true (skipped for hosted providers that reject non-OpenAI
fields and run VAD server-side) plus timestamp_granularities word+segment,
and rebuild cues from word timings: split at speech pauses, sentence ends,
text capacity, and a 7s max duration; cap word-less segments instead of
trusting their reported end; stretch sub-second cues to a readable minimum.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 14:58:54 -04:00

427 lines
14 KiB
TypeScript

import {
Info,
Maximize,
Minimize,
Pause,
PictureInPicture2,
Play,
RotateCcw,
RotateCw,
SkipBack,
SkipForward,
Tags,
} from "lucide-react";
import { CircleButton } from "./CircleButton";
import { SeekBar, formatTime } from "./SeekBar";
import { VolumeControl } from "./VolumeControl";
import { QualityMenu } from "./QualityMenu";
import { SubtitleMenu } from "./SubtitleMenu";
import { AudioTrackMenu } from "./AudioTrackMenu";
import { ChaptersMenu } from "./ChaptersMenu";
import type {
MarkerKind,
MarkerRegionView,
PlayerAudioTrack,
PlayerChapter,
PlayerSubtitleInfo,
QualityOption,
} from "../types";
import type { VersionInfo } from "./QualityMenu";
import type { PlayerConfig } from "../context/PlayerConfigContext";
interface PlayerControlsProps {
// Visibility
visible: boolean;
// Video state
playing: boolean;
currentTime: number;
duration: number;
buffered: TimeRanges | null;
// Seek bar markers
chapters?: PlayerChapter[];
regions?: MarkerRegionView[];
// Marker editing
editing?: boolean;
activeEditKind?: MarkerKind | null;
onRegionEdgeChange?: (kind: MarkerKind, edge: "start" | "end", seconds: number) => void;
markerEditAvailable?: boolean;
markerEditActive?: boolean;
onToggleMarkerEdit?: () => void;
volume: number;
muted: boolean;
isFullscreen: boolean;
// Subtitles
subtitleTracks: PlayerSubtitleInfo[];
activeSubtitleIndex: number | null;
onSubtitleSelect: (index: number | null) => void;
subtitleDelayMs: number;
onSubtitleDelayChange: (ms: number) => void;
mediaFileId?: number;
playerConfig?: PlayerConfig;
onRefreshSubtitles?: () => void;
sessionId?: string;
getSubtitleStartPosition?: () => number;
// Audio
audioTracks: PlayerAudioTrack[];
activeAudioIndex: number;
onAudioSelect?: (index: number, currentPosition: number) => void;
// Quality
qualityOptions: QualityOption[];
activeQualityId: string;
isTranscoding: boolean;
qualityError: string | null;
onQualitySelect: (id: string) => void;
// Version switching
versions?: VersionInfo[];
onSwitchVersion?: (fileId: number) => void;
// PiP
onTogglePiP?: () => void;
// Playback info
showPlaybackInfo: boolean;
onTogglePlaybackInfo: () => void;
// Episode navigation
hasPrevEpisode?: boolean;
hasNextEpisode?: boolean;
onPrevEpisode?: () => void;
onNextEpisode?: () => void;
// Title strip
title?: string;
subtitleLabel?: string;
// Callbacks
onPlayPause: () => void;
onSeek: (seconds: number) => void;
onVolumeChange: (volume: number) => void;
onMutedChange: (muted: boolean) => void;
onFullscreenToggle: () => void;
}
/** Skip amount for the ±seconds buttons, matching keyboard shortcuts. */
const SKIP_BACK_SECONDS = 10;
const SKIP_FORWARD_SECONDS = 30;
export function PlayerControls({
visible,
playing,
currentTime,
duration,
buffered,
chapters,
regions,
editing,
activeEditKind,
onRegionEdgeChange,
markerEditAvailable = false,
markerEditActive = false,
onToggleMarkerEdit,
volume,
muted,
isFullscreen,
subtitleTracks,
activeSubtitleIndex,
onSubtitleSelect,
subtitleDelayMs,
onSubtitleDelayChange,
mediaFileId,
playerConfig,
onRefreshSubtitles,
sessionId,
getSubtitleStartPosition,
audioTracks,
activeAudioIndex,
onAudioSelect,
qualityOptions,
activeQualityId,
isTranscoding,
qualityError,
onQualitySelect,
versions,
onSwitchVersion,
onTogglePiP,
showPlaybackInfo,
onTogglePlaybackInfo,
hasPrevEpisode = false,
hasNextEpisode = false,
onPrevEpisode,
onNextEpisode,
title,
subtitleLabel,
onPlayPause,
onSeek,
onVolumeChange,
onMutedChange,
onFullscreenToggle,
}: PlayerControlsProps) {
const safeDuration = duration > 0 ? duration : 0;
const handleSkipBack = () => onSeek(Math.max(0, currentTime - SKIP_BACK_SECONDS));
const handleSkipForward = () =>
onSeek(Math.min(safeDuration || currentTime, currentTime + SKIP_FORWARD_SECONDS));
// When playing any episode in a series (even the first or last), reserve
// both prev/next slots so the cluster remains symmetric around the play
// button. Movies (no episode nav at all) skip the slots entirely.
const showEpisodeSlots = hasPrevEpisode || hasNextEpisode;
return (
<div
className={`player-controls absolute inset-0 z-10 transition-opacity duration-300 ${
visible ? "opacity-100" : "pointer-events-none opacity-0"
}`}
onClick={onPlayPause}
>
{/* Gradient scrim — darkens the bottom of the frame just enough that
white controls stay legible without washing out the picture. */}
<div className="player-scrim pointer-events-none absolute inset-0" />
{/* ───── BOTTOM HUD ─────
Three-column grid: metadata left, main playback cluster center,
utility rail right. Seek bar spans the full width above the row
so the playhead is always anchored to the frame edge. */}
<div
className="player-hud player-rise absolute inset-x-0 bottom-0 z-10 px-3 pt-4 pb-3 sm:px-6 sm:pb-5"
onClick={(e) => e.stopPropagation()}
>
<SeekBar
currentTime={currentTime}
duration={duration}
buffered={buffered}
chapters={chapters}
regions={regions}
editing={editing}
activeEditKind={activeEditKind}
onRegionEdgeChange={onRegionEdgeChange}
onSeek={onSeek}
/>
{/* Grid keeps the playback cluster visually locked to the centerline
of the frame regardless of how long the title or utility rail is.
`minmax(0,1fr)` forces the side columns to honor 1fr behaviour
rather than growing with their content — the cluster stays put. */}
<div className="mt-2 grid grid-cols-[minmax(0,1fr)_auto_minmax(0,1fr)] items-center gap-3 sm:gap-5">
{/* ─── Left: Title / episode / time ─── */}
<div className="flex min-w-0 flex-col gap-0.5">
{title ? (
<div
className="truncate text-[15px] leading-tight font-semibold tracking-tight text-white sm:text-base"
title={title}
>
{title}
</div>
) : null}
<div className="flex items-center gap-2 text-[10px] leading-tight text-white/55 uppercase">
{subtitleLabel ? (
<>
<span
className="truncate tracking-[0.22em]"
style={{ maxWidth: "38ch" }}
title={subtitleLabel}
>
{subtitleLabel}
</span>
<span className="text-white/25">·</span>
</>
) : null}
<span className="font-mono text-[11px] tracking-[0.12em] text-white/75 normal-case tabular-nums">
{formatTime(currentTime)}
<span className="mx-1 text-white/30">/</span>
{formatTime(duration)}
</span>
</div>
</div>
{/* ─── Center: Main playback cluster ─── */}
{/* When ANY episode navigation exists (series context), both the
prev and next slots are reserved so the play button stays
exactly on the cluster's centerline even at the first or last
episode. For movies the slots are omitted entirely, leaving a
symmetric 3-button cluster that's also perfectly centered. */}
<div className="flex items-center justify-center gap-2 sm:gap-3">
{showEpisodeSlots ? (
hasPrevEpisode ? (
<CircleButton
size="sm"
variant="secondary"
ariaLabel="Previous episode"
onClick={onPrevEpisode}
>
<SkipBack className="h-[18px] w-[18px]" fill="currentColor" />
</CircleButton>
) : (
<ClusterSlotSpacer size="sm" />
)
) : null}
<CircleButton
size="sm"
variant="secondary"
ariaLabel={`Back ${SKIP_BACK_SECONDS} seconds`}
onClick={handleSkipBack}
>
<SkipIcon direction="back" seconds={SKIP_BACK_SECONDS} />
</CircleButton>
<CircleButton
size="md"
variant="primary"
ariaLabel={playing ? "Pause" : "Play"}
onClick={onPlayPause}
data-paused={!playing}
>
{playing ? (
<Pause className="h-6 w-6" strokeWidth={0} fill="currentColor" />
) : (
<Play className="ml-[2px] h-6 w-6" strokeWidth={0} fill="currentColor" />
)}
</CircleButton>
<CircleButton
size="sm"
variant="secondary"
ariaLabel={`Forward ${SKIP_FORWARD_SECONDS} seconds`}
onClick={handleSkipForward}
>
<SkipIcon direction="forward" seconds={SKIP_FORWARD_SECONDS} />
</CircleButton>
{showEpisodeSlots ? (
hasNextEpisode ? (
<CircleButton
size="sm"
variant="secondary"
ariaLabel="Next episode"
onClick={onNextEpisode}
>
<SkipForward className="h-[18px] w-[18px]" fill="currentColor" />
</CircleButton>
) : (
<ClusterSlotSpacer size="sm" />
)
) : null}
</div>
{/* ─── Right: Utility rail ─── */}
<div className="flex items-center justify-end gap-0.5">
<div className="hidden sm:block">
<VolumeControl
volume={volume}
muted={muted}
onVolumeChange={onVolumeChange}
onMutedChange={onMutedChange}
/>
</div>
<div className="player-hud-divider mx-1 hidden sm:block" />
{onAudioSelect && (
<AudioTrackMenu
tracks={audioTracks}
activeIndex={activeAudioIndex}
onSelect={onAudioSelect}
currentPosition={currentTime}
/>
)}
<ChaptersMenu chapters={chapters ?? []} currentTime={currentTime} onSeek={onSeek} />
<SubtitleMenu
tracks={subtitleTracks}
activeIndex={activeSubtitleIndex}
onSelect={onSubtitleSelect}
delayMs={subtitleDelayMs}
onDelayChange={onSubtitleDelayChange}
mediaFileId={mediaFileId}
playerConfig={playerConfig}
onRefreshSubtitles={onRefreshSubtitles}
sessionId={sessionId}
getSubtitleStartPosition={getSubtitleStartPosition}
audioTracks={audioTracks}
/>
<QualityMenu
options={qualityOptions}
activeId={activeQualityId}
isTranscoding={isTranscoding}
error={qualityError}
onSelect={onQualitySelect}
versions={versions}
onSwitchVersion={onSwitchVersion}
/>
{markerEditAvailable && onToggleMarkerEdit && (
<button
type="button"
className="player-utility-btn"
onClick={onToggleMarkerEdit}
aria-label="Edit markers"
aria-pressed={markerEditActive}
title="Edit markers"
data-active={markerEditActive ? "true" : "false"}
>
<Tags className="h-[18px] w-[18px]" />
</button>
)}
<button
type="button"
className="player-utility-btn"
onClick={onTogglePlaybackInfo}
aria-label="Playback info"
data-active={showPlaybackInfo ? "true" : "false"}
>
<Info className="h-[18px] w-[18px]" />
</button>
{onTogglePiP && document.pictureInPictureEnabled && (
<button
type="button"
className="player-utility-btn"
onClick={onTogglePiP}
aria-label="Picture in Picture (P)"
title="Picture in Picture (P)"
>
<PictureInPicture2 className="h-[18px] w-[18px]" />
</button>
)}
<button
type="button"
className="player-utility-btn"
onClick={onFullscreenToggle}
aria-label={isFullscreen ? "Exit fullscreen" : "Fullscreen"}
>
{isFullscreen ? (
<Minimize className="h-[18px] w-[18px]" />
) : (
<Maximize className="h-[18px] w-[18px]" />
)}
</button>
</div>
</div>
</div>
</div>
);
}
/* ─────────────────────────────────────────────────────────────────────
Internal building blocks
───────────────────────────────────────────────────────────────────── */
/** Invisible placeholder that reserves the exact footprint of a CircleButton
* so the playback cluster stays symmetric when a neighboring episode isn't
* available (first/last episode in a series). */
function ClusterSlotSpacer({ size }: { size: "sm" | "md" }) {
const sizing = size === "md" ? "h-12 w-12 sm:h-14 sm:w-14" : "h-10 w-10 sm:h-11 sm:w-11";
return <div aria-hidden="true" className={sizing} />;
}
/** Curved arrow with the skip-seconds number centered in the loop. */
function SkipIcon({ direction, seconds }: { direction: "back" | "forward"; seconds: number }) {
const Arrow = direction === "back" ? RotateCcw : RotateCw;
return (
<span className="relative flex h-7 w-7 items-center justify-center">
<Arrow className="h-7 w-7" strokeWidth={1.6} />
<span className="absolute inset-0 flex items-center justify-center pb-[1px] text-[8.5px] font-semibold tracking-tight tabular-nums">
{seconds}
</span>
</span>
);
}