Files
silo-server/internal/models/media.go
T
39ba284c9d feat(ai): shared AI core — metadata translation, Whisper ASR, per-profile language, on-view translation (#127)
* docs: design + plan for shared AI core, metadata translation, Whisper ASR

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(ai): shared LLM client, segment translator, and job runner packages

internal/ai/llm: OpenAI-compatible chat client moved out of subtitles/ai,
plus /v1/audio/transcriptions (verbose_json) for the ASR work; one shared
retry/backoff loop for both. internal/ai/translate: the batched indexed-JSON
translation protocol generalized to text segments. internal/ai/jobrunner:
dispatch/heartbeat/reaper/cancel lifecycle extracted behind a minimal store
interface, with a semaphore shareable across job services.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(subtitles): consume shared AI core

LLMTranslator becomes a thin cue<->segment adapter over aitranslate; the
service delegates dispatch/heartbeat/reaper/cancel to jobrunner; the local
OpenAI client is gone in favor of internal/ai/llm. Behavior (prompts, wire
protocol, job rows, recovery semantics) is unchanged. NewService now takes
the dispatch semaphore so all AI job services can share one bound.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(config): shared ai.* settings, metadata translation job table, localization provenance columns

ai.* connection keys (chat + optional separate ASR endpoint) load with a
fallback to the legacy subtitle_ai.* rows — those are never renamed in SQL
because encrypted values are GCM-bound to their setting key. New toggles:
subtitle_ai.transcribe_enabled, metadata_ai.enabled. Migration adds
metadata_translation_jobs, per-field provenance (provider|ai|manual) on the
localization tables, and media_folders.auto_translate_metadata.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(catalog): localization field provenance with provider/ai/manual precedence

Provider upserts keep manual values and never blank a field with an empty
incoming value; new UpsertAITranslation/UpsertAIOverview methods write AI
fields only over empty or ai-sourced values (force adds provider, never
manual) — all enforced in single-statement SQL. Serving now merges only
non-empty localized fields onto the base item, since localization rows are
legitimately partial (AI rows carry no titles/artwork).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(metadata): AI translation service, refresh auto-fallback, and admin API

internal/metadata/translation: job service over the shared AI core that
expands an item to its season/episode overviews, skips already-localized
fields (zero model calls on repeat runs), batches paragraphs through the
generic translator, and persists per batch with provenance-aware upserts.
MetadataService gains an AutoTranslator seam invoked after each refresh for
libraries with auto_translate_metadata. Admin endpoints under the metadata
curation guard: enqueue, list (poll), cancel; plus a status probe.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(subtitles): Whisper ASR transcribe and transcribe_translate jobs

New WhisperTranscriber: one ffmpeg pass extracts the audio track to 10-min
16kHz mono WAV chunks (temp dir cleaned on every exit path), each chunk goes
to the OpenAI-compatible /v1/audio/transcriptions endpoint (verbose_json,
per-request timeout sized to 3x chunk duration), segment timestamps are
offset and built into wrapped cues. Chunks process playhead-first and stream
live to the requesting session. The transcript is stored as an ordinary
downloaded subtitle (provider 'transcribed'); transcribe_translate chains
the existing translator and stores the translated track as the job result.
Enqueue accepts an optional kind; status reports transcribe_enabled.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(web): AI services settings, metadata translate action, library auto-translate, generate-from-audio

New AI Services admin page hosts the shared endpoint config (reads fall back
to legacy subtitle_ai.* values, writes target ai.*) and the three feature
toggles; the AI card moves out of Subtitles settings. The metadata editor
gains a Translate-with-AI panel with job polling and force/re-translate. The
library form gains the auto-translate toggle (threaded through the libraries
API). The player translate modal gains a From-audio mode that lists audio
tracks and submits transcribe / transcribe_translate jobs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* style: gofmt import grouping in router and translation tests

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(catalog): per-profile metadata language and viewer-triggered description translation

user_profiles.preferred_metadata_language threads through the access scope
into catalog serving: presentation language now resolves explicit param ->
profile preference -> library metadata language (native API and jellycompat).
ItemDetail gains pending_translation_language when the viewer's language is
missing a localized overview. New metadata_ai.on_view setting (off|button|
auto) gates POST /items/{id}/translate-description: any profile with item
access may request its language, with in-flight dedup and a 15-minute
failure cooldown so page views never hammer a broken endpoint.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(web): on-view description translation with per-profile metadata language

Profile playback settings gain a Metadata language picker (library default
inherit). Detail pages: when the server reports pending_translation_language
and metadata_ai.on_view is 'auto', the description translates on view with a
pulse animation until the refetched detail comes back localized (45s
timeout); in 'button' mode a small Translate chip triggers the same flow.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(web): expose metadata_ai.on_view in AI Services settings

The on-view translation mode had no UI control, so it could only ever be
'off' — viewers got neither the auto translation nor the fallback button.
Adds the off/button/auto selector to the Features card, and the config
loader now warns and falls back to 'off' on a bad row instead of refusing
to start.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai): clear configuration hint when the transcription endpoint is chat-only

A blank Transcription base URL falls back to the chat endpoint; chat-only
gateways reject the multipart upload with an opaque 400 that reads like a
pipeline bug. 400/404/405 transcription failures now carry a hint to set a
Whisper-compatible endpoint in AI Services.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(subtitles): wrap ASR cue text by rune count, not bytes

Arabic/Cyrillic/Greek text is 2+ bytes per character in UTF-8, so byte-based
wrapping broke lines at roughly half the intended visual width.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(web): steer transcription base URL hint away from chat-only gateways

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(ai): block chat-only gateways for transcription, add endpoint presets

llm.IsChatOnlyGateway (OpenRouter et al — no timestamped transcription API)
is enforced in three layers: the settings API rejects ai.asr_base_url values
pointing at one, the router disables ASR with a warning when the blank-URL
fallback would land on one, and llm.Transcribe refuses outright. The AI
Services page gains one-click transcription presets (Groq turbo/accurate,
OpenAI, self-hosted speaches) plus the mirrored client-side check, and the
settings API now also validates metadata_ai.on_view.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(subtitles): tighten ASR subtitle sync

Three systematic timing-error sources addressed: cue offsets now use the
segment muxer's exact per-chunk start times (segment_list CSV) instead of
assuming index*chunk_seconds; the audio stream's start delay relative to the
container timeline (common in TS remuxes) is probed via ffprobe and added to
every cue; and the chunk length is now operator-tunable via
subtitle_ai.asr_chunk_seconds (60-600s, default 600) since shorter chunks
bound Whisper's within-chunk timestamp drift. Playhead-first ordering now
pivots on real chunk starts, and a beyond-end playhead starts at the final
chunk instead of restarting from zero.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai): tolerate base URLs that already include the /v1 segment

Providers like DeepInfra expose their OpenAI-compatible API under a base
that contains the version segment (api.deepinfra.com/v1/openai); always
appending /v1/... mangled those. endpointURL now appends bare paths when
the base already carries /v1.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(web): prefer self-hosted transcription in presets and hints

Preset order becomes self-hosted (recommended) -> Groq turbo -> Groq
large-v3 -> OpenAI, and the settings hint plus the job-error hint lead with
the self-hosted option. The self-hosted preset now fills the turbo CT2 model
to match the recommended speaches setup.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(subtitles): request VAD and word timestamps for ASR cue accuracy

Without vad_filter, faster-whisper servers report wall-to-wall segment
times: cues linger on screen through silence (verified up to 91s) and
paragraph-length segments become single 400+ char cues. Request
vad_filter=true (skipped for hosted providers that reject non-OpenAI
fields and run VAD server-side) plus timestamp_granularities word+segment,
and rebuild cues from word timings: split at speech pauses, sentence ends,
text capacity, and a 7s max duration; cap word-less segments instead of
trusting their reported end; stretch sub-second cues to a readable minimum.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 14:58:54 -04:00

474 lines
17 KiB
Go

package models
import (
"strings"
"time"
)
// MediaFolder represents a row in the media_folders table.
type MediaFolder struct {
ID int
Paths []string // from media_folder_paths child table
Type string // movies, series, mixed
Name string
Enabled bool
MetadataLanguage string // ISO 639-1 code (e.g. "en", "ja")
AutoTranslateMetadata bool // AI-translate descriptions when providers lack this language
ChapterThumbnailsEnabled bool
IntroDetectionEnabled bool
PosterPath string // S3 key for library poster image
LastScannedAt *time.Time // nullable
ScanWarningCode *string
ScanWarningMessage *string
ScanWarningAt *time.Time
AllowEmptyCleanupOnce bool
SortOrder int
}
// MediaFile represents a row in the media_files table.
type MediaFile struct {
ID int
ContentID string // Sonyflake ID (nullable until matched)
EpisodeID string // FK to episodes.content_id (nullable)
SeasonNumber int // parsed from filename (nullable)
EpisodeNumber int // parsed from filename (nullable)
MediaFolderID int
CanonicalRootPath string
ObservedRootPath string
ContentGroupKey string
GroupKeyVersion int
BaseTitle string
BaseYear int
BaseType string
IdentityConfidence string
IdentityJSON []byte
FilePath string
FileSize int64
FileModifiedAt *time.Time
FileHash string // OSHash (16-char hex)
CodecVideo string // h264, hevc, av1
CodecAudio string // aac, opus, flac
Resolution string // 1080p, 2160p
AudioChannels int
HDR bool
Container string // mkv, mp4
Duration int // seconds
Bitrate int // kbps
VideoTracks []VideoTrack // JSONB
AudioTracks []AudioTrack // JSONB
SubtitleTracks []SubtitleTrack // JSONB
ExternalSubtitles []ExternalSubtitle // JSONB
Chapters []MediaChapter // JSONB; nil means not yet probed for chapters
ChapterThumbnailRetryAfter *time.Time
ChapterThumbnailFailureCount int
ChapterThumbnailLastError string
IntroStart *float64
IntroEnd *float64
CreditsStart *float64
CreditsEnd *float64
RecapStart *float64
RecapEnd *float64
PreviewStart *float64
PreviewEnd *float64
MarkersSource *string
MarkersConfidence *float64
IntroMarkersSource *string
IntroMarkersProvider *string
IntroMarkersConfidence *float64
IntroMarkersAlgorithm *string
IntroMarkersDetectedAt *time.Time
CreditsMarkersSource *string
CreditsMarkersProvider *string
CreditsMarkersConfidence *float64
CreditsMarkersAlgorithm *string
CreditsMarkersDetectedAt *time.Time
RecapMarkersSource *string
RecapMarkersProvider *string
RecapMarkersConfidence *float64
RecapMarkersAlgorithm *string
RecapMarkersDetectedAt *time.Time
PreviewMarkersSource *string
PreviewMarkersProvider *string
PreviewMarkersConfidence *float64
PreviewMarkersAlgorithm *string
PreviewMarkersDetectedAt *time.Time
EditionRaw string
EditionKey string
EditionConfidence *float64
EditionSource string
PresentationKind string
PresentationGroupKey string
PresentationPartIndex int
PresentationPartTotal int
MultiEpisodeStart int
MultiEpisodeEnd int
ProbeSource string // arrs, local
ProbeUpdatedAt *time.Time
MatchAttemptedAt *time.Time
MissingSince *time.Time
CreatedAt time.Time
UpdatedAt time.Time
}
// MediaChapter represents a single media chapter derived from embedded file metadata.
type MediaChapter struct {
Index int `json:"index"`
Title string `json:"title"`
StartSeconds float64 `json:"start_seconds"`
EndSeconds float64 `json:"end_seconds"`
Source string `json:"source"`
ThumbnailPath string `json:"thumbnail_path,omitempty"`
ThumbnailThumbhash string `json:"thumbnail_thumbhash,omitempty"`
ThumbnailRetryAfter *time.Time `json:"thumbnail_retry_after,omitempty"`
ThumbnailFailedAt *time.Time `json:"thumbnail_failed_at,omitempty"`
ThumbnailLastError string `json:"thumbnail_last_error,omitempty"`
}
// OverlaySummary is the compact media badge payload shared across API surfaces.
type OverlaySummary struct {
Resolution string `json:"resolution,omitempty"`
HDR string `json:"hdr,omitempty"`
Audio string `json:"audio,omitempty"`
AudioChannels string `json:"audio_channels,omitempty"` // "Stereo", "5.1", "7.1"
VideoCodec string `json:"video_codec,omitempty"` // "H.264", "H.265", "AV1"
Container string `json:"container,omitempty"` // "MKV", "MP4"
AspectRatio string `json:"aspect_ratio,omitempty"` // "16:9", "2.39:1"
ReleaseType string `json:"release_type,omitempty"`
Edition string `json:"edition,omitempty"`
MultiAudio bool `json:"multi_audio,omitempty"` // ≥2 distinct audio languages
MultiSub bool `json:"multi_sub,omitempty"` // ≥1 subtitle track (embedded or external)
}
// VideoTrack represents a probed video stream stored as JSONB.
type VideoTrack struct {
Title string `json:"title,omitempty"`
Codec string `json:"codec,omitempty"`
DolbyVision string `json:"dolby_vision,omitempty"`
Profile string `json:"profile,omitempty"`
Level int `json:"level,omitempty"`
Width int `json:"width,omitempty"`
Height int `json:"height,omitempty"`
AspectRatio string `json:"aspect_ratio,omitempty"`
Interlaced bool `json:"interlaced"`
FrameRate string `json:"frame_rate,omitempty"`
Bitrate int `json:"bitrate,omitempty"`
VideoRange string `json:"video_range,omitempty"`
ColorPrimaries string `json:"color_primaries,omitempty"`
ColorSpace string `json:"color_space,omitempty"`
ColorTransfer string `json:"color_transfer,omitempty"`
BitDepth int `json:"bit_depth,omitempty"`
PixelFormat string `json:"pixel_format,omitempty"`
ReferenceFrames int `json:"reference_frames,omitempty"`
}
// AudioTrack represents a probed audio stream stored as JSONB.
type AudioTrack struct {
Title string `json:"title,omitempty"`
EmbeddedTitle string `json:"embedded_title,omitempty"`
Language string `json:"language,omitempty"`
Codec string `json:"codec,omitempty"`
Layout string `json:"layout,omitempty"`
Channels int `json:"channels,omitempty"`
Bitrate int `json:"bitrate,omitempty"`
SampleRate int `json:"sample_rate,omitempty"`
BitDepth int `json:"bit_depth,omitempty"`
Default bool `json:"default"`
}
// SubtitleTrack represents an embedded subtitle track stored as JSONB.
type SubtitleTrack struct {
Index int `json:"index"`
Language string `json:"language"`
Codec string `json:"codec"`
Title string `json:"title,omitempty"`
EmbeddedTitle string `json:"embedded_title,omitempty"`
Resolution string `json:"resolution,omitempty"`
Forced bool `json:"forced"`
Default bool `json:"default"`
HearingImpaired bool `json:"hearing_impaired"`
External bool `json:"external"`
FileName string `json:"file_name,omitempty"`
}
// ExternalSubtitle represents a sidecar subtitle file stored as JSONB.
type ExternalSubtitle struct {
Path string `json:"path"`
Language string `json:"language"`
Format string `json:"format"` // srt, vtt, ass, ssa, sub
Title string `json:"title,omitempty"`
EmbeddedTitle string `json:"embedded_title,omitempty"`
Resolution string `json:"resolution,omitempty"`
Forced bool `json:"forced"`
Default bool `json:"default"`
HearingImpaired bool `json:"hearing_impaired"`
}
// PersonKind identifies the role type a person has on a media item.
type PersonKind int
const (
PersonKindActor PersonKind = 1
PersonKindDirector PersonKind = 2
PersonKindWriter PersonKind = 3
PersonKindProducer PersonKind = 4
PersonKindGuestStar PersonKind = 5
PersonKindComposer PersonKind = 6
PersonKindAuthor PersonKind = 7
PersonKindNarrator PersonKind = 8
)
// String returns the Jellyfin-compatible type string for this PersonKind.
func (k PersonKind) String() string {
switch k {
case PersonKindActor:
return "Actor"
case PersonKindDirector:
return "Director"
case PersonKindWriter:
return "Writer"
case PersonKindProducer:
return "Producer"
case PersonKindGuestStar:
return "GuestStar"
case PersonKindComposer:
return "Composer"
case PersonKindAuthor:
return "Author"
case PersonKindNarrator:
return "Narrator"
default:
return "Unknown"
}
}
// PersonKindFromJob maps a crew job title to a PersonKind.
func PersonKindFromJob(job string) PersonKind {
switch strings.ToLower(strings.TrimSpace(job)) {
case "director":
return PersonKindDirector
case "writer", "screenplay", "story", "novel":
return PersonKindWriter
case "composer", "original music composer", "music":
return PersonKindComposer
default:
return PersonKindProducer
}
}
// Person represents a deduplicated person entity.
type Person struct {
ID int64
Name string
SortName string
Bio string
BirthDate *time.Time
DeathDate *time.Time
Birthplace string
Homepage string
PhotoPath string
PhotoThumbhash string
TmdbID string
ImdbID string
TvdbID string
PlexGUID string
CreatedAt time.Time
UpdatedAt time.Time
}
// ItemPerson represents a person's credit on a specific media item.
type ItemPerson struct {
Person
Kind PersonKind
Character string
SortOrder int
}
// AudiobookSeriesMembership captures a book's membership in an audiobook
// series and its optional sequence number within that series.
type AudiobookSeriesMembership struct {
Name string
Index *float64
}
// MediaItem represents a row in the media_items table.
type MediaItem struct {
ContentID string // Sonyflake ID (PK)
Type string // movie, series
Title string
SortTitle string
DefaultMetadataLanguage string
OriginalTitle string
Year int
Genres []string
ContentRating string // PG-13, TV-MA
Runtime int // minutes
Overview string
Tagline string
RatingIMDB *float64
RatingTMDB *float64
RatingRTCritic *int
RatingRTAudience *int
ImdbID string
TmdbID string
TvdbID string
PosterPath string // S3 path
PosterThumbhash string
BackdropPath string
BackdropThumbhash string
LogoPath string
MetadataS3Path string
MetadataEtag string
SeasonCount *int // series only
Studios []string
Networks []string
Countries []string
Keywords []string
OriginalLanguage string
ReleaseDate *string // ISO date, nullable
FirstAirDate *string // ISO date (series only), nullable
LastAirDate *string // ISO date (series only), nullable
AirTime *string // Series broadcast time (e.g. "20:00"), nullable
AirTimezone *string // Series broadcast timezone (IANA name, e.g. "America/New_York"), nullable
ShowStatus string // Series lifecycle: "returning", "ended", "cancelled", "in_production", or "" if unknown (series only)
People []ItemPerson
AudiobookSeries []AudiobookSeriesMembership
MatchedAt *time.Time
EpisodeMetadataIncomplete bool
EpisodeMetadataLastCheckedAt *time.Time
LastRefreshed *time.Time `json:"last_refreshed,omitempty"`
RefreshFailures int `json:"refresh_failures"`
LockedFields []int `json:"locked_fields"`
Status string `json:"status"` // pending, matched, unmatched
CreatedAt time.Time
UpdatedAt time.Time
AddedAt *time.Time // populated by browse queries (MIN(mil.first_seen_at))
}
// Season represents a row in the seasons table.
type Season struct {
ContentID string
SeriesID string
SeasonNumber int
Title string
DefaultMetadataLanguage string
Overview string
AirDate *time.Time
PosterPath string
PosterThumbhash string
MetadataS3Path string
MetadataEtag string
MetadataSource string
CreatedAt time.Time
UpdatedAt time.Time
}
// Episode represents a row in the episodes table.
type Episode struct {
ContentID string // episode Sonyflake ID (PK)
SeriesID string
SeasonID string // FK to seasons.content_id
SeasonNumber int
EpisodeNumber int
Title string
DefaultMetadataLanguage string
Overview string
AirDate *time.Time
Runtime int // minutes
RatingIMDB *float64
RatingTMDB *float64
ImdbID string
TmdbID string
TvdbID string
StillPath string
StillThumbhash string
MetadataS3Path string
MetadataEtag string
MetadataSource string
CreatedAt time.Time
UpdatedAt time.Time
}
// MediaItemRoot represents a row in the media_item_roots table.
// It records which content_id owns a given canonical root path within a library folder.
type MediaItemRoot struct {
MediaFolderID int
CanonicalRootPath string
ContentID string
FirstSeenAt time.Time
LastSeenAt time.Time
}
// MediaItemGroup represents a row in the media_item_groups table.
type MediaItemGroup struct {
MediaFolderID int
GroupKeyVersion int
ContentGroupKey string
ContentID string
FirstSeenAt time.Time
LastSeenAt time.Time
}
// MediaItemLibrary represents a row in the media_item_libraries junction table.
type MediaItemLibrary struct {
ContentID string
MediaFolderID int
FirstSeenAt time.Time
}
// EpisodeLibrary represents a row in the episode_libraries junction table.
type EpisodeLibrary struct {
EpisodeID string
MediaFolderID int
FirstSeenAt time.Time
}
// Localization field provenance values. Precedence when writing:
// manual beats provider beats ai — a provider refresh may overwrite an AI
// translation but never a manual edit, and AI never overwrites either
// (except provider, when the admin explicitly forces a re-translation).
const (
LocalizationSourceProvider = "provider"
LocalizationSourceAI = "ai"
LocalizationSourceManual = "manual"
)
type MediaItemLocalization struct {
ContentID string
Language string
Title string
SortTitle string
Overview string
Tagline string
PosterPath string
PosterThumbhash string
BackdropPath string
BackdropThumbhash string
LogoPath string
OverviewSource string // provider | ai | manual
TaglineSource string // provider | ai | manual
CreatedAt time.Time
UpdatedAt time.Time
}
type SeasonLocalization struct {
SeasonContentID string
Language string
Title string
Overview string
PosterPath string
PosterThumbhash string
OverviewSource string // provider | ai | manual
CreatedAt time.Time
UpdatedAt time.Time
}
type EpisodeLocalization struct {
EpisodeContentID string
Language string
Title string
Overview string
OverviewSource string // provider | ai | manual
CreatedAt time.Time
UpdatedAt time.Time
}