Files
silo-server/internal/contentid/contentid.go
T
13c5e0ba2f feat(catalog): deterministic cross-server content_id (#155)
* feat(catalog): deterministic cross-server content_id

Replace per-server Sonyflake content_id with a structured natural key
derived from provider IDs (movie:tmdb:…, series:tvdb:…, episode:…,
local:… fallback), so two servers holding the same title share one
anchor for artwork, watch history, progress, favorites and ratings.

- internal/contentid: derivation core, SeriesIDFromContentID transform,
  frozen precedence, SchemeVersion=1, embedded-series-anchor invariant.
- internal/metadata/service.go: deterministic id at every mint site.
- internal/catalog/history_source.go: resolve show via string transform
  for anchored episode ids; skip the episodes_pkey probe.
- migrations/sql/20260612130000: collision-safe value remap across the
  65-column reference graph + COLLATE "C", FK/trigger handling, audit
  map, working down.

Benchmarked against an exact-cardinality copy of cprod-postgres
(1.93M episodes, 775k history rows): 2.57x faster history page, 1.7x
throughput at 100 concurrent users, 2.7x cheaper per content_id probe.

* feat(catalog): re-ID untagged items to deterministic content_id at first match

Untagged libraries get a path-derived local: content_id at scan time and
only learn their provider IDs later, when the match worker confirms a
result. Previously that id was never folded back in, so untagged-then-matched
items kept a per-server local: placeholder forever and never converged across
servers (re-ID was deferred to a migration rerun).

mergeAndPersist now promotes a local: skeleton to its deterministic
provider-anchored id at the moment of first confirmed match, via a single
new gate (canonicalizeLocalContentID):

  - target id already taken  -> merge onto it (existing rebind machinery)
  - target id free           -> rename in place

The rename is a single SQL function (silo_rename_content_id); FK children
follow via ON UPDATE CASCADE added to the content_id family, so a fresh
skeleton moves a handful of rows rather than the full-table remap the bulk
migration does. The guard is one IsLocal prefix check, so tagged content and
all refreshes pay nothing, and the move is self-healing under retry.

Verified: gofmt/vet/build clean; migrate-validate passes; migration applies
on the real schema (up/down/up), FKs gain ON UPDATE CASCADE while keeping
ON DELETE; functional test confirms series PK move + series_id cascade +
provider-id sweep, and movie rename.

Follow-ups (noted in docs): recomposeSeriesChildIDs for a series that
accumulated episodes before matching; a lockstep test for the soft-ref list.

* fix(catalog): harden content_id parsing and merge per review

Address review feedback on the deterministic content_id work:

- history_source.go: gate the anchored-episode display-id transform on the
  full five-part episode shape (split_part parts 2-5 non-empty), not just the
  'episode:' prefix, so a malformed id can't transform to 'series:broken:' and
  vanish at the media_items join. Shared anchoredEpisodePredicate drives both
  the null-poisoned join key and the series-recovery expression.
- contentid.go: unexport the provider-precedence slices so no package can
  mutate the frozen SchemeVersion ordering at runtime.
- contentid.go: add parseAnchored to validate the exact per-kind arity and
  numeric season/episode suffixes; SeriesIDFromContentID and IsProviderAnchored
  now fail closed on truncated/malformed ids (e.g. "episode:tvdb:296762").
- canonicalize.go: distinguish catalog.ErrItemNotFound from transient lookup
  errors (a real error no longer masquerades as "target free"), and allow a
  matched local source to be consolidated onto the canonical row instead of
  orphaning a duplicate.

* refactor(contentid): URL-safe "-" separator in content_id

Use "-" instead of ":" to join content_id components
(movie-tmdb-228064, episode-tvdb-296762-1-5, local-<hex>). "-" is an RFC 3986
unreserved character, so a content_id is URL-safe verbatim: encodeURIComponent
is a no-op and the id is its own tidy path segment (/item/series-tvdb-296762)
with no %3A escaping. The stored value equals the URL value, so there is no
encode/decode boundary and an operator can grep the id straight out of a URL or
log. Every component is [a-z0-9]+ (or "tt"+digits), so "-" is unambiguous.

Pre-release format finalization: this branch is unmerged, so no deployed data
carries ":" ids — the migration mints the "-" form fresh and no re-migration is
needed. Still SchemeVersion 1.

- contentid.go: single `sep` constant drives construction and parsing so the two
  can never drift; all constructors/parsers and doc examples updated.
- history_source.go: split_part transform and the anchored-episode predicate use
  '-'; kept in lockstep with the package via a code comment.
- 20260612130000_deterministic_content_id.sql: derivation and season/episode
  composition emit '-'; LIKE filters match 'series-%'.
- docs/architecture/deterministic-content-id.md: format spec + rationale for the
  separator choice; this is the design doc the change is derived from.

Client-side: the web frontend treats content_id as an opaque string (no
splitting/regex), so no client changes are required; existing
encodeURIComponent call sites simply stop emitting %3A.

* docs(contentid): show why hash/bigint rejected in probe-cost table

Add Cross-server deterministic / Zero-join show transform / Human-readable
columns to the index-probe-cost comparison so the trade-off is legible at a
glance: the 128-bit hash and bigint surrogate are faster but each give up a
load-bearing property, and the structured key is the only all-checkmark row.

* docs(contentid): order probe-cost table to end on the structured key

* docs(contentid): label fenced blocks and drop stray EOF tags

Per CodeRabbit review: add 'text' language to three fenced code blocks
(MD040) and remove accidental </content></invoke> artifacts at EOF.

* fix(catalog): remap array-valued content_id soft references in deterministic id migration

The value-remap migration (20260612130000) enumerates the reference graph by FK
plus a scalar name+type sweep (text/varchar/bpchar). That misses
trending_discover_snapshots.content_ids: it is text[] (excluded by the type
filter), named content_ids not content_id (excluded by the name list), and
cannot carry an FK — so the bulk remap left those arrays holding stale Sonyflake
ids that resolve to nothing until the snapshot regenerates. A counterexample to
the migration's "self-protecting, cannot orphan" invariant.

Remap the array element-wise in both directions (Up old->new, Down new->old),
preserving order and leaving collision/unmatched elements untouched; a WHERE
EXISTS guard skips empty/unaffected arrays so array_agg never collapses the NOT
NULL column to NULL. Mirror the gap in silo_rename_content_id (20260614120000)
with array_replace for the single-value runtime rename so the two stay in
lockstep.

Verified on PG18: mixed/collision/empty arrays remap correctly and round-trip
clean; runtime array_replace preserves order.

Surfaced reviewing #155. The jellycompat restart-decode regression and the
atomicity-wording nit are posted as review comments, not addressed here.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(jellycompat): pack content_id into compat UUID reversibly so item ids survive restarts

Addresses the restart-decode regression raised in review of #155. With
content_id now a structured string instead of a numeric Sonyflake,
EncodeStringID sent every item/season id down the one-way SHA1 path, making
decode depend on an in-memory reverse map. That map is cold after a process
restart (the codec is a process-lifetime singleton), so a client presenting a
previously-issued item UUID — resume-from-home, deep link, detail page, image,
userdata — got "unknown compat id" until the item was re-listed.

Make the encoding reversible instead of stateful:

- internal/contentid: add Pack/Unpack, a bit-packed, fixed-budget (<=15 byte)
  binary form of a structured or local content_id. digitCount preserves
  provider-id leading zeros (e.g. imdb tt0944947); structured forms are
  self-delimiting; the local form fills the budget exactly. Provider ids that
  overflow uint64 return ok=false.
- Shrink ForLocal to a 112-bit (sha256(path)[:14]) hash so a local id packs
  losslessly into the 15-byte UUID payload. 112 bits is far beyond any single
  server's local-item count. No other code assumed the old width.
- internal/jellycompat: EncodeStringID packs item/season content_ids into the
  UUID (byte 0 = kind, bytes 1..15 = packed, non-zero tag distinguishes it from
  the numeric encoding); DecodeStringID unpacks first and re-packs to confirm,
  so an opaque id whose bytes merely parse is rejected and falls through to the
  map. Numeric ids and arbitrary names (genres, studios) are unchanged.

Net: item/season ids decode by pure computation — stable across restarts and
across instances — with no lookup table. Only the rare unpackable content_id and
non-content names still use the in-memory map.

TDD: round-trip property tests in contentid (all kinds, leading zeros, reject
cases) and a cross-instance decode test in jellycompat that fails on the old
hash+map path. Full contentid + jellycompat suites green; production code
golangci-clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(migrate): make the migration run timeout configurable (SILO_MIGRATE_TIMEOUT)

The boot-path migration runner hardcoded a 5-minute context timeout. The
deterministic-content-id value-remap (20260612130000) does a full-table COLLATE
rewrite + 65-column remap that needs ~20 min on a real dataset (615k items /
2M episodes), so it was cancelled at 5 min. Worse, Postgres keeps the orphaned
backend running (holding AccessExclusive locks) until it notices the dead client
at a statement boundary, while the goose session advisory lock releases on
disconnect — so each 5-min boot retry piled a new attempt behind the previous
one's locks. The migration never applied; the server boot-looped.

Make the timeout configurable via SILO_MIGRATE_TIMEOUT (a Go duration like
"60m"); 0 or negative disables the deadline for a one-off heavy migration. Default
stays 5m. All three entry points (migrate-status, --migrate-only, boot) honor it.

Required for the deterministic-content-id migration to apply on any real-sized
database, not just dev — the 5m cap made the PR undeployable at scale.

Follow-up (not here): on cancellation the runner should actively terminate its
backend so a future timeout cannot orphan a lock-holding statement.

TDD: MigrationTimeout parsing (default/override/zero/invalid) + MigrationContext
deadline behavior.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(contentid): require exact length for local ids in Unpack

Tighten the tagLocal branch of Unpack from `len(body) < localHashLen` to an
equality check. The local form fills the compat-UUID payload exactly (no
padding), so a body of any other length is non-canonical; matching it exactly
keeps Unpack a strict fail-closed inverse of Pack for the fixed-length branch,
which decodes client-supplied UUIDs.

Not applied to the structured branch (a review suggestion proposed the same
change there): structured ids are self-delimiting and the compat layer pads them
with trailing zeros to fill the 15-byte UUID payload, so ignoring trailing bytes
is intentional and documented. Rejecting them would make every structured id
fail to decode — the jellycompat cross-instance test guards against that.

Not a live bug today (the only caller passes u[1:] from a 16-byte UUID, so body
is always exactly localHashLen, and idcodec re-packs to verify), but it is the
correct contract and zero-risk. Adds a regression test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Quick <31828688+Quick104@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 15:34:13 -04:00

334 lines
13 KiB
Go

// Package contentid derives deterministic, cross-server stable content IDs for
// logical media items (movies, series, seasons, episodes).
//
// Historically every logical item was minted a Sonyflake ID — a locally
// generated, time-ordered number that differs per server for the same title.
// content_id is the anchor that artwork, metadata, watch history, progress,
// favorites, ratings, collections and credits hang off, so a per-server ID
// means two servers holding the same movie cannot share any of that state.
//
// This package replaces that with a structured natural key derived from the
// provider IDs already embedded in the library:
//
// movie-<provider>-<id> e.g. movie-tmdb-228064
// series-<provider>-<id> e.g. series-tvdb-296762
// season-<provider>-<seriesId>-<seasonNo> e.g. season-tvdb-296762-1
// episode-<provider>-<seriesId>-<s>-<e> e.g. episode-tvdb-296762-1-5
// local-<hex112> unmatched / local fallback
//
// Provider IDs are unique by construction, so the value is collision-free with
// no hashing. The leading entity-type token domain-separates the namespaces so
// a movie and an episode can never alias on a coincidental number.
//
// The component separator is "-" (an RFC 3986 unreserved character) rather than
// ":". Every component is [a-z0-9]+ (or "tt"+digits for IMDb), so "-" is an
// unambiguous delimiter, and because it needs no percent-encoding the id is its
// own tidy URL path segment — /item/series-tvdb-296762, identical to what is
// stored in the database (no encode/decode, greppable as-is). See sep.
//
// Load-bearing format invariant: an episode/season ID embeds its series anchor,
// so the series content_id is always a pure string transform of the
// episode/season ID (see SeriesIDFromContentID). The watch-history query relies
// on this to resolve a show without an episodes table lookup. Never break it.
//
// See docs/architecture/deterministic-content-id.md for the full rationale.
package contentid
import (
"crypto/sha256"
"encoding/hex"
"regexp"
"strconv"
"strings"
"golang.org/x/text/unicode/norm"
)
// SchemeVersion freezes the provider precedence and the exact string format.
// Changing either re-IDs items, so it is a deliberate, rare event that forces a
// full remap (see the migration machinery). It is intentionally a code constant,
// not a per-row column: content_id has no lasting mixed-version population
// because any scheme change normalizes every row at once.
const SchemeVersion = 1
// sep joins the components of a content_id. It is an RFC 3986 unreserved
// character, so a content_id is URL-safe verbatim (encodeURIComponent is a
// no-op) and survives as a clean path segment with no percent-encoding. It is
// part of the frozen format (SchemeVersion) and mirrored by the literal
// delimiters in 20260612130000_deterministic_content_id.sql and the split_part
// transform in internal/catalog/history_source.go — changing it here without
// changing those re-IDs items inconsistently. No component ever contains it
// (all are [a-z0-9]+ / "tt"+digits), so it is an unambiguous delimiter.
const sep = "-"
// Entity-type tokens. These domain-separate the provider-number namespaces.
const (
kindMovie = "movie"
kindSeries = "series"
kindSeason = "season"
kindEpisode = "episode"
kindLocal = "local"
)
// Canonical provider tokens.
const (
ProviderTMDB = "tmdb"
ProviderIMDB = "imdb"
ProviderTVDB = "tvdb"
)
// movieProviderPrecedence and seriesProviderPrecedence freeze, per the scheme
// version, which provider anchors a logical item when more than one tag is
// present. Two servers that both see the tags therefore pick the same anchor.
// They are unexported and never returned by reference: the precedence is part of
// SchemeVersion and is mirrored by the hard-coded order in
// 20260612130000_deterministic_content_id.sql, so a stray runtime mutation would
// silently mint ids that no longer match the migration.
//
// - Movies prefer TMDB (the richest movie source).
// - Series/Season/Episode prefer TVDB (the traditional episode-canonical
// source).
var (
movieProviderPrecedence = []string{ProviderTMDB, ProviderIMDB, ProviderTVDB}
seriesProviderPrecedence = []string{ProviderTVDB, ProviderTMDB, ProviderIMDB}
)
// ProviderIDs carries the raw provider identifiers for an item as found on the
// denormalized columns or parsed from folder/file tags. Empty fields are
// ignored.
type ProviderIDs struct {
Tmdb string
Imdb string
Tvdb string
}
func (p ProviderIDs) get(provider string) string {
switch provider {
case ProviderTMDB:
return p.Tmdb
case ProviderIMDB:
return p.Imdb
case ProviderTVDB:
return p.Tvdb
}
return ""
}
// numericIDPattern validates a tmdb/tvdb id: a non-empty run of digits.
var numericIDPattern = regexp.MustCompile(`^[0-9]+$`)
// imdbIDPattern validates an imdb id: the literal "tt" followed by digits.
var imdbIDPattern = regexp.MustCompile(`^tt[0-9]+$`)
// normalizeProviderID normalizes and validates a provider id for use in a
// content_id. Returns the normalized id and whether it is usable. IMDb ids are
// lowercased and must retain their "tt" prefix; tmdb/tvdb ids must be a bare
// run of digits with no leading-zero rewriting (the provider's canonical form).
func normalizeProviderID(provider, raw string) (string, bool) {
id := strings.TrimSpace(raw)
if id == "" {
return "", false
}
switch provider {
case ProviderIMDB:
id = strings.ToLower(id)
if !imdbIDPattern.MatchString(id) {
return "", false
}
return id, true
case ProviderTMDB, ProviderTVDB:
if !numericIDPattern.MatchString(id) {
return "", false
}
return id, true
default:
return "", false
}
}
// anchor picks the canonical (provider, id) for an item under the given
// precedence, returning ok=false when no usable provider id is present.
func anchor(ids ProviderIDs, precedence []string) (provider, id string, ok bool) {
for _, p := range precedence {
if norm, valid := normalizeProviderID(p, ids.get(p)); valid {
return p, norm, true
}
}
return "", "", false
}
// ForMovie returns the deterministic content_id for a movie, or ("", false) if
// no provider anchor is present (the caller should fall back to ForLocal).
func ForMovie(ids ProviderIDs) (string, bool) {
provider, id, ok := anchor(ids, movieProviderPrecedence)
if !ok {
return "", false
}
return kindMovie + sep + provider + sep + id, true
}
// ForSeries returns the deterministic content_id for a series, or ("", false)
// if no provider anchor is present.
func ForSeries(ids ProviderIDs) (string, bool) {
provider, id, ok := anchor(ids, seriesProviderPrecedence)
if !ok {
return "", false
}
return kindSeries + sep + provider + sep + id, true
}
// seriesBody returns the "<provider>-<id>" portion of a provider-anchored series
// content_id, e.g. "tvdb-296762" for "series-tvdb-296762". Returns ok=false for
// any id that is not a provider-anchored series (local series, legacy Sonyflake
// ids, etc.) so seasons/episodes correctly fall back instead of composing a
// malformed key.
func seriesBody(seriesContentID string) (string, bool) {
rest, ok := strings.CutPrefix(strings.TrimSpace(seriesContentID), kindSeries+sep)
if !ok || rest == "" {
return "", false
}
// Guard the format invariant: exactly "<provider>-<id>" with a known,
// validated provider and id. Anything else cannot anchor children.
parts := strings.Split(rest, sep)
if len(parts) != 2 {
return "", false
}
if _, valid := normalizeProviderID(parts[0], parts[1]); !valid {
return "", false
}
return rest, true
}
// ForSeason composes a season content_id from its parent series content_id and
// the season number. It relies on the format invariant: the season key embeds
// the series anchor. Returns ("", false) when the series is not provider-
// anchored, so the caller falls back to a local/legacy id.
func ForSeason(seriesContentID string, seasonNumber int) (string, bool) {
body, ok := seriesBody(seriesContentID)
if !ok {
return "", false
}
return kindSeason + sep + body + sep + strconv.Itoa(seasonNumber), true
}
// ForEpisode composes an episode content_id from its parent series content_id
// and the season/episode numbers. Episodes compose from the series anchor plus
// numbers (both universally present in filenames), not their own provider
// episode IDs which are often missing. Returns ("", false) when the series is
// not provider-anchored.
func ForEpisode(seriesContentID string, seasonNumber, episodeNumber int) (string, bool) {
body, ok := seriesBody(seriesContentID)
if !ok {
return "", false
}
return kindEpisode + sep + body + sep +
strconv.Itoa(seasonNumber) + sep + strconv.Itoa(episodeNumber), true
}
// ForLocal returns a content_id in the disjoint "local-" namespace for an item
// with no provider anchor, derived from a normalized path so the same library
// on the same server stays stable across rescans. It can never collide with a
// provider-derived id. An item's local id changes if it is later matched to a
// provider — rare, and such items seldom carry watch state.
//
// The path is NFC-normalized and trimmed before hashing; the hash is the first
// localHashLen bytes (112 bits) of SHA-256, hex-encoded. That width is chosen so
// the id packs losslessly into a Jellyfin-compat UUID (see Pack); 112 bits is
// far beyond any single server's local-item count. Cross-server stability for
// local items is best-effort only (paths differ between servers).
func ForLocal(path string) string {
normalized := norm.NFC.String(strings.TrimSpace(path))
sum := sha256.Sum256([]byte(normalized))
return kindLocal + sep + hex.EncodeToString(sum[:localHashLen])
}
// SeriesIDFromContentID derives the owning series content_id from an episode or
// season content_id by pure string transform — no catalog lookup — per the
// format invariant. For a movie or series id it returns the id unchanged (a
// movie is its own display item; a series id is already a series id). For a
// local episode/season (no embedded series anchor) or any unrecognized id it
// returns ok=false, signaling the caller to fall back to a resolved lookup.
//
// episode-tvdb-296762-1-5 -> series-tvdb-296762
// season-tvdb-296762-1 -> series-tvdb-296762
// movie-tmdb-228064 -> movie-tmdb-228064 (unchanged)
// series-tvdb-296762 -> series-tvdb-296762 (unchanged)
func SeriesIDFromContentID(contentID string) (string, bool) {
id := strings.TrimSpace(contentID)
if strings.HasPrefix(id, kindMovie+sep) || strings.HasPrefix(id, kindSeries+sep) {
// A movie is its own display item; a series id is already a series id.
// Still require the anchor to be well-formed so a truncated "series-"
// cannot pass through unchanged.
if !IsProviderAnchored(id) {
return "", false
}
return id, true
}
// episode/season carry an embedded "<provider>-<seriesId>" anchor; recover it
// by pure string transform. parseAnchored enforces the full per-kind arity, so
// a truncated "episode-tvdb-296762" (missing season/episode) fails closed
// rather than aliasing onto a series.
if provider, seriesID, ok := parseAnchored(id); ok {
return kindSeries + sep + provider + sep + seriesID, true
}
// local ids and legacy Sonyflake ids have no embedded anchor.
return "", false
}
// IsProviderAnchored reports whether the content_id is a fully-formed
// provider-derived key (movie/series/season/episode), as opposed to a local id,
// a legacy Sonyflake id, or a truncated/malformed anchor. Used by migration and
// diagnostics to tell remappable items apart.
func IsProviderAnchored(contentID string) bool {
_, _, ok := parseAnchored(strings.TrimSpace(contentID))
return ok
}
// parseAnchored validates a provider-anchored content_id against the exact
// per-kind arity and component shape, returning the canonical (provider,
// seriesId-or-id) anchor when it is well-formed:
//
// movie-<p>-<id> -> (p, id)
// series-<p>-<id> -> (p, id)
// season-<p>-<sid>-<n> -> (p, sid) n must be a base-10 integer
// episode-<p>-<sid>-<s>-<e> -> (p, sid) s,e must be base-10 integers
//
// It fails closed for any other shape (wrong part count, non-numeric
// season/episode, unknown/invalid provider id, local or legacy ids).
func parseAnchored(id string) (provider, seriesID string, ok bool) {
kind, rest, found := strings.Cut(id, sep)
if !found {
return "", "", false
}
parts := strings.Split(rest, sep)
var wantParts int
switch kind {
case kindMovie, kindSeries:
wantParts = 2 // <provider>-<id>
case kindSeason:
wantParts = 3 // <provider>-<seriesId>-<seasonNo>
case kindEpisode:
wantParts = 4 // <provider>-<seriesId>-<seasonNo>-<episodeNo>
default:
return "", "", false
}
if len(parts) != wantParts {
return "", "", false
}
if _, valid := normalizeProviderID(parts[0], parts[1]); !valid {
return "", "", false
}
// The trailing season/episode numbers must be base-10 integers.
for _, n := range parts[2:] {
if !numericIDPattern.MatchString(n) {
return "", "", false
}
}
return parts[0], parts[1], true
}
// IsLocal reports whether the content_id is in the local fallback namespace.
func IsLocal(contentID string) bool {
return strings.HasPrefix(strings.TrimSpace(contentID), kindLocal+sep)
}