* feat(catalog): deterministic cross-server content_id Replace per-server Sonyflake content_id with a structured natural key derived from provider IDs (movie:tmdb:…, series:tvdb:…, episode:…, local:… fallback), so two servers holding the same title share one anchor for artwork, watch history, progress, favorites and ratings. - internal/contentid: derivation core, SeriesIDFromContentID transform, frozen precedence, SchemeVersion=1, embedded-series-anchor invariant. - internal/metadata/service.go: deterministic id at every mint site. - internal/catalog/history_source.go: resolve show via string transform for anchored episode ids; skip the episodes_pkey probe. - migrations/sql/20260612130000: collision-safe value remap across the 65-column reference graph + COLLATE "C", FK/trigger handling, audit map, working down. Benchmarked against an exact-cardinality copy of cprod-postgres (1.93M episodes, 775k history rows): 2.57x faster history page, 1.7x throughput at 100 concurrent users, 2.7x cheaper per content_id probe. * feat(catalog): re-ID untagged items to deterministic content_id at first match Untagged libraries get a path-derived local: content_id at scan time and only learn their provider IDs later, when the match worker confirms a result. Previously that id was never folded back in, so untagged-then-matched items kept a per-server local: placeholder forever and never converged across servers (re-ID was deferred to a migration rerun). mergeAndPersist now promotes a local: skeleton to its deterministic provider-anchored id at the moment of first confirmed match, via a single new gate (canonicalizeLocalContentID): - target id already taken -> merge onto it (existing rebind machinery) - target id free -> rename in place The rename is a single SQL function (silo_rename_content_id); FK children follow via ON UPDATE CASCADE added to the content_id family, so a fresh skeleton moves a handful of rows rather than the full-table remap the bulk migration does. The guard is one IsLocal prefix check, so tagged content and all refreshes pay nothing, and the move is self-healing under retry. Verified: gofmt/vet/build clean; migrate-validate passes; migration applies on the real schema (up/down/up), FKs gain ON UPDATE CASCADE while keeping ON DELETE; functional test confirms series PK move + series_id cascade + provider-id sweep, and movie rename. Follow-ups (noted in docs): recomposeSeriesChildIDs for a series that accumulated episodes before matching; a lockstep test for the soft-ref list. * fix(catalog): harden content_id parsing and merge per review Address review feedback on the deterministic content_id work: - history_source.go: gate the anchored-episode display-id transform on the full five-part episode shape (split_part parts 2-5 non-empty), not just the 'episode:' prefix, so a malformed id can't transform to 'series:broken:' and vanish at the media_items join. Shared anchoredEpisodePredicate drives both the null-poisoned join key and the series-recovery expression. - contentid.go: unexport the provider-precedence slices so no package can mutate the frozen SchemeVersion ordering at runtime. - contentid.go: add parseAnchored to validate the exact per-kind arity and numeric season/episode suffixes; SeriesIDFromContentID and IsProviderAnchored now fail closed on truncated/malformed ids (e.g. "episode:tvdb:296762"). - canonicalize.go: distinguish catalog.ErrItemNotFound from transient lookup errors (a real error no longer masquerades as "target free"), and allow a matched local source to be consolidated onto the canonical row instead of orphaning a duplicate. * refactor(contentid): URL-safe "-" separator in content_id Use "-" instead of ":" to join content_id components (movie-tmdb-228064, episode-tvdb-296762-1-5, local-<hex>). "-" is an RFC 3986 unreserved character, so a content_id is URL-safe verbatim: encodeURIComponent is a no-op and the id is its own tidy path segment (/item/series-tvdb-296762) with no %3A escaping. The stored value equals the URL value, so there is no encode/decode boundary and an operator can grep the id straight out of a URL or log. Every component is [a-z0-9]+ (or "tt"+digits), so "-" is unambiguous. Pre-release format finalization: this branch is unmerged, so no deployed data carries ":" ids — the migration mints the "-" form fresh and no re-migration is needed. Still SchemeVersion 1. - contentid.go: single `sep` constant drives construction and parsing so the two can never drift; all constructors/parsers and doc examples updated. - history_source.go: split_part transform and the anchored-episode predicate use '-'; kept in lockstep with the package via a code comment. - 20260612130000_deterministic_content_id.sql: derivation and season/episode composition emit '-'; LIKE filters match 'series-%'. - docs/architecture/deterministic-content-id.md: format spec + rationale for the separator choice; this is the design doc the change is derived from. Client-side: the web frontend treats content_id as an opaque string (no splitting/regex), so no client changes are required; existing encodeURIComponent call sites simply stop emitting %3A. * docs(contentid): show why hash/bigint rejected in probe-cost table Add Cross-server deterministic / Zero-join show transform / Human-readable columns to the index-probe-cost comparison so the trade-off is legible at a glance: the 128-bit hash and bigint surrogate are faster but each give up a load-bearing property, and the structured key is the only all-checkmark row. * docs(contentid): order probe-cost table to end on the structured key * docs(contentid): label fenced blocks and drop stray EOF tags Per CodeRabbit review: add 'text' language to three fenced code blocks (MD040) and remove accidental </content></invoke> artifacts at EOF. * fix(catalog): remap array-valued content_id soft references in deterministic id migration The value-remap migration (20260612130000) enumerates the reference graph by FK plus a scalar name+type sweep (text/varchar/bpchar). That misses trending_discover_snapshots.content_ids: it is text[] (excluded by the type filter), named content_ids not content_id (excluded by the name list), and cannot carry an FK — so the bulk remap left those arrays holding stale Sonyflake ids that resolve to nothing until the snapshot regenerates. A counterexample to the migration's "self-protecting, cannot orphan" invariant. Remap the array element-wise in both directions (Up old->new, Down new->old), preserving order and leaving collision/unmatched elements untouched; a WHERE EXISTS guard skips empty/unaffected arrays so array_agg never collapses the NOT NULL column to NULL. Mirror the gap in silo_rename_content_id (20260614120000) with array_replace for the single-value runtime rename so the two stay in lockstep. Verified on PG18: mixed/collision/empty arrays remap correctly and round-trip clean; runtime array_replace preserves order. Surfaced reviewing #155. The jellycompat restart-decode regression and the atomicity-wording nit are posted as review comments, not addressed here. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(jellycompat): pack content_id into compat UUID reversibly so item ids survive restarts Addresses the restart-decode regression raised in review of #155. With content_id now a structured string instead of a numeric Sonyflake, EncodeStringID sent every item/season id down the one-way SHA1 path, making decode depend on an in-memory reverse map. That map is cold after a process restart (the codec is a process-lifetime singleton), so a client presenting a previously-issued item UUID — resume-from-home, deep link, detail page, image, userdata — got "unknown compat id" until the item was re-listed. Make the encoding reversible instead of stateful: - internal/contentid: add Pack/Unpack, a bit-packed, fixed-budget (<=15 byte) binary form of a structured or local content_id. digitCount preserves provider-id leading zeros (e.g. imdb tt0944947); structured forms are self-delimiting; the local form fills the budget exactly. Provider ids that overflow uint64 return ok=false. - Shrink ForLocal to a 112-bit (sha256(path)[:14]) hash so a local id packs losslessly into the 15-byte UUID payload. 112 bits is far beyond any single server's local-item count. No other code assumed the old width. - internal/jellycompat: EncodeStringID packs item/season content_ids into the UUID (byte 0 = kind, bytes 1..15 = packed, non-zero tag distinguishes it from the numeric encoding); DecodeStringID unpacks first and re-packs to confirm, so an opaque id whose bytes merely parse is rejected and falls through to the map. Numeric ids and arbitrary names (genres, studios) are unchanged. Net: item/season ids decode by pure computation — stable across restarts and across instances — with no lookup table. Only the rare unpackable content_id and non-content names still use the in-memory map. TDD: round-trip property tests in contentid (all kinds, leading zeros, reject cases) and a cross-instance decode test in jellycompat that fails on the old hash+map path. Full contentid + jellycompat suites green; production code golangci-clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(migrate): make the migration run timeout configurable (SILO_MIGRATE_TIMEOUT) The boot-path migration runner hardcoded a 5-minute context timeout. The deterministic-content-id value-remap (20260612130000) does a full-table COLLATE rewrite + 65-column remap that needs ~20 min on a real dataset (615k items / 2M episodes), so it was cancelled at 5 min. Worse, Postgres keeps the orphaned backend running (holding AccessExclusive locks) until it notices the dead client at a statement boundary, while the goose session advisory lock releases on disconnect — so each 5-min boot retry piled a new attempt behind the previous one's locks. The migration never applied; the server boot-looped. Make the timeout configurable via SILO_MIGRATE_TIMEOUT (a Go duration like "60m"); 0 or negative disables the deadline for a one-off heavy migration. Default stays 5m. All three entry points (migrate-status, --migrate-only, boot) honor it. Required for the deterministic-content-id migration to apply on any real-sized database, not just dev — the 5m cap made the PR undeployable at scale. Follow-up (not here): on cancellation the runner should actively terminate its backend so a future timeout cannot orphan a lock-holding statement. TDD: MigrationTimeout parsing (default/override/zero/invalid) + MigrationContext deadline behavior. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(contentid): require exact length for local ids in Unpack Tighten the tagLocal branch of Unpack from `len(body) < localHashLen` to an equality check. The local form fills the compat-UUID payload exactly (no padding), so a body of any other length is non-canonical; matching it exactly keeps Unpack a strict fail-closed inverse of Pack for the fixed-length branch, which decodes client-supplied UUIDs. Not applied to the structured branch (a review suggestion proposed the same change there): structured ids are self-delimiting and the compat layer pads them with trailing zeros to fill the 15-byte UUID payload, so ignoring trailing bytes is intentional and documented. Rejecting them would make every structured id fail to decode — the jellycompat cross-instance test guards against that. Not a live bug today (the only caller passes u[1:] from a 16-byte UUID, so body is always exactly localHashLen, and idcodec re-packs to verify), but it is the correct contract and zero-risk. Adds a regression test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Quick <31828688+Quick104@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
242 lines
7.9 KiB
Go
242 lines
7.9 KiB
Go
package jellycompat
|
|
|
|
import (
|
|
"encoding/binary"
|
|
"fmt"
|
|
"strconv"
|
|
"sync"
|
|
|
|
"github.com/google/uuid"
|
|
|
|
"github.com/Silo-Server/silo-server/internal/contentid"
|
|
)
|
|
|
|
// EncodedIDType distinguishes packed compat UUIDs.
|
|
type EncodedIDType byte
|
|
|
|
const (
|
|
EncodedIDLibrary EncodedIDType = 1
|
|
EncodedIDItem EncodedIDType = 2
|
|
EncodedIDMediaSource EncodedIDType = 3
|
|
EncodedIDSeason EncodedIDType = 4
|
|
EncodedIDPlaySession EncodedIDType = 5
|
|
EncodedIDGenre EncodedIDType = 6
|
|
EncodedIDStudio EncodedIDType = 7
|
|
EncodedIDPerson EncodedIDType = 8
|
|
EncodedIDImageProxy EncodedIDType = 9
|
|
EncodedIDCollection EncodedIDType = 10
|
|
)
|
|
|
|
var (
|
|
pseudoUserNamespace = uuid.MustParse("3dfcc388-bf95-5572-bc16-7f1a375992dd")
|
|
stringIDNamespaces = map[EncodedIDType]uuid.UUID{
|
|
EncodedIDItem: uuid.MustParse("0b6716ca-1f61-5987-b17b-f592f04fd6b3"),
|
|
EncodedIDSeason: uuid.MustParse("29831b2b-dad5-5a85-b506-4d1fb2da01ed"),
|
|
EncodedIDPlaySession: uuid.MustParse("75a69ca8-f95f-5e9d-ac0a-d34a37b93eb4"),
|
|
EncodedIDGenre: uuid.MustParse("c0cbb8ea-8331-52c0-b160-15e7cf899fb0"),
|
|
EncodedIDStudio: uuid.MustParse("23712982-b769-592d-9360-b4d3f39654db"),
|
|
EncodedIDPerson: uuid.MustParse("a4e7c1d6-3b8f-5a2e-9c01-7d6f4e8b2a13"),
|
|
EncodedIDCollection: uuid.MustParse("7f3c2a91-5b64-5c1d-8e07-9a2f4d6b1c35"),
|
|
}
|
|
)
|
|
|
|
// DecodedID is a packed compat UUID decoded back to its type and value.
|
|
type DecodedID struct {
|
|
Type EncodedIDType
|
|
Value uint64
|
|
}
|
|
|
|
// ResourceIDCodec encodes numeric IDs directly and keeps reversible mappings
|
|
// for opaque string content IDs used by media items and seasons.
|
|
type ResourceIDCodec struct {
|
|
mu sync.RWMutex
|
|
reverse map[string]registeredID
|
|
mediaSourceOwners map[int64]string
|
|
}
|
|
|
|
type registeredID struct {
|
|
kind EncodedIDType
|
|
value string
|
|
}
|
|
|
|
// PseudoUserID deterministically derives the Jellyfin pseudo-user UUID.
|
|
func PseudoUserID(userID int, profileID string) uuid.UUID {
|
|
return uuid.NewSHA1(pseudoUserNamespace, fmt.Appendf(nil, "%d:%s", userID, profileID))
|
|
}
|
|
|
|
// NewResourceIDCodec creates a new route ID codec.
|
|
func NewResourceIDCodec() *ResourceIDCodec {
|
|
return &ResourceIDCodec{
|
|
reverse: make(map[string]registeredID),
|
|
mediaSourceOwners: make(map[int64]string),
|
|
}
|
|
}
|
|
|
|
// EncodeNumericID packs a numeric Silo identifier into a UUID.
|
|
func EncodeNumericID(kind EncodedIDType, value uint64) uuid.UUID {
|
|
var raw [16]byte
|
|
raw[0] = byte(kind)
|
|
binary.BigEndian.PutUint64(raw[8:], value)
|
|
return uuid.UUID(raw)
|
|
}
|
|
|
|
// EncodeStringID encodes a Silo identifier into a Jellyfin UUID string.
|
|
//
|
|
// Content-id kinds (item, season) whose value is a structured or local
|
|
// content_id are packed into the UUID reversibly (see contentid.Pack), so they
|
|
// decode with no lookup table and survive a server restart. Numeric ids keep
|
|
// their stateless numeric packing. Everything else — arbitrary names such as
|
|
// genres and studios, and the rare content_id too large to pack — falls back to
|
|
// a hashed UUID recorded in the reverse map.
|
|
func (c *ResourceIDCodec) EncodeStringID(kind EncodedIDType, value string) string {
|
|
if numeric, err := strconv.ParseUint(value, 10, 64); err == nil {
|
|
return EncodeNumericID(kind, numeric).String()
|
|
}
|
|
|
|
if isContentIDKind(kind) {
|
|
if packed, ok := packContentIDUUID(kind, value); ok {
|
|
return packed.String()
|
|
}
|
|
}
|
|
|
|
namespace, ok := stringIDNamespaces[kind]
|
|
if !ok {
|
|
namespace = uuid.NameSpaceURL
|
|
}
|
|
encoded := uuid.NewSHA1(namespace, []byte(value))
|
|
|
|
c.mu.Lock()
|
|
c.reverse[encoded.String()] = registeredID{kind: kind, value: value}
|
|
c.mu.Unlock()
|
|
|
|
return encoded.String()
|
|
}
|
|
|
|
// EncodeIntID encodes a native integer ID into a Jellyfin UUID string.
|
|
func (c *ResourceIDCodec) EncodeIntID(kind EncodedIDType, value int64) string {
|
|
return EncodeNumericID(kind, uint64(value)).String()
|
|
}
|
|
|
|
// DecodeStringID decodes a compat UUID back to the original native string ID.
|
|
//
|
|
// A reversibly packed content_id is decoded first (and re-packed to confirm the
|
|
// UUID genuinely came from the packer, so an opaque id whose bytes merely happen
|
|
// to parse is rejected), then the stateless numeric encoding, then the reverse
|
|
// map for hashed ids.
|
|
func (c *ResourceIDCodec) DecodeStringID(kind EncodedIDType, raw string) (string, error) {
|
|
if isContentIDKind(kind) {
|
|
if id, ok := unpackContentIDUUID(kind, raw); ok {
|
|
return id, nil
|
|
}
|
|
}
|
|
|
|
if decoded, err := DecodeID(raw); err == nil && decoded.Type == kind {
|
|
return strconv.FormatUint(decoded.Value, 10), nil
|
|
}
|
|
|
|
c.mu.RLock()
|
|
registered, ok := c.reverse[raw]
|
|
c.mu.RUnlock()
|
|
if !ok || registered.kind != kind {
|
|
return "", fmt.Errorf("unknown compat id %q", raw)
|
|
}
|
|
return registered.value, nil
|
|
}
|
|
|
|
// isContentIDKind reports whether a compat id kind carries a Silo content_id (as
|
|
// opposed to an arbitrary name like a genre or studio). Only these kinds use the
|
|
// reversible content_id packing; everything else keeps the opaque hash + map.
|
|
func isContentIDKind(kind EncodedIDType) bool {
|
|
return kind == EncodedIDItem || kind == EncodedIDSeason
|
|
}
|
|
|
|
// packContentIDUUID packs a structured or local content_id into a compat UUID:
|
|
// byte 0 is the kind and bytes 1..15 hold contentid.Pack output, zero-padded.
|
|
// The pack format tag at byte 1 is always non-zero, which distinguishes a packed
|
|
// UUID from the numeric encoding (whose byte 1 is zero). ok=false when the id
|
|
// does not pack within the 15-byte payload.
|
|
func packContentIDUUID(kind EncodedIDType, contentID string) (uuid.UUID, bool) {
|
|
payload, ok := contentid.Pack(contentID)
|
|
if !ok || len(payload) > 15 {
|
|
return uuid.UUID{}, false
|
|
}
|
|
var u uuid.UUID
|
|
u[0] = byte(kind)
|
|
copy(u[1:], payload)
|
|
return u, true
|
|
}
|
|
|
|
// unpackContentIDUUID reverses packContentIDUUID. It re-packs the decoded id and
|
|
// compares, so a UUID that was not produced by the packer (e.g. an opaque SHA1
|
|
// id whose bytes happen to parse) is rejected and the caller falls through to
|
|
// the reverse map.
|
|
func unpackContentIDUUID(kind EncodedIDType, raw string) (string, bool) {
|
|
u, err := uuid.Parse(raw)
|
|
if err != nil || u[0] != byte(kind) || u[1] == 0 {
|
|
return "", false
|
|
}
|
|
id, ok := contentid.Unpack(u[1:])
|
|
if !ok {
|
|
return "", false
|
|
}
|
|
if check, ok := packContentIDUUID(kind, id); !ok || check != u {
|
|
return "", false
|
|
}
|
|
return id, true
|
|
}
|
|
|
|
// DecodeIntID decodes a compat UUID back to a native integer ID.
|
|
func (c *ResourceIDCodec) DecodeIntID(kind EncodedIDType, raw string) (int64, error) {
|
|
decoded, err := DecodeID(raw)
|
|
if err != nil {
|
|
return 0, err
|
|
}
|
|
if decoded.Type != kind {
|
|
return 0, fmt.Errorf("unexpected compat id type %d", decoded.Type)
|
|
}
|
|
return int64(decoded.Value), nil
|
|
}
|
|
|
|
// RegisterMediaSourceOwner records which content item owns a media-source/file ID.
|
|
func (c *ResourceIDCodec) RegisterMediaSourceOwner(fileID int64, contentID string) {
|
|
c.mu.Lock()
|
|
c.mediaSourceOwners[fileID] = contentID
|
|
c.mu.Unlock()
|
|
}
|
|
|
|
// LookupMediaSourceOwner resolves a media-source/file ID back to its content item.
|
|
func (c *ResourceIDCodec) LookupMediaSourceOwner(fileID int64) (string, bool) {
|
|
c.mu.RLock()
|
|
contentID, ok := c.mediaSourceOwners[fileID]
|
|
c.mu.RUnlock()
|
|
return contentID, ok
|
|
}
|
|
|
|
// mediaSourceIDsEqual reports whether two media-source IDs refer to the same
|
|
// source, tolerating UUID format differences. Silo exposes the canonical
|
|
// dashed compat UUID (e.g. "03000000-0000-0000-0000-00000019e8c2"), but some
|
|
// Jellyfin clients (e.g. Wholphin) echo it back in the compact 32-char hex
|
|
// form ("0300000000000000000000000019e8c2"). Both parse to the same UUID, so
|
|
// matching must compare the parsed values rather than the raw strings.
|
|
func mediaSourceIDsEqual(a, b string) bool {
|
|
if a == b {
|
|
return true
|
|
}
|
|
ua, errA := uuid.Parse(a)
|
|
ub, errB := uuid.Parse(b)
|
|
return errA == nil && errB == nil && ua == ub
|
|
}
|
|
|
|
// DecodeID unpacks a compat UUID into its original numeric value.
|
|
func DecodeID(raw string) (DecodedID, error) {
|
|
parsed, err := uuid.Parse(raw)
|
|
if err != nil {
|
|
return DecodedID{}, fmt.Errorf("parse uuid: %w", err)
|
|
}
|
|
|
|
return DecodedID{
|
|
Type: EncodedIDType(parsed[0]),
|
|
Value: binary.BigEndian.Uint64(parsed[8:]),
|
|
}, nil
|
|
}
|