* feat(catalog): deterministic cross-server content_id Replace per-server Sonyflake content_id with a structured natural key derived from provider IDs (movie:tmdb:…, series:tvdb:…, episode:…, local:… fallback), so two servers holding the same title share one anchor for artwork, watch history, progress, favorites and ratings. - internal/contentid: derivation core, SeriesIDFromContentID transform, frozen precedence, SchemeVersion=1, embedded-series-anchor invariant. - internal/metadata/service.go: deterministic id at every mint site. - internal/catalog/history_source.go: resolve show via string transform for anchored episode ids; skip the episodes_pkey probe. - migrations/sql/20260612130000: collision-safe value remap across the 65-column reference graph + COLLATE "C", FK/trigger handling, audit map, working down. Benchmarked against an exact-cardinality copy of cprod-postgres (1.93M episodes, 775k history rows): 2.57x faster history page, 1.7x throughput at 100 concurrent users, 2.7x cheaper per content_id probe. * feat(catalog): re-ID untagged items to deterministic content_id at first match Untagged libraries get a path-derived local: content_id at scan time and only learn their provider IDs later, when the match worker confirms a result. Previously that id was never folded back in, so untagged-then-matched items kept a per-server local: placeholder forever and never converged across servers (re-ID was deferred to a migration rerun). mergeAndPersist now promotes a local: skeleton to its deterministic provider-anchored id at the moment of first confirmed match, via a single new gate (canonicalizeLocalContentID): - target id already taken -> merge onto it (existing rebind machinery) - target id free -> rename in place The rename is a single SQL function (silo_rename_content_id); FK children follow via ON UPDATE CASCADE added to the content_id family, so a fresh skeleton moves a handful of rows rather than the full-table remap the bulk migration does. The guard is one IsLocal prefix check, so tagged content and all refreshes pay nothing, and the move is self-healing under retry. Verified: gofmt/vet/build clean; migrate-validate passes; migration applies on the real schema (up/down/up), FKs gain ON UPDATE CASCADE while keeping ON DELETE; functional test confirms series PK move + series_id cascade + provider-id sweep, and movie rename. Follow-ups (noted in docs): recomposeSeriesChildIDs for a series that accumulated episodes before matching; a lockstep test for the soft-ref list. * fix(catalog): harden content_id parsing and merge per review Address review feedback on the deterministic content_id work: - history_source.go: gate the anchored-episode display-id transform on the full five-part episode shape (split_part parts 2-5 non-empty), not just the 'episode:' prefix, so a malformed id can't transform to 'series:broken:' and vanish at the media_items join. Shared anchoredEpisodePredicate drives both the null-poisoned join key and the series-recovery expression. - contentid.go: unexport the provider-precedence slices so no package can mutate the frozen SchemeVersion ordering at runtime. - contentid.go: add parseAnchored to validate the exact per-kind arity and numeric season/episode suffixes; SeriesIDFromContentID and IsProviderAnchored now fail closed on truncated/malformed ids (e.g. "episode:tvdb:296762"). - canonicalize.go: distinguish catalog.ErrItemNotFound from transient lookup errors (a real error no longer masquerades as "target free"), and allow a matched local source to be consolidated onto the canonical row instead of orphaning a duplicate. * refactor(contentid): URL-safe "-" separator in content_id Use "-" instead of ":" to join content_id components (movie-tmdb-228064, episode-tvdb-296762-1-5, local-<hex>). "-" is an RFC 3986 unreserved character, so a content_id is URL-safe verbatim: encodeURIComponent is a no-op and the id is its own tidy path segment (/item/series-tvdb-296762) with no %3A escaping. The stored value equals the URL value, so there is no encode/decode boundary and an operator can grep the id straight out of a URL or log. Every component is [a-z0-9]+ (or "tt"+digits), so "-" is unambiguous. Pre-release format finalization: this branch is unmerged, so no deployed data carries ":" ids — the migration mints the "-" form fresh and no re-migration is needed. Still SchemeVersion 1. - contentid.go: single `sep` constant drives construction and parsing so the two can never drift; all constructors/parsers and doc examples updated. - history_source.go: split_part transform and the anchored-episode predicate use '-'; kept in lockstep with the package via a code comment. - 20260612130000_deterministic_content_id.sql: derivation and season/episode composition emit '-'; LIKE filters match 'series-%'. - docs/architecture/deterministic-content-id.md: format spec + rationale for the separator choice; this is the design doc the change is derived from. Client-side: the web frontend treats content_id as an opaque string (no splitting/regex), so no client changes are required; existing encodeURIComponent call sites simply stop emitting %3A. * docs(contentid): show why hash/bigint rejected in probe-cost table Add Cross-server deterministic / Zero-join show transform / Human-readable columns to the index-probe-cost comparison so the trade-off is legible at a glance: the 128-bit hash and bigint surrogate are faster but each give up a load-bearing property, and the structured key is the only all-checkmark row. * docs(contentid): order probe-cost table to end on the structured key * docs(contentid): label fenced blocks and drop stray EOF tags Per CodeRabbit review: add 'text' language to three fenced code blocks (MD040) and remove accidental </content></invoke> artifacts at EOF. * fix(catalog): remap array-valued content_id soft references in deterministic id migration The value-remap migration (20260612130000) enumerates the reference graph by FK plus a scalar name+type sweep (text/varchar/bpchar). That misses trending_discover_snapshots.content_ids: it is text[] (excluded by the type filter), named content_ids not content_id (excluded by the name list), and cannot carry an FK — so the bulk remap left those arrays holding stale Sonyflake ids that resolve to nothing until the snapshot regenerates. A counterexample to the migration's "self-protecting, cannot orphan" invariant. Remap the array element-wise in both directions (Up old->new, Down new->old), preserving order and leaving collision/unmatched elements untouched; a WHERE EXISTS guard skips empty/unaffected arrays so array_agg never collapses the NOT NULL column to NULL. Mirror the gap in silo_rename_content_id (20260614120000) with array_replace for the single-value runtime rename so the two stay in lockstep. Verified on PG18: mixed/collision/empty arrays remap correctly and round-trip clean; runtime array_replace preserves order. Surfaced reviewing #155. The jellycompat restart-decode regression and the atomicity-wording nit are posted as review comments, not addressed here. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(jellycompat): pack content_id into compat UUID reversibly so item ids survive restarts Addresses the restart-decode regression raised in review of #155. With content_id now a structured string instead of a numeric Sonyflake, EncodeStringID sent every item/season id down the one-way SHA1 path, making decode depend on an in-memory reverse map. That map is cold after a process restart (the codec is a process-lifetime singleton), so a client presenting a previously-issued item UUID — resume-from-home, deep link, detail page, image, userdata — got "unknown compat id" until the item was re-listed. Make the encoding reversible instead of stateful: - internal/contentid: add Pack/Unpack, a bit-packed, fixed-budget (<=15 byte) binary form of a structured or local content_id. digitCount preserves provider-id leading zeros (e.g. imdb tt0944947); structured forms are self-delimiting; the local form fills the budget exactly. Provider ids that overflow uint64 return ok=false. - Shrink ForLocal to a 112-bit (sha256(path)[:14]) hash so a local id packs losslessly into the 15-byte UUID payload. 112 bits is far beyond any single server's local-item count. No other code assumed the old width. - internal/jellycompat: EncodeStringID packs item/season content_ids into the UUID (byte 0 = kind, bytes 1..15 = packed, non-zero tag distinguishes it from the numeric encoding); DecodeStringID unpacks first and re-packs to confirm, so an opaque id whose bytes merely parse is rejected and falls through to the map. Numeric ids and arbitrary names (genres, studios) are unchanged. Net: item/season ids decode by pure computation — stable across restarts and across instances — with no lookup table. Only the rare unpackable content_id and non-content names still use the in-memory map. TDD: round-trip property tests in contentid (all kinds, leading zeros, reject cases) and a cross-instance decode test in jellycompat that fails on the old hash+map path. Full contentid + jellycompat suites green; production code golangci-clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(migrate): make the migration run timeout configurable (SILO_MIGRATE_TIMEOUT) The boot-path migration runner hardcoded a 5-minute context timeout. The deterministic-content-id value-remap (20260612130000) does a full-table COLLATE rewrite + 65-column remap that needs ~20 min on a real dataset (615k items / 2M episodes), so it was cancelled at 5 min. Worse, Postgres keeps the orphaned backend running (holding AccessExclusive locks) until it notices the dead client at a statement boundary, while the goose session advisory lock releases on disconnect — so each 5-min boot retry piled a new attempt behind the previous one's locks. The migration never applied; the server boot-looped. Make the timeout configurable via SILO_MIGRATE_TIMEOUT (a Go duration like "60m"); 0 or negative disables the deadline for a one-off heavy migration. Default stays 5m. All three entry points (migrate-status, --migrate-only, boot) honor it. Required for the deterministic-content-id migration to apply on any real-sized database, not just dev — the 5m cap made the PR undeployable at scale. Follow-up (not here): on cancellation the runner should actively terminate its backend so a future timeout cannot orphan a lock-holding statement. TDD: MigrationTimeout parsing (default/override/zero/invalid) + MigrationContext deadline behavior. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(contentid): require exact length for local ids in Unpack Tighten the tagLocal branch of Unpack from `len(body) < localHashLen` to an equality check. The local form fills the compat-UUID payload exactly (no padding), so a body of any other length is non-canonical; matching it exactly keeps Unpack a strict fail-closed inverse of Pack for the fixed-length branch, which decodes client-supplied UUIDs. Not applied to the structured branch (a review suggestion proposed the same change there): structured ids are self-delimiting and the compat layer pads them with trailing zeros to fill the 15-byte UUID payload, so ignoring trailing bytes is intentional and documented. Rejecting them would make every structured id fail to decode — the jellycompat cross-instance test guards against that. Not a live bug today (the only caller passes u[1:] from a 16-byte UUID, so body is always exactly localHashLen, and idcodec re-packs to verify), but it is the correct contract and zero-risk. Adds a regression test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Quick <31828688+Quick104@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
283 lines
8.3 KiB
Go
283 lines
8.3 KiB
Go
package catalog
|
|
|
|
import (
|
|
"context"
|
|
"fmt"
|
|
"strings"
|
|
"time"
|
|
)
|
|
|
|
func historySourceCanUseOptimizedPageQuery(req CatalogRequest) bool {
|
|
if req.Source != CatalogSourceHistory || !req.UseSourceOrder {
|
|
return false
|
|
}
|
|
if strings.TrimSpace(req.SearchQuery) != "" || strings.TrimSpace(req.NamePrefix) != "" {
|
|
return false
|
|
}
|
|
|
|
def := req.Query.Normalize()
|
|
return def.MediaScope == "" &&
|
|
len(def.LibraryIDs) == 0 &&
|
|
len(def.Groups) == 0
|
|
}
|
|
|
|
func (r *CatalogResolver) resolveHistorySourcePage(
|
|
ctx context.Context,
|
|
req CatalogRequest,
|
|
access AccessFilter,
|
|
) (*CatalogResult, error) {
|
|
snapshot := time.Now().UTC()
|
|
if req.SnapshotAt != nil {
|
|
snapshot = *req.SnapshotAt
|
|
}
|
|
|
|
displayIDs, total, hasMore, err := r.loadHistoryDisplayPage(
|
|
ctx,
|
|
access,
|
|
req.Limit,
|
|
req.Offset,
|
|
!req.SkipTotal,
|
|
&snapshot,
|
|
)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
|
|
items, err := r.fetchAccessibleItemsByID(ctx, displayIDs, req, access)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
|
|
return &CatalogResult{
|
|
Items: items,
|
|
Total: total,
|
|
HasMore: hasMore,
|
|
TotalExact: !req.SkipTotal,
|
|
SnapshotAt: snapshot,
|
|
}, nil
|
|
}
|
|
|
|
func (r *CatalogResolver) loadHistoryDisplayPage(
|
|
ctx context.Context,
|
|
access AccessFilter,
|
|
limit int,
|
|
offset int,
|
|
includeTotal bool,
|
|
snapshot *time.Time,
|
|
) ([]string, int, bool, error) {
|
|
if r == nil || r.itemRepo == nil || r.itemRepo.pool == nil {
|
|
return nil, 0, false, fmt.Errorf("catalog resolver requires an item repository")
|
|
}
|
|
if access.UserID <= 0 || strings.TrimSpace(access.ProfileID) == "" {
|
|
return nil, 0, false, fmt.Errorf("%w: history source requires active user scope", ErrInvalidCatalogRequest)
|
|
}
|
|
if limit <= 0 {
|
|
limit = 20
|
|
}
|
|
if offset < 0 {
|
|
offset = 0
|
|
}
|
|
|
|
baseQuery, baseArgs := buildHistoryDisplayBaseQuery(access, snapshot)
|
|
|
|
total := 0
|
|
if includeTotal {
|
|
countQuery := fmt.Sprintf(`WITH history_display AS (%s) SELECT COUNT(*) FROM history_display`, baseQuery)
|
|
if err := r.itemRepo.pool.QueryRow(ctx, countQuery, baseArgs...).Scan(&total); err != nil {
|
|
return nil, 0, false, fmt.Errorf("counting history display rows: %w", err)
|
|
}
|
|
if total == 0 {
|
|
return []string{}, 0, false, nil
|
|
}
|
|
}
|
|
|
|
queryLimit := limit
|
|
if !includeTotal {
|
|
queryLimit++
|
|
}
|
|
|
|
args := append([]any{}, baseArgs...)
|
|
limitArgIdx := len(args) + 1
|
|
args = append(args, queryLimit)
|
|
|
|
offsetClause := ""
|
|
if offset > 0 {
|
|
offsetArgIdx := len(args) + 1
|
|
offsetClause = fmt.Sprintf(" OFFSET $%d", offsetArgIdx)
|
|
args = append(args, offset)
|
|
}
|
|
|
|
pageQuery := fmt.Sprintf(
|
|
`WITH history_display AS (%s)
|
|
SELECT display_id
|
|
FROM history_display
|
|
ORDER BY watched_at DESC, display_id ASC
|
|
LIMIT $%d%s`,
|
|
baseQuery,
|
|
limitArgIdx,
|
|
offsetClause,
|
|
)
|
|
rows, err := r.itemRepo.pool.Query(ctx, pageQuery, args...)
|
|
if err != nil {
|
|
return nil, 0, false, fmt.Errorf("querying history display page: %w", err)
|
|
}
|
|
defer rows.Close()
|
|
|
|
displayIDs := make([]string, 0, limit)
|
|
for rows.Next() {
|
|
var displayID string
|
|
if err := rows.Scan(&displayID); err != nil {
|
|
return nil, 0, false, fmt.Errorf("scanning history display row: %w", err)
|
|
}
|
|
displayIDs = append(displayIDs, displayID)
|
|
}
|
|
if err := rows.Err(); err != nil {
|
|
return nil, 0, false, fmt.Errorf("iterating history display rows: %w", err)
|
|
}
|
|
|
|
hasMore := false
|
|
if includeTotal {
|
|
hasMore = total > offset+len(displayIDs)
|
|
return displayIDs, total, hasMore, nil
|
|
}
|
|
if len(displayIDs) > limit {
|
|
hasMore = true
|
|
displayIDs = displayIDs[:limit]
|
|
}
|
|
return displayIDs, 0, hasMore, nil
|
|
}
|
|
|
|
func buildHistoryDisplayBaseQuery(access AccessFilter, snapshot *time.Time) (string, []any) {
|
|
args := []any{access.UserID, access.ProfileID}
|
|
argIdx := 3
|
|
|
|
conditions := []string{
|
|
"h.user_id = $1",
|
|
"h.profile_id = $2",
|
|
`NOT EXISTS (
|
|
SELECT 1
|
|
FROM user_history_hidden_items hhi
|
|
WHERE hhi.user_id = h.user_id
|
|
AND hhi.profile_id = h.profile_id
|
|
AND hhi.media_item_id = h.media_item_id
|
|
AND h.watched_at <= hhi.hidden_before
|
|
)`,
|
|
}
|
|
|
|
if snapshot != nil {
|
|
conditions = append(conditions, fmt.Sprintf("h.watched_at <= $%d", argIdx))
|
|
args = append(args, *snapshot)
|
|
argIdx++
|
|
}
|
|
|
|
if access.AllowedContentIDs != nil {
|
|
if len(access.AllowedContentIDs) == 0 {
|
|
conditions = append(conditions, "1 = 0")
|
|
} else {
|
|
conditions = append(conditions, fmt.Sprintf("mi.content_id = ANY($%d)", argIdx))
|
|
args = append(args, access.AllowedContentIDs)
|
|
argIdx++
|
|
}
|
|
}
|
|
|
|
if len(access.AllowedLibraryIDs) > 0 {
|
|
conditions = append(conditions, fmt.Sprintf(`EXISTS (
|
|
SELECT 1
|
|
FROM media_item_libraries mil
|
|
WHERE mil.content_id = mi.content_id
|
|
AND mil.media_folder_id = ANY($%d)
|
|
)`, argIdx))
|
|
args = append(args, access.AllowedLibraryIDs)
|
|
argIdx++
|
|
} else if access.AllowedLibraryIDs != nil {
|
|
conditions = append(conditions, "1 = 0")
|
|
}
|
|
|
|
if len(access.DisabledLibraryIDs) > 0 {
|
|
conditions = append(conditions, fmt.Sprintf(`NOT EXISTS (
|
|
SELECT 1
|
|
FROM media_item_libraries mil_disabled
|
|
WHERE mil_disabled.content_id = mi.content_id
|
|
AND mil_disabled.media_folder_id = ANY($%d)
|
|
)`, argIdx))
|
|
args = append(args, access.DisabledLibraryIDs)
|
|
argIdx++
|
|
}
|
|
|
|
ApplySectionAccessFilter("mi", access, &conditions, &args, &argIdx)
|
|
|
|
// display_id resolves a history row (which may be an episode) to its shown
|
|
// item (the series, for episodes). For provider-anchored episode ids the
|
|
// series is a pure string transform of the id (the format invariant), so we
|
|
// skip the episodes_pkey probe entirely. Legacy Sonyflake and local episode
|
|
// ids carry no embedded anchor, so they still fall back to the episodes
|
|
// lookup. The join key is null-poisoned for anchored ids (= NULL is an
|
|
// unsatisfiable b-tree scan key, so the planner never descends the index for
|
|
// them) — an outer-only predicate like NOT LIKE would not actually skip the
|
|
// probe.
|
|
displayIDExpr := fmt.Sprintf(
|
|
"COALESCE(%s, NULLIF(e.series_id, ''), h.media_item_id)",
|
|
seriesFromAnchoredEpisodeExpr("h.media_item_id"),
|
|
)
|
|
|
|
// Null-poison the episodes join key for fully-formed anchored episode ids so
|
|
// the planner skips the episodes_pkey probe for them; everything else (legacy
|
|
// Sonyflake, local, malformed) still falls back to the lookup.
|
|
episodeJoinKey := fmt.Sprintf(
|
|
"CASE WHEN %s THEN NULL ELSE h.media_item_id END",
|
|
anchoredEpisodePredicate("h.media_item_id"),
|
|
)
|
|
|
|
return fmt.Sprintf(
|
|
`SELECT DISTINCT ON (history_events.display_id) history_events.display_id, history_events.watched_at
|
|
FROM (
|
|
SELECT %[1]s AS display_id, h.watched_at
|
|
FROM user_watch_history h
|
|
LEFT JOIN episodes e
|
|
ON e.content_id = %[3]s
|
|
JOIN media_items mi ON mi.content_id = %[1]s
|
|
WHERE %[2]s
|
|
) history_events
|
|
ORDER BY history_events.display_id ASC, history_events.watched_at DESC`,
|
|
displayIDExpr,
|
|
strings.Join(conditions, " AND "),
|
|
episodeJoinKey,
|
|
), args
|
|
}
|
|
|
|
// anchoredEpisodePredicate is the SQL boolean that is TRUE only for a
|
|
// fully-formed provider-anchored episode content_id —
|
|
// episode-<provider>-<seriesId>-<season>-<episode>, i.e. five non-empty
|
|
// "-"-separated components. It deliberately rejects a broader shape like
|
|
// 'episode-broken': matching that on the prefix alone would transform it into
|
|
// 'series-broken-' and skip the episodes fallback, so the row would vanish at
|
|
// the media_items join. split_part is IMMUTABLE. The "-" delimiter matches the
|
|
// content_id format (internal/contentid); keep the two in lockstep.
|
|
func anchoredEpisodePredicate(col string) string {
|
|
return fmt.Sprintf(
|
|
`%[1]s LIKE 'episode-%%' `+
|
|
`AND split_part(%[1]s, '-', 2) <> '' `+
|
|
`AND split_part(%[1]s, '-', 3) <> '' `+
|
|
`AND split_part(%[1]s, '-', 4) <> '' `+
|
|
`AND split_part(%[1]s, '-', 5) <> ''`,
|
|
col,
|
|
)
|
|
}
|
|
|
|
// seriesFromAnchoredEpisodeExpr returns a SQL expression that recovers a show's
|
|
// content_id from a provider-anchored episode content_id by pure string
|
|
// transform — episode-<p>-<sid>-<s>-<e> -> series-<p>-<sid> — per the format
|
|
// invariant in docs/architecture/deterministic-content-id.md. It yields NULL
|
|
// for any id that is not a fully-formed provider-anchored episode (movies,
|
|
// series, local, legacy Sonyflake, or malformed episode ids), so callers
|
|
// COALESCE to the episodes-table lookup for those. split_part/||/CASE are all
|
|
// IMMUTABLE.
|
|
func seriesFromAnchoredEpisodeExpr(col string) string {
|
|
return fmt.Sprintf(
|
|
`CASE WHEN %[2]s `+
|
|
`THEN 'series-' || split_part(%[1]s, '-', 2) || '-' || split_part(%[1]s, '-', 3) END`,
|
|
col,
|
|
anchoredEpisodePredicate(col),
|
|
)
|
|
}
|