* fix(jellycompat): guard aux search paths and index-back person search
Short, recursive type-ahead terms (e.g. a single "a") against the
PostgreSQL people index and the in-memory collection/box-set filter
produced multi-second scans that pegged the server when a client fired
one search per keystroke. Every jellycompat search path except the
Meilisearch-backed /Items media search now rejects a provided SearchTerm
shorter than 3 runes without touching any backend, and caps results at
20 regardless of the client-requested Limit:
- /Persons (PostgreSQL people scan)
- /Search/Hints (catalog search)
- /Items BoxSet (in-memory collection filter)
Shared policy + helpers live in search_guard.go (auxSearchMinTermLen=2,
i.e. reject 1-2 runes / allow 3+, and auxSearchMaxResults=20) with unit
coverage. The 3-rune floor is deliberate: 3 runes is the point where a
pg_trgm trigram index becomes usable, so the gate lines up with the index
and still lets legitimate short titles/names ("300", "Saw", 3-letter
actors) and 3-char type-ahead hints through.
PersonRepository.Search filters with `name ILIKE '%'||$1||'%'` rather
than `LOWER(name) LIKE '%'||LOWER($1)||'%'`. The old expression filtered
on LOWER(name), which the trigram GIN index idx_people_name_trgm (built
on name) could not serve, so every search fell back to an ordered index
scan on idx_people_name that walked the whole table for rare terms
(~300-400ms on the 889k-row production people table). ILIKE on name lets
pg_trgm serve rare 3+ char terms from the trigram index, while the planner
still picks the ordered btree scan with early termination for common
terms. ILIKE is case-insensitive, so behavior is preserved (including the
pre-existing treatment of % and _ in the term as LIKE wildcards).
Verified on production silo-postgres (889,302 people), parameterized query
under both custom and generic plan cache modes:
rare 3-char 'qzx': 304ms -> 1-8ms (trgm bitmap index)
rare 4-char 'zzzz': 329ms -> 1-2ms (trgm bitmap index)
common 3-char 'ann': 15ms -> 7-74ms (btree early-stop / trgm bitmap)
New worst case across all terms is ~83ms (generic-plan common 3-char).
* fix(jellycompat): stop gating meilisearch hints, clamp box-set search
Two review follow-ups on the aux-search guards:
- HandleSearchHints is served by the catalog (Meilisearch-backed) search
provider, which already bounds and short-term-handles its own results.
Gating it with auxSearchTermTooShort contradicted the guard's own
"non-Meilisearch paths only" policy and hid valid 1-2 char titles
("Up", "It") from global type-ahead. Drop the too-short gate there;
keep the empty-term check and the result clamp.
- handleBoxSetsList gated short terms but passed query.limit straight to
slicePage, which treats <=0 as no cap, so a box-set *search* was
uncapped unlike Persons/Hints. Clamp the limit when a search term is
present; empty-term browse keeps its client paging window.
media_items had many columns the Go model (models.MediaItem) declares as
non-pointer string/int fields, but migration 001 left them nullable. Five
scanners (catalog item_repo, catalog browse, jellycompat, sections, and
the catalog API handler) read these straight into the non-pointer fields,
so a NULL row panics with "cannot scan NULL into *string" (or *int).
item_repo papers over a subset (poster/backdrop/logo/metadata paths) with
COALESCE in its SELECT, but the other four scanners read the same columns
raw and crash; sort_title/original_title/etc. are not coalesced anywhere.
No writer stores a meaningful NULL (every insert/upsert passes the Go
field, '' or 0 at worst) and all sort/filter SQL already collapses NULL
and '' (e.g. COALESCE(NULLIF(BTRIM(sort_title), ''), title); "poster_path
IS NULL OR poster_path = ''"). Enforce the invariant the code already
assumes at the schema level rather than scattering COALESCE across every
current and future scanner. Mirrors what later migrations already did for
original_language, show_status, default_metadata_language, and the
*_source_path columns (all NOT NULL DEFAULT '').
- Migration: backfill existing NULLs, then NOT NULL DEFAULT '' on 15 text
columns (sort_title, original_title, content_rating, overview, tagline,
imdb_id/tmdb_id/tvdb_id, poster_path, poster_thumbhash, backdrop_path,
backdrop_thumbhash, logo_path, metadata_s3_path, metadata_etag) and
NOT NULL DEFAULT 0 on year/runtime.
- item_repo: SetLocalPoster and UpdateArtworkIfSourceMatches now store ''
for empty thumbhashes (was NULLIF($,'')) — the only deliberate NULL
writers — matching the upsert path.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(watchsync): sync watchlists with Trakt/Simkl/MDBList
Extend the watch-providers feature to sync a user's watchlist, generalizing
the existing favorites pipeline rather than duplicating it.
What changed
- Generalize the favorites sync into one ListKind-parameterized pipeline
(internal/watchsync/lists.go) driving both favorites and watchlist; the
per-favorites service methods are replaced by kind-generic ones. The shadow
table watch_provider_favorite_items becomes watch_provider_list_items with a
list_kind discriminator.
- Providers: Trakt gains watchlist sync (/sync/watchlist, distinct from
favorites); Simkl gains plan-to-watch sync; MDBList is re-mapped from
favorites to watchlist (its only list is a watchlist) — its capabilities now
report import_favorites=false / import_watchlist=true, and the migration
re-binds existing MDBList connections.
- Auto-remove watched items from the watchlist: a standalone, default-on
profile preference (user_profiles.remove_watched_from_watchlist) removes a
movie when watched and a series once every episode is watched. Implemented as
watchstate.CompletionObserver (internal/watchlist.Maintainer), wired into the
manual mark-watched, playback-stop, and jellycompat mark-played paths.
- Optional MDBList sort-order mirroring: an opt-in, capability-gated toggle
mirrors MDBList's watchlist order into Silo via user_watchlist.sort_index;
ListWatchlist orders by sort_index then added_at, so both /api/v1/watchlist
and the catalog watchlist view inherit it.
- Real-time + scheduled: local add/remove pushes to connected providers
immediately (removals gated by the opt-in removals toggle); the hourly job is
the inbound/import + retry/reconcile path.
- Web: watch-provider settings gain watchlist import/export/removals and
"mirror watchlist order" toggles plus watchlist sync stats.
Why
- The favorites and watchlist pipelines are ~90% identical; generalizing keeps
one code path (per CLAUDE.md's anti-duplication guidance) instead of cloning.
API/compat
- All new fields on ConnectionStatus/Capabilities/ConnectionUpdate/SyncRun and
the web types are additive (Silo v1 additive-only rule). No existing field is
renamed, removed, or retyped.
Risks / follow-up
- MDBList capability flip is intentional and client-visible: silo-android /
silo-apple may need to surface MDBList under the watchlist (not favorites) UI.
- MDBList existing users: their MDBList list previously mirrored Silo favorites
and now mirrors Silo watchlist; the first post-migration sync is a union
(removals default off), so nothing is destructively purged.
- Order mirroring reflects the order MDBList returns from /watchlist/items
(couldn't confirm against their docs — Cloudflare-blocked); if it ever
diverges from the UI sort, a sort param is the small follow-up.
Tests: new maintainer (auto-remove) and watchlist-order unit tests; provider +
service tests updated. go build, go test (affected pkgs), migrate-validate,
verify-local-paths, web prettier/eslint/tsc all pass.
AI-use disclosure: implemented with Claude Code (Claude Opus 4.8).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(watchsync): update list shadow table references
* fix(watchsync): address review — retry/progress + error propagation
Addresses CodeRabbit review on #227:
- maintainer: propagate transient catalog lookup errors instead of silently
treating every items.GetByID failure as "maybe an episode".
- exportList: mark every queued item not confirmed sent (not_found, failed, or
omitted) so the pending loop always advances; the next run's upsert clears the
error and re-attempts, so transient failures still retry.
- removePendingListItems + realtime removal: treat Sent and NotFound as
reconciled; leave true failures pending (no last_error, which would strand
them from the removal query) so the scheduled run retries, using in-memory
dedupe to terminate the loop.
- exportLocalListItems: send the normalized items (with computed
ProviderItemKey), not the original event slice.
- UpdateConnection: clear mirrored watchlist order before persisting the disable
and propagate failures, so a failed clear can't report "disabled" while
sort_index ordering is still active.
- web: include favorite + watchlist removal counts in the exported "sent" total.
- test: align serviceFakeRepo list-state with Postgres (clear last_error on
successful transitions); add maintainer error-propagation test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(collections): sync matches across all bound libraries
Collection sync resolved item membership against only the legacy
library_collections.library_id, so a collection bound to multiple
libraries (N:N library_collection_libraries) had its items collapsed to a
single library on every sync.
Switch the MDBList/TMDB/Trakt sync matchers to the existing
LibraryItemRepository.GetItemsInFolders over collection.LibraryIDs.
GetByID already populates LibraryIDs from the N:N table (falling back to
[LibraryID]), so single-library collections are unaffected. No schema
change; listing and the catalog open path were already N:N-aware.
Unblocks merging duplicated per-library collections into single
multi-library collections.
* fix(collections): dedupe matched items by media_item_id before ReplaceItems
A collection sync resolves source entries (external IDs) to library
content_ids and appends one item per matched entry without deduping by
media_item_id. When two different source entries resolve to the same
library item, the same content_id is appended twice; ReplaceItems then
hits the library_collection_items composite PK (collection_id,
media_item_id) with a duplicate-key violation, failing the whole sync
with an HTTP 500 and zero items.
The multi-library union added in the previous commit (GetItemsInFolders
over collection.LibraryIDs) widens the matched set and so raises the
collision probability, making this latent bug more likely to fire.
Dedupe at the single choke point in ReplaceItems (first occurrence wins,
preserving rank order) so all five sync paths are covered at once, and
renumber position over the surviving rows so it stays dense.
Also log the underlying error in HandleSyncAdminCollection, which
previously mapped every non-sentinel sync error to a generic 500 with
no log line, making this failure invisible on the server.
* fix(jellycompat): authenticate Android TV token-less direct play
Stock Jellyfin Android TV ignores the api_key-bearing DirectStreamUrl
returned from PlaybackInfo and builds its own direct-play URL with no
auth header, no api_key/ApiKey query param, and no PlaySessionId. The
media request arrives via the player's HTTP stack (okhttp) with
auth_kind=none, so PlaybackSessionAuth 401s it — the client retries,
falls back to a transcode that stalls, and surfaces "player error".
Add a third fallback in PlaybackSessionAuth, scoped strictly to the
direct-play video stream routes (/Videos/{id}/stream[.{container}]) via
the chi route pattern so /Items/{id}/Download stays protected: anchor
auth on the PlaybackSession negotiated during the preceding (already
authenticated) PlaybackInfo, looked up by mediaSourceId when present
(else the route item id), and resolve its CompatToken.
Covered by tests: token-less direct play with a matching session
succeeds, no matching session 401s, and Download is not loosened.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(jellycompat): require session item match + expand direct-play tests
Address CodeRabbit review on PR #200:
- Require the matched PlaybackSession's RouteItemID to equal the requested
route item before authorizing, so a mediaSourceId cannot authorize a
stream for a different item.
- Seed the compat session in the 401 tests so they fail on route/session
scoping rather than a missing session.
- Table-drive the positive test across /Videos/{id}/stream and
/Videos/{id}/stream.{container}, plus the route-item lookup branch when
mediaSourceId is absent; add a cross-item denial test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(jellycompat): honor client subtitle selection in PlaybackInfo
Jellyfin clients send the chosen subtitle as SubtitleStreamIndex on
PlaybackInfo, but the compat layer only ever advertised the media's
embedded default subtitle (or none). Picking a subtitle before playback
had no effect and the video started without it — and an explicit
"subtitles off" was ignored when the media had a default subtitle.
Mirror the existing audio-selection plumbing for subtitles:
- parse SubtitleStreamIndex from the PlaybackInfo body and query
- persist it as SelectedSubtitleStreamIndex on the play-session source
- advertise it via DefaultSubtitleStreamIndex and flip IsDefault onto the
chosen stream (embedded, external, or Silo-downloaded)
- honor a negative index as an explicit "subtitles off"
The selection is validated against streamable tracks (bitmap subs that
require burn-in are excluded, matching delivery) and the downloaded
subtitle range, falling back to the media default when invalid.
Out of scope (follow-ups): bitmap-subtitle burn-in transcode wiring and
mid-playback subtitle changes on progress reports.
Part of #217
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(jellycompat): handle downloaded-subtitle lookup failure in selection
Don't let a transient ListDownloadedSubtitles error masquerade as "no
downloaded subtitles", which silently downgraded a valid subtitle
selection to the media default. The lookup error is now logged, and
resolution honors a requested index it cannot validate (it may be a
downloaded subtitle) instead of discarding the user's choice, while
embedded/external selections continue to resolve normally.
Addresses CodeRabbit review feedback on #222.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
catalogSearchMeilisearchSchemaVersion hashed the embedder, canonical dimensions, and index types but not SemanticEnabled. Because attachDocumentVectors omits _vectors when semantic is off, toggling semantic on without a full rebuild left previously indexed (and incrementally synced) documents without vectors while the Postgres coverage gate still reported the type "ready" — silently degrading hybrid ranking until a manual rebuild.
Folding SemanticEnabled into the schema version makes a toggle diverge from the stored version, so the provider falls back to keyword, sync skips, the admin status surfaces the schema mismatch, and a rebuild is required to restore hybrid — consistent with how embedder and index-type changes already behave.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extract listPodcastShowAudioFiles and record each show's audio paths in
seenPaths during ScanPodcastFolder so reconciliation does not treat
still-present episodes as removed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Completes the search-provider-interface wiring that the catalog hardening
commits already call into:
- Skip the transactional search-index-event write path when Meilisearch is
not the active provider (ItemRepository.WithActiveSearchProvider /
SearchIndexEventRepository.disabledByActiveProvider).
- Dead-letter catalog_search_index_events after 10 attempts instead of
retrying forever.
- Track the rebuild high-water mark (MaxEventID / MarkProcessedThrough) and
persist last_processed_event_id in UpdateStateAfterRebuild so a rebuild
reconciles events enqueued during the rebuild.
- Validate (read-only) the embedding lock when embedding a search query
instead of establishing/mutating it.
- Surface total_exact on the legacy /items browse response.
- Wire the active catalog search provider into the scanner and item repo at
startup.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
EmbedAll previously ran the full text-staleness CTE on every page, paying
five item_people LATERAL joins per eligible row just to detect whether an
item's canonical text had drifted - even when the real work was embedding
brand-new (missing) items during active backfill.
Restructure EmbedAll into two passes:
- Pass 1 (cheap): drain missing/model-stale items via the repurposed
ItemsNeedingEmbedding query (single LEFT JOIN, no LATERAL), paged by a
content_id cursor so a failed/skipped item is retried next run instead of
stalling the page.
- Pass 2 (expensive): only once Pass 1 fully drains, run one bounded
ListEmbeddingTextCandidates scan (LIMIT embeddingTextStaleQuotaPerRun=200)
to re-embed text-drifted items. Re-embedding refreshes canonical_text, so
handled rows drop out next run - no Pass 2 cursor needed.
Coverage-first tradeoff (documented in EmbedAll): under steady state Pass 1
drains every run so text-stale items stay fresh; only under pathological
continuous heavy ingest does Pass 2 get skipped, deliberately prioritizing
covering new items over re-embedding changed ones.
Supporting changes:
- ItemsNeedingEmbedding gains an afterID cursor + ORDER BY; SQL extracted to
buildItemsNeedingEmbeddingSQL for a cheap-shape unit test (no item_people /
LATERAL).
- Add an `embedder` interface seam (Engine.embClient) so EmbedAll is testable
with a fake; *embeddings.Client still satisfies it.
- Extract embedBatch to DRY both passes, preserving quota/billing early-return,
single-item fallback, ensureEmbeddingLock-before-upsert, and skip-on-store-error.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a rate-limited Meilisearch embedder capability check and surface a
semantic-coverage block in the admin search status.
Capability validation confirms the active index declares the configured
embedder with source="userProvided" and dimensions matching the canonical
embedding dimension, then runs a unit-vector hybrid probe to confirm the
index accepts the request shape. The whole path (GetSettings + evaluate +
probe) is cached for 5m and NEVER trips the circuit, marks a fallback, or
modifies unhealthyUntil, so a misconfigured embedder cannot take keyword
search down. The pure evaluator (evaluateEmbedderSettings) is unit-tested
across missing/wrong-source/wrong-dimensions/ok with distinct reasons.
The admin status gains a Semantic block read entirely from the in-memory
coverage snapshot (Snapshot + CoverageReady) with no new DB query in
Status(); document_count/vector_document_count are unchanged (additive
only). buildSemanticStatus is pure and unit-tested (nil snapshot and a
populated, deterministically-sorted snapshot).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add per-query search observability to /api/v1/catalog. The internal
CatalogSearchResult now carries Mode + SemanticUsed alongside the existing
Provider + FallbackReason; all four are plumbed through CatalogResult (which
previously dropped them in resolveDirectSearchSource) and exposed as an
additive, omitempty search_diagnostics object on the catalog response.
Two correctness points:
- For Meilisearch, Mode/SemanticUsed are derived from the POST-downgrade
request (baseSearchReq.Hybrid != nil) at the end of searchMeilisearch, so a
hybrid->keyword self-downgrade reports mode="keyword", semantic_used=false.
- search_diagnostics is emitted iff a provider search actually ran
(result.Provider != ""), which naturally omits browse, preview,
non-relevance-sort q=, and group=work paths (none set Provider).
API additive-only: existing response fields keep declaration order and stay
byte-stable; only the new omitempty field is added.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire the semantic-coverage building blocks together so the Meilisearch
provider only goes hybrid once the active embedding model covers enough of
the requested item types.
- Add Coverage SemanticCoverageGate to MeilisearchProviderConfig and consult
it in buildMeilisearchSearchRequest after the wordcount check; when
not-ready, record "semantic_not_ready: <reason>" as a diagnostic fallback
and stay keyword-only (shouldUseSemanticSearch's bool contract unchanged).
- Build and own a semanticCoverageTracker in
NewCatalogSearchServiceFromSettings, but only when semantic is enabled and
a real pool is present. Derive the model provider via a panic-safe comma-ok
helper (semanticModelProvider) so a nil or non-implementing vectorizer
degrades to a not-ready gate instead of asserting a nil interface.
- Add StartCoverageRefresh and call it from NewRouter on deps.AppContext so
the refresher runs for the process lifetime.
Coverage == nil preserves today's behavior exactly; existing hybrid and
SkipTotal tests stay green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add the in-memory coverage gate the search hot path will consult to
decide whether semantic results may be served for a set of item types.
The tracker reads an atomic.Pointer snapshot with no DB and no lock on
the read path, refreshes single-flight under a mutex on a 2-minute
ticker, and applies per-type hysteresis (enable 0.90 / disable 0.80,
holding the previous latch inside the band) to avoid flapping. It
collapses to not-ready when the active embedding model changes so a
stale latch cannot leak across models, and fails safe: a not-yet-
computed snapshot reports not-ready (never panics), and a model-lookup
or count-query error retains the last-good snapshot instead of
publishing zeros.
Refresh counts through the existing catalogSemanticCoverageByType via an
injectable fetch seam, so the behavior is unit-testable with canned
counts and a fake model provider (no pgx.Rows faking). Reuses the
Task-1 coverageQuerier and extends catalogTypeCoverage with Ratio/Ready,
filled by the pure computeCoverageSnapshot helper. Adds the
SemanticCoverageGate interface beside the other provider interfaces;
Task 4 wires the gate and starts the refresher.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add recommendations.Engine.ActiveEmbeddingModel, which reads the
embedding lock from server_settings and returns the locked model (or ""
when no lock is established). Define the catalog
CatalogSemanticModelProvider interface so a later task can scope vector
coverage checks to the active model; the recommendations Engine already
satisfies it. No wiring yet (Task 4 injects recEngine as this provider).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Establish how semantic vector "coverage" is counted: current-model
embeddings over embed-eligible items, per media type, from Postgres.
- Move the embed-eligibility predicate to a single source of truth in
embeddingvectors.ItemEligibilityWhereClause; recommendations now
delegates to it (output byte-identical).
- Add catalogSemanticCoverageByType (per-type eligible/vectorized) and
define the coverageQuerier interface in catalog. Both numerator and
denominator apply the eligibility predicate, so vectorized never
exceeds eligible (C1 guarantee) even with a stale embedding on a
now-unmatched item.
- countCatalogSearchVectorDocuments now takes a coverageQuerier and a
model, applies the eligibility + model filter, and is the per-type
numerator summed across types. Update all four callers (pass "").
- Add idx_media_item_embeddings_model (CONCURRENTLY, self-healing
guard) so the model filter does not scan the embeddings table.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(collections): add profile-scoped display filters
* refactor(collections): dedup display-filter helpers per review
Address code-review feedback on the profile-scoped display filters
without changing behavior:
- Widen CompletedHistoryItemMap to accept ProgressCompletionStore and
drop the duplicate completedHistoryItemMapForProgress copy.
- Extract the duplicated MDBList candidate retry loop into a generic
collectionutil.FetchMDBListWithFallback helper, used by both the user
and library collection syncers, and cover it with unit tests.
- Reuse validateOptionalLibraryIDs in HandleUpdateCollection instead of
an inline positive-ID loop.
- Import the shared COLLECTION_{WATCH,MEDIA}_FILTER_OPTIONS in the
template config form rather than redefining them locally.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(collections): sanitize query_definition library_ids fallback
readSourceConfigLibraryIDs validated source_config.library_ids (finite,
positive, truncated, deduplicated) but returned the query_definition
fallback raw, so legacy rows could surface zero/negative/duplicate IDs
that the backend now rejects on save. Extract a shared sanitizer and
apply it to both paths.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(docs): This makes the agents annoying to work with
* Improve playback session handling
* Support collection source order in catalog filters
* fix(collections): address display filter review feedback
* refactor(catalog): remove duplicate collection query params
* Hide episode media scope for collection overlays
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Silo's ExtractToken only honored the legacy 'api_key' query parameter and
the X-Emby-Token/X-Mediabrowser-Token headers. Real Jellyfin's
AuthorizationContext treats 'ApiKey' (PascalCase) as the current,
always-enabled query token and 'api_key' as legacy (gated behind
EnableLegacyAuthorization). Native Jellyfin clients that build their own
direct-play /Videos/{id}/stream URLs (incl. Jellyfin Android TV) send
'ApiKey', which Silo rejected — the request arrived with auth_kind=none and
the route returned 401, surfacing as a client 'playbackerror'.
Match both 'ApiKey' and 'api_key' case-insensitively in ExtractToken and in
the authKind log classifier. Strictly additive: existing header and
api_key paths are unchanged.
Verified against the live server: /Videos/{id}/stream?...&ApiKey=<tok>
returned 401 before and is accepted after; &api_key=<tok> and the
X-Emby-Token header continue to return 206.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(jellycompat): expose library collections as a Collections library view
Surface server library collections as a top-level Jellyfin "Collections"
library (CollectionType "boxsets") so compat clients see them as the first
library in /UserViews and can browse them by ParentId. The BoxSet machinery
(list/detail/children) already existed; this adds the library wrapper.
- catalog: add LibraryCollectionRepository.AnyVisibleInLibraries, an
index-only EXISTS probe (no item join/aggregation) used to gate the view so
an empty Collections tab never shows. Mirrors collectionVisible semantics
(multi-library scope rows or the legacy single library_id fallback).
- jellycompat: add the synthetic Collections CollectionFolder (fixed Jellyfin
sentinel ID, stable across servers), prepend it to the user's views when a
visible collection exists, and route ParentId/Items-by-ID for that sentinel
to the existing BoxSet listing. ChildCount is left omitted (no per-call
count, no unwatched badge).
AI-use disclosure: implemented with assistance from Claude.
* fix(jellycompat): display collection posters and a generated Collections tile
Library collections surfaced as Jellyfin BoxSets showed blank cards: their
poster_url is frequently a bundled frontend template path
(/images/collection-templates/x.jpg), which the compat image route passed
through unchanged and then rejected in parseRemoteImageURL (no scheme/host),
returning BadGateway. Clients probed Images/Primary and got nothing.
- images: serve app-relative artwork (bundled template posters) straight from
the embedded frontend FS (new ImagesHandler.frontendFS), with content-type
and cache headers. Wired through jellycompat.Dependencies.FrontendFS.
- poster_gen: on-the-fly gradient poster generator (per-title hue, centered
white caption with black outline, gobold/opentype), memoized in a bounded
cache. Used for the synthetic Collections library tile and as a fallback for
collections without usable artwork, so cards are never blank.
- consolidate collection/view image routing in HandleItemImage, authorized by
the signed tag or an authenticated, visibility-checked session.
AI-use disclosure: implemented with assistance from Claude.
* fix(jellycompat): declare 2:3 PrimaryImageAspectRatio on BoxSets and Collections tile
Clients defaulted collection cards to a square and crop the 2:3 poster to fit.
Set PrimaryImageAspectRatio (portrait 2/3) on the BoxSet DTO and the synthetic
Collections library tile so the full poster is shown, matching Jellyfin.
AI-use disclosure: implemented with assistance from Claude.
* feat(metadata): expand provider image cache queue
* fix(metadata): harden provider image cache queue
Addresses bug-review feedback from Codex/CodeRabbit on the metadata image
cache pipeline. All findings validated against the code before fixing;
false positives (rows/connection deadlock, PhotoSourcePath merge coupling)
were confirmed non-issues and left unchanged.
- Honor metadata.cache_images for the background processor. The
cache_metadata_images task was registered whenever S3 was configured,
so merely enabling object storage downloaded the entire provider-artwork
catalog even with caching disabled. Add ImageCacheProcessor.SetEnabled,
gate RunOnce/RunUntilIdle on it, and wire it (with hot reload) from
cfg.Metadata.CacheImages in main.go.
- Guard terminal job updates with lease ownership. EnqueueBatch can
repurpose a running row with a new source; MarkSucceeded/MarkFailed
keyed on id alone let a stale worker finalize the replacement job and
drop the new artwork. Thread locked_by through and add
status='running' AND locked_by=$n guards.
- Avoid uploading stale jobs onto the live artwork key. Verify the
target still references the job's source (CurrentTargetSourcePath)
before CacheImage, so a job whose source an admin/refresh already
replaced cannot overwrite the deterministic storage object.
- COALESCE nullable external IDs in EnqueueExistingProviderArtwork. A
NULL tmdb_id/tvdb_id/imdb_id on any candidate failed the scan and
aborted the whole cache run; matches the existing item_repo pattern.
- Stop re-downloading the catalog every 30 days. Discovery now skips
targets whose *_path is already a cached relative path, making the
cached row the durable dedup marker instead of the prunable job row.
- Decouple catalog sweeps from queue draining. RunOnce no longer runs
discovery per batch; RunUntilIdle sweeps only when the queue drains and
throttles full sweeps to every 15m, so idle installs stop full-scanning
every entity table each minute.
- Requeue claimed-but-unstarted jobs on cancellation. Acquire the
semaphore before spawning workers and RequeueClaimed any jobs not yet
started, instead of leaving them locked until the 15m lease expires.
- Skip the backoff sleep after the final upload attempt in
putObjectWithRetry (saves ~1.5s on permanent failures).
- Add the s3/file/local/upload/generated exclusion to the seasons and
episodes backfill in migration 20260617184537 for consistency with the
later migration (the bad backfill was inert downstream, but the
asymmetry is removed).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
- List seasons or episodes correctly for Jellyfin parent item queries
- Deduplicate episode availability by logical episode identity
- Improve audiobook title fallbacks and existing item updates
* Kindle->EPUB conversion: design + proven wasm build pipeline
Server-side MOBI/AZW/AZW3 -> EPUB conversion so the Android in-app reader
can render Kindle-family ebooks. Conversion runs in-process via libmobi's
mobitool compiled to wasm32-wasi, executed by wazero (pure Go) -- no cgo,
no external binary, arch-independent, sandboxed untrusted input.
This commit lands the design + the validated build artifact (spike done):
- docs/.../2026-06-17-kindle-epub-conversion-design.md (Codex-reviewed;
9 review fixes folded in: failure contract, strong cache key + negative
cache, wazero command-module specifics, FS-sandbox tightening,
double-gated capability, serve headers, .wasm guardrails).
- tools/mobitool-wasm/{Dockerfile,README.md}: reproducible build of
mobitool.wasm (wasi-sdk 25, libmobi 9062742, zlib 1.3.1->wasm), with a
smoke-conversion gate. Build proven on native amd64.
- internal/ebookconvert/mobitool.wasm (+ .sha256): canonical artifact,
built on amd64. go:embed target for the converter package (next).
Spike proven on amd64: -e EPUB path works with --with-libxml2=no (internal
xmlwriter); converts MOBI6/KF8/HUFF-CDIC/unicode -> well-formed EPUB;
verified end-to-end under wazero (WASI preopen + argv + _start). Build
gotcha: link libmobi against real (wasm) zlib, not --with-zlib=no, to avoid
miniz duplicate-symbol clash with mobitool's zip miniz. DRM gotcha:
mobitool prints "Document is encrypted" to stdout but exits 0 -> detect via
stdout + output validation, not exit code.
Not yet implemented: internal/ebookconvert Go package (wazero harness +
cache + singleflight), read-handler wiring, admin flag, client capability.
v1-scope proposal required before PR.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ebookconvert: converter core + cache (Codex-reviewed)
internal/ebookconvert: in-process MOBI/AZW/AZW3 -> EPUB via the embedded
mobitool.wasm on wazero. Converter compiles the module once and instantiates
per conversion (isolated). Cache adds on-disk, singleflighted, size-bounded,
negative-cached conversion keyed by file identity + module fingerprint.
18 tests pass (DRM-free->valid EPUB, DRM->ErrDRMProtected + no output,
oversize/corrupt/missing/timeout/cancel/after-close, 6/8-way concurrent,
EPUB structural validation incl. stored-mimetype + container rootfile,
cache miss/hit/key-change/singleflight/eviction/negative-cache).
Codex review fixes folded in:
- timeout/cancel classified before generic nonzero exit (WithCloseOnContextDone
surfaces sys.ExitError special codes); no more bogus "exit <huge>".
- DRM detection scoped to known mobitool diagnostic LINES (Document is
encrypted / DRM key not found / Invalid DRM pid / DRM expired / DRM support
not included) -> no false-positive on book text; Print Replica -> clear fail.
- WithMemoryLimitPages cap; capped stdout/stderr writers; MaxOutputBytes.
- read-only fs.FS input mount + dedicated writable out dir; documented that
FS isolation ultimately relies on running as a non-root user (memory-safety
is the WASM boundary). validateEpub now requires STORED mimetype + verifies
the container.xml OPF rootfile exists. Atomic moveFile. Closed-guard.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ebookconvert: wire Kindle->EPUB into the read handler + capability endpoint
Server now transparently serves Kindle-family ebooks as EPUB when the admin
flag ebook.kindle_conversion_enabled is on and the WASM converter initialized.
- handlers.EbookConversion (converter + per-request flag predicate) on the read
handler; HandleReadFile -> h.serveEbook. Kindle + enabled -> cached EPUB with
X-Silo-Ebook-Conversion: converted, epub MIME, ETag = exact conversion cache
key, must-revalidate. Failure (DRM/corrupt/oversize/unservable) -> raw
original + X-Silo-Ebook-Conversion: failed + no-store, so the client opens
externally. Context cancel propagates (not a conversion verdict).
- GET /api/v1/ebooks/capability advertises {enabled, source_formats,
served_format, header contract}; enabled only when flag on AND converter
wired (double gate) so the Android client can decide whether to flip
mobi/azw/azw3 to in-app.
- router: buildEbookConversion compiles the module once at startup (feature off
if it fails), cache dir is a sibling of TranscodeDir, flag read per request.
Codex review fixes folded in: ETag derived from the exact SourceKey cache key
(id+size+mtime+oshash+module version), not a weaker hash; no-store on the raw
fallback; open/stat failure of a produced EPUB falls back to raw per the
contract instead of 500. 10 handler tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ebookconvert): harden conversion cache, HEAD path, and artifact verification
Addresses adversarial review + CodeRabbit findings on the Kindle->EPUB feature.
Correctness:
- Stop poisoning the negative cache on transient timeouts. Introduce
ErrConversionTimedOut (distinct, non-wrapping ErrConversionFailed); classify
the per-call timeout as transient and propagate a caller's cancel/deadline
verbatim instead of reclassifying it as a conversion failure. remember() now
only caches deterministic verdicts (DRM / failed), so a one-off timeout under
load no longer wedges a convertible book onto raw-fallback for 6h.
- Detach the singleflight conversion from any single caller's context (DoChan +
context.WithoutCancel), so one caller cancelling no longer aborts the shared
work for the others; the cache is still populated for the next reader.
- enforceBudget never evicts the entry it is about to return, and skips other
conversions' in-flight "converting-*" temp files.
- Cache hits refresh mtime so the mtime-ordered budget eviction is a real LRU,
not FIFO.
Read path:
- HEAD is now cache-only via Cache.Lookup: a hit serves real converted headers,
a negatively-cached source serves the failed contract, a miss advertises the
converted representation cheaply without triggering a (minute-long, ~1 GiB)
conversion. The GET still delivers the body + authoritative verdict.
- The admin flag is read through a short-TTL predicate so the read path and the
capability endpoint no longer hit the DB per request.
Artifact / build:
- Add an in-code provenance test (embedded mobitool.wasm matches its recorded
sha256) and a self-hosted CI job that runs the ebookconvert smoke conversions
+ provenance check, so the committed wasm can't silently rot.
- Pin + checksum-verify wasmtime in the build Dockerfile (drop curl|bash).
Docs: correct the design doc cache-key + setting-name descriptions, document the
HEAD/timeout/LRU semantics and resource limits, note DRM-marker brittleness, and
fix the README markdown table.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* ci: remove ebookconvert workflow
---------
Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: Quick <31828688+Quick104@users.noreply.github.com>
* perf(catalog): fix audiobook detail N+1 + slow people facets
Audiobook detail pages were slow in proportion to track count (up to 433
files/book). Root causes, found by EXPLAIN ANALYZE on the live DB:
- effectiveAudioSelection ran 3-4 user-store queries (profile, audio pref,
library pref) per file inside buildPlaybackInfo's loop, though the results
are invariant across a request. Introduce a request-scoped audioPrefResolver
that memoizes the store lookups (library prefs keyed by folder); a 400-file
audiobook now issues each query once instead of per file. Selection logic is
unchanged (audioPreference returns a copy so the original-language sentinel
is still resolved per file).
- buildAudiobookExtension ran its four independent related-content queries
serially; run them concurrently so latency is the slowest, not the sum.
- author/narrator browse facets did a full people-table scan; add a
(kind, content_id, person_id) index so the facet resolves from an index-only
scan of just that kind's credits (~112ms -> ~49ms on the live library).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf(catalog): cache audiobook author/narrator group browse
The Authors/Narrators audiobook pages were slow on cold load and slow again
after a hard refresh (fast only while the React Query client cache was warm).
Root cause (EXPLAIN ANALYZE on live, 31K-audiobook library): the grouped
browse query is ~234ms/page, there are ~13K distinct authors, and the client
pages through the entire list on every load (sequential 500-row requests). With
no server-side cache each of the ~20 pages re-ran the full aggregation
(COUNT(*) OVER() forces it), so a cold load was ~20x234ms. The client's 60s
staleTime was the only thing making a warm revisit fast; a refresh wiped it.
Fix: AudiobookGroupsCache caches the full sorted group list per (library,
group_by, sort, viewer) for 60s (matching the client staleTime, so no extra
staleness) and serves every page as an in-memory slice — one aggregation per
window instead of one per page, and a refresh is a cache hit. Also raise the
client page size 500->2000 so fewer sequential round-trips are needed now that
a larger page is a cheap slice.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf(settings): throttle per-request device last_seen upserts
Device-setting reads (HandleGetDeviceSetting, HandleGetEffectiveSettings,
HandleGetEffectiveSubtitleAppearance) each registered the request's device — an
INSERT ... ON CONFLICT upsert of last_seen_at on a single per-device row. A page
that fetches many settings fired hundreds of these concurrently; they serialized
on that row's lock (observed 100-237ms each, ~250 per page load in the slow
query log), taxing every settings fetch.
Throttle device registration to one upsert per (profile, device) per 5 minutes
via an in-process TTL cache, marking the device seen before the upsert so a
concurrent burst collapses to a single write. last_seen_at stays fresh to within
the window. Reads no longer issue a contended write on the hot path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf+fix(audiobooks): probe-repair, resume position, cache storm, groups reveal, hot-row + stats index
From the full audiobook code review (EXPLAIN + slow-query trace on live):
- #1 (P0, detail-page killer): NeedsCriticalProbeRepair required video codec/
resolution/tracks, which audio-only files never have, so PlaybackProbeEnsurer
re-ran ffprobe per file on every detail/watch load (up to N serial spawns for
an N-track book) and never converged. Gate video-field checks on the file
actually having a video stream. TDD.
- #3 (P0): abs session-sync rewound the resume cursor — UpdateProgressPosition
did an unconditional SET with no monotonic guard, ignored its error, and
no-op'd when no row existed (first-listen resume lost). Now a finish-preserving
GREATEST upsert; caller logs failures.
- #4 (P0 perf): progress reports fired every ~10s invalidated all of
catalogKeys.all → refetched every active browse/detail query incl the 13k
audiobook group lists. Scope invalidation to the reported item's detail.
- #6 (P1 perf): Authors/Narrators page rendered all ~13k groups + cover images
at once (main-thread freeze). Incremental reveal: render a capped window, grow
on scroll via IntersectionObserver.
- #10: throttle abs TouchToken last_seen upsert (one per token per 5min) — same
hot-row contention class as the device fix.
- #8: index abs_playback_sessions (user_id, profile_id, started_at) for the
listening-stats aggregations.
Deferred (need contract/validation): listening-time idempotency (client delta-vs-
cumulative), scanner deleted-file reconcile, abs session retention job, abs list-
handler batch fetch, scanner-output P2s (need re-backfill).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf(audiobooks): batch-fetch abs list/shelf handlers (kill N+1)
handleSimilarItems, handleItemsInProgress, and handleGetMyProgress called
MediaStore.GetAudiobookByID once per row — up to ~500 single fetches (each a
few queries) on app open. Add GetAudiobooksByIDs (one access-scoped fetch +
people/series hydrated once for the whole set) and look results up from the
returned map, preserving order. Underlying primitives (GetByIDsWithAccess,
hydratePeople/Series) were already batch-capable.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(audiobooks): reconcile deleted files on scan + prune session history
#5: ScanAudiobookFolder only ever upserted — deleted/renamed books leaked
media_items/media_files/memberships forever. Mirror the ebook reconcile: collect
seenPaths during the walk, MarkMissing files no longer on disk, then
reconcileLibraryMemberships. Safety mirrors ebooks/video: an inaccessible root
(unmounted source) is skipped entirely, and a walk that saw zero files while the
DB has rows only reconciles after operator cleanup confirmation
(ebookEmptyCleanupAllowed) — so a flapping mount can't wipe the catalog. Soft
mark only; the existing grace-period purge hard-deletes later. Reconcile runs
only on a fully-completed (non-cancelled) scan. (#9 coarse case already handled:
audiobookFolderShouldSkip skips unchanged folders; per-file reuse deferred.)
#8-retention: abs_playback_sessions grew unbounded (one row per play-start, never
deleted) and fed every listening-stats scan. Add an hourly sweep in
SessionCleaner: close abandoned open sessions (no /close, stopped syncing >24h)
and delete closed sessions older than 90 days. Mirrors the recommendation_cache /
missing-files prune pattern.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(audiobooks): address max-effort code-review findings
From /code-review max on the pre-PR diff:
- DATA RACE (P0): SessionCleaner.lastABSSessionPrune is read+written by both the
15s ticker goroutine and the shutdown-path CleanStale call (main.go defers
Stop() to after that call). Guard the prune-due gate with a mutex. (CleanStale
was stateless before this branch, so concurrent calls were previously safe.)
- ScanAudiobookFolder hardcoded fullScan=true into the empty-walk cleanup guard,
but it's also called from ScanSubtree (incremental scans). An empty subtree
scan would wrongly consume the operator's one-shot empty-cleanup allowance and
warn. Thread a real fullScan flag (true from ScanFolder, false from the two
subtree call sites), mirroring the ebook path.
- Revert UpdateProgressPosition to UPDATE-only (drop the INSERT-on-missing):
keep the monotonic GREATEST + finish guard that fixes the resume rewind, but
restore the no-op-on-missing contract so a stray sync tick can't resurrect
just-cleared progress or create a zero-duration continue-listening row.
- Clamp the audiobook-groups handler limit (paging moved into the cache, leaving
the old 500/page bound stranded); also gofmt the Scanner struct.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(audiobooks): address review feedback for scanner and stats
* fix(audiobooks): address review feedback
* fix(audiobooks): retry failed session prune
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Quick <31828688+Quick104@users.noreply.github.com>
HandleGetWatchDetail populated user_data (resume position) only for
movie/episode/ebook, so an audiobook fetched via /watch returned a null
position_seconds even though progress is recorded the same way. The
/catalog/items/{id} path already includes audiobooks; this brings the watch
endpoint in line so both expose the saved position.
Co-authored-by: rxwatcher <rxwatcher@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
