Compare commits

...
Author SHA1 Message Date
coderabbitai[bot]andGitHub 94a280e89a CodeRabbit Generated Unit Tests: Add unit tests for PR changes 2026-06-27 18:51:29 +00:00
QuickandClaude Opus 4.8 4d2a990de1 chore(skills): harden issue-to-pr with researched best practices
Folds in findings from Anthropic's Claude Code guidance and the agentic-SWE
literature:

- Manual-only trigger (disable-model-invocation) for this side-effecting,
  PR-creating workflow, per Anthropic's fix-issue template.
- Reproduce-first hard gate: confirm the bug exists on current main (red test
  before the fix); stop and report if it doesn't reproduce, rather than patching
  correct/stale code.
- Verification is the primary correctness signal; full-suite run for regressions;
  forbid reaching green by weakening/skipping tests or relaxing CI.
- Adversarial-review loop: confirm each finding is real/grounded before acting,
  fix only correctness/requirement-affecting findings (no gold-plating), and cap
  the loop at 2 (max 3) iterations before escalating.
- Guards against hallucinated APIs, plan-proportional-to-scope, and long-run
  context-loss (progress + failed-approaches notes, re-assert guardrails after
  compaction).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 16:07:36 -04:00
QuickandClaude Opus 4.8 96a279819e chore(skills): add issue-to-pr skill (worktree → fix → adversarial review → PR)
Adds a committed, shared Claude Code skill that takes a GitHub issue number,
creates an isolated worktree off the latest main, finds the root cause or
scopes the feature, implements and verifies the change, runs a Codex
adversarial review, and opens a ready PR.

Carves a narrow exception into the .gitignore /.claude/ rule so only this
skill directory is tracked; everything else under .claude/ stays ignored.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 16:07:36 -04:00
443445f0fe fix(catalog): make media_items scalar columns NOT NULL to stop NULL-scan crashes (#228)
media_items had many columns the Go model (models.MediaItem) declares as
non-pointer string/int fields, but migration 001 left them nullable. Five
scanners (catalog item_repo, catalog browse, jellycompat, sections, and
the catalog API handler) read these straight into the non-pointer fields,
so a NULL row panics with "cannot scan NULL into *string" (or *int).
item_repo papers over a subset (poster/backdrop/logo/metadata paths) with
COALESCE in its SELECT, but the other four scanners read the same columns
raw and crash; sort_title/original_title/etc. are not coalesced anywhere.

No writer stores a meaningful NULL (every insert/upsert passes the Go
field, '' or 0 at worst) and all sort/filter SQL already collapses NULL
and '' (e.g. COALESCE(NULLIF(BTRIM(sort_title), ''), title); "poster_path
IS NULL OR poster_path = ''"). Enforce the invariant the code already
assumes at the schema level rather than scattering COALESCE across every
current and future scanner. Mirrors what later migrations already did for
original_language, show_status, default_metadata_language, and the
*_source_path columns (all NOT NULL DEFAULT '').

- Migration: backfill existing NULLs, then NOT NULL DEFAULT '' on 15 text
  columns (sort_title, original_title, content_rating, overview, tagline,
  imdb_id/tmdb_id/tvdb_id, poster_path, poster_thumbhash, backdrop_path,
  backdrop_thumbhash, logo_path, metadata_s3_path, metadata_etag) and
  NOT NULL DEFAULT 0 on year/runtime.
- item_repo: SetLocalPoster and UpdateArtworkIfSourceMatches now store ''
  for empty thumbhashes (was NULLIF($,'')) — the only deliberate NULL
  writers — matching the upsert path.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 16:06:01 -04:00
02e62767a1 feat(watchsync): sync watchlists with Trakt/Simkl/MDBList (#227)
* feat(watchsync): sync watchlists with Trakt/Simkl/MDBList

Extend the watch-providers feature to sync a user's watchlist, generalizing
the existing favorites pipeline rather than duplicating it.

What changed
- Generalize the favorites sync into one ListKind-parameterized pipeline
  (internal/watchsync/lists.go) driving both favorites and watchlist; the
  per-favorites service methods are replaced by kind-generic ones. The shadow
  table watch_provider_favorite_items becomes watch_provider_list_items with a
  list_kind discriminator.
- Providers: Trakt gains watchlist sync (/sync/watchlist, distinct from
  favorites); Simkl gains plan-to-watch sync; MDBList is re-mapped from
  favorites to watchlist (its only list is a watchlist) — its capabilities now
  report import_favorites=false / import_watchlist=true, and the migration
  re-binds existing MDBList connections.
- Auto-remove watched items from the watchlist: a standalone, default-on
  profile preference (user_profiles.remove_watched_from_watchlist) removes a
  movie when watched and a series once every episode is watched. Implemented as
  watchstate.CompletionObserver (internal/watchlist.Maintainer), wired into the
  manual mark-watched, playback-stop, and jellycompat mark-played paths.
- Optional MDBList sort-order mirroring: an opt-in, capability-gated toggle
  mirrors MDBList's watchlist order into Silo via user_watchlist.sort_index;
  ListWatchlist orders by sort_index then added_at, so both /api/v1/watchlist
  and the catalog watchlist view inherit it.
- Real-time + scheduled: local add/remove pushes to connected providers
  immediately (removals gated by the opt-in removals toggle); the hourly job is
  the inbound/import + retry/reconcile path.
- Web: watch-provider settings gain watchlist import/export/removals and
  "mirror watchlist order" toggles plus watchlist sync stats.

Why
- The favorites and watchlist pipelines are ~90% identical; generalizing keeps
  one code path (per CLAUDE.md's anti-duplication guidance) instead of cloning.

API/compat
- All new fields on ConnectionStatus/Capabilities/ConnectionUpdate/SyncRun and
  the web types are additive (Silo v1 additive-only rule). No existing field is
  renamed, removed, or retyped.

Risks / follow-up
- MDBList capability flip is intentional and client-visible: silo-android /
  silo-apple may need to surface MDBList under the watchlist (not favorites) UI.
- MDBList existing users: their MDBList list previously mirrored Silo favorites
  and now mirrors Silo watchlist; the first post-migration sync is a union
  (removals default off), so nothing is destructively purged.
- Order mirroring reflects the order MDBList returns from /watchlist/items
  (couldn't confirm against their docs — Cloudflare-blocked); if it ever
  diverges from the UI sort, a sort param is the small follow-up.

Tests: new maintainer (auto-remove) and watchlist-order unit tests; provider +
service tests updated. go build, go test (affected pkgs), migrate-validate,
verify-local-paths, web prettier/eslint/tsc all pass.

AI-use disclosure: implemented with Claude Code (Claude Opus 4.8).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(watchsync): update list shadow table references

* fix(watchsync): address review — retry/progress + error propagation

Addresses CodeRabbit review on #227:
- maintainer: propagate transient catalog lookup errors instead of silently
  treating every items.GetByID failure as "maybe an episode".
- exportList: mark every queued item not confirmed sent (not_found, failed, or
  omitted) so the pending loop always advances; the next run's upsert clears the
  error and re-attempts, so transient failures still retry.
- removePendingListItems + realtime removal: treat Sent and NotFound as
  reconciled; leave true failures pending (no last_error, which would strand
  them from the removal query) so the scheduled run retries, using in-memory
  dedupe to terminate the loop.
- exportLocalListItems: send the normalized items (with computed
  ProviderItemKey), not the original event slice.
- UpdateConnection: clear mirrored watchlist order before persisting the disable
  and propagate failures, so a failed clear can't report "disabled" while
  sort_index ordering is still active.
- web: include favorite + watchlist removal counts in the exported "sent" total.
- test: align serviceFakeRepo list-state with Postgres (clear last_error on
  successful transitions); add maintainer error-propagation test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 16:05:53 -04:00
b4a66b29e2 fix(web): preserve untouched metadata levels when toggling a provider (#226)
toggleLevelProvider merged the toggled level into the levelChains state
object, which starts empty. After the first toggle the state collapsed to
a single level, so once chainDirty flipped true activeLevelChains dropped
every untouched level and the dialog rendered them empty.

Seed from the full resolved activeLevelChains map and write it back the
same way reorderLevel already does, so disabling a source on one TV level
no longer blanks the others on the first toggle.

The earlier redesign (b9a9613) only patched the per-level source fallback,
which kept the toggled level's items but did not preserve the rest of the
map; builds containing it still reproduced the issue.

Part of #213

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 12:46:40 -04:00
6854eac80e fix(web/deps): bump vulnerable transitive deps via pnpm overrides (#225)
Resolves 8 open Dependabot alerts in web/pnpm-lock.yaml (all dev-only
transitive dependencies) by pinning patched versions through the existing
pnpm.overrides mechanism:

- undici  7.24.7 -> 7.28.0  (via jsdom; alerts #43-#49, incl. 2 high:
  TLS cert validation bypass & cross-origin routing via SOCKS5 ProxyAgent)
- js-yaml 4.1.1  -> 4.2.0   (via @eslint/eslintrc; alert #39, quadratic DoS)
- @babel/core 7.29.0 -> 7.29.6 (via @vitejs/plugin-react; alert #38,
  arbitrary file read via sourceMappingURL)

No vulnerable versions remain in the lockfile. Web build and lint pass;
the only failing tests (ServerStorageStep ResizeObserver) are pre-existing
on main and unrelated to these bumps.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 12:37:43 -04:00
CoffeeKnyteandGitHub 6414061022 fix(collections): correct multi-library sync — match all bound libraries and dedupe items (#184)
* fix(collections): sync matches across all bound libraries

Collection sync resolved item membership against only the legacy
library_collections.library_id, so a collection bound to multiple
libraries (N:N library_collection_libraries) had its items collapsed to a
single library on every sync.

Switch the MDBList/TMDB/Trakt sync matchers to the existing
LibraryItemRepository.GetItemsInFolders over collection.LibraryIDs.
GetByID already populates LibraryIDs from the N:N table (falling back to
[LibraryID]), so single-library collections are unaffected. No schema
change; listing and the catalog open path were already N:N-aware.

Unblocks merging duplicated per-library collections into single
multi-library collections.

* fix(collections): dedupe matched items by media_item_id before ReplaceItems

A collection sync resolves source entries (external IDs) to library
content_ids and appends one item per matched entry without deduping by
media_item_id. When two different source entries resolve to the same
library item, the same content_id is appended twice; ReplaceItems then
hits the library_collection_items composite PK (collection_id,
media_item_id) with a duplicate-key violation, failing the whole sync
with an HTTP 500 and zero items.

The multi-library union added in the previous commit (GetItemsInFolders
over collection.LibraryIDs) widens the matched set and so raises the
collision probability, making this latent bug more likely to fire.

Dedupe at the single choke point in ReplaceItems (first occurrence wins,
preserving rank order) so all five sync paths are covered at once, and
renumber position over the surviving rows so it stays dense.

Also log the underlying error in HandleSyncAdminCollection, which
previously mapped every non-sentinel sync error to a generic 500 with
no log line, making this failure invisible on the server.
2026-06-26 12:28:14 -04:00
a3aa93534d fix(jellycompat): authenticate Android TV token-less direct play (#200)
* fix(jellycompat): authenticate Android TV token-less direct play

Stock Jellyfin Android TV ignores the api_key-bearing DirectStreamUrl
returned from PlaybackInfo and builds its own direct-play URL with no
auth header, no api_key/ApiKey query param, and no PlaySessionId. The
media request arrives via the player's HTTP stack (okhttp) with
auth_kind=none, so PlaybackSessionAuth 401s it — the client retries,
falls back to a transcode that stalls, and surfaces "player error".

Add a third fallback in PlaybackSessionAuth, scoped strictly to the
direct-play video stream routes (/Videos/{id}/stream[.{container}]) via
the chi route pattern so /Items/{id}/Download stays protected: anchor
auth on the PlaybackSession negotiated during the preceding (already
authenticated) PlaybackInfo, looked up by mediaSourceId when present
(else the route item id), and resolve its CompatToken.

Covered by tests: token-less direct play with a matching session
succeeds, no matching session 401s, and Download is not loosened.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(jellycompat): require session item match + expand direct-play tests

Address CodeRabbit review on PR #200:
- Require the matched PlaybackSession's RouteItemID to equal the requested
  route item before authorizing, so a mediaSourceId cannot authorize a
  stream for a different item.
- Seed the compat session in the 401 tests so they fail on route/session
  scoping rather than a missing session.
- Table-drive the positive test across /Videos/{id}/stream and
  /Videos/{id}/stream.{container}, plus the route-item lookup branch when
  mediaSourceId is absent; add a cross-item denial test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-26 12:27:37 -04:00
QuickandGitHub d83f7fadef fix(audiobooks): mirror ABS playback into live sessions (#203) 2026-06-26 12:27:12 -04:00
185a90b048 fix(jellycompat): honor client subtitle selection in PlaybackInfo (#222)
* fix(jellycompat): honor client subtitle selection in PlaybackInfo

Jellyfin clients send the chosen subtitle as SubtitleStreamIndex on
PlaybackInfo, but the compat layer only ever advertised the media's
embedded default subtitle (or none). Picking a subtitle before playback
had no effect and the video started without it — and an explicit
"subtitles off" was ignored when the media had a default subtitle.

Mirror the existing audio-selection plumbing for subtitles:
- parse SubtitleStreamIndex from the PlaybackInfo body and query
- persist it as SelectedSubtitleStreamIndex on the play-session source
- advertise it via DefaultSubtitleStreamIndex and flip IsDefault onto the
  chosen stream (embedded, external, or Silo-downloaded)
- honor a negative index as an explicit "subtitles off"

The selection is validated against streamable tracks (bitmap subs that
require burn-in are excluded, matching delivery) and the downloaded
subtitle range, falling back to the media default when invalid.

Out of scope (follow-ups): bitmap-subtitle burn-in transcode wiring and
mid-playback subtitle changes on progress reports.

Part of #217

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(jellycompat): handle downloaded-subtitle lookup failure in selection

Don't let a transient ListDownloadedSubtitles error masquerade as "no
downloaded subtitles", which silently downgraded a valid subtitle
selection to the media default. The lookup error is now logged, and
resolution honors a requested index it cannot validate (it may be a
downloaded subtitle) instead of discarding the user's choice, while
embedded/external selections continue to resolve normally.

Addresses CodeRabbit review feedback on #222.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 12:26:27 -04:00
QuickandGitHub 78d4f9acbf Merge pull request #223 from Silo-Server/codex/silodiscordtriage-httpsdiscordappcomchannels148783
Fix semantic-disabled Meilisearch settings
2026-06-26 12:12:31 -04:00
Quick 7f67bb80ad Fix semantic-disabled Meilisearch settings
- Omit Meilisearch embedders unless semantic search is enabled
- Default semantic ratio setting to 0.50
2026-06-26 12:09:45 -04:00
QuickandGitHub b21c3eea37 Merge pull request #221 from Silo-Server/codex/meilisearch-short-title-search
fix(search): tighten short meilisearch title queries
2026-06-26 10:53:11 -04:00
Quick f57c95e813 fix(search): update default semantic ratio for meilisearch settings 2026-06-26 10:47:02 -04:00
Quick a181586139 fix(search): keep meilisearch active during pending sync 2026-06-26 10:46:42 -04:00
Quick dd2784c0ee fix(search): tighten short meilisearch title queries 2026-06-26 10:13:15 -04:00
QuickandGitHub 4473f8c60b Merge pull request #220 from Silo-Server/codex/search-provider-interface
feat(search): add provider interface with initial Meilisearch support
2026-06-26 09:32:53 -04:00
Quick 125e94b429 fix(search): allow zero meilisearch task uid 2026-06-26 09:29:40 -04:00
QuickandClaude Opus 4.8 2a943550d4 fix(search): fold semantic toggle into catalog index schema version
catalogSearchMeilisearchSchemaVersion hashed the embedder, canonical dimensions, and index types but not SemanticEnabled. Because attachDocumentVectors omits _vectors when semantic is off, toggling semantic on without a full rebuild left previously indexed (and incrementally synced) documents without vectors while the Postgres coverage gate still reported the type "ready" — silently degrading hybrid ranking until a manual rebuild.

Folding SemanticEnabled into the schema version makes a toggle diverge from the stored version, so the provider falls back to keyword, sync skips, the admin status surfaces the schema mismatch, and a rebuild is required to restore hybrid — consistent with how embedder and index-type changes already behave.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 08:43:09 -04:00
QuickandClaude Opus 4.8 6b45cfd855 fix(scanner): track podcast show audio paths during folder reconcile
Extract listPodcastShowAudioFiles and record each show's audio paths in
seenPaths during ScanPodcastFolder so reconciliation does not treat
still-present episodes as removed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 08:34:18 -04:00
QuickandClaude Opus 4.8 ed6c084c68 feat(search): gate index events by active provider and harden rebuild reconcile
Completes the search-provider-interface wiring that the catalog hardening
commits already call into:

- Skip the transactional search-index-event write path when Meilisearch is
  not the active provider (ItemRepository.WithActiveSearchProvider /
  SearchIndexEventRepository.disabledByActiveProvider).
- Dead-letter catalog_search_index_events after 10 attempts instead of
  retrying forever.
- Track the rebuild high-water mark (MaxEventID / MarkProcessedThrough) and
  persist last_processed_event_id in UpdateStateAfterRebuild so a rebuild
  reconciles events enqueued during the rebuild.
- Validate (read-only) the embedding lock when embedding a search query
  instead of establishing/mutating it.
- Surface total_exact on the legacy /items browse response.
- Wire the active catalog search provider into the scanner and item repo at
  startup.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 08:34:04 -04:00
QuickandClaude Opus 4.8 91b5105f3c docs(search): add hybrid semantic search hardening plan
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 08:24:14 -04:00
QuickandClaude Opus 4.8 7080491f89 feat(web): surface semantic coverage and embedder capability in search admin
Render the new `semantic` block from the admin catalog-search status
endpoint in the Search Settings page: semantic readiness (with
disabled reason), overall vector coverage %, coverage-updated
timestamp, embedder capability, and a compact per-type coverage list.
Reuses the existing StatusRow/FieldGroup/Badge patterns and adds a
formatPercent helper; keeps the existing Vectorized Documents row.

The TS `semantic?` field is optional and the whole block is guarded so
a new frontend talking to an older backend during rollout degrades
gracefully.

Part of the hybrid-semantic-search hardening work (Task 8).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 00:10:33 -04:00
QuickandClaude Opus 4.8 13ac4b753c perf(recommendations): split embedding backfill into cheap and text-stale passes
EmbedAll previously ran the full text-staleness CTE on every page, paying
five item_people LATERAL joins per eligible row just to detect whether an
item's canonical text had drifted - even when the real work was embedding
brand-new (missing) items during active backfill.

Restructure EmbedAll into two passes:

- Pass 1 (cheap): drain missing/model-stale items via the repurposed
  ItemsNeedingEmbedding query (single LEFT JOIN, no LATERAL), paged by a
  content_id cursor so a failed/skipped item is retried next run instead of
  stalling the page.
- Pass 2 (expensive): only once Pass 1 fully drains, run one bounded
  ListEmbeddingTextCandidates scan (LIMIT embeddingTextStaleQuotaPerRun=200)
  to re-embed text-drifted items. Re-embedding refreshes canonical_text, so
  handled rows drop out next run - no Pass 2 cursor needed.

Coverage-first tradeoff (documented in EmbedAll): under steady state Pass 1
drains every run so text-stale items stay fresh; only under pathological
continuous heavy ingest does Pass 2 get skipped, deliberately prioritizing
covering new items over re-embedding changed ones.

Supporting changes:
- ItemsNeedingEmbedding gains an afterID cursor + ORDER BY; SQL extracted to
  buildItemsNeedingEmbeddingSQL for a cheap-shape unit test (no item_people /
  LATERAL).
- Add an `embedder` interface seam (Engine.embClient) so EmbedAll is testable
  with a fake; *embeddings.Client still satisfies it.
- Extract embedBatch to DRY both passes, preserving quota/billing early-return,
  single-item fallback, ensureEmbeddingLock-before-upsert, and skip-on-store-error.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 00:02:20 -04:00
QuickandClaude Opus 4.8 a5ef464d95 feat(search): validate embedder capability and expose semantic status
Add a rate-limited Meilisearch embedder capability check and surface a
semantic-coverage block in the admin search status.

Capability validation confirms the active index declares the configured
embedder with source="userProvided" and dimensions matching the canonical
embedding dimension, then runs a unit-vector hybrid probe to confirm the
index accepts the request shape. The whole path (GetSettings + evaluate +
probe) is cached for 5m and NEVER trips the circuit, marks a fallback, or
modifies unhealthyUntil, so a misconfigured embedder cannot take keyword
search down. The pure evaluator (evaluateEmbedderSettings) is unit-tested
across missing/wrong-source/wrong-dimensions/ok with distinct reasons.

The admin status gains a Semantic block read entirely from the in-memory
coverage snapshot (Snapshot + CoverageReady) with no new DB query in
Status(); document_count/vector_document_count are unchanged (additive
only). buildSemanticStatus is pure and unit-tested (nil snapshot and a
populated, deterministically-sorted snapshot).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:43:21 -04:00
QuickandClaude Opus 4.8 dadd6f4769 feat(search): surface search diagnostics on catalog responses
Add per-query search observability to /api/v1/catalog. The internal
CatalogSearchResult now carries Mode + SemanticUsed alongside the existing
Provider + FallbackReason; all four are plumbed through CatalogResult (which
previously dropped them in resolveDirectSearchSource) and exposed as an
additive, omitempty search_diagnostics object on the catalog response.

Two correctness points:
- For Meilisearch, Mode/SemanticUsed are derived from the POST-downgrade
  request (baseSearchReq.Hybrid != nil) at the end of searchMeilisearch, so a
  hybrid->keyword self-downgrade reports mode="keyword", semantic_used=false.
- search_diagnostics is emitted iff a provider search actually ran
  (result.Provider != ""), which naturally omits browse, preview,
  non-relevance-sort q=, and group=work paths (none set Provider).

API additive-only: existing response fields keep declaration order and stay
byte-stable; only the new omitempty field is added.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:32:30 -04:00
QuickandClaude Opus 4.8 4cc92c1579 feat(search): gate hybrid on vector coverage readiness
Wire the semantic-coverage building blocks together so the Meilisearch
provider only goes hybrid once the active embedding model covers enough of
the requested item types.

- Add Coverage SemanticCoverageGate to MeilisearchProviderConfig and consult
  it in buildMeilisearchSearchRequest after the wordcount check; when
  not-ready, record "semantic_not_ready: <reason>" as a diagnostic fallback
  and stay keyword-only (shouldUseSemanticSearch's bool contract unchanged).
- Build and own a semanticCoverageTracker in
  NewCatalogSearchServiceFromSettings, but only when semantic is enabled and
  a real pool is present. Derive the model provider via a panic-safe comma-ok
  helper (semanticModelProvider) so a nil or non-implementing vectorizer
  degrades to a not-ready gate instead of asserting a nil interface.
- Add StartCoverageRefresh and call it from NewRouter on deps.AppContext so
  the refresher runs for the process lifetime.

Coverage == nil preserves today's behavior exactly; existing hybrid and
SkipTotal tests stay green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:17:47 -04:00
QuickandClaude Opus 4.8 f40611409d feat(search): semantic coverage tracker with hysteresis gate
Add the in-memory coverage gate the search hot path will consult to
decide whether semantic results may be served for a set of item types.

The tracker reads an atomic.Pointer snapshot with no DB and no lock on
the read path, refreshes single-flight under a mutex on a 2-minute
ticker, and applies per-type hysteresis (enable 0.90 / disable 0.80,
holding the previous latch inside the band) to avoid flapping. It
collapses to not-ready when the active embedding model changes so a
stale latch cannot leak across models, and fails safe: a not-yet-
computed snapshot reports not-ready (never panics), and a model-lookup
or count-query error retains the last-good snapshot instead of
publishing zeros.

Refresh counts through the existing catalogSemanticCoverageByType via an
injectable fetch seam, so the behavior is unit-testable with canned
counts and a fake model provider (no pgx.Rows faking). Reuses the
Task-1 coverageQuerier and extends catalogTypeCoverage with Ratio/Ready,
filled by the pure computeCoverageSnapshot helper. Adds the
SemanticCoverageGate interface beside the other provider interfaces;
Task 4 wires the gate and starts the refresher.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:06:51 -04:00
QuickandClaude Opus 4.8 e21c35cb7f feat(search): expose active embedding model from lock
Add recommendations.Engine.ActiveEmbeddingModel, which reads the
embedding lock from server_settings and returns the locked model (or ""
when no lock is established). Define the catalog
CatalogSemanticModelProvider interface so a later task can scope vector
coverage checks to the active model; the recommendations Engine already
satisfies it. No wiring yet (Task 4 injects recEngine as this provider).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 22:51:36 -04:00
QuickandClaude Opus 4.8 927d7764a4 feat(search): model-filtered embed-eligible coverage counts
Establish how semantic vector "coverage" is counted: current-model
embeddings over embed-eligible items, per media type, from Postgres.

- Move the embed-eligibility predicate to a single source of truth in
  embeddingvectors.ItemEligibilityWhereClause; recommendations now
  delegates to it (output byte-identical).
- Add catalogSemanticCoverageByType (per-type eligible/vectorized) and
  define the coverageQuerier interface in catalog. Both numerator and
  denominator apply the eligibility predicate, so vectorized never
  exceeds eligible (C1 guarantee) even with a stale embedding on a
  now-unmatched item.
- countCatalogSearchVectorDocuments now takes a coverageQuerier and a
  model, applies the eligibility + model filter, and is the per-type
  numerator summed across types. Update all four callers (pass "").
- Add idx_media_item_embeddings_model (CONCURRENTLY, self-healing
  guard) so the model filter does not scan the embeddings table.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 22:43:23 -04:00
Quick 20f36fdc31 fix(search): address provider review comments 2026-06-25 21:31:20 -04:00
Quick 5089e17ee6 fix(scanner): reconcile podcast rescans 2026-06-25 20:57:28 -04:00
Quick 07f2dd5a8f fix(admin): bound autoscan activity backlog 2026-06-25 20:50:45 -04:00
Quick 40329f616d perf(search): speed up catalog query results 2026-06-25 20:46:08 -04:00
Quick fb3a21c332 fix(search): batch orphan delete index events 2026-06-25 17:26:19 -04:00
Quick 86bc69c0a2 fix(search): defer meilisearch key check to runtime 2026-06-25 17:24:08 -04:00
Quick 6df388b25b fix(search): harden search index review paths 2026-06-25 17:00:45 -04:00
Quick 6ca427096b Add catalog search provider support 2026-06-25 16:20:14 -04:00
QuickandGitHub 89c435f046 Merge pull request #219 from Silo-Server/codex/fix-image-resolver-build
fix(plugins): restore image resolver startup build
2026-06-25 15:09:18 -04:00
Quick af52ea7bcc fix(plugins): use scoped installation store for image resolver 2026-06-25 15:08:55 -04:00
QuickandGitHub 13c0dde538 Merge pull request #218 from Silo-Server/codex/image-resolver-capability
feat(plugins): add image resolver registry
2026-06-25 15:01:07 -04:00
Quick 2045b7a0b2 feat(plugins): add image resolver registry 2026-06-25 14:47:48 -04:00
Quick 5c0027b54b fix(metadata): dedupe availability during re-id 2026-06-25 13:29:30 -04:00
87159b0a38 feat(collections): add profile-scoped display filters (#191)
* feat(collections): add profile-scoped display filters

* refactor(collections): dedup display-filter helpers per review

Address code-review feedback on the profile-scoped display filters
without changing behavior:

- Widen CompletedHistoryItemMap to accept ProgressCompletionStore and
  drop the duplicate completedHistoryItemMapForProgress copy.
- Extract the duplicated MDBList candidate retry loop into a generic
  collectionutil.FetchMDBListWithFallback helper, used by both the user
  and library collection syncers, and cover it with unit tests.
- Reuse validateOptionalLibraryIDs in HandleUpdateCollection instead of
  an inline positive-ID loop.
- Import the shared COLLECTION_{WATCH,MEDIA}_FILTER_OPTIONS in the
  template config form rather than redefining them locally.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(collections): sanitize query_definition library_ids fallback

readSourceConfigLibraryIDs validated source_config.library_ids (finite,
positive, truncated, deduplicated) but returned the query_definition
fallback raw, so legacy rows could surface zero/negative/duplicate IDs
that the backend now rejects on save. Extract a shared sanitizer and
apply it to both paths.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor(docs): This makes the agents annoying to work with

* Improve playback session handling

* Support collection source order in catalog filters

* fix(collections): address display filter review feedback

* refactor(catalog): remove duplicate collection query params

* Hide episode media scope for collection overlays

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-24 11:03:38 -04:00
Quick 227986094b Propagate playback client metadata through session sync 2026-06-23 14:26:14 -04:00
Quick 9819f0ff12 Merge branch 'main' of https://github.com/Silo-Server/silo-server 2026-06-23 14:17:21 -04:00
QuickandGitHub dc44e13bfa Filter restricted library sections and share collection queries (#190)
* fix(collections): prevent collection query cache collision

* Filter restricted library sections and share collection queries
2026-06-23 14:16:31 -04:00
Quick 3ad77095c4 fix(collections): prevent collection query cache collision 2026-06-22 18:46:13 -04:00
QuickandClaude Opus 4.8 0726d9430e docs: add Silo trademark & brand policy
Add TRADEMARK.md (canonical brand policy), a brand-asset NOTICE under
web/public, and a License & Trademarks section in the README. Documentation
only; no application code changes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-21 19:24:49 -04:00
544a262924 fix(jellycompat): accept Jellyfin current ApiKey query param for stream auth (#186)
Silo's ExtractToken only honored the legacy 'api_key' query parameter and
the X-Emby-Token/X-Mediabrowser-Token headers. Real Jellyfin's
AuthorizationContext treats 'ApiKey' (PascalCase) as the current,
always-enabled query token and 'api_key' as legacy (gated behind
EnableLegacyAuthorization). Native Jellyfin clients that build their own
direct-play /Videos/{id}/stream URLs (incl. Jellyfin Android TV) send
'ApiKey', which Silo rejected — the request arrived with auth_kind=none and
the route returned 401, surfacing as a client 'playbackerror'.

Match both 'ApiKey' and 'api_key' case-insensitively in ExtractToken and in
the authKind log classifier. Strictly additive: existing header and
api_key paths are unchanged.

Verified against the live server: /Videos/{id}/stream?...&ApiKey=<tok>
returned 401 before and is accepted after; &api_key=<tok> and the
X-Emby-Token header continue to return 206.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-21 16:20:04 -04:00
RXWatcherandGitHub 187f29e163 Merge pull request #183 from Silo-Server/fix/episodes-season-id-index-migration
Fix episodes season index migration
2026-06-19 06:23:21 +02:00
rxwatcher 729468be30 fix: wrap episodes season index migration block 2026-06-19 06:13:26 +02:00
QuickandGitHub de931f784a Fix ebook match dialog to support generic provider IDs (#182)
* fix(metadata): match non-video items with content-aware providers

* fix(web): guard invalid match year parsing
2026-06-18 20:19:47 -04:00
QuickandGitHub b84c040d27 Reinforce first profile empty state layout (#181) 2026-06-18 19:49:06 -04:00
QuickandGitHub cac435c4b9 Fix watch-state unwatch sync across user data and Jellyfin mappings (#179)
* Refine playback session handling and API responses

* fix(watchstate): harden completed-history visibility
2026-06-18 19:17:22 -04:00
dependabot[bot]andGitHub 08997264d2 build(deps): bump react-router from 7.15.0 to 7.15.1 in /web 2026-06-18 16:05:08 -04:00
dependabot[bot]andGitHub 42fd55db08 build(deps-dev): bump vite from 6.4.2 to 6.4.3 in /web 2026-06-18 16:04:45 -04:00
QuickandGitHub 7e532ea982 [codex] fix season page loading (#178)
* fix(catalog): speed up season page loads

* fix: address season loading review feedback
2026-06-18 14:18:35 -04:00
QuickandGitHub b3198276f7 [codex] fix(subtitles): stream live transcribe_translate cues (#177)
* fix(subtitles): stream live transcribe_translate cues

* fix(subtitles): harden live AI transcription
2026-06-18 12:16:27 -04:00
CoffeeKnyteandGitHub 6e1f79e8a9 feat(jellycompat): expose library collections as an auto-shown Collections view (#175)
* feat(jellycompat): expose library collections as a Collections library view

Surface server library collections as a top-level Jellyfin "Collections"
library (CollectionType "boxsets") so compat clients see them as the first
library in /UserViews and can browse them by ParentId. The BoxSet machinery
(list/detail/children) already existed; this adds the library wrapper.

- catalog: add LibraryCollectionRepository.AnyVisibleInLibraries, an
  index-only EXISTS probe (no item join/aggregation) used to gate the view so
  an empty Collections tab never shows. Mirrors collectionVisible semantics
  (multi-library scope rows or the legacy single library_id fallback).
- jellycompat: add the synthetic Collections CollectionFolder (fixed Jellyfin
  sentinel ID, stable across servers), prepend it to the user's views when a
  visible collection exists, and route ParentId/Items-by-ID for that sentinel
  to the existing BoxSet listing. ChildCount is left omitted (no per-call
  count, no unwatched badge).

AI-use disclosure: implemented with assistance from Claude.

* fix(jellycompat): display collection posters and a generated Collections tile

Library collections surfaced as Jellyfin BoxSets showed blank cards: their
poster_url is frequently a bundled frontend template path
(/images/collection-templates/x.jpg), which the compat image route passed
through unchanged and then rejected in parseRemoteImageURL (no scheme/host),
returning BadGateway. Clients probed Images/Primary and got nothing.

- images: serve app-relative artwork (bundled template posters) straight from
  the embedded frontend FS (new ImagesHandler.frontendFS), with content-type
  and cache headers. Wired through jellycompat.Dependencies.FrontendFS.
- poster_gen: on-the-fly gradient poster generator (per-title hue, centered
  white caption with black outline, gobold/opentype), memoized in a bounded
  cache. Used for the synthetic Collections library tile and as a fallback for
  collections without usable artwork, so cards are never blank.
- consolidate collection/view image routing in HandleItemImage, authorized by
  the signed tag or an authenticated, visibility-checked session.

AI-use disclosure: implemented with assistance from Claude.

* fix(jellycompat): declare 2:3 PrimaryImageAspectRatio on BoxSets and Collections tile

Clients defaulted collection cards to a square and crop the 2:3 poster to fit.
Set PrimaryImageAspectRatio (portrait 2/3) on the BoxSet DTO and the synthetic
Collections library tile so the full poster is shown, matching Jellyfin.

AI-use disclosure: implemented with assistance from Claude.
2026-06-18 10:20:48 -04:00
14ffc91dfb [codex] Expand provider image cache queue (#176)
* feat(metadata): expand provider image cache queue

* fix(metadata): harden provider image cache queue

Addresses bug-review feedback from Codex/CodeRabbit on the metadata image
cache pipeline. All findings validated against the code before fixing;
false positives (rows/connection deadlock, PhotoSourcePath merge coupling)
were confirmed non-issues and left unchanged.

- Honor metadata.cache_images for the background processor. The
  cache_metadata_images task was registered whenever S3 was configured,
  so merely enabling object storage downloaded the entire provider-artwork
  catalog even with caching disabled. Add ImageCacheProcessor.SetEnabled,
  gate RunOnce/RunUntilIdle on it, and wire it (with hot reload) from
  cfg.Metadata.CacheImages in main.go.
- Guard terminal job updates with lease ownership. EnqueueBatch can
  repurpose a running row with a new source; MarkSucceeded/MarkFailed
  keyed on id alone let a stale worker finalize the replacement job and
  drop the new artwork. Thread locked_by through and add
  status='running' AND locked_by=$n guards.
- Avoid uploading stale jobs onto the live artwork key. Verify the
  target still references the job's source (CurrentTargetSourcePath)
  before CacheImage, so a job whose source an admin/refresh already
  replaced cannot overwrite the deterministic storage object.
- COALESCE nullable external IDs in EnqueueExistingProviderArtwork. A
  NULL tmdb_id/tvdb_id/imdb_id on any candidate failed the scan and
  aborted the whole cache run; matches the existing item_repo pattern.
- Stop re-downloading the catalog every 30 days. Discovery now skips
  targets whose *_path is already a cached relative path, making the
  cached row the durable dedup marker instead of the prunable job row.
- Decouple catalog sweeps from queue draining. RunOnce no longer runs
  discovery per batch; RunUntilIdle sweeps only when the queue drains and
  throttles full sweeps to every 15m, so idle installs stop full-scanning
  every entity table each minute.
- Requeue claimed-but-unstarted jobs on cancellation. Acquire the
  semaphore before spawning workers and RequeueClaimed any jobs not yet
  started, instead of leaving them locked until the 15m lease expires.
- Skip the backoff sleep after the final upload attempt in
  putObjectWithRetry (saves ~1.5s on permanent failures).
- Add the s3/file/local/upload/generated exclusion to the seasons and
  episodes backfill in migration 20260617184537 for consistency with the
  later migration (the bad backfill was inert downstream, but the
  asymmetry is removed).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-18 10:07:58 -04:00
Quick fbe54ffec5 Merge branch 'codex/notifications-request-burst' 2026-06-17 13:49:18 -04:00
Quick 562ae635d7 feat: improve audiobook groups and notification refresh 2026-06-17 13:48:57 -04:00
Quick cf3e68e02a fix(jellycompat): honor codec profile playback limits 2026-06-17 13:12:18 -04:00
Quick 20e0dd7fe2 Fix Jellyfin parent browsing and release dedupe
- List seasons or episodes correctly for Jellyfin parent item queries
- Deduplicate episode availability by logical episode identity
- Improve audiobook title fallbacks and existing item updates
2026-06-17 13:12:18 -04:00
e99079abf8 Server-side Kindle→EPUB conversion (mobi/azw/azw3) for in-app reading (#171)
* Kindle->EPUB conversion: design + proven wasm build pipeline

Server-side MOBI/AZW/AZW3 -> EPUB conversion so the Android in-app reader
can render Kindle-family ebooks. Conversion runs in-process via libmobi's
mobitool compiled to wasm32-wasi, executed by wazero (pure Go) -- no cgo,
no external binary, arch-independent, sandboxed untrusted input.

This commit lands the design + the validated build artifact (spike done):
- docs/.../2026-06-17-kindle-epub-conversion-design.md (Codex-reviewed;
  9 review fixes folded in: failure contract, strong cache key + negative
  cache, wazero command-module specifics, FS-sandbox tightening,
  double-gated capability, serve headers, .wasm guardrails).
- tools/mobitool-wasm/{Dockerfile,README.md}: reproducible build of
  mobitool.wasm (wasi-sdk 25, libmobi 9062742, zlib 1.3.1->wasm), with a
  smoke-conversion gate. Build proven on native amd64.
- internal/ebookconvert/mobitool.wasm (+ .sha256): canonical artifact,
  built on amd64. go:embed target for the converter package (next).

Spike proven on amd64: -e EPUB path works with --with-libxml2=no (internal
xmlwriter); converts MOBI6/KF8/HUFF-CDIC/unicode -> well-formed EPUB;
verified end-to-end under wazero (WASI preopen + argv + _start). Build
gotcha: link libmobi against real (wasm) zlib, not --with-zlib=no, to avoid
miniz duplicate-symbol clash with mobitool's zip miniz. DRM gotcha:
mobitool prints "Document is encrypted" to stdout but exits 0 -> detect via
stdout + output validation, not exit code.

Not yet implemented: internal/ebookconvert Go package (wazero harness +
cache + singleflight), read-handler wiring, admin flag, client capability.
v1-scope proposal required before PR.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ebookconvert: converter core + cache (Codex-reviewed)

internal/ebookconvert: in-process MOBI/AZW/AZW3 -> EPUB via the embedded
mobitool.wasm on wazero. Converter compiles the module once and instantiates
per conversion (isolated). Cache adds on-disk, singleflighted, size-bounded,
negative-cached conversion keyed by file identity + module fingerprint.

18 tests pass (DRM-free->valid EPUB, DRM->ErrDRMProtected + no output,
oversize/corrupt/missing/timeout/cancel/after-close, 6/8-way concurrent,
EPUB structural validation incl. stored-mimetype + container rootfile,
cache miss/hit/key-change/singleflight/eviction/negative-cache).

Codex review fixes folded in:
- timeout/cancel classified before generic nonzero exit (WithCloseOnContextDone
  surfaces sys.ExitError special codes); no more bogus "exit <huge>".
- DRM detection scoped to known mobitool diagnostic LINES (Document is
  encrypted / DRM key not found / Invalid DRM pid / DRM expired / DRM support
  not included) -> no false-positive on book text; Print Replica -> clear fail.
- WithMemoryLimitPages cap; capped stdout/stderr writers; MaxOutputBytes.
- read-only fs.FS input mount + dedicated writable out dir; documented that
  FS isolation ultimately relies on running as a non-root user (memory-safety
  is the WASM boundary). validateEpub now requires STORED mimetype + verifies
  the container.xml OPF rootfile exists. Atomic moveFile. Closed-guard.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ebookconvert: wire Kindle->EPUB into the read handler + capability endpoint

Server now transparently serves Kindle-family ebooks as EPUB when the admin
flag ebook.kindle_conversion_enabled is on and the WASM converter initialized.

- handlers.EbookConversion (converter + per-request flag predicate) on the read
  handler; HandleReadFile -> h.serveEbook. Kindle + enabled -> cached EPUB with
  X-Silo-Ebook-Conversion: converted, epub MIME, ETag = exact conversion cache
  key, must-revalidate. Failure (DRM/corrupt/oversize/unservable) -> raw
  original + X-Silo-Ebook-Conversion: failed + no-store, so the client opens
  externally. Context cancel propagates (not a conversion verdict).
- GET /api/v1/ebooks/capability advertises {enabled, source_formats,
  served_format, header contract}; enabled only when flag on AND converter
  wired (double gate) so the Android client can decide whether to flip
  mobi/azw/azw3 to in-app.
- router: buildEbookConversion compiles the module once at startup (feature off
  if it fails), cache dir is a sibling of TranscodeDir, flag read per request.

Codex review fixes folded in: ETag derived from the exact SourceKey cache key
(id+size+mtime+oshash+module version), not a weaker hash; no-store on the raw
fallback; open/stat failure of a produced EPUB falls back to raw per the
contract instead of 500. 10 handler tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ebookconvert): harden conversion cache, HEAD path, and artifact verification

Addresses adversarial review + CodeRabbit findings on the Kindle->EPUB feature.

Correctness:
- Stop poisoning the negative cache on transient timeouts. Introduce
  ErrConversionTimedOut (distinct, non-wrapping ErrConversionFailed); classify
  the per-call timeout as transient and propagate a caller's cancel/deadline
  verbatim instead of reclassifying it as a conversion failure. remember() now
  only caches deterministic verdicts (DRM / failed), so a one-off timeout under
  load no longer wedges a convertible book onto raw-fallback for 6h.
- Detach the singleflight conversion from any single caller's context (DoChan +
  context.WithoutCancel), so one caller cancelling no longer aborts the shared
  work for the others; the cache is still populated for the next reader.
- enforceBudget never evicts the entry it is about to return, and skips other
  conversions' in-flight "converting-*" temp files.
- Cache hits refresh mtime so the mtime-ordered budget eviction is a real LRU,
  not FIFO.

Read path:
- HEAD is now cache-only via Cache.Lookup: a hit serves real converted headers,
  a negatively-cached source serves the failed contract, a miss advertises the
  converted representation cheaply without triggering a (minute-long, ~1 GiB)
  conversion. The GET still delivers the body + authoritative verdict.
- The admin flag is read through a short-TTL predicate so the read path and the
  capability endpoint no longer hit the DB per request.

Artifact / build:
- Add an in-code provenance test (embedded mobitool.wasm matches its recorded
  sha256) and a self-hosted CI job that runs the ebookconvert smoke conversions
  + provenance check, so the committed wasm can't silently rot.
- Pin + checksum-verify wasmtime in the build Dockerfile (drop curl|bash).

Docs: correct the design doc cache-key + setting-name descriptions, document the
HEAD/timeout/LRU semantics and resource limits, note DRM-marker brittleness, and
fix the README markdown table.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* ci: remove ebookconvert workflow

---------

Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: Quick <31828688+Quick104@users.noreply.github.com>
2026-06-17 13:09:55 -04:00
9b111649f3 perf+fix(audiobooks): detail page, browse, sessions & scanner (#169)
* perf(catalog): fix audiobook detail N+1 + slow people facets

Audiobook detail pages were slow in proportion to track count (up to 433
files/book). Root causes, found by EXPLAIN ANALYZE on the live DB:

- effectiveAudioSelection ran 3-4 user-store queries (profile, audio pref,
  library pref) per file inside buildPlaybackInfo's loop, though the results
  are invariant across a request. Introduce a request-scoped audioPrefResolver
  that memoizes the store lookups (library prefs keyed by folder); a 400-file
  audiobook now issues each query once instead of per file. Selection logic is
  unchanged (audioPreference returns a copy so the original-language sentinel
  is still resolved per file).
- buildAudiobookExtension ran its four independent related-content queries
  serially; run them concurrently so latency is the slowest, not the sum.
- author/narrator browse facets did a full people-table scan; add a
  (kind, content_id, person_id) index so the facet resolves from an index-only
  scan of just that kind's credits (~112ms -> ~49ms on the live library).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf(catalog): cache audiobook author/narrator group browse

The Authors/Narrators audiobook pages were slow on cold load and slow again
after a hard refresh (fast only while the React Query client cache was warm).

Root cause (EXPLAIN ANALYZE on live, 31K-audiobook library): the grouped
browse query is ~234ms/page, there are ~13K distinct authors, and the client
pages through the entire list on every load (sequential 500-row requests). With
no server-side cache each of the ~20 pages re-ran the full aggregation
(COUNT(*) OVER() forces it), so a cold load was ~20x234ms. The client's 60s
staleTime was the only thing making a warm revisit fast; a refresh wiped it.

Fix: AudiobookGroupsCache caches the full sorted group list per (library,
group_by, sort, viewer) for 60s (matching the client staleTime, so no extra
staleness) and serves every page as an in-memory slice — one aggregation per
window instead of one per page, and a refresh is a cache hit. Also raise the
client page size 500->2000 so fewer sequential round-trips are needed now that
a larger page is a cheap slice.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf(settings): throttle per-request device last_seen upserts

Device-setting reads (HandleGetDeviceSetting, HandleGetEffectiveSettings,
HandleGetEffectiveSubtitleAppearance) each registered the request's device — an
INSERT ... ON CONFLICT upsert of last_seen_at on a single per-device row. A page
that fetches many settings fired hundreds of these concurrently; they serialized
on that row's lock (observed 100-237ms each, ~250 per page load in the slow
query log), taxing every settings fetch.

Throttle device registration to one upsert per (profile, device) per 5 minutes
via an in-process TTL cache, marking the device seen before the upsert so a
concurrent burst collapses to a single write. last_seen_at stays fresh to within
the window. Reads no longer issue a contended write on the hot path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf+fix(audiobooks): probe-repair, resume position, cache storm, groups reveal, hot-row + stats index

From the full audiobook code review (EXPLAIN + slow-query trace on live):

- #1 (P0, detail-page killer): NeedsCriticalProbeRepair required video codec/
  resolution/tracks, which audio-only files never have, so PlaybackProbeEnsurer
  re-ran ffprobe per file on every detail/watch load (up to N serial spawns for
  an N-track book) and never converged. Gate video-field checks on the file
  actually having a video stream. TDD.
- #3 (P0): abs session-sync rewound the resume cursor — UpdateProgressPosition
  did an unconditional SET with no monotonic guard, ignored its error, and
  no-op'd when no row existed (first-listen resume lost). Now a finish-preserving
  GREATEST upsert; caller logs failures.
- #4 (P0 perf): progress reports fired every ~10s invalidated all of
  catalogKeys.all → refetched every active browse/detail query incl the 13k
  audiobook group lists. Scope invalidation to the reported item's detail.
- #6 (P1 perf): Authors/Narrators page rendered all ~13k groups + cover images
  at once (main-thread freeze). Incremental reveal: render a capped window, grow
  on scroll via IntersectionObserver.
- #10: throttle abs TouchToken last_seen upsert (one per token per 5min) — same
  hot-row contention class as the device fix.
- #8: index abs_playback_sessions (user_id, profile_id, started_at) for the
  listening-stats aggregations.

Deferred (need contract/validation): listening-time idempotency (client delta-vs-
cumulative), scanner deleted-file reconcile, abs session retention job, abs list-
handler batch fetch, scanner-output P2s (need re-backfill).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf(audiobooks): batch-fetch abs list/shelf handlers (kill N+1)

handleSimilarItems, handleItemsInProgress, and handleGetMyProgress called
MediaStore.GetAudiobookByID once per row — up to ~500 single fetches (each a
few queries) on app open. Add GetAudiobooksByIDs (one access-scoped fetch +
people/series hydrated once for the whole set) and look results up from the
returned map, preserving order. Underlying primitives (GetByIDsWithAccess,
hydratePeople/Series) were already batch-capable.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(audiobooks): reconcile deleted files on scan + prune session history

#5: ScanAudiobookFolder only ever upserted — deleted/renamed books leaked
media_items/media_files/memberships forever. Mirror the ebook reconcile: collect
seenPaths during the walk, MarkMissing files no longer on disk, then
reconcileLibraryMemberships. Safety mirrors ebooks/video: an inaccessible root
(unmounted source) is skipped entirely, and a walk that saw zero files while the
DB has rows only reconciles after operator cleanup confirmation
(ebookEmptyCleanupAllowed) — so a flapping mount can't wipe the catalog. Soft
mark only; the existing grace-period purge hard-deletes later. Reconcile runs
only on a fully-completed (non-cancelled) scan. (#9 coarse case already handled:
audiobookFolderShouldSkip skips unchanged folders; per-file reuse deferred.)

#8-retention: abs_playback_sessions grew unbounded (one row per play-start, never
deleted) and fed every listening-stats scan. Add an hourly sweep in
SessionCleaner: close abandoned open sessions (no /close, stopped syncing >24h)
and delete closed sessions older than 90 days. Mirrors the recommendation_cache /
missing-files prune pattern.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(audiobooks): address max-effort code-review findings

From /code-review max on the pre-PR diff:
- DATA RACE (P0): SessionCleaner.lastABSSessionPrune is read+written by both the
  15s ticker goroutine and the shutdown-path CleanStale call (main.go defers
  Stop() to after that call). Guard the prune-due gate with a mutex. (CleanStale
  was stateless before this branch, so concurrent calls were previously safe.)
- ScanAudiobookFolder hardcoded fullScan=true into the empty-walk cleanup guard,
  but it's also called from ScanSubtree (incremental scans). An empty subtree
  scan would wrongly consume the operator's one-shot empty-cleanup allowance and
  warn. Thread a real fullScan flag (true from ScanFolder, false from the two
  subtree call sites), mirroring the ebook path.
- Revert UpdateProgressPosition to UPDATE-only (drop the INSERT-on-missing):
  keep the monotonic GREATEST + finish guard that fixes the resume rewind, but
  restore the no-op-on-missing contract so a stray sync tick can't resurrect
  just-cleared progress or create a zero-duration continue-listening row.
- Clamp the audiobook-groups handler limit (paging moved into the cache, leaving
  the old 500/page bound stranded); also gofmt the Scanner struct.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(audiobooks): address review feedback for scanner and stats

* fix(audiobooks): address review feedback

* fix(audiobooks): retry failed session prune

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Quick <31828688+Quick104@users.noreply.github.com>
2026-06-17 09:51:43 -04:00
f207b87ff7 fix(catalog): return user_data for audiobooks on /watch/{id} (#144)
HandleGetWatchDetail populated user_data (resume position) only for
movie/episode/ebook, so an audiobook fetched via /watch returned a null
position_seconds even though progress is recorded the same way. The
/catalog/items/{id} path already includes audiobooks; this brings the watch
endpoint in line so both expose the saved position.

Co-authored-by: rxwatcher <rxwatcher@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 20:32:48 -04:00
c4cbcddeae feat(manga): manga library type — series grouping, reading loop, AniList/MangaDex metadata + status badge (#138)
* docs: design spec for manga library type (host sub-project)

Forks the ebooks library type into a 'manga' type: series detected from the
folder tree as a first-class type='manga' item, .cbz/.cbr chapters stay
readable ebook items linked via a new manga_chapters table, browse shows series
cards, enrichment targets the series item at content level 'manga'. Hands off to
a follow-on plugin spec for the manga metadata source.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: implementation plan for manga library type (host)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(scanner): manga filename index/volume parser

* feat(scanner): manga series-name-from-folder detection

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* docs(plan): align manga DB/scanner tasks to scanner pure-planner pattern (no test-DB)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(scanner): manga parser corpus regression

Add TestParseMangaIndexCorpus — 36 real-world scanlation filenames
covering bare chapter, decimal chapter, v/vol-prefix volume, and
c/ch-prefix chapter patterns; asserts <5% miss rate.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(db): manga_chapters link table

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(scanner): manga_chapters repository + pure chapter-write mapping

Adds mangaChapterWrite (pure, unit-tested), upsertMangaChapter, and
listMangaChapters following the ebook/audiobook thin-SQL pattern.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(scanner): recognize manga library type

Add isMangaLibraryType helper (unexported, matching the style of
isEbookLibraryType / isAudiobookLibraryType) with a corresponding
TestIsMangaLibraryType unit test.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(api): manga library content level

Map library type "manga" to content level ["manga"] in
metadataContentLevelsForLibraryType so that seedDefaultChain seeds a
manga-level metadata provider chain when a manga library is created.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(scanner): route manga libraries to a manga scan path

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(scanner): group manga chapters under a manga series item

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(scanner): give manga series item a library membership so it browses

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(catalog): browse manga libraries as series

Accept "manga" as a valid media_scope so a manga library browses only its
type='manga' series items; the per-chapter type='ebook' items are naturally
excluded because MediaScopeItemTypes("manga") expands to {"manga"}. Add the
manga default library sections (scoped to media_scope='manga') so the library
feed shows series cards. Refresh the two media_scope validation error messages.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(catalog): manga series detail lists chapters

For a type='manga' item, attach its chapters to the detail response via a new
MangaDetailExtension. fetchMangaChapters joins manga_chapters to media_items on
the chapter content ID, scopes to the series, and orders by chapter_index
(NULLS LAST) then sort_title — matching the scanner's chapter ordering.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(web): manga detail types + library browse scoping

Add MangaChapter/MangaDetailExtension TS types mirroring the host
catalog structs, wire manga? onto ItemDetail, and admit "manga" as a
QueryDefinition.media_scope. Scope manga libraries to media_scope=manga
in browse (host expands it to type=manga series items) while reusing the
ebook sort universe via getLibrarySortRelevanceScope. Add isMangaLibraryType.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(web): manga series detail with volume-grouped chapter list

Add MangaContent detail view: a DetailHero series header plus a chapter
list grouped by volume. groupMangaChapters (pure, unit-tested) buckets
chapters by their volume token, orders chapters within a group by
chapter_index (nulls last) and orders groups by their minimum index;
loose (volume-less) chapters collapse into a trailing "Chapters" group.
Each chapter links to the existing ebook reader by content_id alone
(file_id is optional — the reader resolves the file server-side), reusing
buildMediaPlayHref. Admit "manga" into ItemDetail.type and wire the
detail switch. Continue-reading is deferred (needs per-chapter progress
fan-out / a last-read timestamp not in the current payload).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(web): handle manga in playable-type + collection filter-scope unions

Adding "manga" to the shared ItemDetail["type"] and
QueryDefinition["media_scope"] unions leaked into consumers with narrower
local types, breaking the production tsc build. Fixes:

- mediaNavigation: admit "manga" into PlayableMediaType. Manga series are
  not directly playable (you open the detail page and read a chapter,
  itself an ebook item), so buildMediaPlayHref falls through to the item
  href for them, like series/season.
- FilterRuleEditor: add "manga" to FilterRuleMediaScope and relabel
  "watched" -> "Read" for manga as well as ebook (manga is read).
- CollectionGuidedRulesEditor: add "manga" to GuidedFormState.mediaScope,
  a "Manga" media-type option, ebook-like "Read Status" labels, and map
  manga -> ebook sort-relevance scope (manga has no dedicated sort scope).
- CatalogFilterBar (cascading leak surfaced after the above): add a
  "Manga" scope option and map manga -> ebook sort-relevance scope in both
  scope handlers.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(web): offer manga as a library type in the create dialog

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(scanner): strip scene-release junk from manga series names

Add cleanMangaSeriesName which repeatedly strips trailing parenthetical
groups (year, year-range, Digital, release-group tags) then trims any
dangling dash, so folder names like "404 Demons (Digital) (Oak)" resolve
to "404 Demons". Wire it into mangaSeriesFromPath so both the series
title and the mangaSeriesGroupKey identity key use the cleaned value.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(web): flat volume/chapter manga list; nest only multi-chapter volumes

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(scanner): parse manga index after stripping series-name prefix

Numbers inside a series title (e.g. "404 Demons", "365 Days to the
Wedding") were wrongly grabbed as the chapter number because
parseMangaIndex matched the first bare number in the full filename.
mangaIndexForFile now strips the series-name prefix before delegating
to parseMangaIndex, so only the number that follows the title is used.
reconcileMangaFile in manga_scan.go is updated to call mangaIndexForFile
instead of parseMangaIndex directly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(scanner): stop missing-file reconcile from deleting manga series items

Manga series items are file-less virtual parents; the shared
ReconcileFolderMembership swept them every scan because they have no
media_file. Exclude type='manga' from file-presence membership reconciliation,
and add a manga-scan step that deletes only series with zero remaining chapters.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ebooks): exclude manga chapters from individual ebook enrichment

Manga chapters are type='ebook' parts of a series; the ebook enrichment sweep
was searching each one against book sources (Gutenberg/Anna's/etc.) and failing
in a pointless storm. Exclude items with a manga_chapters link; series-level
enrichment is handled separately.

* docs: design spec for manga metadata plugin + series enrichment (sub-project 2)

New silo-plugin-manga-metadata (AniList, high-confidence matching) + a host
MangaEnricher for type='manga' series; default-enabled metadata source for manga
libraries.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: implementation plan for manga metadata plugin + series enrichment

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(db): manga_enrichment_state table

Mirrors ebook_enrichment_state: dedicated failure counter for the manga
enrichment sweep so it does not contend with media_items.refresh_failures.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(manga): series enricher (claims type='manga', resolves manga chain)

* feat(manga): sync_manga_metadata task + enricher wiring

* feat(catalog): expose manga chapter/volume counts in browse

Add manga_chapter_count and manga_volume_count to browse cards so the
frontend can render a Vols N / Ch N chip on manga series. The counts come
from two index-backed correlated subqueries over manga_chapters in the
browse SELECT (mangaCountColumns), scanned positionally before added_at and
nilled out for non-manga rows. Threaded through models.MediaItem and exposed
on the itemListResponse JSON card.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(sections): scope manga home recent sections to type=manga series

A manga library mixes type='manga' series with type='ebook' chapters, so
the auto-generated home 'Recently Added/Released in <Library>' rows surfaced
the junk chapter filenames. Add GeneratedHomeLibraryRecentConfigScoped which
emits the modern QueryDefinition shape (library_ids + media_scope) so a manga
library's generated home rows filter to type='manga' only. Other library
types are unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(catalog): exclude manga chapters from browse/section/search surfaces

Manga CHAPTER items (type='ebook' rows linked into a type='manga' series
via manga_chapters) were leaking into catalog browse, section resolution,
and search as standalone items showing junk filenames. They are internal
sub-units of the series and only the series should appear.

There is no single shared item-listing chokepoint: browse, the query/preview
executor, and search each build their own WHERE. Add a shared, index-backed
anti-join predicate (manga_chapters.chapter_content_id is the PK) via
mangaChapterExclusionWhere and wire it into all three builders. By-id fetch
paths that legitimately resolve chapters (ebook reader, continue-reading,
series detail chapter list) use separate queries and are unaffected.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(scanner): use #NN as the manga volume for Vol.YYYY #NN releases

mangaVolYearIssue early-return was returning the year token (e.g. "Vol.2003")
as the volume label, which the frontend couldn't prettify to "Volume N".
Now returns "v<issue>" (e.g. "v04") so the existing frontend regex ^v?(\d+)$
renders it as "Volume 4" correctly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(web): manga count chip on posters

Add an optional manga_chapter_count / manga_volume_count to the browse
item type and render a top-right "Vols N" / "Ch N" chip on ItemCard,
strictly gated on type==='manga'. The label prefers "Vols" when the
volume count dominates, "Ch" otherwise; the chip is hidden when the
chapter count is missing or non-positive. No other card type renders it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(web): manga reader back returns to series (no loop)

The ebook reader's back action defaulted to the chapter's own item
detail (/item/<chapter>), whose back returned to the reader — an
infinite loop for manga chapters. The reader now honors an explicit
backTo search param when present, navigating there instead. Absent for
normal ebooks, so their back behavior is unchanged. Only manga chapter
rows pass backTo, keeping the fix manga-only.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(web): manga chapter row actions (read/mark-read/download)

Each manga chapter/volume row now offers Read (the existing reader link,
now carrying a backTo to the series), Mark-read (the shared watched-state
mutation per chapter content_id), and Download (lazily fetches the
chapter's file versions on demand and opens the shared
DownloadVersionPicker, gated on user.download_allowed). The
volume-unit / loose-chapter / section structure from buildMangaList is
unchanged. Scoped to MangaContent only; EbookContent is untouched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(web): validate reader backTo param is a safe in-app relative path

Prevents open-redirect / javascript:-URI XSS from a crafted ?backTo= URL.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(catalog): include per-chapter read state in manga detail

Manga chapters are ebook items, so a chapter is "read" when the viewer's
ebook_reader_progress row crosses the finished threshold. fetchMangaChapters
now LEFT JOINs that table scoped to the AccessFilter's user_id/profile_id and
exposes a per-chapter Read bool on MangaChapter, threaded through
buildMangaExtension. The detail payload previously carried no read state, so
the row toggle always started unread.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(web): manga rows reflect read state on load

MangaChapter now carries an optional read flag from the detail payload, and
MangaRow seeds its mark-read toggle from chapter.read instead of always
starting unread. The optimistic toggle + shared watched mutation are
unchanged; only the initial value is seeded.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(sections): exclude manga chapters from recently-added/released/random + other library-listing sections

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(sections): manga recently-added/released cards show the latest volume's cover

* fix(manga): keep enrichment honest about no-match vs enriched, batch 50->200

- sweep stats now separate enriched / no_match / failed: a stamped no-match
  was counted (and logged) as an enrichment, which masked a collapse of the
  real match rate during the backfill
- batch size 50 -> 200 (SILO_MANGA_ENRICH_BATCH overrides): with the plugin
  serving GetMetadata from its search cache an item costs one rate-limited
  AniList request, so a sweep still fits the 5-minute task interval

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(manga): size enrich batch to the 5-minute interval at AniList's real budget

140 items x ~2.1s/request fits the interval; an overlong sweep makes the task
manager drop the next trigger and the effective rate falls below the AniList
budget.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(catalog): manga count chip data missing from library browse

manga_chapter_count/manga_volume_count were only added to BrowseRepository,
but /library/{id}?tab=library flows through previewQuerySource ->
QueryExecutor.PreviewPage, which selects qualifiedListItemColumns and scans
with scanItems - so manga cards never carried the counts and the Vols/Ch
poster chip stayed hidden.

Append mangaCountColumns to the preview-page SELECT and scan them via a new
scanItemsWithMangaCounts (nil for non-manga rows, mirroring scanBrowseItems).
Extract listItemScanDests so the three scan variants share one destination
list instead of duplicating the 48-column scan.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(web): manga chip reads 'X Volumes · X Chapters', menu verbs say Read

- chip: show distinct-volume and loose-chapter counts side by side instead
  of the single 'Vols N'/'Ch N' heuristic; mangaCountColumns now counts
  DISTINCT volume tokens (rows sharing a volume are one volume) and only
  un-volumed rows as chapters
- watched-state labels: type='manga' fell through to the video default, so
  the card dot menu and detail page said 'Mark Watched' - manga now uses
  the ebook reading verbs (Mark Read / Mark Unread, 'Marked as read' toast)
- format MangaContent.test.tsx (pre-existing prettier miss)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(manga): backdrop enrichment - banner hero art + backdrop-only backfill

- cache remote backdrops like posters (cacheRemoteImages generalizes the
  poster-only path; failures keep the provider URL, which still renders)
- claim arm for enriched items missing a backdrop: fetched by stored
  provider ID (search skipped - no rate spend, no re-match risk) and only
  the backdrop is written; stamping after the attempt keeps banner-less
  series from being re-claimed every sweep
- backfill = one-time SQL clearing last_refreshed for poster-set/
  backdrop-empty manga

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(manga): reading-loop UX - continue CTA, next chapter, series-aware cards, file details

Fixes the four high-priority findings from the manga UX review plus a
file-inspector request:

- H1: series hero gets a Continue / Start Reading / Read Again CTA
  targeting the first unread chapter (firstUnreadChapter over the ordered
  list), plus an overflow menu (View Details, admin Refresh Metadata)
- H2: the reader resolves its owning manga series (chapter detail now
  carries series_id/series_title) and offers next-chapter navigation: a
  header next button and an end-of-book floating CTA at >=99.5% progress;
  back defaults to the series even without a backTo param
- H3: chapter rows show a persistent read check + muted title, and the
  mark-read mutation carries series_id so the series detail cache
  invalidates (read states no longer revert on revisit)
- H4: continue-reading cards for manga chapters present the series:
  sections payload resolves chapter->series linkage, the card heading/image
  link to the series, and meta lines launch the reader
- View Details: manga series menus (card dot menu + detail overflow) open
  a file inspector showing folder paths and per-chapter file names/sizes
  via GET /catalog/items/{id}/manga-files; paths are stripped for viewers
  without file-path visibility (item-versions policy)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(manga): UX mediums - richer detail page, smarter list, manga sort scope

Second batch from the manga UX review (M1-M7):

- M1: multi-chapter volume sections are collapsible (fully read sections
  start collapsed) with sticky headers, and long series get a 'Jump to
  <next unread>' anchor above the list
- M2: the series hero shows the author line (HeroCrewLine learns Author
  credits with person links; DetailHero now renders crewLine and genre
  chips independently) and Volumes/Chapters badges
- M3: browse-card count chip abbreviates to '12 Vol - 3 Ch' so it fits
  narrow cards without occluding covers
- M4: manga gets its own sort scope: Duration/Bitrate (meaningless for
  file-less series rows) disappear, reading labels (Date Read / Reads)
  apply, Author stays
- M5: global search labels manga results 'Manga' instead of the raw type
- M6: chapters carry the viewer's reading fraction; part-read rows show an
  inline progress bar + percent
- M7: chapter rows show the extracted cover thumbnail (presigned
  poster_url on the chapters payload) instead of a generic icon

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(manga): UX lows - volume token dedupe, comic reader chrome, empty-state hint

- buildMangaList buckets volumes by canonical numeric token so mixed
  release naming (v01 + 1) yields one Volume 1 instead of duplicates
- cbz/cbr readers start with the side panel closed and hide prose-only
  chrome (reading ruler, TTS, typography/font controls, hyphenation,
  writing mode) while keeping comic-relevant settings (theme, brightness,
  margin, right-to-left, spread, flow)
- manga empty state mentions chapters appear after the library scan

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(manga): publication status badge via new SDK status field

- vendor the unpublished plugin SDK (adds MetadataItem.status) under
  internal/compat/ with a relative go.mod replace, following the
  zishang520-webtransport-go convention; swap to the published module
  before the upstream PR
- map plugin status into MetadataResult.ShowStatus, persist it during
  manga enrichment, and show it as the hero status badge (show_status was
  already on the detail payload and MetadataBadges)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(manga): generalize backdrop pass to secondary fields (backdrop + status)

The backdrop-only claim arm becomes a secondary-fields pass: enriched items
missing a backdrop and/or publication status are claimed, fetched by stored
provider ID, and only the missing secondary fields are written. Lets the
new status field backfill across the already-enriched library instead of
applying only to future enrichments.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(metadata): merge ShowStatus through MergeMetadata/MergeGlobalMetadata

The new MetadataResult.ShowStatus never reached the accumulated result the
manga enricher persists from - the field-by-field merges didn't know it, so
the status backfill pass obtained nothing. Regression-tested on both paths.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(manga): keep scanner identity IDs out of the metadata flow

filterMangaProviderIDs passed the scanner's manga_series identity row
through, so the search-skip-when-already-matched guard saw provider IDs on
every item and never searched: unmatched items went straight to a by-ID
fetch with no usable ID and were stamped as terminal no-match without a
single provider request (and the MangaDex fallback was never consulted).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: gitignore docker-compose.override.yml (local deployment override)

The override unpublishes the bundled redis/postgres host ports
(ports: !override []). It is a per-deployment, local-only file: ignoring it
keeps a rebase from main and git clean -fd from disturbing it, and keeps it
out of any PR. Its accidental absence once exposed Redis to the internet.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(manga): code-review fixes — no-match guard, sort comparator, volume-count consistency

- enrichWithProviders: set accumulator.HasMetadata after a provider result
  merges (MergeMetadata doesn't propagate it). Without this, a confident
  match carrying only genres/authors/status/year but no cover and no overview
  failed the no-match check and was discarded + terminally stamped.
- byChapterIndex: both un-indexed chapters yield POSITIVE_INFINITY, so the
  subtraction was Infinity-Infinity=NaN (Array.sort treats NaN as 0, leaving
  order undefined). Compare explicitly for a stable order.
- MangaContent volume/chapter badges: derive counts from the rendered
  buildMangaList entries (which canonicalize v01 ≡ 1) instead of raw distinct
  volume tokens, so the badge can no longer say '2 Volumes' over one row.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(manga): clarify the enrichment claim's secondary arm is admin-reset-only

The secondary arm (poster present, backdrop/status missing) requires
last_refreshed IS NULL, so it is only reachable when an operator resets
last_refreshed to backfill a newly-added field — not an automatic periodic
re-check (which would re-fetch banner-less series every sweep). Documents the
intent so it does not read as dead code.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(manga): collapse continue-reading chapters per series; batch provider-id lookup

- Continue Reading now collapses multiple in-progress chapters of the same
  manga into one card (most recently read kept), mirroring the episode→series
  collapse. The reading section resolves chapter→series linkage into itemMeta
  (applyMangaChapterSeriesMeta) and runs the shared
  collapseContinueWatchingSeriesCandidates, which the reading path previously
  skipped.
- claimBatch resolves provider IDs for the whole batch in one query via the
  new ProviderIDRepository.GetByContentIDs (content_id = ANY), replacing the
  per-item GetByContentID N+1.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(web): manga publication-status chip on browse cards + more legible chips

- Color-coded publication status pill (Ongoing/Completed/Hiatus/Cancelled/
  Upcoming) in the manga card's top-left corner, mirroring the vol/chapter
  count chip top-right. Strictly manga-gated; show_status was already on the
  browse payload.
- New .glass-chip (78% surface vs glass-subtle's 40%) for the manga count +
  status pills so the labels stay legible over busy cover art.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* build(manga): depend on published silo-plugin-sdk v0.7.0

Replace the vendored internal/compat/silo-plugin-sdk copy with a normal
dependency on the published SDK module at v0.7.0, which adds
MetadataItem.status (publication/airing status) consumed by the manga
status badge at internal/metadata/plugin_provider.go.

- go.mod: pin v0.7.0, drop the local-path replace directive
- remove the vendored internal/compat/silo-plugin-sdk tree
- Dockerfile: drop the vendored-SDK COPY
- strip the manga design docs/plans from docs/superpowers (internal)

Requires Silo-Server/silo-plugin-sdk#4 merged and tagged v0.7.0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(manga): exclude chapters from the matcher's unmatched-item lister

Manga chapters are type='ebook' items that stay status='pending' by
design - provider metadata lives on the type='manga' series item. The
scan-final RetryUnmatchedItemsByFolderAndPathPrefix listed all of them
and ran a rate-limited ebook-plugin search per chapter: 31,564 chapters
x ~1s = 8h46m appended to a 2-minute manga library scan (observed
live), every one a guaranteed no-match. Earlier runs never survived to
completion, so the library's last_scanned_at stayed NULL forever.

Add the same manga_chapters NOT EXISTS guard the ebook enricher's
claim query already uses. Verified live: the same library now scans in
27s with retried_items=0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(scanner): never probe-repair ebook/comic files (ebook+manga detail-page killer)

NeedsCriticalProbeRepair was always true for BaseType 'ebook' files (epub, pdf,
cbz, cbr — incl. manga chapters): buildEbookMediaFile leaves ProbeUpdatedAt nil
and they have no audio/video, so probeEnsurer.Ensure spawned ffprobe per file on
every detail/watch load and never converged (ffprobe errors on zip/rar, result
never persisted). Short-circuit probe-repair for ebook base type — they're read
directly and never use the transcode/playback probe pipeline.

SHARED fix: benefits both the ebooks and manga library types.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf+fix(ebooks): parallelize detail extension + preserve finished read-state

- buildEbookExtension ran its 3 related-content queries (series, also-by-author,
  similar) sequentially; run them concurrently like buildAudiobookExtension so
  ebook detail latency is the slowest query, not their sum.
- PGEbookReaderProgressStore.Upsert did an unconditional SET progress=EXCLUDED;
  a routine autosave (e.g. reopening a finished book) could drop it below the
  0.9 finished threshold and silently un-mark it read (and clear the manga
  chapter checkmark, which rides on the same row). Guard: once finished,
  progress only moves on an explicit unread (row delete); below threshold it
  tracks freely.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf(manga): batch chapter presign, index volume counts, quiet scan log

- fetchMangaChapters presigned each chapter poster individually; a long-running
  series has hundreds of chapters. Batch them in one PresignImageURLs call, and
  add the missing rows.Err() check (was silently returning partial lists).
- The browse manga count chip's count(DISTINCT volume) subquery wasn't covered
  by manga_chapters_series (series_content_id, chapter_index); add
  idx_manga_chapters_series_volume (series_content_id, volume) so both count
  subqueries are index-only.
- Downgrade the per-chapter "manga scan: indexed" log from Info to Debug (one
  line per .cbz; the 500-file progress log already covers operator visibility).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(manga): address PR #138 code-review findings

Folds PR #142 into the manga branch (already done via fast-forward) and
remediates the issues surfaced in the #138 code review.

Correctness:
- Preserve the scanner's manga_series identity anchor through enrichment.
  ReplaceByContentID's DELETE was unconditional, so the first successful
  enrichment wiped the manga_series provider-id row the scanner relies on
  for idempotency, causing duplicate series + metadata loss on the next
  scan. excludedProviderIDs now also means "not deleted", and the DELETE
  preserves those rows. (internal/catalog/provider_id_repo.go)
- Fall back to the series cover when the latest chapter has no poster.
  Poster columns default to '' (not NULL), so the manga series-card poster
  override blanked cards via a plain COALESCE; wrap operands in NULLIF.
  (internal/sections/fetcher.go)
- Keep backTo a real query param on reader links when libraryId is absent.
  It was string-concatenated with '&', producing a malformed URL on
  deep-links; route it through the query helper instead.
  (web/src/lib/mediaNavigation.ts, EbookReader.tsx, MangaContent.tsx)

Quality:
- Hide manga chapters from favorites/watchlist browse, matching the
  exclusion enforced on every other listing surface.
  (internal/catalog/favorites_browse.go)
- Centralize the manga chapter exclusion predicate into a single exported
  catalog.MangaChapterExclusionWhere, removing four duplicated copies.
  (catalog, sections, ebooks)
- Skip the two manga count subqueries on browse scopes that cannot contain
  manga (non-manga type filters), substituting NULL placeholders.
  (internal/catalog/browse.go)
- Normalize provider publication status (AniList/MangaDex/SDK variants)
  into a stable label set so show_status carries one manga value-domain.
  (internal/manga/enrichment.go)

Adds unit tests for the poster NULLIF contract, browse gating, and status
normalization.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore: regenerate go.sum after rebase onto main

Drops stale silo-plugin-sdk v0.6.0 and other leftover hashes from the
intermediate rebased states; go.mod is now on the published v0.7.0 tag.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(scanner): adapt manga scan to ebookFileShouldSkip 3-value signature

main changed ebookFileShouldSkip to also return the existing content ID;
the manga scan path only needs the unchanged flag, so discard the new
return. Resolves a silent semantic conflict from the rebase onto main.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Silo Server Developer <warmasterx555@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 20:13:10 -04:00
63d608f6fc fix(progress): filter continue-watching by viewer access scope (#167)
* fix(progress): filter continue-watching by viewer access scope

The continue-watching list (GET /progress) only filtered by library when an
explicit library_id query param was passed. The global call passes none, so a
restricted profile received progress rows for items outside its scope (e.g. an
XXX title, or a title above the profile's content-rating cap). The web client
then fans out a per-item GET /catalog/items/{id} detail fetch for each row, and
the inaccessible ones return 404 — surfacing as a dead Continue Watching tile
and stray 404s.

Always apply the viewer's access scope to the progress list. Adds
LibraryItemRepository.FilterAccessibleContentIDs, a batched, episode-aware
mirror of the detail endpoint's access predicate (library membership +
content-rating ceiling), and wires it into HandleListProgress for restricted
profiles only (unrestricted viewers are unaffected). ExcludedMediaTypes is
omitted intentionally: the viewer access.Scope does not carry it and the
request path never sets it.

* fix(progress): gate continue-watching episode access on the parent series

FilterAccessibleContentIDs keyed episode access off episode_libraries and
required a media_item_libraries membership even for rating-only viewers, both
of which diverge from the detail/watch path that masks inaccessible items
(DetailService.GetItemDetail → EnsureAccessible(episode.SeriesID)). For shows
whose episodes span multiple library folders this reintroduced the dead tile /
out-of-scope leak this filter exists to prevent, and rating-only profiles
could lose membership-less items the detail endpoint still serves.

Rewrite the predicate to mirror EnsureAccessible exactly: base the lookup on
media_items, join media_item_libraries only when the viewer is
library-restricted, and resolve episodes through their parent series. Extract a
pure buildFilterAccessibleContentIDsSQL so the query shape (placeholder
numbering, the parent-series join, the optional rating predicate) is
unit-tested without a database, and de-duplicate the two progress filter
helpers via progressContentIDs/keepAccessibleEntries.

AI-use disclosure: implemented with AI assistance (Claude).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Quick <31828688+Quick104@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 18:32:52 -04:00
d3v1l1989andGitHub 9d00ce2089 fix(web): start autoplay countdown when entering post-roll early (#162)
When a series episode reaches 30s before its end, the player enters
post-roll and pauses the underlying video to stop HLS from buffering the
tail segment. A paused video never fires the `ended` event, but the
PlayingNextScreen countdown was gated on `postRollVideoEnded`, which only
became true on `ended`. The result: the "Up Next" overlay appears, the
video is paused, and the countdown never starts — it only resumes if the
user manually plays the video (letting it reach the end and fire `ended`).

Arm the countdown gate at the moment we enter post-roll early, since that
is the point at which we've decided playback is over, rather than waiting
for an `ended` event that an intentionally-paused video will never emit.
2026-06-16 18:30:54 -04:00
d3v1l1989andGitHub 4502c7a850 fix(jellycompat): return seasons/episodes when browsing a series via /Items?ParentId= (#160)
* fix(jellycompat): return seasons/episodes when browsing a series via /Items?ParentId=

Clients that browse a show through the generic /Items?ParentId= endpoint with no
IncludeItemTypes (e.g. the VoidTV Android-TV client's getSeasons/getEpisodes) got
the top-level library views instead of the series' seasons or the season's
episodes. HandleItems only honored an item/season ParentId when
IncludeItemTypes=Season was present and otherwise fell through to the
library-views fallback (or a global type browse), so ParentId was effectively
ignored.

Decode a season ParentId into its own parentSeasonID and route item parents:
- season ParentId -> that season's episodes
- series ParentId -> the series' seasons (or its episodes for
  IncludeItemTypes=Episode; an unsatisfiable type filter returns an empty page)

Extract the shared writeSeriesEpisodesResponse used by /Shows/{id}/Episodes and
the new browse paths. The generic /Items browse paths honor StartIndex/Limit
(matching the /Items contract and the sibling seasons listing) while
/Shows/{id}/Episodes keeps its existing whole-season response.

Adds parent_browse_test.go covering series->seasons, season->episodes, the
unsatisfiable-type-filter empty case, paging, and the parser's season/item
ParentId distinction.

* fix(jellycompat): empty result for non-episode type filter on a season ParentId

handleSeasonEpisodeChildren ignored IncludeItemTypes and always returned the
season's episodes. A season's only children are episodes, so a type filter that
excludes Episode (e.g. IncludeItemTypes=Movie) now returns an empty page,
mirroring the series-parent path's handling of unsatisfiable type filters.
Addresses a CodeRabbit review comment.
2026-06-16 18:30:25 -04:00
d3v1l1989andGitHub 623348111c fix(jellycompat): resolve smart-collection BoxSet children at read time (#161)
Browsing into a smart (live-query) collection via /Items?ParentId={boxsetId}
returned 0 children while the BoxSet's ChildCount showed a non-zero count:
handleBoxSetChildren only read materialized library_collection_items via
ListItems, which is empty for smart collections whose membership is a query
evaluated at read time. Confirmed live — smart "Directed by ..." collections
returned TotalRecordCount 0 while materialized collections returned their items.

Branch handleBoxSetChildren on catalog.IsLiveQueryType: smart collections
resolve their members through the query executor (mirroring the web API's
loadLiveCollectionItems — Normalize/Validate/ApplySmartCollectionItemLimit,
library-scope intersection, access-filtered Preview), then feed the resulting
content IDs into the same hydration path as stored collections. A malformed or
invalid query definition degrades to an empty page (never a 500).

Hoist the duplicated intersectCollectionLibraryIDs helper into
catalog.IntersectCollectionLibraryIDs so the web and compat resolvers share one
implementation.

Adds smart_boxset_test.go (smart children resolve in query order, library-scope
intersection reaches the executor, malformed-query and no-executor degrade to
empty).
2026-06-16 18:30:16 -04:00
e084cdd1d6 Add unified literary works for ebooks and audiobooks (#107)
* docs: add literary works design and plan

* feat(literary): add work link schema

* feat(literary): add work domain primitives

* feat(literary): persist work links

* feat(literary): score work matches

* feat(catalog): include literary work summary on item detail

* feat(literary): expose work detail API

* feat(literary): assemble work detail

* feat(literary): add admin work linking primitives

* feat(catalog): group literary items by work

* feat(literary): auto-link works during book scans

* fix(literary): narrow work match candidates

* fix(literary): address work merge blockers

---------

Co-authored-by: rxwatcher <rxwatcher@users.noreply.github.com>
Co-authored-by: Quick <31828688+Quick104@users.noreply.github.com>
2026-06-16 18:17:08 -04:00
CoffeeKnyteandGitHub 9cd00a877b fix(metadata): cap un-completable episode refresh debt as terminal (#168)
Episodes a provider genuinely has no data for stay actionable forever
and, via syncRefreshDebtForItem, pin the whole series item at
RefreshDebtReasonEpisodeIncomplete -> priority 300 (the top band). Those
debt rows never drain and always cut the queue ahead of legitimately-due
content, every backoff interval, indefinitely.

Classify episode-incomplete debt as terminal once its persistent
attempt_count reaches a small cap (3, ~5 days under the existing stepped
backoff): demote it off the priority-300 band (to a still-fixable
reason's band if one is present, else a terminal floor) and re-check it
rarely (90d) instead of every 30d. The signal is purely attempt_count on
the existing row -- no schema change. Demote rather than delete: deleting
would let the next series pass recreate the row at attempt_count 0 and
restart the cycle.

Giving up fast is safe: the library "refresh incomplete" and per-item
refresh endpoints re-fetch on demand and bypass the backoff, and
media_items.episode_metadata_incomplete is left untouched so they still
find these items. Logs the terminal transition (no silent demotion).

AI-use disclosure: implemented with AI assistance (Claude Code).
2026-06-16 18:16:33 -04:00
bf54f040f9 fix(audiobooks): extract scan covers and harden dedupe (#103)
* fix(audiobooks): extract scan covers and harden dedupe

* fix(audiobooks): address scan cover review

---------

Co-authored-by: rxwatcher <rxwatcher@users.noreply.github.com>
Co-authored-by: Quick <31828688+Quick104@users.noreply.github.com>
2026-06-16 17:58:01 -04:00
d3v1l1989andGitHub fe7eb70483 fix(jellycompat): reuse the play session for Static direct play instead of leaking one per request (#165)
Clients that direct-play via /Videos/{id}/stream?Static=true never call
PlaybackInfo, so they send their own client-generated PlaySessionId and repeat
it on every range request. resolvePlaybackRoute looked that id up, missed (the
server never minted it), and returned ErrSessionNotFound — so HandleVideoStream
fell to createStaticPlaySession and started a *new* upstream session for every
request. Each counts toward the per-user max_streams cap and only ages out after
the 45s activity grace, so a single direct play's range requests pile up orphaned
sessions and quickly trip the cap -> 429 TooManyStreams, locking the user out of
their own playback.

When the provided PlaySessionId is unknown (or owned by another caller), fall
through to the existing CompatToken-scoped FindByRoute reuse instead of erroring,
so all of a direct play's requests share one session. Reuse stays scoped to the
caller's own token, so a guessed/foreign id cannot bind another user's session.
Adds a test asserting StartSession runs once across repeated Static requests.
2026-06-16 17:51:42 -04:00
d3v1l1989andGitHub c81de3459e feat(jellycompat): add Filters2, LocalTrailers, UserImage, ClientLog and Sessions endpoints (#164)
* feat(jellycompat): add Filters2, LocalTrailers, UserImage and ClientLog endpoints

Four endpoints that real Jellyfin clients call were unregistered and fell
through to chi's default 404 (or, for Filters2, were swallowed by /Items/{id}).
All are additive and contract-faithful to the Jellyfin C# server:

- GET /Items/Filters2 -> 200 QueryFilters v2 shape (Genres NameGuidPair[],
  Tags, Audio/SubtitleLanguages), empty arrays. Fladder's filter UI 404'd before.
- GET /Items/{id}/LocalTrailers (+ /Users/{userId}/... alias) -> 200 bare
  BaseItemDto[] ([]); Silo indexes no local trailers. Infuse/Moonfin hit this
  on every item-detail load.
- GET|HEAD /UserImage?userId= -> the same anonymous palette avatar as the
  legacy /Users/{id}/Images/Primary route; HandleUserImage now reads the id
  from the query param when the path segment is absent (modern Jellyfin route).
- POST /ClientLog/Document -> 200 {FileName} after draining/discarding the
  body (Silo has no client-log store); 413 over 1 MiB, matching MaxDocumentSize.

Stops recurring 404 noise and lets clients that depend on these (filter sheets,
avatars, crash-log upload) work. Adds handler unit tests for each.

* feat(jellycompat): add GET /Sessions returning a contract-shaped session list

Wholphin and other jellyfin-sdk clients poll GET /Sessions (optionally
?deviceId=) every few seconds during playback; the route was unregistered, so
each poll hit a chi 404 the SDK could not deserialize — a ~289-per-4h 404 storm
in production. Register it under the same [Authorize] group Jellyfin uses and
return a correctly-typed SessionInfoDto[] (currently empty, consistent with the
existing compat stub handlers). This stops the storm and lets clients degrade
cleanly; populating live session/now-playing state from the playback store is a
follow-up.

* refactor(jellycompat): match Jellyfin client-log size limit exactly

Use 1,000,000 bytes (Jellyfin's ClientLogController.MaxDocumentSize, decimal)
instead of 1<<20, and fix the comment that wrongly called it 1 MiB. Behavior is
functionally identical (the body is discarded); this is contract-fidelity only.
Review follow-up.

* test(jellycompat): add router-level coverage for the new endpoints

The per-handler tests call handlers directly and never exercise NewRouter, so
route registration, chi static-vs-{id} ordering, and auth-group placement were
untested — the one thing this change is actually about. Add a full
NewRouter/ServeHTTP test asserting the session-auth-group routes (Filters2,
LocalTrailers x2, Sessions, ClientLog/Document) return 401 unauthenticated
(registered + behind auth, not 404 or accidentally anonymous), /UserImage serves
its anonymous palette avatar, and an authenticated Filters2 reaches the v2
filters handler (not shadowed by /Items/{id}) with /Sessions returning [].
2026-06-16 17:51:00 -04:00
d3v1l1989andGitHub 3f3cf55002 fix(jellycompat): return 404 for HLS segments of a failed transcode (#163)
HandleHLSSegment mapped every non-ErrSegmentNotFound error from the
segment-retrieval/recovery path to a generic 500 "Failed to load segment".
When a transcode process starts and then exits non-zero, WaitForSegment
returns a wrapped playback.ErrTranscodeFailed, which fell through to that
500 — observed in production as repeated 500s on seg_00000.ts that drove an
8x client retry storm and crash-log uploads.

The segment will never materialize once its transcode has died, so this is a
not-found condition: Jellyfin's DynamicHls handler falls through to a
PhysicalFileResult for the absent file, which ASP.NET serves as 404, never
500. Map ErrTranscodeFailed to 404 alongside ErrSegmentNotFound via a small
extracted hlsSegmentErrorResponse helper, reserving 500 for genuinely
unexpected errors. Adds a unit test pinning the mapping.
2026-06-16 11:14:14 -04:00
d3v1l1989andGitHub 190223e030 fix(jellycompat): serve item and user images anonymously to match Jellyfin (#158) 2026-06-15 18:18:10 -04:00
ea6d1d1f00 feat(jellycompat): support DELETE /Videos/ActiveEncodings transcode teardown (#159)
* feat(jellycompat): support DELETE /Videos/ActiveEncodings transcode teardown

* fix(jellycompat): guard ActiveEncodings teardown against not-yet-started sessions

HandleDeleteActiveEncodings omitted the UpstreamSessionID == "" guard that
the sibling Stopped-report path uses. A PlaybackSession is created by
PlaybackInfo with an empty UpstreamSessionID; it is only populated once the
first manifest request reaches ensureUpstreamPlayback. A DELETE
/Videos/ActiveEncodings arriving in that window (with a matching token) passed
the ownership guard and ran teardownPlaySession, which deletes the compat play
session from the store. A subsequent HandleMasterManifest then 404s, and the
teardown was not the "no-op-safe" operation its doc comment claims.

Mirror the Stopped report path by treating an unknown, not-owned, OR
not-yet-started PlaySessionId as a uniform idempotent 204 no-op. Add a
regression test that fails if the guard is removed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Quick <31828688+Quick104@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 18:17:48 -04:00
Quick ae65678ba4 fix(jellycompat): populate episode season parent metadata 2026-06-15 18:10:40 -04:00
d3v1l1989andGitHub 2e422b66df fix(jellycompat): match the Static stream query parameter case-insensitively (#157) 2026-06-15 18:03:58 -04:00
Quick 10a0635331 Increase default migration timeout
- Raise the default migration timeout from 5m to 20m
- Document SILO_MIGRATE_TIMEOUT in the example environment
2026-06-15 16:20:50 -04:00
453 changed files with 45815 additions and 3849 deletions
+264
View File
@@ -0,0 +1,264 @@
---
name: issue-to-pr
disable-model-invocation: true
description: >-
End-to-end implement a GitHub issue and open a ready pull request. Invoke
explicitly as `/issue-to-pr <number|#ref|url>` (this is a side-effecting
workflow — it creates branches and opens PRs — so it does not auto-trigger).
It creates an isolated git worktree off the latest main, reproduces the bug
(or scopes the feature), finds the root cause, implements the smallest correct
change following repo conventions, verifies it (build/lint/tests), runs a Codex
adversarial review and addresses the material findings, then commits and opens
a ready PR. Not for pure triage (use triage-issue), pure review of an existing
PR (use review-pr-or-mr / code-review), or analysis without code changes.
---
# Issue → PR
Turn a single GitHub issue into a verified, reviewed, ready-to-merge pull
request — autonomously — while staying honest about uncertainty and stopping
when a human decision is genuinely required.
The goal is not "produce a diff." It is: **confirm the real problem exists,
make the smallest change that correctly solves it, prove it works by execution,
have an adversary try to break it, and only then ship a PR.** Speed comes from
doing these in order, not from skipping them.
## Inputs
- The issue reference is passed as `$ARGUMENTS` — a number (`123`), a `#123`
reference, or a full issue URL. If none is given, ask; do not guess.
- The repo is the current git repository; PRs target `origin`
(`Silo-Server/silo-server`) and base `main`.
## Operating principles
- **Default to "maybe nothing needs fixing."** Coding agents "fix" already-correct
or stale code in a large fraction of cases because they don't stop to confirm the
problem is real. Confirm the issue reproduces on current `main` before changing
anything — it may already be fixed.
- **Root cause over symptom.** A fix you can't explain is a fix you can't trust.
- **Verify by execution, not by reading.** Build/lint/tests are the primary
correctness signal. The adversarial review is a *second* layer on top — never a
substitute for green checks. If you can't verify it, don't ship it.
- **Smallest correct change.** Match surrounding code's idioms, naming, and comment
density. Put code in the package that owns the behavior (CLAUDE.md); don't create
catch-all helpers or duplicate logic — search for an existing helper before
writing a new one, and extract shared logic instead of copying.
- **Don't trust recall for APIs.** Verify every symbol, function, and signature you
call actually exists by grepping the codebase — agents hallucinate plausible-looking
APIs.
- **Don't gold-plate.** A reviewer told to find gaps will always find some. Address
what affects correctness or the issue's stated requirements; treat the rest as
optional. Extra abstraction "just in case" is a defect, not diligence.
- **Know when to stop.** See [Stop and ask](#stop-and-ask). Autonomy is for
execution, not for inventing product decisions or shipping unverifiable guesses.
Track the phases below with a task list (TaskCreate/TaskUpdate) so progress is
visible and nothing is skipped on long runs.
## Workflow
### 1. Understand the issue
Read everything before writing anything.
```bash
gh issue view <N> --json number,title,body,labels,state,author,url,comments
```
Determine: **bug vs. feature** (labels + body → branch prefix `fix/` vs `feat/`);
**acceptance criteria** (what does "done" mean?); **linked scope** (does it serve
an epic / sub-issue to reference as `Part of #NNN`?); and **client surface** (does
it touch API/auth/playback/session/library/metadata behavior consumed by
`silo-android` / `silo-apple`? — if so, coordinated client follow-up may be needed).
If the issue is vague, underspecified, or needs a product call, stop and ask now —
before spending a worktree on the wrong problem.
### 2. Create a worktree off the latest main
Work in isolation so the user's checkout is untouched. Create it at the **main
repository root**, not inside the current worktree.
```bash
# Resolve the primary repo root (works from inside any worktree)
MAIN_ROOT="$(cd "$(dirname "$(git rev-parse --git-common-dir)")" && pwd)"
git -C "$MAIN_ROOT" fetch origin # always branch off fresh main
SLUG=<short-kebab-slug-from-issue-title>
BRANCH=fix/issue-<N>-$SLUG # use feat/ for features
WT="$MAIN_ROOT/.claude/worktrees/issue-<N>"
git -C "$MAIN_ROOT" worktree add -b "$BRANCH" "$WT" origin/main
cd "$WT"
```
If the branch/worktree already exists from a prior run, reuse it (cd in) rather
than failing — but confirm it's based on current `origin/main`; if it's stale, tell
the user before continuing. From here, **all work happens in `$WT`** (the Bash tool
keeps the working directory between calls); use absolute paths when in doubt.
### 3. Reproduce / confirm — a hard gate
This is the highest-leverage step. Do not skip it.
- **Bug:** reproduce it concretely — ideally a **test that fails on current `main`**
(red), or a precise script/trace through the faulty path. Pin down the exact code
path and the invariant it violates.
- **If you cannot reproduce it, STOP and report** — do not patch. The issue may
already be fixed, be environment-specific, or be a misunderstanding. Shipping a
speculative fix for a non-reproducing bug is the most common way agents make
things worse.
- **Feature:** confirm the capability is actually missing, then map where it fits —
the owning package, patterns to follow, the data flow, the smallest set of files.
**Plan proportionally.** If you could describe the diff in one sentence (a typo, a
log line, a one-line guard), skip formal planning and make the change. Plan first
when the work is multi-file, unfamiliar, or architecturally uncertain — name the
files/interfaces involved, state what's out of scope, and end the plan with the
verification step. For anything large, confirm direction with the user before a big
build-out.
### 4. Implement
Make the change, scoped to this one concern (one concern per PR). Respect the hard
rules:
- **API:** additive-only within `/api/v1` — never rename/remove a response field,
change a field's type, or repurpose a status code. New behavior = new
fields/endpoints; expose capability endpoints for feature detection.
- **Migrations:** new DB changes are Goose SQL migrations via
`make migrate-create NAME=...` (timestamped). Never `goose fix`, never paired
`.up.sql`/`.down.sql`.
- **Maintainability:** extract shared logic instead of duplicating; change existing
code rather than bolting on local workarounds.
For a bug, the failing test from step 3 should now pass.
### 5. Verify — the primary correctness gate
Run the checks and make them genuinely pass. The exact commands and the worktree
quirks (`GOWORK=off`, stubbing `web/dist`, the v2 golangci invocation, the flaky
GPU tests) are in **[references/verification.md](references/verification.md)** —
read it first, because plain `go build ./...` / `make lint` fail inside
`.claude/worktrees`.
At minimum: Go build + the relevant package tests pass; if `web/` changed,
`pnpm run lint` + `pnpm run format:check` + build pass; `make verify-local-paths` is
clean. Run the **full relevant suite**, not just your new test — catch regressions
in code you didn't touch. Show the command output as evidence; don't assert success.
**Never reach green by weakening the checks.** Deleting/skipping/`xfail`-ing tests,
loosening assertions, lowering coverage thresholds, or relaxing lint/CI to make a
red check pass is a blocker, full stop — if a test genuinely must change, that needs
explicit human sign-off, not a quiet edit. If you can't make the checks pass on the
merits, stop and report.
### 6. Adversarial review (Codex)
Commit your work first (so the review sees exactly what will ship), then run a Codex
adversarial review of the branch against `main`. Full invocation, output handling,
the iteration cap, and the fallback are in
**[references/codex-review.md](references/codex-review.md)** — follow it exactly.
The loop, in short:
1. Commit the implementation on the branch.
2. Run the adversarial review (`--base main`, foreground/`--wait`, extended timeout).
3. For each finding: **first confirm it's real and grounded in the actual code**
(reviewers over-flag — don't fix phantom nits). Then **fix** material findings
that affect correctness or the stated requirements (and re-run the review), or
**rebut** them with a defensible, written reason. Note non-material findings as
optional.
4. **Cap the loop at 2 iterations (3 absolute max).** Debugging effectiveness decays
sharply after ~2–3 attempts, so thrashing won't help — if material findings still
stand after the cap, stop and escalate to the user instead of looping.
Do not open the PR while a material, un-rebutted adversarial finding stands.
### 7. Commit, push, open the PR
Conventional Commit subjects scoped to the domain, e.g.
`fix(playback): guard against nil session on reconnect`. End commit messages with:
```
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
```
Push and open a **ready** (non-draft) PR against `main`:
```bash
git push -u origin "$BRANCH"
gh pr create --base main --head "$BRANCH" --title "<conventional subject>" --body "$(cat <<'EOF'
## Problem
<what's broken / missing, grounded in the issue — and how you reproduced it>
## Approach
<what you changed and *why this approach* — the tradeoff, not a file list>
## Verification
<commands run and their result; the failing→passing test; manual checks>
## Adversarial review
<clean, or the material findings and how each was fixed/rebutted>
## Risks & follow-up
<rollout/migration/compatibility risks; any silo-android / silo-apple
follow-up; anything deferred>
Closes #<N>
<add "Part of #<epic>" if the issue serves a capability epic/sub-issue>
## AI-use disclosure
Implemented by Claude Code (issue-to-pr skill) with human oversight.
EOF
)"
```
End the PR body with:
```
🤖 Generated with [Claude Code](https://claude.com/claude-code)
```
For **UI changes**, attach before/after screenshots or a recording (the repo
requires this); if you can't capture them, explicitly flag in the PR that they're
still needed.
### 8. Report back
Give the user the PR URL, a 2–3 line summary, the verification result, the
adversarial-review outcome, any flagged risks or required client follow-up, and the
worktree path so they can inspect it.
## Long or multi-iteration runs
Performance degrades on long sessions, and context compaction can silently drop
constraints. So: keep a short progress note and a "failed approaches" list in the
scratchpad so you don't repeat dead ends; and after any compaction, re-read CLAUDE.md
and re-assert the hard guardrails (additive-only API, migration rules, no weakening
of tests) before continuing.
## Stop and ask
Pause and ask the user (don't push a PR) when:
- The issue is ambiguous, underspecified, or needs a product/UX decision.
- **The bug doesn't reproduce** on current `main`.
- The fix would require a non-additive API change, or a risky/irreversible migration.
- The change balloons into something large or architectural — propose a plan first.
- Checks can't be made to pass on the merits, or material adversarial findings remain
after the iteration cap.
- A test/CI change would be needed to "pass" — get sign-off, don't do it silently.
- The right fix clearly belongs in a sibling repo (`silo-android`, `silo-apple`, a
plugin/SDK repo) — see CLAUDE.md's multi-repo guidance.
In these cases, summarize what you found and what you'd do, and let the user decide.
## Notes
- Authorized to open ready PRs autonomously once checks pass and the adversarial
review is clean — that's the durable instruction from setup. Not authorized to
merge, force-push shared branches, or close issues by hand.
- Never invent a fix you can't verify or explain. An honest "here's what's uncertain"
beats a confident wrong PR.
@@ -0,0 +1,88 @@
# Codex adversarial review
The `/codex:adversarial-review` command is marked `disable-model-invocation: true`,
so you cannot trigger it as a tool/skill. Call the same underlying runtime
directly via Bash: the `codex-companion.mjs` helper, subcommand
`adversarial-review`. It challenges the *approach and design*, not just defects —
it tries to find the strongest reasons the change should not ship.
## 1. Locate the companion script
```bash
CODEX_COMPANION="${CLAUDE_PLUGIN_ROOT:+$CLAUDE_PLUGIN_ROOT/scripts/codex-companion.mjs}"
if [ -z "$CODEX_COMPANION" ] || [ ! -f "$CODEX_COMPANION" ]; then
CODEX_COMPANION="$(find "$HOME/.claude/plugins/marketplaces" "$HOME/.claude/plugins/cache" \
-name codex-companion.mjs 2>/dev/null | head -n1)"
fi
echo "$CODEX_COMPANION"
```
If nothing is found, the Codex plugin isn't installed/set up. Don't silently skip
the review — tell the user, point them at `/codex:setup`, and ask whether to
proceed without it (or fall back to a `feature-dev:code-reviewer` agent pass).
The user asked specifically for the Codex adversarial review, so treat its absence
as a real blocker, not a detail.
## 2. Commit first, then review the branch against main
Review what will actually ship. Commit the implementation on the branch, then
diff against `main`:
```bash
git add -A && git commit -m "<conventional subject>"
```
Run it in the **foreground** so results come back this turn, and set an extended
Bash timeout (reviews can take several minutes) — pass `timeout: 600000` on the
Bash call:
```bash
node "$CODEX_COMPANION" adversarial-review --wait --base main "optional focus text"
```
- `--wait` runs foreground and prints the result to stdout.
- `--base main` reviews the whole branch diff (`main...HEAD`).
- Append focus text to steer it (e.g. the specific failure mode, the risky
concurrency path, the migration). Keep the adversarial framing — don't soften it.
- Do **not** use `--background` here; you need the findings in-hand to act on them.
## 3. Interpret the output
The review returns a verdict plus findings:
- **`approve` / no material findings** → green light for the PR.
- **`needs-attention`** → there is at least one material risk. Each finding names
a file, a line range, an impact, and a concrete recommendation.
LLM reviewers systematically **over-flag** — they report gaps even when the work is
sound. So for every finding, in order:
1. **Confirm it's real.** Open the cited file/lines and verify the finding is
grounded in the actual code and actually reachable. Drop findings that reference
code that doesn't exist, lines outside the diff, or behavior the code doesn't have
— don't fix phantom nits.
2. For a confirmed finding, do one of two things — never silently ignore it:
- **Fix it** if it affects correctness or the issue's stated requirements, then
re-run the review (go back to step 2 after committing the fix); or
- **Rebut it** with a defensible, written reason it doesn't apply, and capture
that reasoning in the PR's "Adversarial review" section.
3. Treat style/naming/speculative/low-value findings as **optional** — don't
gold-plate. The adversarial prompt is told not to emit these, but if any slip
through, note and move on; don't add abstraction or defensive code for cases that
can't happen.
## 4. Loop — but cap it
Repeat implement → verify → commit → review until the review is `approve` or every
material finding is fixed or defensibly rebutted.
**Cap the loop at 2 iterations (3 absolute max).** Debugging effectiveness decays
sharply after ~2–3 attempts — if material findings still stand after the cap, stop
and escalate to the user with a summary rather than thrashing or shipping over a
standing objection. Only open the PR once the review is clean within the cap.
## 5. Record it in the PR
Summarize the outcome in the PR body's **Adversarial review** section: either
"clean — no material findings" or a terse list of what was raised and how each was
fixed or rebutted. This gives reviewers the adversary's perspective for free.
@@ -0,0 +1,77 @@
# Verification inside a worktree
Commands assume the repository root is the cwd (the new worktree, `$WT`).
## Why the obvious commands fail here
A `.claude/worktrees/*` checkout lives under the parent Silo workspace, whose
`go.work` only lists the primary `./silo-server`, not worktree checkouts. So:
- `go env GOWORK` resolves to the parent workspace from anywhere under `SiloServer/`.
- Plain `go build ./...` / `go test ./...` fail with
*"directory prefix . does not contain modules listed in go.work"*.
- `go build ./...` also needs `web/dist` to exist (the embed directive).
- `golangci-lint` is not installed locally.
## Go
Prefix every Go command with `GOWORK=off`.
```bash
# Build the parts that matter without needing web/dist:
GOWORK=off go build ./internal/... ./cmd/...
# Run the tests for the package(s) you touched (fast, targeted):
GOWORK=off go test ./internal/<pkg>/...
# If you need a full ./... build, stub the embed target first:
mkdir -p web/dist && touch web/dist/.gitkeep
GOWORK=off go build ./...
```
Lint Go without installing golangci-lint — must be the **v2** module path (the
repo's `.golangci` config rejects v1):
```bash
GOWORK=off go run github.com/golangci/golangci-lint/v2/cmd/golangci-lint@latest run ./internal/...
```
**Known flake, not a real failure:** `internal/playback` GPU-probe tests
(`TestResolveHWAccel*`, the NVENC smoke test) can fail with `signal: killed`
under parallel load on macOS, but pass in isolation. If they're the only
failures, re-run them alone before treating them as real:
```bash
GOWORK=off go test ./internal/playback/ -run TestResolveHWAccel -count=1
```
## Frontend (only if `web/` changed)
```bash
cd web
pnpm install # if deps changed or node_modules is absent
pnpm run lint
pnpm run format:check
pnpm run build # catches type/build errors before the PR
cd ..
```
The repo requires `pnpm run lint` and `pnpm run format:check` to be clean before
a merge request.
## Repo-wide
```bash
make verify-local-paths # must be clean; CLAUDE.md forbids absolute paths in docs/specs
```
## Migrations
If the change needs schema work, create a timestamped Goose migration — never
hand-number, never `goose fix`, never paired up/down files:
```bash
make migrate-create NAME=add_thing
make migrate-validate
make migrate-status
```
+1
View File
@@ -76,3 +76,4 @@ POSTGRES_SHM_SIZE=8gb
# Optional server mode override for source runs or standalone worker nodes.
# MODE=integrated
#SILO_MIGRATE_TIMEOUT=60m
+11 -1
View File
@@ -50,7 +50,11 @@ docs/inspiration
.agents/
/.claude/
# Ignore everything in .claude/ except committed, shared skills.
/.claude/*
!/.claude/skills/
/.claude/skills/*
!/.claude/skills/issue-to-pr/
.cursor/
.superpowers/
.codex/
@@ -59,5 +63,11 @@ node_modules/
logs/
.playwright-mcp/
demo/
docker-compose.dev.yml
# Local-only deployment override (unpublishes bundled redis/postgres host
# ports via `ports: !override []`). Kept out of git so a rebase from main
# never disturbs it and it never lands in a PR; its absence once exposed
# Redis to the internet, so it must persist on the box.
docker-compose.override.yml
.playwright-cli/
output/
+1 -7
View File
@@ -91,10 +91,6 @@ Recent history follows Conventional Commit-style subjects such as `feat(playback
## Silo v1 Process (scope lock & API rules)
Silo is working toward a locked v1 feature set so client teams can plan against a stable core. These rules apply to every change.
**Scope gate.** Before implementing a new user-facing capability, check `docs/architecture/v1-scope.md`. If the capability is not listed there — or the file says scope is not yet locked — do NOT open a feature PR. File a **v1 capability proposal** issue (template in this repo) and wait for triage. Bug fixes, refactors, and work on already-locked capabilities proceed normally.
**API rules — additive-only within `/api/v1`.**
- Never rename or remove a response field, change a field's type, or repurpose a status code on an existing endpoint.
- New functionality adds new fields/endpoints; removals go through the existing Deprecation/Sunset header flow only.
@@ -103,6 +99,4 @@ Silo is working toward a locked v1 feature set so client teams can plan against
**PR requirements.**
- Link the capability epic or sub-issue the PR serves (`Part of #NNN`). PRs with no linked scope item will be questioned at review.
- One concern per PR; Conventional Commit subject; AI-use disclosure in the PR body.
**Pre-push checklist.** `make lint` · `cd web && pnpm run lint && pnpm run format:check` · `make verify-local-paths` · Go tests in a libvips-capable container (a bare-host `go test ./...` silently skips CGO packages, including `internal/api/handlers`).
- One concern per PR; Conventional Commit subject; AI-use disclosure in the PR body.
+10 -2
View File
@@ -252,9 +252,17 @@ cd web && bun run lint
cd web && bun run format:check
```
## License
## License & Trademarks
Silo is licensed under `AGPL-3.0-or-later`. See [LICENSE](LICENSE).
Silo's source code is licensed under the **GNU Affero General Public License
v3.0 or later** (`AGPL-3.0-or-later`) — see [LICENSE](LICENSE).
The **Silo name, logo, and wordmark are trademarks of Silo Media L.L.C.** and
are **not** covered by the AGPL. You're free to fork and redistribute the code,
but forks and redistributions must not use the Silo brand as their identity and
must remove or replace the brand assets. Publishing a Silo-branded app to an app
store requires written permission. See [TRADEMARK.md](TRADEMARK.md) for what's
permitted — including referential use like "compatible with Silo."
## Configuration
+85
View File
@@ -0,0 +1,85 @@
# Silo Trademark & Brand Policy
Silo's source code is free and open source under the **GNU Affero General Public
License v3.0 or later** (`AGPL-3.0-or-later`). You are welcome to use, study,
modify, and redistribute it under those terms.
**That freedom applies to the code — not to the Silo name and brand.** The Silo
trademarks identify the official project and its official builds. This policy
explains how they may and may not be used. It exists for a simple reason:
anyone should be free to fork the software, while users can still trust that
something called "Silo" actually comes from us.
## Marks covered
The following are trademarks of **Silo Media L.L.C.** ("Silo Media"), whether or
not registered:
- the names **Silo** and **Silo Server**
- the **Silo logo** and **wordmark**
- Silo **product and component names**, including **SiloRemote**, **SiloMac**,
and other "Silo…" names
- confusingly similar variations of the above
The AGPL license over the source code **does not grant any license to these
marks.** This reservation is expressly permitted by AGPL-3.0 §7(e).
## Uses that are always fine — no permission needed
- **Referring to Silo truthfully** — "compatible with Silo," "a plugin for
Silo," "built on Silo," "a fork of Silo." Use the word to refer to *our*
project, not to brand *yours*.
- **Redistributing official, unmodified builds** under the Silo name, with all
notices kept intact.
- **Running, modifying, and self-hosting** Silo for yourself or your
organization.
- **Editorial and educational** use — articles, tutorials, talks, reviews.
## Uses that require our written permission
- Distributing a **modified version under the Silo name** (see *Forks* below).
- **Publishing an app to any app store** — Apple App Store, Google Play,
F-Droid, or similar — under the Silo name, logo, or a confusingly similar
identity.
- Using the marks in a **company, product, domain, or app name**, or in any way
that implies Silo Media operates or endorses your offering.
- **Merchandise** or other commercial use of the logo or wordmark.
## Forks must rebrand
You may fork Silo — that is your right under the AGPL. But a **distributed** fork
must not use the Silo name or logo as its identity. Pick your own name and logo,
remove the Silo brand assets (below), and you may state factually that your
project "is based on Silo" or "is a fork of Silo." This is the same model used by
Mozilla, Rust, and many other open-source projects: **open code, protected
identity.**
## Brand assets are not under the AGPL
The Silo logo, wordmark, and icon files are **© 2026 Silo Media L.L.C., all
rights reserved**, and are **not** licensed under the AGPL or any other
open-source license. Any redistribution or fork must **remove or replace** them.
See the `NOTICE` file in the brand-asset directories.
## Official sources
The only official Silo software comes from:
- the **Silo-Server** organization on GitHub — <https://github.com/Silo-Server>
- the official website — <https://siloserver.org>
- official app-store listings published by **Silo Media L.L.C.**
Any app-store listing, download, or "Silo" app that does not originate from the
above is **not** official and is **not** authorized to use the Silo marks.
## Requesting permission / reporting misuse
Email **trademark@siloserver.org** with the details. We're generally glad to
grant reasonable requests — the goal of this policy is only to prevent user
confusion and unauthorized "official-looking" releases.
---
*Silo and the Silo logo are trademarks of Silo Media L.L.C. The **™** symbol is
used with the Silo wordmark until federal registration issues, after which **®**
applies.*
+256 -26
View File
@@ -57,9 +57,11 @@ import (
"github.com/Silo-Server/silo-server/internal/intromarkers"
"github.com/Silo-Server/silo-server/internal/jellycompat"
"github.com/Silo-Server/silo-server/internal/libraryingest"
"github.com/Silo-Server/silo-server/internal/literaryworks"
"github.com/Silo-Server/silo-server/internal/logfilter"
"github.com/Silo-Server/silo-server/internal/logstream"
"github.com/Silo-Server/silo-server/internal/mail"
"github.com/Silo-Server/silo-server/internal/manga"
"github.com/Silo-Server/silo-server/internal/markers"
"github.com/Silo-Server/silo-server/internal/mdblist"
"github.com/Silo-Server/silo-server/internal/metadata"
@@ -93,6 +95,7 @@ import (
"github.com/Silo-Server/silo-server/internal/userdb"
"github.com/Silo-Server/silo-server/internal/userstore"
"github.com/Silo-Server/silo-server/internal/userstore/pgstore"
"github.com/Silo-Server/silo-server/internal/watchlist"
"github.com/Silo-Server/silo-server/internal/watchstate"
"github.com/Silo-Server/silo-server/internal/watchsync"
watchmdblist "github.com/Silo-Server/silo-server/internal/watchsync/providers/mdblist"
@@ -518,6 +521,12 @@ func main() {
// still encrypts/decrypts — no raw settings repo may escape into later wiring.
settingsRepo = catalog.NewEncryptedSettingsRepo(catalog.NewServerSettingsRepo(pool), dataCipher)
nodeID := resolveNodeIdentity()
catalogSearchStartupSettings, err := catalog.CatalogSearchSettingsFromMap(settings)
if err != nil {
slog.Warn("catalog search: failed to load settings for startup wiring; using postgres", "err", err)
catalogSearchStartupSettings = catalog.DefaultCatalogSearchSettings()
}
activeCatalogSearchProvider := catalog.ActiveCatalogSearchProvider(catalogSearchStartupSettings)
// Step 9: Validate
if err := cfg.Validate(); err != nil {
@@ -802,6 +811,11 @@ func main() {
configureS3Clients(cfg, &deps)
}
var literaryWorkService *literaryworks.Service
if deps.DB != nil {
literaryWorkService = literaryworks.NewService(literaryworks.NewRepository(deps.DB))
}
// Step 4: Create scanner (if needed).
if needsScanner && deps.DB != nil {
folderRepo := catalog.NewFolderRepository(deps.DB)
@@ -811,9 +825,11 @@ func main() {
ffprobePath := scanner.FFprobePathFromFFmpeg(cfg.Playback.FFmpegPath)
s := scanner.NewScanner(fileRepo, ffprobePath, deps.S3Public, cfg.Scanner.Workers, cfg.Scanner.EmptyTrashAfterScan)
s.SetSearchIndexProvider(activeCatalogSearchProvider)
configWatcher.OnChange(func(_, updated *config.Config) {
s.SetWorkers(updated.Scanner.Workers)
})
s.SetLiteraryWorkLinker(literaryWorkService)
deps.Scanner = s
deps.ProbeEnsurer = scanner.NewPlaybackProbeEnsurer(fileRepo, ffprobePath, 10*time.Second)
slog.Info("scanner initialized")
@@ -1034,6 +1050,7 @@ func main() {
// Step 4b: Create metadata service and match worker (if needed).
var metadataService *metadata.MetadataService
var metadataImageCacheProcessor *metadata.ImageCacheProcessor
var personRefreshService *metadata.PersonRefreshService
var matchWorker *metadata.MatchWorker
var libraryIngestExecutor *libraryingest.Executor
@@ -1051,10 +1068,11 @@ func main() {
var episodeRepo *catalog.EpisodeRepository
var audiobookEnricher *audiobooks.Enricher
var ebookEnricher *ebooks.Enricher
var mangaEnricher *manga.Enricher
if needsWorkers && deps.DB != nil && deps.FileRepo != nil {
chainRepo := metadata.NewChainRepository(deps.DB)
skippedRootRepo = metadata.NewSkippedRootRepository(deps.DB)
itemRepo = catalog.NewItemRepository(deps.DB)
itemRepo = catalog.NewItemRepository(deps.DB).WithActiveSearchProvider(activeCatalogSearchProvider)
episodeRepo = catalog.NewEpisodeRepository(deps.DB)
seasonRepo = catalog.NewSeasonRepository(deps.DB)
personRepo := catalog.NewPersonRepository(deps.DB)
@@ -1064,33 +1082,14 @@ func main() {
<-pluginAutoUpdateDone
imageResolver := metadata.NewPluginImageResolver()
if pluginService != nil {
// Register image resolver sources for all enabled plugin metadata providers.
rows, err := deps.DB.Query(appCtx,
`SELECT pc.plugin_installation_id, pc.capability_id
FROM plugin_capabilities pc
JOIN plugin_installations pi ON pi.id = pc.plugin_installation_id
WHERE pc.capability_type = 'metadata_provider.v1' AND pi.enabled = true`)
if err != nil {
slog.Warn("failed to list capabilities for image resolver registration", "error", err)
} else {
defer rows.Close()
for rows.Next() {
var instID int
var capID string
if err := rows.Scan(&instID, &capID); err != nil {
slog.Warn("failed to scan capability for image resolver", "error", err)
continue
}
source := metadata.NewPluginClientSource(instID, capID, func(
ctx context.Context, installationID int, capabilityID string,
) (metadata.PluginMetadataClient, error) {
return pluginService.MetadataProviderClient(ctx, installationID, capabilityID)
})
imageResolver.RegisterSource(capID, source)
slog.Info("registered plugin image resolver", "capability_id", capID, "installation_id", instID)
if pluginService != nil && pluginInstallationStore != nil {
reloadImageResolvers := func(ctx context.Context) {
if err := reloadPluginImageResolvers(ctx, pluginInstallationStore, imageResolver, pluginService); err != nil {
slog.Warn("failed to reload plugin image resolvers", "error", err)
}
}
pluginService.AddLifecycleHook(reloadImageResolvers)
reloadImageResolvers(appCtx)
}
if deps.S3Public != nil {
presignTTL := cfg.S3.MetadataPresignExpiry
@@ -1140,6 +1139,16 @@ func main() {
personRepo,
providerIDRepo,
)
audiobookEnricher.SetLiteraryWorkLinker(literaryWorkService)
ebookEnricher.SetLiteraryWorkLinker(literaryWorkService)
mangaEnricher = manga.NewEnricher(
deps.DB,
chainRepo,
pluginResolver,
itemRepo,
personRepo,
providerIDRepo,
)
// Always wire the image resolver so plugin-prefixed URLs (e.g.
// metadb://) can be resolved to presigned HTTP URLs in API responses.
@@ -1150,23 +1159,47 @@ func main() {
if deps.S3Public != nil {
imageCacher := imagecache.New(deps.S3Public)
metadataService.SetImageCacher(imageCacher)
imageCacheJobs := metadata.NewImageCacheJobRepository(deps.DB)
metadataService.SetImageCacheJobEnqueuer(imageCacheJobs)
metadataImageCacheProcessor = metadata.NewImageCacheProcessorWithTargets(
imageCacheJobs,
imageCacher,
imageResolver,
metadata.ImageCacheProcessorTargets{
Items: itemRepo,
Seasons: seasonRepo,
Episodes: episodeRepo,
ItemLocalizations: catalog.NewMediaItemLocalizationRepository(deps.DB),
SeasonLocalizations: catalog.NewSeasonLocalizationRepository(deps.DB),
People: personRepo,
},
)
metadataService.SetAutoCacheImages(cfg.Metadata.CacheImages)
metadataImageCacheProcessor.SetEnabled(cfg.Metadata.CacheImages)
configWatcher.OnChange(func(_, updated *config.Config) {
metadataService.SetAutoCacheImages(updated.Metadata.CacheImages)
metadataImageCacheProcessor.SetEnabled(updated.Metadata.CacheImages)
})
if deps.Scanner != nil {
deps.Scanner.SetImageCacher(imageCacher)
}
if cfg.Metadata.CacheImages {
personRefreshService.SetImageCacher(imageCacher)
personRefreshService.SetImageCacheJobEnqueuer(imageCacheJobs)
slog.Info("metadata image caching enabled")
}
if audiobookEnricher != nil {
audiobookEnricher.SetImageCacher(imageCacher)
audiobookEnricher.SetImageCacheJobEnqueuer(imageCacheJobs)
audiobookEnricher.SetFFmpegPath(scanner.FFmpegPathFromFFprobe(scanner.FFprobePathFromFFmpeg(cfg.Playback.FFmpegPath)))
}
if ebookEnricher != nil {
ebookEnricher.SetImageCacher(imageCacher)
ebookEnricher.SetImageCacheJobEnqueuer(imageCacheJobs)
}
if mangaEnricher != nil {
mangaEnricher.SetImageCacher(imageCacher)
mangaEnricher.SetImageCacheJobEnqueuer(imageCacheJobs)
}
}
@@ -1417,6 +1450,15 @@ func main() {
}
})
}
// Auto-remove fully-watched items from the watchlist (standalone behavior,
// default-on per profile), propagating removals to connected providers.
if itemRepo != nil && episodeRepo != nil && userStoreProvider != nil {
maintainer := watchlist.NewMaintainer(userStoreProvider, itemRepo, episodeRepo)
if watchProviderService != nil {
maintainer.WithListEventDispatcher(watchProviderService)
}
deps.WatchCompletionObserver = maintainer
}
deps.SessionMgr = sessionMgr
deps.PlaybackRealtimeHub = playback.NewRealtimeHub()
if chapterThumbService != nil && deps.S3Public != nil {
@@ -1480,6 +1522,7 @@ func main() {
cfg.Recommendations,
)
deps.Recommender = recEngine
deps.CatalogSearchVectorizer = recEngine
var err error
recWorker, err = recommendations.NewWorker(
@@ -1675,6 +1718,9 @@ func main() {
taskMgr.Register(tasks.NewScanLibrariesTask(deps.FolderRepo, deps.LibraryScanQueue, deps.EventBus))
}
taskMgr.Register(tasks.NewCleanupOrphanedMediaItemsTask(catalog.NewOrphanedProvisionalCleaner(deps.DB)))
catalogSearchIndexer := catalog.NewCatalogSearchIndexer(deps.DB, settingsRepo)
taskMgr.Register(tasks.NewSyncCatalogSearchIndexTask(catalogSearchIndexer))
taskMgr.Register(tasks.NewRebuildCatalogSearchIndexTask(catalogSearchIndexer))
if deps.IntroAnalyzer != nil {
taskMgr.Register(tasks.NewDetectIntroMarkersTask(deps.IntroAnalyzer, settingsRepo))
}
@@ -1699,6 +1745,9 @@ func main() {
if refreshWorker != nil && metadataService != nil {
taskMgr.Register(tasks.NewRefreshMetadataTask(refreshWorker, metadataService))
}
if metadataImageCacheProcessor != nil {
taskMgr.Register(tasks.NewCacheMetadataImagesTask(metadataImageCacheProcessor))
}
if pluginAutoUpdater != nil {
taskMgr.Register(tasks.NewCheckPluginUpdatesTask(pluginAutoUpdater))
}
@@ -1775,6 +1824,9 @@ func main() {
if ebookEnricher != nil {
taskMgr.Register(tasks.NewSyncEbookMetadataTask(ebookEnricher))
}
if mangaEnricher != nil {
taskMgr.Register(tasks.NewSyncMangaMetadataTask(mangaEnricher))
}
if pluginInstallationStore != nil && pluginRuntimeConfigStore != nil && pluginService != nil {
pluginTasks, err := plugins.NewTaskRegistryWithTypedResolver(pluginInstallationStore, pluginRuntimeConfigStore, pluginService).Tasks(appCtx)
if err != nil {
@@ -1839,6 +1891,8 @@ func main() {
AccessResolver: audiobooks.NewABSAccessResolver(absUserRepo, userStoreProvider),
Recs: recommendations.NewRepo(deps.DB),
Detail: absDetailSvc,
SessionMgr: sessionMgr,
SessionSyncer: deps.SessionSyncer,
}
absH := audiobooksService.BuildABSHandler(absHDeps)
deps.ABSHandler = absH
@@ -2092,6 +2146,7 @@ func main() {
NodePlanner: deps.NodePlanner,
JWTSecret: cfg.Auth.JWTSecret,
RecWorker: recWorker,
FrontendFS: deps.FrontendFS,
}
// Wire direct dependencies when DB is available.
@@ -2127,6 +2182,7 @@ func main() {
compatDeps.FolderRepo = folderRepo
compatDeps.SessionMgr = sessionMgr
compatDeps.UserStoreProvider = userStoreProvider
compatDeps.WatchCompletionObserver = deps.WatchCompletionObserver
compatDeps.SettingsRepo = settingsRepo
compatDeps.PersonRepo = personRepo
@@ -2433,6 +2489,180 @@ func configureS3Clients(cfg *config.Config, deps *api.Dependencies) {
}
}
type pluginImageResolverCapabilityStore interface {
ListEnabled(ctx context.Context) ([]*plugins.Installation, error)
ListCapabilities(ctx context.Context, installationID int) ([]*plugins.Capability, error)
}
func reloadPluginImageResolvers(
ctx context.Context,
store pluginImageResolverCapabilityStore,
resolver *metadata.PluginImageResolver,
service *plugins.Service,
) error {
if resolver == nil {
return nil
}
if store == nil || service == nil {
resolver.ReplaceSources(nil)
return nil
}
installations, err := store.ListEnabled(ctx)
if err != nil {
return fmt.Errorf("list enabled plugin installations: %w", err)
}
sort.Slice(installations, func(i, j int) bool {
if installations[i] == nil {
return false
}
if installations[j] == nil {
return true
}
return installations[i].ID < installations[j].ID
})
var registrations []metadata.PluginImageResolverSourceRegistration
for _, installation := range installations {
if installation == nil {
continue
}
capabilities, err := store.ListCapabilities(ctx, installation.ID)
if err != nil {
return fmt.Errorf("list image resolver capabilities for installation %d: %w", installation.ID, err)
}
sort.Slice(capabilities, func(i, j int) bool {
if capabilities[i] == nil {
return false
}
if capabilities[j] == nil {
return true
}
if capabilities[i].Type != capabilities[j].Type {
return capabilities[i].Type < capabilities[j].Type
}
return capabilities[i].ID < capabilities[j].ID
})
for _, capability := range capabilities {
if capability == nil {
continue
}
switch capability.Type {
case sdkcapability.ImageResolver:
schemes, priority := imageResolverCapabilityConfig(capability)
if len(schemes) == 0 {
slog.Warn("plugin image resolver capability has no valid schemes",
"installation_id", installation.ID,
"capability_id", capability.ID)
continue
}
for _, scheme := range schemes {
source := metadata.NewPluginClientSource(installation.ID, capability.ID, func(
ctx context.Context, installationID int, capabilityID string,
) (metadata.PluginMetadataClient, error) {
return service.ImageResolverClient(ctx, installationID, capabilityID)
})
registrations = append(registrations, metadata.PluginImageResolverSourceRegistration{
Scheme: scheme,
Source: source,
Kind: metadata.PluginImageResolverSourceExplicit,
Priority: priority,
InstallationID: installation.ID,
CapabilityID: capability.ID,
})
}
case sdkcapability.MetadataProvider:
scheme := strings.TrimSpace(capability.ID)
if !metadata.ValidImageResolverScheme(scheme) {
slog.Warn("skipping legacy metadata image resolver with invalid scheme",
"installation_id", installation.ID,
"capability_id", capability.ID)
continue
}
source := metadata.NewPluginClientSource(installation.ID, capability.ID, func(
ctx context.Context, installationID int, capabilityID string,
) (metadata.PluginMetadataClient, error) {
return service.MetadataProviderClient(ctx, installationID, capabilityID)
})
registrations = append(registrations, metadata.PluginImageResolverSourceRegistration{
Scheme: scheme,
Source: source,
Kind: metadata.PluginImageResolverSourceLegacy,
InstallationID: installation.ID,
CapabilityID: capability.ID,
})
}
}
}
resolver.ReplaceSources(registrations)
slog.Info("reloaded plugin image resolvers", "sources", len(registrations))
return nil
}
func imageResolverCapabilityConfig(capability *plugins.Capability) ([]string, int) {
if capability == nil {
return nil, 0
}
meta := capabilityMetadataFields(capability.Metadata)
return metadataStringList(meta["schemes"]), metadataInt(meta["priority"])
}
func capabilityMetadataFields(raw map[string]any) map[string]any {
if raw == nil {
return nil
}
if nested, ok := raw["metadata"]; ok {
switch typed := nested.(type) {
case map[string]any:
return typed
}
}
return raw
}
func metadataStringList(value any) []string {
var out []string
switch typed := value.(type) {
case []string:
for _, item := range typed {
if scheme := strings.TrimSpace(item); metadata.ValidImageResolverScheme(scheme) {
out = append(out, scheme)
}
}
case []any:
for _, item := range typed {
text, ok := item.(string)
if !ok {
continue
}
if scheme := strings.TrimSpace(text); metadata.ValidImageResolverScheme(scheme) {
out = append(out, scheme)
}
}
}
return out
}
func metadataInt(value any) int {
switch typed := value.(type) {
case int:
return typed
case int32:
return int(typed)
case int64:
return int(typed)
case float64:
return int(typed)
case json.Number:
n, _ := typed.Int64()
return int(n)
default:
return 0
}
}
type markerPluginCapabilityStore interface {
ListEnabled(ctx context.Context) ([]*plugins.Installation, error)
ListCapabilities(ctx context.Context, installationID int) ([]*plugins.Capability, error)
+3
View File
@@ -23,6 +23,9 @@ func buildLiveSessionSync(s *playback.Session, reportingNode string) worker.Sess
PlayMethod: string(s.PlayMethod),
ReportingNode: reportingNode,
ClientIP: s.ClientIP,
ClientName: s.ClientName,
ClientVersion: s.ClientVersion,
ClientUserAgent: s.ClientUserAgent,
AudioTrackIndex: s.AudioTrackIndex,
TranscodeAudio: s.TranscodeAudio,
StreamBitrateKbps: s.StreamBitrateKbps,
+20
View File
@@ -31,6 +31,26 @@ services:
timeout: 3s
retries: 5
meilisearch:
image: ${MEILISEARCH_IMAGE:-getmeili/meilisearch:latest}
restart: unless-stopped
profiles: [search]
entrypoint:
- /bin/sh
- -euc
- |
if [ -z "$${MEILI_MASTER_KEY:-}" ]; then
echo "MEILI_MASTER_KEY is required when enabling the search profile" >&2
exit 1
fi
exec /bin/meilisearch
environment:
MEILI_MASTER_KEY: ${MEILI_MASTER_KEY:-}
ports:
- "${MEILISEARCH_HOST:-127.0.0.1}:${MEILISEARCH_PORT:-7700}:7700"
volumes:
- ${SILO_DATA_ROOT:-/opt/silo}/meilisearch:/meili_data
silo:
image: ${SILO_IMAGE:-ghcr.io/silo-server/silo-server:latest}
restart: unless-stopped
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,311 @@
# Plan: interleave `transcribe_translate` so live AI subtitles follow the playhead
Issue: https://github.com/Silo-Server/silo-server/issues/154
Area: `internal/subtitles/ai` (+ `internal/playback` extraction, `internal/config`)
Type: bug fix to an existing (shipped) capability — makes the chained AI-subtitle
path deliver its designed live behavior. No new client-facing contract.
Commands assume the repository root is the cwd.
## Problem (confirmed)
`Service.runTranscribe` (`internal/subtitles/ai/service.go`) runs the chained
`transcribe_translate` kind as two **global, sequential** stages:
1. Transcribe the **entire** file (progress 5%→70%). Transcript cues are not
streamed for the chained kind —
`streamTranscript := streaming && job.Kind == JobKindTranscribe`
(`service.go:417`), so it is `false` for `transcribe_translate`.
2. Only then translate **all** cues (70%→95%), streaming each translated batch
as it lands (`service.go:482-493`).
Because translation cannot start until step 1 finishes for the whole file, the
first translated cue streams minutes into the job, long past the playhead.
### Two distinct bottlenecks (both must be addressed)
- **Transcription stage is global.** Translation waits for the whole-file ASR
pass. This is the dominant term for a feature film (~20 min ASR) and the
primary cause. Fixed by interleaving translation per chunk (Phase 1).
- **Extraction is global and precedes any callback.** `Transcribe` calls
`t.extract(...)` (`transcriber.go:124`) before the `onChunk` loop, and
`ExtractAudioChunks` blocks on a single whole-track ffmpeg pass
(`internal/playback/audio_extract.go:34,65`). So even with per-chunk
interleave and small chunks, the first cue still waits for full-file
extraction (~1–2 min for a 2h film). Fixed by playhead-first incremental
extraction (Phase 2).
The transcriber already exposes the per-chunk seam: `Transcribe` invokes
`onChunk(cues, done, total)` per chunk, playhead-first (`transcriber.go:111-174`),
and plain `transcribe` already streams through it (`service.go:426-429`).
Secondary defect found while confirming: `WhisperTranscriber.SetExtraction`
clamps any `chunkSeconds` outside `[60, 600]` to the **default 600**
(`transcriber.go:100-102`), so a small live chunk value silently becomes 10 min
instead of clamping to the floor.
## Goal & honest latency targets
Translated cues stream playhead-first for a session-attached
`transcribe_translate` job, instead of waiting for whole-file transcription.
Time-to-first-translated-cue, by phase:
- **Today:** `extract(full) + transcribe(full) + translate(first batch)` — many
minutes (≈ full ASR pass).
- **After Phase 1 (interleave, existing full extraction):**
`extract(full) + transcribe(first chunk) + translate(first chunk)` — removes
the whole-file ASR pass from the critical path; ≈ extraction time (~1–2 min for
a 2h film), then cues follow the playhead. Already usable for live after a
one-time wait.
- **After Phase 2 (playhead-first incremental extraction):**
`extract(one chunk near playhead) + transcribe + translate` — single-digit
seconds. This is what the issue means by "within seconds."
Phase 1 delivers most of the value with low risk; Phase 2 closes the remaining
extraction latency. Ship Phase 1 first; do not claim "within seconds" until
Phase 2 lands.
### Non-goals (this plan)
- Pacing/throttling to the playhead (rolling window). Optional in the issue;
follow-up below.
- Cross-chunk translation-context continuity beyond batch context neighbours.
## Decision: interleave only when streaming
**Session-attached (live) jobs** use the interleaved per-chunk path.
**Background (non-streaming) `transcribe_translate` jobs keep the current
whole-file transcribe-then-translate-at-end path unchanged.** Rationale: the
issue is live-only; this preserves background output byte-for-byte (full
cross-file translation context, no incremental-extraction cost) and keeps the
two behaviors cleanly separated by the existing `streaming` boolean. The
shared transcript-storage step is factored so it is not duplicated.
---
## Phase 1 — interleave translation per chunk (streaming jobs)
### 1a. `internal/subtitles/ai/service.go` — `runTranscribe`
For the chained kind **when `streaming`**, replace the kind-specific `onChunk`
body (`service.go:423-430`) with per-chunk translate-and-stream:
- Capture a translate error in a closure: `var translateErr error` and an
accumulator `var translatedAccum []SubtitleCue`.
- In `onChunk(chunk, done, total)`:
- If `translateErr != nil` or `len(chunk) == 0`, return (skip silence; stop
translating after a prior failure). Do **not** abort transcription.
- Else call `s.translator.Translate` on **that chunk's** cues. On error: set
`translateErr`, emit `TranslationFailed` once, and return (let transcription
continue). On success: stream translated cues via `TranslationCues` and
append to `translatedAccum`.
- Progress: single 5%→95% band keyed on chunk `done/total`
(message "Transcribing & translating").
- Keep `onChunk` signature unchanged (no error return). Aborting on translate
failure via the callback is intentionally avoided — see 1b.
After `Transcribe` returns:
- **Always store the transcript track** (unchanged, `service.go:447-466`,
including the transcript `SubtitleReady` broadcast). This must run regardless
of `translateErr`.
- If `translateErr != nil`: `finishWithError(ctx, job, translateErr)`. The
transcript is already persisted (cache preserved), and the live session
already received `TranslationFailed`.
- Else: assemble the translated track from `translatedAccum` (`sortCuesByStart`
+ `StoreSubtitle`, `service.go:499-514`), `CompleteJob`, and
`TranslationCompleted` (`service.go:515-522`).
- Remove the whole-file translate-at-end (`service.go:479-497`) for the
streaming path only.
Non-streaming chained jobs fall through to the existing whole-file path,
untouched.
### 1b. Transcript-on-translation-failure (resolves review P1)
Today the full transcript is stored before translation starts
(`service.go:447` then `:482`), so a translation failure still leaves a
transcript/cache row and an already-broadcast `SubtitleReady`. The interleaved
path preserves this exactly because:
- a per-chunk translate failure sets `translateErr` but lets the ASR pass run to
completion (it does **not** return an error from `onChunk`), so `Transcribe`
returns the full cue set normally;
- the transcript is then stored on the same post-transcribe path as today,
before the job is failed for the translated track.
An ASR failure or context cancellation still returns an error from `Transcribe`
and yields no transcript — identical to today.
### 1c. Live progress semantics (resolves review P2)
`SubtitleTranslationCuesPayload.Done/Total` is documented as overall progress
(`internal/playback/realtime.go:172-181`). For live `transcribe_translate`:
- `TranslationCues` carries **`done = chunks completed, total = total chunk
count`** — the same chunk-granular convention plain `transcribe` already emits
(`service.go:427-428`). Monotonic and stable across chunks.
- `TranslationStarted.TotalCues = 0` (indeterminate), consistent with the
transcribe path today (`service.go:412`), since the cue total is unknown before
ASR completes.
- Cues carry absolute Start/End, so the client places them regardless of
arrival order; unpause is driven by cue arrival near the playhead, not by
`Done/Total`. Confirm with client teams that `Done/Total` is treated as a
fraction, not absolute cue indices (clients already receive chunk counts from
the plain `transcribe` path, so this should hold).
### 1d. Small live chunks + clamp fix
- Add `TranscribeJobRequest.ChunkSeconds int` (0 = configured default). In
`Transcribe`, use `req.ChunkSeconds` when `> 0`, else the atomic default
(`transcriber.go:121-122`); validate/clamp identically to `SetExtraction`.
- Fix the clamp in `SetExtraction` (`transcriber.go:99-105`): clamp
out-of-range to the nearest bound, not to the 600 default.
- Lower `minASRChunkSeconds` from 60 to 15 (`transcriber.go:26`); update the
doc comment about request count / boundary word-clip tradeoff.
- Config: add `subtitle_ai.live_asr_chunk_seconds` (default 30):
- `internal/config/config.go`: new field beside `ASRChunkSeconds`
(`config.go:274-277`).
- `internal/config/db_loader.go`: load it next to `asr_chunk_seconds`
(`db_loader.go:472-476`).
- `internal/subtitles/ai/config.go`: add `LiveASRChunkSeconds int` to the
service `Config` so the service can set `req.ChunkSeconds` for streaming jobs.
- `internal/api/router.go`: plumb through `effectiveSubtitleAIConfig` and the
`OnConfigChange` `UpdateConfig` (`router.go:1024-1055`).
### 1e. Phase 1 tests
- `service_test.go`: fake `Transcriber` emitting several chunks playhead-first
via `onChunk`; fake `Translator` recording calls. For a streaming
`transcribe_translate` job assert: one `Translate` per non-empty chunk; first
`TranslationCues` streamed before the last chunk is transcribed; empty chunks
skipped; final stored translated track = all translated cues sorted by start;
`Done/Total` is chunk-granular and monotonic.
- Failure case: a translate error on chunk N sets the failure but transcription
completes, the transcript track **is** stored, `TranslationFailed` is emitted,
and the job ends via `finishWithError`.
- Non-streaming `transcribe_translate`: unchanged whole-file path; correct final
track; no `TranslationCues`.
- `transcriber_test.go`: `SetExtraction` clamp fix and `ChunkSeconds` override
(incl. the new 15s floor).
---
## Phase 2 — playhead-first incremental extraction (true seconds latency)
Phase 1 still waits on full-file extraction before the first cue. Phase 2
removes that by extracting and consuming chunks incrementally, seeked to the
playhead.
### 2a. New incremental extractor in `internal/playback`
Add a streaming, seek-based extractor alongside `ExtractAudioChunks`:
```
ExtractAudioChunksFrom(ctx, filePath, audioTrackIndex, dir, ffmpegPath,
startSec float64, chunkSeconds int,
onSegment func(AudioChunk) error) error
```
- Runs one ffmpeg pass with `-ss <startSec> -i <file> ... -f segment
-segment_list segments.csv` covering `startSec`→end.
- Consumes segments **as they complete**: poll the segment-list CSV (ffmpeg
appends `filename,start,end` when it closes each segment) on a short interval;
for each new row, invoke `onSegment(AudioChunk{Path, Start: startSec + csvStart})`.
Reconcile any trailing rows after `cmd.Wait()` so the final segment is not
missed.
- Honors `ctx` (CommandContext kills ffmpeg on cancel) and `onSegment` errors
(stop the pass). Caller owns `dir`; segments are deleted by the consumer after
ASR to cap disk at one extraction.
- Preserves exact per-segment starts from the CSV, so timing does not accumulate
drift within a pass (only the initial `-ss` seek is packet-accurate, sub-second,
and is further corrected by the existing `ProbeAudioStartOffset`).
Rejected alternative — **per-chunk seek extraction** (`-ss s -t d` per chunk):
simpler control flow but O(n) ffmpeg spawns and packet-aligned per-chunk seeks
that can clip words at every seam. The single-pass streaming consumer keeps the
existing segment-muxer boundary quality and far fewer process spawns.
### 2b. Transcriber incremental mode
- Add `TranscribeJobRequest.Incremental bool`, set by the service for streaming
jobs.
- When `Incremental`, `Transcribe` drives playhead-first in two streaming passes
(reproducing today's `chunkOrderForPosition` order): pass A `startSec =
playhead → end`, then pass B `0 → playhead`. Each completed segment is fed
straight into the existing per-chunk ASR loop body (transcribe → `onChunk`),
which Phase 1 already wires to translate+stream. The full cue set is still
accumulated and returned for transcript storage.
- When not `Incremental` (background, and plain transcribe unless we opt it in),
keep the existing whole-file `ExtractAudioChunks` + `chunkOrderForPosition`
path.
Note: timing composition (`-ss` reset PTS + segment CSV start + probe offset)
must be verified with a fixture; see tests.
### 2c. Phase 2 tests
- `audio_extract` test (or transcriber test with an injected extractor) covering
incremental emission order, exact-start mapping with a `startSec` offset, and
ctx-cancel mid-pass.
- Transcriber test asserting `Incremental` yields playhead-first chunk order
across the two passes and returns the complete cue set for transcript storage.
- Confirm first `onChunk` fires after ~one chunk of audio, not after full-file
extraction (timing/ordering assertion via a fake extractor, not wall-clock).
---
## Multi-repo / client impact
None expected to the contract. Reuses the existing notifier protocol
(`TranslationStarted` / `TranslationCues` / `TranslationCompleted` /
`TranslationFailed`, `notifier.go`) under the same translated-track `trackKey`.
Translated cues simply begin arriving far earlier, and `Done/Total` keeps the
chunk-granular meaning plain `transcribe` already uses (1c). Worth a smoke test
on one `silo-android` / `silo-apple` client to confirm the live track renders
cues arriving before the transcript track completes, and that `Done/Total` is
treated as a fraction.
## Risks & tradeoffs
- **Translation context at chunk boundaries (live only).** Per-chunk translation
resets the batch context-neighbour window at each ~30s boundary — accepted
quality tradeoff for live; background jobs are unchanged.
- **More/smaller ASR requests for live jobs.** 30s chunks mean more requests;
per-chunk timeout (`chunkSeconds * asrChunkTimeoutFactor`) and per-spawn
overhead scale; the 15s floor bounds the worst case.
- **Incremental extraction reliability (Phase 2).** Segment-list polling + the
`-ss` seek path need fixtures for boundary timing, the final-segment race, and
cancel/cleanup. Single ffmpeg pass per direction keeps cost low.
- **Boundary word-clips.** Smaller live chunks add more fixed-length boundaries
where a straddling word can clip — already a documented v1 limitation; the
silence-aligned follow-up would address it.
## Out of scope / follow-ups
- **Pace to playback (issue item 3):** rolling window ahead of the playhead to
bound work; separate change once interleave + incremental extraction land.
- **Cross-chunk translation context:** feed the previous chunk's tail source
cues as untranslated context into each chunk's translate call.
- **Silence-aligned chunk boundaries:** reduce boundary word-clips that smaller
live chunks make more frequent.
## v1 scope note
`docs/architecture/v1-scope.md` is **NOT LOCKED**; the issue carries
`v1-proposed`. This is a **bug fix** to an already-shipped capability (no new
user-facing capability, endpoint, or response field — only an internal config
key and corrected streaming behavior), so it proceeds under the "bug fixes
proceed normally" clause. Link the PR to the AI-subtitle capability item per the
v1 PR requirements.
## Pre-push checklist
- `make lint`
- `cd web && pnpm run lint && pnpm run format:check` (run only if web touched —
none expected)
- `make verify-local-paths`
- Go tests for `internal/subtitles/ai` and `internal/playback` in a
libvips-capable container (a bare-host `go test ./...` silently skips CGO
packages).
@@ -0,0 +1,188 @@
# Personal Collection Catalog Filter Standardization - Implementation Plan
- **Date:** 2026-06-23
- **Spec:** `docs/superpowers/specs/2026-06-23-personal-collection-catalog-filter-standardization-design.md` (rev. 3, post-adversarial-review). The spec owns the *why* and the decisions; this plan owns the *how*, with concrete files and ordering.
- **Note:** All paths are repository-relative. Commands assume the repository root is the cwd.
## Approach & PR Boundaries
Three concerns, landed in order. PR 1 is independently valuable and must merge first because PR 2 depends on its hardened predicate for correctness.
| PR | Concern | Depends on |
|----|---------|-----------|
| **PR 1** | Harden the catalog `watched` predicate (series/season rollup + row-type branching) and pin equivalence with item user-data. | — |
| **PR 2** | Replace the `watch_filter` / `media_filter` enum vocabulary with a `display_query_definition` fragment end to end (migration → storage → API → execution → web). | PR 1 |
| **PR 3** (deferred) | Wire display filters for smart/live collections via an executor-level `(source) AND (display)` conjunction. | PR 2 |
**Why PR 3 is deferred:** display-filter controls are currently exposed only for manual/exact and imported collections (`CollectionBuilder.tsx` renders them only when `collection_type === "manual"`; imported uses `ImportedCollectionEditor.tsx`). Smart collections never reach `FilterUserCollectionDisplayItems` today (`catalog_resolver.go` branches smart collections to `resolveQuerySource` before the filter call). So the OR-leak the adversarial review found is a guardrail for *future* wiring, not a v1 path. Build PR 2 so the same conjunction helper is reusable, but do not expose smart-collection display filters until PR 3.
---
## PR 1 — Catalog Watched Predicate Hardening
**Goal:** make the catalog `watched` rule produce the same played/read verdict as item user-data (`internal/api/handlers/user_state.go`) for every item kind, under both scoped and unscoped queries. This is the single-semantic foundation; everything else routes through it.
### Current state (verified)
- `internal/catalog/query_builder.go`:
- `buildWatchedClause` (≈900–912) wraps `userStateCompletionClause` and negates for `value:false`.
- `userStateCompletionClause` (≈1119–1171) checks completion against **the row's own `content_id`** (`uwp.media_item_id = mi.content_id` / `uwh.media_item_id = mi.content_id`). No episode rollup → a series row is effectively never watched.
- `ebookUserStateCompletionClause` (≈1173–1193) is selected **only when `qb.mediaScope == "ebook"`** → an ebook under an unscoped query is misclassified by the video clause.
- `internal/api/handlers/user_state.go` `allEpisodesCompleted` (≈171–185) defines the canonical rollup over the episode repository's set.
- Episode set (the parity target): `internal/catalog/episode_repo.go` `ListBySeriesIDs` / `ListBySeasonIDs` select from `episodes` filtered by `series_id` / `season_id` **AND** `episodeAvailabilityPredicate` = `EXISTS (SELECT 1 FROM episode_libraries el WHERE el.episode_id = episodes.content_id)`. No specials/season-0 exclusion, no media-file requirement. Links: `episodes.series_id`, `episodes.season_id`.
- The base relation always exposes `mi.type` (`media_items mi`, or the episode derived table that hard-codes `'episode'::text AS type`), so a `CASE` on row type is feasible.
### Changes
1. **Extract a reusable completion predicate.** In `internal/catalog/query_builder.go`, refactor `userStateCompletionClause` into a helper that, given a row alias and the linking column, emits the leaf completion EXISTS pair (progress + history, with hidden-history guard). Keep `ebookUserStateCompletionClause` as the ebook leaf. This is the shared building block for the branches below.
2. **Add series/season rollup branches with repository-parity episode sets.** Emit, for a series row, "≥1 child episode exists AND no child episode is incomplete," expressed so the child-episode set is **identical** to `episodeAvailabilityPredicate`:
```sql
EXISTS (SELECT 1 FROM episodes e
WHERE e.series_id = mi.content_id
AND EXISTS (SELECT 1 FROM episode_libraries el WHERE el.episode_id = e.content_id))
AND NOT EXISTS (
SELECT 1 FROM episodes e
WHERE e.series_id = mi.content_id
AND EXISTS (SELECT 1 FROM episode_libraries el WHERE el.episode_id = e.content_id)
AND NOT (<leaf completion predicate for e.content_id>))
```
Season rollup is the same with `e.season_id = mi.content_id`. The leaf predicate inside the `NOT EXISTS` reuses the helper from step 1 keyed on `e.content_id`. **The `episode_libraries` EXISTS must match `episodeAvailabilityPredicate` verbatim** — if `episode_repo.go` ever changes that predicate, this SQL changes with it. Add a code comment cross-referencing `episodeAvailabilityPredicate` so the coupling is discoverable.
3. **Branch by row type for unscoped queries.** When `qb.mediaScope` is empty/`video` (rows can be mixed), select the completion source from `mi.type` rather than from `mediaScope`:
- `movie` / `episode` / video leaf → leaf predicate,
- `series` → series rollup, `season` → season rollup,
- `ebook` → ebook leaf,
- `manga` → deferred; treat as never-completed for now (documented), do not invent a collection-only interpretation.
When `mediaScope` is a single concrete kind (e.g. `series`, `ebook`), the optimizer-friendly path may emit just that branch. Implement as a `CASE mi.type WHEN ... END` or chained `(mi.type = 'x' AND <clause>) OR ...`; verify argIdx accounting (each leaf/ebook clause increments `qb.argIdx` by 2 — the rollups reuse the leaf, so count the placeholders carefully and add a builder test that the produced arg count matches bound args).
4. **`last_watched` stays separate.** Do not route `last_watched` through the new predicate; it is a `user_last_watched` CTE timestamp aggregate. Leave as-is; add a one-line comment noting the intentional divergence.
### Tests (PR 1)
- `internal/catalog/query_builder_test.go`: assert the `watched` clause contains the series/season rollup EXISTS only when the scope/row-type can include series/seasons; assert arg count == bound args for each branch; assert the `episode_libraries` predicate text is present in the rollup.
- **Equivalence test (highest value):** seed a Postgres fixture with a series whose episodes are variously completed/not, a season, a movie, and an ebook for one profile; assert the catalog `watched` rule's matched set equals exactly the items whose `user_state.go` `Played` flag is true. Because `user_state.go` lives in `internal/api/handlers` (CGO/libvips), run this in the libvips-capable container; the test can drive both the catalog executor and the item-user-state builder against the same pool and diff the verdicts. If a same-package home is awkward, place the comparison test in `internal/api/handlers` where both sides are reachable.
### Release note (PR 1)
Catalog `watched` filtering for series/seasons changes from "never matches" to "matches when all available episodes are completed." Call this out in the PR body and release notes per the spec's semantics-change risk.
---
## PR 2 — Replace the Enum Vocabulary with `display_query_definition`
One concern: swap the persisted representation and every surface that reads/writes it. Clean replacement — no aliases, no backfill (the enums are branch-only and unreleased; see spec Current Branch State). Land the steps below together so no layer reads a column another layer has dropped.
### Step 2.1 — Migration reshape
Edit `migrations/sql/20260623184858_user_collection_display_filters.sql` in place so the canonical schema has no legacy columns:
- **Up:** `ALTER TABLE public.user_personal_collections ADD COLUMN IF NOT EXISTS display_query_definition jsonb;` (nullable; `NULL`/absent = no display filter). Remove the `watch_filter` / `media_filter` `ADD COLUMN` and their CHECK constraints.
- **Down:** `ALTER TABLE ... DROP COLUMN IF EXISTS display_query_definition;`
**Operational gotcha:** Goose keys applied migrations by version, so editing an already-applied migration will **not** re-run it. Any dev DB that already applied the original `20260623184858` (which added `watch_filter`/`media_filter`) must be rolled back to before it and re-applied: `make migrate-down` to the prior version, then `make migrate-up`. If a shared dev DB makes in-place editing risky, the alternative is a *new* timestamped migration (`make migrate-create NAME=replace_collection_display_filters`) that drops the two columns and adds `display_query_definition`; the spec prefers in-place reshape since nothing is merged, but call the choice out in the PR.
### Step 2.2 — Backend storage & types
- `internal/userstore/types.go`: in `Collection` (≈185–213), `CreateCollectionInput` (≈238–255), `UpdateCollectionInput` (≈257–278), remove `WatchFilter` / `MediaFilter`; add `DisplayQueryDefinition string` (and `*string` on the update input), mirroring the existing `QueryDefinition` fields.
- `internal/userstore/pgstore/collections.go`:
- `collectionSelectColumns` (≈15–18): drop `watch_filter, media_filter`; add `display_query_definition`.
- `scanCollection` (≈20–42) and the `ListCollections` scan (≈192–196): drop the two scan targets; add `&c.DisplayQueryDefinition`.
- `CreateCollection` INSERT (≈79–94): drop the two columns/args; add `display_query_definition` + arg.
- `UpdateCollection` (≈270–283): replace the two `add("watch_filter"…)` / `add("media_filter"…)` blocks with a single `display_query_definition` block driven by `input.DisplayQueryDefinition != nil`.
- The normalization at create-time (≈59–68) moves from enum normalization to **fragment normalization** (Step 2.5).
- `internal/userstore/collection_filters.go`: repurpose into the fragment normalizer's home, or delete the enum constants/`Normalize*` funcs if nothing else uses them after the swap. Keep the file's tests meaningful — convert `collection_filters_test.go` to cover the new normalizer (Step 2.5) rather than the deleted enums.
### Step 2.3 — API surface
- `internal/api/handlers/collections.go`:
- `createCollectionRequest` (≈39–50), `updateCollectionRequest` (≈52–67), `collectionResponse` (≈73–101): drop `watch_filter` / `media_filter`; add `display_query_definition json.RawMessage` following the existing `query_definition` handling (response assembly mirrors `defaultJSON([]byte(c.QueryDefinition))` at ≈843).
- `HandleCreateCollection` (≈270–279) / `HandleUpdateCollection` (≈341–356): replace enum validation with fragment normalization/validation (Step 2.5); reject invalid fragments with `400`.
- `collectionCapabilitiesResponse` (≈108–111) + `HandleCapabilities` (≈224–229): replace `watch_filters`/`media_filters` value lists with capability metadata describing the supported display-query fields (`type`, `watched`) and the UI presets (`all/watched/unwatched`, `all/movies/series`). Keep it additive and feature-detection-oriented per the v1 API rules.
- `internal/api/handlers/user_collection_imports.go`: `userImportSharedFields` (≈67–77) and `createImportedCollection` validation (≈214–223) — same swap to `display_query_definition`.
### Step 2.4 — Exact-collection execution through the catalog executor
Replace the custom Go filter pass and the metadata-only in-Go matcher with one SQL path.
- Delete `internal/catalog/user_collection_display_filters.go` (`FilterUserCollectionDisplayItems` and helpers) and its test, plus the duplicated `userCollectionAllEpisodesCompleted`.
- Call sites to rewrite:
- `internal/catalog/catalog_resolver.go` `resolveUserCollectionSource` (≈479–526, filter call at ≈517).
- `internal/sections/fetcher.go` `fetchUserCollection` (filter call at ≈1371).
- New behavior for exact collections (per spec Execution Model):
1. Load member IDs in source order (`user_personal_collection_items`, `ORDER BY position`) — unchanged.
2. Parse `collection.DisplayQueryDefinition` into a `QueryDefinition`, **force `Limit = nil`** and drop any fragment `sort`/`library_ids` defensively before execution.
3. Run the executor with `AccessFilter.AllowedContentIDs = memberIDs` (the pattern `resolveCandidateItemsWithQuery` already uses at `catalog_resolver.go` ≈606–607). The executor's `conditions` slice AND-joins `AllowedContentIDs` with the display fragment, so membership ∧ filter is automatic — no OR-leak risk here.
4. For source-ordered requests: request the **full** match set (no limit), collect matching IDs into a set, reapply original order, then apply request offset/limit. Pagination happens after reordering, never inside the executor.
5. Stop using `filterCatalogItems` (`catalog_resolver.go` ≈1772) for these collections — replace the `UseSourceOrder` branch in `resolveExactOrderedMediaItems` (≈571–586) with the match-set-then-reorder path. **Do not delete `filterCatalogItems`:** it has a second caller at `catalog_resolver.go:244` (the query-source path), so it stays; only the exact-collection call at ≈575 changes.
- Empty display fragment ⇒ skip the executor filter entirely and return members in source order (preserves today's "no filter" performance for unfiltered collections).
### Step 2.5 — Fragment normalization (shared backend helper)
A `display_query_definition` is a **filter-only** `QueryDefinition` fragment. Add one normalizer (e.g. in `internal/catalog` next to `QueryDefinition`, or `internal/userstore`) used by every write path (create/update/import):
- Accept only `match` + `groups`, with rules limited to the allowed display vocabulary (v1: `type`, `watched`).
- Reject (preferred, on write) or strip `limit`, `sort`, `library_ids`, `media_scope`.
- Reject unknown fields/ops so clients learn the contract.
- Return a canonical JSON string for storage. Used by `pgstore` create/update and the API handlers.
### Step 2.6 — Web
The frontend already has the machinery: `QueryDefinition` types, `createEmptyQueryDefinition` / `normalizeQueryDefinition` (`web/src/api/types.ts` ≈1318–1361), and `CollectionGuidedRulesEditor.tsx` which already converts `watched` rules ↔ a `watchStatus` field (`queryDefinitionToGuidedState` ≈93–254, `guidedStateToQueryDefinition` ≈257–374). Reuse it; do not add a parallel state model.
- `web/src/lib/collectionDisplayFilters.ts`: replace the enum option lists / `normalize*` helpers with converters **preset ⇄ `QueryDefinition` fragment**:
- `watched`/`unwatched`/`all` ⇄ a `{field:"watched",op:"is",value:true|false}` rule (or none),
- `movie`/`series`/`all` ⇄ a `{field:"type",op:"is",value:"movie"|"series"}` rule (or none),
- a builder that assembles the two presets into a single filter-only fragment, and a reader that derives the two preset values back from a fragment (for editing).
- `web/src/api/types.ts`: drop `UserCollectionWatchFilter` / `UserCollectionMediaFilter` and the `watch_filter` / `media_filter` fields on `Collection` (≈1270–1271), `CreateCollectionRequest` (≈1403–1404), `UpdateCollectionRequest` (≈1420–1421); add `display_query_definition?: QueryDefinition`.
- Editors — swap the two `<Select>` controls' wiring to read/write through the helper, keeping the same visible labels:
- `web/src/components/collections/CollectionBuilder.tsx` (value shape ≈45–57, UI ≈259–313, submit ≈88–101),
- `web/src/pages/ImportedCollectionEditor.tsx` (state ≈118–125, UI ≈277–354, save ≈184–214),
- `web/src/components/CollectionTemplateGallery/UserCollectionTemplateConfigForm.tsx` (state ≈99–100, submit ≈119–167).
- `web/src/hooks/queries/collections.ts` (`useCreateCollection` ≈85–101, `useUpdateCollection` ≈103–127): payloads now carry `display_query_definition` instead of the enums (the generic `buildUserCollectionPayload` needs no change beyond the type swap).
### Tests (PR 2)
- `internal/api/handlers/collections_test.go`: create/update/get round-trips `display_query_definition`; `watch_filter`/`media_filter` are absent from request and response; an invalid fragment (e.g. containing `limit` or an unknown field) returns `400`.
- Resolver/executor tests:
- exact collection routes through the catalog `watched` rule and preserves source order across pages;
- an exact collection larger than a page returns the full match set in source order regardless of fragment contents (guards the limit-truncation finding);
- a mixed-media exact collection (movie + ebook) with `watched`/`unwatched` classifies the ebook via `ebook_reader_progress`, not video history (relies on PR 1 row-type branching).
- Fragment-normalizer unit tests (replacing `collection_filters_test.go`): strips/rejects `limit`/`sort`/`library_ids`/`media_scope`; round-trips `type` + `watched` rules.
- Frontend: helper test for preset ⇄ fragment conversion (both directions); one editor test asserting the submit payload carries the canonical `display_query_definition`.
---
## PR 3 — Smart/Live Display Filters (deferred)
Only when smart/live collections expose display-filter controls. Implements the spec's executor-level conjunction so `(source) AND (display)` holds regardless of the source's top-level `match`.
- Seam: `internal/catalog/query_executor.go`, after the `AllowedContentIDs` handling (≈322) and before `ApplySectionAccessFilter`. Build the display fragment with a second `NewQueryBuilder("mi").WithArgIdx(argIdx)`, `rebindSQLPlaceholders` its output to the running `argIdx`, append the rebound clause to `conditions`, and advance `argIdx`/`args`. Because `conditions` is AND-joined (≈353), this yields `(sourceWhere) AND (displayWhere)` with no group-list manipulation and no OR-leak.
- Do **not** append the display group into `def.Groups` — that is the OR-leak the adversarial review found (`QueryBuilder.Build` joins top-level groups with the def's single `match`, ≈84–111).
- Tests: a `match: any` source plus a display filter never returns rows the source alone excludes; a contradictory `watched` overlay on a `match: any` source returns zero rows.
---
## Verification (every PR, pre-push)
Per `CLAUDE.md`:
- `make lint`
- `cd web && pnpm run lint && pnpm run format:check`
- `make verify-local-paths`
- Go tests in a **libvips-capable container** (a bare-host `go test ./...` silently skips CGO packages including `internal/api/handlers`, where the equivalence and DTO tests live). Prefix `GOWORK=off` for builds/tests in this repo.
- Targeted: `go test ./internal/catalog/... ./internal/api/handlers/... ./internal/userstore/...` and the relevant `web` helper/editor tests.
## Sequencing summary
1. PR 1 (predicate + equivalence test) → merge.
2. PR 2 (migration → types/storage → API/imports → fragment normalizer → exact execution → web), landed together → merge.
3. PR 3 only if/when smart-collection display filters are wired.
Link each PR to the capability sub-issue (`Part of #NNN`) per the v1 process; keep one concern per PR.
## Open items to confirm before/while implementing
- Confirm no `silo-android` / `silo-apple` build already consumes the branch's `watch_filter` / `media_filter` fields (spec assumes unreleased; if pinned, fall back to the contingency backfill path in the spec's Migration Strategy).
- Confirm `episodeAvailabilityPredicate` is the intended "child episode" definition to match (Open Question #4 in the spec) — it currently includes specials/season-0 if they have library links. Match it exactly; change item user-data and the catalog rule together if it should differ.
- Decide the fragment normalizer's package home (`internal/catalog` vs `internal/userstore`) based on import direction — it needs the `QueryDefinition` type without creating an import cycle.
@@ -0,0 +1,327 @@
# Hybrid Semantic Search Hardening Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. Commands assume the repository root is the cwd.
**Goal:** Make Meilisearch hybrid semantic search safe to enable broadly by gating it on *trustworthy* per-type vector coverage, validating embedder capability, and exposing per-query/admin diagnostics — without regressing the hot search path.
**Architecture:** Semantic search stays gated behind the existing `SemanticEnabled && wordCount >= 3` rule, plus a new **coverage gate** that reads an in-memory, periodically-refreshed per-type coverage snapshot (never a per-request DB query). Coverage = `current-model embeddings ÷ embed-eligible items`, computed entirely from Postgres and latched with hysteresis. Existing provider/fallback fields are plumbed through the resolver into an additive `search_diagnostics` response object. A rate-limited capability probe validates the Meili embedder without downing keyword search.
**Tech Stack:** Go (pgx/pgxpool), Meilisearch hybrid search (`userProvided` embedder), React/TypeScript admin UI, Goose migrations.
## Global Constraints
- **API additive-only within `/api/v1`** — never rename/remove a response field, change a type, or repurpose a status code. New fields are `omitempty` additions. (CLAUDE.md)
- **Performance first, reliability first, predictable under load** — the search request path must not gain a synchronous DB count, lock wait, or remote probe. Fail **safe** (keyword-only), never panic. (CLAUDE.md Core Priorities)
- **One additive index migration** is in scope (revised — see Assumptions): a btree on `media_item_embeddings(model)` to support the periodic coverage query. No schema/column changes.
- **Canonical embedding dimensions** = `embeddingvectors.CanonicalDimensions` (= 3072; re-exported as `recommendations.CanonicalEmbeddingDimensions`). Never hardcode the number.
- **Single source of truth for embed-eligibility** — the predicate `(status='matched' OR type IN ('audiobook','ebook'))` must not be duplicated. It moves to the neutral `internal/embeddingvectors` package (imported by both `catalog` and `recommendations`; `recommendations` already imports `catalog`, so `catalog` cannot import `recommendations`).
- **Single phase** — measurement *and* enforcement ship together. The diagnostics shipped here are what let us retune thresholds later without code changes-in-anger.
- Build/test in worktrees with `GOWORK=off` and a stubbed `web/dist` (see worktree-build-quirks memory).
---
## Corrected Baseline (what actually exists today — verified)
Work from this, not from intuition. Each point was confirmed against the tree.
1. **The only semantic skip rule is `SemanticEnabled && len(Fields(normalizedQuery)) >= 3`** (`internal/catalog/search_meilisearch_provider.go:329`). There is **no** `SkipTotal`/preview skip, and `TestMeilisearchSearchRequestBuildsHybridForApproximateInteractiveSearch` (`search_provider_test.go:213`) asserts hybrid **IS** used with `SkipTotal: true`. → **Do not add a `SkipTotal` skip.** Layer the coverage gate onto the existing rule; remove nothing.
2. **`CatalogSearchResult` already carries `Provider` and `FallbackReason`** (`search_provider.go:64-71`). Diagnostics work is *plumbing these through* `CatalogResult` (which drops them, `catalog_resolver.go:24-30,291-296`) — not inventing provider fields.
3. **The provider already self-downgrades hybrid→keyword on a Meili error** (`search_meilisearch_provider.go:221-228`). Diagnostics must report the *post-downgrade* reality.
4. **`countCatalogSearchVectorDocuments` ignores `model`** (`search_indexer.go:757`), counts the *indexed* population (no eligibility filter), and its companion denominator (`document_count`) comes from **Meili index stats** (`search_service.go:129`), not Postgres. It has **four callers** — `search_service.go:139`, `search_indexer.go:133`, `:203`, `:310` — all must be updated when its signature changes.
5. **Indexed population ≠ embed-eligible population.** `LoadDocumentsAfter` (`search_indexer.go:534`) indexes every type-matched, non-manga row regardless of `status`; embeddings are only written for `(status='matched' OR audiobook OR ebook)` (`recommendations/repo.go:154-160`, `recommendationItemEligibilityWhereClause`). A denominator of "indexable items" is therefore structurally < 1.0 on any library with unmatched video → **the gate would never open.** Coverage must use the **embed-eligible** population for both numerator and denominator.
6. **Status fields that already exist:** `document_count`, `vector_document_count` (`search_provider.go:347-348`). The earlier draft's `index_document_count` does **not** exist — keep real names; add new fields alongside.
7. **`UpsertEmbedding` already enqueues a Meili index event after every write** (`recommendations/repo.go:186`) — verified correct, **no change**.
8. **Catalog search settings are restart-bound** (`config/restart_keys.go:75-85`, *"Catalog search provider construction is intentionally startup-bound in v1"*). The service is built **once** in `NewRouter` and never live-rebuilt. → the coverage refresher is a single process-lifetime goroutine; no live-rebuild leak exists in v1.
9. **`catalogSearchService` is a local var in `NewRouter`** (`router.go:437`), wired into handlers but **never returned** to `main.go`. The long-lived `deps.AppContext` (`router.go:88`) is the refresher's context.
10. **Legacy `/search` plain-`q` re-route is descoped** — see [Descoped](#descoped--not-in-this-pass).
---
## Design Decisions
### D1 — Coverage = current-model embeddings ÷ embed-eligible items (single Postgres source, per type)
Let `ELIG(alias)` = `embeddingvectors.ItemEligibilityWhereClause(alias)` → `(alias.status='matched' OR alias.type IN ('audiobook','ebook'))` (moved here from `recommendations`; `recommendations.recommendationItemEligibilityWhereClause` is refactored to call it — no behavior change, no duplication).
- **Denominator** (embed-eligible items, per type):
```sql
SELECT mi.type, COUNT(*) AS eligible
FROM media_items mi
WHERE NOT EXISTS (SELECT 1 FROM manga_chapters mc WHERE mc.chapter_content_id = mi.content_id)
AND ($1::text[] IS NULL OR mi.type = ANY($1))
AND (mi.status = 'matched' OR mi.type IN ('audiobook','ebook')) -- ELIG("mi")
GROUP BY mi.type
```
- **Numerator** (eligible items with a *current-model* embedding, per type):
```sql
SELECT mi.type, COUNT(*) AS vectorized
FROM media_item_embeddings e
JOIN media_items mi ON mi.content_id = e.media_item_id
WHERE NOT EXISTS (SELECT 1 FROM manga_chapters mc WHERE mc.chapter_content_id = mi.content_id)
AND ($1::text[] IS NULL OR mi.type = ANY($1))
AND (mi.status = 'matched' OR mi.type IN ('audiobook','ebook')) -- ELIG("mi")
AND ($2 = '' OR e.model = $2)
GROUP BY mi.type
```
Applying `ELIG` to **both** guarantees numerator ⊆ denominator → ratio ∈ `[0,1]` (an embedding left behind on an item that became unmatched can't push a type over 1.0). Unmatched items are excluded from both, so they never drag coverage. As backfill completes, ratio → 1.0.
**Active model (`$2`)** is read from the embedding lock (`recommendations.EmbeddingLock.Model`, `embedding_lock.go:16`). No lock ⇒ `$2 = ""` and the snapshot is marked not-ready (never gate semantic on for an unestablished embedding space).
> **Why model-filtering is load-bearing:** right after a model change, old vectors persist but are mismatched to the new query embedder. A model-blind count reads ~100% at the most dangerous moment. Filtering by the locked model collapses coverage to ~0 until re-embedding catches up.
**Index (migration, in scope):** `media_item_embeddings` has only an HNSW index on `embedding` (`migrations/sql/001_schema.sql:1857`, `014_...:25`) — nothing on `model`. Add `CREATE INDEX CONCURRENTLY idx_media_item_embeddings_model ON public.media_item_embeddings (model)` so the numerator filter is index-supported. (Goose: timestamped file via `make migrate-create`, with `-- +goose NO TRANSACTION` because `CONCURRENTLY` cannot run in a txn.)
### D2 — Coverage lives in an in-memory snapshot, refreshed off the request path
A `semanticCoverageTracker` (new, `internal/catalog/semantic_coverage.go`) holds an `atomic.Pointer[semanticCoverageSnapshot]`.
- **Refresh cadence:** a single background goroutine every `semanticCoverageRefreshInterval` (**2 minutes** — long enough that two `GROUP BY` aggregates are negligible steady-state load even at 1M rows; short enough to bound the post-model-change danger window, which also fails safe via the model-collapse rule below). The interval is a named const; **there is no after-sync trigger** (the indexer and service share no handle — see Descoped rationale; the interval alone is sufficient and avoids a cross-component wiring + race).
- **Single-flight & immutability:** `Refresh` holds a `sync.Mutex` across read-prev → query → compute → publish (background path; contention irrelevant). Each refresh builds a **fresh** `PerType` map and publishes a new immutable snapshot pointer; the published snapshot is **never mutated** after publish (avoids a `map` data race against concurrent `CoverageReady` reads). A `-race` test fires refreshes concurrently with reads.
- **Fail-safe reads:** `CoverageReady` reads only the atomic pointer — **no DB, no lock, no remote call.** If the snapshot is `nil` (boot, before first refresh) it returns `(false, "coverage not yet computed")` — **never** dereferences a nil map (that would be a hot-path panic).
- **Error retention:** on a refresh query error, retain the last-good snapshot (transient DB blips must not flip semantic off); log; do not publish a zeroed snapshot. Admin status flags `UpdatedAt` staleness.
- **Model-change collapse:** the snapshot records the `Model` it was computed for. When `Refresh` observes the active model differs from the published snapshot's model, it **immediately publishes an all-not-ready snapshot** (safe) and recomputes real coverage under the new model on the same pass. This bounds the mismatched-vector window to ≤ one interval and fails safe within it.
### D3 — Per-type readiness with hysteresis; scope = AND of its types; empty scope = all types
Each type's `ready` flag is **latched**: flips **true** at `ratio >= semanticCoverageEnableRatio` (0.90), flips **false** only at `ratio < semanticCoverageDisableRatio` (0.80). The snapshot carries prior latched state so each refresh applies hysteresis — preventing flap (identical queries returning differently-ranked results).
`CoverageReady(itemTypes)`:
- **Non-empty scope:** ready iff **every** requested type is latched-ready (`min` semantics — a weak subtype blocks the mixed scope). The reason names the first failing type.
- **Empty/nil scope** (the common "search everything" case — `MediaScopeItemTypes("") → nil`, so `ItemTypes == nil` reaches the gate): ready iff **every type in the snapshot** is latched-ready. This deliberately mirrors the empty-config semantics and refuses to let an unscoped search bypass a weak type. Reason names the first failing type, or `"coverage not yet computed"` if the snapshot is nil/empty.
### D4 — Diagnostics: existing fields, plumbed, reported post-downgrade, scoped to the provider path
- Add `Mode string` (`"keyword"`/`"hybrid"`) and `SemanticUsed bool` to `CatalogSearchResult`; the Meili provider sets them to reflect the **final** request after the internal hybrid→keyword downgrade.
- Add `Provider`, `Mode`, `SemanticUsed`, `FallbackReason` to `CatalogResult`; copy them in `resolveDirectSearchSource`.
- Add an `omitempty` `search_diagnostics` object to `catalogResponse`.
- **Scope:** diagnostics are produced **only** on the direct provider-backed search path (`useDirectSearchPath` = relevance sort, desc, no advanced rules — `catalog_resolver.go:1416-1427`). They are **omitted** for: browse, non-relevance-sorted `q=` searches (which never hit the provider), and `group=work` responses (the grouped wrapper builds a fresh `CatalogResult` at `catalog.go:287-293` and is already approximate — `TotalExact=false`). Each exclusion gets an explicit handler test.
### D5 — Capability validation: narrow, rate-limited, non-fatal, admin-path only
Embedder settings are written by Silo itself, so validation guards only **external drift** (Meili volume reset, manual edits). Add `GetSettings` to the Meili client; verify the configured embedder exists, `source == "userProvided"`, and `dimensions == embeddingvectors.CanonicalDimensions`. The hybrid probe:
- runs **only on the admin status path** (`HandleGetCatalogSearchStatus`), **not** on `CheckConnection`/health (which only does `GET /health`), and the admin query has no `refetchInterval` (`settings.ts`, `staleTime: 15_000`);
- runs **only when `state.ActiveIndexUID != ""`**, and is **cached for `semanticCapabilityProbeTTL` (5m)** → ≤ 1 probe / 5m regardless of refresh rate;
- uses a **non-degenerate unit sample vector** (`v[0]=1.0`, rest 0, length `CanonicalDimensions`) — an all-zero vector has zero norm and can spuriously fail cosine hybrid; targets `ActiveIndexUID` with `Hybrid.Embedder = config.Embedder`, `limit:1`.
Failures surface in admin status and force keyword-only; they **never** flip provider `Healthy` or trip the circuit.
### D6 — Backfill split: cheap discovery vs. bounded text-staleness, with fairness
The expensive part of `ListEmbeddingTextCandidates` is **building `current_text` for every eligible row** via 5 `LATERAL` joins to `item_people` (`recommendations/repo.go:544-589`) — *not* the `IS DISTINCT FROM` comparison.
- **Cheap discovery** (`ListMissingOrModelStaleEmbeddingIDs`): `LEFT JOIN media_item_embeddings`, filter `ELIG("mi") AND (e.media_item_id IS NULL OR e.model = '' OR e.model <> $currentModel)`, `ORDER BY mi.content_id`, `LIMIT`. **No `LATERAL`, no `current_text`.** The SQL is built by a package-private `buildMissingOrModelStaleEmbeddingIDsSQL() string` helper so a unit test can assert it contains neither `item_people` nor `LATERAL` (mirrors the existing `embeddingEligibilityWhereClause()` string-helper test pattern). Returns IDs; `current_text` is then built **only for that bounded batch** via `BuildEmbeddingTextForIDs(ids)`.
- **Text-staleness** (existing `ListEmbeddingTextCandidates`): keep, but run as a **bounded, unconditional quota per cycle** (`embeddingTextStaleQuotaPerRun`) so canonical-text-stale items progress even when the cheap backlog never empties (fairness against starvation).
- **Cursors:** the cheap pass advances `afterID` from the **cheap query's** last id (it is the ordered cursor), independent of `BuildEmbeddingTextForIDs` row order (`ANY($1)` does not preserve slice order, and a dropped id must not desync the cursor). The text-stale pass keeps its **own** `afterID`. Both monotonic.
---
## File Structure
| File | Responsibility | Change |
| --- | --- | --- |
| `migrations/sql/<ts>_media_item_embeddings_model_index.sql` | `(model)` btree, `CONCURRENTLY` / `NO TRANSACTION` | Create |
| `internal/embeddingvectors/eligibility.go` | `ItemEligibilityWhereClause(alias)` — single source of truth | Create |
| `internal/catalog/semantic_coverage.go` | tracker, snapshot, querier seam, per-type SQL, hysteresis, single-flight refresher | Create |
| `internal/catalog/search_provider.go` | gate + model-provider interfaces; `Mode`/`SemanticUsed`; semantic status structs | Modify |
| `internal/catalog/search_meilisearch_provider.go` | consult gate; set `Mode`/`SemanticUsed` (incl. downgrade); capability probe (unit vector) | Modify |
| `internal/catalog/search_meilisearch_client.go` | `GetSettings` + settings DTOs | Modify |
| `internal/catalog/search_service.go` | build/own tracker; `StartCoverageRefresh`; comma-ok model-provider; `Status()` reads snapshot | Modify |
| `internal/catalog/search_indexer.go` | `ELIG`+`model` numerator & eligible denominator; update **all 4** `countCatalogSearchVectorDocuments` callers | Modify |
| `internal/catalog/catalog_resolver.go` | carry diagnostics through `CatalogResult` (direct path only) | Modify |
| `internal/recommendations/repo.go` | delegate eligibility to `embeddingvectors`; `ListMissingOrModelStaleEmbeddingIDs` (+SQL helper); `BuildEmbeddingTextForIDs` | Modify |
| `internal/recommendations/similar.go` | `EmbedAll`: cheap drain + unconditional text-stale quota; dual cursors | Modify |
| `internal/recommendations/engine.go` | `ActiveEmbeddingModel(ctx)` from the lock | Modify |
| `internal/api/handlers/catalog.go` | `searchDiagnostics` DTO; populate on direct path; omit for grouped | Modify |
| `internal/api/router.go` | start `service.StartCoverageRefresh(deps.AppContext)` inside `NewRouter` | Modify |
| `web/src/hooks/queries/admin/settings.ts`, `web/src/pages/admin-settings/SearchSettings.tsx` | surface readiness/ratio/per-type/capability | Modify |
---
## Tasks
### Task 1: Shared eligibility predicate, model+eligibility coverage counts, supporting index
**Files:** Create `internal/embeddingvectors/eligibility.go`, `migrations/sql/<ts>_media_item_embeddings_model_index.sql`; Modify `internal/recommendations/repo.go:150-160`, `internal/catalog/search_indexer.go:757-782` (+ callers `:133,:203,:310`, `search_service.go:139`); Test `internal/embeddingvectors/eligibility_test.go`, `internal/catalog/search_indexer_test.go`.
**Interfaces — Produces:**
- `embeddingvectors.ItemEligibilityWhereClause(alias string) string`
- `countCatalogSearchVectorDocuments(ctx, q coverageQuerier, itemTypes []string, model string) (int, error)` (adds `model`; applies `ELIG`)
- `catalogSemanticCoverageByType(ctx, q coverageQuerier, itemTypes []string, model string) ([]catalogTypeCoverage, error)` → `{Type string; Eligible, Vectorized int}`
- [ ] Migration: `make migrate-create NAME=media_item_embeddings_model_index`; body uses `-- +goose NO TRANSACTION` then a self-healing guard for a leftover invalid index from a crashed build (follow the `20260618164519_add_episodes_season_id_index.sql` precedent: a `DO $$ ... NOT i.indisvalid ... DROP INDEX $$` block), then `CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_media_item_embeddings_model ON public.media_item_embeddings (model);` (down: `DROP INDEX CONCURRENTLY IF EXISTS`). The runner honors `NO TRANSACTION` (three shipping precedents). Run `make migrate-up`; verify with `make migrate-status`.
- [ ] Move the eligibility predicate to `embeddingvectors`; refactor `recommendationItemEligibilityWhereClause` to delegate (assert output string unchanged).
- [ ] Failing tests: (a) `model` filter excludes rows whose `e.model` differs; (b) an **unmatched, non-book** item inflates neither numerator nor denominator (the C1 regression guard); (c) denominator counts eligible items with no embedding row; (d) ratio ∈ [0,1] when a stale embedding exists on a now-unmatched item.
- [ ] Implement both queries with `ELIG`; merge per type in Go; update **all four** callers (pass active model where available, else `""`).
- [ ] `GOWORK=off go test ./internal/embeddingvectors ./internal/catalog -run 'Eligib|Coverage' -count=1` → PASS. Commit.
### Task 2: Active embedding model from the lock
**Files:** Modify recommendations `Engine` (`engine.go`), `internal/catalog/search_provider.go`; Test `internal/recommendations/*_test.go`.
**Produces:** `recommendations: func (e *Engine) ActiveEmbeddingModel(ctx) (string, error)` (→ `GetEmbeddingLock`; `lock.Model` or `""`); `catalog: type CatalogSemanticModelProvider interface { ActiveEmbeddingModel(ctx context.Context) (string, error) }`.
- [ ] Test: lock present → its `Model`; no lock → `("", nil)`.
- [ ] Implement; confirm `recEngine` satisfies both `CatalogSearchQueryVectorizer` (the real interface name, `search_provider.go:77`) and `CatalogSemanticModelProvider`.
- [ ] Run recommendations tests → PASS. Commit.
### Task 3: Coverage tracker — querier seam, snapshot, hysteresis, single-flight refresher
**Files:** Create `internal/catalog/semantic_coverage.go`, `semantic_coverage_test.go`; Modify `internal/catalog/search_provider.go` (consts + gate interface).
**Produces:**
```go
const (
semanticCoverageEnableRatio = 0.90
semanticCoverageDisableRatio = 0.80
semanticCoverageRefreshInterval = 2 * time.Minute
)
// coverageQuerier is the seam so Refresh is unit-testable without a real *pgxpool.Pool.
type coverageQuerier interface {
QueryRow(ctx context.Context, sql string, args ...any) pgx.Row
Query(ctx context.Context, sql string, args ...any) (pgx.Rows, error)
}
type catalogTypeCoverage struct { Type string; Eligible, Vectorized int; Ratio float64; Ready bool }
type semanticCoverageSnapshot struct { PerType map[string]catalogTypeCoverage; Overall float64; Model string; UpdatedAt time.Time }
type SemanticCoverageGate interface { CoverageReady(itemTypes []string) (ready bool, reason string) }
type semanticCoverageTracker struct { /* q coverageQuerier; indexTypes []string; models CatalogSemanticModelProvider; mu sync.Mutex; snap atomic.Pointer[semanticCoverageSnapshot]; clock func() time.Time */ }
func (t *semanticCoverageTracker) Refresh(ctx context.Context) error
func (t *semanticCoverageTracker) CoverageReady(itemTypes []string) (bool, string) // nil-snapshot ⇒ (false, "coverage not yet computed")
func (t *semanticCoverageTracker) Snapshot() *semanticCoverageSnapshot
func (t *semanticCoverageTracker) Run(ctx context.Context) // ticker; returns on ctx.Done()
```
- [ ] Test (no panic / fail-safe): `CoverageReady` on a freshly-constructed tracker (no `Refresh` yet) returns `(false, "coverage not yet computed")` and touches the querier **zero** times (querier stub fails the test if called).
- [ ] Test (hysteresis): 0.85→not ready; →0.92 ready; →0.85 **stays** ready; →0.79 not ready.
- [ ] Test (scope): `CoverageReady(["movie","series"])` false when `series` < threshold though `movie` ready; reason names `series`. Empty scope requires all snapshot types ready.
- [ ] Test (model collapse): snapshot computed for model A; active model becomes B → next `Refresh` publishes all-not-ready before recomputing.
- [ ] Test (`-race`): concurrent `Refresh` + `CoverageReady` loops; assert monotonic `UpdatedAt`, no race.
- [ ] Test (error retention): a `Refresh` whose query errors retains the prior snapshot.
- [ ] Implement single-flight mutex, fresh-map immutability, nil-safe reads, model-collapse, retain-last-good, ticker `Run`.
- [ ] `GOWORK=off go test ./internal/catalog -run Coverage -race -count=1` → PASS. Commit.
### Task 4: Wire gate + model provider; comma-ok degradation; start refresher in NewRouter
**Files:** Modify `internal/catalog/search_meilisearch_provider.go:27-66,329-334` (`MeilisearchProviderConfig` opens at `:27`), `internal/catalog/search_service.go:31-74`, `internal/api/router.go:437-552`.
**Consumes:** `SemanticCoverageGate`, `CatalogSemanticModelProvider`. **Produces:** `MeilisearchProviderConfig.Coverage SemanticCoverageGate`; `func (s *CatalogSearchService) StartCoverageRefresh(ctx context.Context)`.
- [ ] `shouldUseSemanticSearch` → `SemanticEnabled && wordCount>=3 && (p.config.Coverage == nil || ready)`; not-ready yields fallback reason `"semantic_not_ready: " + reason` (a **diagnostic, not an error**). Keep the `SkipTotal`-builds-hybrid test green.
- [ ] In `NewCatalogSearchServiceFromSettings`: only when `SemanticEnabled`, derive the model provider via **comma-ok** `mp, ok := queryVectorizer.(CatalogSemanticModelProvider)`. If `!ok` or `queryVectorizer == nil` (semantic enabled but recommendations disabled — a real config), build the tracker with a model provider that yields `""` ⇒ every type not-ready (reason `"embedding model provider unavailable"`); **never** assert without comma-ok (nil-interface assertion panics). Store the tracker on the service; pass it as `Coverage`.
- [ ] `StartCoverageRefresh` runs `tracker.Run(ctx)` once; no-op when there is no tracker (postgres/semantic-off). Call it from `NewRouter` after the service is built: `if catalogSearchService != nil { catalogSearchService.StartCoverageRefresh(deps.AppContext) }`. (The service is local to `NewRouter` and not returned; `deps.AppContext` is the process-lifetime ctx — Baseline #8/#9.)
- [ ] Test: stub gate not-ready ⇒ long query stays keyword-only; ready gate + vectorizer ⇒ hybrid emitted; nil vectorizer + semantic-enabled ⇒ not-ready, no panic.
- [ ] `GOWORK=off go test ./internal/catalog -count=1` → PASS. Commit.
### Task 5: Diagnostics plumbing (provider → resolver → handler), correctly scoped
**Files:** Modify `internal/catalog/search_provider.go:64-71`, `internal/catalog/search_meilisearch_provider.go:189-237`, `internal/catalog/catalog_resolver.go:24-30,269-297`, `internal/api/handlers/catalog.go:55-61,91-134`; Test `search_provider_test.go`, `catalog_test.go`.
**Produces:** `CatalogSearchResult{+Mode string; +SemanticUsed bool}`; `CatalogResult{+Provider,Mode string; +SemanticUsed bool; +FallbackReason string}`; handler `type searchDiagnostics struct { Provider, Mode string; SemanticUsed bool; FallbackReason string \`json:"fallback_reason,omitempty"\` }`; `catalogResponse{+SearchDiagnostics *searchDiagnostics \`json:"search_diagnostics,omitempty"\`}`.
- [ ] Provider: `Mode="hybrid", SemanticUsed=true` only when the hybrid request is issued **and survives**; on the downgrade (`:221-228`) set `Mode="keyword", SemanticUsed=false, FallbackReason=semanticFallback`.
- [ ] Resolver: copy the four fields in `resolveDirectSearchSource` (direct path only).
- [ ] Handler: populate `search_diagnostics` only on the non-grouped direct path; **omit** for `group=work` and for non-relevance-sort `q=` requests.
- [ ] Tests: (a) hybrid error ⇒ `semantic_used:false, mode:"keyword", fallback_reason` set (post-downgrade truth); (b) existing `total/total_exact/has_more/items/snapshot` byte-stable; (c) field **absent** for browse, for `group=work`, and for a `q=`+title-sort request.
- [ ] `GOWORK=off go test ./internal/catalog ./internal/api/handlers -count=1` → PASS. Commit.
### Task 6: Capability validation + rate-limited probe; admin status from snapshot
**Files:** Modify `internal/catalog/search_meilisearch_client.go:167+`, `internal/catalog/search_meilisearch_provider.go`, `internal/catalog/search_service.go:90-145`, `internal/catalog/search_provider.go` (status structs).
**Produces:**
```go
func (c *meilisearchClient) GetSettings(ctx, uid string) (meilisearchIndexSettings, error) // GET /indexes/{uid}/settings
type meilisearchIndexSettings struct { Embedders map[string]meilisearchEmbedderSettings `json:"embedders"` }
type meilisearchEmbedderSettings struct { Source string `json:"source"`; Dimensions int `json:"dimensions"` }
type CatalogSearchSemanticStatus struct {
Ready bool `json:"ready"`; DisabledReason string `json:"disabled_reason,omitempty"`
CoverageRatio float64 `json:"vector_coverage_ratio"`; CoverageUpdatedAt *time.Time `json:"coverage_updated_at,omitempty"`
PerType []CatalogSearchTypeCoverage `json:"per_type,omitempty"`; Capability CatalogSearchSemanticCapability `json:"capability"`
}
type CatalogSearchTypeCoverage struct { Type string `json:"type"`; Eligible int `json:"eligible"`; Vectorized int `json:"vectorized"`; CoverageRatio float64 `json:"vector_coverage_ratio"`; Ready bool `json:"ready"` }
type CatalogSearchSemanticCapability struct { OK bool `json:"ok"`; Reason string `json:"reason,omitempty"`; Embedder string `json:"embedder,omitempty"`; Dimensions int `json:"dimensions,omitempty"` }
const semanticCapabilityProbeTTL = 5 * time.Minute
```
- [ ] Add `Semantic CatalogSearchSemanticStatus` to `CatalogSearchRuntimeStatus`. **Keep** `document_count`/`vector_document_count` unchanged (additive only). The `Semantic` block (ratio, per-type, ready) is read from `tracker.Snapshot()` — **no fresh query** in `Status()`; if no tracker, `Ready=false, DisabledReason="semantic disabled"`.
- [ ] Capability: embedder present, `source=="userProvided"`, `dimensions==embeddingvectors.CanonicalDimensions`; each failure → distinct `Reason`. Probe (unit vector, `limit:1`, `Embedder=config.Embedder`) only when `ActiveIndexUID != ""`, cached `semanticCapabilityProbeTTL`.
- [ ] Tests: missing embedder / wrong source / wrong dimensions / probe failure each distinct; provider `Healthy` and circuit untouched in all four.
- [ ] `GOWORK=off go test ./internal/catalog -count=1` → PASS. Commit.
### Task 7: Backfill cheap/expensive split with fairness
**Files:** Modify `internal/recommendations/repo.go:472-621`, `internal/recommendations/similar.go:251-362`; Test `repo_test.go`, `similar_test.go`.
**Produces:** `buildMissingOrModelStaleEmbeddingIDsSQL() string`; `ListMissingOrModelStaleEmbeddingIDs(ctx, afterID, currentModel string, limit int) ([]string, error)`; `BuildEmbeddingTextForIDs(ctx, ids []string) ([]EmbeddingTextCandidate, error)`; `EmbedAll` two-pass loop with dual cursors and an unconditional `embeddingTextStaleQuotaPerRun`.
- [ ] Test (SQL shape): `buildMissingOrModelStaleEmbeddingIDsSQL()` contains neither `"item_people"` nor `"LATERAL"`.
- [ ] Test (behavior, no real DB needed for the shape test; behavior tests use existing recommendations test seams): an item that is *only* canonical-text-stale (people changed, model current) is **not** returned by the cheap query; a missing/model-stale item **is**.
- [ ] Test (parity): `BuildEmbeddingTextForIDs(ids)` reproduces the same `canonical_text` as the legacy full CTE for the same ids, regardless of `ids` order.
- [ ] Test (no starvation): with a permanently non-empty cheap backlog, a text-stale item is still embedded within a bounded number of `EmbedAll` runs (quota is unconditional); both cursors stay monotonic.
- [ ] Implement; cheap `afterID` advances from the cheap query's last id.
- [ ] `GOWORK=off go test ./internal/recommendations -count=1` → PASS. Commit.
### Task 8: Admin UI surfacing
**Files:** Modify `web/src/hooks/queries/admin/settings.ts:21-54`, `web/src/pages/admin-settings/SearchSettings.tsx:202-261`.
- [ ] Extend the `CatalogSearchStatus` TS type with the `semantic` block (ready, disabled_reason, vector_coverage_ratio, coverage_updated_at, per_type, capability).
- [ ] Render a "Semantic readiness" row (ready/disabled + reason), overall coverage %, a per-type coverage table, a capability badge, and a "coverage updated" timestamp. Keep "Vectorized Documents" (`vector_document_count`).
- [ ] `cd web && pnpm run lint && pnpm run format:check` → PASS. Commit.
---
## Test Plan
**Go unit (`internal/embeddingvectors`, `internal/catalog`):**
- Eligibility predicate single-sourced; unmatched non-book item excluded from numerator **and** denominator (C1 guard); ratio ∈ [0,1].
- Hybrid omitted when gate not ready; emitted when ready + capable + long query + vectorizer present.
- **Model-stale safety:** only-old-model embeddings ⇒ coverage ~0 ⇒ keyword-only; model-change collapse publishes not-ready within one interval.
- Hysteresis latch; mixed-scope AND; empty-scope = all-types; nil-snapshot returns not-ready with **no panic** and **no DB call**; `-race` refresh/read; error retention.
- Capability: missing embedder / wrong source / wrong dimensions / probe failure each distinct; keyword stays up.
**Go unit (`internal/recommendations`):**
- Cheap SQL excludes `item_people`/`LATERAL`; people-only-stale row excluded from cheap pass; per-ID text build matches legacy output order-independently; no text-stale starvation; dual cursors monotonic.
**Go unit (`internal/api/handlers`):**
- `search_diagnostics` present on direct path; **absent** for browse, `group=work`, and non-relevance-sort `q=`; `semantic_used` reflects post-downgrade reality; existing fields byte-stable.
**Admin status:** `semantic.ready`, `disabled_reason`, `vector_coverage_ratio`, `per_type`, `capability`, `coverage_updated_at` sourced from the snapshot (no fresh query).
**Focused command:**
```bash
GOWORK=off go test ./internal/embeddingvectors ./internal/catalog ./internal/api/handlers ./internal/recommendations -race -count=1
```
---
## Descoped — not in this pass
**Routing legacy `/search?q=` through the provider pipeline.** Sunset **Wed, 01 Jul 2026** (`internal/api/handlers/legacy_read_routes.go:10`, `legacyReadSunset`) — days away; hardening a path scheduled for deletion is negative ROI. The typed legacy path already routes through the resolver; only the bare-`q` fallback hits `itemRepo.Search`, which the sunset removes. Re-routing also risks `browseResponse`↔`catalogResponse` shape drift. Reopen as a one-task follow-up **only if the sunset slips**, with a byte-for-byte `browseResponse` assertion.
**After-sync coverage refresh trigger.** The indexer (`main.go:1730`) and the search service (`router.go:457`) share no handle, so an after-drain trigger would need new cross-component wiring and would race the interval refresh. The 2-minute interval plus model-change collapse is sufficient; the trigger is intentionally omitted.
---
## Assumptions
- **One additive index migration** (`media_item_embeddings(model)`, `CONCURRENTLY`). No schema/column changes. *(Revised from the original "no migration": the model-filtered numerator is only cheap with this index, and performance is priority #1.)*
- All runtime coverage state is **in-memory** (atomic per-type snapshot), rebuilt on boot and refreshed every 2 minutes; not persisted.
- Coverage is computed **entirely from Postgres** over the **embed-eligible** population (`status='matched' OR audiobook/ebook`), model-filtered, manga excluded, restricted to configured index types; it **never** uses the Meili `document_count`.
- Active embedding model comes from `recommendations.EmbeddingLock.Model`; **no lock / no model provider ⇒ semantic not ready.**
- A model change is reflected within ≤ one refresh interval and **fails safe** within that window via the model-collapse rule; the residual ≤2-minute mismatch window is acceptable for a rare, admin-gated reconfiguration.
- Thresholds (enable 0.90 / disable 0.80) and the 2-minute interval are named consts and starting heuristics; the diagnostics shipped here are the instrument for retuning them. Promoting them to settings is a possible follow-up, not this pass.
- `semantic_ratio` stays `0.30` (`DefaultMeilisearchSemanticRatio`).
- Scope is hybrid-search hardening only — not conversational search.
---
## Self-Review (writing-plans checklist)
- **Findings coverage:** C1 eligibility denominator (D1, Task 1) · C2 empty-scope semantics (D3, Task 3) · missing `model` index (D1 migration, Task 1) · 4-caller signature change (Baseline #4, Task 1) · refresher wiring in `NewRouter`/`AppContext` (Baseline #9, Task 4) · concurrent-refresh single-flight + immutability + `-race` (D2, Task 3) · nil-snapshot/error fail-safe (D2, Task 3) · model-change collapse (D2, Task 3) · comma-ok nil vectorizer (Task 4) · querier seam + SQL-string helper for testability (Task 3, Task 7) · grouped/non-relevance diagnostics scoping (D4, Task 5) · unit probe vector + admin-only/5m (D5, Task 6) · `Status()` reads snapshot (Task 6) · backfill cursors + fairness (D6, Task 7) · legacy & after-sync descoped (Descoped) — all mapped.
- **Type consistency:** `SemanticCoverageGate`/`CatalogSemanticModelProvider`/`coverageQuerier`/`semanticCoverageTracker`/`catalogTypeCoverage`/`CatalogSearchSemanticStatus` used identically across tasks; `Eligible` (not `Indexable`) is the denominator field name throughout.
- **No placeholders:** every new symbol has a signature; SQL is concrete; the migration body is specified.
@@ -0,0 +1,248 @@
# Unified Literary Works Design
## Goal
Silo should expose a server-owned literary work identity that links multiple format-specific catalog items for the same book, especially ebooks and audiobooks, without collapsing those items into one `media_items` row. Clients can then render one detail page with format tabs/actions while existing item detail, playback, reading, download, and catalog APIs keep working.
The implementation should target the ebook stack head, currently `work/ebook-reader-ruler-profiles`, which includes the ebook foundation, #98 local-cover/grouping work, and #99 reader profile/ruler work. That stack already has ebook scanning, ebook series metadata, ebook reader progress, and ebook detail extensions. The work layer should sit above those existing format-specific systems.
## Existing Context
`media_items.content_id` is the stable identity for each catalog item. Audiobooks are `media_items.type = 'audiobook'`; ebooks are introduced as `media_items.type = 'ebook'` in the ebook stack. These identities must remain independent because each format has different files, progress, playback/reader behavior, external compatibility, and library membership.
Audiobooks already store author and narrator credits in `item_people`, series data in `audiobook_series`, file data in `media_files`, and listening progress in existing playback/watch progress tables. The ebook stack adds ebook author credits, `ebook_series`, `ebook_reader_progress`, reader state/config, and ebook file serving. #98 also groups sibling ebook file formats by book identity during scan; that is an intra-ebook item grouping behavior, not the cross-format work identity.
The current catalog detail endpoint is `GET /api/v1/catalog/items/{id}` through `CatalogResourceHandler.HandleGetItemDetail`, backed by `catalog.DetailService.GetItemDetail`. Catalog browse/search remains item-oriented by default through catalog query/browse code.
## Data Model
Add a neutral work layer:
```sql
CREATE TABLE literary_works (
work_id TEXT PRIMARY KEY,
canonical_title TEXT NOT NULL,
sort_title TEXT,
normalized_title TEXT NOT NULL,
primary_author_key TEXT NOT NULL DEFAULT '',
primary_cover_content_id TEXT REFERENCES media_items(content_id) ON DELETE SET NULL,
description TEXT,
published_date DATE,
publisher TEXT,
genres TEXT[] NOT NULL DEFAULT '{}',
metadata JSONB NOT NULL DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE TABLE literary_work_items (
work_id TEXT NOT NULL REFERENCES literary_works(work_id) ON DELETE CASCADE,
content_id TEXT NOT NULL REFERENCES media_items(content_id) ON DELETE CASCADE,
format_type TEXT NOT NULL,
link_source TEXT NOT NULL,
confidence DOUBLE PRECISION NOT NULL DEFAULT 1,
confirmed_at TIMESTAMPTZ,
ignored_at TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
PRIMARY KEY (work_id, content_id),
UNIQUE (content_id),
CHECK (format_type IN ('ebook', 'audiobook', 'comic', 'manga')),
CHECK (link_source IN ('manual', 'external_id', 'metadata_match', 'series_match', 'scan_seed'))
);
CREATE TABLE literary_work_match_decisions (
source_content_id TEXT NOT NULL REFERENCES media_items(content_id) ON DELETE CASCADE,
target_content_id TEXT NOT NULL REFERENCES media_items(content_id) ON DELETE CASCADE,
decision TEXT NOT NULL,
created_by INTEGER REFERENCES users(id) ON DELETE SET NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
PRIMARY KEY (source_content_id, target_content_id),
CHECK (source_content_id <> target_content_id),
CHECK (decision IN ('confirmed', 'ignored'))
);
```
Every ebook or audiobook can belong to at most one work through `UNIQUE (content_id)`. A work can contain many format-specific items. Manual/admin actions update `literary_work_items` and `literary_work_match_decisions`; automatic matching must not override an ignored pair or move a manually linked item without an explicit admin request.
Work IDs should be generated by the existing server ID generator, not derived directly from title text. Matching keys can change when metadata improves, but the `work_id` remains stable.
## Matching Rules
Matching runs in a `literaryworks` service after scan/enrichment or as an admin-triggered repair job. It considers literary item types only: `ebook`, `audiobook`, and later `comic`/`manga`.
Authority order:
1. Manual confirmed links and manual unlinks/ignored pairs.
2. Shared external IDs from `media_item_provider_ids`, excluding format-specific aliases that are not true work identity.
3. Normalized title plus normalized author.
4. Normalized series name plus series index plus author.
5. Publication year/date and publisher as supporting evidence only.
Automatic matching should be conservative. Same title alone is not enough. Same title plus different primary author should not link. Same title plus same author can link unless there is strong conflicting series/index evidence. Same series/index plus same author can link even if titles differ slightly due subtitle/edition formatting. Narrator is never required for ebook matching and must not prevent an ebook/audiobook work link; narrator remains audiobook-format metadata.
The matcher should produce scored candidates. Scores at or above the automatic threshold can create links when neither item has conflicting manual state. Lower scores are exposed as possible matches for admin review.
## Service Boundaries
Create an `internal/literaryworks` package with:
- `Repository`: reads/writes `literary_works`, memberships, decisions, and candidate rows.
- `Matcher`: normalizes item metadata, scores candidates, and applies conservative automatic links.
- `Service`: public orchestration used by handlers and background jobs.
The package depends on catalog repositories and the database pool, but catalog detail should depend on a small interface so item detail can include work data without creating a large import cycle.
The matcher should reuse existing item metadata:
- Titles from `media_items`.
- Authors from `item_people.kind = author`.
- Audiobook narrators from `item_people.kind = narrator` only for display inside audiobook format details.
- Series from `audiobook_series` and `ebook_series`.
- External IDs from `media_item_provider_ids`.
- File metadata from `media_files`.
## API Contract
Add a work endpoint:
`GET /api/v1/works/{work_id}`
Response:
```json
{
"work_id": "work_123",
"work_title": "Project Hail Mary",
"authors": [{ "name": "Andy Weir", "person_id": "42" }],
"formats": [
{
"type": "ebook",
"content_id": "ebook_1",
"library_id": 1,
"available_files": [
{
"file_id": 10,
"original_filename": "Project Hail Mary.epub",
"format": "epub",
"mime_type": "application/epub+zip",
"size": 12345
}
],
"progress": {
"kind": "reading",
"progress": 0.42,
"updated_at": "2026-06-09T12:00:00Z"
}
},
{
"type": "audiobook",
"content_id": "audio_1",
"library_id": 2,
"available_files": [
{
"file_id": 20,
"original_filename": "Project Hail Mary.m4b",
"format": "m4b",
"mime_type": "audio/mp4",
"size": 987654321,
"duration_seconds": 58200
}
],
"progress": {
"kind": "listening",
"position_seconds": 1200,
"duration_seconds": 58200,
"updated_at": "2026-06-09T12:00:00Z"
}
}
],
"primary_cover_url": "/api/v1/images/...",
"metadata": {
"description": "...",
"series": { "name": "The Series", "index": 1 },
"genres": ["Science Fiction"],
"published_date": "2021-05-04",
"publisher": "Ballantine Books"
}
}
```
Extend `GET /api/v1/catalog/items/{content_id}` additively:
```json
{
"content_id": "ebook_1",
"type": "ebook",
"work_id": "work_123",
"work_title": "Project Hail Mary",
"work_formats": [
{ "type": "ebook", "content_id": "ebook_1", "library_id": 1 },
{ "type": "audiobook", "content_id": "audio_1", "library_id": 2 }
]
}
```
Existing fields and behavior remain unchanged. Clients that do not know about works can ignore the new fields.
## Catalog Grouping
Add optional work grouping for literary media:
`GET /api/v1/catalog?type=reading&group=work`
The grouped response should return one card per work when a linked work exists, with format badges such as `ebook` and `audiobook`. Unlinked literary items still appear as single-item cards with an implicit one-item work projection. Non-literary media remain item-based.
Do not change default catalog/search behavior. Work grouping must be opt-in so current clients and Jellyfin/ABS compatibility routes are unaffected.
## Progress and Files
Progress remains per format and per existing content identity:
- Ebook progress reads from `ebook_reader_progress`.
- Audiobook progress reads from existing playback/watch progress storage.
The work response aggregates those format-specific progress rows without merging them.
Available files come from `media_files`, grouped by format content ID. File metadata should include `file_id`, original basename, extension/container, MIME type when known, byte size, and duration for audio. Android can continue using existing file/download routes by `file_id`; the work endpoint only tells it what is available.
## Admin Primitives
Add admin endpoints:
- `GET /api/v1/admin/literary-works/items/{content_id}/matches`: list possible work/item matches with score and evidence.
- `POST /api/v1/admin/literary-works/link`: link two or more content IDs into one work.
- `POST /api/v1/admin/literary-works/{work_id}/unlink`: unlink one content ID from a work.
- `POST /api/v1/admin/literary-works/matches/ignore`: mark a candidate pair ignored.
- `POST /api/v1/admin/literary-works/matches/confirm`: confirm a candidate pair and link them.
Admin linking must validate that every content ID is a supported literary type and accessible to the admin. Unlinking the last item can delete the empty work.
## Backward Compatibility
All existing item detail, catalog, reader, playback, download, Jellyfin compatibility, and ABS compatibility routes remain valid. The new tables cascade from `media_items`, so library deletion and orphan cleanup continue to remove format items; work cleanup should delete empty works after membership deletion.
No caller is required to supply `work_id` for existing format actions. Work APIs are additive and read-oriented except for admin management endpoints.
## Testing Strategy
Repository and matcher tests should cover:
- Ebook and audiobook with same normalized title and author link to one work.
- Same title with different authors does not auto-link.
- Same author and series/index links despite title subtitle differences.
- Shared external ID links above fuzzy metadata.
- Manual ignored pair blocks automatic linking.
- Manual confirmed link wins over automatic scoring.
- Unlink removes one item without deleting the format media item.
- Existing item detail JSON still includes normal fields when no work exists.
- Existing catalog browse/search remains item-based unless `group=work`.
- Work endpoint returns separate ebook and audiobook progress.
- Work endpoint returns file metadata for EPUB/PDF/CBZ and M4B/MP3 without changing download behavior.
Handler tests should assert route status codes and response shape. SQL tests should include cascade/empty-work cleanup behavior.
## Rollout Plan
Implement the schema and service behind additive APIs first. Then wire automatic matching after scan/enrichment. Finally add grouped catalog support. This keeps default behavior stable while giving clients and admin tooling a safe way to adopt works.
@@ -0,0 +1,96 @@
# Server-side Kindle→EPUB conversion (MOBI / AZW / AZW3) — design
**Goal:** Let the silo-server render Kindle-family ebooks (mobi, azw, azw3) in the Android in-app reader by converting them to EPUB **server-side, in-process, on demand**, served transparently through the existing read endpoint. No external converter binary, no cgo.
**Status:** Approach approved by the user: convert via **libmobi**, run **built-in** to the server as **WASM executed by `wazero`** (a pure-Go runtime). On-demand + cached, admin-gated, DRM-free only. This is the MOBI half of the "all 9 server formats in-app" goal; the CBR (client-side) half is a separate Android spec.
> **Linchpin PROVEN (build spike, 2026-06-17, linux/amd64).** libmobi + `mobitool` cross-compile to `wasm32-wasi` (wasi-sdk 25, libmobi `9062742`, zlib 1.3.1 built to wasm); `-e` (create EPUB) is present; the module converts real MOBI6 / KF8-hybrid / HUFF-CDIC / unicode samples to **well-formed EPUB** (`mimetype`=`application/epub+zip` first, `META-INF/container.xml`, `OEBPS/*`). Verified end-to-end under **wazero** (the prod runtime) with WASI preopen + argv + `_start` + exit handling. Reproducible build stage committed at `tools/mobitool-wasm/Dockerfile` (with a smoke-conversion gate). Build-config gotcha resolved: build libmobi against a **wasm-compiled zlib** — `--with-zlib=no` makes libmobi vendor miniz that collides with mobitool's own miniz at `wasm-ld`. **DRM finding:** `mobitool` prints `Document is encrypted` to **stdout** but still **exits 0** and writes a (garbage) EPUB — so detect failure by capturing stdout + validating output, never by exit code alone.
## Why this shape
- Pure-Go MOBI/KF8 *parsers* don't exist at usable quality (the Go MOBI packages only *write*). Calibre is the gold standard but ~300MB+ of Python/Qt — rejected as too heavy. **libmobi** (mature C library behind the `mobitool` CLI) handles MOBI6 + KF8/AZW3 + AZW, decompression, and EPUB generation.
- Rather than shell out to an external `mobitool` binary (a deploy moving part) or bind via cgo (forces `CGO_ENABLED=1`, a C toolchain, painful cross-compile, and an LGPLv3 static-link obligation), we compile **`mobitool` to a WASI `.wasm`** once and **run it in-process with `wazero`**.
- This keeps the main Go build **`CGO_ENABLED=0`** and pure; one **architecture-independent** `.wasm` runs on amd64/arm64 servers alike; the `.wasm` is `go:embed`-ed into the binary (self-contained); libmobi's LGPL is cleanly isolated as a separate embedded artifact; and — a real bonus — **untrusted ebook bytes are parsed inside the WASM sandbox**, so a memory-safety bug in libmobi on a malicious file cannot reach the host (only the mapped temp dir).
## Components
### 1. The WASM module (build-time artifact)
- Compile libmobi + its `mobitool` tool to `wasm32-wasi` using the **wasi-sdk** (clang). Output: `mobitool.wasm`.
- Build libmobi **without libxml2** (`--with-libxml2=no` / `--disable-xmlwriter` as available) so the module has no external native deps; mobitool's EPUB writer uses a bundled miniz, which is portable C and compiles to WASI cleanly. (Feasibility of the no-libxml2 build for the `-e` path is the #1 thing to verify in the build spike — see Risks.)
- Build it in a **dedicated Docker build stage** (or a committed `Makefile`/script under `tools/mobitool-wasm/`). **Commit the resulting `mobitool.wasm`** into the repo so the normal Go build does not need the wasi-sdk — it only `go:embed`s the prebuilt artifact. Rebuild the `.wasm` only when bumping libmobi; record the libmobi version + build flags next to it.
### 2. `internal/ebookconvert` (new package) — the in-process converter
```go
// Converter converts a Kindle-family ebook file to EPUB bytes, in-process via wazero.
type Converter interface {
// Convert reads srcPath (mobi/azw/azw3) and returns the path to a converted .epub.
// Returns ErrDRMProtected / ErrUnsupported / ErrConversionFailed as appropriate.
Convert(ctx context.Context, srcPath string) (epubPath string, err error)
}
```
- `//go:embed mobitool.wasm` → compile the module **once** at startup with `wazero.NewRuntime` + `CompileModule` (the compiled module is cached and reused; only cheap instantiation happens per conversion).
- **wazero command-module specifics** (from Codex review): `wasi_snapshot_preview1.MustInstantiate` the host module once; `Runtime.CompileModule` once at startup; per conversion instantiate with `ModuleConfig.WithArgs(...)` (argv ≈ `mobitool -e -o <outdir> <input>`), `WithFSConfig`, `WithStdout`+`WithStderr` (capture **both** — `mobitool.c` prints many failures with `printf` to stdout, not stderr), and the default `_start`. Use `WithName("")` (or a unique name) for repeated instantiation. Non-zero exits surface as `*sys.ExitError`. Honor ctx timeout via `RuntimeConfig.WithCloseOnContextDone(true)`.
- **Filesystem sandboxing** (from Codex review): wazero's `WithDirMount`/`WithReadOnlyDirMount` still permit relative `../../` traversal *within host permissions*, so **never preopen the library directory or the cache root**. Per conversion: copy the source into a private scratch tree, mount input **read-only via an `fs.FS`** (`WithFSMount(os.DirFS(in), …)` — confined, no `..` escape) and a dedicated empty **writable** output dir (`WithDirMount`). **The WASM sandbox is a memory-safety boundary, not a hard filesystem jail** — a writable dir mount can still be traversed by a compromised guest within the server user's reach. Filesystem isolation therefore depends on a **deployment requirement: run silo-server as a constrained, non-root service user (the container already does), so the converter's blast radius is minimal.** This is documented here and in `internal/ebookconvert/converter.go`.
- Enforce a wall-clock **timeout** (ctx), a **max source size**, and a **concurrency semaphore** (N simultaneous conversions). WASM memory is bounded by wazero config. **Resource note:** the defaults are `DefaultMaxMemoryPages` = 1 GiB linear memory per conversion × `DefaultConcurrency` = 2, i.e. up to ~2 GiB of wasm memory under saturation (plus the 256 MiB source / 512 MiB output caps). Generous for a small server; tune `ebookconvert.Options` for tight deployments.
- Map mobitool's exit code / captured stdout+stderr to typed errors. **Deterministic** verdicts (cacheable): DRM-protected input → `ErrDRMProtected`; corrupt / unconvertible layout / oversize-output → `ErrConversionFailed`. **Transient** verdicts (never negatively cached, retried next read): the per-call timeout firing → `ErrConversionTimedOut`; source over the size cap → `ErrSourceTooLarge`; caller cancellation/deadline → the context error propagated verbatim.
### 3. Conversion cache (on-demand, keyed)
- Cache converted EPUBs on disk under a configured cache root, keyed by **strong source identity + converter identity** — `file_id + size + mtime_ns + content hash` (use the scanner checksum if one exists) **plus** the converter **module fingerprint** (`moduleVersion`: a sha256 of the embedded `mobitool.wasm`, which already pins the libmobi commit, build flags, and wasi-sdk used to produce it). (Plain `path|size|mtime` can false-hit when a file is replaced with preserved mtime/size — Codex.) A stale key misses and reconverts.
- **Singleflight** per key so concurrent reads of the same just-opened book convert once. The shared conversion runs on a **detached context** (decoupled from any one caller), so a single caller cancelling its request never aborts the work for the others.
- **Negative caching**: remember only **deterministic-bad** sources — DRM (`ErrDRMProtected`) and conversion failures (`ErrConversionFailed`) — so every reader open does not reconvert a known-bad file. Transient outcomes (timeout, cancellation, oversize source) are **not** cached so they retry.
- A cache **hit refreshes the entry's mtime**, so the mtime-ordered budget eviction acts as a real LRU; eviction never deletes the entry it is handing back or another conversion's in-flight temp.
- Bounded by an LRU / total-size budget with opportunistic eviction (mirrors the existing reader-file/image cache conventions in the codebase).
### 4. Wiring into the read path
- `serveEbookInline` (`internal/api/handlers/ebook_reader.go:602`) currently `os.Open`s `file.FilePath` and `http.ServeContent`s it with `ebookMimeType(...)`.
- Change: when the source format is **kindle-family AND conversion is enabled**, resolve the converted EPUB via the cache (convert-on-miss), then serve **that** file with `Content-Type: application/epub+zip`. All other formats are served exactly as today (raw, byte-range, unchanged).
- Byte-range/`http.ServeContent` semantics are preserved — we serve the cached EPUB `*os.File` the same way, so resume/range still works.
- **Response headers** (from Codex review): `Content-Type: application/epub+zip`, `Content-Disposition` with an `.epub` filename, a strong `ETag` derived from the conversion cache key, and — because the *same URL* can return raw or converted bytes depending on the setting/failure — `Cache-Control: private, max-age=0, must-revalidate` (or `no-store`) to prevent stale client/proxy caching of the wrong representation.
- **HEAD is cache-only.** A `HEAD /read` must never trigger a (possibly minute-long, ~1 GiB) conversion. It answers from the cache: a hit serves the real converted headers + `Content-Length`; a negatively-cached source serves the `failed` contract; a miss advertises the converted headers (`Content-Type`, the cache-key `ETag`, `X-Silo-Ebook-Conversion: converted`) **optimistically with no body and no conversion** — the subsequent `GET` produces the body and the authoritative DRM/failure verdict, which then populates the cache that later HEADs read. *(Client note: a HEAD miss intentionally omits `Content-Length`; coordinate with the Android/Apple clients if any relies on it.)*
### 5. Admin gate + client capability
- An **admin-toggleable setting** (`ebook.kindle_conversion_enabled`, default off) gates the whole feature, via the existing server settings mechanism (`internal/api/handlers/settings.go` / `admin.go` `HandleUpdateSetting`). The flag is read with a short TTL cache so the hot read path (and the capability endpoint clients poll) does not hit the DB per request, while toggles still apply within seconds without a restart. When off: kindle-family files serve raw as today and the capability is not advertised.
- **Two gates to advertise the capability** (from Codex review): advertise conversion availability to clients **only when** (a) the admin flag is on **and** (b) the embedded WASM module compiled + instantiated successfully at startup. The Android client flips mobi/azw/azw3 from `ExternalOnly`→`InApp` **only when the flag is advertised**; otherwise it keeps today's external-open fallback. (Android-side change is small and tracked separately; this spec is server-side.)
- **Capability endpoint (implemented):** `GET /api/v1/ebooks/capability` → `{enabled, source_formats:["mobi","azw","azw3"], served_format:"epub", header:"X-Silo-Ebook-Conversion", header_failed_value:"failed"}`. `enabled` is true only when both gates hold (`EbookConversion.active`). The read response carries `X-Silo-Ebook-Conversion: converted|failed` so the client knows whether the body is EPUB or the raw original (→ external-open). ETag on the converted response is derived from the **exact conversion cache key** (`SourceKey.CacheKey()`) so a validator can never disagree with the cache; the failed/raw fallback is `Cache-Control: no-store` since the same URL flips representations with the flag.
## Data flow (read of a `.azw3`, conversion enabled)
1. `GET /api/v1/ebooks/{cid}/files/{fid}/read` → authorize → `file` is `azw3`, conversion enabled.
2. Cache lookup by source identity+version. Hit → serve cached `.epub`. Miss → singleflight: `Converter.Convert` (wazero runs `mobitool.wasm` on a temp copy) → store under key.
3. `http.ServeContent` the `.epub` with `application/epub+zip`, byte-range supported.
4. Client sees EPUB → existing reflow `EpubReader` renders it. Reader progress stays keyed to the **source** `fileId` (the EPUB is a derived view of the same file), so progress/annotations are unaffected.
## Error handling — explicit client failure contract
Transparent raw fallback is **not** safe on its own: once the server advertises conversion and the Android in-app EPUB path opens `/read`, returning raw MOBI/AZW bytes hands the EPUB reader a non-EPUB body (Codex). So failures need a contract the client can detect:
- On conversion failure (DRM, corrupt, timeout, oversize) the read endpoint returns a **structured signal** the Android client maps to its external-open fallback — preferred: a distinct status (e.g. `422 Unprocessable` / `415`) with an error code, **or** serve raw bytes **plus** an explicit `X-Silo-Ebook-Conversion: failed` (with the original MIME) header that the client checks. Pick one and make both sides honor it. Never silently hand raw bytes to the EPUB path.
- `ErrDRMProtected` vs `ErrConversionFailed` are distinguished in logs + (optionally) the error code, so DRM books get the "open externally" message and transient failures can be retried.
- Module compile/instantiate failure at startup (corrupt embed) → capability **not advertised** (gate b), kindle files serve raw as today, logged loudly.
## Testing
- `internal/ebookconvert` unit tests with **small DRM-free MOBI6, AZW, and KF8/AZW3 fixtures**: `Convert` produces a valid EPUB (zip with `mimetype` first + `META-INF/container.xml` + OPF + at least one content doc), deterministic across runs; corrupt input → `ErrConversionFailed`; oversize → rejected before invocation; timeout honored (ctx cancel).
- Cache tests: miss→convert→hit; key changes on source mtime/size and on module-version bump; singleflight collapses concurrent conversions.
- Handler tests: enabled+kindle → `application/epub+zip` body is a valid EPUB; disabled → raw bytes + original MIME; non-kindle → unchanged; DRM/failed → graceful fallback path.
- A build-stage check (CI) that the committed `mobitool.wasm` matches a rebuild from the pinned libmobi (or at least that it instantiates + converts a fixture), so the artifact can't silently rot.
## Sequencing — spike first (Codex-recommended)
**The first PR is only the build spike + a tiny Go harness that converts fixtures** — no handler/cache/wiring until the linchpin is proven. Codex confirmed libmobi documents both deps as optional (`--with-zlib=no` → bundled `miniz.c`; `--with-libxml2=no` → internal xmlwriter) and that `mobitool.c` gates `-e` behind `USE_XMLWRITER`, not `HAVE_LIBXML2` — so the no-libxml2 wasm build is *plausible*, but the spike must **assert** `mobitool -h` exposes `-e` and a fixture actually converts. Only then design the handler/cache/wiring PR.
## Risks / spike-first items
1. **libmobi+mobitool compiles to `wasm32-wasi` with the `-e` (EPUB) path working without libxml2.** The linchpin. If libxml2 turns out mandatory for `-e`, fall back to compiling libxml2 to wasm too (heavier) or assembling the EPUB in Go from libmobi's `MOBIRawml` parts.
2. **wazero WASI preopen file IO** for the file sizes involved (tens of MB) — confirm mapping + memory/time limits.
3. mobitool CLI surface as a wasm **command module** (argv + preopens) — confirm wazero runs it via `_start` and we capture exit status + stdout + stderr.
## Committed `mobitool.wasm` — guardrails (Codex)
Commit the artifact (so normal Go builds need no wasi-sdk) **with**: the pinned libmobi source commit, the Dockerfile/build script that produced it, its SHA256, the build flags, and the LGPL license notice. Add a **CI smoke test** that instantiates the embedded module and converts a fixture. Do not gate CI on byte-for-byte rebuild reproducibility — `mobitool -v` embeds `__DATE__`/`__TIME__`, so a rebuild won't match byte-for-byte unless normalized; rely on the fixture-conversion smoke + metadata hash instead.
## Process gate (Codex)
`docs/architecture/v1-scope.md` indicates v1 scope is not locked and new user-facing capabilities may need a proposal / project-tracking entry **before** a feature PR. Confirm this process step with the maintainer before opening the conversion PR. *(Decision deferred to the user.)*
## Out of scope
- CBR (separate Android client-side spec).
- DRM removal (never).
- Converting at scan/ingest time (on-demand only).
- Surfacing a separate "EPUB version" in the catalog (we serve transparently through `/read`).
- Reusing the WASM converter for other formats (later, if useful).
## Commit / deploy rules (project)
- silo-server changes go on a **feature branch, committed locally, PR-ready — NOT pushed** (per standing rule; server changes land via PR).
@@ -0,0 +1,246 @@
# Personal Collection Catalog Filter Standardization - Design
- **Date:** 2026-06-23
- **Status:** Draft (rev. 3 after adversarial review). Incorporates three findings: `match: any` merge safety, display-query fragment normalization, and ebook misclassification under unscoped watched filters.
- **Scope:** `silo-server` backend and web collection UI. Client follow-ups in `silo-android` and `silo-apple` are optional unless they adopt the same collection editing surface.
- **Note:** All paths are repository-relative. Commands assume the repository root is the cwd.
## Current Branch State
This design lives on the in-flight `profile-scoped-collection-filters` branch, which has **already implemented** the first cut of the feature but has **not merged to `main` and has not shipped to any client**. Concretely, on this branch today:
- A migration adds `watch_filter` and `media_filter` as `text` columns on `user_personal_collections`, each with a CHECK constraint (`migrations/sql/20260623184858_user_collection_display_filters.sql`).
- `internal/catalog/user_collection_display_filters.go` implements a custom Go filter pass (`FilterUserCollectionDisplayItems`) called from `internal/sections/fetcher.go` and `internal/catalog/catalog_resolver.go`.
- Web controls in `web/src/components/collections/CollectionBuilder.tsx`, `web/src/pages/ImportedCollectionEditor.tsx`, and helpers in `web/src/lib/collectionDisplayFilters.ts` already emit the legacy enums.
Because nothing is merged or released, this is a **clean-replacement** opportunity, not a backward-compatibility problem. The migration can be reshaped before merge; no client depends on the legacy enums. This resolves Open Question #1 below.
## Problem
Personal collection display filtering is growing its own filter model alongside two pre-existing completion implementations. There are **three** places that answer "is this item played/read for this profile?":
- the custom watched/media pass in `internal/catalog/user_collection_display_filters.go` (exact user-collection reads),
- the personalized `watched` query rule in `internal/catalog/query_builder.go` (`userStateCompletionClause`),
- the item user-data path in `internal/api/handlers/user_state.go` (`allEpisodesCompleted`), which is the canonical played/read state surfaced to clients.
That creates more than one way to answer the same product question: "should this item be visible for this profile based on watched state?" If multiple paths persist, they will drift in semantics, performance, and bug fixes. The drift is already visible *and already duplicated in code*:
- The custom personal collection helper treats series/seasons as watched only when all child episodes are complete; `userCollectionAllEpisodesCompleted` is a near-verbatim copy of `allEpisodesCompleted` in `user_state.go`.
- The catalog `watched` rule, by contrast, checks completion for the row's own `content_id` only (`uwp.media_item_id = mi.content_id`), so a series row is effectively never "watched" because progress/history is keyed to episode IDs, not the series ID.
Consolidating onto the catalog predicate therefore removes one Go copy outright and requires teaching the catalog predicate the rollup the other two already perform.
## Goals
- Use the catalog query system as the canonical representation and executor for watched filters.
- Make personal collection display filtering, catalog filtering, and collection-builder UI speak the same `QueryDefinition` / `QueryRule` language.
- Preserve exact collection source order when a collection is rendered as a shelf or source-ordered list.
- Preserve existing access controls and profile scoping.
- Avoid persisting a second watched-filter vocabulary unless needed as a compatibility shim.
- Keep the implementation narrow enough to land as a refactor/standardization of an already intended collection-filter feature.
## Non-Goals
- No new broad smart-collection DSL.
- No change to external sync membership. Imported lists still sync exact item membership before display filtering is applied.
- No change to collection ownership, sharing, or profile visibility rules.
- No client lockstep requirement. Existing clients can ignore new display-filter capabilities until they adopt the collection editor.
- No attempt to optimize very large exact collections beyond the current exact-source behavior in this pass.
## Scope Gate
`docs/architecture/v1-scope.md` says v1 is not locked and the proposal window is open. This design should be treated as a standardization of the profile-scoped collection filter work already underway, not as a separate feature expansion. If the final implementation exposes a new public capability beyond personal collection display filtering, file a v1 capability proposal before opening a feature PR.
## Design Principle
One feature gets one semantic implementation:
```json
{ "field": "watched", "op": "is", "value": true }
```
is the only canonical expression for "show watched items"; `value: false` is the canonical expression for "show unwatched items".
UI controls may present simpler labels such as "All", "Watched", and "Unwatched", but those controls must translate into catalog query rules before execution. Backend code should not add another watched evaluator for personal collections.
## Canonical Semantics
The canonical `watched` filter should match the user-facing played/read state shown by item user data:
| Item kind | `watched is true` should mean |
|---|---|
| Movie / episode / video leaf | Completed progress or visible completed history exists for the active profile. |
| Series | At least one child episode exists, and every child episode is completed for the active profile. |
| Season | At least one child episode exists, and every child episode in the season is completed for the active profile. |
| Ebook | Reader progress is at or above `models.EbookFinishedProgressThreshold`. |
| Manga | Use the same read-state source the catalog uses for manga/chapters once manga catalog filtering is finalized; do not add a collection-only interpretation. |
This intentionally aligns with `internal/api/handlers/user_state.go`, where series and season user data are derived by `allEpisodesCompleted`. If this definition is too expensive for catalog queries, we should optimize the catalog predicate rather than keep a separate personal collection definition.
For this table to hold when a query is *unscoped* (mixed-media exact collections), the predicate must pick the completion source from each row's own type, not from a single query-wide `media_scope` — see Catalog Watched Predicate Work. Picking the source by `media_scope` alone misclassifies any row whose type differs from the scope (notably ebooks under an unscoped query).
## Storage Model
Recommended canonical storage for personal collection display filters:
- Add a dedicated JSON column named for display filtering, `display_query_definition`. Do **not** overload the existing `query_definition` field (see Risks: it already carries `library_ids` for live/smart collections and is fragile enough to have needed a recent sanitization fix).
- Store the standard catalog `QueryDefinition` *shape* in that field, normalized to a filter-only fragment (see normalization rules below).
- For the current "media filter" control, store a **`type` rule**, not `media_scope`:
- `all` -> no `type` rule
- `movie` -> `{ "field": "type", "op": "is", "value": "movie" }`
- `series` -> `{ "field": "type", "op": "is", "value": "series" }`
- For the current "watch filter" control, store a `watched` rule:
- `all` -> no watched rule
- `watched` -> `{ "field": "watched", "op": "is", "value": true }`
- `unwatched` -> `{ "field": "watched", "op": "is", "value": false }`
**Why `type` rule, not `media_scope`:** `media_scope` is a single string on `QueryDefinition`, not a rule, and there is no logic to intersect two scopes (only `intersectCatalogDefinitionLibraries` exists, for library IDs). A `type` rule composes as an ordinary AND group and avoids any collision with a smart collection's own `media_scope`. `type` is already a supported, executable query field, so this needs no new merge machinery. Reserve `media_scope` for the source query's own scoping.
**`display_query_definition` is a filter-only fragment, not a full query.** A full `QueryDefinition` carries fields that are meaningless — and actively dangerous — for a display overlay. In particular the executor honors `def.Limit` internally, so a persisted display query with `limit` set would cap the match set *before* the exact-collection source-order reorder, silently and permanently hiding matching items and producing wrong totals/pages. `sort` and `library_ids` are likewise the source/collection's concern, not the overlay's. Normalize on the API boundary before persisting and before executing:
- Persist only `match` and `groups` (with rules drawn from the allowed display vocabulary — initially `type` and `watched`).
- Reject or strip `limit`, `sort`, `library_ids`, and `media_scope` from the fragment. Prefer reject-with-error on write so clients learn the contract; strip defensively on read for forward-compat.
- Storing the `QueryDefinition` *shape* keeps the door open to a richer rule set later; it does not mean accepting the whole `QueryDefinition` surface.
Compatibility: none required. The legacy `watch_filter` / `media_filter` enums exist only on this unmerged branch (see Current Branch State), so they are dropped rather than aliased. If — and only if — a future build ships the enums to clients before this lands, revisit the compatibility shim; until then there is no legacy surface to preserve.
## Execution Model
### Smart/live collections
Smart and live collections already resolve through `CatalogResolver.resolveQuerySource`. Their stored query is already a catalog `QueryDefinition`. Display filters should be merged into that query before execution.
**Merge must guarantee `(source) AND (display)` — appending a group does not.** `QueryBuilder.Build` joins *all* top-level groups with a single joiner derived from the definition's top-level `match` (`query_builder.go`: `topJoiner` is ` OR ` when `match == "any"`). So a smart/live source with `match: any` and groups `G1, G2` becomes `G1 OR G2`; appending a display group `D` yields `G1 OR G2 OR D` — which *widens* the result past the collection source and lets contradictory watched rules return rows instead of none. The QueryDefinition model has only one top-level `match` and only two levels (groups of rules), so there is no in-model way to express `(G1 OR G2) AND D`.
Merge rule:
- Keep library scope, `media_scope`, limit, and sort from the collection's source query.
- Combine the source definition and the display fragment with a true conjunction that is independent of either side's `match`. The robust implementation is an **executor-level conjunction**: build the source predicate and the display predicate as two separate SQL fragments and join them with ` AND ` (each side already parenthesized), rather than merging group lists. This is net-new merge logic, not an existing capability.
- The append-as-group shortcut is correct **only** when the source's top-level `match == "all"`. When `match == "any"` it is incorrect. Do not special-case "all" and forget "any"; implement the general conjunction and test both.
- Because the media control is stored as a `type` rule (not `media_scope`), it ANDs in through the same conjunction. If the source query's `media_scope` already excludes that type (e.g. a movies-only smart collection plus a `type is series` display filter), the result is empty — no scope-intersection code is needed. (There is no existing media-scope intersection logic to lean on, which is the main reason the media control is modeled as a `type` rule rather than a second `media_scope`.)
- If both the source and display layers contain `watched` rules, the conjunction preserves both; contradictory rules naturally return no rows.
### Exact/manual/imported collections
Exact collections have a source item set from `user_personal_collection_items`. Display filtering is applied after membership is known but before pagination.
**This path replaces two existing filters, not one.** Today the source-ordered exact path runs the in-Go `filterCatalogItems` (a metadata-only matcher that cannot evaluate personalized fields — it returns `false` for `watched`) *and* the separate `FilterUserCollectionDisplayItems` Go helper for the watched dimension. Neither is the SQL executor. Routing through the executor folds both metadata and personalized filtering into one SQL path; `FilterUserCollectionDisplayItems` is deleted and `filterCatalogItems` is no longer used for these collections.
Recommended execution:
1. Load collection item IDs in source order (from `user_personal_collection_items`, `ORDER BY position`).
2. Build a catalog `QueryDefinition` from the normalized display fragment plus any request overlay. Force `Limit = nil` and drop any fragment-level `sort`/`library_ids` so the executor cannot truncate or reorder the match set out from under step 4. (Defense in depth: the fragment is already normalized at persistence per the Storage Model, but the executor call must not reintroduce a limit.)
3. Run the normal catalog executor with `AccessFilter.AllowedContentIDs = itemIDs` so all filtering, including `watched`, uses the catalog query path. (`resolveCandidateItemsWithQuery` already establishes this pattern.)
4. For source-ordered requests, retrieve the *full* matching set (no limit), collect the matching IDs into a set, reapply the original source order, and only then apply request offset/limit. Request pagination is applied after reordering, never inside the executor.
5. For requests that sort via the query, allow the catalog executor to sort normally. **Note:** exact user collections are hardcoded to source order today (`UseSourceOrder: true`); this branch exists for forward-compatibility and is not exercised by current callers. Mark it clearly so it is not mistaken for present behavior.
This preserves exact collection order without duplicating watched logic. Step 4 is a new shape (run the executor for the match-set, then reorder), distinct from the sorted-pagination shape `resolveCandidateItemsWithQuery` uses today — call out the differing limit handling in the implementation.
## Catalog Watched Predicate Work
The existing catalog `watched` rule is already implemented but needs one semantic hardening before it becomes the shared path:
- Extract a reusable played/read predicate for `QueryBuilder`.
- Keep the existing leaf-item checks for progress, completed history, hidden history, and ebook reader progress.
- Add series and season rollup branches so `watched` matches item user-data semantics.
- Add SQL tests that assert series/season predicates are present only when relevant.
- Reuse the same predicate for `last_watched` only if the semantics truly match; otherwise leave `last_watched` as a leaf/history timestamp filter and document the difference. (`last_watched` is a timestamp aggregate built from the `user_last_watched` CTE — different machinery from the completion predicate — so it most likely stays separate.)
**Episode-set parity is the load-bearing requirement, not the rollup branch itself.** The Go rollups (`allEpisodesCompleted` in `user_state.go` and its copy in the collection helper) define "all episodes complete" over the episode set returned by the episode repository (`ListBySeriesIDs` / `ListBySeasonIDs`). If the SQL rollup's notion of "child episodes" differs — specials, unaired, or unavailable episodes included or excluded differently — the catalog `watched` rule will *disagree* with item user-data, recreating the exact drift this design removes, just relocated into SQL. The hardening must therefore reproduce the repository's child-episode set exactly (same joins, same filters), and a test must pin that equivalence (see Testing Strategy). Treat Open Question #4 (all-available vs all-aired) as a question about *what that repository set already is*, and match it.
**Row-type branching is mandatory for unscoped queries, including ebooks.** The current predicate selects the ebook source (`ebook_reader_progress`) *only* when `mediaScope == "ebook"`; otherwise it uses video watch progress/history (`userStateCompletionClause` branches on `qb.mediaScope == "ebook"`). Exact collection membership is not type-restricted, so an exact collection can contain an ebook while the query runs unscoped (the `all` media control emits no `type` rule). Under today's code that ebook would be evaluated against video progress/history — i.e. classified as never-watched regardless of read state — reintroducing exactly the kind of drift this design removes. The hardened predicate must therefore branch by the row's own type when the query is unscoped:
- movie / episode / video leaf -> watch progress + completed history,
- series / season -> episode rollup (with repository-parity episode sets, above),
- ebook -> `ebook_reader_progress >= EbookFinishedProgressThreshold`,
- manga -> deferred (see Non-Goals / Canonical Semantics).
If row-type branching cannot land in the first cut, v1 must instead *enforce* a video-only scope at execution for exact collections (so ebooks are never silently classified by the video source) and say so explicitly — but row-type branching is the correct end state because it is what makes the canonical semantics table hold under unscoped queries.
The first implementation may still gate series/season rollup behavior by `mediaScope == "series"` and exact known season rows, but it must not assume a single item kind for unscoped/video queries that can return movies, series, and ebooks together.
## API Shape
Preferred API model:
- Add `display_query_definition?: QueryDefinition` to personal collection create/update/response types.
- Replace the branch's `watch_filter` / `media_filter` request and response fields with `display_query_definition`. Since the enums are unreleased (see Current Branch State), no aliasing or deprecation header flow is required.
- Keep collection capabilities additive:
- expose supported display-query fields using the existing catalog field vocabulary,
- optionally expose UI presets for "all/watched/unwatched" and "all/movies/series" as client convenience.
Do not add a separate `filter_watched` query parameter to catalog endpoints. Catalog already has the watched rule.
## UI Model
The web UI can keep simple controls:
- media segmented control: All / Movies / Series,
- watched segmented control: All / Unwatched / Watched.
Those controls should edit a `QueryDefinition` fragment using shared helpers, not a separate collection-only state model.
Recommended helper location:
- `web/src/lib/collectionDisplayFilters.ts` should convert between UI presets and catalog `QueryDefinition`.
- `web/src/components/collections/CollectionGuidedRulesEditor.tsx` remains the broader catalog-query editor.
- `web/src/components/collections/CollectionBuilder.tsx` and `web/src/pages/ImportedCollectionEditor.tsx` use the shared helper.
## Migration Strategy
The legacy `watch_filter` / `media_filter` enums exist only on this unmerged branch and have shipped to no client (see Current Branch State), so the clean-replacement path applies. There is no backfill, dual-read, or deprecation window:
- Reshape the existing migration `20260623184858_user_collection_display_filters.sql` rather than layering a second migration on top: drop the `watch_filter` / `media_filter` columns and their CHECK constraints, and add the canonical `display_query_definition jsonb` column (nullable; `NULL` / absent means "no display filter"). Because the branch is unmerged, editing the not-yet-applied migration in place is cleaner than a forward-then-backfill sequence.
- Update API and web types to use display-query-backed helpers; delete the legacy enum request/response fields.
- No server code reads or writes the legacy columns after this change.
A backfill/dual-read migration would only be warranted if a build ships the legacy enums to clients before this lands. If that happens, fall back to: add `display_query_definition`, backfill (`watch_filter = 'watched'` -> watched rule true; `'unwatched'` -> false; `media_filter = 'movie'`/`'series'` -> `type` rule; `all` omits the rule), read legacy as fallback, stop writing legacy, and remove later under the API deprecation flow. Treat this as a contingency, not the plan of record.
## Risks
- **Merge OR-leak (smart/live):** appending a display group to a `match: any` source query widens results past the collection source instead of constraining them. Mitigate with the executor-level conjunction (Execution Model) and a test that a `match: any` source plus a display filter never returns rows the source alone would exclude.
- **Display fragment truncation:** a persisted `limit` (or stray `sort`) on `display_query_definition` would cap or reorder the exact-collection match set before the source-order reorder, hiding items and corrupting totals. Mitigate by normalizing the fragment to filter-only on write and forcing `Limit = nil` on the match-set execution.
- **Ebook misclassification under unscoped queries:** an ebook in a mixed-media exact collection evaluated by an unscoped `watched` rule is classified by the video source today (`media_scope`-gated ebook predicate). Mitigate with row-type branching in the predicate, or enforce a video-only execution scope for v1 (Catalog Watched Predicate Work).
- **Episode-set divergence (highest risk):** if the SQL rollup's child-episode set differs from the repository's `ListBySeriesIDs` / `ListBySeasonIDs` listing, the catalog `watched` rule will disagree with item user-data and reintroduce drift in a new place. Mitigate by reproducing the repository's joins/filters exactly and pinning the equivalence with a test. This, not raw SQL cost, is the thing most likely to ship subtly wrong.
- **Series/season SQL cost:** all-episodes-complete checks are more expensive than leaf checks. Mitigate by only emitting rollup branches when item types can include series/seasons and by relying on indexes on episode parent IDs and progress/history media IDs.
- **Semantics change for catalog series filters:** extending catalog `watched` to match displayed played state can change existing catalog results for series (today they are effectively never "watched"). This is likely correct, but it must be called out in release notes or PR text.
- **Exact collection performance:** source-order rendering requires evaluating the full exact collection before pagination (the executor must return the complete match-set to reorder). This matches current behavior but should not regress further; very large exact collections remain out of scope this pass (see Non-Goals).
- **Field overloading (resolved → dedicated column):** code inspection confirms `query_definition` already carries `library_ids` for live/smart collections and recently needed a `library_ids` sanitization fix. Reusing it for display filters would entangle two concerns; use a dedicated `display_query_definition` column.
## Testing Strategy
Backend tests are justified because this changes shared personalized filtering semantics:
- **Equivalence test (highest value):** for the same series/season/profile fixtures, assert the catalog `watched` rule and `user_state.go`'s `Played` flag agree. This is the "one semantic" goal made executable and is the test most likely to catch episode-set divergence; it should fail loudly if the two definitions drift.
- `internal/catalog/query_builder_test.go`: `watched` emits leaf completion checks and series/season rollup checks as appropriate.
- `internal/catalog/query_executor_test.go` or focused resolver tests: exact collection filtering routes through catalog `watched` and preserves source order; the executor returns the full match-set so reordering is stable across pages.
- `internal/catalog/catalog_resolver_test.go`: contradictory display/source filters return no rows; a `type is series` display filter on a movies-only smart collection yields no rows without any scope-intersection code.
- **Merge safety:** a smart/live source with top-level `match: any` plus a display filter must never return rows the source alone excludes (guards against the OR-leak); a contradictory `watched` overlay on a `match: any` source returns zero rows.
- **Fragment normalization:** a `display_query_definition` carrying `limit`/`sort`/`library_ids` is rejected on write (or stripped on read); an exact collection larger than any plausible page returns the full match set in source order regardless of fragment contents.
- **Ebook under unscoped watched:** a mixed-media exact collection (movie + ebook) with an `unwatched`/`watched` filter classifies the ebook via `ebook_reader_progress`, not video history (guards against the misclassification finding).
- `internal/api/handlers/collections_test.go`: create/update round-trips `display_query_definition`; the removed `watch_filter` / `media_filter` fields are gone from request and response.
- Migration test only if the contingency backfill path is used (it is not, on the plan of record).
Frontend tests should stay targeted:
- helper tests for converting simple UI controls to/from `QueryDefinition`,
- one editor test proving the submit payload carries the canonical display query.
## Implementation Outline
1. Harden catalog `watched` semantics (series/season rollup with repository-parity episode sets) and land the equivalence test. This is independently valuable and reviewable; consider shipping it as its own PR first.
2. Reshape migration `20260623184858` to drop `watch_filter` / `media_filter` and add `display_query_definition jsonb`.
3. Add display-query types and normalization helpers in backend collection handling.
4. Replace both the custom `FilterUserCollectionDisplayItems` pass and the in-Go `filterCatalogItems` source-order path with catalog-query execution constrained by exact item IDs (match-set, then reorder).
5. Update web controls to emit catalog query fragments through shared helpers; delete the legacy enum fields.
6. Run focused Go tests for catalog/query/user-collection paths and targeted frontend helper tests.
Per the v1 process, link the capability sub-issue (`Part of #NNN`) and keep PRs to one concern — predicate-hardening (step 1) is a natural standalone PR separate from the storage/UI reshape (steps 2–5).
## Open Questions
- **Resolved.** The legacy enums exist only on this unmerged, unreleased branch, so it replaces them cleanly with no compatibility shim (see Current Branch State and Migration Strategy).
- **Decided:** the standardized *controls* expose video item scopes (movie/series/episode) for v1; manga read-state filtering is deferred until catalog manga filtering is finalized, then adopts the same catalog read-state source rather than a collection-only interpretation. Note this constrains the *control vocabulary*, not collection *membership* — exact collections can still contain ebooks, so the watched predicate must branch by row type (see Catalog Watched Predicate Work) regardless of what the control offers.
- **Decided:** first version exposes only the `type` and `watched` controls, stored as a **filter-only `QueryDefinition` fragment** (`match` + `groups`; no `limit`/`sort`/`library_ids`/`media_scope`). Keeping the fragment shape lets a richer rule set turn on later without a storage change, while normalization prevents the fragment from carrying execution-controlling fields (see Storage Model).
- Should catalog `watched` rollup for series mean "all available episodes complete" or "all aired episodes complete"? This is a question about *what the episode repository already returns* — match that set exactly (see Catalog Watched Predicate Work / episode-set parity). Only change the definition by changing the repository path and item user-data together, never the catalog rule alone.
- **New:** confirm no `silo-android` / `silo-apple` build has already consumed the branch's `watch_filter` / `media_filter` fields. The design assumes not (the branch is unreleased); if a client has pinned them, the contingency backfill path applies instead.
+2 -1
View File
@@ -31,6 +31,7 @@ require (
github.com/oklog/ulid/v2 v2.1.0
github.com/pgvector/pgvector-go v0.3.0
github.com/pressly/goose/v3 v3.27.1
github.com/tetratelabs/wazero v1.12.0
github.com/wneessen/go-mail v0.7.3
github.com/zishang520/socket.io/v2 v2.5.0
go.n16f.net/thumbhash v1.1.0
@@ -76,7 +77,7 @@ require (
)
require (
github.com/Silo-Server/silo-plugin-sdk v0.6.0
github.com/Silo-Server/silo-plugin-sdk v0.8.1
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.8 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.21 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.21 // indirect
+4 -2
View File
@@ -4,8 +4,8 @@ github.com/PuerkitoBio/goquery v1.8.0 h1:PJTF7AmFCFKk1N6V6jmKfrNH9tV5pNE6lZMkG0g
github.com/PuerkitoBio/goquery v1.8.0/go.mod h1:ypIiRMtY7COPGk+I/YbZLbxsxn9g5ejnI2HSMtkjZvI=
github.com/SherClockHolmes/webpush-go v1.4.0 h1:ocnzNKWN23T9nvHi6IfyrQjkIc0oJWv1B1pULsf9i3s=
github.com/SherClockHolmes/webpush-go v1.4.0/go.mod h1:XSq8pKX11vNV8MJEMwjrlTkxhAj1zKfxmyhdV7Pd6UA=
github.com/Silo-Server/silo-plugin-sdk v0.6.0 h1:Gi9TdH9kt7b8X4xRXH493/nSYb9n0GO4VCWmlll0hKI=
github.com/Silo-Server/silo-plugin-sdk v0.6.0/go.mod h1:etqmxLTwjxpFH9goAjBDfNDoqHMv2/sqUXu8yx3hNfA=
github.com/Silo-Server/silo-plugin-sdk v0.8.1 h1:6DLm2s/O3N9KTx2d09Nq3bbV8z4u9HJwO2gueQDPyPw=
github.com/Silo-Server/silo-plugin-sdk v0.8.1/go.mod h1:etqmxLTwjxpFH9goAjBDfNDoqHMv2/sqUXu8yx3hNfA=
github.com/abadojack/whatlanggo v1.0.1 h1:19N6YogDnf71CTHm3Mp2qhYfkRdyvbgwWdd2EPxJRG4=
github.com/abadojack/whatlanggo v1.0.1/go.mod h1:66WiQbSbJBIlOZMsvbKe5m6pzQovxCH9B/K8tQB2uoc=
github.com/andybalholm/brotli v1.2.1 h1:R+f5xP285VArJDRgowrfb9DqL18yVK0gKAW/F+eTWro=
@@ -197,6 +197,8 @@ github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/
github.com/stretchr/testify v1.7.2/go.mod h1:R6va5+xMeoiuVRoj+gSkQ7d3FALtqAAGI1FQKckRals=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/tetratelabs/wazero v1.12.0 h1:DuWcpNu/FzgEXgGBDp8J1Spc+CWOvvtvVyjKlaZopYU=
github.com/tetratelabs/wazero v1.12.0/go.mod h1:LvKtzl2RqO4gyF27BiXU+nKAjcV8f38U+kP/q2vgxh0=
github.com/tmthrgd/go-hex v0.0.0-20190904060850-447a3041c3bc h1:9lRDQMhESg+zvGYmW5DyG0UqvY96Bu5QYsTLvCHdrgo=
github.com/tmthrgd/go-hex v0.0.0-20190904060850-447a3041c3bc/go.mod h1:bciPuU6GHm1iF1pBvUfxfsH0Wmnc2VbpgvbI9ZWuIRs=
github.com/uptrace/bun v1.1.12 h1:sOjDVHxNTuM6dNGaba0wUuz7KvDE1BmNu9Gqs2gJSXQ=
+14 -2
View File
@@ -99,6 +99,9 @@ func (s stubStore) ListHistory(context.Context, string, int, int) ([]userstore.W
func (s stubStore) ListCompletedHistory(context.Context, userstore.CompletedHistoryQuery) ([]userstore.WatchHistoryEntry, error) {
panic("unused")
}
func (s stubStore) ListCompletedHistoryItems(context.Context, userstore.CompletedHistoryItemQuery) ([]userstore.CompletedHistoryItem, error) {
panic("unused")
}
func (s stubStore) RemoveHistoryItems(context.Context, string, []string, time.Time) error {
panic("unused")
}
@@ -127,9 +130,18 @@ func (s stubStore) ListFavorites(context.Context, string, int, int) ([]userstore
func (s stubStore) ListFavoritesByMediaItems(context.Context, string, []string) (map[string]bool, error) {
panic("unused")
}
func (s stubStore) IsFavorite(context.Context, string, string) (bool, error) { panic("unused") }
func (s stubStore) AddToWatchlist(context.Context, string, string) error { panic("unused") }
func (s stubStore) IsFavorite(context.Context, string, string) (bool, error) { panic("unused") }
func (s stubStore) AddToWatchlist(context.Context, string, string) error { panic("unused") }
func (s stubStore) AddToWatchlistAt(context.Context, string, string, time.Time) error {
panic("unused")
}
func (s stubStore) RemoveWatchedFromWatchlist(context.Context, string) (bool, error) {
return true, nil
}
func (s stubStore) RemoveFromWatchlist(context.Context, string, string) error { panic("unused") }
func (s stubStore) ReplaceWatchlistOrder(context.Context, string, []string) error {
panic("unused")
}
func (s stubStore) ListWatchlist(context.Context, string, int, int) ([]userstore.WatchlistEntry, error) {
panic("unused")
}
+103
View File
@@ -0,0 +1,103 @@
package api
import (
"context"
"log/slog"
"os"
"path/filepath"
"strings"
"sync"
"time"
"github.com/Silo-Server/silo-server/internal/api/handlers"
"github.com/Silo-Server/silo-server/internal/catalog"
"github.com/Silo-Server/silo-server/internal/config"
"github.com/Silo-Server/silo-server/internal/ebookconvert"
)
// ebookKindleConversionSettingKey is the admin flag that gates Kindle->EPUB
// conversion. Off by default; toggled via PUT /admin/settings/{key}.
const ebookKindleConversionSettingKey = "ebook.kindle_conversion_enabled"
// buildEbookConversion wires the in-process Kindle->EPUB converter for the read
// handler. It compiles the embedded WASM module once at startup; if that fails
// the feature stays off (returns nil) and the raw original is served as before.
// The admin flag is read with a short TTL (see ebookFlagPredicate), so it can be
// toggled without a restart yet does not hit the DB on every read.
//
// The Converter (a wazero runtime, ~no external resources — per-conversion
// scratch dirs are cleaned as they go) is intentionally owned for the process
// lifetime: it is not Close()d on shutdown because process exit reclaims it and
// there is nothing to flush. Memory is bounded per conversion by
// ebookconvert.DefaultMaxMemoryPages × DefaultConcurrency (see the design doc's
// resource notes); tune via ebookconvert.Options if the deployment is tight.
func buildEbookConversion(deps Dependencies, settings catalog.SettingsStore) *handlers.EbookConversion {
if settings == nil {
return nil
}
converter, err := ebookconvert.NewConverter(context.Background(), ebookconvert.Options{})
if err != nil {
slog.Warn("ebook Kindle->EPUB conversion unavailable: converter init failed", "error", err)
return nil
}
cache, err := ebookconvert.NewCache(converter, ebookconvert.CacheOptions{Dir: ebookConversionCacheDir(deps.CurrentConfig())})
if err != nil {
slog.Warn("ebook Kindle->EPUB conversion unavailable: cache init failed", "error", err)
_ = converter.Close(context.Background())
return nil
}
slog.Info("ebook Kindle->EPUB conversion ready (admin-flag gated)", "setting", ebookKindleConversionSettingKey)
return &handlers.EbookConversion{
Converter: cache,
Enabled: ebookFlagPredicate(settings, ebookKindleConversionSettingKey),
}
}
// ebookFlagPredicate returns a cached boolean reader for a server setting. The
// read path (every ebook read + the capability endpoint, which clients may
// poll) would otherwise issue a DB query per request; a short TTL collapses
// bursts to one query while keeping admin toggles effectively instant. On a read
// error it keeps the last known value rather than flapping the feature off.
func ebookFlagPredicate(settings catalog.SettingsStore, key string) func(context.Context) bool {
const ttl = 3 * time.Second
var (
mu sync.Mutex
value bool
fetched time.Time
hasVal bool
)
return func(ctx context.Context) bool {
mu.Lock()
defer mu.Unlock()
if hasVal && time.Since(fetched) < ttl {
return value
}
v, err := settings.Get(ctx, key)
if err != nil {
return value // keep last known (false until the first successful read)
}
value = isTruthySetting(v)
fetched = time.Now()
hasVal = true
return value
}
}
// ebookConversionCacheDir derives the converted-EPUB cache directory, a sibling
// of the transcode dir so it lives alongside other derived media.
func ebookConversionCacheDir(cfg *config.Config) string {
base := os.TempDir()
if cfg != nil && strings.TrimSpace(cfg.Playback.TranscodeDir) != "" {
base = filepath.Dir(cfg.Playback.TranscodeDir)
}
return filepath.Join(base, "silo-ebook-epub")
}
func isTruthySetting(v string) bool {
switch strings.ToLower(strings.TrimSpace(v)) {
case "true", "1", "yes", "on", "enabled":
return true
default:
return false
}
}
+39
View File
@@ -0,0 +1,39 @@
package api
import (
"context"
"sync/atomic"
"testing"
)
// stubSettings counts Get calls so we can assert the flag predicate caches.
type stubSettings struct {
value string
gets int64
}
func (s *stubSettings) Get(_ context.Context, _ string) (string, error) {
atomic.AddInt64(&s.gets, 1)
return s.value, nil
}
func (s *stubSettings) Set(context.Context, string, string) error { return nil }
func (s *stubSettings) GetAll(context.Context) (map[string]string, error) {
return map[string]string{}, nil
}
// The flag predicate must not hit the settings store on every call: the read
// path (every ebook read + capability poll) would otherwise issue a DB query
// per request.
func TestEbookFlagPredicate_CachesWithinTTL(t *testing.T) {
s := &stubSettings{value: "true"}
pred := ebookFlagPredicate(s, "ebook.kindle_conversion_enabled")
for i := 0; i < 50; i++ {
if !pred(context.Background()) {
t.Fatalf("call %d: predicate returned false for a truthy flag", i)
}
}
if got := atomic.LoadInt64(&s.gets); got > 2 {
t.Fatalf("settings.Get called %d times across 50 predicate calls; expected it to be cached", got)
}
}
+91
View File
@@ -8,7 +8,9 @@ import (
"fmt"
"io"
"log/slog"
"math"
"net/http"
"net/url"
"slices"
"sort"
"strconv"
@@ -99,6 +101,7 @@ type AdminHandler struct {
OnUserSessionsRevoked func(ctx context.Context, userID int)
OnServerSettingUpdated func(ctx context.Context, key, value string)
RestartStatus *ServerRestartStatusTracker
CatalogSearchStatus catalog.CatalogSearchStatusProvider
}
// NewAdminHandler creates a new AdminHandler backed by the given
@@ -2138,6 +2141,94 @@ func (h *AdminHandler) HandleUpdateSetting(w http.ResponseWriter, r *http.Reques
"subtitle_ai.transcribe_quota_period must be day, week, or month")
return
}
case catalog.SearchSettingProvider:
switch strings.TrimSpace(strings.ToLower(req.Value)) {
case catalog.SearchProviderPostgres, catalog.SearchProviderMeilisearch:
req.Value = strings.TrimSpace(strings.ToLower(req.Value))
default:
writeError(w, http.StatusBadRequest, "bad_request", "catalog.search.provider must be postgres or meilisearch")
return
}
case catalog.SearchSettingMeilisearchURL:
value := strings.TrimSpace(req.Value)
if value != "" {
parsed, err := url.Parse(value)
if err != nil || parsed.Scheme == "" || parsed.Host == "" {
writeError(w, http.StatusBadRequest, "bad_request", "catalog.search.meilisearch.url must include scheme and host")
return
}
}
req.Value = value
case catalog.SearchSettingMeilisearchIndex:
req.Value = strings.TrimSpace(req.Value)
if req.Value == "" {
writeError(w, http.StatusBadRequest, "bad_request", "catalog.search.meilisearch.index is required")
return
}
case catalog.SearchSettingMeilisearchTimeoutMS:
n, err := strconv.Atoi(strings.TrimSpace(req.Value))
if err != nil || n <= 0 {
writeError(w, http.StatusBadRequest, "bad_request", "catalog.search.meilisearch.timeout_ms must be an integer greater than 0")
return
}
req.Value = strconv.Itoa(n)
case catalog.SearchSettingMeilisearchMatchingStrategy:
switch strings.TrimSpace(strings.ToLower(req.Value)) {
case "last", "all":
req.Value = strings.TrimSpace(strings.ToLower(req.Value))
default:
writeError(w, http.StatusBadRequest, "bad_request", "catalog.search.meilisearch.matching_strategy must be last or all")
return
}
case catalog.SearchSettingMeilisearchSyncBatchSize:
n, err := strconv.Atoi(strings.TrimSpace(req.Value))
if err != nil || n < 1 || n > catalog.MaxMeilisearchSyncBatchSize {
writeError(w, http.StatusBadRequest, "bad_request", "catalog.search.meilisearch.sync_batch_size must be an integer between 1 and 10000")
return
}
req.Value = strconv.Itoa(n)
case catalog.SearchSettingMeilisearchRebuildBatchSize:
n, err := strconv.Atoi(strings.TrimSpace(req.Value))
if err != nil || n < 1 || n > catalog.MaxMeilisearchRebuildBatchSize {
writeError(w, http.StatusBadRequest, "bad_request", "catalog.search.meilisearch.rebuild_batch_size must be an integer between 1 and 25000")
return
}
req.Value = strconv.Itoa(n)
case catalog.SearchSettingMeilisearchRebuildQueue:
n, err := strconv.Atoi(strings.TrimSpace(req.Value))
if err != nil || n < 1 || n > catalog.MaxMeilisearchRebuildQueueDepth {
writeError(w, http.StatusBadRequest, "bad_request", "catalog.search.meilisearch.rebuild_task_queue_depth must be an integer between 1 and 16")
return
}
req.Value = strconv.Itoa(n)
case catalog.SearchSettingMeilisearchIndexTypes:
itemTypes, err := catalog.NormalizeCatalogSearchIndexTypesValue(req.Value)
if err != nil {
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
return
}
req.Value = catalog.FormatCatalogSearchIndexTypesValue(itemTypes)
case catalog.SearchSettingMeilisearchSemanticEnabled:
enabled, err := strconv.ParseBool(strings.TrimSpace(req.Value))
if err != nil {
writeError(w, http.StatusBadRequest, "bad_request", "catalog.search.meilisearch.semantic_enabled must be true or false")
return
}
req.Value = strconv.FormatBool(enabled)
case catalog.SearchSettingMeilisearchSemanticRatio:
ratio, err := strconv.ParseFloat(strings.TrimSpace(req.Value), 64)
if err != nil || math.IsNaN(ratio) || ratio < 0 || ratio > 1 {
writeError(w, http.StatusBadRequest, "bad_request", "catalog.search.meilisearch.semantic_ratio must be a number between 0 and 1")
return
}
req.Value = strconv.FormatFloat(ratio, 'f', -1, 64)
case catalog.SearchSettingMeilisearchEmbedder:
embedder, err := catalog.NormalizeCatalogSearchEmbedderName(req.Value)
if err != nil {
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
return
}
req.Value = embedder
}
if err := h.SettingsRepo.Set(r.Context(), key, req.Value); err != nil {
@@ -0,0 +1,30 @@
package handlers
import (
"net/http"
"github.com/Silo-Server/silo-server/internal/catalog"
)
func (h *AdminHandler) HandleGetCatalogSearchStatus(w http.ResponseWriter, r *http.Request) {
if h == nil || h.CatalogSearchStatus == nil {
writeJSON(w, http.StatusOK, catalog.CatalogSearchRuntimeStatus{
ConfiguredProvider: catalog.SearchProviderPostgres,
ActiveProvider: catalog.SearchProviderPostgres,
Meilisearch: catalog.CatalogSearchMeiliStatus{
Configured: false,
Healthy: false,
CircuitState: "not_configured",
},
Index: catalog.CatalogSearchIndexStateStatus{
ExpectedSchemaVersion: catalog.SearchMeilisearchSchemaVersion,
},
Tasks: []catalog.CatalogSearchTaskLink{
{Key: "sync_catalog_search_index", Name: "Sync Catalog Search Index", Href: "/admin/tasks/sync_catalog_search_index"},
{Key: "rebuild_catalog_search_index", Name: "Rebuild Catalog Search Index", Href: "/admin/tasks/rebuild_catalog_search_index"},
},
})
return
}
writeJSON(w, http.StatusOK, h.CatalogSearchStatus.Status(r.Context()))
}
+29 -16
View File
@@ -6,6 +6,7 @@ import (
"fmt"
"log/slog"
"net/http"
"strings"
"github.com/go-chi/chi/v5"
"github.com/jackc/pgx/v5/pgxpool"
@@ -56,12 +57,13 @@ func NewAdminMatchHandler(
// --- Request/Response types ---
type matchSearchRequest struct {
Title string `json:"title"`
Year int `json:"year"`
ImdbID string `json:"imdb_id"`
TmdbID string `json:"tmdb_id"`
TvdbID string `json:"tvdb_id"`
LibraryID *int `json:"library_id,omitempty"`
Title string `json:"title"`
Year int `json:"year"`
ImdbID string `json:"imdb_id"`
TmdbID string `json:"tmdb_id"`
TvdbID string `json:"tvdb_id"`
ProviderIDs map[string]string `json:"provider_ids,omitempty"`
LibraryID *int `json:"library_id,omitempty"`
}
type matchSearchResponse struct {
@@ -119,7 +121,7 @@ func (h *AdminMatchHandler) HandleSearchItemMatchCandidates(w http.ResponseWrite
Title: req.Title,
Year: req.Year,
ContentType: item.Type,
ProviderIDs: make(map[string]string),
ProviderIDs: normalizeMatchProviderIDs(req.ProviderIDs),
}
if query.Title == "" {
query.Title = item.Title
@@ -129,15 +131,9 @@ func (h *AdminMatchHandler) HandleSearchItemMatchCandidates(w http.ResponseWrite
}
// Inject any provider IDs from the request.
if req.ImdbID != "" {
query.ProviderIDs["imdb"] = req.ImdbID
}
if req.TmdbID != "" {
query.ProviderIDs["tmdb"] = req.TmdbID
}
if req.TvdbID != "" {
query.ProviderIDs["tvdb"] = req.TvdbID
}
setMatchProviderID(query.ProviderIDs, "imdb", req.ImdbID)
setMatchProviderID(query.ProviderIDs, "tmdb", req.TmdbID)
setMatchProviderID(query.ProviderIDs, "tvdb", req.TvdbID)
candidates, err := h.metadata.SearchAndNormalize(r.Context(), query, folderID)
if err != nil {
@@ -211,6 +207,23 @@ func (h *AdminMatchHandler) HandleApplyItemMatch(w http.ResponseWriter, r *http.
})
}
func normalizeMatchProviderIDs(input map[string]string) map[string]string {
out := make(map[string]string, len(input))
for provider, providerID := range input {
setMatchProviderID(out, provider, providerID)
}
return out
}
func setMatchProviderID(providerIDs map[string]string, provider string, providerID string) {
provider = strings.ToLower(strings.TrimSpace(provider))
providerID = strings.TrimSpace(providerID)
if provider == "" || providerID == "" {
return
}
providerIDs[provider] = providerID
}
// PoolFolderLookup implements MatchFolderLookup using a direct pgx pool query
// against the media_item_libraries table.
type PoolFolderLookup struct {
+56 -2
View File
@@ -199,6 +199,58 @@ func TestAdminMatchSearch_FallsBackToItemMetadata(t *testing.T) {
}
}
func TestAdminMatchSearch_AcceptsGenericProviderIDsAndLibraryID(t *testing.T) {
items := &fakeMatchItemLookup{
items: map[string]*models.MediaItem{
"ebook-1": {ContentID: "ebook-1", Title: "Example Book", Year: 2024, Type: "ebook"},
},
}
folders := &fakeMatchFolderLookup{
folderIDs: map[string][]int{"ebook-1": {7, 8}},
}
captureSvc := &capturingMetadataService{
inner: &fakeMatchMetadataService{searchResults: []metadata.MatchCandidate{}},
}
h := NewAdminMatchHandler(items, folders, captureSvc)
router := buildMatchRouter(h)
libraryID := 8
body, _ := json.Marshal(matchSearchRequest{
ProviderIDs: map[string]string{
" ISBN ": " 9780000000001 ",
"goodreads": " ",
},
TmdbID: "12345",
LibraryID: &libraryID,
})
req := httptest.NewRequest(http.MethodPost, "/admin/items/ebook-1/match/search", bytes.NewReader(body))
req.Header.Set("Content-Type", "application/json")
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("expected 200, got %d: %s", rec.Code, rec.Body.String())
}
if captureSvc.lastFolderID != 8 {
t.Fatalf("folderID = %d, want 8", captureSvc.lastFolderID)
}
if captureSvc.lastQuery.ContentType != "ebook" {
t.Fatalf("ContentType = %q, want ebook", captureSvc.lastQuery.ContentType)
}
if got := captureSvc.lastQuery.ProviderIDs["isbn"]; got != "9780000000001" {
t.Fatalf("ProviderIDs[isbn] = %q, want 9780000000001", got)
}
if got := captureSvc.lastQuery.ProviderIDs["tmdb"]; got != "12345" {
t.Fatalf("ProviderIDs[tmdb] = %q, want 12345", got)
}
if _, ok := captureSvc.lastQuery.ProviderIDs["goodreads"]; ok {
t.Fatalf("blank provider IDs should be ignored, got %v", captureSvc.lastQuery.ProviderIDs)
}
}
func TestAdminMatchApply_PreservesContentID(t *testing.T) {
items := &fakeMatchItemLookup{
items: map[string]*models.MediaItem{
@@ -293,12 +345,14 @@ func TestAdminMatchApply_ItemNotFound(t *testing.T) {
// capturingMetadataService wraps a fake to capture the query passed to SearchAndNormalize.
type capturingMetadataService struct {
inner *fakeMatchMetadataService
lastQuery metadata.SearchQuery
inner *fakeMatchMetadataService
lastQuery metadata.SearchQuery
lastFolderID int
}
func (c *capturingMetadataService) SearchAndNormalize(ctx context.Context, query metadata.SearchQuery, folderID int) ([]metadata.MatchCandidate, error) {
c.lastQuery = query
c.lastFolderID = folderID
return c.inner.SearchAndNormalize(ctx, query, folderID)
}
@@ -12,6 +12,7 @@ import (
"github.com/redis/go-redis/v9"
"github.com/Silo-Server/silo-server/internal/cache"
"github.com/Silo-Server/silo-server/internal/catalog"
"github.com/Silo-Server/silo-server/internal/config"
"github.com/Silo-Server/silo-server/internal/recommendations/embeddings"
"github.com/Silo-Server/silo-server/internal/s3client"
@@ -116,6 +117,8 @@ func (h *AdminHandler) HandleCheckSettingsConnection(w http.ResponseWriter, r *h
response = checkRedisConnection(r.Context(), cfg)
case "recommendations_embedding":
response = checkRecommendationsEmbeddingConnection(r.Context(), cfg)
case "meilisearch":
response = checkMeilisearchConnection(r.Context(), effectiveSettings)
default:
writeError(w, http.StatusBadRequest, "bad_request", "Unsupported connection check kind")
return
@@ -124,6 +127,22 @@ func (h *AdminHandler) HandleCheckSettingsConnection(w http.ResponseWriter, r *h
writeJSON(w, http.StatusOK, response)
}
func checkMeilisearchConnection(ctx context.Context, settings map[string]string) connectionCheckResponse {
searchSettings, err := catalog.CatalogSearchSettingsFromMap(settings)
if err != nil {
return connectionCheckResponse{Success: false, Message: err.Error()}
}
if searchSettings.MeilisearchURL == "" {
return connectionCheckResponse{Success: false, Message: "Meilisearch URL is required"}
}
searchSettings.Provider = catalog.SearchProviderMeilisearch
indexer := catalog.NewCatalogSearchIndexer(nil, nil)
if err := indexer.CheckConnection(ctx, searchSettings); err != nil {
return connectionCheckResponse{Success: false, Message: err.Error()}
}
return connectionCheckResponse{Success: true, Message: "Meilisearch connection successful"}
}
func (h *AdminHandler) effectiveSettingsForConnectionCheck(
ctx context.Context,
req adminSettingsConnectionCheckRequest,
@@ -11,6 +11,7 @@ import (
"github.com/go-chi/chi/v5"
"github.com/Silo-Server/silo-server/internal/catalog"
"github.com/Silo-Server/silo-server/internal/config"
"github.com/Silo-Server/silo-server/internal/s3client"
)
@@ -376,6 +377,74 @@ func TestAdminUpdateSettingReportsRestartRequired(t *testing.T) {
}
}
func TestAdminUpdateCatalogSearchSemanticSettingsValidation(t *testing.T) {
valid := []struct {
key string
value string
want string
}{
{key: catalog.SearchSettingMeilisearchSemanticEnabled, value: "TRUE", want: "true"},
{key: catalog.SearchSettingMeilisearchSemanticRatio, value: ".30", want: "0.3"},
{key: catalog.SearchSettingMeilisearchEmbedder, value: "", want: catalog.DefaultMeilisearchEmbedder},
{key: catalog.SearchSettingMeilisearchEmbedder, value: "custom_embedder-2", want: "custom_embedder-2"},
}
for _, tc := range valid {
t.Run("valid "+tc.key, func(t *testing.T) {
settings := &fakeServerSettingsStore{}
handler := &AdminHandler{SettingsRepo: settings, RestartStatus: NewServerRestartStatusTracker()}
req := httptest.NewRequest(
http.MethodPut,
"/admin/settings/"+tc.key,
strings.NewReader(`{"value":"`+tc.value+`"}`),
)
req = withChiParam(req, "key", tc.key)
rec := httptest.NewRecorder()
handler.HandleUpdateSetting(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("status = %d, want 200; body=%s", rec.Code, rec.Body.String())
}
if settings.values[tc.key] != tc.want {
t.Fatalf("stored value = %q, want %q", settings.values[tc.key], tc.want)
}
})
}
invalid := []struct {
key string
value string
}{
{key: catalog.SearchSettingMeilisearchSemanticEnabled, value: "sometimes"},
{key: catalog.SearchSettingMeilisearchSemanticRatio, value: "-0.01"},
{key: catalog.SearchSettingMeilisearchSemanticRatio, value: "1.01"},
{key: catalog.SearchSettingMeilisearchSemanticRatio, value: "NaN"},
{key: catalog.SearchSettingMeilisearchEmbedder, value: "bad.name"},
}
for _, tc := range invalid {
t.Run("invalid "+tc.key, func(t *testing.T) {
settings := &fakeServerSettingsStore{}
handler := &AdminHandler{SettingsRepo: settings}
req := httptest.NewRequest(
http.MethodPut,
"/admin/settings/"+tc.key,
strings.NewReader(`{"value":"`+tc.value+`"}`),
)
req = withChiParam(req, "key", tc.key)
rec := httptest.NewRecorder()
handler.HandleUpdateSetting(rec, req)
if rec.Code != http.StatusBadRequest {
t.Fatalf("status = %d, want 400; body=%s", rec.Code, rec.Body.String())
}
if _, ok := settings.values[tc.key]; ok {
t.Fatalf("invalid setting was stored: %#v", settings.values)
}
})
}
}
func performSettingsCheckRequest(
t *testing.T,
handler *AdminHandler,
+55 -13
View File
@@ -18,8 +18,10 @@ type audiobookGroupResponse struct {
}
type audiobookGroupsResponse struct {
Total int `json:"total"`
Groups []audiobookGroupResponse `json:"groups"`
Total int `json:"total"`
TotalExact bool `json:"total_exact"`
HasMore bool `json:"has_more"`
Groups []audiobookGroupResponse `json:"groups"`
}
// HandleGetAudiobookGroups — GET /api/v1/catalog/audiobook-groups
@@ -28,7 +30,7 @@ type audiobookGroupsResponse struct {
// aggregate stats (book count, total duration, per-profile progress counts)
// and up to four poster URLs for cover stacks. Query parameters:
// library_id=<id> (required), group_by=author|narrator|series (required),
// sort=name|count|duration (default name), limit, offset.
// sort=name|count|duration (default name), limit, offset, include_total, q.
//
// Group names round-trip into the corresponding catalog filter fields
// (author / narrator / series), which match case-insensitively.
@@ -68,16 +70,27 @@ func (h *CatalogHandler) HandleGetAudiobookGroups(w http.ResponseWriter, r *http
limit = n
}
offset := max(catalog.ParseIntParam(r.URL.Query().Get("offset")), 0)
includeTotal := true
if raw := strings.TrimSpace(r.URL.Query().Get("include_total")); raw != "" {
parsed, parseErr := strconv.ParseBool(raw)
if parseErr != nil {
writeError(w, http.StatusBadRequest, "bad_request", "include_total must be true or false")
return
}
includeTotal = parsed
}
groups, total, err := catalog.ListAudiobookGroups(
result, err := catalog.ListAudiobookGroups(
r.Context(),
h.itemsH.browseRepo.Pool(),
catalog.AudiobookGroupsQuery{
LibraryID: libraryID,
GroupBy: groupBy,
Sort: sort,
Limit: limit,
Offset: offset,
LibraryID: libraryID,
GroupBy: groupBy,
SearchPrefix: strings.TrimSpace(r.URL.Query().Get("q")),
IncludeTotal: includeTotal,
Sort: sort,
Limit: limit,
Offset: offset,
},
h.itemsH.accessFilter(r),
)
@@ -86,12 +99,18 @@ func (h *CatalogHandler) HandleGetAudiobookGroups(w http.ResponseWriter, r *http
return
}
resp := audiobookGroupsResponse{Total: total, Groups: make([]audiobookGroupResponse, 0, len(groups))}
for _, g := range groups {
resolvedPosters := h.resolveAudiobookGroupPosterURLs(r, result.Groups)
resp := audiobookGroupsResponse{
Total: result.Total,
TotalExact: result.TotalExact,
HasMore: result.HasMore,
Groups: make([]audiobookGroupResponse, 0, len(result.Groups)),
}
for _, g := range result.Groups {
posterURLs := make([]string, 0, len(g.PosterPaths))
for _, path := range g.PosterPaths {
if url := h.itemsH.presignURL(r, cardThumbnailPath(path), "card"); url != "" {
posterURLs = append(posterURLs, url)
if resolved := resolvedPosters[cardThumbnailPath(path)]; resolved.URL != "" {
posterURLs = append(posterURLs, resolved.URL)
}
}
resp.Groups = append(resp.Groups, audiobookGroupResponse{
@@ -106,3 +125,26 @@ func (h *CatalogHandler) HandleGetAudiobookGroups(w http.ResponseWriter, r *http
writeJSON(w, http.StatusOK, resp)
}
func (h *CatalogHandler) resolveAudiobookGroupPosterURLs(r *http.Request, groups []catalog.AudiobookGroup) map[string]catalog.ResolvedImageURL {
if h == nil || h.itemsH == nil || h.itemsH.detailSvc == nil || len(groups) == 0 {
return map[string]catalog.ResolvedImageURL{}
}
paths := make([]string, 0, len(groups)*4)
seen := make(map[string]struct{}, len(groups)*4)
for _, group := range groups {
for _, path := range group.PosterPaths {
normalized := cardThumbnailPath(path)
if normalized == "" {
continue
}
if _, ok := seen[normalized]; ok {
continue
}
seen[normalized] = struct{}{}
paths = append(paths, normalized)
}
}
return h.itemsH.detailSvc.PresignURLsWithExpiry(r.Context(), paths, "card")
}
+315 -34
View File
@@ -8,6 +8,7 @@ import (
"net/http"
"strconv"
"strings"
"sync"
"time"
apimw "github.com/Silo-Server/silo-server/internal/api/middleware"
@@ -16,9 +17,28 @@ import (
"github.com/Silo-Server/silo-server/internal/sections"
)
// audiobookGroupsCacheTTL matches the client's React Query staleTime: the
// grouped author/narrator browse is cached server-side for the same window so
// the sequential page fetches and a quick refresh reuse one aggregation.
const audiobookGroupsCacheTTL = 60 * time.Second
type CatalogHandler struct {
resolver *catalog.CatalogResolver
itemsH *ItemsHandler
resolver *catalog.CatalogResolver
itemsH *ItemsHandler
workSummary catalog.WorkSummaryProvider
groupsCacheOnce sync.Once
groupsCache *catalog.AudiobookGroupsCache
}
// audiobookGroups returns the lazily-initialized grouped-browse cache. Built on
// first use (only the route-mounted singleton handler serves audiobook-groups)
// so the per-request CatalogHandler instances never spawn a cache sweeper.
func (h *CatalogHandler) audiobookGroups() *catalog.AudiobookGroupsCache {
h.groupsCacheOnce.Do(func() {
h.groupsCache = catalog.NewAudiobookGroupsCache(h.itemsH.browseRepo.Pool(), audiobookGroupsCacheTTL)
})
return h.groupsCache
}
func NewCatalogHandler(resolver *catalog.CatalogResolver, itemsH *ItemsHandler) *CatalogHandler {
@@ -28,12 +48,31 @@ func NewCatalogHandler(resolver *catalog.CatalogResolver, itemsH *ItemsHandler)
}
}
func (h *CatalogHandler) SetWorkSummaryProvider(provider catalog.WorkSummaryProvider) {
h.workSummary = provider
}
type catalogResponse struct {
Total int `json:"total"`
TotalExact bool `json:"total_exact"`
HasMore bool `json:"has_more"`
Items []itemListResponse `json:"items"`
Snapshot string `json:"snapshot,omitempty"`
Total int `json:"total"`
TotalExact bool `json:"total_exact"`
HasMore bool `json:"has_more"`
Items []itemListResponse `json:"items"`
Snapshot string `json:"snapshot,omitempty"`
SearchDiagnostics *searchDiagnostics `json:"search_diagnostics,omitempty"`
}
// searchDiagnostics is an additive, per-query observability object emitted on
// /api/v1/catalog only when a relevance-sorted search actually ran through a
// CatalogSearchProvider. mode/semantic_used reflect POST-downgrade reality
// (a hybrid request that fell back to keyword reports mode="keyword",
// semantic_used=false). fallback_reason and index_pending_updates are omitted
// when empty.
type searchDiagnostics struct {
Provider string `json:"provider"`
Mode string `json:"mode"`
SemanticUsed bool `json:"semantic_used"`
FallbackReason string `json:"fallback_reason,omitempty"`
IndexPendingUpdates int `json:"index_pending_updates,omitempty"`
}
type catalogFiltersResponse struct {
@@ -62,8 +101,36 @@ func (h *CatalogHandler) HandleGetCatalog(w http.ResponseWriter, r *http.Request
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
return
}
accessFilter := h.itemsH.accessFilter(r)
groupedByWork := strings.EqualFold(strings.TrimSpace(r.URL.Query().Get("group")), "work")
if groupedByWork {
result, entries, err := h.resolveGroupedCatalogByWork(r, req, accessFilter)
if err != nil {
if errors.Is(err, catalog.ErrInvalidCatalogRequest) {
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
return
}
if errors.Is(err, catalog.ErrCatalogSourceNotFound) {
writeError(w, http.StatusNotFound, "not_found", "Catalog source not found")
return
}
slog.Error("catalog: resolve grouped by work failed", "err_msg", err.Error())
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to resolve catalog")
return
}
result, err := h.resolver.Resolve(r.Context(), req, h.itemsH.accessFilter(r))
resultItems := groupedCatalogItems(entries)
items := h.catalogItemResponses(r, resultItems, catalog.NormalizeQuerySort(req.Query.Sort).Field, accessFilter)
for i := range items {
if i < len(entries) && entries[i].summary != nil {
applyWorkSummaryToCatalogItem(&items[i], entries[i].summary)
}
}
h.writeCatalogResponse(w, result, items, groupedByWork)
return
}
result, err := h.resolver.Resolve(r.Context(), req, accessFilter)
if err != nil {
if errors.Is(err, catalog.ErrInvalidCatalogRequest) {
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
@@ -77,53 +144,267 @@ func (h *CatalogHandler) HandleGetCatalog(w http.ResponseWriter, r *http.Request
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to resolve catalog")
return
}
items := h.catalogItemResponses(r, result.Items, catalog.NormalizeQuerySort(req.Query.Sort).Field, accessFilter)
h.writeCatalogResponse(w, result, items, groupedByWork)
}
overlaySummaries := h.itemsH.listOverlaySummaries(r.Context(), result.Items, h.itemsH.accessFilter(r))
userStates := h.itemsH.listItemUserStates(r, result.Items)
episodeMetadata := h.itemsH.listEpisodeBrowseMetadata(r.Context(), result.Items)
sortField := catalog.NormalizeQuerySort(req.Query.Sort).Field
store, profileID, _ := h.itemsH.userStoreForRequest(r)
sortMetrics := h.itemsH.listSortMetrics(
r.Context(),
result.Items,
sortField,
h.itemsH.accessFilter(r),
overlaySummaries,
store,
apimw.GetUserID(r.Context()),
profileID,
func (h *CatalogHandler) catalogItemResponses(r *http.Request, resultItems []*models.MediaItem, sortField string, accessFilter catalog.AccessFilter) []itemListResponse {
var (
localizedItems []*models.MediaItem
overlaySummaries map[string]*models.OverlaySummary
userStates map[string]*itemUserStateResponse
episodeMetadata map[string]struct {
SeriesTitle string
SeasonNumber *int
EpisodeNumber *int
}
)
items := make([]itemListResponse, 0, len(result.Items))
for _, item := range result.Items {
resp := h.itemsH.toItemListResponseWithOverlay(
r,
item,
overlaySummaries[item.ContentID],
userStates[item.ContentID],
var enrichWG sync.WaitGroup
enrichWG.Add(4)
go func() {
defer enrichWG.Done()
localizedItems = h.itemsH.localizeItemListModels(r.Context(), resultItems, accessFilter)
}()
go func() {
defer enrichWG.Done()
overlaySummaries = h.itemsH.listOverlaySummaries(r.Context(), resultItems, accessFilter)
}()
go func() {
defer enrichWG.Done()
userStates = h.itemsH.listItemUserStates(r, resultItems)
}()
go func() {
defer enrichWG.Done()
episodeMetadata = h.itemsH.listEpisodeBrowseMetadata(r.Context(), resultItems)
}()
enrichWG.Wait()
var (
imageURLs map[string]itemListImageURLs
sortMetrics map[string]*sortMetricsResponse
)
store, profileID, _ := h.itemsH.userStoreForRequest(r)
var responseWG sync.WaitGroup
responseWG.Add(2)
go func() {
defer responseWG.Done()
imageURLs = h.itemsH.itemListCardImageURLs(r.Context(), localizedItems)
}()
go func() {
defer responseWG.Done()
sortMetrics = h.itemsH.listSortMetrics(
r.Context(),
resultItems,
sortField,
accessFilter,
overlaySummaries,
store,
apimw.GetUserID(r.Context()),
profileID,
)
}()
responseWG.Wait()
items := make([]itemListResponse, 0, len(localizedItems))
for _, item := range localizedItems {
if item == nil {
continue
}
resp := itemListResponseShell(item, overlaySummaries[item.ContentID], userStates[item.ContentID])
if meta, ok := episodeMetadata[item.ContentID]; ok {
resp.SeriesTitle = meta.SeriesTitle
resp.SeasonNumber = meta.SeasonNumber
resp.EpisodeNumber = meta.EpisodeNumber
}
resp.PosterURL = imageURLs[item.ContentID].posterURL
resp.BackdropURL = imageURLs[item.ContentID].backdropURL
resp.SortMetrics = sortMetrics[item.ContentID]
items = append(items, resp)
}
return items
}
func (h *CatalogHandler) writeCatalogResponse(w http.ResponseWriter, result *catalog.CatalogResult, items []itemListResponse, groupedByWork bool) {
var snapshot string
if !result.SnapshotAt.IsZero() {
snapshot = result.SnapshotAt.Format(time.RFC3339Nano)
}
// A non-empty Provider is the single gate: only the direct-search path sets
// it. Browse / preview / non-relevance-sort q= (which never run a provider)
// and group=work (fresh CatalogResult with empty Provider) all omit it.
var diag *searchDiagnostics
if result.Provider != "" {
diag = &searchDiagnostics{
Provider: result.Provider,
Mode: result.Mode,
SemanticUsed: result.SemanticUsed,
FallbackReason: result.FallbackReason,
IndexPendingUpdates: result.IndexPendingEvents,
}
}
writeJSON(w, http.StatusOK, catalogResponse{
Total: result.Total,
TotalExact: result.TotalExact,
HasMore: result.HasMore,
Items: items,
Snapshot: snapshot,
Total: result.Total,
TotalExact: result.TotalExact && !groupedByWork,
HasMore: result.HasMore,
Items: items,
Snapshot: snapshot,
SearchDiagnostics: diag,
})
}
type groupedCatalogEntry struct {
item *models.MediaItem
summary *catalog.WorkSummary
}
func (h *CatalogHandler) resolveGroupedCatalogByWork(r *http.Request, req catalog.CatalogRequest, accessFilter catalog.AccessFilter) (*catalog.CatalogResult, []groupedCatalogEntry, error) {
fetchReq := req
fetchReq.Offset = 0
fetchReq.Limit = groupedCatalogFetchLimit(req.Limit)
fetchReq.SkipTotal = true
seen := map[string]struct{}{}
entries := make([]groupedCatalogEntry, 0, req.Limit+1)
groupIndex := 0
var snapshot time.Time
for {
result, err := h.resolver.Resolve(r.Context(), fetchReq, accessFilter)
if err != nil {
return nil, nil, err
}
if snapshot.IsZero() {
snapshot = result.SnapshotAt
if !snapshot.IsZero() {
fetchReq.SnapshotAt = &snapshot
}
}
summaries, err := h.workSummariesForItems(r, result.Items, accessFilter)
if err != nil {
return nil, nil, err
}
for _, item := range result.Items {
summary := summaries[item.ContentID]
key := groupedCatalogEntryKey(item, summary)
if _, ok := seen[key]; ok {
continue
}
seen[key] = struct{}{}
if groupIndex >= req.Offset {
entries = append(entries, groupedCatalogEntry{item: item, summary: summary})
}
groupIndex++
if len(entries) > req.Limit {
break
}
}
if len(entries) > req.Limit || !result.HasMore || len(result.Items) == 0 {
break
}
fetchReq.Offset += len(result.Items)
}
hasMore := len(entries) > req.Limit
if hasMore {
entries = entries[:req.Limit]
}
total := req.Offset + len(entries)
if hasMore {
total++
}
result := &catalog.CatalogResult{
Items: groupedCatalogItems(entries),
Total: total,
HasMore: hasMore,
TotalExact: false,
SnapshotAt: snapshot,
}
return result, entries, nil
}
func groupedCatalogFetchLimit(limit int) int {
switch {
case limit <= 0:
return 100
case limit >= 100:
return 100
default:
return min(100, limit*4+10)
}
}
func (h *CatalogHandler) workSummariesForItems(r *http.Request, items []*models.MediaItem, filter catalog.AccessFilter) (map[string]*catalog.WorkSummary, error) {
summaries := map[string]*catalog.WorkSummary{}
if h == nil || h.workSummary == nil || len(items) == 0 {
return summaries, nil
}
contentIDs := make([]string, 0, len(items))
seen := map[string]struct{}{}
for _, item := range items {
if !catalogItemCanGroupByWork(item) {
continue
}
if _, ok := seen[item.ContentID]; ok {
continue
}
seen[item.ContentID] = struct{}{}
contentIDs = append(contentIDs, item.ContentID)
}
if len(contentIDs) == 0 {
return summaries, nil
}
if batch, ok := h.workSummary.(catalog.WorkSummaryBatchProvider); ok {
return batch.ListSummariesForContentIDs(r.Context(), contentIDs, filter)
}
for _, contentID := range contentIDs {
summary, err := h.workSummary.GetSummaryForContentID(r.Context(), contentID, filter)
if err != nil {
return nil, err
}
if summary != nil && summary.WorkID != "" {
summaries[contentID] = summary
}
}
return summaries, nil
}
func groupedCatalogItems(entries []groupedCatalogEntry) []*models.MediaItem {
items := make([]*models.MediaItem, 0, len(entries))
for _, entry := range entries {
items = append(items, entry.item)
}
return items
}
func groupedCatalogEntryKey(item *models.MediaItem, summary *catalog.WorkSummary) string {
if catalogItemCanGroupByWork(item) && summary != nil && summary.WorkID != "" {
return "work:" + summary.WorkID
}
if item == nil {
return "item:"
}
return "item:" + item.ContentID
}
func catalogItemCanGroupByWork(item *models.MediaItem) bool {
return item != nil && (item.Type == "ebook" || item.Type == "audiobook")
}
func applyWorkSummaryToCatalogItem(item *itemListResponse, summary *catalog.WorkSummary) {
if item == nil || summary == nil || summary.WorkID == "" {
return
}
item.Type = "work"
item.WorkID = summary.WorkID
item.WorkTitle = summary.Title
item.WorkFormats = summary.Formats
if summary.Title != "" {
item.Title = summary.Title
}
}
func (h *CatalogHandler) HandleGetCatalogFilters(w http.ResponseWriter, r *http.Request) {
if h == nil || h.resolver == nil || h.itemsH == nil {
writeError(w, http.StatusInternalServerError, "internal_error", "Catalog is not configured")
@@ -0,0 +1,123 @@
package handlers
import (
"encoding/json"
"net/http/httptest"
"testing"
"github.com/Silo-Server/silo-server/internal/catalog"
)
// decodeCatalogResponse runs writeCatalogResponse against an httptest recorder
// and returns the decoded JSON body as a generic map so individual keys can be
// asserted for presence/absence (search_diagnostics is omitempty).
func decodeCatalogResponse(t *testing.T, result *catalog.CatalogResult, grouped bool) map[string]any {
t.Helper()
rec := httptest.NewRecorder()
// writeCatalogResponse uses no handler state, so a zero-value handler is fine.
(&CatalogHandler{}).writeCatalogResponse(rec, result, []itemListResponse{}, grouped)
var body map[string]any
if err := json.Unmarshal(rec.Body.Bytes(), &body); err != nil {
t.Fatalf("decoding catalog response: %v\nbody: %s", err, rec.Body.String())
}
return body
}
func TestWriteCatalogResponse_DiagnosticsKeywordFallback(t *testing.T) {
body := decodeCatalogResponse(t, &catalog.CatalogResult{
Total: 3,
TotalExact: true,
HasMore: false,
Provider: catalog.SearchProviderMeilisearch,
Mode: "keyword",
SemanticUsed: false,
FallbackReason: `semantic_not_ready: type "movie" coverage 40% below threshold`,
}, false)
// Existing keys must stay present and unchanged (byte-stable contract).
for _, key := range []string{"total", "total_exact", "has_more", "items"} {
if _, ok := body[key]; !ok {
t.Fatalf("response missing existing key %q: %v", key, body)
}
}
if body["total"].(float64) != 3 {
t.Fatalf("total = %v, want 3", body["total"])
}
if body["total_exact"].(bool) != true {
t.Fatalf("total_exact = %v, want true", body["total_exact"])
}
diagRaw, ok := body["search_diagnostics"]
if !ok {
t.Fatalf("expected search_diagnostics in response: %v", body)
}
diag := diagRaw.(map[string]any)
if diag["provider"] != catalog.SearchProviderMeilisearch {
t.Fatalf("provider = %v, want %q", diag["provider"], catalog.SearchProviderMeilisearch)
}
if diag["mode"] != "keyword" {
t.Fatalf("mode = %v, want keyword", diag["mode"])
}
if diag["semantic_used"].(bool) != false {
t.Fatalf("semantic_used = %v, want false", diag["semantic_used"])
}
if diag["fallback_reason"] != `semantic_not_ready: type "movie" coverage 40% below threshold` {
t.Fatalf("fallback_reason = %v", diag["fallback_reason"])
}
}
func TestWriteCatalogResponse_DiagnosticsHybridOmitsFallbackReason(t *testing.T) {
body := decodeCatalogResponse(t, &catalog.CatalogResult{
Provider: catalog.SearchProviderMeilisearch,
Mode: "hybrid",
SemanticUsed: true,
IndexPendingEvents: 7,
}, false)
diagRaw, ok := body["search_diagnostics"]
if !ok {
t.Fatalf("expected search_diagnostics in response: %v", body)
}
diag := diagRaw.(map[string]any)
if diag["semantic_used"].(bool) != true {
t.Fatalf("semantic_used = %v, want true", diag["semantic_used"])
}
if diag["mode"] != "hybrid" {
t.Fatalf("mode = %v, want hybrid", diag["mode"])
}
if _, ok := diag["fallback_reason"]; ok {
t.Fatalf("fallback_reason should be omitted when empty: %v", diag)
}
if diag["index_pending_updates"].(float64) != 7 {
t.Fatalf("index_pending_updates = %v, want 7", diag["index_pending_updates"])
}
}
func TestWriteCatalogResponse_NoProviderOmitsDiagnostics(t *testing.T) {
// Browse / preview / non-relevance q= paths never set Provider.
body := decodeCatalogResponse(t, &catalog.CatalogResult{
Total: 5,
TotalExact: true,
}, false)
if _, ok := body["search_diagnostics"]; ok {
t.Fatalf("search_diagnostics should be omitted when no provider search ran: %v", body)
}
}
func TestWriteCatalogResponse_GroupedByWorkOmitsDiagnostics(t *testing.T) {
// group=work builds a fresh CatalogResult with an empty Provider.
body := decodeCatalogResponse(t, &catalog.CatalogResult{
Total: 2,
TotalExact: true,
}, true)
if _, ok := body["search_diagnostics"]; ok {
t.Fatalf("grouped response should omit search_diagnostics: %v", body)
}
// Grouped responses force total_exact false regardless of result.TotalExact.
if body["total_exact"].(bool) != false {
t.Fatalf("grouped total_exact = %v, want false", body["total_exact"])
}
}
@@ -0,0 +1,53 @@
package handlers
import (
"context"
"testing"
"github.com/Silo-Server/silo-server/internal/catalog"
"github.com/Silo-Server/silo-server/internal/models"
)
type fakeCatalogWorkSummaryProvider struct{}
func (fakeCatalogWorkSummaryProvider) GetSummaryForContentID(ctx context.Context, contentID string, filter catalog.AccessFilter) (*catalog.WorkSummary, error) {
if contentID != "ebook-1" && contentID != "audio-1" {
return nil, nil
}
return &catalog.WorkSummary{
WorkID: "work-1",
Title: "Project Hail Mary",
Formats: []catalog.WorkFormatSummary{
{Type: "ebook", ContentID: "ebook-1", LibraryID: 1},
{Type: "audiobook", ContentID: "audio-1", LibraryID: 2},
},
}, nil
}
func TestGroupedCatalogEntryKeyDeduplicatesLinkedFormats(t *testing.T) {
summary := &catalog.WorkSummary{
WorkID: "work-1",
Title: "Project Hail Mary",
Formats: []catalog.WorkFormatSummary{
{Type: "ebook", ContentID: "ebook-1", LibraryID: 1},
{Type: "audiobook", ContentID: "audio-1", LibraryID: 2},
},
}
ebookKey := groupedCatalogEntryKey(&models.MediaItem{ContentID: "ebook-1", Type: "ebook"}, summary)
audioKey := groupedCatalogEntryKey(&models.MediaItem{ContentID: "audio-1", Type: "audiobook"}, summary)
if ebookKey != audioKey {
t.Fatalf("linked format keys differ: %q vs %q", ebookKey, audioKey)
}
movieKey := groupedCatalogEntryKey(&models.MediaItem{ContentID: "movie-1", Type: "movie"}, nil)
if movieKey != "item:movie-1" {
t.Fatalf("movie key = %q, want item key", movieKey)
}
resp := itemListResponse{ContentID: "ebook-1", Type: "ebook", Title: "Project Hail Mary"}
applyWorkSummaryToCatalogItem(&resp, summary)
if resp.Type != "work" || resp.WorkID != "work-1" || len(resp.WorkFormats) != 2 {
t.Fatalf("work response = %#v", resp)
}
}
+73 -6
View File
@@ -87,6 +87,37 @@ func (h *CatalogResourceHandler) HandleGetItemVersions(w http.ResponseWriter, r
writeJSON(w, http.StatusOK, detail.Versions)
}
// HandleGetMangaFiles returns the local file listing for a manga series (the
// series "View Details" dialog): folder paths plus per-chapter file rows.
// Folder and file paths are stripped for viewers without file-path visibility,
// matching the item-versions policy; file names and sizes remain.
func (h *CatalogResourceHandler) HandleGetMangaFiles(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
if id == "" {
writeError(w, http.StatusBadRequest, "bad_request", "Item ID is required")
return
}
files, err := h.items.detailSvc.GetMangaChapterFiles(r.Context(), id, h.items.accessFilter(r))
if err != nil {
if isNotFound(err) {
writeError(w, http.StatusNotFound, "not_found", "Item not found")
return
}
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to get manga files")
return
}
if !h.items.requestCanViewFilePaths(r) {
files.FolderPaths = nil
for i := range files.Files {
files.Files[i].FilePath = ""
}
}
writeJSON(w, http.StatusOK, files)
}
func (h *CatalogResourceHandler) HandleGetItemEpisodes(w http.ResponseWriter, r *http.Request) {
filter := h.items.accessFilter(r)
id := chi.URLParam(r, "id")
@@ -197,12 +228,24 @@ func (h *CatalogResourceHandler) HandleGetSeasons(w http.ResponseWriter, r *http
}
if len(seasons) > 0 {
episodesBySeason, err := h.items.episodeRepo.ListBySeriesGroupedBySeason(r.Context(), id)
if err != nil {
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to list seasons")
return
}
progressMap, hasProgressMap := h.items.progressMapForEpisodes(r, flattenEpisodeGroups(episodesBySeason))
resp := make([]seasonResponse, 0, len(seasons))
for _, s := range seasons {
sr := h.items.toSeasonResponse(r, id, s)
if sr.EpisodeCount == 0 {
episodes := episodesBySeason[s.SeasonNumber]
if len(episodes) == 0 {
continue
}
var userData *catalog.SeasonUserData
if hasProgressMap {
userData = catalog.EpisodeRollupUserData(episodes, progressMap)
}
sr := h.items.toSeasonResponseFromEpisodes(r, id, s, episodes, userData)
resp = append(resp, sr)
}
@@ -224,13 +267,14 @@ func (h *CatalogResourceHandler) HandleGetSeasons(w http.ResponseWriter, r *http
title = "Specials"
}
episodes, _ := h.items.episodeRepo.ListBySeason(r.Context(), id, s.SeasonNumber)
userData := h.items.getAggregateUserData(r, episodes)
resp = append(resp, seasonResponse{
ContentID: fmt.Sprintf("%s-S%02d", id, s.SeasonNumber),
SeasonNumber: s.SeasonNumber,
IsSpecials: s.SeasonNumber == 0,
EpisodeCount: s.EpisodeCount,
Title: title,
UserData: h.items.getAggregateUserData(r, episodes),
UserData: userData,
})
}
@@ -273,10 +317,27 @@ func (h *CatalogResourceHandler) HandleGetSeason(w http.ResponseWriter, r *http.
season, err := h.items.seasonRepo.GetBySeriesAndNumber(r.Context(), id, num)
switch {
case err == nil:
episodes, _ := h.items.episodeRepo.ListBySeasonID(r.Context(), season.ContentID)
episodes, err := h.items.episodeRepo.ListBySeasonID(r.Context(), season.ContentID)
if err != nil {
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to get season")
return
}
if len(episodes) == 0 {
episodes, err = h.items.episodeRepo.ListBySeason(r.Context(), id, season.SeasonNumber)
if err != nil {
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to get season")
return
}
}
h.items.maybeRequestStaleSeasonMetadataRefresh(r.Context(), season.ContentID, episodes)
writeJSON(w, http.StatusOK, seasonDetailResponse{
Season: h.items.toSeasonResponse(r, id, season),
Season: h.items.toSeasonResponseFromEpisodes(
r,
id,
season,
episodes,
h.items.getAggregateUserData(r, episodes),
),
})
return
case !errors.Is(err, catalog.ErrSeasonNotFound):
@@ -388,7 +449,13 @@ func (h *CatalogResourceHandler) syntheticSeasonDetail(r *http.Request, seasonID
} else {
season.Title = "Season " + strconv.Itoa(seasonNum)
}
seasonResp := h.items.toSeasonResponse(r, seriesID, season)
seasonResp := h.items.toSeasonResponseFromEpisodes(
r,
seriesID,
season,
episodes,
h.items.getAggregateUserData(r, episodes),
)
return &catalog.ItemDetail{
ContentID: seasonID,
Type: "season",
+58 -1
View File
@@ -43,6 +43,7 @@ type createCollectionRequest struct {
AllowedProfileIDs []string `json:"allowed_profile_ids"`
QueryDefinition json.RawMessage `json:"query_definition"`
SortConfig json.RawMessage `json:"sort_config"`
DisplayQueryDefinition json.RawMessage `json:"display_query_definition"`
IncludeInServerCollections bool `json:"include_in_server_collections"`
PosterSourceURL string `json:"poster_source_url"`
}
@@ -56,6 +57,8 @@ type updateCollectionRequest struct {
SortConfig json.RawMessage `json:"sort_config"`
SourceURL *string `json:"source_url"`
MaxItems *int `json:"max_items"`
LibraryIDs *[]int `json:"library_ids"`
DisplayQueryDefinition json.RawMessage `json:"display_query_definition"`
IncludeInServerCollections *bool `json:"include_in_server_collections"`
PosterSourceURL *string `json:"poster_source_url"`
GroupID optionalNullableString `json:"group_id"`
@@ -85,6 +88,7 @@ type collectionResponse struct {
LastSyncAt string `json:"last_sync_at,omitempty"`
LastSyncStatus string `json:"last_sync_status,omitempty"`
LastSyncMessage string `json:"last_sync_message,omitempty"`
DisplayQueryDefinition json.RawMessage `json:"display_query_definition,omitempty"`
ItemCount int `json:"item_count"`
IncludeInServerCollections bool `json:"include_in_server_collections"`
PosterURL string `json:"poster_url,omitempty"`
@@ -98,6 +102,19 @@ type collectionListResponse struct {
Groups []collectionGroupResponse `json:"groups"`
}
type collectionCapabilitiesResponse struct {
// DisplayFilterFields are the catalog query fields a personal-collection
// display filter may use. Clients build a display_query_definition fragment
// from these rather than a bespoke enum.
DisplayFilterFields []string `json:"display_filter_fields"`
DisplayFilterPresets collectionDisplayFilterPresets `json:"display_filter_presets"`
}
type collectionDisplayFilterPresets struct {
Watched []string `json:"watched"`
Media []string `json:"media"`
}
type collectionGroupResponse struct {
ID string `json:"id"`
Name string `json:"name"`
@@ -208,6 +225,17 @@ func (h *CollectionHandler) HandleListCollections(w http.ResponseWriter, r *http
writeJSON(w, http.StatusOK, resp)
}
// HandleCapabilities exposes additive feature support for collection clients.
func (h *CollectionHandler) HandleCapabilities(w http.ResponseWriter, r *http.Request) {
writeJSON(w, http.StatusOK, collectionCapabilitiesResponse{
DisplayFilterFields: []string{"type", "watched"},
DisplayFilterPresets: collectionDisplayFilterPresets{
Watched: []string{"all", "watched", "unwatched"},
Media: []string{"all", "movie", "series"},
},
})
}
// HandleCreateCollection handles POST /collections.
func (h *CollectionHandler) HandleCreateCollection(w http.ResponseWriter, r *http.Request) {
userID := apimw.GetUserID(r.Context())
@@ -247,6 +275,11 @@ func (h *CollectionHandler) HandleCreateCollection(w http.ResponseWriter, r *htt
}
queryDefinition := string(queryDefinitionJSON)
sortConfig := string(defaultJSON(req.SortConfig))
displayQueryDefinition, err := catalog.NormalizeDisplayQueryFragment(req.DisplayQueryDefinition)
if err != nil {
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
return
}
collection, err := store.CreateCollection(r.Context(), userstore.CreateCollectionInput{
CreatorProfileID: profileID,
Name: req.Name,
@@ -255,6 +288,7 @@ func (h *CollectionHandler) HandleCreateCollection(w http.ResponseWriter, r *htt
AllowedProfileIDs: req.AllowedProfileIDs,
QueryDefinition: queryDefinition,
SortConfig: sortConfig,
DisplayQueryDefinition: displayQueryDefinition,
IncludeInServerCollections: req.IncludeInServerCollections,
})
if err != nil {
@@ -306,6 +340,14 @@ func (h *CollectionHandler) HandleUpdateCollection(w http.ResponseWriter, r *htt
AllowedProfileIDs: req.AllowedProfileIDs,
IncludeInServerCollections: req.IncludeInServerCollections,
}
if req.DisplayQueryDefinition != nil {
displayQueryDefinition, err := catalog.NormalizeDisplayQueryFragment(req.DisplayQueryDefinition)
if err != nil {
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
return
}
input.DisplayQueryDefinition = &displayQueryDefinition
}
if len(req.QueryDefinition) > 0 {
normalized, err := normalizeSmartCollectionQueryDefinitionJSON(req.QueryDefinition, true, true)
if err != nil {
@@ -341,7 +383,7 @@ func (h *CollectionHandler) HandleUpdateCollection(w http.ResponseWriter, r *htt
// Source URL and max items both live inside source_config (Limit / URL).
// Load the existing collection and re-marshal so the unaffected fields
// (preset, media_type, etc.) survive untouched.
if req.SourceURL != nil || req.MaxItems != nil {
if req.SourceURL != nil || req.MaxItems != nil || req.LibraryIDs != nil {
existing, err := store.GetCollection(r.Context(), collectionID)
if err != nil {
if strings.Contains(err.Error(), "not found") {
@@ -356,6 +398,18 @@ func (h *CollectionHandler) HandleUpdateCollection(w http.ResponseWriter, r *htt
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to parse source config")
return
}
if req.LibraryIDs != nil {
if !catalog.IsSyncableType(existing.CollectionType) {
writeError(w, http.StatusBadRequest, "bad_request", "library_ids can only be edited for imported collections")
return
}
if !validateOptionalLibraryIDs(*req.LibraryIDs, w) {
return
}
cfg.LibraryIDs = append([]int(nil), (*req.LibraryIDs)...)
emptyQuery := "{}"
input.QueryDefinition = &emptyQuery
}
if req.MaxItems != nil {
if *req.MaxItems < 0 {
writeError(w, http.StatusBadRequest, "bad_request", "max_items must be zero or positive")
@@ -805,6 +859,9 @@ func toCollectionResponse(c userstore.Collection) collectionResponse {
CreatedAt: c.CreatedAt,
UpdatedAt: c.UpdatedAt,
}
if strings.TrimSpace(c.DisplayQueryDefinition) != "" {
resp.DisplayQueryDefinition = json.RawMessage(c.DisplayQueryDefinition)
}
if c.SourceConfig != "" && c.SourceConfig != "{}" {
resp.SourceConfig = json.RawMessage(c.SourceConfig)
}
@@ -0,0 +1,184 @@
package handlers
import (
"context"
"errors"
"fmt"
"net/http"
"os"
"path/filepath"
"strings"
"github.com/Silo-Server/silo-server/internal/ebookconvert"
"github.com/Silo-Server/silo-server/internal/models"
)
// ConversionHeader tells the client how a kindle-family read was served.
// "converted" => body is EPUB; "failed" => body is the raw original and the
// client should fall back to opening it externally. Absent => no conversion
// was attempted (feature off or non-kindle format), serve/behave as before.
const ConversionHeader = "X-Silo-Ebook-Conversion"
// EbookConverter produces (and caches) a converted EPUB for a source file.
// Implemented by *ebookconvert.Cache; an interface here for testability.
type EbookConverter interface {
GetOrConvert(ctx context.Context, srcPath string, key ebookconvert.SourceKey) (string, error)
// Lookup reports a cached conversion WITHOUT converting (used by HEAD so a
// metadata probe never kicks off a full conversion): a path on hit, a
// non-nil error for a negatively-cached source, else a miss.
Lookup(key ebookconvert.SourceKey) (string, error, bool)
}
// EbookConversion bundles the converter with the admin-flag predicate. A nil
// *EbookConversion (or nil Converter) means the feature is off.
type EbookConversion struct {
Converter EbookConverter
// Enabled reports whether the admin flag is on for this request.
Enabled func(ctx context.Context) bool
}
func (c *EbookConversion) active(ctx context.Context) bool {
return c != nil && c.Converter != nil && c.Enabled != nil && c.Enabled(ctx)
}
// kindleReaderFormats are the Kindle-family formats the converter handles.
var kindleReaderFormats = map[string]bool{"mobi": true, "azw": true, "azw3": true}
func isKindleReaderFile(file *models.MediaFile) bool {
return file != nil && kindleReaderFormats[ebookReaderFormat(file.FilePath, file.Container)]
}
// serveEbook serves an ebook file, converting Kindle-family formats to EPUB
// when the feature is enabled. On conversion failure it serves the raw original
// with the ConversionHeader=failed contract so the client opens it externally.
func (h *EbookReaderHandler) serveEbook(w http.ResponseWriter, r *http.Request, file *models.MediaFile) error {
if h.Conversion.active(r.Context()) && isKindleReaderFile(file) {
// Strong identity for both the conversion cache and the response ETag:
// id + size + mtime (stat) + content hash + module version.
key, statErr := ebookconvert.SourceKeyFromStat(file.ID, file.FilePath)
if statErr != nil {
return h.serveRawKindleFallback(w, r, file)
}
key.Checksum = file.FileHash
// HEAD must not trigger a (possibly minute-long, 1 GiB) conversion. Answer
// from cache only; on a miss, advertise the converted representation
// cheaply and let the GET produce the body + authoritative verdict.
if r.Method == http.MethodHead {
return h.serveKindleHead(w, r, file, key)
}
epubPath, err := h.Conversion.Converter.GetOrConvert(r.Context(), file.FilePath, key)
switch {
case err == nil:
if serveErr := serveConvertedEpub(w, r, file, epubPath, key); serveErr != nil {
// Produced but unservable (open/stat) — honor the contract, not a 500.
return h.serveRawKindleFallback(w, r, file)
}
return nil
case errors.Is(err, context.Canceled), errors.Is(err, context.DeadlineExceeded):
// Caller went away / request cancelled — not a conversion verdict.
return err
default:
// DRM, corrupt, oversize, timed out, unavailable: fall back to the raw
// original and tell the client so it can open externally.
return h.serveRawKindleFallback(w, r, file)
}
}
// Feature off or non-kindle format: serve the raw original with its native
// MIME and no conversion header. This path deliberately omits the failed
// contract — when conversion is disabled the capability endpoint already
// reports `enabled:false`, so clients keep their external-open path and never
// expect an EPUB here. (No representation flip to guard against.)
return serveEbookInline(w, r, file)
}
// serveRawKindleFallback serves the raw original with the failed-conversion
// contract. no-store because the same URL flips raw/converted with the flag.
func (h *EbookReaderHandler) serveRawKindleFallback(w http.ResponseWriter, r *http.Request, file *models.MediaFile) error {
w.Header().Set(ConversionHeader, "failed")
w.Header().Set("Cache-Control", "no-store")
return serveEbookInline(w, r, file)
}
// serveKindleHead answers a HEAD for a kindle-family file without converting:
// - cache hit → real converted headers (and ServeContent's Content-Length);
// - neg-cached → the failed contract (raw original headers);
// - miss → converted headers advertised optimistically (no body, no
// Content-Length, no conversion). A HEAD is only a hint; the GET delivers
// the body and the authoritative DRM/failure verdict, which then populates
// the (positive or negative) cache that subsequent HEADs read.
func (h *EbookReaderHandler) serveKindleHead(w http.ResponseWriter, r *http.Request, file *models.MediaFile, key ebookconvert.SourceKey) error {
path, lookupErr, ok := h.Conversion.Converter.Lookup(key)
switch {
case ok:
if serveErr := serveConvertedEpub(w, r, file, path, key); serveErr != nil {
return h.serveRawKindleFallback(w, r, file)
}
return nil
case lookupErr != nil:
return h.serveRawKindleFallback(w, r, file)
default:
setConvertedEpubHeaders(w, file, key)
w.WriteHeader(http.StatusOK)
return nil
}
}
// setConvertedEpubHeaders writes the headers for a converted-EPUB response.
// Because the same URL can return raw or converted bytes depending on the
// flag/outcome, the ETag is derived from the exact conversion cache key (so a
// validator can never disagree with the cache) with a revalidate policy that
// keeps clients/proxies from serving a stale representation.
func setConvertedEpubHeaders(w http.ResponseWriter, file *models.MediaFile, key ebookconvert.SourceKey) {
name := strings.TrimSuffix(filepath.Base(file.FilePath), filepath.Ext(file.FilePath)) + ".epub"
w.Header().Set("Content-Type", "application/epub+zip")
w.Header().Set("Content-Disposition", inlineContentDisposition(name))
w.Header().Set("X-Content-Type-Options", "nosniff")
w.Header().Set(ConversionHeader, "converted")
w.Header().Set("ETag", fmt.Sprintf("%q", "epubconv-"+key.CacheKey()))
w.Header().Set("Cache-Control", "private, max-age=0, must-revalidate")
}
// serveConvertedEpub serves the cached EPUB with EPUB headers (see
// setConvertedEpubHeaders). For a HEAD, ServeContent emits the headers and
// Content-Length without a body.
func serveConvertedEpub(w http.ResponseWriter, r *http.Request, file *models.MediaFile, epubPath string, key ebookconvert.SourceKey) error {
f, err := os.Open(epubPath)
if err != nil {
return fmt.Errorf("opening converted epub: %w", err)
}
defer f.Close()
stat, err := f.Stat()
if err != nil {
return fmt.Errorf("stat converted epub: %w", err)
}
name := strings.TrimSuffix(filepath.Base(file.FilePath), filepath.Ext(file.FilePath)) + ".epub"
setConvertedEpubHeaders(w, file, key)
http.ServeContent(w, r, name, stat.ModTime(), f)
return nil
}
// ebookConversionCapability is the client-facing advertisement for the
// Kindle->EPUB feature. Clients flip mobi/azw/azw3 to in-app reading only when
// Enabled is true; otherwise they keep the external-open path.
type ebookConversionCapability struct {
Enabled bool `json:"enabled"`
SourceFormats []string `json:"source_formats"`
ServedFormat string `json:"served_format"`
Header string `json:"header"`
HeaderOnFailed string `json:"header_failed_value"`
}
// HandleConversionCapability reports whether Kindle->EPUB conversion is active
// (admin flag on AND the converter initialized). GET /api/v1/ebooks/capability.
func (h *EbookReaderHandler) HandleConversionCapability(w http.ResponseWriter, r *http.Request) {
writeJSON(w, http.StatusOK, ebookConversionCapability{
Enabled: h.Conversion.active(r.Context()),
SourceFormats: []string{"mobi", "azw", "azw3"},
ServedFormat: "epub",
Header: ConversionHeader,
HeaderOnFailed: "failed",
})
}
@@ -0,0 +1,271 @@
package handlers
import (
"context"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"github.com/Silo-Server/silo-server/internal/ebookconvert"
"github.com/Silo-Server/silo-server/internal/models"
)
type fakeConverter struct {
epubPath string
err error
calls int
lookupPath string
lookupErr error
lookupOK bool
}
func (f *fakeConverter) GetOrConvert(_ context.Context, _ string, _ ebookconvert.SourceKey) (string, error) {
f.calls++
return f.epubPath, f.err
}
func (f *fakeConverter) Lookup(_ ebookconvert.SourceKey) (string, error, bool) {
return f.lookupPath, f.lookupErr, f.lookupOK
}
func writeTemp(t *testing.T, name, content string) string {
t.Helper()
p := filepath.Join(t.TempDir(), name)
if err := os.WriteFile(p, []byte(content), 0o644); err != nil {
t.Fatal(err)
}
return p
}
func serveAndRecord(t *testing.T, h *EbookReaderHandler, file *models.MediaFile) *httptest.ResponseRecorder {
t.Helper()
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodGet, "/read", nil)
if err := h.serveEbook(rec, req, file); err != nil {
t.Fatalf("serveEbook: %v", err)
}
return rec
}
func enabledConversion(conv EbookConverter, on bool) *EbookConversion {
return &EbookConversion{Converter: conv, Enabled: func(context.Context) bool { return on }}
}
func TestServeEbook_ConvertsKindleWhenEnabled(t *testing.T) {
epub := writeTemp(t, "out.epub", "PK\x03\x04 fake-epub-bytes")
src := writeTemp(t, "book.azw3", "raw-azw3-source")
conv := &fakeConverter{epubPath: epub}
h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)}
file := &models.MediaFile{ID: 1, FilePath: src, Container: "azw3", FileHash: "abc123"}
rec := serveAndRecord(t, h, file)
if got := rec.Header().Get("Content-Type"); got != "application/epub+zip" {
t.Fatalf("Content-Type = %q, want application/epub+zip", got)
}
if got := rec.Header().Get(ConversionHeader); got != "converted" {
t.Fatalf("%s = %q, want converted", ConversionHeader, got)
}
if rec.Header().Get("ETag") == "" {
t.Fatal("expected an ETag on converted response")
}
if rec.Body.String() != "PK\x03\x04 fake-epub-bytes" {
t.Fatalf("body = %q, want the epub bytes", rec.Body.String())
}
if conv.calls != 1 {
t.Fatalf("converter called %d times, want 1", conv.calls)
}
}
func TestServeEbook_ConversionFailureFallsBackToRawWithHeader(t *testing.T) {
raw := writeTemp(t, "book.mobi", "raw-mobi-bytes")
conv := &fakeConverter{err: ebookconvert.ErrDRMProtected}
h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)}
file := &models.MediaFile{ID: 2, FilePath: raw, Container: "mobi"}
rec := serveAndRecord(t, h, file)
if got := rec.Header().Get(ConversionHeader); got != "failed" {
t.Fatalf("%s = %q, want failed", ConversionHeader, got)
}
if got := rec.Header().Get("Content-Type"); got != "application/x-mobipocket-ebook" {
t.Fatalf("Content-Type = %q, want raw mobi mime", got)
}
if rec.Body.String() != "raw-mobi-bytes" {
t.Fatalf("body = %q, want raw mobi bytes", rec.Body.String())
}
}
func TestServeEbook_DisabledServesRaw(t *testing.T) {
raw := writeTemp(t, "book.mobi", "raw-mobi-bytes")
conv := &fakeConverter{epubPath: "should-not-be-used"}
h := &EbookReaderHandler{Conversion: enabledConversion(conv, false)}
file := &models.MediaFile{ID: 3, FilePath: raw, Container: "mobi"}
rec := serveAndRecord(t, h, file)
if got := rec.Header().Get(ConversionHeader); got != "" {
t.Fatalf("%s = %q, want empty (no conversion attempted)", ConversionHeader, got)
}
if got := rec.Header().Get("Content-Type"); got != "application/x-mobipocket-ebook" {
t.Fatalf("Content-Type = %q, want raw mobi mime", got)
}
if conv.calls != 0 {
t.Fatalf("converter called %d times while disabled, want 0", conv.calls)
}
}
func TestServeEbook_NonKindleUntouched(t *testing.T) {
raw := writeTemp(t, "book.epub", "epub-bytes")
conv := &fakeConverter{epubPath: "should-not-be-used"}
h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)}
file := &models.MediaFile{ID: 4, FilePath: raw, Container: "epub"}
rec := serveAndRecord(t, h, file)
if rec.Header().Get(ConversionHeader) != "" {
t.Fatal("non-kindle file must not trigger conversion")
}
if conv.calls != 0 {
t.Fatalf("converter called %d times for epub, want 0", conv.calls)
}
if got := rec.Header().Get("Content-Type"); got != "application/epub+zip" {
t.Fatalf("Content-Type = %q, want epub", got)
}
}
func TestServeEbook_NilConversionServesRaw(t *testing.T) {
raw := writeTemp(t, "book.mobi", "raw")
h := &EbookReaderHandler{} // Conversion nil
file := &models.MediaFile{ID: 5, FilePath: raw, Container: "mobi"}
rec := serveAndRecord(t, h, file)
if rec.Header().Get(ConversionHeader) != "" {
t.Fatal("nil Conversion must not set the conversion header")
}
}
func TestHandleConversionCapability(t *testing.T) {
for _, tc := range []struct {
name string
conv *EbookConversion
want bool
}{
{"enabled", enabledConversion(&fakeConverter{}, true), true},
{"flag-off", enabledConversion(&fakeConverter{}, false), false},
{"not-wired", nil, false},
} {
t.Run(tc.name, func(t *testing.T) {
h := &EbookReaderHandler{Conversion: tc.conv}
rec := httptest.NewRecorder()
h.HandleConversionCapability(rec, httptest.NewRequest(http.MethodGet, "/ebooks/capability", nil))
if rec.Code != http.StatusOK {
t.Fatalf("status = %d", rec.Code)
}
body := rec.Body.String()
wantField := `"enabled":false`
if tc.want {
wantField = `"enabled":true`
}
if !strings.Contains(body, wantField) {
t.Fatalf("body %q missing %s", body, wantField)
}
if !strings.Contains(body, `"served_format":"epub"`) || !strings.Contains(body, "mobi") {
t.Fatalf("capability body incomplete: %s", body)
}
})
}
}
// A HEAD on a cache miss must NOT trigger a (possibly minute-long) conversion;
// it advertises the converted representation cheaply and lets the GET do the work.
func TestServeEbook_HeadDoesNotConvertOnMiss(t *testing.T) {
src := writeTemp(t, "book.azw3", "raw-azw3-source")
conv := &fakeConverter{lookupOK: false} // cache miss
h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)}
file := &models.MediaFile{ID: 1, FilePath: src, Container: "azw3"}
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodHead, "/read", nil)
if err := h.serveEbook(rec, req, file); err != nil {
t.Fatalf("serveEbook HEAD: %v", err)
}
if conv.calls != 0 {
t.Fatalf("HEAD on a cache miss triggered %d conversions, want 0", conv.calls)
}
if got := rec.Header().Get(ConversionHeader); got != "converted" {
t.Fatalf("%s = %q, want converted (advertised on HEAD miss)", ConversionHeader, got)
}
if got := rec.Header().Get("Content-Type"); got != "application/epub+zip" {
t.Fatalf("Content-Type = %q, want application/epub+zip", got)
}
if rec.Header().Get("ETag") == "" {
t.Fatal("HEAD miss should still carry the conversion ETag")
}
}
// A HEAD that hits the cache serves the converted headers from the cached file,
// without converting.
func TestServeEbook_HeadServesCachedConverted(t *testing.T) {
epub := writeTemp(t, "out.epub", "PK\x03\x04 cached-epub")
src := writeTemp(t, "book.mobi", "raw-mobi-source")
conv := &fakeConverter{lookupPath: epub, lookupOK: true}
h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)}
file := &models.MediaFile{ID: 2, FilePath: src, Container: "mobi"}
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodHead, "/read", nil)
if err := h.serveEbook(rec, req, file); err != nil {
t.Fatalf("serveEbook HEAD: %v", err)
}
if conv.calls != 0 {
t.Fatalf("HEAD on a cache hit triggered %d conversions, want 0", conv.calls)
}
if got := rec.Header().Get(ConversionHeader); got != "converted" {
t.Fatalf("%s = %q, want converted", ConversionHeader, got)
}
if got := rec.Header().Get("Content-Type"); got != "application/epub+zip" {
t.Fatalf("Content-Type = %q, want application/epub+zip", got)
}
}
// A HEAD on a negatively-cached (DRM/failed) source reports the failed contract
// without converting.
func TestServeEbook_HeadDRMReturnsFailed(t *testing.T) {
src := writeTemp(t, "book.mobi", "raw-mobi-source")
conv := &fakeConverter{lookupErr: ebookconvert.ErrDRMProtected}
h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)}
file := &models.MediaFile{ID: 3, FilePath: src, Container: "mobi"}
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodHead, "/read", nil)
if err := h.serveEbook(rec, req, file); err != nil {
t.Fatalf("serveEbook HEAD: %v", err)
}
if conv.calls != 0 {
t.Fatalf("HEAD on a negatively-cached source triggered %d conversions, want 0", conv.calls)
}
if got := rec.Header().Get(ConversionHeader); got != "failed" {
t.Fatalf("%s = %q, want failed", ConversionHeader, got)
}
}
func TestServeEbook_ContextCancelPropagates(t *testing.T) {
src := writeTemp(t, "book.mobi", "raw-mobi-source")
conv := &fakeConverter{err: context.Canceled}
h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)}
file := &models.MediaFile{ID: 6, FilePath: src, Container: "mobi"}
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodGet, "/read", nil)
err := h.serveEbook(rec, req, file)
if err == nil {
t.Fatal("expected context cancellation to propagate, not fall back")
}
if rec.Header().Get(ConversionHeader) == "failed" {
t.Fatal("cancellation must not be reported as a conversion failure")
}
}
+17 -4
View File
@@ -96,6 +96,9 @@ type EbookReaderHandler struct {
ProgressStore EbookReaderProgressStore
ConfigStore EbookReaderConfigStore
AnnotationStore EbookReaderAnnotationStore
// Conversion, when set and enabled, transparently converts Kindle-family
// files to EPUB on read. Nil means the feature is off.
Conversion *EbookConversion
}
func NewEbookReaderHandler(authorizer *MediaFileAuthorizer) *EbookReaderHandler {
@@ -130,7 +133,7 @@ func (h *EbookReaderHandler) HandleReadFile(w http.ResponseWriter, r *http.Reque
return
}
if err := serveEbookInline(w, r, file); err != nil {
if err := h.serveEbook(w, r, file); err != nil {
if errors.Is(err, catalog.ErrItemNotFound) {
writeError(w, http.StatusNotFound, "not_found", "Ebook file not found")
return
@@ -1046,15 +1049,25 @@ func (s *PGEbookReaderProgressStore) Upsert(ctx context.Context, progress EbookR
if s == nil || s.pool == nil {
return fmt.Errorf("ebook reader progress store is not configured")
}
if _, err := s.pool.Exec(ctx, `
// A routine autosave (e.g. reopening a finished book) must not silently drop
// a "finished" item below the threshold and un-mark it read; once finished,
// progress only moves on an explicit unread (which deletes the row). Below
// the threshold, progress tracks freely. Manga chapter ✓ marks ride on this
// same row, so the guard protects them too.
query := fmt.Sprintf(`
INSERT INTO ebook_reader_progress
(user_id, profile_id, content_id, file_id, location, progress, updated_at)
VALUES ($1, $2, $3, $4, $5, $6, $7)
ON CONFLICT (user_id, profile_id, content_id) DO UPDATE SET
file_id = EXCLUDED.file_id,
location = EXCLUDED.location,
progress = EXCLUDED.progress,
updated_at = EXCLUDED.updated_at`,
progress = CASE
WHEN ebook_reader_progress.progress >= %[1]v AND EXCLUDED.progress < %[1]v
THEN ebook_reader_progress.progress
ELSE EXCLUDED.progress
END,
updated_at = EXCLUDED.updated_at`, models.EbookFinishedProgressThreshold)
if _, err := s.pool.Exec(ctx, query,
progress.UserID,
progress.ProfileID,
progress.ContentID,
+13 -6
View File
@@ -28,8 +28,10 @@ type taskInfoLister interface {
ListTasks(includeHidden bool) []taskmanager.TaskInfo
}
const maxRealtimeScanSnapshotRuns = 500
type activeScanLister interface {
ListActive(ctx context.Context) ([]evt.ScanRun, error)
ListActiveSnapshot(ctx context.Context, limit int) ([]evt.ScanRun, error)
}
type EventsHandler struct {
@@ -414,16 +416,21 @@ func (h *EventsHandler) snapshotForChannel(
if h == nil {
return json.RawMessage("[]"), nil
}
runs := make([]evt.ScanRun, 0)
if h.persistedScans != nil {
persisted, err := h.persistedScans.ListActive(r.Context())
runs := make([]evt.ScanRun, 0, maxRealtimeScanSnapshotRuns)
remaining := maxRealtimeScanSnapshotRuns
if h.persistedScans != nil && remaining > 0 {
persisted, err := h.persistedScans.ListActiveSnapshot(r.Context(), remaining)
if err != nil {
return nil, err
}
if len(persisted) > remaining {
persisted = persisted[:remaining]
}
runs = append(runs, persisted...)
remaining -= len(persisted)
}
if h.scans != nil {
runs = append(runs, h.scans.ListActive()...)
if h.scans != nil && remaining > 0 {
runs = append(runs, h.scans.ListActiveLimit(remaining)...)
}
return marshalJSON(runs), nil
case evt.ChannelHistoryImport:
+79
View File
@@ -0,0 +1,79 @@
package handlers
import (
"context"
"encoding/json"
"net/http/httptest"
"strconv"
"strings"
"testing"
evt "github.com/Silo-Server/silo-server/internal/events"
)
type fakeActiveScanSnapshotLister struct {
runs []evt.ScanRun
limit int
}
func (f *fakeActiveScanSnapshotLister) ListActiveSnapshot(_ context.Context, limit int) ([]evt.ScanRun, error) {
f.limit = limit
if limit > 0 && len(f.runs) > limit {
return f.runs[:limit], nil
}
return f.runs, nil
}
func TestScanSnapshotCapsActiveRuns(t *testing.T) {
persisted := &fakeActiveScanSnapshotLister{runs: makeScanRuns("persisted-", 300)}
registry := evt.NewScanRegistry()
for _, run := range makeScanRuns("registry-", 250) {
registry.Upsert(run)
}
registry.Upsert(evt.ScanRun{ID: "registry-completed", Status: "completed"})
handler := &EventsHandler{
persistedScans: persisted,
scans: registry,
}
raw, err := handler.snapshotForChannel(httptest.NewRequest("GET", "/", nil), nil, "", evt.ChannelScans)
if err != nil {
t.Fatalf("snapshotForChannel: %v", err)
}
var runs []evt.ScanRun
if err := json.Unmarshal(raw, &runs); err != nil {
t.Fatalf("decode snapshot: %v", err)
}
if len(runs) != maxRealtimeScanSnapshotRuns {
t.Fatalf("snapshot length = %d, want %d", len(runs), maxRealtimeScanSnapshotRuns)
}
if persisted.limit != maxRealtimeScanSnapshotRuns {
t.Fatalf("persisted snapshot limit = %d, want %d", persisted.limit, maxRealtimeScanSnapshotRuns)
}
registryRuns := 0
for _, run := range runs {
if strings.HasPrefix(run.ID, "registry-") {
registryRuns++
}
}
if registryRuns != maxRealtimeScanSnapshotRuns-len(persisted.runs) {
t.Fatalf("registry runs in snapshot = %d, want %d", registryRuns, maxRealtimeScanSnapshotRuns-len(persisted.runs))
}
}
func makeScanRuns(prefix string, count int) []evt.ScanRun {
runs := make([]evt.ScanRun, 0, count)
for i := 0; i < count; i++ {
runs = append(runs, evt.ScanRun{
ID: prefix + strconv.Itoa(i),
LibraryID: 1,
Mode: "file",
Path: "/library/file-" + strconv.Itoa(i) + ".mkv",
Trigger: "autoscan",
Status: "accepted",
})
}
return runs
}
+16 -13
View File
@@ -25,8 +25,8 @@ type personalDataItemRepository interface {
EnsureAccessible(ctx context.Context, contentID string, filter catalog.AccessFilter) error
}
type LocalFavoriteEventDispatcher interface {
HandleLocalFavoriteEvent(ctx context.Context, event watchsync.LocalFavoriteEvent) error
type LocalListEventDispatcher interface {
HandleLocalListEvent(ctx context.Context, event watchsync.LocalListEvent) error
}
// PersonalDataHandler handles favorites, watchlist, and history endpoints.
@@ -37,7 +37,7 @@ type PersonalDataHandler struct {
seasonRepo *catalog.SeasonRepository
detailSvc *catalog.DetailService
EventsHub *evt.Hub
localFavoriteDispatcher LocalFavoriteEventDispatcher
localListDispatcher LocalListEventDispatcher
profileStaler ProfileStaler
profileRefreshRequester ProfileRefreshRequester
ebookProgressStore EbookReaderProgressLister
@@ -78,8 +78,8 @@ func (h *PersonalDataHandler) SetSeasonRepo(repo *catalog.SeasonRepository) {
h.seasonRepo = repo
}
func (h *PersonalDataHandler) SetLocalFavoriteEventDispatcher(dispatcher LocalFavoriteEventDispatcher) {
h.localFavoriteDispatcher = dispatcher
func (h *PersonalDataHandler) SetLocalListEventDispatcher(dispatcher LocalListEventDispatcher) {
h.localListDispatcher = dispatcher
}
// --- Response types ---
@@ -219,7 +219,7 @@ func (h *PersonalDataHandler) HandleAddFavorite(w http.ResponseWriter, r *http.R
return
}
h.dispatchLocalFavoriteEvent(r.Context(), watchsync.LocalFavoriteEventAdded, userID, profileID, itemID)
h.dispatchLocalListEvent(r.Context(), watchsync.ListKindFavorites, watchsync.ListChangeAdded, userID, profileID, itemID)
triggerProfileRefresh(r.Context(), h.profileStaler, h.profileRefreshRequester, userID, profileID)
publishUserStateEvent(r.Context(), h.EventsHub, userID, profileID, itemID, "", "favorite", userStateEventState{
IsFavorite: boolPtr(true),
@@ -249,7 +249,7 @@ func (h *PersonalDataHandler) HandleRemoveFavorite(w http.ResponseWriter, r *htt
return
}
h.dispatchLocalFavoriteEvent(r.Context(), watchsync.LocalFavoriteEventRemoved, userID, profileID, itemID)
h.dispatchLocalListEvent(r.Context(), watchsync.ListKindFavorites, watchsync.ListChangeRemoved, userID, profileID, itemID)
triggerProfileRefresh(r.Context(), h.profileStaler, h.profileRefreshRequester, userID, profileID)
publishUserStateEvent(r.Context(), h.EventsHub, userID, profileID, itemID, "", "favorite", userStateEventState{
IsFavorite: boolPtr(false),
@@ -257,15 +257,15 @@ func (h *PersonalDataHandler) HandleRemoveFavorite(w http.ResponseWriter, r *htt
w.WriteHeader(http.StatusNoContent)
}
func (h *PersonalDataHandler) dispatchLocalFavoriteEvent(ctx context.Context, kind watchsync.LocalFavoriteEventKind, userID int, profileID string, itemID string) {
if h == nil || h.localFavoriteDispatcher == nil || h.itemRepo == nil {
func (h *PersonalDataHandler) dispatchLocalListEvent(ctx context.Context, list watchsync.ListKind, change watchsync.ListChange, userID int, profileID string, itemID string) {
if h == nil || h.localListDispatcher == nil || h.itemRepo == nil {
return
}
item, err := h.itemRepo.GetByID(ctx, itemID)
if err != nil || item == nil {
return
}
favorite := watchsync.LocalFavorite{
listItem := watchsync.LocalFavorite{
MediaItemID: item.ContentID,
Kind: item.Type,
Title: item.Title,
@@ -275,11 +275,12 @@ func (h *PersonalDataHandler) dispatchLocalFavoriteEvent(ctx context.Context, ki
TVDBID: item.TvdbID,
FavoritedAt: time.Now().UTC(),
}
_ = h.localFavoriteDispatcher.HandleLocalFavoriteEvent(ctx, watchsync.LocalFavoriteEvent{
Kind: kind,
_ = h.localListDispatcher.HandleLocalListEvent(ctx, watchsync.LocalListEvent{
List: list,
Change: change,
UserID: userID,
ProfileID: profileID,
Favorites: []watchsync.LocalFavorite{favorite},
Items: []watchsync.LocalFavorite{listItem},
})
}
@@ -374,6 +375,7 @@ func (h *PersonalDataHandler) HandleAddToWatchlist(w http.ResponseWriter, r *htt
return
}
h.dispatchLocalListEvent(r.Context(), watchsync.ListKindWatchlist, watchsync.ListChangeAdded, userID, profileID, itemID)
triggerProfileRefresh(r.Context(), h.profileStaler, h.profileRefreshRequester, userID, profileID)
publishUserStateEvent(r.Context(), h.EventsHub, userID, profileID, itemID, "", "watchlist", userStateEventState{
InWatchlist: boolPtr(true),
@@ -403,6 +405,7 @@ func (h *PersonalDataHandler) HandleRemoveFromWatchlist(w http.ResponseWriter, r
return
}
h.dispatchLocalListEvent(r.Context(), watchsync.ListKindWatchlist, watchsync.ListChangeRemoved, userID, profileID, itemID)
triggerProfileRefresh(r.Context(), h.profileStaler, h.profileRefreshRequester, userID, profileID)
publishUserStateEvent(r.Context(), h.EventsHub, userID, profileID, itemID, "", "watchlist", userStateEventState{
InWatchlist: boolPtr(false),
+281 -85
View File
@@ -38,6 +38,11 @@ type batchEpisodeFileProvider interface {
ListByEpisodeIDs(ctx context.Context, episodeIDs []string) (map[string][]*models.MediaFile, error)
}
type overlayFileProvider interface {
ListOverlayFilesByContentIDs(ctx context.Context, contentIDs []string) (map[string][]*models.MediaFile, error)
ListOverlayFilesByEpisodeIDs(ctx context.Context, episodeIDs []string) (map[string][]*models.MediaFile, error)
}
type MetadataRefreshRequester interface {
RequestStaleMetadataRefresh(ctx context.Context, targetType, contentID string) error
}
@@ -102,6 +107,15 @@ func NewItemsHandler(
}
}
// SetCompletionObserver wires an optional observer notified when a watch
// completes (used to auto-remove fully-watched items from the watchlist).
func (h *ItemsHandler) SetCompletionObserver(obs watchstate.CompletionObserver) {
if h == nil || h.watchState == nil || obs == nil {
return
}
h.watchState.WithCompletionObserver(obs)
}
// SetProfileStaler configures an optional staleness trigger for taste profiles.
func (h *ItemsHandler) SetProfileStaler(ps ProfileStaler) {
h.profileStaler = ps
@@ -116,6 +130,13 @@ func (h *ItemsHandler) SetMetadataRefreshRequester(requester MetadataRefreshRequ
h.metadataRefreshRequester = requester
}
func (h *ItemsHandler) SetCatalogSearchProvider(provider catalog.CatalogSearchProvider) {
if h == nil || h.catalogResolver == nil || provider == nil {
return
}
h.catalogResolver.WithSearchProvider(provider)
}
func (h *ItemsHandler) SetLocalWatchEventDispatcher(dispatcher LocalWatchEventDispatcher) {
h.localWatchDispatcher = dispatcher
}
@@ -195,37 +216,42 @@ func (h *ItemsHandler) requestStaleMetadataRefresh(ctx context.Context, targetTy
// itemListResponse is the shape of a single item in browse/search list responses.
type itemListResponse struct {
ContentID string `json:"content_id"`
Type string `json:"type"`
Title string `json:"title"`
SeriesTitle string `json:"series_title,omitempty"`
SeasonNumber *int `json:"season_number,omitempty"`
EpisodeNumber *int `json:"episode_number,omitempty"`
Year int `json:"year,omitempty"`
Runtime int `json:"runtime,omitempty"`
Genres []string `json:"genres"`
Keywords []string `json:"keywords"`
Studios []string `json:"studios,omitempty"`
Networks []string `json:"networks,omitempty"`
ContentRating string `json:"content_rating,omitempty"`
Status string `json:"status"`
ShowStatus string `json:"show_status,omitempty"`
RatingIMDB *float64 `json:"rating_imdb,omitempty"`
RatingTMDB *float64 `json:"rating_tmdb,omitempty"`
RatingRTCritic *int `json:"rating_rt_critic,omitempty"`
RatingRTAudience *int `json:"rating_rt_audience,omitempty"`
OriginalLanguage string `json:"original_language,omitempty"`
Overview string `json:"overview,omitempty"`
PosterURL string `json:"poster_url,omitempty"`
PosterThumbhash string `json:"poster_thumbhash,omitempty"`
BackdropURL string `json:"backdrop_url,omitempty"`
BackdropThumbhash string `json:"backdrop_thumbhash,omitempty"`
ReleaseDate *string `json:"release_date,omitempty"`
LastAirDate *string `json:"last_air_date,omitempty"`
AddedAt *time.Time `json:"added_at,omitempty"`
OverlaySummary *models.OverlaySummary `json:"overlay_summary,omitempty"`
SortMetrics *sortMetricsResponse `json:"sort_metrics,omitempty"`
UserState *itemUserStateResponse `json:"user_state,omitempty"`
ContentID string `json:"content_id"`
Type string `json:"type"`
Title string `json:"title"`
SeriesTitle string `json:"series_title,omitempty"`
SeasonNumber *int `json:"season_number,omitempty"`
EpisodeNumber *int `json:"episode_number,omitempty"`
Year int `json:"year,omitempty"`
Runtime int `json:"runtime,omitempty"`
Genres []string `json:"genres"`
Keywords []string `json:"keywords"`
Studios []string `json:"studios,omitempty"`
Networks []string `json:"networks,omitempty"`
ContentRating string `json:"content_rating,omitempty"`
Status string `json:"status"`
ShowStatus string `json:"show_status,omitempty"`
RatingIMDB *float64 `json:"rating_imdb,omitempty"`
RatingTMDB *float64 `json:"rating_tmdb,omitempty"`
RatingRTCritic *int `json:"rating_rt_critic,omitempty"`
RatingRTAudience *int `json:"rating_rt_audience,omitempty"`
OriginalLanguage string `json:"original_language,omitempty"`
Overview string `json:"overview,omitempty"`
PosterURL string `json:"poster_url,omitempty"`
PosterThumbhash string `json:"poster_thumbhash,omitempty"`
BackdropURL string `json:"backdrop_url,omitempty"`
BackdropThumbhash string `json:"backdrop_thumbhash,omitempty"`
ReleaseDate *string `json:"release_date,omitempty"`
LastAirDate *string `json:"last_air_date,omitempty"`
AddedAt *time.Time `json:"added_at,omitempty"`
MangaChapterCount *int `json:"manga_chapter_count,omitempty"`
MangaVolumeCount *int `json:"manga_volume_count,omitempty"`
OverlaySummary *models.OverlaySummary `json:"overlay_summary,omitempty"`
SortMetrics *sortMetricsResponse `json:"sort_metrics,omitempty"`
UserState *itemUserStateResponse `json:"user_state,omitempty"`
WorkID string `json:"work_id,omitempty"`
WorkTitle string `json:"work_title,omitempty"`
WorkFormats []catalog.WorkFormatSummary `json:"work_formats,omitempty"`
}
type sortMetricsResponse struct {
@@ -241,11 +267,17 @@ type sortMetricsResponse struct {
SeriesName string `json:"series_name,omitempty"`
}
type itemListImageURLs struct {
posterURL string
backdropURL string
}
// browseResponse is the paginated response for the /items endpoint.
type browseResponse struct {
Total int `json:"total"`
HasMore bool `json:"has_more"`
Items []itemListResponse `json:"items"`
Total int `json:"total"`
TotalExact bool `json:"total_exact"`
HasMore bool `json:"has_more"`
Items []itemListResponse `json:"items"`
}
type itemFiltersResponse struct {
@@ -387,7 +419,7 @@ func (h *ItemsHandler) HandleGetWatchDetail(w http.ResponseWriter, r *http.Reque
}
}
if detail.Type == "movie" || detail.Type == "episode" || detail.Type == "ebook" {
if detail.Type == "movie" || detail.Type == "episode" || detail.Type == "ebook" || detail.Type == "audiobook" {
detail.UserData = h.getLeafUserData(r, detail.ContentID, detail.Type)
applyEffectiveEditionPreference(detail.UserData, &detail.EffectiveVersionEditionKey)
}
@@ -578,9 +610,10 @@ func (h *ItemsHandler) writeCatalogBrowseResponse(w http.ResponseWriter, r *http
}
writeJSON(w, http.StatusOK, browseResponse{
Total: result.Total,
HasMore: result.HasMore,
Items: items,
Total: result.Total,
TotalExact: result.TotalExact,
HasMore: result.HasMore,
Items: items,
})
return true
}
@@ -628,6 +661,13 @@ func (h *ItemsHandler) toItemListResponseWithOverlay(r *http.Request, item *mode
item = localized
}
}
resp := itemListResponseShell(item, overlaySummary, userState)
resp.PosterURL = h.presignURL(r, cardThumbnailPath(item.PosterPath), "card")
resp.BackdropURL = h.presignURL(r, cardThumbnailPath(item.BackdropPath), "card")
return resp
}
func itemListResponseShell(item *models.MediaItem, overlaySummary *models.OverlaySummary, userState *itemUserStateResponse) itemListResponse {
resp := itemListResponse{
ContentID: item.ContentID,
Type: item.Type,
@@ -654,10 +694,10 @@ func (h *ItemsHandler) toItemListResponseWithOverlay(r *http.Request, item *mode
}
resp.AddedAt = item.AddedAt
resp.MangaChapterCount = item.MangaChapterCount
resp.MangaVolumeCount = item.MangaVolumeCount
resp.ReleaseDate = item.ReleaseDate
resp.LastAirDate = item.LastAirDate
resp.PosterURL = h.presignURL(r, cardThumbnailPath(item.PosterPath), "card")
resp.BackdropURL = h.presignURL(r, cardThumbnailPath(item.BackdropPath), "card")
if resp.Genres == nil {
resp.Genres = []string{}
}
@@ -668,6 +708,67 @@ func (h *ItemsHandler) toItemListResponseWithOverlay(r *http.Request, item *mode
return resp
}
func (h *ItemsHandler) localizeItemListModels(ctx context.Context, items []*models.MediaItem, filter catalog.AccessFilter) []*models.MediaItem {
if h == nil || h.detailSvc == nil || len(items) == 0 {
return items
}
localized, err := h.detailSvc.LocalizeItemModels(ctx, items, filter)
if err != nil || len(localized) != len(items) {
return items
}
return localized
}
func (h *ItemsHandler) itemListCardImageURLs(ctx context.Context, items []*models.MediaItem) map[string]itemListImageURLs {
urls := make(map[string]itemListImageURLs, len(items))
if h == nil || h.detailSvc == nil || len(items) == 0 {
return urls
}
type pendingImages struct {
contentID string
posterPath string
backdropPath string
}
pending := make([]pendingImages, 0, len(items))
paths := make([]string, 0, len(items)*2)
seenPaths := make(map[string]struct{}, len(items)*2)
addPath := func(path string) {
if path == "" || path == "-" {
return
}
if _, ok := seenPaths[path]; ok {
return
}
seenPaths[path] = struct{}{}
paths = append(paths, path)
}
for _, item := range items {
if item == nil || item.ContentID == "" {
continue
}
images := pendingImages{
contentID: item.ContentID,
posterPath: cardThumbnailPath(item.PosterPath),
backdropPath: cardThumbnailPath(item.BackdropPath),
}
pending = append(pending, images)
addPath(images.posterPath)
addPath(images.backdropPath)
}
resolved := h.detailSvc.PresignURLsWithExpiry(ctx, paths, "card")
for _, images := range pending {
urls[images.contentID] = itemListImageURLs{
posterURL: resolved[images.posterPath].URL,
backdropURL: resolved[images.backdropPath].URL,
}
}
return urls
}
func (h *ItemsHandler) listEpisodeBrowseMetadata(
ctx context.Context,
items []*models.MediaItem,
@@ -954,7 +1055,7 @@ func (h *ItemsHandler) listOverlaySummaries(ctx context.Context, items []*models
return summaries
}
groupedFiles := h.listBrowseItemFiles(ctx, items, filter)
groupedFiles := h.listBrowseItemOverlayFiles(ctx, items, filter)
for contentID, files := range groupedFiles {
if summary := overlays.BuildSummary(files); summary != nil {
summaries[contentID] = summary
@@ -1256,21 +1357,7 @@ func (h *ItemsHandler) listBrowseItemFiles(ctx context.Context, items []*models.
contentIDs := make([]string, 0, len(items))
episodeIDs := make([]string, 0)
seen := make(map[string]struct{}, len(items))
for _, item := range items {
if item == nil || item.ContentID == "" {
continue
}
if _, ok := seen[item.ContentID]; ok {
continue
}
seen[item.ContentID] = struct{}{}
if item.Type == "episode" {
episodeIDs = append(episodeIDs, item.ContentID)
} else {
contentIDs = append(contentIDs, item.ContentID)
}
}
collectBrowseFileIDs(items, &contentIDs, &episodeIDs)
if len(contentIDs) > 0 {
contentFiles, err := h.fileRepo.ListByContentIDs(ctx, contentIDs)
@@ -1295,6 +1382,60 @@ func (h *ItemsHandler) listBrowseItemFiles(ctx context.Context, items []*models.
return grouped
}
func (h *ItemsHandler) listBrowseItemOverlayFiles(ctx context.Context, items []*models.MediaItem, filter catalog.AccessFilter) map[string][]*models.MediaFile {
if h.fileRepo == nil || len(items) == 0 {
return map[string][]*models.MediaFile{}
}
overlayProvider, ok := h.fileRepo.(overlayFileProvider)
if !ok {
return h.listBrowseItemFiles(ctx, items, filter)
}
contentIDs := make([]string, 0, len(items))
episodeIDs := make([]string, 0)
collectBrowseFileIDs(items, &contentIDs, &episodeIDs)
grouped := make(map[string][]*models.MediaFile, len(items))
if len(contentIDs) > 0 {
contentFiles, err := overlayProvider.ListOverlayFilesByContentIDs(ctx, contentIDs)
if err != nil {
return h.listBrowseItemFiles(ctx, items, filter)
}
for contentID, files := range contentFiles {
grouped[contentID] = catalog.FilterMediaFilesByAccess(files, filter)
}
}
if len(episodeIDs) > 0 {
episodeFiles, err := overlayProvider.ListOverlayFilesByEpisodeIDs(ctx, episodeIDs)
if err != nil {
return h.listBrowseItemFiles(ctx, items, filter)
}
for episodeID, files := range episodeFiles {
grouped[episodeID] = catalog.FilterMediaFilesByAccess(files, filter)
}
}
return grouped
}
func collectBrowseFileIDs(items []*models.MediaItem, contentIDs *[]string, episodeIDs *[]string) {
seen := make(map[string]struct{}, len(items))
for _, item := range items {
if item == nil || item.ContentID == "" {
continue
}
if _, ok := seen[item.ContentID]; ok {
continue
}
seen[item.ContentID] = struct{}{}
if item.Type == "episode" {
*episodeIDs = append(*episodeIDs, item.ContentID)
} else {
*contentIDs = append(*contentIDs, item.ContentID)
}
}
}
func firstNonBlankPtr(values ...*string) *string {
for _, value := range values {
if value == nil || strings.TrimSpace(*value) == "" {
@@ -1318,12 +1459,22 @@ func maxFileBitrate(files []*models.MediaFile) int {
// toSeasonResponse converts a Season model to an API response.
func (h *ItemsHandler) toSeasonResponse(r *http.Request, seriesID string, s *models.Season) seasonResponse {
episodes, _ := h.episodeRepo.ListBySeason(r.Context(), seriesID, s.SeasonNumber)
return h.toSeasonResponseFromEpisodes(r, seriesID, s, episodes, h.getAggregateUserData(r, episodes))
}
func (h *ItemsHandler) toSeasonResponseFromEpisodes(
r *http.Request,
seriesID string,
s *models.Season,
episodes []*models.Episode,
userData *catalog.SeasonUserData,
) seasonResponse {
if h.detailSvc != nil {
if localized, err := h.detailSvc.LocalizeSeasonModel(r.Context(), s, h.accessFilter(r)); err == nil && localized != nil {
s = localized
}
}
episodes, _ := h.episodeRepo.ListBySeason(r.Context(), seriesID, s.SeasonNumber)
resp := seasonResponse{
ContentID: s.ContentID,
@@ -1338,7 +1489,7 @@ func (h *ItemsHandler) toSeasonResponse(r *http.Request, seriesID string, s *mod
resp.AirDate = s.AirDate.Format("2006-01-02")
}
resp.PosterURL = h.presignURL(r, featuredPosterPath(s.PosterPath), "featured")
resp.UserData = h.getAggregateUserData(r, episodes)
resp.UserData = userData
return resp
}
@@ -1353,12 +1504,11 @@ func (h *ItemsHandler) getLeafUserData(r *http.Request, contentID string, itemTy
return nil
}
progress, err := store.GetProgress(r.Context(), profileID, contentID)
if err != nil || progress == nil {
progress, err := userstore.GetProgressWithCompletedHistory(r.Context(), store, profileID, contentID)
if err != nil {
return nil
}
return leafUserDataFromProgress(*progress)
return leafUserDataFromProgress(progress)
}
// listLeafUserData batch-fetches watch progress for the given content IDs in a
@@ -1369,19 +1519,23 @@ func (h *ItemsHandler) listLeafUserData(r *http.Request, contentIDs []string) ma
return nil
}
progressMap, err := store.ListProgressByMediaItems(r.Context(), profileID, contentIDs)
progressMap, err := userstore.ListProgressWithCompletedHistory(r.Context(), store, profileID, contentIDs)
if err != nil {
return nil
}
result := make(map[string]*catalog.SeasonUserData, len(progressMap))
for contentID, progress := range progressMap {
result[contentID] = leafUserDataFromProgress(progress)
progressCopy := progress
result[contentID] = leafUserDataFromProgress(&progressCopy)
}
return result
}
func leafUserDataFromProgress(progress userstore.WatchProgress) *catalog.SeasonUserData {
func leafUserDataFromProgress(progress *userstore.WatchProgress) *catalog.SeasonUserData {
if progress == nil {
return nil
}
return &catalog.SeasonUserData{
PositionSeconds: progress.PositionSeconds,
DurationSeconds: progress.DurationSeconds,
@@ -1446,29 +1600,71 @@ func (h *ItemsHandler) getAggregateUserData(r *http.Request, episodes []*models.
return nil
}
var watchedCount int
var inProgressCount int
for _, ep := range episodes {
progress, progressErr := store.GetProgress(r.Context(), profileID, ep.ContentID)
if progressErr != nil || progress == nil {
continue
}
if progress.Completed {
watchedCount++
continue
}
if progress.PositionSeconds > 0 {
inProgressCount++
}
progressMap, err := h.listProgressForEpisodeIDs(r.Context(), store, profileID, episodeContentIDs(episodes))
if err != nil {
return nil
}
return catalog.EpisodeRollupUserData(episodes, progressMap)
}
unplayedCount := len(episodes) - watchedCount
return &catalog.SeasonUserData{
WatchedCount: watchedCount,
UnplayedCount: unplayedCount,
InProgressCount: inProgressCount,
Played: watchedCount == len(episodes),
func (h *ItemsHandler) progressMapForEpisodes(r *http.Request, episodes []*models.Episode) (map[string]userstore.WatchProgress, bool) {
store, profileID, ok := h.userStoreForRequest(r)
if !ok {
return nil, false
}
episodeIDs := episodeContentIDs(episodes)
progressMap, err := h.listProgressForEpisodeIDs(r.Context(), store, profileID, episodeIDs)
if err != nil {
return nil, false
}
return progressMap, true
}
func (h *ItemsHandler) listProgressForEpisodeIDs(ctx context.Context, store userstore.UserStore, profileID string, episodeIDs []string) (map[string]userstore.WatchProgress, error) {
const chunkSize = 500
progressMap := make(map[string]userstore.WatchProgress, len(episodeIDs))
for start := 0; start < len(episodeIDs); start += chunkSize {
end := start + chunkSize
if end > len(episodeIDs) {
end = len(episodeIDs)
}
chunk, err := userstore.ListProgressWithCompletedHistory(ctx, store, profileID, episodeIDs[start:end])
if err != nil {
return nil, err
}
for id, progress := range chunk {
progressMap[id] = progress
}
}
return progressMap, nil
}
func episodeContentIDs(episodes []*models.Episode) []string {
ids := make([]string, 0, len(episodes))
seen := make(map[string]struct{}, len(episodes))
for _, ep := range episodes {
if ep == nil || ep.ContentID == "" {
continue
}
if _, ok := seen[ep.ContentID]; ok {
continue
}
seen[ep.ContentID] = struct{}{}
ids = append(ids, ep.ContentID)
}
return ids
}
func flattenEpisodeGroups(groups map[int][]*models.Episode) []*models.Episode {
var total int
for _, episodes := range groups {
total += len(episodes)
}
flattened := make([]*models.Episode, 0, total)
for _, episodes := range groups {
flattened = append(flattened, episodes...)
}
return flattened
}
// requestProfileID resolves the active profile for a request, falling back to
@@ -0,0 +1,185 @@
package handlers
import (
"context"
"encoding/json"
"strings"
"testing"
"github.com/Silo-Server/silo-server/internal/catalog"
"github.com/Silo-Server/silo-server/internal/models"
)
func TestBrowseResponseIncludesTotalExact(t *testing.T) {
data, err := json.Marshal(browseResponse{
Total: 3,
TotalExact: true,
HasMore: false,
Items: []itemListResponse{},
})
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(data), `"total_exact":true`) {
t.Fatalf("browse response missing total_exact: %s", data)
}
}
type countingItemListImageResolver struct {
singleCalls int
batchCalls int
batchPaths []string
}
func (r *countingItemListImageResolver) ResolveImageURL(_ context.Context, path string, variant string) string {
r.singleCalls++
return "single:" + variant + ":" + path
}
func (r *countingItemListImageResolver) ResolveImageURLs(_ context.Context, paths []string, variant string) map[string]string {
resolved := r.ResolveImageURLsWithExpiry(context.Background(), paths, variant)
out := make(map[string]string, len(resolved))
for path, value := range resolved {
out[path] = value.URL
}
return out
}
func (r *countingItemListImageResolver) ResolveImageURLWithExpiry(_ context.Context, path string, variant string) catalog.ResolvedImageURL {
r.singleCalls++
return catalog.ResolvedImageURL{URL: "single:" + variant + ":" + path}
}
func (r *countingItemListImageResolver) ResolveImageURLsWithExpiry(_ context.Context, paths []string, variant string) map[string]catalog.ResolvedImageURL {
r.batchCalls++
r.batchPaths = append(r.batchPaths[:0], paths...)
out := make(map[string]catalog.ResolvedImageURL, len(paths))
for _, path := range paths {
out[path] = catalog.ResolvedImageURL{URL: "batch:" + variant + ":" + path}
}
return out
}
func TestItemListCardImageURLsUsesBatchResolver(t *testing.T) {
resolver := &countingItemListImageResolver{}
detailSvc := &catalog.DetailService{}
detailSvc.SetImageResolver(resolver)
handler := &ItemsHandler{detailSvc: detailSvc}
items := []*models.MediaItem{
{
ContentID: "movie-1",
PosterPath: "plugin://poster-1/original.jpg",
BackdropPath: "plugin://backdrop-1/original.jpg",
},
{
ContentID: "movie-2",
PosterPath: "https://cdn.example/poster-2.jpg",
BackdropPath: "plugin://backdrop-1/original.jpg",
},
}
urls := handler.itemListCardImageURLs(context.Background(), items)
if resolver.singleCalls != 0 {
t.Fatalf("single resolver calls = %d, want 0", resolver.singleCalls)
}
if resolver.batchCalls != 1 {
t.Fatalf("batch resolver calls = %d, want 1", resolver.batchCalls)
}
if got := urls["movie-1"].posterURL; got != "batch:card:plugin://poster-1/original.jpg" {
t.Fatalf("movie-1 poster URL = %q", got)
}
if got := urls["movie-1"].backdropURL; got != "batch:card:plugin://backdrop-1/original.jpg" {
t.Fatalf("movie-1 backdrop URL = %q", got)
}
if got := urls["movie-2"].posterURL; got != "https://cdn.example/poster-2.jpg" {
t.Fatalf("movie-2 poster URL = %q", got)
}
if got := len(resolver.batchPaths); got != 2 {
t.Fatalf("batch resolver path count = %d, want 2", got)
}
}
type overlayFastPathFileRepo struct {
fullContentCalls int
overlayContentCalls int
overlayEpisodeCalls int
}
func (r *overlayFastPathFileRepo) GetByContentID(context.Context, string) ([]*models.MediaFile, error) {
return nil, nil
}
func (r *overlayFastPathFileRepo) GetByEpisodeID(context.Context, string) ([]*models.MediaFile, error) {
return nil, nil
}
func (r *overlayFastPathFileRepo) ListByContentIDs(context.Context, []string) (map[string][]*models.MediaFile, error) {
r.fullContentCalls++
return nil, nil
}
func (r *overlayFastPathFileRepo) ListOverlayFilesByContentIDs(context.Context, []string) (map[string][]*models.MediaFile, error) {
r.overlayContentCalls++
return map[string][]*models.MediaFile{
"movie-1": {
{
ContentID: "movie-1",
Resolution: "4k",
CodecVideo: "hevc",
CodecAudio: "eac3",
AudioChannels: 6,
HDR: true,
Container: "mkv",
MediaFolderID: 1,
AudioTracks: []models.AudioTrack{{Codec: "eac3", Channels: 6, Default: true}},
SubtitleTracks: []models.SubtitleTrack{{Language: "en"}},
},
},
}, nil
}
func (r *overlayFastPathFileRepo) ListOverlayFilesByEpisodeIDs(context.Context, []string) (map[string][]*models.MediaFile, error) {
r.overlayEpisodeCalls++
return map[string][]*models.MediaFile{
"episode-1": {
{
EpisodeID: "episode-1",
Resolution: "1080p",
CodecVideo: "h264",
Container: "mp4",
MediaFolderID: 1,
},
},
}, nil
}
func TestListOverlaySummariesUsesOverlayFileProjection(t *testing.T) {
repo := &overlayFastPathFileRepo{}
handler := &ItemsHandler{fileRepo: repo}
summaries := handler.listOverlaySummaries(context.Background(), []*models.MediaItem{
{ContentID: "movie-1", Type: "movie"},
{ContentID: "episode-1", Type: "episode"},
}, catalog.AccessFilter{})
if repo.fullContentCalls != 0 {
t.Fatalf("full content file calls = %d, want 0", repo.fullContentCalls)
}
if repo.overlayContentCalls != 1 {
t.Fatalf("overlay content calls = %d, want 1", repo.overlayContentCalls)
}
if repo.overlayEpisodeCalls != 1 {
t.Fatalf("overlay episode calls = %d, want 1", repo.overlayEpisodeCalls)
}
if got := summaries["movie-1"].Resolution; got != "2160p" {
t.Fatalf("movie overlay resolution = %q, want 2160p", got)
}
if got := summaries["movie-1"].VideoCodec; got != "H.265" {
t.Fatalf("movie overlay video codec = %q, want H.265", got)
}
if got := summaries["episode-1"].VideoCodec; got != "H.264" {
t.Fatalf("episode overlay video codec = %q, want H.264", got)
}
}
@@ -1,12 +1,18 @@
package handlers
import (
"context"
"errors"
"net/http"
"net/http/httptest"
"strconv"
"testing"
"time"
apimw "github.com/Silo-Server/silo-server/internal/api/middleware"
"github.com/Silo-Server/silo-server/internal/auth"
"github.com/Silo-Server/silo-server/internal/models"
"github.com/Silo-Server/silo-server/internal/userstore"
)
func TestGetLeafUserDataUsesEbookReaderProgress(t *testing.T) {
@@ -45,3 +51,168 @@ func TestGetLeafUserDataUsesEbookReaderProgress(t *testing.T) {
t.Fatalf("completed ebook user data = %#v", complete)
}
}
// Audiobooks resolve user_data through the same watch-progress store as
// movies/episodes (not the ebook reader store), so an in-progress audiobook
// reports its saved position_seconds — which is what lets the player resume.
func TestGetLeafUserDataReturnsAudiobookProgress(t *testing.T) {
store := newPlaybackTestStore(t)
if err := store.SetProgress(context.Background(), "profile-1", "audiobook-1", 1234, 5000, userstore.ProgressThresholds{}); err != nil {
t.Fatalf("seed progress: %v", err)
}
handler := &ItemsHandler{storeProvider: testUserStoreProvider{store: store}}
req := httptest.NewRequest("GET", "/watch/audiobook-1", nil)
ctx := apimw.SetClaims(req.Context(), &auth.Claims{UserID: 1})
ctx = apimw.SetProfileID(ctx, "profile-1")
req = req.WithContext(ctx)
progress := handler.getLeafUserData(req, "audiobook-1", "audiobook")
if progress == nil {
t.Fatal("audiobook user data = nil, want saved progress")
}
if progress.PositionSeconds != 1234 {
t.Fatalf("PositionSeconds = %v, want 1234", progress.PositionSeconds)
}
}
func TestGetLeafUserDataUsesCompletedHistoryWhenProgressMissing(t *testing.T) {
store := newPlaybackTestStore(t)
addCompletedHistoryForUserDataTest(t, store, "movie-history-only")
handler := &ItemsHandler{storeProvider: testUserStoreProvider{store: store}}
req := authorizedUserDataRequest()
userData := handler.getLeafUserData(req, "movie-history-only", "movie")
if userData == nil {
t.Fatal("movie user data = nil, want history-backed watched state")
}
if !userData.Played {
t.Fatalf("Played = false, want true from completed history")
}
if userData.PositionSeconds != 0 || userData.IsInProgress {
t.Fatalf("history-only user data = %+v, want no resume position", userData)
}
}
func TestGetLeafUserDataPreservesResumeWhenCompletedHistoryExists(t *testing.T) {
store := newPlaybackTestStore(t)
if err := store.SetProgressAt(
context.Background(),
"profile-1",
"movie-rewatch",
600,
7200,
false,
time.Date(2026, 5, 5, 12, 0, 0, 0, time.UTC),
); err != nil {
t.Fatalf("seed progress: %v", err)
}
addCompletedHistoryForUserDataTest(t, store, "movie-rewatch")
handler := &ItemsHandler{storeProvider: testUserStoreProvider{store: store}}
req := authorizedUserDataRequest()
userData := handler.getLeafUserData(req, "movie-rewatch", "movie")
if userData == nil {
t.Fatal("movie user data = nil, want progress-backed user data")
}
if !userData.Played {
t.Fatalf("Played = false, want true from completed history")
}
if userData.PositionSeconds != 600 || userData.DurationSeconds != 7200 || !userData.IsInProgress {
t.Fatalf("resume fields = %+v, want in-progress resume preserved", userData)
}
}
func TestGetAggregateUserDataReturnsNilWhenProgressBatchFails(t *testing.T) {
store := &failingBatchProgressStore{}
handler := &ItemsHandler{storeProvider: testUserStoreProvider{store: store}}
req := httptest.NewRequest("GET", "/series/series-1/seasons/1", nil)
ctx := apimw.SetClaims(req.Context(), &auth.Claims{UserID: 1})
ctx = apimw.SetProfileID(ctx, "profile-1")
req = req.WithContext(ctx)
episodes := make([]*models.Episode, 501)
for i := range episodes {
episodes[i] = &models.Episode{ContentID: "episode-" + strconv.Itoa(i+1)}
}
if userData := handler.getAggregateUserData(req, episodes); userData != nil {
t.Fatalf("getAggregateUserData() = %#v, want nil after batch failure", userData)
}
if store.calls != 2 {
t.Fatalf("ListProgressByMediaItems calls = %d, want 2", store.calls)
}
}
func TestGetAggregateUserDataCountsCompletedHistory(t *testing.T) {
store := newPlaybackTestStore(t)
if err := store.SetProgressAt(
context.Background(),
"profile-1",
"episode-progress-complete",
0,
1800,
true,
time.Date(2026, 5, 5, 12, 0, 0, 0, time.UTC),
); err != nil {
t.Fatalf("seed progress: %v", err)
}
addCompletedHistoryForUserDataTest(t, store, "episode-history-complete")
handler := &ItemsHandler{storeProvider: testUserStoreProvider{store: store}}
req := authorizedUserDataRequest()
userData := handler.getAggregateUserData(req, []*models.Episode{
{ContentID: "episode-progress-complete"},
{ContentID: "episode-history-complete"},
{ContentID: "episode-unplayed"},
})
if userData == nil {
t.Fatal("aggregate user data = nil, want counts")
}
if userData.WatchedCount != 2 || userData.UnplayedCount != 1 || userData.Played {
t.Fatalf("aggregate user data = %+v, want two watched and one unplayed", userData)
}
}
type failingBatchProgressStore struct {
userstore.UserStore
calls int
}
func (s *failingBatchProgressStore) ListProgressByMediaItems(
_ context.Context,
_ string,
mediaItemIDs []string,
) (map[string]userstore.WatchProgress, error) {
s.calls++
if s.calls == 2 {
return nil, errors.New("progress batch failed")
}
progress := make(map[string]userstore.WatchProgress, len(mediaItemIDs))
for _, id := range mediaItemIDs {
progress[id] = userstore.WatchProgress{MediaItemID: id, Completed: true}
}
return progress, nil
}
func authorizedUserDataRequest() *http.Request {
req := httptest.NewRequest("GET", "/items/movie-1", nil)
ctx := apimw.SetClaims(req.Context(), &auth.Claims{UserID: 1})
ctx = apimw.SetProfileID(ctx, "profile-1")
return req.WithContext(ctx)
}
func addCompletedHistoryForUserDataTest(t *testing.T, store userstore.UserStore, mediaItemID string) {
t.Helper()
if err := store.AddHistory(context.Background(), userstore.WatchHistoryEntry{
ProfileID: "profile-1",
MediaItemID: mediaItemID,
WatchedAt: "2026-05-04T12:00:00Z",
DurationSeconds: 7200,
Completed: true,
Source: userstore.WatchHistorySourceTrakt,
}); err != nil {
t.Fatalf("seed completed history: %v", err)
}
}
+2
View File
@@ -2102,6 +2102,8 @@ func metadataContentLevelsForLibraryType(libraryType string) []string {
return []string{"audiobook"}
case "ebooks", "ebook":
return []string{"ebook"}
case "manga":
return []string{"manga"}
case "mixed":
return []string{"movie", "series", "season", "episode", "audiobook", "ebook"}
default:
@@ -14,6 +14,7 @@ func TestMetadataContentLevelsForLibraryTypeIncludesEbooks(t *testing.T) {
{name: "plural ebooks", libraryType: "ebooks", want: []string{"ebook"}},
{name: "singular ebook", libraryType: "ebook", want: []string{"ebook"}},
{name: "mixed includes ebook", libraryType: "mixed", want: []string{"movie", "series", "season", "episode", "audiobook", "ebook"}},
{name: "manga", libraryType: "manga", want: []string{"manga"}},
}
for _, tc := range cases {
+9 -31
View File
@@ -795,7 +795,8 @@ func (h *LibraryCollectionHandler) HandleListLibraryUserCollections(w http.Respo
}
userID := apimw.GetUserID(r.Context())
collections, err := usercollections.ListServerVisibleByLibrary(r.Context(), h.UserCollectionPool, userID, libraryID)
profileID := apimw.GetProfileID(r.Context())
collections, err := usercollections.ListServerVisibleByLibrary(r.Context(), h.UserCollectionPool, userID, profileID, libraryID)
if err != nil {
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to load user collections")
return
@@ -833,6 +834,7 @@ func (h *LibraryCollectionHandler) HandleListLibraryCollections(w http.ResponseW
}
userID := apimw.GetUserID(r.Context())
profileID := apimw.GetProfileID(r.Context())
groups, err := h.GroupRepo.ListByLibrary(r.Context(), libraryID)
if err != nil {
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to load groups")
@@ -855,7 +857,7 @@ func (h *LibraryCollectionHandler) HandleListLibraryCollections(w http.ResponseW
continue
}
if !userCollectionsLoaded {
loadedUserCollections, loadErr := usercollections.ListServerVisibleByLibrary(r.Context(), h.UserCollectionPool, userID, libraryID)
loadedUserCollections, loadErr := usercollections.ListServerVisibleByLibrary(r.Context(), h.UserCollectionPool, userID, profileID, libraryID)
if loadErr != nil {
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to load user collections")
return
@@ -1564,6 +1566,7 @@ func (h *LibraryCollectionHandler) HandleSyncAdminCollection(w http.ResponseWrit
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
return
}
slog.Error("collection sync failed", "collection_id", collectionID, "error", err)
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to sync collection")
return
}
@@ -3131,12 +3134,12 @@ func (h *LibraryCollectionHandler) loadLiveCollectionItems(r *http.Request, coll
switch {
case len(collection.LibraryIDs) > 0:
def.LibraryIDs = intersectCollectionLibraryIDs(def.LibraryIDs, collection.LibraryIDs)
def.LibraryIDs = catalog.IntersectCollectionLibraryIDs(def.LibraryIDs, collection.LibraryIDs)
if len(def.LibraryIDs) == 0 {
return []itemListResponse{}, nil
}
case collection.LibraryID > 0:
def.LibraryIDs = intersectCollectionLibraryIDs(def.LibraryIDs, []int{collection.LibraryID})
def.LibraryIDs = catalog.IntersectCollectionLibraryIDs(def.LibraryIDs, []int{collection.LibraryID})
if len(def.LibraryIDs) == 0 {
return []itemListResponse{}, nil
}
@@ -3175,33 +3178,8 @@ func (e smartCollectionQueryError) Unwrap() error {
return e.err
}
func intersectCollectionLibraryIDs(existing, required []int) []int {
if len(required) == 0 {
return append([]int(nil), existing...)
}
if len(existing) == 0 {
return append([]int(nil), required...)
}
allowed := make(map[int]struct{}, len(required))
for _, id := range required {
allowed[id] = struct{}{}
}
result := make([]int, 0, len(existing))
seen := make(map[int]struct{}, len(existing))
for _, id := range existing {
if _, ok := allowed[id]; !ok {
continue
}
if _, ok := seen[id]; ok {
continue
}
seen[id] = struct{}{}
result = append(result, id)
}
return result
}
// intersectCollectionLibraryIDs moved to catalog.IntersectCollectionLibraryIDs
// so the web API and the Jellyfin-compat resolver share one implementation.
func (h *LibraryCollectionHandler) toItemListResponse(r *http.Request, item *models.MediaItem) itemListResponse {
resp := itemListResponse{
+182
View File
@@ -0,0 +1,182 @@
package handlers
import (
"context"
"encoding/json"
"errors"
"net/http"
"strconv"
"strings"
"github.com/go-chi/chi/v5"
apimw "github.com/Silo-Server/silo-server/internal/api/middleware"
"github.com/Silo-Server/silo-server/internal/catalog"
"github.com/Silo-Server/silo-server/internal/literaryworks"
)
type LiteraryWorkService interface {
GetWork(ctx context.Context, workID string, filter catalog.AccessFilter) (*literaryworks.DetailResponse, error)
ListCandidates(ctx context.Context, contentID string, limit int) ([]literaryworks.Candidate, error)
LinkItems(ctx context.Context, workID string, contentIDs []string) (string, error)
UnlinkItem(ctx context.Context, workID, contentID string) error
ConfirmMatch(ctx context.Context, sourceContentID, targetContentID string, userID int) (string, error)
IgnoreMatch(ctx context.Context, sourceContentID, targetContentID string, userID int) error
}
type LiteraryWorkHandler struct {
Service LiteraryWorkService
}
type literaryWorkLinkRequest struct {
WorkID string `json:"work_id"`
ContentIDs []string `json:"content_ids"`
}
type literaryWorkDecisionRequest struct {
SourceContentID string `json:"source_content_id"`
TargetContentID string `json:"target_content_id"`
}
func (h *LiteraryWorkHandler) HandleGetWork(w http.ResponseWriter, r *http.Request) {
workID := strings.TrimSpace(chi.URLParam(r, "work_id"))
if workID == "" {
writeError(w, http.StatusBadRequest, "bad_request", "work_id is required")
return
}
if h == nil || h.Service == nil {
writeError(w, http.StatusServiceUnavailable, "unavailable", "Literary works are not configured")
return
}
resp, err := h.Service.GetWork(r.Context(), workID, requestAccessFilter(r))
if err != nil {
if errors.Is(err, literaryworks.ErrWorkNotFound) {
writeError(w, http.StatusNotFound, "not_found", "Work not found")
return
}
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to load work")
return
}
writeJSON(w, http.StatusOK, resp)
}
func (h *LiteraryWorkHandler) HandleListCandidates(w http.ResponseWriter, r *http.Request) {
contentID := strings.TrimSpace(chi.URLParam(r, "content_id"))
if contentID == "" {
writeError(w, http.StatusBadRequest, "bad_request", "content_id is required")
return
}
if h == nil || h.Service == nil {
writeError(w, http.StatusServiceUnavailable, "unavailable", "Literary works are not configured")
return
}
limit := 20
if raw := strings.TrimSpace(r.URL.Query().Get("limit")); raw != "" {
if parsed, err := strconv.Atoi(raw); err == nil && parsed > 0 && parsed <= 100 {
limit = parsed
}
}
candidates, err := h.Service.ListCandidates(r.Context(), contentID, limit)
if err != nil {
if errors.Is(err, literaryworks.ErrWorkNotFound) {
writeError(w, http.StatusNotFound, "not_found", "Item not found")
return
}
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to list work candidates")
return
}
writeJSON(w, http.StatusOK, map[string]any{"candidates": candidates})
}
func (h *LiteraryWorkHandler) HandleLinkItems(w http.ResponseWriter, r *http.Request) {
if h == nil || h.Service == nil {
writeError(w, http.StatusServiceUnavailable, "unavailable", "Literary works are not configured")
return
}
var req literaryWorkLinkRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "bad_request", "Invalid request body")
return
}
workID, err := h.Service.LinkItems(r.Context(), req.WorkID, req.ContentIDs)
if err != nil {
if errors.Is(err, literaryworks.ErrWorkNotFound) {
writeError(w, http.StatusNotFound, "not_found", "Item not found")
return
}
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to link work items")
return
}
writeJSON(w, http.StatusOK, map[string]string{"work_id": workID})
}
func (h *LiteraryWorkHandler) HandleUnlinkItem(w http.ResponseWriter, r *http.Request) {
workID := strings.TrimSpace(chi.URLParam(r, "work_id"))
contentID := strings.TrimSpace(chi.URLParam(r, "content_id"))
if workID == "" || contentID == "" {
writeError(w, http.StatusBadRequest, "bad_request", "work_id and content_id are required")
return
}
if h == nil || h.Service == nil {
writeError(w, http.StatusServiceUnavailable, "unavailable", "Literary works are not configured")
return
}
if err := h.Service.UnlinkItem(r.Context(), workID, contentID); err != nil {
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to unlink work item")
return
}
w.WriteHeader(http.StatusNoContent)
}
func (h *LiteraryWorkHandler) HandleConfirmMatch(w http.ResponseWriter, r *http.Request) {
h.handleDecision(w, r, true)
}
func (h *LiteraryWorkHandler) HandleIgnoreMatch(w http.ResponseWriter, r *http.Request) {
h.handleDecision(w, r, false)
}
func (h *LiteraryWorkHandler) handleDecision(w http.ResponseWriter, r *http.Request, confirm bool) {
if h == nil || h.Service == nil {
writeError(w, http.StatusServiceUnavailable, "unavailable", "Literary works are not configured")
return
}
var req literaryWorkDecisionRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "bad_request", "Invalid request body")
return
}
req.SourceContentID = strings.TrimSpace(req.SourceContentID)
req.TargetContentID = strings.TrimSpace(req.TargetContentID)
if req.SourceContentID == "" || req.TargetContentID == "" || req.SourceContentID == req.TargetContentID {
writeError(w, http.StatusBadRequest, "bad_request", "source_content_id and target_content_id are required")
return
}
claims := apimw.GetClaims(r.Context())
userID := 0
if claims != nil {
userID = claims.UserID
}
var (
workID string
err error
)
if confirm {
workID, err = h.Service.ConfirmMatch(r.Context(), req.SourceContentID, req.TargetContentID, userID)
} else {
err = h.Service.IgnoreMatch(r.Context(), req.SourceContentID, req.TargetContentID, userID)
}
if err != nil {
if errors.Is(err, literaryworks.ErrWorkNotFound) {
writeError(w, http.StatusNotFound, "not_found", "Item not found")
return
}
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to record work match decision")
return
}
if confirm {
writeJSON(w, http.StatusOK, map[string]string{"status": "ok", "work_id": workID})
return
}
writeJSON(w, http.StatusOK, map[string]string{"status": "ok"})
}
@@ -0,0 +1,73 @@
package handlers
import (
"context"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/go-chi/chi/v5"
"github.com/Silo-Server/silo-server/internal/catalog"
"github.com/Silo-Server/silo-server/internal/literaryworks"
)
type fakeLiteraryWorkService struct {
work *literaryworks.DetailResponse
err error
}
func (f fakeLiteraryWorkService) GetWork(ctx context.Context, workID string, filter catalog.AccessFilter) (*literaryworks.DetailResponse, error) {
return f.work, f.err
}
func (f fakeLiteraryWorkService) ListCandidates(ctx context.Context, contentID string, limit int) ([]literaryworks.Candidate, error) {
return []literaryworks.Candidate{{SourceContentID: contentID, TargetContentID: "audio-1", Score: 0.9}}, f.err
}
func (f fakeLiteraryWorkService) LinkItems(ctx context.Context, workID string, contentIDs []string) (string, error) {
if workID != "" {
return workID, f.err
}
return "work-1", f.err
}
func (f fakeLiteraryWorkService) UnlinkItem(ctx context.Context, workID, contentID string) error {
return f.err
}
func (f fakeLiteraryWorkService) ConfirmMatch(ctx context.Context, sourceContentID, targetContentID string, userID int) (string, error) {
return "work-1", f.err
}
func (f fakeLiteraryWorkService) IgnoreMatch(ctx context.Context, sourceContentID, targetContentID string, userID int) error {
return f.err
}
func TestLiteraryWorkHandlerGetWork(t *testing.T) {
handler := &LiteraryWorkHandler{Service: fakeLiteraryWorkService{
work: &literaryworks.DetailResponse{
WorkID: "work-1",
WorkTitle: "Project Hail Mary",
Formats: []literaryworks.FormatResponse{
{Type: "ebook", ContentID: "ebook-1"},
{Type: "audiobook", ContentID: "audio-1"},
},
},
}}
req := httptest.NewRequest(http.MethodGet, "/api/v1/works/work-1", nil)
rctx := chi.NewRouteContext()
rctx.URLParams.Add("work_id", "work-1")
req = req.WithContext(context.WithValue(req.Context(), chi.RouteCtxKey, rctx))
rec := httptest.NewRecorder()
handler.HandleGetWork(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("status = %d body=%s", rec.Code, rec.Body.String())
}
if !strings.Contains(rec.Body.String(), `"formats"`) || !strings.Contains(rec.Body.String(), `"audiobook"`) {
t.Fatalf("body = %s, want work formats", rec.Body.String())
}
}
+27 -8
View File
@@ -103,6 +103,7 @@ type PlaybackHandler struct {
StoreProvider userstore.UserStoreProvider // optional; enables progress/history persistence
WatchScrobbler PlaybackWatchScrobbler
StableIdentityResolver *watchstate.StableIdentityResolver
CompletionObserver watchstate.CompletionObserver // optional; auto-removes watched items from the watchlist
profileStaler ProfileStaler
profileRefreshRequester ProfileRefreshRequester
AdminStore PlaybackAdminStore // optional; enables admin playback history/live session cleanup
@@ -134,12 +135,12 @@ type PlaybackHandler struct {
// hwaccel, transcode dir). Wired to the live config in integrated mode
// so admin changes apply to newly started transcodes. Read it through
// playbackConfig(), which falls back to defaults when unset.
PlaybackConfig func() config.PlaybackConfig
FFmpegLogSink playback.FFmpegLogSink
transcodeMu sync.RWMutex
transcodes map[string]*playback.TranscodeSession
realtimeCommandMu sync.Mutex
realtimeCommands map[string]playbackCommandRecord
PlaybackConfig func() config.PlaybackConfig
FFmpegLogSink playback.FFmpegLogSink
transcodeMu sync.RWMutex
transcodes map[string]*playback.TranscodeSession
realtimeCommandMu sync.Mutex
realtimeCommands map[string]playbackCommandRecord
}
type PlaybackWatchScrobbler interface {
@@ -826,7 +827,9 @@ func (h *PlaybackHandler) persistStopAndHistory(ctx context.Context, session *pl
duration := float64(file.Duration)
thresholds := h.playbackThresholds(ctx)
watchSvc := watchstate.NewService(h.StoreProvider).WithStableIdentityResolver(h.StableIdentityResolver)
watchSvc := watchstate.NewService(h.StoreProvider).
WithStableIdentityResolver(h.StableIdentityResolver).
WithCompletionObserver(h.CompletionObserver)
stoppedAt := time.Now().UTC()
result, err := watchSvc.RecordPlaybackStop(ctx, session.UserID, session.ProfileID, targetID, duration, session.Position, stoppedAt, userstore.VersionHints{
FileID: file.ID,
@@ -1262,10 +1265,12 @@ func (h *PlaybackHandler) HandleStartPlayback(w http.ResponseWriter, r *http.Req
return
}
clientInfo := playbackClientInfoFromRequest(r)
sessionCtx := playback.WithClientInfo(r.Context(), clientInfo)
var session *playback.Session
if starter, ok := h.sessionMgr.(sessionStarterWithFilesContext); ok {
session, err = starter.StartSessionWithFilesContext(
r.Context(),
sessionCtx,
userID,
profileID,
effectiveFile.ID,
@@ -1321,6 +1326,9 @@ func (h *PlaybackHandler) HandleStartPlayback(w http.ResponseWriter, r *http.Req
AudioTrackIndex: audioTrackIndex,
TranscodeAudio: session.TranscodeAudio,
ClientIP: clientip.FromContext(r.Context()),
ClientName: clientInfo.Name,
ClientVersion: clientInfo.Version,
ClientUserAgent: clientInfo.UserAgent,
StreamBitrateKbps: streamBitrateKbps,
TargetAudioCodec: targetAudioCodec,
}); err != nil {
@@ -1442,6 +1450,17 @@ func (h *PlaybackHandler) HandleStartPlayback(w http.ResponseWriter, r *http.Req
writeJSON(w, http.StatusCreated, resp)
}
func playbackClientInfoFromRequest(r *http.Request) playback.ClientInfo {
if r == nil {
return playback.ClientInfo{}
}
return playback.ClientInfo{
Name: strings.TrimSpace(r.Header.Get("X-Silo-Client")),
Version: strings.TrimSpace(r.Header.Get("X-Silo-Client-Version")),
UserAgent: r.UserAgent(),
}
}
// subtitleURLExt returns the URL file extension for a subtitle codec.
// ASS/SSA tracks get ".ass" so the frontend can request raw ASS data for
// client-side rendering (JASSUB); PGS tracks get ".sup" for client-side
+145 -1
View File
@@ -46,6 +46,10 @@ type playbackSessionRow struct {
IsPaused bool `json:"is_paused"`
HasPlaybackControl bool `json:"has_playback_control"`
ClientIP string `json:"client_ip,omitempty"`
ClientName string `json:"client_name,omitempty"`
ClientVersion string `json:"client_version,omitempty"`
ClientLabel string `json:"client_label,omitempty"`
ClientUserAgent string `json:"client_user_agent,omitempty"`
AudioTrackIndex int `json:"audio_track_index"`
TranscodeAudio bool `json:"transcode_audio"`
StreamBitrateKbps *int `json:"stream_bitrate_kbps"`
@@ -149,6 +153,9 @@ func (l *PlaybackSessionsLoader) Load(
COALESCE(s.is_paused, FALSE),
COALESCE(s.has_websocket, FALSE),
COALESCE(HOST(s.client_ip), ''),
COALESCE(s.client_name, ''),
COALESCE(s.client_version, ''),
COALESCE(s.client_user_agent, ''),
COALESCE(s.audio_track_index, 0),
COALESCE(s.transcode_audio, FALSE),
s.stream_bitrate_kbps,
@@ -203,7 +210,8 @@ func (l *PlaybackSessionsLoader) Load(
&s.MediaTitle, &s.MediaType, &s.SeriesName, &s.EpisodeName, &s.SeasonNumber, &s.EpisodeNumber,
&posterPath,
&s.PlayMethod, &s.ReportingNode, &s.NodeDisplayName, &s.FileDuration, &s.StartedAt, &s.UpdatedAt,
&s.PositionSeconds, &s.IsPaused, &s.HasPlaybackControl, &s.ClientIP, &s.AudioTrackIndex, &s.TranscodeAudio, &streamBitrateKbps,
&s.PositionSeconds, &s.IsPaused, &s.HasPlaybackControl, &s.ClientIP, &s.ClientName, &s.ClientVersion,
&s.ClientUserAgent, &s.AudioTrackIndex, &s.TranscodeAudio, &streamBitrateKbps,
&s.TranscodeNodeURL, &s.TargetResolution, &s.TargetVideoCodec, &s.TargetAudioCodec, &targetBitrateKbps,
&s.TranscodeHWAccel, &s.SourceContainer, &sourceBitrateKbps, &s.SourceVideoCodec, &s.SourceVideoResolution,
&s.SourceAudioCodec, &sourceAudioChannels, &audioTracksJSON, &s.RequestedVideoCodec, &s.RequestedVideoResolution,
@@ -215,6 +223,7 @@ func (l *PlaybackSessionsLoader) Load(
s.TargetBitrateKbps = targetBitrateKbps
s.SourceBitrateKbps = sourceBitrateKbps
s.SourceAudioChannels = sourceAudioChannels
s.ClientLabel = playbackClientDisplayName(s.ClientName, s.ClientVersion, s.ClientUserAgent)
enrichPlaybackSessionRow(&s, audioTracksJSON)
sessions = append(sessions, s)
}
@@ -335,6 +344,141 @@ func firstNonEmptyValue(values ...string) string {
return ""
}
func playbackClientDisplayName(name, version, userAgent string) string {
name = strings.TrimSpace(name)
version = shortPlaybackClientVersion(version)
if name != "" {
if version != "" {
return name + " " + version
}
return name
}
userAgent = strings.TrimSpace(userAgent)
if userAgent == "" {
return ""
}
rules := []struct {
label string
tokens []string
versionTokens []string
}{
{label: "Infuse", tokens: []string{"infuse"}, versionTokens: []string{"infuse-direct", "infuse"}},
{label: "Findroid", tokens: []string{"findroid"}, versionTokens: []string{"findroid"}},
{label: "Streamyfin", tokens: []string{"streamyfin"}, versionTokens: []string{"streamyfin"}},
{label: "Swiftfin", tokens: []string{"swiftfin"}, versionTokens: []string{"swiftfin"}},
{label: "Jellyfin", tokens: []string{"jellyfin"}, versionTokens: []string{"jellyfin"}},
{label: "JellyCon", tokens: []string{"jellycon"}, versionTokens: []string{"jellycon"}},
{label: "Wholphin", tokens: []string{"wholphin"}, versionTokens: []string{"wholphin"}},
{label: "Fladder", tokens: []string{"fladder"}, versionTokens: []string{"fladder"}},
{label: "VidHub", tokens: []string{"vidhub"}, versionTokens: []string{"vidhub"}},
{label: "SenPlayer", tokens: []string{"senplayer"}, versionTokens: []string{"senplayer"}},
{label: "Kodi", tokens: []string{"kodi"}, versionTokens: []string{"kodi"}},
{label: "MPV", tokens: []string{"mpv"}, versionTokens: []string{"mpv"}},
{label: "Edge", tokens: []string{"edg/"}, versionTokens: []string{"edg"}},
{label: "Opera", tokens: []string{"opr/", "opera"}, versionTokens: []string{"opr", "opera"}},
{label: "Firefox", tokens: []string{"firefox/", "fxios/"}, versionTokens: []string{"firefox", "fxios"}},
{label: "Chrome", tokens: []string{"chrome/", "crios/"}, versionTokens: []string{"chrome", "crios"}},
{label: "Safari", tokens: []string{"safari/"}, versionTokens: []string{"version"}},
}
lower := strings.ToLower(userAgent)
for _, rule := range rules {
if !containsAny(lower, rule.tokens) {
continue
}
if version := firstProductVersion(userAgent, rule.versionTokens); version != "" {
return rule.label + " " + version
}
return rule.label
}
switch {
case strings.Contains(lower, "applecoremedia"):
return "Apple player"
case strings.Contains(lower, "okhttp"):
return "Android client"
case strings.Contains(lower, "dart/"):
return "Flutter client"
case strings.Contains(lower, "go-http-client"):
return "Go client"
case strings.Contains(lower, "curl/"):
return "curl"
case strings.Contains(lower, "python-requests"):
return "Python requests"
default:
return firstUserAgentProduct(userAgent)
}
}
func containsAny(value string, needles []string) bool {
for _, needle := range needles {
if strings.Contains(value, needle) {
return true
}
}
return false
}
func firstProductVersion(userAgent string, tokens []string) string {
for _, field := range strings.Fields(userAgent) {
name, version, ok := strings.Cut(field, "/")
if !ok {
continue
}
name = strings.TrimSpace(strings.ToLower(name))
for _, token := range tokens {
if name == strings.TrimSpace(strings.ToLower(token)) {
return shortPlaybackClientVersion(version)
}
}
}
return ""
}
func shortPlaybackClientVersion(version string) string {
version = strings.Trim(strings.TrimSpace(version), `";),`)
if version == "" {
return ""
}
version = strings.Map(func(r rune) rune {
if (r >= '0' && r <= '9') || r == '.' {
return r
}
return -1
}, version)
parts := strings.Split(version, ".")
filtered := parts[:0]
for _, part := range parts {
if part != "" {
filtered = append(filtered, part)
}
}
if len(filtered) == 0 {
return ""
}
if len(filtered) > 2 {
filtered = filtered[:2]
}
if len(filtered) == 2 && filtered[1] == "0" {
filtered = filtered[:1]
}
return strings.Join(filtered, ".")
}
func firstUserAgentProduct(userAgent string) string {
for _, field := range strings.Fields(userAgent) {
name, _, ok := strings.Cut(field, "/")
if ok {
name = strings.Trim(name, `";(),`)
if name != "" && !strings.EqualFold(name, "Mozilla") {
return name
}
}
}
return "Unknown client"
}
func (l *PlaybackSessionsLoader) populateProfileNames(ctx context.Context, sessions []playbackSessionRow) {
if l == nil || l.storeProv == nil || len(sessions) == 0 {
return
-37
View File
@@ -503,7 +503,6 @@ func (h *PluginHandler) HandleCreateInstallation(w http.ResponseWriter, r *http.
}
h.syncMetadataProviders(r.Context(), result.Installation)
h.syncImageResolvers(r.Context(), result.Installation)
response, err := h.buildInstallationResponse(r.Context(), result.Installation, result.Manifest)
if err != nil {
@@ -662,7 +661,6 @@ func (h *PluginHandler) installUploadedPlugin(ctx context.Context, path string)
func (h *PluginHandler) writeUploadedPluginResponse(w http.ResponseWriter, r *http.Request, result *plugins.InstallResult) {
h.syncMetadataProviders(r.Context(), result.Installation)
h.syncImageResolvers(r.Context(), result.Installation)
response, err := h.buildInstallationResponse(r.Context(), result.Installation, result.Manifest)
if err != nil {
@@ -703,35 +701,6 @@ func (h *PluginHandler) syncMetadataProviders(ctx context.Context, installation
}
}
// syncImageResolvers registers image resolver sources for any metadata_provider.v1
// capabilities from the given installation so that plugin-prefixed image URLs
// (e.g. "tmdb://poster/abc.jpg") can be resolved at runtime.
func (h *PluginHandler) syncImageResolvers(ctx context.Context, installation *plugins.Installation) {
if h.imageResolver == nil || h.service == nil {
return
}
caps, err := h.installations.ListCapabilities(ctx, installation.ID)
if err != nil {
slog.Error("listing capabilities for image resolver sync",
"installation_id", installation.ID, "error", err)
return
}
for _, cap := range caps {
if cap.Type != "metadata_provider.v1" {
continue
}
source := metadata.NewPluginClientSource(installation.ID, cap.ID, func(
ctx context.Context, installationID int, capabilityID string,
) (metadata.PluginMetadataClient, error) {
return h.service.MetadataProviderClient(ctx, installationID, capabilityID)
})
h.imageResolver.RegisterSource(cap.ID, source)
slog.Info("registered plugin image resolver", "capability_id", cap.ID, "installation_id", installation.ID)
}
}
func isZipUpload(data []byte) bool {
if len(data) < 4 {
return false
@@ -849,11 +818,6 @@ func (h *PluginHandler) HandleUpdateInstallation(w http.ResponseWriter, r *http.
return
}
// When a plugin is being enabled, register its image resolvers.
if req.Enabled != nil && *req.Enabled && !currentInstallation.Enabled {
h.syncImageResolvers(r.Context(), installation)
}
response, err := h.buildInstallationResponse(r.Context(), installation, nil)
if err != nil {
slog.Error("building updated plugin installation response", "error", err)
@@ -883,7 +847,6 @@ func (h *PluginHandler) HandleApplyUpdate(w http.ResponseWriter, r *http.Request
}
h.syncMetadataProviders(r.Context(), installation)
h.syncImageResolvers(r.Context(), installation)
response, err := h.buildInstallationResponse(r.Context(), installation, nil)
if err != nil {
+65 -12
View File
@@ -6,6 +6,7 @@ import (
"net/http"
"strconv"
"github.com/Silo-Server/silo-server/internal/access"
apimw "github.com/Silo-Server/silo-server/internal/api/middleware"
evt "github.com/Silo-Server/silo-server/internal/events"
"github.com/Silo-Server/silo-server/internal/userstore"
@@ -14,6 +15,9 @@ import (
// ProgressLibraryLookup resolves which progress items belong to a library.
type ProgressLibraryLookup interface {
GetItemsInFolder(ctx context.Context, contentIDs []string, folderID int) (map[string]bool, error)
// FilterAccessibleContentIDs returns the subset of contentIDs the viewer
// may access given their library scope and content-rating ceiling.
FilterAccessibleContentIDs(ctx context.Context, contentIDs []string, allowedFolderIDs, disabledFolderIDs []int, maxContentRating string) (map[string]bool, error)
}
// ProgressHandler handles watch progress and sync endpoints.
@@ -102,6 +106,25 @@ func (h *ProgressHandler) HandleListProgress(w http.ResponseWriter, r *http.Requ
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to list progress")
return
}
// Drop entries the viewer can't access before they reach the client.
// Without this, a library-restricted profile receives progress rows for
// items outside its scope (e.g. an XXX title) and the client then fans out
// per-item detail fetches that 404 — a dead Continue Watching tile. Only
// runs for restricted profiles; unrestricted viewers are unaffected.
if scope, ok := access.GetScope(r.Context()); ok &&
(scope.AllowedLibraryIDs != nil || len(scope.DisabledLibraryIDs) > 0 || scope.MaxContentRating != "") {
if h.LibraryLookup == nil {
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to apply access filter")
return
}
entries, err = filterProgressEntriesByAccess(r.Context(), entries, scope, h.LibraryLookup)
if err != nil {
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to apply access filter")
return
}
}
if libraryID > 0 {
if h.LibraryLookup == nil {
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to apply library filter")
@@ -144,6 +167,27 @@ func parseLibraryIDParam(r *http.Request) (int, error) {
return libraryID, nil
}
// progressContentIDs collects the media item IDs from a progress slice.
func progressContentIDs(entries []userstore.WatchProgress) []string {
contentIDs := make([]string, 0, len(entries))
for _, entry := range entries {
contentIDs = append(contentIDs, entry.MediaItemID)
}
return contentIDs
}
// keepAccessibleEntries returns, in order, the entries whose media item ID maps
// to true in accessible.
func keepAccessibleEntries(entries []userstore.WatchProgress, accessible map[string]bool) []userstore.WatchProgress {
filtered := make([]userstore.WatchProgress, 0, len(entries))
for _, entry := range entries {
if accessible[entry.MediaItemID] {
filtered = append(filtered, entry)
}
}
return filtered
}
func filterProgressEntriesByLibrary(
ctx context.Context,
entries []userstore.WatchProgress,
@@ -154,24 +198,33 @@ func filterProgressEntriesByLibrary(
return entries, nil
}
contentIDs := make([]string, 0, len(entries))
for _, entry := range entries {
contentIDs = append(contentIDs, entry.MediaItemID)
}
allowed, err := lookup.GetItemsInFolder(ctx, contentIDs, libraryID)
allowed, err := lookup.GetItemsInFolder(ctx, progressContentIDs(entries), libraryID)
if err != nil {
return nil, err
}
filtered := make([]userstore.WatchProgress, 0, len(entries))
for _, entry := range entries {
if allowed[entry.MediaItemID] {
filtered = append(filtered, entry)
}
return keepAccessibleEntries(entries, allowed), nil
}
// filterProgressEntriesByAccess removes progress entries whose item falls
// outside the viewer's access scope (allowed/disabled libraries and the
// content-rating ceiling).
func filterProgressEntriesByAccess(
ctx context.Context,
entries []userstore.WatchProgress,
scope access.Scope,
lookup ProgressLibraryLookup,
) ([]userstore.WatchProgress, error) {
if len(entries) == 0 {
return entries, nil
}
return filtered, nil
accessible, err := lookup.FilterAccessibleContentIDs(ctx, progressContentIDs(entries), scope.AllowedLibraryIDs, scope.DisabledLibraryIDs, scope.MaxContentRating)
if err != nil {
return nil, err
}
return keepAccessibleEntries(entries, accessible), nil
}
// HandleSyncProgress handles POST /sync/progress.
@@ -0,0 +1,111 @@
package handlers
import (
"context"
"slices"
"testing"
"github.com/Silo-Server/silo-server/internal/access"
"github.com/Silo-Server/silo-server/internal/userstore"
)
// fakeProgressLookup records the arguments it was called with and returns a
// fixed accessibility map.
type fakeProgressLookup struct {
accessible map[string]bool
gotContentIDs []string
gotAllowed []int
gotDisabled []int
gotRating string
}
func (f *fakeProgressLookup) GetItemsInFolder(context.Context, []string, int) (map[string]bool, error) {
return nil, nil
}
func (f *fakeProgressLookup) FilterAccessibleContentIDs(
_ context.Context, contentIDs []string, allowedFolderIDs, disabledFolderIDs []int, maxContentRating string,
) (map[string]bool, error) {
f.gotContentIDs = contentIDs
f.gotAllowed = allowedFolderIDs
f.gotDisabled = disabledFolderIDs
f.gotRating = maxContentRating
return f.accessible, nil
}
func entries(ids ...string) []userstore.WatchProgress {
out := make([]userstore.WatchProgress, 0, len(ids))
for _, id := range ids {
out = append(out, userstore.WatchProgress{MediaItemID: id})
}
return out
}
func ids(entries []userstore.WatchProgress) []string {
out := make([]string, 0, len(entries))
for _, e := range entries {
out = append(out, e.MediaItemID)
}
return out
}
func TestFilterProgressEntriesByAccess(t *testing.T) {
// access.Scope sets DisabledLibraryIDs only when AllowedLibraryIDs is nil
// (see access.Scope docs), so the two restriction shapes are tested
// separately rather than as one impossible combined scope.
cases := []struct {
name string
scope access.Scope
}{
{
name: "allowed libraries + rating",
scope: access.Scope{AllowedLibraryIDs: []int{1, 2}, MaxContentRating: "PG-13"},
},
{
name: "disabled libraries + rating",
scope: access.Scope{DisabledLibraryIDs: []int{9}, MaxContentRating: "PG-13"},
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
lookup := &fakeProgressLookup{accessible: map[string]bool{"a": true, "c": true}}
got, err := filterProgressEntriesByAccess(context.Background(), entries("a", "b", "c"), tc.scope, lookup)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
want := []string{"a", "c"}
if g := ids(got); len(g) != len(want) || g[0] != want[0] || g[1] != want[1] {
t.Fatalf("filtered entries = %v, want %v", g, want)
}
// Scope must be forwarded verbatim to the lookup.
if !slices.Equal(lookup.gotAllowed, tc.scope.AllowedLibraryIDs) {
t.Errorf("allowed folders = %v, want %v", lookup.gotAllowed, tc.scope.AllowedLibraryIDs)
}
if !slices.Equal(lookup.gotDisabled, tc.scope.DisabledLibraryIDs) {
t.Errorf("disabled folders = %v, want %v", lookup.gotDisabled, tc.scope.DisabledLibraryIDs)
}
if len(lookup.gotContentIDs) != 3 {
t.Errorf("content ids = %v, want 3 entries", lookup.gotContentIDs)
}
if lookup.gotRating != tc.scope.MaxContentRating {
t.Errorf("max content rating = %q, want %q", lookup.gotRating, tc.scope.MaxContentRating)
}
})
}
}
func TestFilterProgressEntriesByAccessEmpty(t *testing.T) {
lookup := &fakeProgressLookup{accessible: map[string]bool{}}
got, err := filterProgressEntriesByAccess(context.Background(), nil, access.Scope{}, lookup)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if len(got) != 0 {
t.Fatalf("expected no entries, got %v", ids(got))
}
}
+100
View File
@@ -764,6 +764,8 @@ func (h *SectionHandler) loadResolvedHomeSections(r *http.Request) ([]sections.R
}
}
resolved = filterResolvedSectionsByAccess(resolved, accessFilter)
return resolved, libraryIDs, accessFilter, profileID, nil
}
@@ -1047,6 +1049,7 @@ func (h *SectionHandler) HandleSectionSettings(w http.ResponseWriter, r *http.Re
}
resolved := sections.ResolveForSettings(adminSections, overrides)
resolved = filterResolvedSectionsByAccess(resolved, requestAccessFilter(r))
type settingsEntry struct {
ID string `json:"id"`
@@ -1080,6 +1083,64 @@ func (h *SectionHandler) HandleSectionSettings(w http.ResponseWriter, r *http.Re
writeJSON(w, http.StatusOK, map[string][]settingsEntry{"sections": entries})
}
func filterResolvedSectionsByAccess(resolved []sections.ResolvedSection, filter catalog.AccessFilter) []sections.ResolvedSection {
if filter.AllowedLibraryIDs == nil && len(filter.DisabledLibraryIDs) == 0 {
return resolved
}
out := resolved[:0]
for _, section := range resolved {
if sectionAllowedByAccess(section, filter) {
out = append(out, section)
}
}
return out
}
func sectionAllowedByAccess(section sections.ResolvedSection, filter catalog.AccessFilter) bool {
configLibraryIDs := sections.ParseConfigFilters(section.Config).LibraryIDs()
if len(configLibraryIDs) == 0 {
return true
}
if filter.AllowedLibraryIDs != nil {
return intSlicesIntersect(configLibraryIDs, filter.AllowedLibraryIDs)
}
for _, libraryID := range configLibraryIDs {
if !intSliceContains(filter.DisabledLibraryIDs, libraryID) {
return true
}
}
return false
}
func intSlicesIntersect(left, right []int) bool {
if len(left) == 0 || len(right) == 0 {
return false
}
set := make(map[int]struct{}, len(right))
for _, value := range right {
set[value] = struct{}{}
}
for _, value := range left {
if _, ok := set[value]; ok {
return true
}
}
return false
}
func intSliceContains(values []int, target int) bool {
for _, value := range values {
if value == target {
return true
}
}
return false
}
// applyDiversityFilter removes items from sections whose recipe has
// AvoidDuplicates=true if the same content ID was already surfaced by an
// earlier section in the same render. Operates in-place — preserves order.
@@ -1179,6 +1240,7 @@ func (h *SectionHandler) buildSectionsResponse(r *http.Request, withItems []sect
userStates := h.listSectionItemUserStates(r, allItems)
imageURLs := h.resolveSectionItemImageURLs(r.Context(), withItems)
episodeMeta := h.listSectionEpisodeItemMeta(r.Context(), withItems, requestAccessFilter(r))
mangaChapterMeta := h.listSectionMangaChapterItemMeta(r.Context(), allItems)
for _, s := range withItems {
items := make([]sectionItemResponse, 0, len(s.Items))
for _, item := range s.Items {
@@ -1193,6 +1255,17 @@ func (h *SectionHandler) buildSectionsResponse(r *http.Request, withItems []sect
meta = &value
}
}
// Manga chapters carry their series linkage on top of whatever
// meta (e.g. reading progress) the section already resolved, so
// continue-reading cards can head to the series.
if value, ok := mangaChapterMeta[item.ContentID]; ok {
if meta == nil {
empty := sections.SectionItemMeta{}
meta = &empty
}
meta.SeriesID = value.SeriesID
meta.SeriesTitle = value.SeriesTitle
}
imageKey := sectionItemImageKey{sectionID: s.ID, contentID: item.ContentID}
items = append(items, h.toSectionItemResponse(s.SectionType, item, meta, overlaySummaries[item.ContentID], userStates[item.ContentID], imageURLs[imageKey]))
}
@@ -1211,6 +1284,33 @@ func (h *SectionHandler) buildSectionsResponse(r *http.Request, withItems []sect
return resp
}
// listSectionMangaChapterItemMeta resolves series linkage for every manga
// chapter (type='ebook' linked via manga_chapters) among the section items.
// Non-chapter ebooks simply get no entry.
func (h *SectionHandler) listSectionMangaChapterItemMeta(ctx context.Context, items []*models.MediaItem) map[string]sections.SectionItemMeta {
if h == nil || h.fetcher == nil {
return map[string]sections.SectionItemMeta{}
}
ids := make([]string, 0)
seen := make(map[string]struct{})
for _, item := range items {
if item == nil || item.Type != "ebook" || strings.TrimSpace(item.ContentID) == "" {
continue
}
if _, ok := seen[item.ContentID]; ok {
continue
}
seen[item.ContentID] = struct{}{}
ids = append(ids, item.ContentID)
}
meta, err := h.fetcher.FetchMangaChapterSeriesMeta(ctx, ids)
if err != nil {
slog.Warn("loading section manga chapter metadata", "error", err)
return map[string]sections.SectionItemMeta{}
}
return meta
}
func (h *SectionHandler) listSectionEpisodeItemMeta(ctx context.Context, withItems []sections.SectionWithItems, filter catalog.AccessFilter) map[string]sections.SectionItemMeta {
if h == nil || h.episodeFetcher == nil {
return map[string]sections.SectionItemMeta{}
+62
View File
@@ -390,6 +390,68 @@ func TestLibraryDefaultSectionsUsesFolderType(t *testing.T) {
}
}
func TestFilterResolvedSectionsByAccessHidesBlockedLibraryRows(t *testing.T) {
sectionsIn := []sections.ResolvedSection{
{
ID: "continue",
SectionType: sections.SectionContinueWatching,
Title: "Continue Watching",
Config: sections.ContinueTypeConfig(sections.ContinueTypeWatching),
},
{
ID: "allowed-library",
SectionType: sections.SectionRecentlyAdded,
Title: "Recently Added in Allowed",
Config: sections.GeneratedHomeLibraryRecentConfig(11),
},
{
ID: "blocked-library",
SectionType: sections.SectionRecentlyAdded,
Title: "Recently Added in Blocked",
Config: sections.GeneratedHomeLibraryRecentConfig(42),
},
}
got := filterResolvedSectionsByAccess(sectionsIn, catalog.AccessFilter{
AllowedLibraryIDs: []int{11},
})
if len(got) != 2 {
t.Fatalf("filtered sections length = %d, want 2", len(got))
}
if got[0].ID != "continue" || got[1].ID != "allowed-library" {
t.Fatalf("filtered section ids = [%s %s], want [continue allowed-library]", got[0].ID, got[1].ID)
}
}
func TestFilterResolvedSectionsByAccessHidesDisabledOnlyRows(t *testing.T) {
sectionsIn := []sections.ResolvedSection{
{
ID: "disabled-library",
SectionType: sections.SectionRecentlyReleased,
Title: "Recently Released in Disabled",
Config: sections.GeneratedHomeLibraryRecentConfig(42),
},
{
ID: "mixed-libraries",
SectionType: sections.SectionRecentlyReleased,
Title: "Mixed",
Config: []byte(`{"filter_library_ids":[11,42]}`),
},
}
got := filterResolvedSectionsByAccess(sectionsIn, catalog.AccessFilter{
DisabledLibraryIDs: []int{42},
})
if len(got) != 1 {
t.Fatalf("filtered sections length = %d, want 1", len(got))
}
if got[0].ID != "mixed-libraries" {
t.Fatalf("filtered section id = %s, want mixed-libraries", got[0].ID)
}
}
func TestApplyDiversityFilterSkipsDuplicatesInLaterAvoidSection(t *testing.T) {
in := []sections.SectionWithItems{
{ResolvedSection: sections.ResolvedSection{ID: "ra", SectionType: sections.SectionRecentlyAdded}, Items: []*models.MediaItem{{ContentID: "abc"}}},
+40 -6
View File
@@ -9,13 +9,23 @@ import (
"net/http"
"strconv"
"strings"
"time"
"github.com/go-chi/chi/v5"
apimw "github.com/Silo-Server/silo-server/internal/api/middleware"
"github.com/Silo-Server/silo-server/internal/cache"
"github.com/Silo-Server/silo-server/internal/userstore"
)
// deviceSeenThrottle bounds how often a device's last_seen_at is refreshed from
// request traffic. Device-setting reads each registered the device (an upsert
// on a single per-device row), so a page that fetches many settings serialized
// hundreds of upserts on that row's lock. Skipping the upsert when the device
// was seen within this window removes the contention while keeping last_seen
// fresh to within the window.
const deviceSeenThrottle = 5 * time.Minute
const subtitleAppearanceSettingKey = "subtitle_appearance"
const (
libraryPageStateSettingKey = "ui.library_page_state"
@@ -38,11 +48,32 @@ type ServerSettingReader interface {
type SettingsHandler struct {
storeProvider userstore.UserStoreProvider
serverSettings ServerSettingReader
deviceSeen *cache.TTLCache[struct{}]
}
// NewSettingsHandler creates a new SettingsHandler.
func NewSettingsHandler(provider userstore.UserStoreProvider) *SettingsHandler {
return &SettingsHandler{storeProvider: provider}
return &SettingsHandler{
storeProvider: provider,
deviceSeen: cache.NewTTLCache[struct{}](),
}
}
// shouldRegisterDevice reports whether the device's last_seen_at should be
// refreshed now, throttling to one upsert per deviceSeenThrottle window per
// (profile, device). It marks the device seen before returning true so a burst
// of concurrent reads collapses to a single upsert instead of contending on the
// device row.
func (h *SettingsHandler) shouldRegisterDevice(profileID, deviceID string) bool {
if h == nil || h.deviceSeen == nil {
return true
}
key := profileID + "\x00" + deviceID
if _, seen := h.deviceSeen.Get(key); seen {
return false
}
h.deviceSeen.Set(key, struct{}{}, deviceSeenThrottle)
return true
}
// SetServerSettings configures the optional server settings reader for overlay config etc.
@@ -382,7 +413,7 @@ func (h *SettingsHandler) HandleGetDeviceSetting(w http.ResponseWriter, r *http.
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to access user store")
return
}
registerRequestDevice(r.Context(), store, profileID, device)
h.registerRequestDevice(r.Context(), store, profileID, device)
value, err := store.GetDeviceSetting(r.Context(), profileID, device.DeviceID, key)
if err != nil {
@@ -482,7 +513,7 @@ func (h *SettingsHandler) HandleDeleteDeviceSetting(w http.ResponseWriter, r *ht
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to access user store")
return
}
registerRequestDevice(r.Context(), store, profileID, device)
h.registerRequestDevice(r.Context(), store, profileID, device)
if err := store.DeleteDeviceSetting(r.Context(), profileID, device.DeviceID, key); err != nil {
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to delete device setting")
@@ -512,7 +543,7 @@ func (h *SettingsHandler) HandleGetEffectiveSettings(w http.ResponseWriter, r *h
}
device := deviceMetadataFromRequest(r)
registerRequestDevice(r.Context(), store, profileID, device)
h.registerRequestDevice(r.Context(), store, profileID, device)
keys := parseSettingKeys(keysParam)
resp := effectiveSettingsResponse{
Settings: make([]effectiveSettingResponse, 0, len(keys)),
@@ -543,7 +574,7 @@ func (h *SettingsHandler) HandleGetEffectiveSubtitleAppearance(w http.ResponseWr
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to access user store")
return
}
registerRequestDevice(r.Context(), store, profileID, device)
h.registerRequestDevice(r.Context(), store, profileID, device)
resolved, err := h.resolveEffectiveSetting(r.Context(), store, profileID, device, subtitleAppearanceSettingKey)
if err != nil {
@@ -604,7 +635,7 @@ func deviceMetadataFromRequest(r *http.Request) requestDeviceMetadata {
}
}
func registerRequestDevice(
func (h *SettingsHandler) registerRequestDevice(
ctx context.Context,
store userstore.UserStore,
profileID string,
@@ -616,6 +647,9 @@ func registerRequestDevice(
if store == nil {
return
}
if !h.shouldRegisterDevice(profileID, device.DeviceID) {
return
}
registry, ok := store.(userstore.DeviceRegistry)
if !ok {
return
@@ -48,7 +48,8 @@ func (r testAdminUserRepo) GetByID(_ context.Context, id int) (*models.User, err
}
func TestRegisterRequestDeviceNilStore(t *testing.T) {
registerRequestDevice(context.Background(), nil, "profile-1", requestDeviceMetadata{
h := NewSettingsHandler(nil)
h.registerRequestDevice(context.Background(), nil, "profile-1", requestDeviceMetadata{
DeviceID: "device-1",
DeviceName: "Living Room",
DevicePlatform: "web",
@@ -0,0 +1,28 @@
package handlers
import (
"testing"
"github.com/Silo-Server/silo-server/internal/cache"
)
// Device-setting reads register the device (last_seen_at upsert). A page that
// fetches many settings must not issue one upsert per read for the same device
// — they contend on a single row. The throttle collapses repeats within the
// window to one registration.
func TestShouldRegisterDevice_ThrottlesRepeatWithinWindow(t *testing.T) {
h := &SettingsHandler{deviceSeen: cache.NewTTLCache[struct{}]()}
if !h.shouldRegisterDevice("p1", "dev1") {
t.Fatal("first sighting of a device should register")
}
if h.shouldRegisterDevice("p1", "dev1") {
t.Fatal("repeat sighting within the window should be throttled (no upsert)")
}
if !h.shouldRegisterDevice("p1", "dev2") {
t.Fatal("a different device on the same profile should register")
}
if !h.shouldRegisterDevice("p2", "dev1") {
t.Fatal("the same device id under a different profile should register")
}
}
@@ -14,6 +14,7 @@ import (
"github.com/go-chi/chi/v5"
apimw "github.com/Silo-Server/silo-server/internal/api/middleware"
"github.com/Silo-Server/silo-server/internal/catalog"
"github.com/Silo-Server/silo-server/internal/collections/templates"
"github.com/Silo-Server/silo-server/internal/mdblist"
"github.com/Silo-Server/silo-server/internal/s3client"
@@ -65,13 +66,14 @@ func (h *UserCollectionImportHandler) HandleListTemplates(w http.ResponseWriter,
}
type userImportSharedFields struct {
Title string `json:"title"`
Description string `json:"description"`
Limit *int `json:"limit,omitempty"`
SyncSchedule string `json:"sync_schedule"`
IsShared bool `json:"is_shared"`
PosterURL string `json:"poster_url"`
LibraryIDs []int `json:"library_ids,omitempty"`
Title string `json:"title"`
Description string `json:"description"`
Limit *int `json:"limit,omitempty"`
SyncSchedule string `json:"sync_schedule"`
IsShared bool `json:"is_shared"`
PosterURL string `json:"poster_url"`
LibraryIDs []int `json:"library_ids,omitempty"`
DisplayQueryDefinition json.RawMessage `json:"display_query_definition,omitempty"`
}
type userImportMDBListRequest struct {
@@ -200,26 +202,35 @@ func (h *UserCollectionImportHandler) createImportedCollection(
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
return
}
if !validateOptionalLibraryIDs(cfg.LibraryIDs, w) {
return
}
sourceConfigJSON, err := usercollections.MarshalSourceConfig(cfg)
if err != nil {
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to encode source config")
return
}
displayQueryDefinition, err := catalog.NormalizeDisplayQueryFragment(shared.DisplayQueryDefinition)
if err != nil {
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
return
}
collection, err := store.CreateCollection(r.Context(), userstore.CreateCollectionInput{
CreatorProfileID: profileID,
Name: strings.TrimSpace(shared.Title),
Description: strings.TrimSpace(shared.Description),
CollectionType: collectionType,
IsShared: shared.IsShared,
QueryDefinition: "{}",
SortConfig: "{}",
SourceURL: cfg.DisplayURL(),
SourceConfig: sourceConfigJSON,
SyncSchedule: schedule,
NextSyncAt: usercollections.InitialNextSyncAt(schedule),
PosterURL: strings.TrimSpace(shared.PosterURL),
CreatorProfileID: profileID,
Name: strings.TrimSpace(shared.Title),
Description: strings.TrimSpace(shared.Description),
CollectionType: collectionType,
IsShared: shared.IsShared,
QueryDefinition: "{}",
SortConfig: "{}",
SourceURL: cfg.DisplayURL(),
SourceConfig: sourceConfigJSON,
SyncSchedule: schedule,
NextSyncAt: usercollections.InitialNextSyncAt(schedule),
DisplayQueryDefinition: displayQueryDefinition,
PosterURL: strings.TrimSpace(shared.PosterURL),
})
if err != nil {
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to create collection")
@@ -425,3 +436,13 @@ func validateOptionalLimit(limit *int, w http.ResponseWriter) bool {
}
return true
}
func validateOptionalLibraryIDs(libraryIDs []int, w http.ResponseWriter) bool {
for _, id := range libraryIDs {
if id <= 0 {
writeError(w, http.StatusBadRequest, "bad_request", "library_ids must contain positive IDs")
return false
}
}
return true
}
+1 -1
View File
@@ -123,7 +123,7 @@ func resolveItemUserStatesWithOptions(
}
}
progressMap, err := store.ListProgressByMediaItems(ctx, profileID, progressIDs)
progressMap, err := userstore.ListProgressWithCompletedHistory(ctx, store, profileID, progressIDs)
if err != nil {
return nil, err
}
+30
View File
@@ -6,6 +6,7 @@ import (
"time"
"github.com/Silo-Server/silo-server/internal/models"
"github.com/Silo-Server/silo-server/internal/userstore"
)
func TestResolveItemUserStatesIncludesCompletedEbookReaderProgress(t *testing.T) {
@@ -94,3 +95,32 @@ func TestResolveItemUserStatesExcludesHiddenEbookReaderProgress(t *testing.T) {
t.Fatalf("ebook updated after hidden_before = %+v, want played again", states["ebook-reread"])
}
}
func TestResolveItemUserStatesIncludesCompletedHistory(t *testing.T) {
ctx := context.Background()
store := newProfileTestStore(t)
addCompletedHistoryForUserDataTest(t, store, "movie-history-only")
items := []*models.MediaItem{
{ContentID: "movie-history-only", Type: "movie", Title: "Imported Movie"},
}
states, err := resolveItemUserStates(ctx, store, "profile-1", nil, items)
if err != nil {
t.Fatalf("resolveItemUserStates: %v", err)
}
if states["movie-history-only"] == nil || !states["movie-history-only"].Played {
t.Fatalf("history-only movie state = %+v, want played", states["movie-history-only"])
}
}
func TestAllEpisodesCompletedIncludesCompletedHistory(t *testing.T) {
episodes := []*models.Episode{{ContentID: "episode-progress"}, {ContentID: "episode-history"}}
progress := map[string]userstore.WatchProgress{
"episode-progress": {MediaItemID: "episode-progress", Completed: true},
"episode-history": {MediaItemID: "episode-history", Completed: true},
}
if !allEpisodesCompleted(episodes, progress) {
t.Fatal("allEpisodesCompleted = false, want completed from progress plus history")
}
}
+60 -3
View File
@@ -37,6 +37,7 @@ import (
"github.com/Silo-Server/silo-server/internal/historyimport"
"github.com/Silo-Server/silo-server/internal/intromarkers"
"github.com/Silo-Server/silo-server/internal/libraryingest"
"github.com/Silo-Server/silo-server/internal/literaryworks"
"github.com/Silo-Server/silo-server/internal/logstream"
"github.com/Silo-Server/silo-server/internal/mail"
"github.com/Silo-Server/silo-server/internal/markers"
@@ -113,6 +114,7 @@ type Dependencies struct {
AdminStatsProvider handlers.AdminStatsSource
Recommender recommendations.Recommender // nil when disabled
RecWorker *recommendations.Worker // nil when disabled
CatalogSearchVectorizer catalog.CatalogSearchQueryVectorizer
RatingsRepo *catalog.RatingsRepo
PersonRepo *catalog.PersonRepository
PersonRefreshQueue handlers.PersonRefreshQueue
@@ -141,6 +143,7 @@ type Dependencies struct {
MarkerContributionStore *markers.ContributionStore
MarkerContributionService *markers.ContributionService
WatchProviderService handlers.WatchProviderService
WatchCompletionObserver watchstate.CompletionObserver
PluginService *plugins.Service
PluginHTTPProxy *plugins.HTTPProxy
PluginUserConfig *plugins.UserConfigStore
@@ -424,6 +427,7 @@ func NewRouter(deps Dependencies) chi.Router {
var itemsHandler *handlers.ItemsHandler
var catalogResourceHandler *handlers.CatalogResourceHandler
var catalogHandler *handlers.CatalogHandler
var literaryWorkHandler *handlers.LiteraryWorkHandler
var peopleHandler *handlers.PeopleHandler
var itemRepo *catalog.ItemRepository
var episodeRepo *catalog.EpisodeRepository
@@ -431,6 +435,7 @@ func NewRouter(deps Dependencies) chi.Router {
var seasonRepo *catalog.SeasonRepository
var detailSvc *catalog.DetailService
var calendarRepo *catalog.CalendarRepository
var catalogSearchService *catalog.CatalogSearchService
var webhookSyncHandler *handlers.WebhookSyncHandler
var requestHandler *handlers.RequestsHandler
var autoscanHandler *handlers.AutoscanHandler
@@ -444,6 +449,27 @@ func NewRouter(deps Dependencies) chi.Router {
ebookAnnotationStore = handlers.NewPGEbookReaderAnnotationStore(deps.DB)
browseRepo := catalog.NewBrowseRepository(deps.DB)
itemRepo = catalog.NewItemRepository(deps.DB)
catalogSearchSettings, err := catalog.LoadCatalogSearchSettings(context.Background(), settingsRepo)
if err != nil {
slog.Warn("catalog search: failed to load settings; using postgres", "err", err)
catalogSearchSettings = catalog.DefaultCatalogSearchSettings()
}
searchIndexEvents := catalog.NewSearchIndexEventRepository(deps.DB)
catalogSearchService = catalog.NewCatalogSearchServiceFromSettings(
catalogSearchSettings,
itemRepo,
searchIndexEvents,
deps.CatalogSearchVectorizer,
)
if catalogSearchService != nil {
catalogSearchService.StartCoverageRefresh(deps.AppContext)
}
activeSearchProvider := catalog.SearchProviderPostgres
if _, ok := catalogSearchService.Provider().(*catalog.MeilisearchSearchProvider); ok {
activeSearchProvider = catalog.SearchProviderMeilisearch
}
searchIndexEvents.WithActiveProvider(activeSearchProvider)
itemRepo.WithSearchIndexEvents(searchIndexEvents)
episodeRepo = catalog.NewEpisodeRepository(deps.DB)
providerIDRepo = catalog.NewProviderIDRepository(deps.DB)
calendarRepo = catalog.NewCalendarRepository(deps.DB)
@@ -463,10 +489,13 @@ func NewRouter(deps Dependencies) chi.Router {
rootClaimRepo := catalog.NewRootClaimRepository(deps.DB)
groupClaimRepo := catalog.NewGroupClaimRepository(deps.DB)
literaryRepo := literaryworks.NewRepository(deps.DB)
literaryWorkHandler = &handlers.LiteraryWorkHandler{Service: literaryworks.NewService(literaryRepo)}
detailSvc = catalog.NewDetailService(itemRepo, episodeRepo, seasonRepo, deps.PersonRepo, fileFetcher)
detailSvc.SetFolderRepository(folderRepo)
detailSvc.SetRootClaimRepository(rootClaimRepo)
detailSvc.SetGroupClaimRepository(groupClaimRepo)
detailSvc.SetWorkSummaryProvider(literaryRepo)
detailSvc.SetProbeEnsurer(deps.ProbeEnsurer)
detailSvc.SetChapterThumbnailQueuer(deps.ChapterThumbnailQueuer)
if deps.ImageResolver != nil {
@@ -484,6 +513,9 @@ func NewRouter(deps Dependencies) chi.Router {
detailSvc,
providerIDRepo,
)
if catalogSearchService != nil {
itemsHandler.SetCatalogSearchProvider(catalogSearchService.Provider())
}
itemsHandler.EventsHub = deps.EventsHub
itemsHandler.UserRepo = userRepo
if requester, ok := deps.MetadataService.(handlers.MetadataRefreshRequester); ok {
@@ -492,6 +524,9 @@ func NewRouter(deps Dependencies) chi.Router {
if dispatcher, ok := deps.WatchProviderService.(handlers.LocalWatchEventDispatcher); ok {
itemsHandler.SetLocalWatchEventDispatcher(dispatcher)
}
if deps.WatchCompletionObserver != nil {
itemsHandler.SetCompletionObserver(deps.WatchCompletionObserver)
}
if ebookProgressStore != nil {
itemsHandler.SetEbookReaderProgressStore(ebookProgressStore)
}
@@ -510,14 +545,19 @@ func NewRouter(deps Dependencies) chi.Router {
if ebookAnnotationStore != nil {
ebookReaderHandler.AnnotationStore = ebookAnnotationStore
}
if conv := buildEbookConversion(deps, settingsRepo); conv != nil {
ebookReaderHandler.Conversion = conv
}
}
catalogResourceHandler = handlers.NewCatalogResourceHandler(itemsHandler)
catalogHandler = handlers.NewCatalogHandler(
catalog.NewCatalogResolver(browseRepo, itemRepo).
WithEpisodeRepository(episodeRepo).
WithUserStoreProvider(deps.UserStoreProvider),
WithUserStoreProvider(deps.UserStoreProvider).
WithSearchProvider(catalogSearchService.Provider()),
itemsHandler,
)
catalogHandler.SetWorkSummaryProvider(literaryRepo)
tmdbAPIKey := ""
if deps.Config != nil {
@@ -602,8 +642,8 @@ func NewRouter(deps Dependencies) chi.Router {
personalDataHandler.SetEpisodeRepo(episodeRepo)
personalDataHandler.SetSeasonRepo(seasonRepo)
personalDataHandler.EventsHub = deps.EventsHub
if dispatcher, ok := deps.WatchProviderService.(handlers.LocalFavoriteEventDispatcher); ok {
personalDataHandler.SetLocalFavoriteEventDispatcher(dispatcher)
if dispatcher, ok := deps.WatchProviderService.(handlers.LocalListEventDispatcher); ok {
personalDataHandler.SetLocalListEventDispatcher(dispatcher)
}
progressHandler = handlers.NewProgressHandler(deps.UserStoreProvider)
progressHandler.EventsHub = deps.EventsHub
@@ -698,6 +738,7 @@ func NewRouter(deps Dependencies) chi.Router {
playbackHandler.StoreProvider = deps.UserStoreProvider
}
playbackHandler.StableIdentityResolver = watchstate.NewStableIdentityResolver(itemRepo, episodeRepo, providerIDRepo)
playbackHandler.CompletionObserver = deps.WatchCompletionObserver
if scrobbler, ok := deps.WatchProviderService.(handlers.PlaybackWatchScrobbler); ok {
playbackHandler.WatchScrobbler = scrobbler
}
@@ -822,6 +863,7 @@ func NewRouter(deps Dependencies) chi.Router {
adminHandler.BootstrapSensitiveConfigured = deps.BootstrapSensitiveConfigured
adminHandler.BootstrapSensitiveValues = deps.BootstrapSensitiveValues
adminHandler.RestartStatus = restartStatus
adminHandler.CatalogSearchStatus = catalogSearchService
if settingsRepo != nil {
adminHandler.SettingsRepo = settingsRepo
}
@@ -1688,10 +1730,14 @@ func NewRouter(deps Dependencies) chi.Router {
r.Get("/catalog/filters/search", catalogHandler.HandleGetCatalogFacetSearch)
r.Get("/catalog/audiobook-groups", catalogHandler.HandleGetAudiobookGroups)
r.Post("/catalog/query", catalogHandler.HandlePostCatalogQuery)
if literaryWorkHandler != nil {
r.Get("/works/{work_id}", literaryWorkHandler.HandleGetWork)
}
if catalogResourceHandler != nil {
r.Get("/catalog/items/{id}", catalogResourceHandler.HandleGetItemDetail)
r.Get("/catalog/items/{id}/episodes", catalogResourceHandler.HandleGetItemEpisodes)
r.Get("/catalog/items/{id}/versions", catalogResourceHandler.HandleGetItemVersions)
r.Get("/catalog/items/{id}/manga-files", catalogResourceHandler.HandleGetMangaFiles)
r.Get("/catalog/series/{id}/seasons", catalogResourceHandler.HandleGetSeasons)
r.Get("/catalog/series/{id}/seasons/{num}", catalogResourceHandler.HandleGetSeason)
r.Get("/catalog/series/{id}/seasons/{num}/episodes", catalogResourceHandler.HandleGetEpisodes)
@@ -1805,6 +1851,7 @@ func NewRouter(deps Dependencies) chi.Router {
r.Route("/collections", func(r chi.Router) {
r.Use(apimw.RequireProfile)
r.Get("/", collectionHandler.HandleListCollections)
r.Get("/capabilities", collectionHandler.HandleCapabilities)
if libraryCollectionHandler != nil {
// Aggregated server (admin-curated) collections across
// every accessible library. Separate from "/" (personal,
@@ -1983,6 +2030,7 @@ func NewRouter(deps Dependencies) chi.Router {
if ebookReaderHandler != nil {
r.Route("/ebooks", func(r chi.Router) {
r.Use(apimw.RequireProfile)
r.Get("/capability", ebookReaderHandler.HandleConversionCapability)
r.Get("/{content_id}/files/{file_id}/read", ebookReaderHandler.HandleReadFile)
r.Head("/{content_id}/files/{file_id}/read", ebookReaderHandler.HandleReadFile)
r.Get("/{content_id}/progress", ebookReaderHandler.HandleGetProgress)
@@ -2221,6 +2269,14 @@ func NewRouter(deps Dependencies) chi.Router {
r.Get("/unmatched", adminHandler.HandleListUnmatched)
r.Get("/stats", adminHandler.HandleGetStats)
r.Get("/server/status", adminHandler.HandleGetServerStatus)
r.Get("/catalog/search/status", adminHandler.HandleGetCatalogSearchStatus)
if literaryWorkHandler != nil {
r.Get("/literary-works/items/{content_id}/candidates", literaryWorkHandler.HandleListCandidates)
r.Post("/literary-works/link", literaryWorkHandler.HandleLinkItems)
r.Delete("/literary-works/{work_id}/items/{content_id}", literaryWorkHandler.HandleUnlinkItem)
r.Post("/literary-works/matches/confirm", literaryWorkHandler.HandleConfirmMatch)
r.Post("/literary-works/matches/ignore", literaryWorkHandler.HandleIgnoreMatch)
}
r.Post("/server/restart", serverControlHandler.HandleRestart)
r.Get("/jellyfin-compat/status", adminHandler.HandleGetJellyfinCompatStatus)
r.Patch("/jellyfin-compat/settings", adminHandler.HandleUpdateJellyfinCompatSettings)
@@ -2894,6 +2950,7 @@ func effectiveSubtitleAIConfig(cfg *config.Config) (subtitleai.Config, string) {
ASRModel: cfg.AI.ASRModel,
BatchSize: cfg.SubtitleAI.BatchSize,
ContextNeighbors: cfg.SubtitleAI.ContextNeighbors,
LiveASRChunkSeconds: cfg.SubtitleAI.LiveASRChunkSeconds,
TranscribeQuotaJobs: cfg.SubtitleAI.TranscribeQuotaJobs,
TranscribeQuotaPeriod: cfg.SubtitleAI.TranscribeQuotaPeriod,
}, disabledGateway
@@ -174,6 +174,22 @@ func (s *stubMediaStore) GetAudiobookByID(_ context.Context, id string, _ catalo
}
return nil, nil
}
func (s *stubMediaStore) GetAudiobooksByIDs(_ context.Context, ids []string, _ catalog.AccessFilter) (map[string]*models.MediaItem, error) {
if s.lookupErr != nil {
return nil, s.lookupErr
}
out := make(map[string]*models.MediaItem, len(ids))
for _, id := range ids {
if it, ok := s.known[id]; ok {
if it == nil {
out[id] = &models.MediaItem{ContentID: id}
} else {
out[id] = it
}
}
}
return out, nil
}
func (s *stubMediaStore) GetAuthorByID(_ context.Context, id string, _ catalog.AccessFilter) (Author, error) {
return Author{}, ErrNotFound
}
+3
View File
@@ -172,6 +172,9 @@ func (h *Handler) handlePublicTrack(w http.ResponseWriter, r *http.Request) {
http.Error(w, "session expired", http.StatusGone)
return
}
if h.beginNativePlaybackTransport(sid) {
defer h.endNativePlaybackTransport(sid)
}
access, err := h.accessFilterForAuth(r.Context(), ctxAuth{UserID: sess.UserID, ProfileID: sess.ProfileID})
if err != nil {
+11
View File
@@ -41,6 +41,9 @@ type AudiobookLibrary struct {
// a small adapter struct added in a later stage.
type MediaStore interface {
GetAudiobookByID(ctx context.Context, contentID string, access catalog.AccessFilter) (*models.MediaItem, error)
// GetAudiobooksByIDs batch-fetches audiobooks by content_id (people + series
// hydrated once), keyed by content_id, for list/shelf handlers.
GetAudiobooksByIDs(ctx context.Context, contentIDs []string, access catalog.AccessFilter) (map[string]*models.MediaItem, error)
// ListAudiobooks returns a page of audiobooks. When libraryID is non-zero
// it filters to items in that media_folder; 0 means all audiobook items.
ListAudiobooks(ctx context.Context, libraryID int64, limit, offset int, access catalog.AccessFilter) ([]*models.MediaItem, int, error)
@@ -255,6 +258,14 @@ type Dependencies struct {
// SocketIO is the Socket.io server mounted at /abs/socket.io/. May be nil;
// the route is only registered when a non-nil value is supplied.
SocketIO SocketIOServer
// NativeSessions mirrors ABS playback into Silo's native playback session
// manager so shared live-session views, limits, and stale-session cleanup
// see Audiobookshelf-compatible clients. May be nil; ABS playback still
// functions, but admin live-session visibility is unavailable.
NativeSessions PlaybackSessionManager
// NativeSessionSyncer flushes native session-manager state into the shared
// admin live-session table after ABS play/sync/close events.
NativeSessionSyncer PlaybackSessionSyncer
// CoverResolver translates a raw silo poster path (e.g.
// "local/audiobooks/.../original.webp") into a fully-qualified URL
// the ABS client can fetch. Optional; when nil, /api/items/{id}/cover
+21 -5
View File
@@ -108,10 +108,15 @@ func (h *Handler) handleSimilarItems(w http.ResponseWriter, r *http.Request) {
}
lib := h.resolveDefaultLibrary(r.Context(), access)
baseURL := h.absBaseURL(r)
byID, err := h.deps.MediaStore.GetAudiobooksByIDs(r.Context(), ids, access)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
out := make([]LibraryItem, 0, len(ids))
for _, id := range ids {
si, err := h.deps.MediaStore.GetAudiobookByID(r.Context(), id, access)
if err != nil || si == nil {
for _, id := range ids { // preserve recommender order
si := byID[id]
if si == nil {
continue
}
out = append(out, siloItemToLibraryItem(si, lib, baseURL))
@@ -149,13 +154,24 @@ func (h *Handler) handleItemsInProgress(w http.ResponseWriter, r *http.Request)
}
lib := h.resolveDefaultLibrary(r.Context(), access)
baseURL := h.absBaseURL(r)
ids := make([]string, 0, len(rows))
for _, p := range rows {
if !p.IsFinished && p.CurrentSeconds > 0 {
ids = append(ids, p.ContentID)
}
}
byID, err := h.deps.MediaStore.GetAudiobooksByIDs(r.Context(), ids, access)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
items := make([]any, 0, len(rows))
for _, p := range rows {
if p.IsFinished || p.CurrentSeconds <= 0 {
continue
}
si, err := h.deps.MediaStore.GetAudiobookByID(r.Context(), p.ContentID, access)
if err != nil || si == nil {
si := byID[p.ContentID]
if si == nil {
continue
}
li := siloItemToLibraryItem(si, lib, baseURL)
@@ -23,6 +23,9 @@ type noopMediaStore struct{}
func (noopMediaStore) GetAudiobookByID(context.Context, string, catalog.AccessFilter) (*models.MediaItem, error) {
return nil, nil
}
func (noopMediaStore) GetAudiobooksByIDs(context.Context, []string, catalog.AccessFilter) (map[string]*models.MediaItem, error) {
return map[string]*models.MediaItem{}, nil
}
func (noopMediaStore) ListAudiobooks(context.Context, int64, int, int, catalog.AccessFilter) ([]*models.MediaItem, int, error) {
return nil, 0, nil
}
+226
View File
@@ -0,0 +1,226 @@
package abs
import (
"context"
"errors"
"fmt"
"log/slog"
"net"
"net/http"
"strconv"
"strings"
"github.com/Silo-Server/silo-server/internal/clientip"
"github.com/Silo-Server/silo-server/internal/models"
"github.com/Silo-Server/silo-server/internal/playback"
)
// PlaybackSessionManager is the native Silo playback-session surface ABS needs
// to make compatibility playback visible to live admin/session monitoring.
type PlaybackSessionManager interface {
StartSessionWithFilesContext(ctx context.Context, userID int, profileID string, effectiveFileID int, requestedFileID int, method playback.PlayMethod, transcodeAudio bool) (*playback.Session, error)
UpdateProgress(sessionID string, position float64, isPaused bool) error
UpdateStreamState(sessionID string, state playback.SessionStreamState) error
BeginTransport(sessionID string) error
EndTransport(sessionID string) error
SetProgressPersistenceDisabled(sessionID string, disabled bool) error
StopSession(sessionID string) error
}
// PlaybackSessionSyncer flushes the local native-session snapshot into the
// shared admin live-session table.
type PlaybackSessionSyncer interface {
SyncNow(ctx context.Context) error
}
func (h *Handler) startNativePlaybackSession(
r *http.Request,
a ctxAuth,
files []*models.MediaFile,
startPosition float64,
) (*playback.Session, error) {
if h == nil || h.deps.NativeSessions == nil || len(files) == 0 {
return nil, nil
}
userID, err := strconv.Atoi(a.UserID)
if err != nil {
return nil, fmt.Errorf("invalid ABS user id %q: %w", a.UserID, err)
}
file := files[0]
sessionCtx := playback.WithClientInfo(r.Context(), absPlaybackClientInfoFromRequest(r))
session, err := h.deps.NativeSessions.StartSessionWithFilesContext(
sessionCtx,
userID,
a.ProfileID,
file.ID,
file.ID,
playback.PlayDirect,
false,
)
if err != nil {
return nil, err
}
if err := h.deps.NativeSessions.SetProgressPersistenceDisabled(session.ID, true); err != nil {
slog.Warn("abs play: disable native progress persistence failed",
"session_id", session.ID, "error", err)
} else {
session.DisableProgressPersistence = true
}
if startPosition > 0 {
if err := h.deps.NativeSessions.UpdateProgress(session.ID, startPosition, false); err != nil {
slog.Warn("abs play: seed native session progress failed",
"session_id", session.ID, "position", startPosition, "error", err)
} else {
session.Position = startPosition
session.IsPaused = false
}
}
streamBitrateKbps := 0
if file.Bitrate > 0 {
streamBitrateKbps = file.Bitrate
}
if err := h.deps.NativeSessions.UpdateStreamState(session.ID, playback.SessionStreamState{
PlayMethod: playback.PlayDirect,
BasePlayMethod: playback.PlayDirect,
ClientIP: requestClientIP(r),
ClientName: session.ClientName,
ClientVersion: session.ClientVersion,
ClientUserAgent: session.ClientUserAgent,
StreamBitrateKbps: streamBitrateKbps,
}); err != nil {
slog.Warn("abs play: update native session stream state failed",
"session_id", session.ID, "error", err)
} else {
session.PlayMethod = playback.PlayDirect
session.BasePlayMethod = playback.PlayDirect
session.ClientIP = requestClientIP(r)
session.StreamBitrateKbps = streamBitrateKbps
}
h.syncNativeSessionsNow(r.Context(), "abs_start")
return session, nil
}
func (h *Handler) updateNativePlaybackProgress(ctx context.Context, sessionID string, position float64) {
if h == nil || h.deps.NativeSessions == nil || sessionID == "" {
return
}
if err := h.deps.NativeSessions.UpdateProgress(sessionID, position, false); err != nil {
if !errors.Is(err, playback.ErrSessionNotFound) {
slog.Warn("abs session sync: update native session progress failed",
"session_id", sessionID, "position", position, "error", err)
}
return
}
h.syncNativeSessionsNow(ctx, "abs_progress")
}
func (h *Handler) stopNativePlaybackSession(ctx context.Context, sessionID string) {
if h == nil || h.deps.NativeSessions == nil || sessionID == "" {
return
}
if err := h.deps.NativeSessions.StopSession(sessionID); err != nil {
if !errors.Is(err, playback.ErrSessionNotFound) {
slog.Warn("abs session close: stop native session failed",
"session_id", sessionID, "error", err)
}
return
}
h.syncNativeSessionsNow(ctx, "abs_close")
}
func (h *Handler) beginNativePlaybackTransport(sessionID string) bool {
if h == nil || h.deps.NativeSessions == nil || sessionID == "" {
return false
}
if err := h.deps.NativeSessions.BeginTransport(sessionID); err != nil {
if !errors.Is(err, playback.ErrSessionNotFound) {
slog.Warn("abs public track: begin native transport failed",
"session_id", sessionID, "error", err)
}
return false
}
return true
}
func (h *Handler) endNativePlaybackTransport(sessionID string) {
if h == nil || h.deps.NativeSessions == nil || sessionID == "" {
return
}
if err := h.deps.NativeSessions.EndTransport(sessionID); err != nil && !errors.Is(err, playback.ErrSessionNotFound) {
slog.Warn("abs public track: end native transport failed",
"session_id", sessionID, "error", err)
}
}
func (h *Handler) syncNativeSessionsNow(ctx context.Context, reason string) {
if h == nil || h.deps.NativeSessionSyncer == nil {
return
}
if err := h.deps.NativeSessionSyncer.SyncNow(ctx); err != nil {
slog.Error("abs: failed to sync native playback sessions", "reason", reason, "error", err)
}
}
func writeNativePlaybackStartError(w http.ResponseWriter, err error) {
switch {
case errors.Is(err, playback.ErrTooManyStreams):
http.Error(w, "too many concurrent streams", http.StatusTooManyRequests)
case errors.Is(err, playback.ErrTooManyTranscodes):
http.Error(w, "too many concurrent transcodes", http.StatusTooManyRequests)
default:
slog.Error("abs play: start native playback session failed", "error", err)
http.Error(w, "failed to start playback session", http.StatusInternalServerError)
}
}
func absPlaybackClientInfoFromRequest(r *http.Request) playback.ClientInfo {
if r == nil {
return playback.ClientInfo{Name: "Audiobookshelf"}
}
name := firstHeaderValue(r,
"X-Silo-Client",
"X-Client-Name",
"X-Device-Name",
"X-Emby-Client",
)
if name == "" {
name = "Audiobookshelf"
}
return playback.ClientInfo{
Name: name,
Version: firstHeaderValue(r,
"X-Silo-Client-Version",
"X-Client-Version",
"X-Emby-Client-Version",
),
UserAgent: r.UserAgent(),
}
}
func firstHeaderValue(r *http.Request, names ...string) string {
for _, name := range names {
if value := strings.TrimSpace(r.Header.Get(name)); value != "" {
return value
}
}
return ""
}
func requestClientIP(r *http.Request) string {
if r == nil {
return ""
}
if ip := clientip.FromContext(r.Context()); ip != "" {
return ip
}
host, _, err := net.SplitHostPort(r.RemoteAddr)
if err == nil {
return host
}
return strings.TrimSpace(r.RemoteAddr)
}
@@ -0,0 +1,229 @@
package abs
import (
"context"
"encoding/json"
"net/http"
"strings"
"testing"
"time"
"github.com/Silo-Server/silo-server/internal/catalog"
"github.com/Silo-Server/silo-server/internal/models"
"github.com/Silo-Server/silo-server/internal/playback"
)
type playStartMediaStore struct {
noopMediaStore
item *models.MediaItem
files []*models.MediaFile
}
func (s *playStartMediaStore) GetAudiobookByID(_ context.Context, id string, _ catalog.AccessFilter) (*models.MediaItem, error) {
if s.item != nil && s.item.ContentID == id {
return s.item, nil
}
return nil, nil
}
func (s *playStartMediaStore) GetMediaFiles(_ context.Context, contentID string, _ catalog.AccessFilter) ([]*models.MediaFile, error) {
if s.item != nil && s.item.ContentID == contentID {
return s.files, nil
}
return nil, nil
}
type recordingPlaybackSessionSyncer struct {
calls int
}
func (s *recordingPlaybackSessionSyncer) SyncNow(context.Context) error {
s.calls++
return nil
}
func TestHandlePlayStartCreatesNativePlaybackSession(t *testing.T) {
now := time.Now()
media := &playStartMediaStore{
item: &models.MediaItem{
ContentID: "book-1",
Type: "audiobook",
Title: "Native Session Book",
UpdatedAt: now,
AddedAt: &now,
},
files: []*models.MediaFile{{
ID: 42,
ContentID: "book-1",
FilePath: "/tmp/book.mp3",
FileSize: 1024,
Duration: 3600,
Bitrate: 128,
CodecAudio: "mp3",
}},
}
absSessions := &fakePlaybackSessionStore{}
nativeSessions := playback.NewSessionManager(0, 0)
syncer := &recordingPlaybackSessionSyncer{}
progress := &fakeProgressStore{row: &ProgressRow{
UserID: "1",
ProfileID: "profile-1",
ContentID: "book-1",
CurrentSeconds: 123.5,
DurationSeconds: 3600,
UpdatedAt: now,
}}
h := New(Dependencies{
MediaStore: media,
ProgressStore: progress,
PlaybackSessionStore: absSessions,
NativeSessions: nativeSessions,
NativeSessionSyncer: syncer,
})
rec := dispatchABSWithParams(
http.MethodPost,
"/api/items/book-1/play",
map[string]string{"libraryItemId": "book-1"},
nil,
"1",
"profile-1",
h.handlePlayStart,
)
if rec.Code != http.StatusOK {
t.Fatalf("status = %d, want 200; body=%s", rec.Code, rec.Body.String())
}
var body map[string]any
if err := json.Unmarshal(rec.Body.Bytes(), &body); err != nil {
t.Fatalf("decode response: %v", err)
}
sessionID, _ := body["id"].(string)
if sessionID == "" {
t.Fatalf("response id is empty: %#v", body["id"])
}
native, err := nativeSessions.GetSession(sessionID)
if err != nil {
t.Fatalf("native session %q missing: %v", sessionID, err)
}
if native.MediaFileID != 42 || native.RequestedMediaFileID != 42 {
t.Fatalf("native file ids = (%d, %d), want (42, 42)", native.MediaFileID, native.RequestedMediaFileID)
}
if !native.DisableProgressPersistence {
t.Fatalf("native session should disable progress persistence")
}
if native.Position != 123.5 {
t.Fatalf("native position = %v, want 123.5", native.Position)
}
if syncer.calls == 0 {
t.Fatalf("native session syncer was not called")
}
absSession, err := absSessions.GetPlaybackSession(context.Background(), sessionID)
if err != nil {
t.Fatalf("ABS session %q missing: %v", sessionID, err)
}
if absSession.CurrentPositionSeconds != 123.5 {
t.Fatalf("ABS session position = %v, want 123.5", absSession.CurrentPositionSeconds)
}
tracks, _ := body["audioTracks"].([]any)
if len(tracks) != 1 {
t.Fatalf("audioTracks length = %d, want 1", len(tracks))
}
track, _ := tracks[0].(map[string]any)
if got, _ := track["contentUrl"].(string); got == "" || !strings.Contains(got, "/abs/public/session/"+sessionID+"/track/1") {
t.Fatalf("contentUrl = %q, want session-scoped URL", got)
}
}
func TestHandleSessionSyncUpdatesNativePlaybackSession(t *testing.T) {
media := &playStartMediaStore{
item: &models.MediaItem{ContentID: "book-1", Type: "audiobook", Title: "Book", UpdatedAt: time.Now()},
}
absSessions := &fakePlaybackSessionStore{}
nativeSessions := playback.NewSessionManager(0, 0)
native, err := nativeSessions.StartSessionWithFilesContext(context.Background(), 1, "profile-1", 42, 42, playback.PlayDirect, false)
if err != nil {
t.Fatalf("start native session: %v", err)
}
_ = absSessions.InsertPlaybackSession(context.Background(), ABSPlaybackSession{
ID: native.ID,
UserID: "1",
ProfileID: "profile-1",
ContentID: "book-1",
})
syncer := &recordingPlaybackSessionSyncer{}
h := New(Dependencies{
MediaStore: media,
ProgressStore: &fakeProgressStore{},
PlaybackSessionStore: absSessions,
NativeSessions: nativeSessions,
NativeSessionSyncer: syncer,
})
rec := dispatchABSWithParams(
http.MethodPatch,
"/api/session/"+native.ID,
map[string]string{"sid": native.ID},
[]byte(`{"currentTime":55.25,"timeListening":10}`),
"1",
"profile-1",
h.handleSessionSync,
)
if rec.Code != http.StatusOK {
t.Fatalf("status = %d, want 200; body=%s", rec.Code, rec.Body.String())
}
updated, err := nativeSessions.GetSession(native.ID)
if err != nil {
t.Fatalf("native session missing: %v", err)
}
if updated.Position != 55.25 {
t.Fatalf("native position = %v, want 55.25", updated.Position)
}
if updated.IsPaused {
t.Fatalf("native session should be marked playing")
}
if syncer.calls == 0 {
t.Fatalf("native session syncer was not called")
}
}
func TestHandleSessionCloseStopsNativePlaybackSession(t *testing.T) {
absSessions := &fakePlaybackSessionStore{}
nativeSessions := playback.NewSessionManager(0, 0)
native, err := nativeSessions.StartSessionWithFilesContext(context.Background(), 1, "profile-1", 42, 42, playback.PlayDirect, false)
if err != nil {
t.Fatalf("start native session: %v", err)
}
_ = absSessions.InsertPlaybackSession(context.Background(), ABSPlaybackSession{
ID: native.ID,
UserID: "1",
ProfileID: "profile-1",
ContentID: "book-1",
})
syncer := &recordingPlaybackSessionSyncer{}
h := New(Dependencies{
MediaStore: noopMediaStore{},
PlaybackSessionStore: absSessions,
NativeSessions: nativeSessions,
NativeSessionSyncer: syncer,
})
rec := dispatchABSWithParams(
http.MethodPost,
"/api/session/"+native.ID+"/close",
map[string]string{"sid": native.ID},
nil,
"1",
"profile-1",
h.handleSessionClose,
)
if rec.Code != http.StatusNoContent {
t.Fatalf("status = %d, want 204; body=%s", rec.Code, rec.Body.String())
}
if _, err := nativeSessions.GetSession(native.ID); err == nil {
t.Fatalf("native session still exists after close")
}
if syncer.calls == 0 {
t.Fatalf("native session syncer was not called")
}
}
+28 -21
View File
@@ -59,21 +59,45 @@ func (h *Handler) handlePlayStart(w http.ResponseWriter, r *http.Request) {
return
}
// currentTime seeds the audio element's initial position so cross-device
// resume works. Lookup is best-effort: any error returns position 0,
// which is always correct for a first listen.
//
// Note: we deliberately do NOT emit a "progress" (0.0-1.0) field on the
// play session response. The canonical continuum-plugin handler omits it
// too — "progress" belongs to /me/progress responses, not playbackSession.
// The spec's Phase 0 row mentioning "currentTime AND progress fields" was
// over-specified; matching the canonical wire shape is the load-bearing
// requirement.
var currentTime float64
currentTime, err = resolveResumeTime(r.Context(), h.deps.ProgressStore, a.UserID, a.ProfileID, contentID)
if err != nil {
slog.Debug("play: progress lookup failed", "user", a.UserID, "item", contentID, "err", err)
// currentTime is already 0 on error path; safe to continue.
}
// The {ino} parameter used by handleFileStream is a 0-based file index, but
// we want iOS clients to resolve it via the stable MD5 derivation. Emit inos
// that handleFileStream can reverse without a database lookup.
baseURL := h.absBaseURL(r)
sessionID := ulid.Make().String()
if nativeSession, err := h.startNativePlaybackSession(r, a, files, currentTime); err != nil {
writeNativePlaybackStartError(w, err)
return
} else if nativeSession != nil {
sessionID = nativeSession.ID
}
// Persist the session row so subsequent PATCH /session/{sid} heartbeats
// and POST /session/{sid}/close can find it. Without this, the session
// ID is returned to the client but every sync/close lookup 404s.
if h.deps.PlaybackSessionStore != nil {
sess := ABSPlaybackSession{
ID: sessionID,
UserID: a.UserID,
ProfileID: a.ProfileID,
ContentID: contentID,
ID: sessionID,
UserID: a.UserID,
ProfileID: a.ProfileID,
ContentID: contentID,
CurrentPositionSeconds: currentTime,
}
if len(files) > 0 {
fid := files[0].ID
@@ -111,23 +135,6 @@ func (h *Handler) handlePlayStart(w http.ResponseWriter, r *http.Request) {
dateStr := now.UTC().Format("2006-01-02")
dayOfWeek := now.UTC().Weekday().String()
// currentTime seeds the audio element's initial position so cross-device
// resume works. Lookup is best-effort: any error returns position 0,
// which is always correct for a first listen.
//
// Note: we deliberately do NOT emit a "progress" (0.0-1.0) field on the
// play session response. The canonical continuum-plugin handler omits it
// too — "progress" belongs to /me/progress responses, not playbackSession.
// The spec's Phase 0 row mentioning "currentTime AND progress fields" was
// over-specified; matching the canonical wire shape is the load-bearing
// requirement.
var currentTime float64
currentTime, err = resolveResumeTime(r.Context(), h.deps.ProgressStore, a.UserID, a.ProfileID, contentID)
if err != nil {
slog.Debug("play: progress lookup failed", "user", a.UserID, "item", contentID, "err", err)
// currentTime is already 0 on error path; safe to continue.
}
playbackSession := map[string]any{
"id": sessionID,
"userId": a.UserID,
+20 -4
View File
@@ -3,6 +3,7 @@ package abs
import (
"context"
"encoding/json"
"log/slog"
"net/http"
"time"
@@ -139,10 +140,20 @@ func (h *Handler) handleGetMyProgress(w http.ResponseWriter, r *http.Request) {
http.Error(w, "resolve access: "+err.Error(), http.StatusForbidden)
return
}
ids := make([]string, 0, len(rows))
for _, p := range rows {
ids = append(ids, p.ContentID)
}
// One batch fetch acts as the access/existence gate (the item itself isn't
// rendered here), instead of one query per progress row.
byID, err := h.deps.MediaStore.GetAudiobooksByIDs(r.Context(), ids, access)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
out := make([]map[string]any, 0, len(rows))
for _, p := range rows {
item, err := h.deps.MediaStore.GetAudiobookByID(r.Context(), p.ContentID, access)
if err != nil || item == nil {
if byID[p.ContentID] == nil {
continue
}
out = append(out, progressRowToABS(p))
@@ -358,10 +369,14 @@ func (h *Handler) handleSessionSync(w http.ResponseWriter, r *http.Request) {
// Update position in user_watch_progress. Must NOT be a full upsert —
// see comment in handleSetItemProgress re: not overwriting is_finished.
if h.deps.ProgressStore != nil {
_ = h.deps.ProgressStore.UpdateProgressPosition(
if err := h.deps.ProgressStore.UpdateProgressPosition(
r.Context(), a.UserID, a.ProfileID, sess.ContentID, p.CurrentTime,
)
); err != nil {
slog.Warn("abs session sync: update progress position failed",
"session_id", sid, "content_id", sess.ContentID, "error", err)
}
}
h.updateNativePlaybackProgress(r.Context(), sid, p.CurrentTime)
// Realtime push to other connected clients.
h.publish(a.UserID, "user_item_progress_updated", map[string]any{
@@ -407,6 +422,7 @@ func (h *Handler) handleSessionClose(w http.ResponseWriter, r *http.Request) {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
h.stopNativePlaybackSession(r.Context(), sid)
h.publish(a.UserID, "user_session_closed", map[string]any{
"id": sid,
+11 -6
View File
@@ -172,10 +172,14 @@ func (s *ABSProgressStore) UpsertProgress(ctx context.Context, row abs.ProgressR
return nil
}
// UpdateProgressPosition updates only the position_seconds column for an
// existing row. If no row exists this is a no-op (the session-sync path
// that calls this only needs to move the cursor, not create a progress row
// for the first time; that's done when the user explicitly sets progress).
// UpdateProgressPosition advances the resume cursor from a session-sync tick.
// It is monotonic and finish-preserving: position only moves forward
// (GREATEST) and a completed row is never moved or un-finished. Without the
// GREATEST guard an out-of-order tick or a second device rewound the saved
// position. It deliberately stays UPDATE-only (no row created when none
// exists): the row is created by the explicit progress-report path, so a stray
// sync tick can't resurrect progress the user just cleared, nor create a
// zero-duration row.
func (s *ABSProgressStore) UpdateProgressPosition(ctx context.Context, userID, profileID, contentID string, positionSeconds float64) error {
uid, err := strconv.Atoi(userID)
if err != nil {
@@ -183,9 +187,10 @@ func (s *ABSProgressStore) UpdateProgressPosition(ctx context.Context, userID, p
}
_, err = s.Pool.Exec(ctx, `
UPDATE user_watch_progress
SET position_seconds = $4,
SET position_seconds = GREATEST(position_seconds, $4),
updated_at = now()
WHERE user_id = $1 AND profile_id = $2 AND media_item_id = $3`,
WHERE user_id = $1 AND profile_id = $2 AND media_item_id = $3
AND NOT completed`,
uid, profileID, contentID, positionSeconds,
)
if err != nil {
+20 -1
View File
@@ -7,20 +7,31 @@ import (
"errors"
"fmt"
"strconv"
"sync"
"time"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/Silo-Server/silo-server/internal/audiobooks/abs"
"github.com/Silo-Server/silo-server/internal/cache"
)
// absTokenTouchThrottle bounds how often a token's last_seen_at is refreshed.
// TouchToken fires on every authenticated /abs/* request and updates a single
// per-token row; without throttling a request burst serialized on that row's
// lock (the same hot-row pattern as device last_seen).
const absTokenTouchThrottle = 5 * time.Minute
// ABSSessionStore implements abs.TokenStore on the abs_sessions table
// (migration 147). Each JTI is stored as a SHA-256 `token_hash`; revocation
// sets `revoked_at`. The table stores integer user_id, so we parse the
// string UserID from ABSToken back to int.
type ABSSessionStore struct {
Pool *pgxpool.Pool
touchOnce sync.Once
touched *cache.TTLCache[struct{}]
}
// InsertToken inserts a newly minted JTI into abs_sessions.
@@ -143,9 +154,17 @@ func (s *ABSSessionStore) RevokeTokensForPrincipal(ctx context.Context, userID,
// TouchToken bumps last_seen_at for active-session bookkeeping.
// Errors are logged by the caller; we never gate a valid request on this.
func (s *ABSSessionStore) TouchToken(ctx context.Context, jti string) error {
hash := absTokenHash(jti)
s.touchOnce.Do(func() { s.touched = cache.NewTTLCache[struct{}]() })
if _, seen := s.touched.Get(hash); seen {
// last_seen_at was refreshed within the throttle window; skip the
// per-request hot-row UPDATE.
return nil
}
s.touched.Set(hash, struct{}{}, absTokenTouchThrottle)
_, err := s.Pool.Exec(ctx, `
UPDATE abs_sessions SET last_seen_at = now()
WHERE token_hash = $1`, absTokenHash(jti))
WHERE token_hash = $1`, hash)
if err != nil {
return fmt.Errorf("abs_session_store: touch token: %w", err)
}
+96 -12
View File
@@ -93,16 +93,22 @@ type enrichmentItemRow struct {
// Enricher drives the audiobook metadata enrichment sweep.
type Enricher struct {
pool *pgxpool.Pool
chainRepo *metadata.ChainRepository
resolver *metadata.PluginResolverAdapter
itemRepo *catalog.ItemRepository
personRepo *catalog.PersonRepository
providerIDs *catalog.ProviderIDRepository
imageCacher audiobookCoverCacher
ffmpegPath string
batchSize int
workers int
pool *pgxpool.Pool
chainRepo *metadata.ChainRepository
resolver *metadata.PluginResolverAdapter
itemRepo *catalog.ItemRepository
personRepo *catalog.PersonRepository
providerIDs *catalog.ProviderIDRepository
imageCacher audiobookCoverCacher
imageCacheJobs metadata.ImageCacheJobEnqueuer
workLinker literaryWorkLinker
ffmpegPath string
batchSize int
workers int
}
type literaryWorkLinker interface {
AutoLinkContent(ctx context.Context, contentID string) (workID string, linked bool, err error)
}
// NewEnricher constructs an Enricher. All fields are required; a nil pool or
@@ -139,6 +145,20 @@ func (e *Enricher) SetImageCacher(cacher audiobookCoverCacher) {
e.imageCacher = cacher
}
func (e *Enricher) SetImageCacheJobEnqueuer(enqueuer metadata.ImageCacheJobEnqueuer) {
if e == nil {
return
}
e.imageCacheJobs = enqueuer
}
func (e *Enricher) SetLiteraryWorkLinker(linker literaryWorkLinker) {
if e == nil {
return
}
e.workLinker = linker
}
// SetFFmpegPath installs the ffmpeg binary path used by the deferred
// local-cover fallback. Empty string disables the fallback.
func (e *Enricher) SetFFmpegPath(path string) {
@@ -456,12 +476,12 @@ func (e *Enricher) enrichItem(ctx context.Context, item enrichmentItemRow) error
return e.stampLastRefreshed(ctx, item.ContentID)
}
e.cacheRemotePoster(ctx, item.ContentID, accumulator)
// Phase 3: Persist.
if err := e.persist(ctx, item.ContentID, accumulatedIDs, accumulator); err != nil {
return fmt.Errorf("persisting enrichment for %s: %w", item.ContentID, err)
}
e.enqueueRemoteArtwork(ctx, item.ContentID, accumulator)
e.autoLinkLiteraryWork(ctx, item.ContentID)
slog.Info("audiobook enrichment: enriched",
"content_id", item.ContentID,
@@ -474,6 +494,20 @@ func (e *Enricher) enrichItem(ctx context.Context, item enrichmentItemRow) error
return nil
}
func (e *Enricher) autoLinkLiteraryWork(ctx context.Context, contentID string) {
if e == nil || e.workLinker == nil || strings.TrimSpace(contentID) == "" {
return
}
workID, linked, err := e.workLinker.AutoLinkContent(ctx, contentID)
if err != nil {
slog.Warn("audiobook enrichment: literary work auto-link failed", "content_id", contentID, "error", err)
return
}
if linked {
slog.Info("audiobook enrichment: literary work auto-linked", "content_id", contentID, "work_id", workID)
}
}
// maybeApplyCoverFallback wraps applyLocalCoverFallback with the nil-guards
// and warn-logging, suitable for `defer` at the top of enrichItem so every
// exit path (success and early returns alike) gets a fallback attempt.
@@ -550,18 +584,27 @@ func (e *Enricher) persist(ctx context.Context, contentID string, providerIDs ma
if result.PosterPath != "" {
upd.PosterPath = &result.PosterPath
if isRemoteHTTPImage(result.PosterPath) {
upd.PosterSourcePath = &result.PosterPath
}
}
if result.PosterThumbhash != "" {
upd.PosterThumbhash = &result.PosterThumbhash
}
if result.BackdropPath != "" {
upd.BackdropPath = &result.BackdropPath
if isRemoteHTTPImage(result.BackdropPath) {
upd.BackdropSourcePath = &result.BackdropPath
}
}
if result.BackdropThumbhash != "" {
upd.BackdropThumbhash = &result.BackdropThumbhash
}
if result.LogoPath != "" {
upd.LogoPath = &result.LogoPath
if isRemoteHTTPImage(result.LogoPath) {
upd.LogoSourcePath = &result.LogoPath
}
}
if result.Overview != "" {
upd.Overview = &result.Overview
@@ -620,6 +663,47 @@ func (e *Enricher) persist(ctx context.Context, contentID string, providerIDs ma
return nil
}
func (e *Enricher) enqueueRemoteArtwork(ctx context.Context, contentID string, result *metadata.MetadataResult) {
if e == nil || e.imageCacheJobs == nil || result == nil || contentID == "" {
return
}
inputs := make([]metadata.EnqueueImageCacheJobInput, 0, 3)
add := func(sourcePath string, imageType metadata.ImageType) {
if !isRemoteHTTPImage(sourcePath) {
return
}
inputs = append(inputs, metadata.EnqueueImageCacheJobInput{
TargetType: metadata.ImageCacheTargetItem,
TargetContentID: contentID,
SeriesID: contentID,
SourcePath: sourcePath,
ProviderID: audiobookMetadataImageProviderID,
ProviderContentID: contentID,
ContentType: "audiobooks",
ImageType: metadata.ImageTypeToString(imageType),
})
}
add(result.PosterPath, metadata.ImagePoster)
add(result.BackdropPath, metadata.ImageBackdrop)
add(result.LogoPath, metadata.ImageLogo)
if len(inputs) == 0 {
return
}
enqueueCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)
defer cancel()
if _, err := e.imageCacheJobs.EnqueueBatch(enqueueCtx, inputs); err != nil {
slog.Warn("audiobook enrichment: failed to enqueue image cache jobs",
"content_id", contentID,
"count", len(inputs),
"error", err,
)
}
}
func isRemoteHTTPImage(path string) bool {
return strings.HasPrefix(path, "http://") || strings.HasPrefix(path, "https://")
}
// updateMetadataAndTimestamps runs UpdateMetadata and also stamps
// last_refreshed and matched_at so the item is skipped on future sweeps.
func (e *Enricher) updateMetadataAndTimestamps(ctx context.Context, contentID string, upd *catalog.MetadataUpdate) error {
+36
View File
@@ -54,6 +54,42 @@ func (s *ABSMediaStore) GetAudiobookByID(ctx context.Context, contentID string,
return item, nil
}
// GetAudiobooksByIDs batch-fetches audiobooks by content_id, hydrating people
// and series once for the whole set. List/shelf handlers (continue-listening,
// similar, my-progress) previously called GetAudiobookByID per row, issuing a
// handful of queries per item — up to ~500 single fetches on app open. Returns
// a map keyed by content_id; missing or non-audiobook ids are omitted.
func (s *ABSMediaStore) GetAudiobooksByIDs(ctx context.Context, contentIDs []string, access catalog.AccessFilter) (map[string]*models.MediaItem, error) {
if len(contentIDs) == 0 {
return map[string]*models.MediaItem{}, nil
}
items, err := s.Items.GetByIDsWithAccess(ctx, contentIDs, access)
if err != nil {
return nil, fmt.Errorf("abs_media_store: get audiobooks: %w", err)
}
books := make([]*models.MediaItem, 0, len(items))
for _, item := range items {
if item != nil && item.Type == "audiobook" {
books = append(books, item)
}
}
if len(books) == 0 {
return map[string]*models.MediaItem{}, nil
}
// Hydrate the whole set in two queries rather than per item.
if err := s.hydratePeople(ctx, books); err != nil {
_ = err
}
if err := s.hydrateAudiobookSeries(ctx, books); err != nil {
_ = err
}
out := make(map[string]*models.MediaItem, len(books))
for _, b := range books {
out[b.ContentID] = b
}
return out, nil
}
// ListAudiobooks returns a page of media_items with type='audiobook'.
// When libraryID is non-zero, results are filtered to items in that
// media_folder (via the media_item_libraries junction); 0 means all libraries.
+6 -1
View File
@@ -7,6 +7,7 @@ import (
"github.com/Silo-Server/silo-server/internal/audiobooks/abs"
"github.com/Silo-Server/silo-server/internal/audiobooks/abssocket"
"github.com/Silo-Server/silo-server/internal/catalog"
"github.com/Silo-Server/silo-server/internal/playback"
"github.com/Silo-Server/silo-server/internal/recommendations"
"github.com/Silo-Server/silo-server/internal/scanner"
@@ -54,7 +55,9 @@ type ABSHandlerDeps struct {
// Detail resolves audiobook poster S3 paths into fully-qualified URLs
// that ABS clients can fetch. Optional; when nil, /api/items/{id}/cover
// 404s rather than redirecting to an unreachable storage path.
Detail *catalog.DetailService
Detail *catalog.DetailService
SessionMgr *playback.SessionManager
SessionSyncer abs.PlaybackSessionSyncer
}
// absAuthAdapter is the narrow slice of internal/auth that BuildABSHandler
@@ -158,6 +161,8 @@ func (s *Service) BuildABSHandler(deps ABSHandlerDeps) *abs.Handler {
SmartCollectionStore: smartCollectionStore,
RSSFeedStore: rssFeedStore,
SocketIO: socketServer,
NativeSessions: deps.SessionMgr,
NativeSessionSyncer: deps.SessionSyncer,
CoverResolver: func(ctx context.Context, path, variant string) string {
if deps.Detail == nil {
return ""
+70 -7
View File
@@ -10,7 +10,14 @@ import (
"github.com/Silo-Server/silo-server/internal/scantrigger"
)
const scanTrigger = "autoscan"
const (
scanTrigger = "autoscan"
// Autoscan sources can occasionally report a very large marker window after
// downtime or provider recovery. Keep those windows bounded in the scan
// queue by falling back to one full-library scan per affected library.
maxAutoscanTargetsPerPoll = 1000
)
// Store is the persistence surface the engine needs. The repository implements
// the full CRUD surface; this interface is the read/bookkeeping subset the poll
@@ -194,15 +201,20 @@ func (s *Service) PollOnce(ctx context.Context) error {
rewritten := rewriteChanges(changes, src.PathRewrites)
targets, claimed, resolvedAny, stats := s.resolveAndClaim(ctx, rewritten, ttl)
if len(targets) > maxAutoscanTargetsPerPoll {
collapsed := collapseTargetsToLibraryScans(targets)
slog.WarnContext(ctx, "autoscan: collapsed large scan target batch to library scans",
"source_id", src.ID,
"targets", len(targets),
"collapsed_targets", len(collapsed),
"limit", maxAutoscanTargetsPerPoll,
)
targets = collapsed
}
var enqueue EnqueueResult
if len(targets) > 0 {
var eerr error
if eventID != 0 {
enqueue.Created, enqueue.Reused, eerr = s.queue.EnqueueAutoscanScans(ctx, targets, eventID)
} else {
eerr = s.queue.EnqueueScans(ctx, targets)
enqueue.Created = len(targets)
}
enqueue, eerr = s.enqueueScanTargets(ctx, targets, eventID)
if eerr != nil {
s.releaseClaims(ctx, claimed)
slog.WarnContext(ctx, "autoscan: enqueue failed", "source_id", src.ID, "err", eerr)
@@ -278,6 +290,34 @@ func (s *Service) PollOnce(ctx context.Context) error {
return nil
}
func (s *Service) enqueueScanTargets(ctx context.Context, targets []scantrigger.Target, eventID int64) (EnqueueResult, error) {
var result EnqueueResult
if len(targets) == 0 {
return result, nil
}
for start := 0; start < len(targets); start += maxAutoscanTargetsPerPoll {
end := start + maxAutoscanTargetsPerPoll
if end > len(targets) {
end = len(targets)
}
chunk := targets[start:end]
if eventID != 0 {
created, reused, err := s.queue.EnqueueAutoscanScans(ctx, chunk, eventID)
if err != nil {
return result, err
}
result.Created += created
result.Reused += reused
continue
}
if err := s.queue.EnqueueScans(ctx, chunk); err != nil {
return result, err
}
result.Created += len(chunk)
}
return result, nil
}
func (s *Service) createEvent(ctx context.Context, src Source, marker string, startedAt time.Time) (int64, bool) {
if s == nil || s.store == nil {
return 0, true
@@ -382,6 +422,29 @@ func (s *Service) resolveAndClaim(ctx context.Context, changes []Change, ttl tim
return targets, claimed, resolvedAny, stats
}
func collapseTargetsToLibraryScans(targets []scantrigger.Target) []scantrigger.Target {
if len(targets) == 0 {
return []scantrigger.Target{}
}
seen := make(map[int]struct{})
collapsed := make([]scantrigger.Target, 0)
for _, target := range targets {
if target.Folder == nil {
continue
}
if _, ok := seen[target.Folder.ID]; ok {
continue
}
seen[target.Folder.ID] = struct{}{}
collapsed = append(collapsed, scantrigger.Target{
Folder: target.Folder,
Mode: scantrigger.ModeLibrary,
Trigger: scanTrigger,
})
}
return collapsed
}
func (s *Service) resolveChange(ctx context.Context, change Change) (*scantrigger.Target, bool) {
if change.SourcePath == "" {
return nil, false
+117
View File
@@ -4,6 +4,7 @@ import (
"context"
"errors"
"path/filepath"
"strconv"
"strings"
"testing"
"time"
@@ -172,6 +173,38 @@ func (fakeResolver) ResolveMissingSubtree(_ context.Context, subtreePath, trigge
return nil, &scantrigger.RequestError{Status: 400, Code: "bad_request", Message: "outside media folders"}
}
type distinctFolderResolver struct{}
func (distinctFolderResolver) Resolve(_ context.Context, req scantrigger.Request) (*scantrigger.Target, error) {
if !strings.HasPrefix(req.Path, "/mnt/media/library-") {
return nil, &scantrigger.RequestError{Status: 400, Code: "bad_request", Message: "outside media folders"}
}
rest := strings.TrimPrefix(req.Path, "/mnt/media/library-")
idPart := rest
if slash := strings.IndexByte(rest, '/'); slash >= 0 {
idPart = rest[:slash]
}
id, err := strconv.Atoi(idPart)
if err != nil {
return nil, &scantrigger.RequestError{Status: 400, Code: "bad_request", Message: "outside media folders"}
}
return &scantrigger.Target{
Folder: &models.MediaFolder{ID: id},
Mode: scantrigger.ModeFile,
Path: req.Path,
Trigger: req.Trigger,
}, nil
}
func (distinctFolderResolver) ResolveMissingSubtree(_ context.Context, subtreePath, trigger string) (*scantrigger.Target, error) {
return &scantrigger.Target{
Folder: &models.MediaFolder{ID: 1},
Mode: scantrigger.ModeSubtree,
Path: subtreePath,
Trigger: trigger,
}, nil
}
// unresolvableResolver treats every path as outside Silo's media folders,
// returning a RequestError — the "none resolved → misconfiguration" signal.
type unresolvableResolver struct{}
@@ -194,16 +227,19 @@ func (denySuppressor) Release(context.Context, string) error { return nil }
type recordingQueuer struct {
enqueued []scantrigger.Target
batches [][]scantrigger.Target
autoscanEvents []int64
createdCount *int
reusedCount int
}
func (q *recordingQueuer) EnqueueScans(_ context.Context, targets []scantrigger.Target) error {
q.batches = append(q.batches, append([]scantrigger.Target(nil), targets...))
q.enqueued = append(q.enqueued, targets...)
return nil
}
func (q *recordingQueuer) EnqueueAutoscanScans(_ context.Context, targets []scantrigger.Target, eventID int64) (int, int, error) {
q.batches = append(q.batches, append([]scantrigger.Target(nil), targets...))
q.enqueued = append(q.enqueued, targets...)
q.autoscanEvents = append(q.autoscanEvents, eventID)
if q.createdCount != nil {
@@ -452,6 +488,87 @@ func TestPollOnceStructuredSubtreeChangeEnqueuesExactSubtree(t *testing.T) {
}
}
func TestPollOnceCollapsesLargeTargetBatchToLibraryScans(t *testing.T) {
store := &fakeStore{
settings: Settings{Enabled: true, DefaultPollIntervalSeconds: 600, DebounceSeconds: 60},
sources: []Source{{
ID: "s1", PluginID: "silo.autoscan.cephfs", CapabilityID: "cephfs", Enabled: true,
}},
}
changes := make([]Change, 0, maxAutoscanTargetsPerPoll+1)
for i := 0; i <= maxAutoscanTargetsPerPoll; i++ {
changes = append(changes, Change{
SourcePath: "/mnt/media/Show/S01/Episode" + strconv.Itoa(i) + ".mkv",
Scope: ChangeScopeFile,
})
}
prov := &fakeProvider{changes: map[string][]Change{"cephfs": changes}, nextMarker: "m1"}
q := &recordingQueuer{}
svc := newService(store, prov, q, allowSuppressor{})
if err := svc.PollOnce(context.Background()); err != nil {
t.Fatalf("PollOnce: %v", err)
}
if len(q.enqueued) != 1 {
t.Fatalf("expected one collapsed library scan, got %d: %+v", len(q.enqueued), q.enqueued)
}
target := q.enqueued[0]
if target.Folder == nil || target.Folder.ID != 7 || target.Mode != scantrigger.ModeLibrary || target.Path != "" {
t.Fatalf("unexpected collapsed target: %+v", target)
}
if got, ok := store.advanced["s1"]; !ok || got != "m1" {
t.Fatalf("marker must advance after collapsed enqueue, got %q ok=%v", got, ok)
}
if len(store.events) != 1 {
t.Fatalf("expected one finished event, got %d", len(store.events))
}
event := store.events[0]
if event.ChangesReturned != maxAutoscanTargetsPerPoll+1 || event.TargetsClaimed != maxAutoscanTargetsPerPoll+1 {
t.Fatalf("event should preserve original burst counts, got %+v", event)
}
if event.ScansCreated != 1 || event.ScansReused != 0 {
t.Fatalf("event should record collapsed scan counts, got %+v", event)
}
}
func TestPollOnceChunksCollapsedLibraryScansAtTargetCap(t *testing.T) {
store := &fakeStore{
settings: Settings{Enabled: true, DefaultPollIntervalSeconds: 600, DebounceSeconds: 60},
sources: []Source{{
ID: "s1", PluginID: "silo.autoscan.cephfs", CapabilityID: "cephfs", Enabled: true,
}},
}
changes := make([]Change, 0, maxAutoscanTargetsPerPoll+1)
for i := 0; i <= maxAutoscanTargetsPerPoll; i++ {
changes = append(changes, Change{
SourcePath: "/mnt/media/library-" + strconv.Itoa(i+1) + "/Episode.mkv",
Scope: ChangeScopeFile,
})
}
prov := &fakeProvider{changes: map[string][]Change{"cephfs": changes}, nextMarker: "m1"}
q := &recordingQueuer{}
svc := NewService(store, prov, passthroughConnRes{}, distinctFolderResolver{}, q, allowSuppressor{}, nil)
if err := svc.PollOnce(context.Background()); err != nil {
t.Fatalf("PollOnce: %v", err)
}
if len(q.enqueued) != maxAutoscanTargetsPerPoll+1 {
t.Fatalf("enqueued scans = %d, want %d", len(q.enqueued), maxAutoscanTargetsPerPoll+1)
}
if len(q.batches) != 2 {
t.Fatalf("queue batches = %d, want 2", len(q.batches))
}
if len(q.batches[0]) != maxAutoscanTargetsPerPoll {
t.Fatalf("first batch size = %d, want %d", len(q.batches[0]), maxAutoscanTargetsPerPoll)
}
if len(q.batches[1]) != 1 {
t.Fatalf("second batch size = %d, want 1", len(q.batches[1]))
}
if got, ok := store.advanced["s1"]; !ok || got != "m1" {
t.Fatalf("marker must advance after all chunks enqueue, got %q ok=%v", got, ok)
}
}
func TestPollOnceScansDistinctPathsUnderSameFolder(t *testing.T) {
// Two imported subtrees resolve to the SAME folder ID (7) but DIFFERENT
// target paths. Keying suppression on folder ID alone would drop the second;
+209 -51
View File
@@ -35,14 +35,25 @@ func ParseAudiobookGroupBy(raw string) (AudiobookGroupBy, bool) {
// AudiobookGroupsQuery controls the grouped audiobook browse lookup.
type AudiobookGroupsQuery struct {
LibraryID int
GroupBy AudiobookGroupBy
LibraryID int
GroupBy AudiobookGroupBy
SearchPrefix string
IncludeTotal bool
// Sort is one of "name" (default), "count", "duration".
Sort string
Limit int
Offset int
}
// AudiobookGroupsResult is the paged grouped browse response before API image
// URL resolution.
type AudiobookGroupsResult struct {
Groups []AudiobookGroup
Total int
HasMore bool
TotalExact bool
}
// AudiobookGroup is one grouped browse row: an author, narrator, or series
// with aggregate stats over the audiobooks visible to the viewer.
type AudiobookGroup struct {
@@ -56,28 +67,131 @@ type AudiobookGroup struct {
PosterPaths []string
}
type audiobookGroupsSQLPlan struct {
SQL string
Args []any
Limit int
FetchLimit int
Offset int
IncludeTotal bool
}
// audiobookGroupsPageCap bounds externally-requested pages.
const audiobookGroupsPageCap = 500
// audiobookGroupsFullCap preserves the full-list path used by the legacy
// short-lived cache; the endpoint now uses true paged queries instead.
const audiobookGroupsFullCap = 50000
// ListAudiobookGroups returns grouped browse rows for an audiobook library.
// Groups are aggregated per person name (authors/narrators) or per series
// name, matching the case-insensitive semantics of the corresponding catalog
// filters, so a group name can be fed straight back into an author/narrator/
// series filter rule. Progress counts are scoped to the requesting profile via
// filter.UserID / filter.ProfileID.
func ListAudiobookGroups(ctx context.Context, pool *pgxpool.Pool, q AudiobookGroupsQuery, filter AccessFilter) ([]AudiobookGroup, int, error) {
if pool == nil {
return nil, 0, fmt.Errorf("audiobook groups: no database pool")
func ListAudiobookGroups(ctx context.Context, pool *pgxpool.Pool, q AudiobookGroupsQuery, filter AccessFilter) (AudiobookGroupsResult, error) {
return listAudiobookGroupsWithLimit(ctx, pool, q, filter, audiobookGroupsPageCap)
}
// listAllAudiobookGroups fetches a complete grouped list for the older cache
// helper. New callers should prefer ListAudiobookGroups so each page can avoid
// recomputing exact totals and poster stacks for groups outside the page.
func listAllAudiobookGroups(ctx context.Context, pool *pgxpool.Pool, q AudiobookGroupsQuery, filter AccessFilter) ([]AudiobookGroup, int, error) {
full := q
full.Limit = audiobookGroupsFullCap
full.Offset = 0
full.IncludeTotal = true
result, err := listAudiobookGroupsWithLimit(ctx, pool, full, filter, audiobookGroupsFullCap)
if err != nil {
return nil, 0, err
}
return result.Groups, result.Total, nil
}
func listAudiobookGroupsWithLimit(ctx context.Context, pool *pgxpool.Pool, q AudiobookGroupsQuery, filter AccessFilter, maxLimit int) (AudiobookGroupsResult, error) {
if pool == nil {
return AudiobookGroupsResult{}, fmt.Errorf("audiobook groups: no database pool")
}
plan, err := buildAudiobookGroupsSQLWithLimit(q, filter, maxLimit)
if err != nil {
return AudiobookGroupsResult{}, err
}
if plan.SQL == "" {
return AudiobookGroupsResult{Groups: []AudiobookGroup{}, TotalExact: plan.IncludeTotal}, nil
}
rows, err := pool.Query(ctx, plan.SQL, plan.Args...)
if err != nil {
return AudiobookGroupsResult{}, fmt.Errorf("querying audiobook groups: %w", err)
}
defer rows.Close()
total := 0
groups := make([]AudiobookGroup, 0, plan.Limit)
for rows.Next() {
var g AudiobookGroup
var posterPaths []string
if plan.IncludeTotal {
if err := rows.Scan(&g.Name, &g.ItemCount, &g.TotalDurationSeconds, &g.InProgressCount, &g.FinishedCount, &posterPaths, &total); err != nil {
return AudiobookGroupsResult{}, fmt.Errorf("scanning audiobook group: %w", err)
}
} else if err := rows.Scan(&g.Name, &g.ItemCount, &g.TotalDurationSeconds, &g.InProgressCount, &g.FinishedCount, &posterPaths); err != nil {
return AudiobookGroupsResult{}, fmt.Errorf("scanning audiobook group: %w", err)
}
if posterPaths == nil {
posterPaths = []string{}
}
g.PosterPaths = posterPaths
groups = append(groups, g)
}
if err := rows.Err(); err != nil {
return AudiobookGroupsResult{}, fmt.Errorf("iterating audiobook groups: %w", err)
}
hasMore := false
if plan.IncludeTotal {
hasMore = plan.Offset+len(groups) < total
} else if len(groups) > plan.Limit {
hasMore = true
groups = groups[:plan.Limit]
total = plan.Offset + len(groups) + 1
} else {
total = plan.Offset + len(groups)
}
return AudiobookGroupsResult{
Groups: groups,
Total: total,
HasMore: hasMore,
TotalExact: plan.IncludeTotal,
}, nil
}
func buildAudiobookGroupsSQL(q AudiobookGroupsQuery, filter AccessFilter) (audiobookGroupsSQLPlan, error) {
return buildAudiobookGroupsSQLWithLimit(q, filter, audiobookGroupsPageCap)
}
func buildAudiobookGroupsSQLWithLimit(q AudiobookGroupsQuery, filter AccessFilter, maxLimit int) (audiobookGroupsSQLPlan, error) {
if q.LibraryID <= 0 {
return nil, 0, fmt.Errorf("audiobook groups: library id is required")
return audiobookGroupsSQLPlan{}, fmt.Errorf("audiobook groups: library id is required")
}
limit := q.Limit
if limit <= 0 {
limit = 200
}
if limit > 500 {
limit = 500
if maxLimit <= 0 {
maxLimit = audiobookGroupsPageCap
}
if limit > maxLimit {
limit = maxLimit
}
offset := max(q.Offset, 0)
fetchLimit := limit
if !q.IncludeTotal {
fetchLimit++
}
args := []any{q.LibraryID}
argIdx := 2
@@ -87,7 +201,12 @@ func ListAudiobookGroups(ctx context.Context, pool *pgxpool.Pool, q AudiobookGro
"mil.media_folder_id = $1",
}
if !appendAudiobookItemAccessConditions("mi", filter, &conditions, &args, &argIdx) {
return []AudiobookGroup{}, 0, nil
return audiobookGroupsSQLPlan{
Limit: limit,
FetchLimit: fetchLimit,
Offset: offset,
IncludeTotal: q.IncludeTotal,
}, nil
}
var joinClause, nameExpr, groupExpr, posterOrder string
@@ -110,7 +229,18 @@ func ListAudiobookGroups(ctx context.Context, pool *pgxpool.Pool, q AudiobookGro
groupExpr = "LOWER(BTRIM(s.series_name))"
posterOrder = "s.series_index NULLS LAST, b.sort_title"
default:
return nil, 0, fmt.Errorf("audiobook groups: unsupported group_by %q", q.GroupBy)
return audiobookGroupsSQLPlan{}, fmt.Errorf("audiobook groups: unsupported group_by %q", q.GroupBy)
}
groupFilters := make([]string, 0, 1)
if search := audiobookGroupSearchPattern(q.SearchPrefix); search != "" {
groupFilters = append(groupFilters, fmt.Sprintf(`%s LIKE $%d ESCAPE '\'`, groupExpr, argIdx))
args = append(args, search)
argIdx++
}
groupWhereClause := ""
if len(groupFilters) > 0 {
groupWhereClause = "WHERE " + strings.Join(groupFilters, " AND ")
}
userArg := fmt.Sprintf("$%d", argIdx)
@@ -129,7 +259,14 @@ func ListAudiobookGroups(ctx context.Context, pool *pgxpool.Pool, q AudiobookGro
case "duration":
orderClause = "total_duration_seconds DESC, LOWER(name)"
default:
return nil, 0, fmt.Errorf("audiobook groups: unsupported sort %q", q.Sort)
return audiobookGroupsSQLPlan{}, fmt.Errorf("audiobook groups: unsupported sort %q", q.Sort)
}
totalSelect := ""
totalColumn := ""
if q.IncludeTotal {
totalSelect = ", COUNT(*) OVER ()::int AS total_groups"
totalColumn = ", pg.total_groups"
}
query := fmt.Sprintf(`
@@ -137,65 +274,86 @@ func ListAudiobookGroups(ctx context.Context, pool *pgxpool.Pool, q AudiobookGro
SELECT
mi.content_id,
mi.poster_path,
mi.sort_title,
COALESCE((
SELECT SUM(mf.duration)
FROM media_files mf
WHERE mf.content_id = mi.content_id AND mf.missing_since IS NULL
), 0) AS duration_seconds
COALESCE(NULLIF(mi.sort_title, ''), mi.title) AS sort_title,
COALESCE(afs.duration_seconds, 0)::bigint AS duration_seconds
FROM media_items mi
JOIN media_item_libraries mil ON mil.content_id = mi.content_id
LEFT JOIN audiobook_item_file_stats afs
ON afs.media_folder_id = mil.media_folder_id
AND afs.content_id = mi.content_id
WHERE %s
),
grouped AS (
SELECT
%s AS group_key,
%s AS name,
COUNT(*)::int AS item_count,
COALESCE(SUM(b.duration_seconds), 0)::bigint AS total_duration_seconds,
COUNT(*) FILTER (WHERE uwp.media_item_id IS NOT NULL AND NOT uwp.completed)::int AS in_progress_count,
COUNT(*) FILTER (WHERE uwp.completed)::int AS finished_count,
(ARRAY_AGG(b.poster_path ORDER BY %s) FILTER (WHERE NULLIF(b.poster_path, '') IS NOT NULL))[1:4] AS poster_paths
COUNT(*) FILTER (WHERE uwp.completed)::int AS finished_count
FROM books b
%s
LEFT JOIN user_watch_progress uwp
ON uwp.media_item_id = b.content_id
AND uwp.user_id = %s
AND uwp.profile_id = %s
%s
GROUP BY %s
),
paged_groups AS (
SELECT
group_key,
name,
item_count,
total_duration_seconds,
in_progress_count,
finished_count%s
FROM grouped
ORDER BY %s
LIMIT $%d OFFSET $%d
)
SELECT name, item_count, total_duration_seconds, in_progress_count, finished_count, poster_paths,
COUNT(*) OVER ()::int AS total_groups
FROM grouped
ORDER BY %s
LIMIT $%d OFFSET $%d`,
SELECT
pg.name,
pg.item_count,
pg.total_duration_seconds,
pg.in_progress_count,
pg.finished_count,
COALESCE(posters.poster_paths, '{}'::text[]) AS poster_paths%s
FROM paged_groups pg
LEFT JOIN LATERAL (
SELECT ARRAY(
SELECT b.poster_path
FROM books b
%s
WHERE %s = pg.group_key
AND NULLIF(b.poster_path, '') IS NOT NULL
ORDER BY %s
LIMIT 4
) AS poster_paths
) posters ON TRUE
ORDER BY %s`,
strings.Join(conditions, " AND "),
nameExpr, posterOrder, joinClause, userArg, profileArg, groupExpr, orderClause,
argIdx, argIdx+1,
groupExpr, nameExpr, joinClause, userArg, profileArg, groupWhereClause, groupExpr,
totalSelect, orderClause, argIdx, argIdx+1,
totalColumn, joinClause, groupExpr, posterOrder, orderClause,
)
args = append(args, limit, offset)
args = append(args, fetchLimit, offset)
rows, err := pool.Query(ctx, query, args...)
if err != nil {
return nil, 0, fmt.Errorf("querying audiobook groups: %w", err)
}
defer rows.Close()
total := 0
groups := make([]AudiobookGroup, 0, limit)
for rows.Next() {
var g AudiobookGroup
var posterPaths []string
if err := rows.Scan(&g.Name, &g.ItemCount, &g.TotalDurationSeconds, &g.InProgressCount, &g.FinishedCount, &posterPaths, &total); err != nil {
return nil, 0, fmt.Errorf("scanning audiobook group: %w", err)
}
if posterPaths == nil {
posterPaths = []string{}
}
g.PosterPaths = posterPaths
groups = append(groups, g)
}
if err := rows.Err(); err != nil {
return nil, 0, fmt.Errorf("iterating audiobook groups: %w", err)
}
return groups, total, nil
return audiobookGroupsSQLPlan{
SQL: query,
Args: args,
Limit: limit,
FetchLimit: fetchLimit,
Offset: offset,
IncludeTotal: q.IncludeTotal,
}, nil
}
func audiobookGroupSearchPattern(raw string) string {
raw = strings.ToLower(strings.TrimSpace(raw))
if raw == "" {
return ""
}
replacer := strings.NewReplacer(`\`, `\\`, `%`, `\%`, `_`, `\_`)
return replacer.Replace(raw) + "%"
}
+142
View File
@@ -0,0 +1,142 @@
package catalog
import (
"context"
"fmt"
"sort"
"strconv"
"strings"
"time"
"github.com/jackc/pgx/v5/pgxpool"
"golang.org/x/sync/singleflight"
"github.com/Silo-Server/silo-server/internal/cache"
)
type groupsCacheEntry struct {
groups []AudiobookGroup
total int
}
// audiobookGroupsFetcher fetches the complete grouped list for a query.
type audiobookGroupsFetcher func(ctx context.Context, q AudiobookGroupsQuery, filter AccessFilter) ([]AudiobookGroup, int, error)
// AudiobookGroupsCache serves paged audiobook-group browse results from a
// short-lived in-process cache of the full grouped list.
//
// The author/narrator grouping is an expensive aggregation over the whole
// library, and the client pages through the entire result on every load
// (sequential 500-row requests until the total is reached). Without a cache
// each page re-runs the full aggregation, so a cold load is N pages times the
// per-page cost; caching the full sorted list per (library, group_by, sort,
// viewer) lets one computation serve every page and survive a quick refresh.
type AudiobookGroupsCache struct {
cache *cache.TTLCache[*groupsCacheEntry]
ttl time.Duration
fetch audiobookGroupsFetcher
group singleflight.Group
}
// NewAudiobookGroupsCache builds a cache that warms itself from the given pool.
func NewAudiobookGroupsCache(pool *pgxpool.Pool, ttl time.Duration) *AudiobookGroupsCache {
return &AudiobookGroupsCache{
cache: cache.NewTTLCache[*groupsCacheEntry](),
ttl: ttl,
fetch: func(ctx context.Context, q AudiobookGroupsQuery, filter AccessFilter) ([]AudiobookGroup, int, error) {
return listAllAudiobookGroups(ctx, pool, q, filter)
},
}
}
// Close stops the cache's background expiry sweeper.
func (c *AudiobookGroupsCache) Close() {
if c != nil && c.cache != nil {
c.cache.Close()
}
}
// Page returns the offset/limit slice of the grouped list plus the full group
// count.
func (c *AudiobookGroupsCache) Page(ctx context.Context, q AudiobookGroupsQuery, filter AccessFilter) ([]AudiobookGroup, int, error) {
key := audiobookGroupsCacheKey(q, filter)
if entry, ok := c.cache.Get(key); ok {
return sliceGroups(entry.groups, q.Offset, q.Limit), entry.total, nil
}
value, err, _ := c.group.Do(key, func() (any, error) {
if entry, ok := c.cache.Get(key); ok {
return entry, nil
}
groups, total, err := c.fetch(ctx, q, filter)
if err != nil {
return nil, err
}
entry := &groupsCacheEntry{groups: groups, total: total}
c.cache.Set(key, entry, c.ttl)
return entry, nil
})
if err != nil {
return nil, 0, err
}
entry := value.(*groupsCacheEntry)
return sliceGroups(entry.groups, q.Offset, q.Limit), entry.total, nil
}
func sliceGroups(groups []AudiobookGroup, offset, limit int) []AudiobookGroup {
if offset < 0 {
offset = 0
}
if offset >= len(groups) {
return []AudiobookGroup{}
}
end := len(groups)
if limit > 0 && offset+limit < end {
end = offset + limit
}
return groups[offset:end]
}
// audiobookGroupsCacheKey identifies a cached full list. It includes every
// AccessFilter field that changes the rows or the per-profile progress counts
// so two viewers (or two access scopes) never share an entry.
func audiobookGroupsCacheKey(q AudiobookGroupsQuery, filter AccessFilter) string {
sortKey := strings.ToLower(strings.TrimSpace(q.Sort))
if sortKey == "" {
sortKey = "name"
}
searchKey := strings.ToLower(strings.TrimSpace(q.SearchPrefix))
var b strings.Builder
fmt.Fprintf(&b, "%d|%s|%s|q=%s|u=%d|p=%s|cr=%s", q.LibraryID, q.GroupBy, sortKey, searchKey, filter.UserID, filter.ProfileID, filter.MaxContentRating)
b.WriteString("|allow=")
b.WriteString(joinSortedInts(filter.AllowedLibraryIDs))
b.WriteString("|deny=")
b.WriteString(joinSortedInts(filter.DisabledLibraryIDs))
b.WriteString("|cids=")
b.WriteString(strings.Join(sortedCopy(filter.AllowedContentIDs), ","))
b.WriteString("|excluded_types=")
b.WriteString(strings.Join(sortedCopy(filter.ExcludedMediaTypes), ","))
return b.String()
}
func joinSortedInts(values []int) string {
if len(values) == 0 {
return ""
}
cp := append([]int(nil), values...)
sort.Ints(cp)
parts := make([]string, len(cp))
for i, v := range cp {
parts[i] = strconv.Itoa(v)
}
return strings.Join(parts, ",")
}
func sortedCopy(values []string) []string {
if len(values) == 0 {
return nil
}
cp := append([]string(nil), values...)
sort.Strings(cp)
return cp
}
@@ -0,0 +1,180 @@
package catalog
import (
"context"
"fmt"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/Silo-Server/silo-server/internal/cache"
)
func newTestGroups(n int) []AudiobookGroup {
groups := make([]AudiobookGroup, n)
for i := range groups {
groups[i] = AudiobookGroup{Name: "author-" + string(rune('a'+i)), ItemCount: i + 1}
}
return groups
}
// The grouped list is the expensive aggregation; paging through it (the client
// fetches every page on each load) must compute it once, not once per page.
func TestAudiobookGroupsCache_ComputesFullListOncePerKey(t *testing.T) {
full := newTestGroups(5)
var fetches int
c := &AudiobookGroupsCache{
cache: cache.NewTTLCache[*groupsCacheEntry](),
ttl: time.Minute,
fetch: func(context.Context, AudiobookGroupsQuery, AccessFilter) ([]AudiobookGroup, int, error) {
fetches++
return full, len(full), nil
},
}
defer c.Close()
q := AudiobookGroupsQuery{LibraryID: 7, GroupBy: AudiobookGroupByAuthor, Sort: "name"}
filter := AccessFilter{UserID: 1, ProfileID: "p1"}
page1, total, err := c.Page(context.Background(), AudiobookGroupsQuery{LibraryID: q.LibraryID, GroupBy: q.GroupBy, Sort: q.Sort, Limit: 2, Offset: 0}, filter)
if err != nil {
t.Fatalf("page1: %v", err)
}
page2, _, err := c.Page(context.Background(), AudiobookGroupsQuery{LibraryID: q.LibraryID, GroupBy: q.GroupBy, Sort: q.Sort, Limit: 2, Offset: 2}, filter)
if err != nil {
t.Fatalf("page2: %v", err)
}
page3, _, err := c.Page(context.Background(), AudiobookGroupsQuery{LibraryID: q.LibraryID, GroupBy: q.GroupBy, Sort: q.Sort, Limit: 2, Offset: 4}, filter)
if err != nil {
t.Fatalf("page3: %v", err)
}
if fetches != 1 {
t.Fatalf("full list computed %d times across 3 pages of one key; want 1", fetches)
}
if total != 5 {
t.Fatalf("total = %d, want 5", total)
}
if len(page1) != 2 || len(page2) != 2 || len(page3) != 1 {
t.Fatalf("page sizes = %d,%d,%d; want 2,2,1", len(page1), len(page2), len(page3))
}
if page1[0].Name != "author-a" || page2[0].Name != "author-c" || page3[0].Name != "author-e" {
t.Fatalf("slice boundaries wrong: %q %q %q", page1[0].Name, page2[0].Name, page3[0].Name)
}
}
// A different viewer must not be served another profile's cached counts.
func TestAudiobookGroupsCache_KeyedByViewer(t *testing.T) {
var fetches int
c := &AudiobookGroupsCache{
cache: cache.NewTTLCache[*groupsCacheEntry](),
ttl: time.Minute,
fetch: func(context.Context, AudiobookGroupsQuery, AccessFilter) ([]AudiobookGroup, int, error) {
fetches++
return newTestGroups(3), 3, nil
},
}
defer c.Close()
q := AudiobookGroupsQuery{LibraryID: 7, GroupBy: AudiobookGroupByAuthor, Sort: "name", Limit: 10}
if _, _, err := c.Page(context.Background(), q, AccessFilter{UserID: 1, ProfileID: "p1"}); err != nil {
t.Fatalf("viewer1: %v", err)
}
if _, _, err := c.Page(context.Background(), q, AccessFilter{UserID: 2, ProfileID: "p2"}); err != nil {
t.Fatalf("viewer2: %v", err)
}
if fetches != 2 {
t.Fatalf("distinct viewers shared a cache entry: fetches=%d, want 2", fetches)
}
}
func TestAudiobookGroupsCache_DeduplicatesConcurrentMisses(t *testing.T) {
var fetches atomic.Int32
started := make(chan struct{})
release := make(chan struct{})
var startedOnce sync.Once
c := &AudiobookGroupsCache{
cache: cache.NewTTLCache[*groupsCacheEntry](),
ttl: time.Minute,
fetch: func(context.Context, AudiobookGroupsQuery, AccessFilter) ([]AudiobookGroup, int, error) {
fetches.Add(1)
startedOnce.Do(func() { close(started) })
<-release
return newTestGroups(4), 4, nil
},
}
defer c.Close()
q := AudiobookGroupsQuery{LibraryID: 7, GroupBy: AudiobookGroupByAuthor, Sort: "name", Limit: 2}
filter := AccessFilter{UserID: 1, ProfileID: "p1"}
const callers = 8
ready := make(chan struct{}, callers)
begin := make(chan struct{})
errs := make(chan error, callers)
var wg sync.WaitGroup
wg.Add(callers)
for range callers {
go func() {
defer wg.Done()
ready <- struct{}{}
<-begin
groups, total, err := c.Page(context.Background(), q, filter)
if err != nil {
errs <- err
return
}
if total != 4 || len(groups) != 2 {
errs <- fmt.Errorf("page result = len %d total %d, want len 2 total 4", len(groups), total)
}
}()
}
for range callers {
<-ready
}
close(begin)
select {
case <-started:
case <-time.After(time.Second):
t.Fatal("timed out waiting for cache fetch to start")
}
time.Sleep(50 * time.Millisecond)
close(release)
wg.Wait()
close(errs)
for err := range errs {
if err != nil {
t.Fatalf("concurrent page call failed: %v", err)
}
}
if got := fetches.Load(); got != 1 {
t.Fatalf("concurrent miss fetched %d times; want 1", got)
}
}
func TestAudiobookGroupsCache_KeyIncludesExcludedMediaTypes(t *testing.T) {
var fetches int
c := &AudiobookGroupsCache{
cache: cache.NewTTLCache[*groupsCacheEntry](),
ttl: time.Minute,
fetch: func(context.Context, AudiobookGroupsQuery, AccessFilter) ([]AudiobookGroup, int, error) {
fetches++
return newTestGroups(3), 3, nil
},
}
defer c.Close()
q := AudiobookGroupsQuery{LibraryID: 7, GroupBy: AudiobookGroupByAuthor, Sort: "name", Limit: 10}
if _, _, err := c.Page(context.Background(), q, AccessFilter{UserID: 1, ProfileID: "p1", ExcludedMediaTypes: []string{"podcast"}}); err != nil {
t.Fatalf("podcast excluded: %v", err)
}
if _, _, err := c.Page(context.Background(), q, AccessFilter{UserID: 1, ProfileID: "p1", ExcludedMediaTypes: []string{"audiobook"}}); err != nil {
t.Fatalf("audiobook excluded: %v", err)
}
if _, _, err := c.Page(context.Background(), q, AccessFilter{UserID: 1, ProfileID: "p1", ExcludedMediaTypes: []string{"audiobook"}}); err != nil {
t.Fatalf("audiobook excluded cached: %v", err)
}
if fetches != 2 {
t.Fatalf("excluded media type variants shared a cache entry: fetches=%d, want 2", fetches)
}
}
+96
View File
@@ -0,0 +1,96 @@
package catalog
import (
"strings"
"testing"
)
func TestBuildAudiobookGroupsSQL_UsesItemFileStatsInsteadOfMediaFilesSubquery(t *testing.T) {
plan, err := buildAudiobookGroupsSQL(AudiobookGroupsQuery{
LibraryID: 10,
GroupBy: AudiobookGroupByAuthor,
Sort: "count",
Limit: 60,
Offset: 0,
IncludeTotal: true,
}, AccessFilter{})
if err != nil {
t.Fatalf("buildAudiobookGroupsSQL error: %v", err)
}
if !strings.Contains(plan.SQL, "audiobook_item_file_stats") {
t.Fatalf("audiobook groups query must use maintained item file stats; got:\n%s", plan.SQL)
}
if strings.Contains(plan.SQL, "FROM media_files mf") || strings.Contains(plan.SQL, "SELECT SUM(mf.duration)") {
t.Fatalf("audiobook groups query must not use the old correlated media_files duration lookup; got:\n%s", plan.SQL)
}
}
func TestBuildAudiobookGroupsSQL_SkipTotalOmitsWindowCountAndFetchesLimitPlusOne(t *testing.T) {
plan, err := buildAudiobookGroupsSQL(AudiobookGroupsQuery{
LibraryID: 10,
GroupBy: AudiobookGroupByNarrator,
Sort: "name",
Limit: 60,
Offset: 120,
IncludeTotal: false,
}, AccessFilter{})
if err != nil {
t.Fatalf("buildAudiobookGroupsSQL error: %v", err)
}
if strings.Contains(plan.SQL, "COUNT(*) OVER ()") {
t.Fatalf("include_total=false must omit window counts; got:\n%s", plan.SQL)
}
if strings.Contains(plan.SQL, "total_groups") {
t.Fatalf("include_total=false must omit exact total selection; got:\n%s", plan.SQL)
}
if plan.FetchLimit != 61 {
t.Fatalf("FetchLimit = %d, want limit+1 for has_more detection", plan.FetchLimit)
}
if plan.Limit != 60 {
t.Fatalf("Limit = %d, want requested page size", plan.Limit)
}
}
func TestBuildAudiobookGroupsSQL_SearchAppliesNormalizedPrefixFilter(t *testing.T) {
for _, tc := range []struct {
name string
groupBy AudiobookGroupBy
want string
}{
{name: "author", groupBy: AudiobookGroupByAuthor, want: "LOWER(BTRIM(p.name)) LIKE $"},
{name: "narrator", groupBy: AudiobookGroupByNarrator, want: "LOWER(BTRIM(p.name)) LIKE $"},
{name: "series", groupBy: AudiobookGroupBySeries, want: "LOWER(BTRIM(s.series_name)) LIKE $"},
} {
t.Run(tc.name, func(t *testing.T) {
plan, err := buildAudiobookGroupsSQL(AudiobookGroupsQuery{
LibraryID: 10,
GroupBy: tc.groupBy,
Sort: "name",
Limit: 60,
SearchPrefix: "Neil",
IncludeTotal: true,
}, AccessFilter{})
if err != nil {
t.Fatalf("buildAudiobookGroupsSQL error: %v", err)
}
if !strings.Contains(plan.SQL, tc.want) {
t.Fatalf("expected normalized prefix predicate %q; got:\n%s", tc.want, plan.SQL)
}
if !containsArg(plan.Args, "neil%") {
t.Fatalf("search args = %#v, want neil%%", plan.Args)
}
})
}
}
func containsArg(args []any, want any) bool {
for _, arg := range args {
if arg == want {
return true
}
}
return false
}
+77 -1
View File
@@ -344,6 +344,10 @@ func (r *BrowseRepository) buildBrowsePlan(filters BrowseFilters) (browseQueryPl
applyAccessFilter("mi", AccessFilter{MaxContentRating: filters.MaxContentRating}, &conditions, &args, &argIdx)
// Manga chapters (type='ebook' rows linked into a manga series) are internal
// sub-units and must never surface as standalone catalog items.
conditions = append(conditions, MangaChapterExclusionWhere("mi"))
if filters.SnapshotAt != nil {
conditions = append(conditions, fmt.Sprintf("mi.created_at <= $%d", argIdx))
args = append(args, *filters.SnapshotAt)
@@ -372,7 +376,14 @@ func (r *BrowseRepository) buildBrowsePlan(filters BrowseFilters) (browseQueryPl
orderBy, orderArgs := buildOrderByPlan(filters.Sort, filters.Order, filters.SnapshotAt, argIdx, singleLibraryNoDedup, browseFiltersAreMovieOnly(filters))
argIdx += len(orderArgs)
selectClause := browseItemColumns("mi")
// Only run the manga count subqueries when the scope can contain manga
// series; a non-manga type filter rules them out, so substitute NULL
// placeholders and skip two correlated subqueries per row on the hot path.
mangaCounts := mangaCountColumns("mi")
if !browseScopeMayContainManga(filters) {
mangaCounts = nullMangaCountColumns()
}
selectClause := browseItemColumns("mi") + ", " + mangaCounts
groupByClause := ""
switch {
case singleLibraryNoDedup:
@@ -1123,6 +1134,63 @@ func browseItemColumns(alias string) string {
return strings.Join(prefixed, ", ")
}
// mangaCountColumns returns two index-backed correlated subqueries feeding the
// "X Volumes · X Chapters" poster chip: distinct volume tokens (many chapter
// rows can share one volume) and loose chapter rows without a volume token.
// They return 0 for non-manga rows (no matching manga_chapters), which the
// scan path nils out so only manga cards carry the counts. The subqueries are
// functionally dependent on alias.content_id (the media_items PK, which leads
// browseGroupByColumns), so they remain valid under the dedup GROUP BY without
// being listed there.
func mangaCountColumns(alias string) string {
return "(SELECT count(*) FROM manga_chapters mc WHERE mc.series_content_id = " + alias + ".content_id AND (mc.volume IS NULL OR mc.volume = '')) AS manga_chapter_count, " +
"(SELECT count(DISTINCT mc.volume) FROM manga_chapters mc WHERE mc.series_content_id = " + alias + ".content_id AND mc.volume IS NOT NULL AND mc.volume <> '') AS manga_volume_count"
}
// nullMangaCountColumns substitutes NULL placeholders for the manga count
// subqueries when the browse scope cannot contain manga series. Column names
// and order match mangaCountColumns so the scan path is unchanged.
func nullMangaCountColumns() string {
return "NULL::bigint AS manga_chapter_count, NULL::bigint AS manga_volume_count"
}
// browseScopeMayContainManga reports whether a browse with these filters could
// return type='manga' rows. An empty type filter (all types) or one that
// includes "manga" keeps the counts; any other explicit type filter rules
// manga out, letting the caller skip the count subqueries.
func browseScopeMayContainManga(filters BrowseFilters) bool {
if filters.Type == "" {
return true
}
for _, t := range strings.Split(filters.Type, ",") {
if strings.TrimSpace(t) == "manga" {
return true
}
}
return false
}
// MangaChapterExclusionWhere returns a WHERE predicate that hides manga CHAPTER
// items (type='ebook' rows linked into a type='manga' series via the
// manga_chapters table) from catalog listing surfaces — browse, section
// resolution, and search. Chapters are internal sub-units of a manga series and
// must never appear as standalone catalog items; only the series should.
//
// It is index-backed: manga_chapters.chapter_content_id is the table's primary
// key, so the anti-join is a cheap unique-index probe. The predicate is global
// and harmless for every other row: regular ebooks have no manga_chapters link,
// and non-ebook types never match either, so they all pass. It is redundant
// (but harmless) for type='manga' browse scopes, whose series rows are linked
// via series_content_id, not chapter_content_id.
//
// By-id fetch paths that legitimately resolve chapters — the ebook reader,
// continue-reading (ebook_reader_progress / watch-progress), and the series
// detail chapter list (mangaChaptersQuery) — use separate queries and must NOT
// call this.
func MangaChapterExclusionWhere(alias string) string {
return "NOT EXISTS (SELECT 1 FROM manga_chapters mc WHERE mc.chapter_content_id = " + alias + ".content_id)"
}
// browseGroupByColumns returns the columns needed for GROUP BY when joining
// with the junction table.
func browseGroupByColumns(alias string) string {
@@ -1200,11 +1268,19 @@ func scanBrowseItems(rows pgx.Rows) ([]*models.MediaItem, error) {
&item.Status,
&item.CreatedAt,
&item.UpdatedAt,
&item.MangaChapterCount,
&item.MangaVolumeCount,
&item.AddedAt,
)
if err != nil {
return nil, fmt.Errorf("scanning browse item row: %w", err)
}
// The manga count subqueries return 0 for non-manga rows; drop them so
// only manga cards carry the counts (movies/series stay clean).
if item.Type != "manga" {
item.MangaChapterCount = nil
item.MangaVolumeCount = nil
}
items = append(items, &item)
}
if err := rows.Err(); err != nil {
@@ -0,0 +1,97 @@
package catalog
import (
"strings"
"testing"
)
// TestMangaCountColumns pins the browse-card manga count contract: two
// index-backed correlated subqueries over manga_chapters, scoped to the series
// content ID and aliased so the scan paths can read them positionally. The
// card chip reads "X Volumes · X Chapters", so manga_volume_count must count
// DISTINCT volume tokens (many chapter rows can share one volume) and
// manga_chapter_count must count only loose rows without a volume token.
func TestMangaCountColumns(t *testing.T) {
cols := mangaCountColumns("mi")
for _, want := range []string{
"FROM manga_chapters mc",
"mc.series_content_id = mi.content_id",
"count(DISTINCT mc.volume)",
"mc.volume IS NOT NULL AND mc.volume <> ''",
"AS manga_volume_count",
"(mc.volume IS NULL OR mc.volume = '')",
"AS manga_chapter_count",
} {
if !strings.Contains(cols, want) {
t.Fatalf("manga count columns missing %q\ngot: %s", want, cols)
}
}
// Both counts must be present (two correlated subqueries).
if got := strings.Count(cols, "FROM manga_chapters mc"); got != 2 {
t.Fatalf("expected 2 manga count subqueries, got %d\ngot: %s", got, cols)
}
}
// TestBrowseScopeMayContainManga pins the gating that lets browse skip the two
// manga count subqueries when the scope cannot return manga rows. An empty type
// filter (all types) or one that includes "manga" keeps them; any other
// explicit type filter rules manga out.
func TestBrowseScopeMayContainManga(t *testing.T) {
cases := []struct {
typeFilter string
want bool
}{
{"", true},
{"manga", true},
{"movie,manga", true},
{" manga ", true},
{"movie", false},
{"movie,series,episode", false},
{"ebook", false},
}
for _, c := range cases {
if got := browseScopeMayContainManga(BrowseFilters{Type: c.typeFilter}); got != c.want {
t.Fatalf("browseScopeMayContainManga(%q) = %v, want %v", c.typeFilter, got, c.want)
}
}
}
// nullMangaCountColumns must keep the exact column names and order of
// mangaCountColumns so the shared scan path is unchanged when the subqueries
// are skipped.
func TestNullMangaCountColumnsMatchScanContract(t *testing.T) {
null := nullMangaCountColumns()
for _, want := range []string{"AS manga_chapter_count", "AS manga_volume_count"} {
if !strings.Contains(null, want) {
t.Fatalf("null manga count columns missing %q\ngot: %s", want, null)
}
}
if strings.Contains(null, "FROM manga_chapters") {
t.Fatalf("null manga count columns must not run subqueries\ngot: %s", null)
}
if a, b := strings.Index(null, "manga_chapter_count"), strings.Index(null, "manga_volume_count"); a > b {
t.Fatalf("column order must match mangaCountColumns (chapter then volume)\ngot: %s", null)
}
}
// The library page browses through the catalog query preview path
// (previewQuerySource -> QueryExecutor.PreviewPage), not BrowseRepository, so
// the preview-page SELECT must carry the same manga count columns or manga
// cards in /library/{id}?tab=library render without the Vols/Ch chip.
func TestPreviewPageSQLIncludesMangaCounts(t *testing.T) {
sql, _, err := (&QueryExecutor{}).buildPreviewPageSQL(
QueryDefinition{MediaScope: "manga"},
AccessFilter{},
20, 0, true,
)
if err != nil {
t.Fatalf("buildPreviewPageSQL error: %v", err)
}
for _, want := range []string{"AS manga_chapter_count", "AS manga_volume_count"} {
if !strings.Contains(sql, want) {
t.Fatalf("preview-page SQL missing %q\ngot: %s", want, sql)
}
}
}
+7 -3
View File
@@ -126,10 +126,14 @@ func ParseCatalogRequest(values url.Values) (CatalogRequest, error) {
if req.CollectionID == "" {
return CatalogRequest{}, fmt.Errorf("collection_id is required")
}
if hasCatalogOverlayParams(values, false) {
return CatalogRequest{}, fmt.Errorf("source %q does not allow overlay params", req.Source)
overlay, err := parseCatalogOverlay(values)
if err != nil {
return CatalogRequest{}, err
}
req.UseSourceOrder = true
req.NamePrefix = overlay.namePrefix
req.SearchQuery = overlay.searchQuery
req.Query = overlay.query
req.UseSourceOrder = !overlay.hasExplicitSort
default:
return CatalogRequest{}, fmt.Errorf("unsupported catalog source %q", req.Source)
}
+48 -1
View File
@@ -1,9 +1,56 @@
package catalog
import "testing"
import (
"net/url"
"testing"
)
func TestParseCatalogMediaScope_AllowsEbook(t *testing.T) {
if got := parseCatalogMediaScope(" ebook "); got != "ebook" {
t.Fatalf("expected ebook media scope, got %q", got)
}
}
func TestParseCatalogMediaScope_AllowsManga(t *testing.T) {
if got := parseCatalogMediaScope(" manga "); got != "manga" {
t.Fatalf("expected manga media scope, got %q", got)
}
}
func TestParseCatalogRequest_AllowsCollectionOverlayParams(t *testing.T) {
values := url.Values{
"source": {"user_collection"},
"collection_id": {"col-7"},
"type": {"movie"},
"sort": {"title"},
"order": {"asc"},
"groups[0][match]": {"all"},
"groups[0][rules][0][field]": {"watched"},
"groups[0][rules][0][op]": {"is"},
"groups[0][rules][0][value]": {"false"},
"groups[1][match]": {"all"},
"groups[1][rules][0][field]": {"resolution"},
"groups[1][rules][0][op]": {"is"},
"groups[1][rules][0][value]": {"4K"},
}
req, err := ParseCatalogRequest(values)
if err != nil {
t.Fatalf("ParseCatalogRequest returned error: %v", err)
}
if req.Source != CatalogSourceUserCollection || req.CollectionID != "col-7" {
t.Fatalf("unexpected collection source: source=%q collection=%q", req.Source, req.CollectionID)
}
if req.UseSourceOrder {
t.Fatal("explicit sort should opt out of source order")
}
if req.Query.MediaScope != "movie" {
t.Fatalf("media scope = %q, want movie", req.Query.MediaScope)
}
if req.Query.Sort != (QuerySort{Field: "title", Order: "asc"}) {
t.Fatalf("sort = %+v, want title asc", req.Query.Sort)
}
if len(req.Query.Groups) != 2 {
t.Fatalf("groups = %+v, want two overlay groups", req.Query.Groups)
}
}
+318 -19
View File
@@ -27,6 +27,16 @@ type CatalogResult struct {
HasMore bool
TotalExact bool
SnapshotAt time.Time // pagination fence timestamp
// Provider, Mode, SemanticUsed, FallbackReason and IndexPendingEvents are
// per-query search diagnostics. They are only populated on the direct-search
// path (where a CatalogSearchProvider actually ran); browse / preview /
// grouped paths leave them zero-valued so the handler omits
// search_diagnostics.
Provider string
Mode string
SemanticUsed bool
FallbackReason string
IndexPendingEvents int
}
type CatalogFiltersResult struct {
@@ -137,11 +147,12 @@ type previewExecutor interface {
}
type CatalogResolver struct {
browseRepo *BrowseRepository
itemRepo *ItemRepository
episodeRepo *EpisodeRepository
storeProvider userstore.UserStoreProvider
facets facetFetcher
browseRepo *BrowseRepository
itemRepo *ItemRepository
episodeRepo *EpisodeRepository
searchProvider CatalogSearchProvider
storeProvider userstore.UserStoreProvider
facets facetFetcher
// previewExecutorForScope, when non-nil, is used by previewQuerySource
// instead of the default queryExecutorForScope. Tests inject a stub here
// to observe how many times the executor is asked for a result page.
@@ -150,8 +161,9 @@ type CatalogResolver struct {
func NewCatalogResolver(browseRepo *BrowseRepository, itemRepo *ItemRepository) *CatalogResolver {
r := &CatalogResolver{
browseRepo: browseRepo,
itemRepo: itemRepo,
browseRepo: browseRepo,
itemRepo: itemRepo,
searchProvider: NewPostgresSearchProvider(itemRepo),
}
if browseRepo != nil {
r.facets = &pgxFacetFetcher{pool: browseRepo.pool}
@@ -175,6 +187,16 @@ func (r *CatalogResolver) WithEpisodeRepository(repo *EpisodeRepository) *Catalo
return r
}
func (r *CatalogResolver) WithSearchProvider(provider CatalogSearchProvider) *CatalogResolver {
if r == nil {
return nil
}
if provider != nil {
r.searchProvider = provider
}
return r
}
func (r *CatalogResolver) queryExecutorForScope(scope string, snapshot *time.Time) *QueryExecutor {
return &QueryExecutor{
Pool: r.itemRepo.pool,
@@ -200,12 +222,12 @@ func (r *CatalogResolver) Resolve(ctx context.Context, req CatalogRequest, acces
}
return r.resolveSectionSource(ctx, req, access)
case CatalogSourceLibraryCollection:
if err := validateCatalogExactCollectionRequest(req); err != nil {
if err := validateCatalogCollectionRequest(req, strings.TrimSpace(access.ProfileID) != ""); err != nil {
return nil, err
}
return r.resolveLibraryCollectionSource(ctx, req, access)
case CatalogSourceUserCollection:
if err := validateCatalogExactCollectionRequest(req); err != nil {
if err := validateCatalogCollectionRequest(req, strings.TrimSpace(access.ProfileID) != ""); err != nil {
return nil, err
}
return r.resolveUserCollectionSource(ctx, req, access)
@@ -260,16 +282,32 @@ func (r *CatalogResolver) resolveDirectSearchSource(ctx context.Context, req Cat
return &CatalogResult{Items: []*models.MediaItem{}, Total: 0, HasMore: false, TotalExact: true}, nil
}
items, total, err := r.itemRepo.Search(ctx, req.SearchQuery, itemTypes, req.Limit, req.Offset, searchAccess)
provider := r.searchProvider
if provider == nil {
provider = NewPostgresSearchProvider(r.itemRepo)
}
result, err := provider.Search(ctx, CatalogSearchRequest{
Query: req.SearchQuery,
ItemTypes: itemTypes,
Limit: req.Limit,
Offset: req.Offset,
Access: searchAccess,
SkipTotal: req.SkipTotal,
})
if err != nil {
return nil, fmt.Errorf("searching catalog items: %w", err)
}
return &CatalogResult{
Items: items,
Total: total,
HasMore: total > req.Offset+len(items),
TotalExact: true,
Items: result.Items,
Total: result.Total,
HasMore: result.HasMore,
TotalExact: result.TotalExact,
Provider: result.Provider,
Mode: result.Mode,
SemanticUsed: result.SemanticUsed,
FallbackReason: result.FallbackReason,
IndexPendingEvents: result.IndexPendingEvents,
}, nil
}
@@ -467,6 +505,13 @@ func (r *CatalogResolver) resolveLiveLibraryCollectionSource(ctx context.Context
def.LibraryIDs = intersectCatalogDefinitionLibraries(def.LibraryIDs, []int{collection.LibraryID})
}
def = ApplySmartCollectionItemLimit(def)
if catalogRequestHasOverlay(req) {
items, err := r.resolveCollectionQueryBaseItems(ctx, def, stripCatalogUserScope(access))
if err != nil {
return nil, err
}
return r.resolveExactOrderedMediaItems(ctx, items, req, access)
}
return r.resolveQuerySource(ctx, CatalogRequest{
Source: CatalogSourceQuery,
Query: def,
@@ -487,12 +532,23 @@ func (r *CatalogResolver) resolveUserCollectionSource(ctx context.Context, req C
return nil, ErrCatalogSourceNotFound
}
if strings.EqualFold(strings.TrimSpace(collection.CollectionType), "smart") || catalogCollectionUsesLiveQuery([]byte(collection.QueryDefinition)) {
if IsLiveQueryType(collection.CollectionType) {
def, err := parseCatalogCollectionQueryDefinition([]byte(collection.QueryDefinition))
if err != nil {
return nil, fmt.Errorf("%w: parsing user collection query_definition: %v", ErrInvalidCatalogRequest, err)
}
def = ApplySmartCollectionItemLimit(def)
if catalogRequestHasOverlay(req) || strings.TrimSpace(collection.DisplayQueryDefinition) != "" {
items, err := r.resolveCollectionQueryBaseItems(ctx, def, access)
if err != nil {
return nil, err
}
items, err = FilterCollectionItemsByDisplayQuery(ctx, r.itemRepo.pool, items, collection.DisplayQueryDefinition, access)
if err != nil {
return nil, err
}
return r.resolveExactOrderedMediaItems(ctx, items, req, access)
}
return r.resolveQuerySource(ctx, CatalogRequest{
Source: CatalogSourceQuery,
Query: def,
@@ -510,7 +566,15 @@ func (r *CatalogResolver) resolveUserCollectionSource(ctx context.Context, req C
for _, item := range items {
contentIDs = append(contentIDs, item.MediaItemID)
}
return r.resolveExactOrderedItems(ctx, contentIDs, req, access)
mediaItems, err := r.fetchAccessibleItemsByID(ctx, contentIDs, catalogBaseCollectionRequest(req), access)
if err != nil {
return nil, err
}
mediaItems, err = FilterCollectionItemsByDisplayQuery(ctx, r.itemRepo.pool, mediaItems, collection.DisplayQueryDefinition, access)
if err != nil {
return nil, err
}
return r.resolveExactOrderedMediaItems(ctx, mediaItems, req, access)
}
func (r *CatalogResolver) resolvePersonalSource(ctx context.Context, req CatalogRequest, access AccessFilter) (*CatalogResult, error) {
@@ -548,15 +612,27 @@ func (r *CatalogResolver) resolvePersonSource(ctx context.Context, req CatalogRe
}
func (r *CatalogResolver) resolveExactOrderedItems(ctx context.Context, contentIDs []string, req CatalogRequest, access AccessFilter) (*CatalogResult, error) {
items, err := r.fetchAccessibleItemsByID(ctx, contentIDs, req, access)
fetchReq := req
if req.Source == CatalogSourceLibraryCollection || req.Source == CatalogSourceUserCollection {
fetchReq = catalogBaseCollectionRequest(req)
}
items, err := r.fetchAccessibleItemsByID(ctx, contentIDs, fetchReq, access)
if err != nil {
return nil, err
}
return r.resolveExactOrderedMediaItems(ctx, items, req, access)
}
func (r *CatalogResolver) resolveExactOrderedMediaItems(ctx context.Context, items []*models.MediaItem, req CatalogRequest, access AccessFilter) (*CatalogResult, error) {
req = normalizeExactCollectionOverlayRequest(req, items)
items = filterCatalogSearchItems(items, req.SearchQuery)
items = filterCatalogNamePrefix(items, req.NamePrefix)
if req.UseSourceOrder {
items = filterCatalogItems(items, req.Query)
var err error
items, err = r.filterExactSourceItemsByQuery(ctx, items, req.Query, access)
if err != nil {
return nil, err
}
total := len(items)
paged := paginateCatalogItems(items, req.Offset, req.Limit)
return &CatalogResult{
@@ -569,6 +645,83 @@ func (r *CatalogResolver) resolveExactOrderedItems(ctx context.Context, contentI
return r.resolveCandidateItemsWithQuery(ctx, req, access, items, false)
}
func normalizeExactCollectionOverlayRequest(req CatalogRequest, items []*models.MediaItem) CatalogRequest {
if req.Source != CatalogSourceLibraryCollection && req.Source != CatalogSourceUserCollection {
return req
}
if !isEpisodeCatalogScope(req.Query.MediaScope) {
return req
}
for _, item := range items {
if item != nil && item.Type == "episode" {
return req
}
}
req.Query.MediaScope = ""
return req
}
func (r *CatalogResolver) resolveCollectionQueryBaseItems(ctx context.Context, def QueryDefinition, access AccessFilter) ([]*models.MediaItem, error) {
limit := DefaultSmartCollectionItemLimit
if def.Limit != nil && *def.Limit > 0 {
limit = *def.Limit
}
executor := r.queryExecutorForScope(def.MediaScope, nil)
items, _, err := executor.Preview(ctx, def, access, limit)
if err != nil {
return nil, err
}
return items, nil
}
func (r *CatalogResolver) filterExactSourceItemsByQuery(ctx context.Context, items []*models.MediaItem, def QueryDefinition, access AccessFilter) ([]*models.MediaItem, error) {
if !catalogQueryHasFilter(def) || len(items) == 0 {
return items, nil
}
ids := make([]string, 0, len(items))
for _, item := range items {
if item != nil && strings.TrimSpace(item.ContentID) != "" {
ids = append(ids, item.ContentID)
}
}
if len(ids) == 0 {
return []*models.MediaItem{}, nil
}
filterDef := def
filterDef.Sort = QuerySort{}
filterDef.Limit = nil
queryAccess := access
queryAccess.AllowedContentIDs = ids
executor := r.queryExecutorForScope(filterDef.MediaScope, nil)
matched, _, err := executor.Preview(ctx, filterDef, queryAccess, len(ids))
if err != nil {
return nil, err
}
matchedIDs := make(map[string]struct{}, len(matched))
for _, item := range matched {
if item != nil {
matchedIDs[item.ContentID] = struct{}{}
}
}
filtered := make([]*models.MediaItem, 0, len(items))
for _, item := range items {
if item == nil {
continue
}
if _, ok := matchedIDs[item.ContentID]; ok {
filtered = append(filtered, item)
}
}
if def.Limit != nil && *def.Limit > 0 && len(filtered) > *def.Limit {
filtered = filtered[:*def.Limit]
}
return filtered, nil
}
func (r *CatalogResolver) resolveCandidateItemsWithQuery(
ctx context.Context,
req CatalogRequest,
@@ -698,6 +851,19 @@ func (r *CatalogResolver) ListFiltersWithOptions(ctx context.Context, req Catalo
return nil, err
}
filters.PersonID = req.PersonID
case CatalogSourceLibraryCollection, CatalogSourceUserCollection:
if err := validateCatalogCollectionRequest(req, strings.TrimSpace(access.ProfileID) != ""); err != nil {
return nil, err
}
contentIDs, err := r.loadCollectionSourceIDs(ctx, req, access)
if err != nil {
return nil, err
}
filters, earlyEmpty, err = catalogBrowseFilters(req, access)
if err != nil {
return nil, err
}
filters.ContentIDs = contentIDs
default:
return nil, fmt.Errorf("%w: source %q is not supported", ErrInvalidCatalogRequest, req.Source)
}
@@ -789,6 +955,19 @@ func (r *CatalogResolver) SearchFacet(ctx context.Context, req CatalogRequest, a
return nil, err
}
filters.PersonID = req.PersonID
case CatalogSourceLibraryCollection, CatalogSourceUserCollection:
if err := validateCatalogCollectionRequest(req, strings.TrimSpace(access.ProfileID) != ""); err != nil {
return nil, err
}
contentIDs, err := r.loadCollectionSourceIDs(ctx, req, access)
if err != nil {
return nil, err
}
filters, earlyEmpty, err = catalogBrowseFilters(req, access)
if err != nil {
return nil, err
}
filters.ContentIDs = contentIDs
default:
return nil, fmt.Errorf("%w: source %q is not supported", ErrInvalidCatalogRequest, req.Source)
}
@@ -1088,9 +1267,30 @@ func validateCatalogExactCollectionRequest(req CatalogRequest) error {
return nil
}
func validateCatalogCollectionRequest(req CatalogRequest, allowPersonalizedSorts bool) error {
if err := validateCatalogExactCollectionRequest(req); err != nil {
return err
}
return validateCatalogOverlayQuery(req.SearchQuery, req.Query, catalogPersonalRuleFields, QuerySortFieldSet(allowPersonalizedSorts), false)
}
func catalogRequestHasOverlay(req CatalogRequest) bool {
return strings.TrimSpace(req.SearchQuery) != "" ||
strings.TrimSpace(req.NamePrefix) != "" ||
catalogQueryHasFilter(req.Query) ||
strings.TrimSpace(req.Query.Sort.Field) != ""
}
func catalogQueryHasFilter(def QueryDefinition) bool {
return strings.TrimSpace(def.MediaScope) != "" ||
len(def.LibraryIDs) > 0 ||
len(def.Groups) > 0 ||
def.Limit != nil
}
func validateCatalogOverlayQuery(searchQuery string, def QueryDefinition, ruleFields, sortFields map[string]bool, allowRelevance bool) error {
if !IsValidMediaScope(def.MediaScope) {
return fmt.Errorf("%w: media_scope must be 'movie', 'series', 'episode', 'audiobook', 'ebook', or 'video'", ErrInvalidCatalogRequest)
return fmt.Errorf("%w: media_scope must be 'movie', 'series', 'episode', 'audiobook', 'ebook', 'manga', or 'video'", ErrInvalidCatalogRequest)
}
if def.Match != "" && def.Match != "all" && def.Match != "any" {
return fmt.Errorf("%w: match must be 'all' or 'any'", ErrInvalidCatalogRequest)
@@ -1502,6 +1702,105 @@ func (r *CatalogResolver) loadPersonalSourceIDs(ctx context.Context, store users
}
}
func (r *CatalogResolver) loadCollectionSourceIDs(ctx context.Context, req CatalogRequest, access AccessFilter) ([]string, error) {
items, err := r.loadCollectionSourceBaseItems(ctx, req, access)
if err != nil {
return nil, err
}
return contentIDsFromMediaItems(items), nil
}
func (r *CatalogResolver) loadCollectionSourceBaseItems(ctx context.Context, req CatalogRequest, access AccessFilter) ([]*models.MediaItem, error) {
switch req.Source {
case CatalogSourceLibraryCollection:
collectionRepo := NewLibraryCollectionRepository(r.itemRepo.pool)
collection, err := collectionRepo.GetByID(ctx, req.CollectionID)
if err != nil || collection.Visibility != "visible" {
return nil, ErrCatalogSourceNotFound
}
if IsLiveQueryType(collection.CollectionType) || catalogCollectionUsesLiveQuery(collection.QueryDefinition) {
def, err := parseCatalogCollectionQueryDefinition(collection.QueryDefinition)
if err != nil {
return nil, fmt.Errorf("%w: parsing library collection query_definition: %v", ErrInvalidCatalogRequest, err)
}
if len(collection.LibraryIDs) > 0 {
def.LibraryIDs = intersectCatalogDefinitionLibraries(def.LibraryIDs, collection.LibraryIDs)
} else if collection.LibraryID > 0 {
def.LibraryIDs = intersectCatalogDefinitionLibraries(def.LibraryIDs, []int{collection.LibraryID})
}
return r.resolveCollectionQueryBaseItems(ctx, ApplySmartCollectionItemLimit(def), stripCatalogUserScope(access))
}
collectionItems, err := collectionRepo.ListItems(ctx, collection.ID)
if err != nil {
return nil, err
}
contentIDs := make([]string, 0, len(collectionItems))
for _, item := range collectionItems {
contentIDs = append(contentIDs, item.MediaItemID)
}
return r.fetchAccessibleItemsByID(ctx, contentIDs, catalogBaseCollectionRequest(req), access)
case CatalogSourceUserCollection:
store, err := r.catalogStoreForAccess(ctx, access)
if err != nil {
return nil, err
}
collection, err := store.GetCollection(ctx, req.CollectionID)
if err != nil || !catalogProfileCanAccessCollection(collection, access.ProfileID) {
return nil, ErrCatalogSourceNotFound
}
var items []*models.MediaItem
if IsLiveQueryType(collection.CollectionType) {
def, err := parseCatalogCollectionQueryDefinition([]byte(collection.QueryDefinition))
if err != nil {
return nil, fmt.Errorf("%w: parsing user collection query_definition: %v", ErrInvalidCatalogRequest, err)
}
items, err = r.resolveCollectionQueryBaseItems(ctx, ApplySmartCollectionItemLimit(def), access)
if err != nil {
return nil, err
}
} else {
collectionItems, err := store.ListCollectionItems(ctx, collection.ID)
if err != nil {
return nil, err
}
contentIDs := make([]string, 0, len(collectionItems))
for _, item := range collectionItems {
contentIDs = append(contentIDs, item.MediaItemID)
}
items, err = r.fetchAccessibleItemsByID(ctx, contentIDs, catalogBaseCollectionRequest(req), access)
if err != nil {
return nil, err
}
}
return FilterCollectionItemsByDisplayQuery(ctx, r.itemRepo.pool, items, collection.DisplayQueryDefinition, access)
default:
return nil, fmt.Errorf("%w: source %q is not a collection source", ErrInvalidCatalogRequest, req.Source)
}
}
func catalogBaseCollectionRequest(req CatalogRequest) CatalogRequest {
return CatalogRequest{
Source: req.Source,
CollectionID: req.CollectionID,
Limit: req.Limit,
Offset: req.Offset,
SkipTotal: req.SkipTotal,
UseSourceOrder: true,
}
}
func contentIDsFromMediaItems(items []*models.MediaItem) []string {
ids := make([]string, 0, len(items))
for _, item := range items {
if item == nil || strings.TrimSpace(item.ContentID) == "" {
continue
}
ids = append(ids, item.ContentID)
}
return ids
}
func (r *CatalogResolver) fetchAccessibleItemsByID(ctx context.Context, contentIDs []string, req CatalogRequest, access AccessFilter) ([]*models.MediaItem, error) {
if len(contentIDs) == 0 {
return []*models.MediaItem{}, nil
+113
View File
@@ -54,6 +54,21 @@ func TestValidateCatalogQueryRequest_AllowsEbookMediaScope(t *testing.T) {
}
}
func TestValidateCatalogQueryRequest_AllowsMangaMediaScope(t *testing.T) {
req := CatalogRequest{
Source: CatalogSourceQuery,
Query: QueryDefinition{
MediaScope: "manga",
Match: "all",
Sort: QuerySort{Field: "title", Order: "asc"},
},
}
if err := validateCatalogQueryRequest(req, true); err != nil {
t.Fatalf("expected manga media scope to be accepted, got %v", err)
}
}
func TestValidateCatalogQueryRequest_AllowsAddedAtFilter(t *testing.T) {
req := CatalogRequest{
Source: CatalogSourceQuery,
@@ -354,6 +369,104 @@ func newTestResolver(exec previewExecutor) *CatalogResolver {
}
}
// fakeSearchProvider is a CatalogSearchProvider stub that returns a fixed
// CatalogSearchResult so resolveDirectSearchSource's field plumbing can be
// exercised without a database or live search backend.
type fakeSearchProvider struct {
result *CatalogSearchResult
}
func (f *fakeSearchProvider) Search(_ context.Context, _ CatalogSearchRequest) (*CatalogSearchResult, error) {
return f.result, nil
}
// TestResolveDirectSearchSource_PlumbsDiagnostics asserts that the four
// diagnostics fields (Provider, Mode, SemanticUsed, FallbackReason) carried by
// the provider's CatalogSearchResult are copied onto the resolver's
// CatalogResult on the direct-search path. Covers both the downgraded-to-keyword
// case (semantic_not_ready) and the hybrid-survived case.
func TestResolveDirectSearchSource_PlumbsDiagnostics(t *testing.T) {
cases := []struct {
name string
result *CatalogSearchResult
}{
{
name: "keyword fallback carries reason",
result: &CatalogSearchResult{
Items: []*models.MediaItem{},
Provider: SearchProviderMeilisearch,
Mode: "keyword",
SemanticUsed: false,
FallbackReason: `semantic_not_ready: type "movie" coverage 40% below threshold`,
},
},
{
name: "hybrid survived",
result: &CatalogSearchResult{
Items: []*models.MediaItem{},
Provider: SearchProviderMeilisearch,
Mode: "hybrid",
SemanticUsed: true,
IndexPendingEvents: 7,
},
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
resolver := &CatalogResolver{
searchProvider: &fakeSearchProvider{result: tc.result},
}
got, err := resolver.resolveDirectSearchSource(
context.Background(),
CatalogRequest{Source: CatalogSourceQuery, SearchQuery: "dune", Limit: 20},
AccessFilter{},
)
if err != nil {
t.Fatalf("resolveDirectSearchSource error: %v", err)
}
if got.Provider != tc.result.Provider {
t.Fatalf("Provider = %q, want %q", got.Provider, tc.result.Provider)
}
if got.Mode != tc.result.Mode {
t.Fatalf("Mode = %q, want %q", got.Mode, tc.result.Mode)
}
if got.SemanticUsed != tc.result.SemanticUsed {
t.Fatalf("SemanticUsed = %v, want %v", got.SemanticUsed, tc.result.SemanticUsed)
}
if got.FallbackReason != tc.result.FallbackReason {
t.Fatalf("FallbackReason = %q, want %q", got.FallbackReason, tc.result.FallbackReason)
}
if got.IndexPendingEvents != tc.result.IndexPendingEvents {
t.Fatalf("IndexPendingEvents = %d, want %d", got.IndexPendingEvents, tc.result.IndexPendingEvents)
}
})
}
}
// TestResolveDirectSearchSource_EarlyEmptyOmitsDiagnostics asserts that the
// early-empty path (no accessible libraries) returns a zero-valued Provider so
// the handler omits search_diagnostics for it.
func TestResolveDirectSearchSource_EarlyEmptyOmitsDiagnostics(t *testing.T) {
resolver := &CatalogResolver{
searchProvider: &fakeSearchProvider{result: &CatalogSearchResult{Provider: SearchProviderMeilisearch}},
}
got, err := resolver.resolveDirectSearchSource(
context.Background(),
CatalogRequest{Source: CatalogSourceQuery, SearchQuery: "dune", Limit: 20},
// AllowedLibraryIDs empty (non-nil) => effectiveCatalogLibraryIDs early-empties.
AccessFilter{AllowedLibraryIDs: []int{}},
)
if err != nil {
t.Fatalf("resolveDirectSearchSource error: %v", err)
}
if got.Provider != "" {
t.Fatalf("early-empty Provider = %q, want empty", got.Provider)
}
}
// TestPreviewQuerySource_NamePrefix_DoesNotFetchAllRows asserts that the
// preview path makes a single PreviewPage call with NamePrefix forwarded into
// AccessFilter, instead of the previous fetch-all + Go-side filter pattern
+32
View File
@@ -17,3 +17,35 @@ func IsSyncableType(collectionType string) bool {
return false
}
}
// IntersectCollectionLibraryIDs narrows a smart collection's query library scope
// (existing) to the libraries the collection is bound to (required), returning a
// copy of the other set when one side is empty. Shared by the web API and the
// Jellyfin-compat collection-item resolvers so their scoping stays identical.
func IntersectCollectionLibraryIDs(existing, required []int) []int {
if len(required) == 0 {
return append([]int(nil), existing...)
}
if len(existing) == 0 {
return append([]int(nil), required...)
}
allowed := make(map[int]struct{}, len(required))
for _, id := range required {
allowed[id] = struct{}{}
}
result := make([]int, 0, len(existing))
seen := make(map[int]struct{}, len(existing))
for _, id := range existing {
if _, ok := allowed[id]; !ok {
continue
}
if _, ok := seen[id]; ok {
continue
}
seen[id] = struct{}{}
result = append(result, id)
}
return result
}
+373 -41
View File
@@ -10,6 +10,7 @@ import (
"sort"
"strconv"
"strings"
"sync"
"time"
"github.com/Silo-Server/silo-server/internal/access"
@@ -61,6 +62,26 @@ type expiringImageResolver interface {
ResolveImageURLsWithExpiry(ctx context.Context, paths []string, variant string) map[string]ResolvedImageURL
}
type WorkSummaryProvider interface {
GetSummaryForContentID(ctx context.Context, contentID string, filter AccessFilter) (*WorkSummary, error)
}
type WorkSummaryBatchProvider interface {
ListSummariesForContentIDs(ctx context.Context, contentIDs []string, filter AccessFilter) (map[string]*WorkSummary, error)
}
type WorkSummary struct {
WorkID string
Title string
Formats []WorkFormatSummary
}
type WorkFormatSummary struct {
Type string `json:"type"`
ContentID string `json:"content_id"`
LibraryID int `json:"library_id,omitempty"`
}
// ItemDetail is the full detail response for a single media item, including
// metadata, file versions, subtitles, intro/credits markers, and presigned image URLs.
type ItemDetail struct {
@@ -108,6 +129,11 @@ type ItemDetail struct {
BackdropThumbhash string `json:"backdrop_thumbhash,omitempty"`
LogoURL string `json:"logo_url,omitempty"`
// Literary work linkage. Present for linked ebook/audiobook items.
WorkID string `json:"work_id,omitempty"`
WorkTitle string `json:"work_title,omitempty"`
WorkFormats []WorkFormatSummary `json:"work_formats,omitempty"`
// Series-specific.
SeasonCount *int `json:"season_count,omitempty"`
@@ -164,6 +190,9 @@ type ItemDetail struct {
// Ebook-specific detail. Present only when Type == "ebook".
Ebook *EbookDetailExtension `json:"ebook,omitempty"`
// Manga-specific detail. Present only when Type == "manga".
Manga *MangaDetailExtension `json:"manga,omitempty"`
}
type AudiobookDetailExtension struct {
@@ -215,6 +244,29 @@ type EbookDetailExtension struct {
Related AudiobookRelatedContent `json:"related"`
}
// MangaDetailExtension is the manga-series detail payload. A manga series item
// (media_items.type='manga') owns a set of readable chapter items
// (media_items.type='ebook') linked via the manga_chapters table.
type MangaDetailExtension struct {
Chapters []MangaChapter `json:"chapters"`
}
// MangaChapter is one chapter of a manga series, ordered by chapter index.
type MangaChapter struct {
ContentID string `json:"content_id"`
Title string `json:"title"`
ChapterIndex *float64 `json:"chapter_index,omitempty"`
Volume string `json:"volume,omitempty"`
// Read is true when the current viewer has finished this chapter, mirroring
// ebook read state: ebook_reader_progress.progress >= the finished threshold.
Read bool `json:"read"`
// Progress is the viewer's reading position as a 0..1 fraction, present
// only when a progress row exists. The row progress bar uses it.
Progress *float64 `json:"progress,omitempty"`
// PosterURL is the chapter's extracted cover (presigned), for row thumbnails.
PosterURL string `json:"poster_url,omitempty"`
}
// ItemUserState is per-profile viewer state included in item detail responses.
type ItemUserState struct {
Played bool `json:"played"`
@@ -497,6 +549,7 @@ type DetailService struct {
groupClaimRepo *GroupClaimRepository
imageResolver ImageResolver
userStoreProvider userstore.UserStoreProvider
workSummary WorkSummaryProvider
originalLangFn func(context.Context, string) string
probeEnsurer PlaybackProbeEnsurer
chapterThumbs ChapterThumbnailQueuer
@@ -533,6 +586,12 @@ func (s *DetailService) SetUserStoreProvider(provider userstore.UserStoreProvide
s.userStoreProvider = provider
}
func (s *DetailService) SetWorkSummaryProvider(provider WorkSummaryProvider) {
if s != nil {
s.workSummary = provider
}
}
func (s *DetailService) SetProbeEnsurer(ensurer PlaybackProbeEnsurer) {
s.probeEnsurer = ensurer
}
@@ -657,6 +716,53 @@ func (s *DetailService) LocalizeItemModel(ctx context.Context, item *models.Medi
return applyItemLocalization(item, loc), nil
}
// LocalizeItemModels applies presentation-language localization to a batch of
// media items with a single lookup. Items without a localization row are
// returned unchanged; the result preserves input order and length.
func (s *DetailService) LocalizeItemModels(ctx context.Context, items []*models.MediaItem, filter AccessFilter) ([]*models.MediaItem, error) {
if len(items) == 0 {
return items, nil
}
localized := make([]*models.MediaItem, len(items))
for i, item := range items {
localized[i] = cloneMediaItem(item)
}
language, err := s.resolvePresentationLanguage(ctx, filter)
if err != nil || language == "" || s.itemLocRepo == nil {
return localized, err
}
ids := make([]string, 0, len(items))
seen := make(map[string]struct{}, len(items))
for _, item := range items {
if item == nil || item.ContentID == "" || sameMetadataLanguage(item.DefaultMetadataLanguage, language) {
continue
}
if _, ok := seen[item.ContentID]; ok {
continue
}
seen[item.ContentID] = struct{}{}
ids = append(ids, item.ContentID)
}
if len(ids) == 0 {
return localized, nil
}
locs, err := s.itemLocRepo.GetByContentIDs(ctx, ids, language)
if err != nil || len(locs) == 0 {
return localized, err
}
for i, item := range items {
if item == nil {
continue
}
if loc := locs[item.ContentID]; loc != nil {
localized[i] = applyItemLocalization(item, loc)
}
}
return localized, nil
}
func (s *DetailService) LocalizeSeasonModel(ctx context.Context, season *models.Season, filter AccessFilter) (*models.Season, error) {
if season == nil {
return nil, nil
@@ -960,6 +1066,16 @@ func (s *DetailService) buildMediaItemDetail(ctx context.Context, item *models.M
}
if item.Type == "ebook" {
detail.Ebook = s.buildEbookExtension(ctx, item, crewCredits, filter)
// A manga chapter is an ebook item linked to its series; exposing the
// linkage lets the reader navigate back/next within the series and
// continue-reading cards show the series instead of the chapter.
if seriesID, seriesTitle, ok := s.lookupMangaSeriesForChapter(ctx, item.ContentID); ok {
detail.SeriesID = seriesID
detail.SeriesTitle = seriesTitle
}
}
if item.Type == "manga" {
detail.Manga = s.buildMangaExtension(ctx, item, filter)
}
// Series folder paths from confirmed claims when available, otherwise from
@@ -999,9 +1115,23 @@ func (s *DetailService) buildMediaItemDetail(ctx context.Context, item *models.M
}
}
applyWorkSummary(ctx, detail, s.workSummary, filter)
return detail, nil
}
func applyWorkSummary(ctx context.Context, detail *ItemDetail, provider WorkSummaryProvider, filter AccessFilter) {
if detail == nil || provider == nil || detail.ContentID == "" {
return
}
summary, err := provider.GetSummaryForContentID(ctx, detail.ContentID, filter)
if err != nil || summary == nil {
return
}
detail.WorkID = summary.WorkID
detail.WorkTitle = summary.Title
detail.WorkFormats = summary.Formats
}
// personCredits converts ItemPerson slice to PersonCredit slice with presigned URLs.
func (s *DetailService) personCredits(ctx context.Context, people []models.ItemPerson) []PersonCredit {
credits := make([]PersonCredit, 0, len(people))
@@ -1088,16 +1218,32 @@ func (s *DetailService) buildAudiobookExtension(
}
}
// The four related-content lookups are independent read-only queries;
// run them concurrently so detail latency is the slowest one, not their sum.
var (
series *AudiobookSeriesGroup
otherNarr []AudiobookNarration
alsoByAuthor []AudiobookRelatedItem
similar []AudiobookRelatedItem
wg sync.WaitGroup
)
wg.Add(4)
go func() { defer wg.Done(); series = s.fetchAudiobookSeries(ctx, item.ContentID, filter) }()
go func() { defer wg.Done(); otherNarr = s.fetchAudiobookOtherNarrations(ctx, item.ContentID, filter) }()
go func() { defer wg.Done(); alsoByAuthor = s.fetchAudiobookAlsoByAuthor(ctx, item.ContentID, filter) }()
go func() { defer wg.Done(); similar = s.fetchAudiobookSimilarByGenres(ctx, item.ContentID, filter) }()
wg.Wait()
return &AudiobookDetailExtension{
Authors: audiobookPeopleFromCrew(crew, models.PersonKindAuthor.String()),
Narrators: audiobookPeopleFromCrew(crew, models.PersonKindNarrator.String()),
Publisher: firstNonEmptyString(item.Studios),
TotalDurationSeconds: totalDuration,
Series: s.fetchAudiobookSeries(ctx, item.ContentID, filter),
OtherNarrations: s.fetchAudiobookOtherNarrations(ctx, item.ContentID, filter),
Series: series,
OtherNarrations: otherNarr,
Related: AudiobookRelatedContent{
AlsoByAuthor: s.fetchAudiobookAlsoByAuthor(ctx, item.ContentID, filter),
Similar: s.fetchAudiobookSimilarByGenres(ctx, item.ContentID, filter),
AlsoByAuthor: alsoByAuthor,
Similar: similar,
},
}
}
@@ -1111,17 +1257,117 @@ func (s *DetailService) buildEbookExtension(
if item == nil {
return nil
}
// The three related-content lookups are independent read-only queries; run
// them concurrently so detail latency is the slowest one, not their sum
// (mirrors buildAudiobookExtension).
var (
series *AudiobookSeriesGroup
alsoByAuthor []AudiobookRelatedItem
similar []AudiobookRelatedItem
wg sync.WaitGroup
)
wg.Add(3)
go func() { defer wg.Done(); series = s.fetchEbookSeries(ctx, item.ContentID, filter) }()
go func() { defer wg.Done(); alsoByAuthor = s.fetchEbookAlsoByAuthor(ctx, item.ContentID, filter) }()
go func() { defer wg.Done(); similar = s.fetchEbookSimilarByGenres(ctx, item.ContentID, filter) }()
wg.Wait()
return &EbookDetailExtension{
Authors: audiobookPeopleFromCrew(crew, models.PersonKindAuthor.String()),
Publisher: firstNonEmptyString(item.Studios),
Series: s.fetchEbookSeries(ctx, item.ContentID, filter),
Series: series,
Related: AudiobookRelatedContent{
AlsoByAuthor: s.fetchEbookAlsoByAuthor(ctx, item.ContentID, filter),
Similar: s.fetchEbookSimilarByGenres(ctx, item.ContentID, filter),
AlsoByAuthor: alsoByAuthor,
Similar: similar,
},
}
}
// buildMangaExtension assembles the manga-series detail payload by listing the
// series' chapters (ebook items linked via manga_chapters).
func (s *DetailService) buildMangaExtension(ctx context.Context, item *models.MediaItem, filter AccessFilter) *MangaDetailExtension {
if item == nil {
return nil
}
return &MangaDetailExtension{
Chapters: s.fetchMangaChapters(ctx, item.ContentID, filter),
}
}
// mangaChaptersQuery is the SQL listing a manga series' chapters in reading
// order. Chapters with a parsed index sort first (ascending); those without
// fall back to sort_title. Kept as a package var so the ordering contract can
// be asserted without a database.
//
// Manga chapters are ebook items, so per-chapter read state mirrors the ebook
// surfaces: a chapter is read when the current viewer's ebook_reader_progress
// row has progress >= the finished threshold. The LEFT JOIN is scoped by the
// viewer's user_id + profile_id ($2/$3) and yields false when no row exists.
var mangaChaptersQuery = fmt.Sprintf(`
SELECT m.content_id, m.title, mc.chapter_index, mc.volume,
COALESCE(erp.progress >= %s, false) AS read,
erp.progress::double precision,
COALESCE(m.poster_path, '') AS poster_path
FROM manga_chapters mc
JOIN media_items m ON m.content_id = mc.chapter_content_id
LEFT JOIN ebook_reader_progress erp
ON erp.content_id = mc.chapter_content_id
AND erp.user_id = $2
AND erp.profile_id = $3
WHERE mc.series_content_id = $1
ORDER BY mc.chapter_index NULLS LAST, m.sort_title
`, EbookFinishedProgressThresholdSQL)
// fetchMangaChapters returns the ordered chapters for a manga series. It never
// returns nil so the JSON payload always carries a (possibly empty) array. The
// access filter supplies the viewer (user_id/profile_id) used to resolve each
// chapter's per-viewer read state.
func (s *DetailService) fetchMangaChapters(ctx context.Context, seriesContentID string, filter AccessFilter) []MangaChapter {
chapters := make([]MangaChapter, 0, 16)
if s == nil || s.itemRepo == nil || s.itemRepo.pool == nil {
return chapters
}
rows, err := s.itemRepo.pool.Query(ctx, mangaChaptersQuery, seriesContentID, filter.UserID, filter.ProfileID)
if err != nil {
return chapters
}
defer rows.Close()
posterPaths := make([]string, 0, 16)
for rows.Next() {
var (
ch MangaChapter
index *float64
volume *string
progress *float64
posterPath string
)
if err := rows.Scan(&ch.ContentID, &ch.Title, &index, &volume, &ch.Read, &progress, &posterPath); err != nil {
return chapters
}
ch.ChapterIndex = index
if volume != nil {
ch.Volume = *volume
}
ch.Progress = progress
ch.PosterURL = posterPath // raw path; resolved in one batch below
if posterPath != "" {
posterPaths = append(posterPaths, posterPath)
}
chapters = append(chapters, ch)
}
if err := rows.Err(); err != nil {
slog.Warn("manga chapters: row iteration error", "series", seriesContentID, "error", err)
}
// Presign every chapter poster in one batch rather than per chapter — a
// long-running series has hundreds of chapters.
resolved := s.PresignImageURLs(ctx, posterPaths, "poster", "")
for i := range chapters {
chapters[i].PosterURL = resolved[chapters[i].PosterURL]
}
return chapters
}
func audiobookPeopleFromCrew(crew []CrewCredit, job string) []AudiobookPerson {
out := make([]AudiobookPerson, 0)
for _, credit := range crew {
@@ -1930,16 +2176,126 @@ func resolveSelectedAudioLanguage(
return ""
}
// audioPrefResolver memoizes the user-store lookups behind
// effectiveAudioSelection that are invariant across the files of a single
// detail request — the user store, the profile, the item's audio preference,
// and the resolved original language — plus per-library playback preferences.
// A multi-track item (e.g. a several-hundred-file audiobook) thus issues each
// of those queries once per request instead of once per file.
type audioPrefResolver struct {
svc *DetailService
store userstore.UserStore
valid bool
profileID string
contentID string
profileDone bool
profileLang string
audioDone bool
audioPref *playback.AudioTrackPreference
originalDone bool
originalLang string
libLang map[int]string
}
// newAudioPrefResolver resolves the per-user store once and prepares the
// memoization caches. The returned resolver is not safe for concurrent use; it
// is intended to be threaded through a single sequential file loop.
func (s *DetailService) newAudioPrefResolver(ctx context.Context, filter AccessFilter, audioPreferenceContentID string) *audioPrefResolver {
r := &audioPrefResolver{
svc: s,
profileID: filter.ProfileID,
contentID: audioPreferenceContentID,
libLang: map[int]string{},
}
if s.userStoreProvider == nil || filter.UserID == 0 || filter.ProfileID == "" {
return r
}
store, err := s.userStoreProvider.ForUser(ctx, filter.UserID)
if err != nil || store == nil {
return r
}
r.store = store
r.valid = true
return r
}
// audioPreference returns a fresh copy of the item's stored audio preference
// (or nil). A copy is returned each call so the caller can resolve the
// original-language sentinel in place without mutating the memoized value.
func (r *audioPrefResolver) audioPreference(ctx context.Context) *playback.AudioTrackPreference {
if !r.audioDone {
r.audioDone = true
if strings.TrimSpace(r.contentID) != "" {
if pref, prefErr := r.store.GetAudioPreference(ctx, r.profileID, r.contentID); prefErr == nil && pref != nil {
r.audioPref = &playback.AudioTrackPreference{
AudioTrackIndex: pref.AudioTrackIndex,
AudioLanguage: pref.AudioLanguage,
TrackSignature: pref.TrackSignature,
}
}
}
}
if r.audioPref == nil {
return nil
}
cp := *r.audioPref
return &cp
}
func (r *audioPrefResolver) profileLanguage(ctx context.Context) string {
if !r.profileDone {
r.profileDone = true
if profile, profileErr := r.store.GetProfile(ctx, r.profileID); profileErr == nil && profile != nil {
r.profileLang = strings.TrimSpace(profile.Language)
}
}
return r.profileLang
}
func (r *audioPrefResolver) libraryAudioLanguage(ctx context.Context, libraryID int) string {
if lang, ok := r.libLang[libraryID]; ok {
return lang
}
lang := ""
if pref, prefErr := r.store.GetLibraryPlaybackPreference(ctx, r.profileID, libraryID); prefErr == nil && pref != nil {
lang = strings.TrimSpace(pref.AudioLanguage)
}
r.libLang[libraryID] = lang
return lang
}
func (r *audioPrefResolver) originalLanguage(ctx context.Context) string {
if !r.originalDone {
r.originalDone = true
r.originalLang = r.svc.resolveOriginalLanguage(ctx, r.contentID)
}
return r.originalLang
}
func (s *DetailService) effectiveAudioSelection(
ctx context.Context,
filter AccessFilter,
audioPreferenceContentID string,
file *models.MediaFile,
) effectiveAudioSelection {
return s.effectiveAudioSelectionWith(ctx, s.newAudioPrefResolver(ctx, filter, audioPreferenceContentID), file)
}
// effectiveAudioSelectionWith computes the selection for one file using a
// resolver whose request-invariant lookups are shared across every file.
func (s *DetailService) effectiveAudioSelectionWith(
ctx context.Context,
r *audioPrefResolver,
file *models.MediaFile,
) effectiveAudioSelection {
if file == nil || len(file.AudioTracks) == 0 {
return effectiveAudioSelection{}
}
if s.userStoreProvider == nil || filter.UserID == 0 || filter.ProfileID == "" {
if r == nil || !r.valid {
index := playback.SelectAudioTrack(file.AudioTracks, "", nil)
return effectiveAudioSelection{
Index: index,
@@ -1947,42 +2303,19 @@ func (s *DetailService) effectiveAudioSelection(
}
}
store, err := s.userStoreProvider.ForUser(ctx, filter.UserID)
if err != nil || store == nil {
index := playback.SelectAudioTrack(file.AudioTracks, "", nil)
return effectiveAudioSelection{
Index: index,
Language: resolveSelectedAudioLanguage(file, index, "", false),
}
}
seriesPref := r.audioPreference(ctx)
var seriesPref *playback.AudioTrackPreference
if strings.TrimSpace(audioPreferenceContentID) != "" {
if pref, prefErr := store.GetAudioPreference(ctx, filter.ProfileID, audioPreferenceContentID); prefErr == nil && pref != nil {
seriesPref = &playback.AudioTrackPreference{
AudioTrackIndex: pref.AudioTrackIndex,
AudioLanguage: pref.AudioLanguage,
TrackSignature: pref.TrackSignature,
}
}
}
preferredLang := ""
if profile, profileErr := store.GetProfile(ctx, filter.ProfileID); profileErr == nil && profile != nil {
preferredLang = strings.TrimSpace(profile.Language)
}
preferredLang := r.profileLanguage(ctx)
libraryAudioLang := ""
if seriesPref == nil {
if pref, prefErr := store.GetLibraryPlaybackPreference(ctx, filter.ProfileID, file.MediaFolderID); prefErr == nil && pref != nil {
libraryAudioLang = strings.TrimSpace(pref.AudioLanguage)
}
libraryAudioLang = r.libraryAudioLanguage(ctx, file.MediaFolderID)
}
originalLanguage := ""
resolveOriginalLanguage := func() string {
if originalLanguage == "" {
originalLanguage = s.resolveOriginalLanguage(ctx, audioPreferenceContentID)
originalLanguage = r.originalLanguage(ctx)
}
return originalLanguage
}
@@ -2118,16 +2451,15 @@ func (s *DetailService) buildPlaybackInfo(
subtitleSet := make(map[string]SubtitleInfo)
var firstIntro, firstCredits, firstRecap, firstPreview *Marker
// Resolve the request-invariant audio preferences once; a multi-track item
// would otherwise re-query the profile/preference rows for every file.
audioResolver := s.newAudioPrefResolver(ctx, filter, audioPreferenceContentID)
for _, f := range files {
if f == nil {
continue
}
effectiveAudioSelection := s.effectiveAudioSelection(
ctx,
filter,
audioPreferenceContentID,
f,
)
effectiveAudioSelection := s.effectiveAudioSelectionWith(ctx, audioResolver, f)
versionIntro := markerFromRange(f.IntroStart, f.IntroEnd)
if versionIntro != nil && firstIntro == nil {
firstIntro = versionIntro
@@ -0,0 +1,68 @@
package catalog
import (
"context"
"sync/atomic"
"testing"
"github.com/Silo-Server/silo-server/internal/models"
"github.com/Silo-Server/silo-server/internal/userstore"
)
// countingUserStore wraps a real UserStore and counts the per-file preference
// lookups that effectiveAudioSelection performs, so a test can assert they are
// resolved once per detail request rather than once per media file.
type countingUserStore struct {
userstore.UserStore
getProfile int32
getAudioPref int32
getLibraryPref int32
}
func (c *countingUserStore) GetProfile(ctx context.Context, id string) (*userstore.Profile, error) {
atomic.AddInt32(&c.getProfile, 1)
return c.UserStore.GetProfile(ctx, id)
}
func (c *countingUserStore) GetAudioPreference(ctx context.Context, profileID, seriesID string) (*userstore.AudioPreference, error) {
atomic.AddInt32(&c.getAudioPref, 1)
return c.UserStore.GetAudioPreference(ctx, profileID, seriesID)
}
func (c *countingUserStore) GetLibraryPlaybackPreference(ctx context.Context, profileID string, libraryID int) (*userstore.LibraryPlaybackPreference, error) {
atomic.AddInt32(&c.getLibraryPref, 1)
return c.UserStore.GetLibraryPlaybackPreference(ctx, profileID, libraryID)
}
// A multi-file audiobook must not re-issue the file-invariant preference
// queries once per track. Before the request-scoped memo this scaled O(files),
// which is why a many-track audiobook detail page was slow.
func TestBuildPlaybackInfo_AudioPrefLookupsDoNotScaleWithFileCount(t *testing.T) {
counting := &countingUserStore{UserStore: newDetailTestStore(t)}
service := &DetailService{}
service.SetUserStoreProvider(testDetailUserStoreProvider{store: counting})
const fileCount = 8
files := make([]*models.MediaFile, 0, fileCount)
for i := 0; i < fileCount; i++ {
files = append(files, &models.MediaFile{
ID: i + 1,
MediaFolderID: 42,
AudioTracks: []models.AudioTrack{{Language: "eng", Default: true}},
})
}
filter := AccessFilter{UserID: 1, ProfileID: "profile-1"}
service.buildPlaybackInfo(context.Background(), files, filter, "book-1")
if got := atomic.LoadInt32(&counting.getProfile); got != 1 {
t.Fatalf("GetProfile called %d times for %d files; want 1 (lookup must not scale with file count)", got, fileCount)
}
if got := atomic.LoadInt32(&counting.getAudioPref); got != 1 {
t.Fatalf("GetAudioPreference called %d times for %d files; want 1", got, fileCount)
}
if got := atomic.LoadInt32(&counting.getLibraryPref); got != 1 {
t.Fatalf("GetLibraryPlaybackPreference called %d times for one folder; want 1", got)
}
}
@@ -0,0 +1,27 @@
package catalog
import (
"context"
"testing"
)
type fakeWorkSummaryProvider struct{}
func (fakeWorkSummaryProvider) GetSummaryForContentID(ctx context.Context, contentID string, filter AccessFilter) (*WorkSummary, error) {
return &WorkSummary{
WorkID: "work-1",
Title: "Project Hail Mary",
Formats: []WorkFormatSummary{
{Type: "ebook", ContentID: contentID, LibraryID: 1},
{Type: "audiobook", ContentID: "audio-1", LibraryID: 2},
},
}, nil
}
func TestItemDetailIncludesWorkSummaryWhenProviderConfigured(t *testing.T) {
detail := &ItemDetail{ContentID: "ebook-1", Type: "ebook", Title: "Project Hail Mary"}
applyWorkSummary(context.Background(), detail, fakeWorkSummaryProvider{}, AccessFilter{})
if detail.WorkID != "work-1" || len(detail.WorkFormats) != 2 {
t.Fatalf("detail work fields = %#v", detail)
}
}
+49
View File
@@ -0,0 +1,49 @@
package catalog
import (
"context"
"strings"
"testing"
)
// TestMangaChaptersQueryOrdering pins the manga chapter listing contract: join
// manga_chapters to media_items on the chapter content ID, scope to the series,
// and order by chapter_index (NULLs last) then sort_title. A wrong ORDER BY
// would surface chapters out of reading order in the series detail.
func TestMangaChaptersQueryOrdering(t *testing.T) {
q := strings.Join(strings.Fields(mangaChaptersQuery), " ")
for _, want := range []string{
"FROM manga_chapters mc",
"JOIN media_items m ON m.content_id = mc.chapter_content_id",
"WHERE mc.series_content_id = $1",
"ORDER BY mc.chapter_index NULLS LAST, m.sort_title",
// Per-chapter read state: viewer-scoped LEFT JOIN onto ebook progress.
"LEFT JOIN ebook_reader_progress erp",
"AND erp.user_id = $2",
"AND erp.profile_id = $3",
"AS read",
} {
if !strings.Contains(q, want) {
t.Fatalf("manga chapters query missing %q\nquery: %s", want, q)
}
}
}
// TestFetchMangaChaptersNilSafe asserts the helper never returns nil (the JSON
// payload must always carry an array) and tolerates an unconfigured pool.
func TestFetchMangaChaptersNilSafe(t *testing.T) {
var s *DetailService
if got := s.fetchMangaChapters(context.Background(), "series-1", AccessFilter{}); got == nil {
t.Fatal("nil receiver should yield an empty slice, not nil")
}
s = &DetailService{}
got := s.fetchMangaChapters(context.Background(), "series-1", AccessFilter{})
if got == nil {
t.Fatal("unconfigured pool should yield an empty slice, not nil")
}
if len(got) != 0 {
t.Fatalf("expected no chapters without a pool, got %d", len(got))
}
}
+6
View File
@@ -54,6 +54,8 @@ func buildRatingThresholdQuery(f RatingFilter) (string, []any) {
applyAccessFilter("mi", f.Filter, &conditions, &args, &argIdx)
conditions = append(conditions, MangaChapterExclusionWhere("mi"))
query := fmt.Sprintf(
"SELECT %s FROM media_items mi WHERE %s ORDER BY mi.rating_imdb DESC NULLS LAST, mi.content_id ASC",
qualifiedItemColumns("mi"),
@@ -134,6 +136,8 @@ func buildUnplayedHighRatedQuery(f UnplayedFilter) (string, []any) {
applyAccessFilter("mi", f.Filter, &conditions, &args, &argIdx)
conditions = append(conditions, MangaChapterExclusionWhere("mi"))
query := fmt.Sprintf(
"SELECT %s FROM media_items mi WHERE %s ORDER BY mi.rating_imdb DESC NULLS LAST, mi.content_id ASC",
qualifiedItemColumns("mi"),
@@ -226,6 +230,8 @@ func buildForgottenFavoritesQuery(f ForgottenFavoritesFilter) (string, []any) {
applyAccessFilter("mi", f.Filter, &conditions, &args, &argIdx)
conditions = append(conditions, MangaChapterExclusionWhere("mi"))
query := fmt.Sprintf(
"SELECT %s FROM media_items mi WHERE %s ORDER BY mi.rating_imdb DESC NULLS LAST, mi.content_id ASC",
qualifiedItemColumns("mi"),
+196
View File
@@ -0,0 +1,196 @@
package catalog
import (
"bytes"
"context"
"encoding/json"
"fmt"
"strings"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/Silo-Server/silo-server/internal/models"
)
const (
displayFilterFieldType = "type"
displayFilterFieldWatched = "watched"
)
// displayFilterFields are the QueryRule fields a personal-collection display
// filter may use. Display filters are intentionally a small, fixed vocabulary
// (the "watched" and "movies/series" controls); richer rules belong in a
// collection's own query_definition, not its display overlay.
var displayFilterFields = map[string]bool{
displayFilterFieldType: true,
displayFilterFieldWatched: true,
}
// NormalizeDisplayQueryFragment validates a personal-collection display filter
// and returns its canonical JSON form, or ("", nil) when the fragment is absent
// or carries no rules.
//
// A display filter is a FILTER-ONLY QueryDefinition fragment: it may set only
// match + groups, with rules drawn from displayFilterFields. Fields that control
// execution rather than filtering — library_ids, media_scope, sort, and limit —
// are rejected, because a stored display filter must never reorder or truncate a
// collection's match set (the exact-collection reader relies on retrieving the
// full match set in source order before paginating).
func NormalizeDisplayQueryFragment(raw []byte) (string, error) {
trimmed := bytes.TrimSpace(raw)
if len(trimmed) == 0 {
return "", nil
}
// A JSON null unmarshals to a zero QueryDefinition with no groups, which the
// empty-groups guard below already collapses to "".
var def QueryDefinition
if err := json.Unmarshal(trimmed, &def); err != nil {
return "", fmt.Errorf("display_query_definition must be a query definition: %w", err)
}
if len(def.LibraryIDs) > 0 || strings.TrimSpace(def.MediaScope) != "" || def.Limit != nil || def.Sort != (QuerySort{}) {
return "", fmt.Errorf("display_query_definition is filter-only and must not set library_ids, media_scope, sort, or limit")
}
cleanGroups := make([]QueryGroup, 0, len(def.Groups))
for _, group := range def.Groups {
rules := make([]QueryRule, 0, len(group.Rules))
for _, rule := range group.Rules {
field := strings.ToLower(strings.TrimSpace(rule.Field))
if !displayFilterFields[field] {
return "", fmt.Errorf("display_query_definition does not support field %q", rule.Field)
}
rule.Field = field
if field == displayFilterFieldType {
if value, ok := rule.Value.(string); ok {
rule.Value = strings.ToLower(strings.TrimSpace(value))
}
}
if err := validateDisplayFilterValue(field, rule); err != nil {
return "", err
}
if field == displayFilterFieldType && rule.Value == "all" {
continue
}
rules = append(rules, rule)
}
if len(rules) == 0 {
continue
}
cleanGroups = append(cleanGroups, QueryGroup{Match: normalizeMatch(group.Match), Rules: rules})
}
if len(cleanGroups) == 0 {
return "", nil
}
// Validate field/op structure against the catalog vocabulary, then emit a
// minimal {match, groups} object so stored display filters never carry the
// default sort/limit a full QueryDefinition would serialize.
canonical := QueryDefinition{Match: normalizeMatch(def.Match), Groups: cleanGroups}
if err := canonical.Validate(); err != nil {
return "", fmt.Errorf("invalid display_query_definition: %w", err)
}
encoded, err := json.Marshal(struct {
Match string `json:"match"`
Groups []QueryGroup `json:"groups"`
}{Match: canonical.Match, Groups: cleanGroups})
if err != nil {
return "", fmt.Errorf("encoding display_query_definition: %w", err)
}
return string(encoded), nil
}
func validateDisplayFilterValue(field string, rule QueryRule) error {
switch field {
case displayFilterFieldWatched:
if _, ok := rule.Value.(bool); !ok {
return fmt.Errorf("display_query_definition watched value must be a boolean")
}
case displayFilterFieldType:
value, ok := rule.Value.(string)
if !ok || value == "" {
return fmt.Errorf("display_query_definition type value must be a non-empty string")
}
switch value {
case "all", "movie", "series":
default:
return fmt.Errorf("display_query_definition type value must be one of %q, %q, or %q", "all", "movie", "series")
}
}
return nil
}
// FilterCollectionItemsByDisplayQuery applies a personal collection's stored
// display filter (a filter-only QueryDefinition fragment) to items already
// loaded in source order, returning the matching items in their original order.
//
// Filtering runs through the catalog query executor constrained to the member
// IDs, so watched/type semantics match catalog browsing and item user-data
// exactly — no separate watched evaluator. The fragment is forced filter-only
// before execution (no scope/sort/library/limit) so it can never reorder or
// truncate the match set; source order and pagination are the caller's job. An
// empty fragment returns items unchanged.
func FilterCollectionItemsByDisplayQuery(
ctx context.Context,
pool *pgxpool.Pool,
items []*models.MediaItem,
displayQueryDefinition string,
access AccessFilter,
) ([]*models.MediaItem, error) {
trimmed := strings.TrimSpace(displayQueryDefinition)
if trimmed == "" || len(items) == 0 {
return items, nil
}
var def QueryDefinition
if err := json.Unmarshal([]byte(trimmed), &def); err != nil {
return nil, fmt.Errorf("parsing collection display_query_definition: %w", err)
}
// Filter-only: a stored fragment must never scope, reorder, or truncate the
// collection's match set.
def.MediaScope = ""
def.LibraryIDs = nil
def.Sort = QuerySort{}
def.Limit = nil
if len(def.Groups) == 0 {
return items, nil
}
ids := make([]string, 0, len(items))
for _, item := range items {
if item != nil && strings.TrimSpace(item.ContentID) != "" {
ids = append(ids, item.ContentID)
}
}
if len(ids) == 0 {
return items, nil
}
queryAccess := access
queryAccess.AllowedContentIDs = ids
executor := &QueryExecutor{Pool: pool}
matched, _, err := executor.Preview(ctx, def, queryAccess, len(ids))
if err != nil {
return nil, fmt.Errorf("evaluating collection display_query_definition: %w", err)
}
matchedIDs := make(map[string]struct{}, len(matched))
for _, item := range matched {
if item != nil {
matchedIDs[item.ContentID] = struct{}{}
}
}
filtered := make([]*models.MediaItem, 0, len(items))
for _, item := range items {
if item == nil {
continue
}
if _, ok := matchedIDs[item.ContentID]; ok {
filtered = append(filtered, item)
}
}
return filtered, nil
}
@@ -0,0 +1,184 @@
package catalog
import (
"context"
"encoding/json"
"fmt"
"os"
"strings"
"testing"
"time"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/Silo-Server/silo-server/internal/models"
)
func TestNormalizeDisplayQueryFragment_EmptyInputs(t *testing.T) {
for _, raw := range []string{
"",
" ",
"null",
`{"match":"all","groups":[]}`,
`{"match":"all","groups":[{"match":"all","rules":[]}]}`,
`{"match":"all","groups":[{"match":"all","rules":[{"field":"type","op":"is","value":"all"}]}]}`,
} {
got, err := NormalizeDisplayQueryFragment([]byte(raw))
if err != nil {
t.Fatalf("NormalizeDisplayQueryFragment(%q) returned error: %v", raw, err)
}
if got != "" {
t.Fatalf("NormalizeDisplayQueryFragment(%q) = %q, want empty", raw, got)
}
}
}
func TestNormalizeDisplayQueryFragment_CanonicalizesTypeAndWatched(t *testing.T) {
raw := `{"match":"all","groups":[{"match":"all","rules":[
{"field":"watched","op":"is","value":true},
{"field":"type","op":"is","value":"movie"}
]}]}`
got, err := NormalizeDisplayQueryFragment([]byte(raw))
if err != nil {
t.Fatalf("NormalizeDisplayQueryFragment returned error: %v", err)
}
// Canonical output is filter-only: no sort/limit/library_ids/media_scope.
for _, banned := range []string{"sort", "limit", "library_ids", "media_scope"} {
if strings.Contains(got, banned) {
t.Fatalf("canonical fragment must not contain %q, got %q", banned, got)
}
}
var def QueryDefinition
if err := json.Unmarshal([]byte(got), &def); err != nil {
t.Fatalf("canonical fragment does not round-trip: %v", err)
}
if def.Match != "all" || len(def.Groups) != 1 || len(def.Groups[0].Rules) != 2 {
t.Fatalf("unexpected canonical shape: %+v", def)
}
}
func TestNormalizeDisplayQueryFragment_RejectsExecutionControllingFields(t *testing.T) {
cases := map[string]string{
"limit": `{"match":"all","limit":5,"groups":[{"match":"all","rules":[{"field":"watched","op":"is","value":true}]}]}`,
"sort": `{"match":"all","sort":{"field":"title","order":"asc"},"groups":[{"match":"all","rules":[{"field":"watched","op":"is","value":true}]}]}`,
"library_ids": `{"match":"all","library_ids":[3],"groups":[{"match":"all","rules":[{"field":"watched","op":"is","value":true}]}]}`,
"media_scope": `{"match":"all","media_scope":"movie","groups":[{"match":"all","rules":[{"field":"watched","op":"is","value":true}]}]}`,
}
for name, raw := range cases {
if _, err := NormalizeDisplayQueryFragment([]byte(raw)); err == nil {
t.Fatalf("%s: expected rejection of execution-controlling field, got nil error", name)
}
}
}
func TestNormalizeDisplayQueryFragment_RejectsUnsupportedFieldsAndValues(t *testing.T) {
cases := map[string]string{
"unsupported field": `{"match":"all","groups":[{"match":"all","rules":[{"field":"genre","op":"is","value":"Drama"}]}]}`,
"watched non-bool": `{"match":"all","groups":[{"match":"all","rules":[{"field":"watched","op":"is","value":"yes"}]}]}`,
"type non-string": `{"match":"all","groups":[{"match":"all","rules":[{"field":"type","op":"is","value":3}]}]}`,
"type empty string": `{"match":"all","groups":[{"match":"all","rules":[{"field":"type","op":"is","value":" "}]}]}`,
"type unsupported": `{"match":"all","groups":[{"match":"all","rules":[{"field":"type","op":"is","value":"ebook"}]}]}`,
"malformed json": `{"match":"all",`,
}
for name, raw := range cases {
if _, err := NormalizeDisplayQueryFragment([]byte(raw)); err == nil {
t.Fatalf("%s: expected error, got nil", name)
}
}
}
func TestFilterCollectionItemsByDisplayQuery(t *testing.T) {
ctx := context.Background()
input := []*models.MediaItem{
{ContentID: "movie-b"},
{ContentID: "series-1"},
{ContentID: "movie-a"},
}
empty, err := FilterCollectionItemsByDisplayQuery(ctx, nil, input, " ", AccessFilter{})
if err != nil {
t.Fatalf("empty display query returned error: %v", err)
}
if got := contentIDsForTest(empty); strings.Join(got, ",") != "movie-b,series-1,movie-a" {
t.Fatalf("empty display query order = %v", got)
}
pool := newDisplayQueryFilterTestPool(t)
suffix := time.Now().UnixNano()
movieA := fmt.Sprintf("display-filter-movie-a-%d", suffix)
movieB := fmt.Sprintf("display-filter-movie-b-%d", suffix)
series := fmt.Sprintf("display-filter-series-%d", suffix)
outsideMovie := fmt.Sprintf("display-filter-outside-movie-%d", suffix)
ids := []string{movieA, movieB, series, outsideMovie}
t.Cleanup(func() {
_, _ = pool.Exec(ctx, `DELETE FROM media_items WHERE content_id = ANY($1)`, ids)
})
seedDisplayFilterMediaItem(t, pool, movieA, "movie")
seedDisplayFilterMediaItem(t, pool, movieB, "movie")
seedDisplayFilterMediaItem(t, pool, series, "series")
seedDisplayFilterMediaItem(t, pool, outsideMovie, "movie")
items := []*models.MediaItem{
{ContentID: movieB},
{ContentID: series},
{ContentID: movieA},
}
filtered, err := FilterCollectionItemsByDisplayQuery(
ctx,
pool,
items,
`{"match":"all","groups":[{"match":"all","rules":[{"field":"type","op":"is","value":"movie"}]}]}`,
AccessFilter{},
)
if err != nil {
t.Fatalf("FilterCollectionItemsByDisplayQuery returned error: %v", err)
}
got := contentIDsForTest(filtered)
want := []string{movieB, movieA}
if strings.Join(got, ",") != strings.Join(want, ",") {
t.Fatalf("filtered content IDs = %v, want %v", got, want)
}
}
func newDisplayQueryFilterTestPool(t *testing.T) *pgxpool.Pool {
t.Helper()
dsn := os.Getenv("SILO_TEST_DATABASE_URL")
if dsn == "" {
t.Skip("SILO_TEST_DATABASE_URL is not set")
}
ctx := context.Background()
pool, err := pgxpool.New(ctx, dsn)
if err != nil {
t.Fatalf("connect test database: %v", err)
}
t.Cleanup(pool.Close)
var tableName *string
if err := pool.QueryRow(ctx, `SELECT to_regclass('public.media_items')::text`).Scan(&tableName); err != nil {
t.Fatalf("check media_items table: %v", err)
}
if tableName == nil || *tableName == "" {
t.Skip("test database has not applied base schema")
}
return pool
}
func seedDisplayFilterMediaItem(t *testing.T, pool *pgxpool.Pool, contentID string, mediaType string) {
t.Helper()
if _, err := pool.Exec(context.Background(), `
INSERT INTO media_items (content_id, type, title, genres)
VALUES ($1, $2, $3, '{}'::text[])
`, contentID, mediaType, contentID); err != nil {
t.Fatalf("seed media item %s: %v", contentID, err)
}
}
func contentIDsForTest(items []*models.MediaItem) []string {
ids := make([]string, 0, len(items))
for _, item := range items {
if item != nil {
ids = append(ids, item.ContentID)
}
}
return ids
}
@@ -80,6 +80,10 @@ var SensitiveSettingKeys = map[string]bool{
"recommendations.embedding_auth_token": true,
"recommendations.openai_api_key": true,
// Optional catalog search provider auth. Do not rename this key after
// shipping; encrypted values are bound to the setting name.
"catalog.search.meilisearch.api_key": true,
// Watch-sync OAuth client credentials. The client_id entries are low-value
// but kept as a harmless safe superset and to match existing redaction.
"watchsync.trakt.client_id": true,
@@ -32,8 +32,10 @@ const episodeCatalogSelectBody = `(
''::text AS poster_source_path,
COALESCE(NULLIF(s.poster_thumbhash, ''), NULLIF(si.poster_thumbhash, ''), NULLIF(e.still_thumbhash, ''), '') AS poster_thumbhash,
COALESCE(si.backdrop_path, '') AS backdrop_path,
COALESCE(si.backdrop_source_path, '') AS backdrop_source_path,
COALESCE(si.backdrop_thumbhash, '') AS backdrop_thumbhash,
COALESCE(si.logo_path, '') AS logo_path,
COALESCE(si.logo_source_path, '') AS logo_source_path,
COALESCE(e.metadata_s3_path, '') AS metadata_s3_path,
COALESCE(e.metadata_etag, '') AS metadata_etag,
NULL::integer AS season_count,
+43 -8
View File
@@ -33,7 +33,7 @@ const episodeColumns = `content_id, series_id, season_id, season_number, episode
title, default_metadata_language, overview, air_date, runtime,
rating_imdb, rating_tmdb,
imdb_id, tmdb_id, tvdb_id,
still_path, still_thumbhash,
still_path, still_source_path, still_thumbhash,
metadata_s3_path, metadata_etag, metadata_source,
created_at, updated_at`
@@ -113,6 +113,7 @@ func scanEpisode(row pgx.Row) (*models.Episode, error) {
var tmdbID *string
var tvdbID *string
var stillPath *string
var stillSourcePath *string
var stillThumbhash *string
var metadataS3Path *string
var metadataEtag *string
@@ -133,6 +134,7 @@ func scanEpisode(row pgx.Row) (*models.Episode, error) {
&tmdbID,
&tvdbID,
&stillPath,
&stillSourcePath,
&stillThumbhash,
&metadataS3Path,
&metadataEtag,
@@ -161,6 +163,9 @@ func scanEpisode(row pgx.Row) (*models.Episode, error) {
if stillPath != nil {
ep.StillPath = *stillPath
}
if stillSourcePath != nil {
ep.StillSourcePath = *stillSourcePath
}
if stillThumbhash != nil {
ep.StillThumbhash = *stillThumbhash
}
@@ -191,6 +196,7 @@ func scanEpisodes(rows pgx.Rows) ([]*models.Episode, error) {
var tmdbID *string
var tvdbID *string
var stillPath *string
var stillSourcePath *string
var stillThumbhash *string
var metadataS3Path *string
var metadataEtag *string
@@ -211,6 +217,7 @@ func scanEpisodes(rows pgx.Rows) ([]*models.Episode, error) {
&tmdbID,
&tvdbID,
&stillPath,
&stillSourcePath,
&stillThumbhash,
&metadataS3Path,
&metadataEtag,
@@ -239,6 +246,9 @@ func scanEpisodes(rows pgx.Rows) ([]*models.Episode, error) {
if stillPath != nil {
ep.StillPath = *stillPath
}
if stillSourcePath != nil {
ep.StillSourcePath = *stillSourcePath
}
if stillThumbhash != nil {
ep.StillThumbhash = *stillThumbhash
}
@@ -297,15 +307,15 @@ func (r *EpisodeRepository) Upsert(ctx context.Context, ep *models.Episode) erro
title, default_metadata_language, overview, air_date, runtime,
rating_imdb, rating_tmdb,
imdb_id, tmdb_id, tvdb_id,
still_path, still_thumbhash,
still_path, still_source_path, still_thumbhash,
metadata_s3_path, metadata_etag, metadata_source
) VALUES (
$1, $2, $3, $4, $5,
$6, $7, $8, $9, $10,
$11, $12,
$13, $14, $15,
$16, $17,
$18, $19, $20
$16, $17, $18,
$19, $20, $21
)
ON CONFLICT (series_id, season_number, episode_number) DO UPDATE SET
season_id = COALESCE(EXCLUDED.season_id, episodes.season_id),
@@ -320,6 +330,7 @@ func (r *EpisodeRepository) Upsert(ctx context.Context, ep *models.Episode) erro
tmdb_id = COALESCE(NULLIF(EXCLUDED.tmdb_id, ''), episodes.tmdb_id),
tvdb_id = COALESCE(NULLIF(EXCLUDED.tvdb_id, ''), episodes.tvdb_id),
still_path = EXCLUDED.still_path,
still_source_path = EXCLUDED.still_source_path,
still_thumbhash = EXCLUDED.still_thumbhash,
metadata_s3_path = EXCLUDED.metadata_s3_path,
metadata_etag = EXCLUDED.metadata_etag,
@@ -345,6 +356,7 @@ func (r *EpisodeRepository) Upsert(ctx context.Context, ep *models.Episode) erro
ep.TmdbID,
ep.TvdbID,
ep.StillPath,
ep.StillSourcePath,
ep.StillThumbhash,
ep.MetadataS3Path,
ep.MetadataEtag,
@@ -439,6 +451,7 @@ func (r *EpisodeRepository) BulkUpsert(ctx context.Context, seriesID string, epi
epTmdbIDs := make([]string, len(episodes))
epTvdbIDs := make([]string, len(episodes))
stillPaths := make([]string, len(episodes))
stillSourcePaths := make([]string, len(episodes))
stillThumbs := make([]string, len(episodes))
metaS3Paths := make([]string, len(episodes))
metaEtags := make([]string, len(episodes))
@@ -463,6 +476,7 @@ func (r *EpisodeRepository) BulkUpsert(ctx context.Context, seriesID string, epi
epTmdbIDs[i] = ep.TmdbID
epTvdbIDs[i] = ep.TvdbID
stillPaths[i] = ep.StillPath
stillSourcePaths[i] = ep.StillSourcePath
stillThumbs[i] = ep.StillThumbhash
metaS3Paths[i] = ep.MetadataS3Path
metaEtags[i] = ep.MetadataEtag
@@ -475,7 +489,7 @@ func (r *EpisodeRepository) BulkUpsert(ctx context.Context, seriesID string, epi
title, default_metadata_language, overview, air_date, runtime,
rating_imdb, rating_tmdb,
imdb_id, tmdb_id, tvdb_id,
still_path, still_thumbhash,
still_path, still_source_path, still_thumbhash,
metadata_s3_path, metadata_etag, metadata_source
)
SELECT * FROM UNNEST(
@@ -483,8 +497,8 @@ func (r *EpisodeRepository) BulkUpsert(ctx context.Context, seriesID string, epi
$6::text[], $7::text[], $8::text[], $9::date[], $10::int[],
$11::float8[], $12::float8[],
$13::text[], $14::text[], $15::text[],
$16::text[], $17::text[],
$18::text[], $19::text[], $20::text[]
$16::text[], $17::text[], $18::text[],
$19::text[], $20::text[], $21::text[]
)
ON CONFLICT (series_id, season_number, episode_number) DO UPDATE SET
season_id = COALESCE(EXCLUDED.season_id, episodes.season_id),
@@ -499,6 +513,7 @@ func (r *EpisodeRepository) BulkUpsert(ctx context.Context, seriesID string, epi
tmdb_id = COALESCE(NULLIF(EXCLUDED.tmdb_id, ''), episodes.tmdb_id),
tvdb_id = COALESCE(NULLIF(EXCLUDED.tvdb_id, ''), episodes.tvdb_id),
still_path = EXCLUDED.still_path,
still_source_path = EXCLUDED.still_source_path,
still_thumbhash = EXCLUDED.still_thumbhash,
metadata_s3_path = EXCLUDED.metadata_s3_path,
metadata_etag = EXCLUDED.metadata_etag,
@@ -511,7 +526,7 @@ func (r *EpisodeRepository) BulkUpsert(ctx context.Context, seriesID string, epi
titles, defaultMetadataLanguages, overviews, airDates, runtimes,
ratingsIMDB, ratingsTMDB,
epImdbIDs, epTmdbIDs, epTvdbIDs,
stillPaths, stillThumbs,
stillPaths, stillSourcePaths, stillThumbs,
metaS3Paths, metaEtags, metaSources,
)
if err != nil {
@@ -900,6 +915,10 @@ func (r *EpisodeRepository) UpdateMetadata(ctx context.Context, contentID string
addString("tmdb_id", upd.TmdbID)
addString("tvdb_id", upd.TvdbID)
addString("still_path", upd.StillPath)
if upd.StillPath != nil && upd.StillSourcePath == nil {
setClauses = append(setClauses, "still_source_path = ''")
}
addString("still_source_path", upd.StillSourcePath)
addString("still_thumbhash", upd.StillThumbhash)
setClauses = append(setClauses, "updated_at = NOW()")
@@ -917,3 +936,19 @@ func (r *EpisodeRepository) UpdateMetadata(ctx context.Context, contentID string
}
return nil
}
func (r *EpisodeRepository) UpdateStillIfSourceMatches(ctx context.Context, contentID, sourcePath, cachedPath, thumbhash string) (bool, error) {
tag, err := r.pool.Exec(ctx, `
UPDATE episodes
SET still_path = $3,
still_source_path = $2,
still_thumbhash = $4,
updated_at = NOW()
WHERE content_id = $1
AND still_source_path = $2
`, contentID, sourcePath, cachedPath, thumbhash)
if err != nil {
return false, fmt.Errorf("updating episode cached still: %w", err)
}
return tag.RowsAffected() > 0, nil
}

Some files were not shown because too many files have changed in this diff Show More