From 91b5105f3c914b0b48cbef1ce321f557a5cf089f Mon Sep 17 00:00:00 2001 From: Quick <31828688+Quick104@users.noreply.github.com> Date: Fri, 26 Jun 2026 08:24:14 -0400 Subject: [PATCH] docs(search): add hybrid semantic search hardening plan Co-Authored-By: Claude Opus 4.8 --- ...-06-25-hybrid-semantic-search-hardening.md | 327 ++++++++++++++++++ 1 file changed, 327 insertions(+) create mode 100644 docs/superpowers/plans/2026-06-25-hybrid-semantic-search-hardening.md diff --git a/docs/superpowers/plans/2026-06-25-hybrid-semantic-search-hardening.md b/docs/superpowers/plans/2026-06-25-hybrid-semantic-search-hardening.md new file mode 100644 index 00000000..5c9f52bb --- /dev/null +++ b/docs/superpowers/plans/2026-06-25-hybrid-semantic-search-hardening.md @@ -0,0 +1,327 @@ +# Hybrid Semantic Search Hardening Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. Commands assume the repository root is the cwd. + +**Goal:** Make Meilisearch hybrid semantic search safe to enable broadly by gating it on *trustworthy* per-type vector coverage, validating embedder capability, and exposing per-query/admin diagnostics — without regressing the hot search path. + +**Architecture:** Semantic search stays gated behind the existing `SemanticEnabled && wordCount >= 3` rule, plus a new **coverage gate** that reads an in-memory, periodically-refreshed per-type coverage snapshot (never a per-request DB query). Coverage = `current-model embeddings ÷ embed-eligible items`, computed entirely from Postgres and latched with hysteresis. Existing provider/fallback fields are plumbed through the resolver into an additive `search_diagnostics` response object. A rate-limited capability probe validates the Meili embedder without downing keyword search. + +**Tech Stack:** Go (pgx/pgxpool), Meilisearch hybrid search (`userProvided` embedder), React/TypeScript admin UI, Goose migrations. + +## Global Constraints + +- **API additive-only within `/api/v1`** — never rename/remove a response field, change a type, or repurpose a status code. New fields are `omitempty` additions. (CLAUDE.md) +- **Performance first, reliability first, predictable under load** — the search request path must not gain a synchronous DB count, lock wait, or remote probe. Fail **safe** (keyword-only), never panic. (CLAUDE.md Core Priorities) +- **One additive index migration** is in scope (revised — see Assumptions): a btree on `media_item_embeddings(model)` to support the periodic coverage query. No schema/column changes. +- **Canonical embedding dimensions** = `embeddingvectors.CanonicalDimensions` (= 3072; re-exported as `recommendations.CanonicalEmbeddingDimensions`). Never hardcode the number. +- **Single source of truth for embed-eligibility** — the predicate `(status='matched' OR type IN ('audiobook','ebook'))` must not be duplicated. It moves to the neutral `internal/embeddingvectors` package (imported by both `catalog` and `recommendations`; `recommendations` already imports `catalog`, so `catalog` cannot import `recommendations`). +- **Single phase** — measurement *and* enforcement ship together. The diagnostics shipped here are what let us retune thresholds later without code changes-in-anger. +- Build/test in worktrees with `GOWORK=off` and a stubbed `web/dist` (see worktree-build-quirks memory). + +--- + +## Corrected Baseline (what actually exists today — verified) + +Work from this, not from intuition. Each point was confirmed against the tree. + +1. **The only semantic skip rule is `SemanticEnabled && len(Fields(normalizedQuery)) >= 3`** (`internal/catalog/search_meilisearch_provider.go:329`). There is **no** `SkipTotal`/preview skip, and `TestMeilisearchSearchRequestBuildsHybridForApproximateInteractiveSearch` (`search_provider_test.go:213`) asserts hybrid **IS** used with `SkipTotal: true`. → **Do not add a `SkipTotal` skip.** Layer the coverage gate onto the existing rule; remove nothing. +2. **`CatalogSearchResult` already carries `Provider` and `FallbackReason`** (`search_provider.go:64-71`). Diagnostics work is *plumbing these through* `CatalogResult` (which drops them, `catalog_resolver.go:24-30,291-296`) — not inventing provider fields. +3. **The provider already self-downgrades hybrid→keyword on a Meili error** (`search_meilisearch_provider.go:221-228`). Diagnostics must report the *post-downgrade* reality. +4. **`countCatalogSearchVectorDocuments` ignores `model`** (`search_indexer.go:757`), counts the *indexed* population (no eligibility filter), and its companion denominator (`document_count`) comes from **Meili index stats** (`search_service.go:129`), not Postgres. It has **four callers** — `search_service.go:139`, `search_indexer.go:133`, `:203`, `:310` — all must be updated when its signature changes. +5. **Indexed population ≠ embed-eligible population.** `LoadDocumentsAfter` (`search_indexer.go:534`) indexes every type-matched, non-manga row regardless of `status`; embeddings are only written for `(status='matched' OR audiobook OR ebook)` (`recommendations/repo.go:154-160`, `recommendationItemEligibilityWhereClause`). A denominator of "indexable items" is therefore structurally < 1.0 on any library with unmatched video → **the gate would never open.** Coverage must use the **embed-eligible** population for both numerator and denominator. +6. **Status fields that already exist:** `document_count`, `vector_document_count` (`search_provider.go:347-348`). The earlier draft's `index_document_count` does **not** exist — keep real names; add new fields alongside. +7. **`UpsertEmbedding` already enqueues a Meili index event after every write** (`recommendations/repo.go:186`) — verified correct, **no change**. +8. **Catalog search settings are restart-bound** (`config/restart_keys.go:75-85`, *"Catalog search provider construction is intentionally startup-bound in v1"*). The service is built **once** in `NewRouter` and never live-rebuilt. → the coverage refresher is a single process-lifetime goroutine; no live-rebuild leak exists in v1. +9. **`catalogSearchService` is a local var in `NewRouter`** (`router.go:437`), wired into handlers but **never returned** to `main.go`. The long-lived `deps.AppContext` (`router.go:88`) is the refresher's context. +10. **Legacy `/search` plain-`q` re-route is descoped** — see [Descoped](#descoped--not-in-this-pass). + +--- + +## Design Decisions + +### D1 — Coverage = current-model embeddings ÷ embed-eligible items (single Postgres source, per type) + +Let `ELIG(alias)` = `embeddingvectors.ItemEligibilityWhereClause(alias)` → `(alias.status='matched' OR alias.type IN ('audiobook','ebook'))` (moved here from `recommendations`; `recommendations.recommendationItemEligibilityWhereClause` is refactored to call it — no behavior change, no duplication). + +- **Denominator** (embed-eligible items, per type): + ```sql + SELECT mi.type, COUNT(*) AS eligible + FROM media_items mi + WHERE NOT EXISTS (SELECT 1 FROM manga_chapters mc WHERE mc.chapter_content_id = mi.content_id) + AND ($1::text[] IS NULL OR mi.type = ANY($1)) + AND (mi.status = 'matched' OR mi.type IN ('audiobook','ebook')) -- ELIG("mi") + GROUP BY mi.type + ``` +- **Numerator** (eligible items with a *current-model* embedding, per type): + ```sql + SELECT mi.type, COUNT(*) AS vectorized + FROM media_item_embeddings e + JOIN media_items mi ON mi.content_id = e.media_item_id + WHERE NOT EXISTS (SELECT 1 FROM manga_chapters mc WHERE mc.chapter_content_id = mi.content_id) + AND ($1::text[] IS NULL OR mi.type = ANY($1)) + AND (mi.status = 'matched' OR mi.type IN ('audiobook','ebook')) -- ELIG("mi") + AND ($2 = '' OR e.model = $2) + GROUP BY mi.type + ``` + +Applying `ELIG` to **both** guarantees numerator ⊆ denominator → ratio ∈ `[0,1]` (an embedding left behind on an item that became unmatched can't push a type over 1.0). Unmatched items are excluded from both, so they never drag coverage. As backfill completes, ratio → 1.0. + +**Active model (`$2`)** is read from the embedding lock (`recommendations.EmbeddingLock.Model`, `embedding_lock.go:16`). No lock ⇒ `$2 = ""` and the snapshot is marked not-ready (never gate semantic on for an unestablished embedding space). + +> **Why model-filtering is load-bearing:** right after a model change, old vectors persist but are mismatched to the new query embedder. A model-blind count reads ~100% at the most dangerous moment. Filtering by the locked model collapses coverage to ~0 until re-embedding catches up. + +**Index (migration, in scope):** `media_item_embeddings` has only an HNSW index on `embedding` (`migrations/sql/001_schema.sql:1857`, `014_...:25`) — nothing on `model`. Add `CREATE INDEX CONCURRENTLY idx_media_item_embeddings_model ON public.media_item_embeddings (model)` so the numerator filter is index-supported. (Goose: timestamped file via `make migrate-create`, with `-- +goose NO TRANSACTION` because `CONCURRENTLY` cannot run in a txn.) + +### D2 — Coverage lives in an in-memory snapshot, refreshed off the request path + +A `semanticCoverageTracker` (new, `internal/catalog/semantic_coverage.go`) holds an `atomic.Pointer[semanticCoverageSnapshot]`. + +- **Refresh cadence:** a single background goroutine every `semanticCoverageRefreshInterval` (**2 minutes** — long enough that two `GROUP BY` aggregates are negligible steady-state load even at 1M rows; short enough to bound the post-model-change danger window, which also fails safe via the model-collapse rule below). The interval is a named const; **there is no after-sync trigger** (the indexer and service share no handle — see Descoped rationale; the interval alone is sufficient and avoids a cross-component wiring + race). +- **Single-flight & immutability:** `Refresh` holds a `sync.Mutex` across read-prev → query → compute → publish (background path; contention irrelevant). Each refresh builds a **fresh** `PerType` map and publishes a new immutable snapshot pointer; the published snapshot is **never mutated** after publish (avoids a `map` data race against concurrent `CoverageReady` reads). A `-race` test fires refreshes concurrently with reads. +- **Fail-safe reads:** `CoverageReady` reads only the atomic pointer — **no DB, no lock, no remote call.** If the snapshot is `nil` (boot, before first refresh) it returns `(false, "coverage not yet computed")` — **never** dereferences a nil map (that would be a hot-path panic). +- **Error retention:** on a refresh query error, retain the last-good snapshot (transient DB blips must not flip semantic off); log; do not publish a zeroed snapshot. Admin status flags `UpdatedAt` staleness. +- **Model-change collapse:** the snapshot records the `Model` it was computed for. When `Refresh` observes the active model differs from the published snapshot's model, it **immediately publishes an all-not-ready snapshot** (safe) and recomputes real coverage under the new model on the same pass. This bounds the mismatched-vector window to ≤ one interval and fails safe within it. + +### D3 — Per-type readiness with hysteresis; scope = AND of its types; empty scope = all types + +Each type's `ready` flag is **latched**: flips **true** at `ratio >= semanticCoverageEnableRatio` (0.90), flips **false** only at `ratio < semanticCoverageDisableRatio` (0.80). The snapshot carries prior latched state so each refresh applies hysteresis — preventing flap (identical queries returning differently-ranked results). + +`CoverageReady(itemTypes)`: +- **Non-empty scope:** ready iff **every** requested type is latched-ready (`min` semantics — a weak subtype blocks the mixed scope). The reason names the first failing type. +- **Empty/nil scope** (the common "search everything" case — `MediaScopeItemTypes("") → nil`, so `ItemTypes == nil` reaches the gate): ready iff **every type in the snapshot** is latched-ready. This deliberately mirrors the empty-config semantics and refuses to let an unscoped search bypass a weak type. Reason names the first failing type, or `"coverage not yet computed"` if the snapshot is nil/empty. + +### D4 — Diagnostics: existing fields, plumbed, reported post-downgrade, scoped to the provider path + +- Add `Mode string` (`"keyword"`/`"hybrid"`) and `SemanticUsed bool` to `CatalogSearchResult`; the Meili provider sets them to reflect the **final** request after the internal hybrid→keyword downgrade. +- Add `Provider`, `Mode`, `SemanticUsed`, `FallbackReason` to `CatalogResult`; copy them in `resolveDirectSearchSource`. +- Add an `omitempty` `search_diagnostics` object to `catalogResponse`. +- **Scope:** diagnostics are produced **only** on the direct provider-backed search path (`useDirectSearchPath` = relevance sort, desc, no advanced rules — `catalog_resolver.go:1416-1427`). They are **omitted** for: browse, non-relevance-sorted `q=` searches (which never hit the provider), and `group=work` responses (the grouped wrapper builds a fresh `CatalogResult` at `catalog.go:287-293` and is already approximate — `TotalExact=false`). Each exclusion gets an explicit handler test. + +### D5 — Capability validation: narrow, rate-limited, non-fatal, admin-path only + +Embedder settings are written by Silo itself, so validation guards only **external drift** (Meili volume reset, manual edits). Add `GetSettings` to the Meili client; verify the configured embedder exists, `source == "userProvided"`, and `dimensions == embeddingvectors.CanonicalDimensions`. The hybrid probe: +- runs **only on the admin status path** (`HandleGetCatalogSearchStatus`), **not** on `CheckConnection`/health (which only does `GET /health`), and the admin query has no `refetchInterval` (`settings.ts`, `staleTime: 15_000`); +- runs **only when `state.ActiveIndexUID != ""`**, and is **cached for `semanticCapabilityProbeTTL` (5m)** → ≤ 1 probe / 5m regardless of refresh rate; +- uses a **non-degenerate unit sample vector** (`v[0]=1.0`, rest 0, length `CanonicalDimensions`) — an all-zero vector has zero norm and can spuriously fail cosine hybrid; targets `ActiveIndexUID` with `Hybrid.Embedder = config.Embedder`, `limit:1`. + +Failures surface in admin status and force keyword-only; they **never** flip provider `Healthy` or trip the circuit. + +### D6 — Backfill split: cheap discovery vs. bounded text-staleness, with fairness + +The expensive part of `ListEmbeddingTextCandidates` is **building `current_text` for every eligible row** via 5 `LATERAL` joins to `item_people` (`recommendations/repo.go:544-589`) — *not* the `IS DISTINCT FROM` comparison. + +- **Cheap discovery** (`ListMissingOrModelStaleEmbeddingIDs`): `LEFT JOIN media_item_embeddings`, filter `ELIG("mi") AND (e.media_item_id IS NULL OR e.model = '' OR e.model <> $currentModel)`, `ORDER BY mi.content_id`, `LIMIT`. **No `LATERAL`, no `current_text`.** The SQL is built by a package-private `buildMissingOrModelStaleEmbeddingIDsSQL() string` helper so a unit test can assert it contains neither `item_people` nor `LATERAL` (mirrors the existing `embeddingEligibilityWhereClause()` string-helper test pattern). Returns IDs; `current_text` is then built **only for that bounded batch** via `BuildEmbeddingTextForIDs(ids)`. +- **Text-staleness** (existing `ListEmbeddingTextCandidates`): keep, but run as a **bounded, unconditional quota per cycle** (`embeddingTextStaleQuotaPerRun`) so canonical-text-stale items progress even when the cheap backlog never empties (fairness against starvation). +- **Cursors:** the cheap pass advances `afterID` from the **cheap query's** last id (it is the ordered cursor), independent of `BuildEmbeddingTextForIDs` row order (`ANY($1)` does not preserve slice order, and a dropped id must not desync the cursor). The text-stale pass keeps its **own** `afterID`. Both monotonic. + +--- + +## File Structure + +| File | Responsibility | Change | +| --- | --- | --- | +| `migrations/sql/_media_item_embeddings_model_index.sql` | `(model)` btree, `CONCURRENTLY` / `NO TRANSACTION` | Create | +| `internal/embeddingvectors/eligibility.go` | `ItemEligibilityWhereClause(alias)` — single source of truth | Create | +| `internal/catalog/semantic_coverage.go` | tracker, snapshot, querier seam, per-type SQL, hysteresis, single-flight refresher | Create | +| `internal/catalog/search_provider.go` | gate + model-provider interfaces; `Mode`/`SemanticUsed`; semantic status structs | Modify | +| `internal/catalog/search_meilisearch_provider.go` | consult gate; set `Mode`/`SemanticUsed` (incl. downgrade); capability probe (unit vector) | Modify | +| `internal/catalog/search_meilisearch_client.go` | `GetSettings` + settings DTOs | Modify | +| `internal/catalog/search_service.go` | build/own tracker; `StartCoverageRefresh`; comma-ok model-provider; `Status()` reads snapshot | Modify | +| `internal/catalog/search_indexer.go` | `ELIG`+`model` numerator & eligible denominator; update **all 4** `countCatalogSearchVectorDocuments` callers | Modify | +| `internal/catalog/catalog_resolver.go` | carry diagnostics through `CatalogResult` (direct path only) | Modify | +| `internal/recommendations/repo.go` | delegate eligibility to `embeddingvectors`; `ListMissingOrModelStaleEmbeddingIDs` (+SQL helper); `BuildEmbeddingTextForIDs` | Modify | +| `internal/recommendations/similar.go` | `EmbedAll`: cheap drain + unconditional text-stale quota; dual cursors | Modify | +| `internal/recommendations/engine.go` | `ActiveEmbeddingModel(ctx)` from the lock | Modify | +| `internal/api/handlers/catalog.go` | `searchDiagnostics` DTO; populate on direct path; omit for grouped | Modify | +| `internal/api/router.go` | start `service.StartCoverageRefresh(deps.AppContext)` inside `NewRouter` | Modify | +| `web/src/hooks/queries/admin/settings.ts`, `web/src/pages/admin-settings/SearchSettings.tsx` | surface readiness/ratio/per-type/capability | Modify | + +--- + +## Tasks + +### Task 1: Shared eligibility predicate, model+eligibility coverage counts, supporting index + +**Files:** Create `internal/embeddingvectors/eligibility.go`, `migrations/sql/_media_item_embeddings_model_index.sql`; Modify `internal/recommendations/repo.go:150-160`, `internal/catalog/search_indexer.go:757-782` (+ callers `:133,:203,:310`, `search_service.go:139`); Test `internal/embeddingvectors/eligibility_test.go`, `internal/catalog/search_indexer_test.go`. + +**Interfaces — Produces:** +- `embeddingvectors.ItemEligibilityWhereClause(alias string) string` +- `countCatalogSearchVectorDocuments(ctx, q coverageQuerier, itemTypes []string, model string) (int, error)` (adds `model`; applies `ELIG`) +- `catalogSemanticCoverageByType(ctx, q coverageQuerier, itemTypes []string, model string) ([]catalogTypeCoverage, error)` → `{Type string; Eligible, Vectorized int}` + +- [ ] Migration: `make migrate-create NAME=media_item_embeddings_model_index`; body uses `-- +goose NO TRANSACTION` then a self-healing guard for a leftover invalid index from a crashed build (follow the `20260618164519_add_episodes_season_id_index.sql` precedent: a `DO $$ ... NOT i.indisvalid ... DROP INDEX $$` block), then `CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_media_item_embeddings_model ON public.media_item_embeddings (model);` (down: `DROP INDEX CONCURRENTLY IF EXISTS`). The runner honors `NO TRANSACTION` (three shipping precedents). Run `make migrate-up`; verify with `make migrate-status`. +- [ ] Move the eligibility predicate to `embeddingvectors`; refactor `recommendationItemEligibilityWhereClause` to delegate (assert output string unchanged). +- [ ] Failing tests: (a) `model` filter excludes rows whose `e.model` differs; (b) an **unmatched, non-book** item inflates neither numerator nor denominator (the C1 regression guard); (c) denominator counts eligible items with no embedding row; (d) ratio ∈ [0,1] when a stale embedding exists on a now-unmatched item. +- [ ] Implement both queries with `ELIG`; merge per type in Go; update **all four** callers (pass active model where available, else `""`). +- [ ] `GOWORK=off go test ./internal/embeddingvectors ./internal/catalog -run 'Eligib|Coverage' -count=1` → PASS. Commit. + +### Task 2: Active embedding model from the lock + +**Files:** Modify recommendations `Engine` (`engine.go`), `internal/catalog/search_provider.go`; Test `internal/recommendations/*_test.go`. + +**Produces:** `recommendations: func (e *Engine) ActiveEmbeddingModel(ctx) (string, error)` (→ `GetEmbeddingLock`; `lock.Model` or `""`); `catalog: type CatalogSemanticModelProvider interface { ActiveEmbeddingModel(ctx context.Context) (string, error) }`. + +- [ ] Test: lock present → its `Model`; no lock → `("", nil)`. +- [ ] Implement; confirm `recEngine` satisfies both `CatalogSearchQueryVectorizer` (the real interface name, `search_provider.go:77`) and `CatalogSemanticModelProvider`. +- [ ] Run recommendations tests → PASS. Commit. + +### Task 3: Coverage tracker — querier seam, snapshot, hysteresis, single-flight refresher + +**Files:** Create `internal/catalog/semantic_coverage.go`, `semantic_coverage_test.go`; Modify `internal/catalog/search_provider.go` (consts + gate interface). + +**Produces:** +```go +const ( + semanticCoverageEnableRatio = 0.90 + semanticCoverageDisableRatio = 0.80 + semanticCoverageRefreshInterval = 2 * time.Minute +) + +// coverageQuerier is the seam so Refresh is unit-testable without a real *pgxpool.Pool. +type coverageQuerier interface { + QueryRow(ctx context.Context, sql string, args ...any) pgx.Row + Query(ctx context.Context, sql string, args ...any) (pgx.Rows, error) +} + +type catalogTypeCoverage struct { Type string; Eligible, Vectorized int; Ratio float64; Ready bool } +type semanticCoverageSnapshot struct { PerType map[string]catalogTypeCoverage; Overall float64; Model string; UpdatedAt time.Time } + +type SemanticCoverageGate interface { CoverageReady(itemTypes []string) (ready bool, reason string) } + +type semanticCoverageTracker struct { /* q coverageQuerier; indexTypes []string; models CatalogSemanticModelProvider; mu sync.Mutex; snap atomic.Pointer[semanticCoverageSnapshot]; clock func() time.Time */ } +func (t *semanticCoverageTracker) Refresh(ctx context.Context) error +func (t *semanticCoverageTracker) CoverageReady(itemTypes []string) (bool, string) // nil-snapshot ⇒ (false, "coverage not yet computed") +func (t *semanticCoverageTracker) Snapshot() *semanticCoverageSnapshot +func (t *semanticCoverageTracker) Run(ctx context.Context) // ticker; returns on ctx.Done() +``` + +- [ ] Test (no panic / fail-safe): `CoverageReady` on a freshly-constructed tracker (no `Refresh` yet) returns `(false, "coverage not yet computed")` and touches the querier **zero** times (querier stub fails the test if called). +- [ ] Test (hysteresis): 0.85→not ready; →0.92 ready; →0.85 **stays** ready; →0.79 not ready. +- [ ] Test (scope): `CoverageReady(["movie","series"])` false when `series` < threshold though `movie` ready; reason names `series`. Empty scope requires all snapshot types ready. +- [ ] Test (model collapse): snapshot computed for model A; active model becomes B → next `Refresh` publishes all-not-ready before recomputing. +- [ ] Test (`-race`): concurrent `Refresh` + `CoverageReady` loops; assert monotonic `UpdatedAt`, no race. +- [ ] Test (error retention): a `Refresh` whose query errors retains the prior snapshot. +- [ ] Implement single-flight mutex, fresh-map immutability, nil-safe reads, model-collapse, retain-last-good, ticker `Run`. +- [ ] `GOWORK=off go test ./internal/catalog -run Coverage -race -count=1` → PASS. Commit. + +### Task 4: Wire gate + model provider; comma-ok degradation; start refresher in NewRouter + +**Files:** Modify `internal/catalog/search_meilisearch_provider.go:27-66,329-334` (`MeilisearchProviderConfig` opens at `:27`), `internal/catalog/search_service.go:31-74`, `internal/api/router.go:437-552`. + +**Consumes:** `SemanticCoverageGate`, `CatalogSemanticModelProvider`. **Produces:** `MeilisearchProviderConfig.Coverage SemanticCoverageGate`; `func (s *CatalogSearchService) StartCoverageRefresh(ctx context.Context)`. + +- [ ] `shouldUseSemanticSearch` → `SemanticEnabled && wordCount>=3 && (p.config.Coverage == nil || ready)`; not-ready yields fallback reason `"semantic_not_ready: " + reason` (a **diagnostic, not an error**). Keep the `SkipTotal`-builds-hybrid test green. +- [ ] In `NewCatalogSearchServiceFromSettings`: only when `SemanticEnabled`, derive the model provider via **comma-ok** `mp, ok := queryVectorizer.(CatalogSemanticModelProvider)`. If `!ok` or `queryVectorizer == nil` (semantic enabled but recommendations disabled — a real config), build the tracker with a model provider that yields `""` ⇒ every type not-ready (reason `"embedding model provider unavailable"`); **never** assert without comma-ok (nil-interface assertion panics). Store the tracker on the service; pass it as `Coverage`. +- [ ] `StartCoverageRefresh` runs `tracker.Run(ctx)` once; no-op when there is no tracker (postgres/semantic-off). Call it from `NewRouter` after the service is built: `if catalogSearchService != nil { catalogSearchService.StartCoverageRefresh(deps.AppContext) }`. (The service is local to `NewRouter` and not returned; `deps.AppContext` is the process-lifetime ctx — Baseline #8/#9.) +- [ ] Test: stub gate not-ready ⇒ long query stays keyword-only; ready gate + vectorizer ⇒ hybrid emitted; nil vectorizer + semantic-enabled ⇒ not-ready, no panic. +- [ ] `GOWORK=off go test ./internal/catalog -count=1` → PASS. Commit. + +### Task 5: Diagnostics plumbing (provider → resolver → handler), correctly scoped + +**Files:** Modify `internal/catalog/search_provider.go:64-71`, `internal/catalog/search_meilisearch_provider.go:189-237`, `internal/catalog/catalog_resolver.go:24-30,269-297`, `internal/api/handlers/catalog.go:55-61,91-134`; Test `search_provider_test.go`, `catalog_test.go`. + +**Produces:** `CatalogSearchResult{+Mode string; +SemanticUsed bool}`; `CatalogResult{+Provider,Mode string; +SemanticUsed bool; +FallbackReason string}`; handler `type searchDiagnostics struct { Provider, Mode string; SemanticUsed bool; FallbackReason string \`json:"fallback_reason,omitempty"\` }`; `catalogResponse{+SearchDiagnostics *searchDiagnostics \`json:"search_diagnostics,omitempty"\`}`. + +- [ ] Provider: `Mode="hybrid", SemanticUsed=true` only when the hybrid request is issued **and survives**; on the downgrade (`:221-228`) set `Mode="keyword", SemanticUsed=false, FallbackReason=semanticFallback`. +- [ ] Resolver: copy the four fields in `resolveDirectSearchSource` (direct path only). +- [ ] Handler: populate `search_diagnostics` only on the non-grouped direct path; **omit** for `group=work` and for non-relevance-sort `q=` requests. +- [ ] Tests: (a) hybrid error ⇒ `semantic_used:false, mode:"keyword", fallback_reason` set (post-downgrade truth); (b) existing `total/total_exact/has_more/items/snapshot` byte-stable; (c) field **absent** for browse, for `group=work`, and for a `q=`+title-sort request. +- [ ] `GOWORK=off go test ./internal/catalog ./internal/api/handlers -count=1` → PASS. Commit. + +### Task 6: Capability validation + rate-limited probe; admin status from snapshot + +**Files:** Modify `internal/catalog/search_meilisearch_client.go:167+`, `internal/catalog/search_meilisearch_provider.go`, `internal/catalog/search_service.go:90-145`, `internal/catalog/search_provider.go` (status structs). + +**Produces:** +```go +func (c *meilisearchClient) GetSettings(ctx, uid string) (meilisearchIndexSettings, error) // GET /indexes/{uid}/settings +type meilisearchIndexSettings struct { Embedders map[string]meilisearchEmbedderSettings `json:"embedders"` } +type meilisearchEmbedderSettings struct { Source string `json:"source"`; Dimensions int `json:"dimensions"` } +type CatalogSearchSemanticStatus struct { + Ready bool `json:"ready"`; DisabledReason string `json:"disabled_reason,omitempty"` + CoverageRatio float64 `json:"vector_coverage_ratio"`; CoverageUpdatedAt *time.Time `json:"coverage_updated_at,omitempty"` + PerType []CatalogSearchTypeCoverage `json:"per_type,omitempty"`; Capability CatalogSearchSemanticCapability `json:"capability"` +} +type CatalogSearchTypeCoverage struct { Type string `json:"type"`; Eligible int `json:"eligible"`; Vectorized int `json:"vectorized"`; CoverageRatio float64 `json:"vector_coverage_ratio"`; Ready bool `json:"ready"` } +type CatalogSearchSemanticCapability struct { OK bool `json:"ok"`; Reason string `json:"reason,omitempty"`; Embedder string `json:"embedder,omitempty"`; Dimensions int `json:"dimensions,omitempty"` } +const semanticCapabilityProbeTTL = 5 * time.Minute +``` + +- [ ] Add `Semantic CatalogSearchSemanticStatus` to `CatalogSearchRuntimeStatus`. **Keep** `document_count`/`vector_document_count` unchanged (additive only). The `Semantic` block (ratio, per-type, ready) is read from `tracker.Snapshot()` — **no fresh query** in `Status()`; if no tracker, `Ready=false, DisabledReason="semantic disabled"`. +- [ ] Capability: embedder present, `source=="userProvided"`, `dimensions==embeddingvectors.CanonicalDimensions`; each failure → distinct `Reason`. Probe (unit vector, `limit:1`, `Embedder=config.Embedder`) only when `ActiveIndexUID != ""`, cached `semanticCapabilityProbeTTL`. +- [ ] Tests: missing embedder / wrong source / wrong dimensions / probe failure each distinct; provider `Healthy` and circuit untouched in all four. +- [ ] `GOWORK=off go test ./internal/catalog -count=1` → PASS. Commit. + +### Task 7: Backfill cheap/expensive split with fairness + +**Files:** Modify `internal/recommendations/repo.go:472-621`, `internal/recommendations/similar.go:251-362`; Test `repo_test.go`, `similar_test.go`. + +**Produces:** `buildMissingOrModelStaleEmbeddingIDsSQL() string`; `ListMissingOrModelStaleEmbeddingIDs(ctx, afterID, currentModel string, limit int) ([]string, error)`; `BuildEmbeddingTextForIDs(ctx, ids []string) ([]EmbeddingTextCandidate, error)`; `EmbedAll` two-pass loop with dual cursors and an unconditional `embeddingTextStaleQuotaPerRun`. + +- [ ] Test (SQL shape): `buildMissingOrModelStaleEmbeddingIDsSQL()` contains neither `"item_people"` nor `"LATERAL"`. +- [ ] Test (behavior, no real DB needed for the shape test; behavior tests use existing recommendations test seams): an item that is *only* canonical-text-stale (people changed, model current) is **not** returned by the cheap query; a missing/model-stale item **is**. +- [ ] Test (parity): `BuildEmbeddingTextForIDs(ids)` reproduces the same `canonical_text` as the legacy full CTE for the same ids, regardless of `ids` order. +- [ ] Test (no starvation): with a permanently non-empty cheap backlog, a text-stale item is still embedded within a bounded number of `EmbedAll` runs (quota is unconditional); both cursors stay monotonic. +- [ ] Implement; cheap `afterID` advances from the cheap query's last id. +- [ ] `GOWORK=off go test ./internal/recommendations -count=1` → PASS. Commit. + +### Task 8: Admin UI surfacing + +**Files:** Modify `web/src/hooks/queries/admin/settings.ts:21-54`, `web/src/pages/admin-settings/SearchSettings.tsx:202-261`. + +- [ ] Extend the `CatalogSearchStatus` TS type with the `semantic` block (ready, disabled_reason, vector_coverage_ratio, coverage_updated_at, per_type, capability). +- [ ] Render a "Semantic readiness" row (ready/disabled + reason), overall coverage %, a per-type coverage table, a capability badge, and a "coverage updated" timestamp. Keep "Vectorized Documents" (`vector_document_count`). +- [ ] `cd web && pnpm run lint && pnpm run format:check` → PASS. Commit. + +--- + +## Test Plan + +**Go unit (`internal/embeddingvectors`, `internal/catalog`):** +- Eligibility predicate single-sourced; unmatched non-book item excluded from numerator **and** denominator (C1 guard); ratio ∈ [0,1]. +- Hybrid omitted when gate not ready; emitted when ready + capable + long query + vectorizer present. +- **Model-stale safety:** only-old-model embeddings ⇒ coverage ~0 ⇒ keyword-only; model-change collapse publishes not-ready within one interval. +- Hysteresis latch; mixed-scope AND; empty-scope = all-types; nil-snapshot returns not-ready with **no panic** and **no DB call**; `-race` refresh/read; error retention. +- Capability: missing embedder / wrong source / wrong dimensions / probe failure each distinct; keyword stays up. + +**Go unit (`internal/recommendations`):** +- Cheap SQL excludes `item_people`/`LATERAL`; people-only-stale row excluded from cheap pass; per-ID text build matches legacy output order-independently; no text-stale starvation; dual cursors monotonic. + +**Go unit (`internal/api/handlers`):** +- `search_diagnostics` present on direct path; **absent** for browse, `group=work`, and non-relevance-sort `q=`; `semantic_used` reflects post-downgrade reality; existing fields byte-stable. + +**Admin status:** `semantic.ready`, `disabled_reason`, `vector_coverage_ratio`, `per_type`, `capability`, `coverage_updated_at` sourced from the snapshot (no fresh query). + +**Focused command:** +```bash +GOWORK=off go test ./internal/embeddingvectors ./internal/catalog ./internal/api/handlers ./internal/recommendations -race -count=1 +``` + +--- + +## Descoped — not in this pass + +**Routing legacy `/search?q=` through the provider pipeline.** Sunset **Wed, 01 Jul 2026** (`internal/api/handlers/legacy_read_routes.go:10`, `legacyReadSunset`) — days away; hardening a path scheduled for deletion is negative ROI. The typed legacy path already routes through the resolver; only the bare-`q` fallback hits `itemRepo.Search`, which the sunset removes. Re-routing also risks `browseResponse`↔`catalogResponse` shape drift. Reopen as a one-task follow-up **only if the sunset slips**, with a byte-for-byte `browseResponse` assertion. + +**After-sync coverage refresh trigger.** The indexer (`main.go:1730`) and the search service (`router.go:457`) share no handle, so an after-drain trigger would need new cross-component wiring and would race the interval refresh. The 2-minute interval plus model-change collapse is sufficient; the trigger is intentionally omitted. + +--- + +## Assumptions + +- **One additive index migration** (`media_item_embeddings(model)`, `CONCURRENTLY`). No schema/column changes. *(Revised from the original "no migration": the model-filtered numerator is only cheap with this index, and performance is priority #1.)* +- All runtime coverage state is **in-memory** (atomic per-type snapshot), rebuilt on boot and refreshed every 2 minutes; not persisted. +- Coverage is computed **entirely from Postgres** over the **embed-eligible** population (`status='matched' OR audiobook/ebook`), model-filtered, manga excluded, restricted to configured index types; it **never** uses the Meili `document_count`. +- Active embedding model comes from `recommendations.EmbeddingLock.Model`; **no lock / no model provider ⇒ semantic not ready.** +- A model change is reflected within ≤ one refresh interval and **fails safe** within that window via the model-collapse rule; the residual ≤2-minute mismatch window is acceptable for a rare, admin-gated reconfiguration. +- Thresholds (enable 0.90 / disable 0.80) and the 2-minute interval are named consts and starting heuristics; the diagnostics shipped here are the instrument for retuning them. Promoting them to settings is a possible follow-up, not this pass. +- `semantic_ratio` stays `0.30` (`DefaultMeilisearchSemanticRatio`). +- Scope is hybrid-search hardening only — not conversational search. + +--- + +## Self-Review (writing-plans checklist) + +- **Findings coverage:** C1 eligibility denominator (D1, Task 1) · C2 empty-scope semantics (D3, Task 3) · missing `model` index (D1 migration, Task 1) · 4-caller signature change (Baseline #4, Task 1) · refresher wiring in `NewRouter`/`AppContext` (Baseline #9, Task 4) · concurrent-refresh single-flight + immutability + `-race` (D2, Task 3) · nil-snapshot/error fail-safe (D2, Task 3) · model-change collapse (D2, Task 3) · comma-ok nil vectorizer (Task 4) · querier seam + SQL-string helper for testability (Task 3, Task 7) · grouped/non-relevance diagnostics scoping (D4, Task 5) · unit probe vector + admin-only/5m (D5, Task 6) · `Status()` reads snapshot (Task 6) · backfill cursors + fairness (D6, Task 7) · legacy & after-sync descoped (Descoped) — all mapped. +- **Type consistency:** `SemanticCoverageGate`/`CatalogSemanticModelProvider`/`coverageQuerier`/`semanticCoverageTracker`/`catalogTypeCoverage`/`CatalogSearchSemanticStatus` used identically across tasks; `Eligible` (not `Indexable`) is the denominator field name throughout. +- **No placeholders:** every new symbol has a signature; SQL is concrete; the migration body is specified.