Findings from a full review of the Meilisearch implementation:
- Give the indexer its own 2m HTTP timeout instead of reusing the 800ms
search-path timeout_ms, so large document uploads to a non-loopback
Meilisearch stop timing out.
- Delete superseded indexes after a rebuild (previous active + leftover
<prefix>_rebuild_* partials); every rebuild previously leaked a full
copy of the catalog on the Meilisearch instance.
- Cache the index state row + pending count for 3s on the search hot
path (was two Postgres round trips per search request); a failed
search invalidates the cache immediately.
- Swap the active-index pointer before marking outbox events processed
so a crash between the two replays events instead of losing them.
- End pagination only on a short page; estimatedTotalHits is an
estimate and could truncate results.
- Latch the startup-resolved provider process-wide so package-level
enqueue helpers stop querying server_settings in write transactions.
- Surface dead-lettered outbox events in the admin status + web UI.
- Remove unwired provider config knobs, dedupe the manga-chapter
exclusion predicate, split the vector cache onto its own mutex, real
rebuild progress percentages, and expand client/coalesce test
coverage.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Add the in-memory coverage gate the search hot path will consult to
decide whether semantic results may be served for a set of item types.
The tracker reads an atomic.Pointer snapshot with no DB and no lock on
the read path, refreshes single-flight under a mutex on a 2-minute
ticker, and applies per-type hysteresis (enable 0.90 / disable 0.80,
holding the previous latch inside the band) to avoid flapping. It
collapses to not-ready when the active embedding model changes so a
stale latch cannot leak across models, and fails safe: a not-yet-
computed snapshot reports not-ready (never panics), and a model-lookup
or count-query error retains the last-good snapshot instead of
publishing zeros.
Refresh counts through the existing catalogSemanticCoverageByType via an
injectable fetch seam, so the behavior is unit-testable with canned
counts and a fake model provider (no pgx.Rows faking). Reuses the
Task-1 coverageQuerier and extends catalogTypeCoverage with Ratio/Ready,
filled by the pure computeCoverageSnapshot helper. Adds the
SemanticCoverageGate interface beside the other provider interfaces;
Task 4 wires the gate and starts the refresher.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Establish how semantic vector "coverage" is counted: current-model
embeddings over embed-eligible items, per media type, from Postgres.
- Move the embed-eligibility predicate to a single source of truth in
embeddingvectors.ItemEligibilityWhereClause; recommendations now
delegates to it (output byte-identical).
- Add catalogSemanticCoverageByType (per-type eligible/vectorized) and
define the coverageQuerier interface in catalog. Both numerator and
denominator apply the eligibility predicate, so vectorized never
exceeds eligible (C1 guarantee) even with a stale embedding on a
now-unmatched item.
- countCatalogSearchVectorDocuments now takes a coverageQuerier and a
model, applies the eligibility + model filter, and is the per-type
numerator summed across types. Update all four callers (pass "").
- Add idx_media_item_embeddings_model (CONCURRENTLY, self-healing
guard) so the model filter does not scan the embeddings table.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
