* fix(jellycompat): sync activity dashboard immediately on compat playback start/stop
Compat (Infuse/Jellyfin) sessions were only reconciled into
playback_sessions_sync on the periodic 15s reconciler tick, so stopped
streams lingered in the admin activity dashboard and overlapped with
newly started ones as ghost sessions. Native playback handlers already
trigger an immediate SessionSyncer.SyncNow on start/stop; wire the same
syncer into the jellycompat playback handler and flush after
teardownPlaySession (Stopped report + ActiveEncodings teardown) and
after a new upstream session starts.
The stop-path sync detaches from request cancellation
(context.WithoutCancel) because clients often drop the connection right
after reporting a stop.
Also adds the ListProgressSince method to the jellycompat test fake so
the package's tests compile again after the downloads-v2 interface
change.
Part of #205
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(worker): serialize SyncNow and bound compat request-path session sync
Adversarial review follow-ups: SyncNow snapshots could commit out of
order when request-path syncs raced the periodic tick (an older
snapshot committing last would resurrect stopped sessions or drop
fresh ones), and the detached stop-path sync had no deadline, letting
a stalled DB pin request goroutines. Serialize snapshot capture +
reconcile under one lock and cap request-path syncs at 5s so failures
degrade to the periodic tick.
Part of #205
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(worker): coalesce immediate session syncs instead of queueing on a lock
Review iteration 2: a stop-triggered sync queued on a bare mutex could
expire its 5s deadline waiting behind a slow tick and fail before
removing the stopped row. Coalesce instead: one owner reconciles at a
time, callers that arrive mid-flight return immediately after flagging
a follow-up pass, and the owner re-captures a fresh (post-change)
snapshot afterwards. Request goroutines never block behind another
sync, snapshots still commit in capture order, and an expired-context
owner leaves the queued pass for the next tick instead of burning it.
Part of #205
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(web): detect HDR via video-dynamic-range media query for Firefox
Firefox's (dynamic-range: high) query reflects the browser canvas and
reports standard even on HDR-enabled displays, so the player capability
check always sent hdr=false. Firefox 116+ exposes the video plane via
(video-dynamic-range: high); accept either query. Unknown media features
parse as non-matching, so the extra query is safe in other browsers.
Closes#206
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(web): keep Window receiver when passing matchMedia to HDR helper
Invoking a detached matchMedia reference throws (Illegal invocation /
TypeError) in some browsers, crashing capability detection instead of
reporting HDR. Wrap it in an arrow so the Window receiver is preserved.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* perf(literaryworks): cut Postgres load on ebook↔audiobook auto-linking
Rescans and candidate matching were driving high Postgres CPU on book
libraries.
- AutoLinkContent now checks literary_work_items with a cheap indexed
lookup before GetMatchItem, so unchanged already-linked books skip the
heavy triple-lateral query on every rescan.
- ListMatchCandidates is split into an indexable ID-selection phase
(title / provider EXISTS / series EXISTS) and a hydration phase, so the
per-row lateral aggregates run only for the LIMIT candidates kept rather
than for every opposite-format book.
- Add a partial LOWER(title) index scoped to ebook/audiobook so the OR of
title/provider/series filters can be driven entirely by indexes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(migrations): drop invalid leftover index before concurrent rebuild
An interrupted CREATE INDEX CONCURRENTLY (cancel/restart) can leave an
INVALID idx_media_items_books_title_lower; IF NOT EXISTS would then skip
the rebuild while Goose records success. Add the preflight invalid-index
drop used by the repo's other concurrent-index migrations.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(downloads): offline sync for mobile (downloads v2)
Replace internal/download with a unified internal/downloads package and add
fully-offline download + watch-sync support for mobile clients, across five
independently-shippable phases:
- Phase 0: reshape the downloads table and the /downloads contract to be
device- and format-aware; add GET /downloads/capability; extend
DownloadConfig (default-off keys); update the web download hooks/components in
lockstep. This is the one approved pre-lock exception to the additive-only
/api/v1 rule (the web app is the only consumer and is updated together).
- Phase 1: managed device-library entries (create/list/PATCH/delete/serve),
keyed on the X-Silo-Device-Id header.
- Phase 2: offline playback manifest plus artwork/subtitle proxy endpoints that
strip every presigned URL (inline thumbhashes + authenticated proxies).
- Phase 3: prepare-to-file (remux + transcode-to-single-file) as a durable,
leased artifact queue with startup recovery, hosted on the task manager;
playback.PrepareFile emits one +faststart MP4. Adds the admin transcode
toggle and per-artifact LRU cleanup.
- Phase 4: offline progress reconciliation -- a clamped event_at LWW key plus a
server-assigned synced_seq cursor on watch_progress; an optional clamped
updated_at on POST /sync/progress and an opaque ?since= cursor on
GET /progress (additive; existing callers unaffected).
Security & reliability invariants, each with an acceptance test:
1. Server-owned sync ordering: ?since= delta delivery is driven only by the
server-assigned synced_seq; the client clock is bounded (event_at, clamped
to now+skew) and used only for last-write-wins on the caller's own profile.
2. Full profile+device authorization on every managed endpoint, with a
per-profile content/library access re-check before serving any bytes/assets.
3. Durable artifact recovery: a transactionally-claimed (FOR UPDATE SKIP
LOCKED), lease-heartbeat, attempt-counted queue with a startup sweep, so no
crash strands a download in preparing and concurrent workers never
double-encode.
Migrations are timestamped Goose files: reshape downloads (device/format);
download_artifacts (durable queue); watch_progress event_at/synced_seq.
DB-backed acceptance tests skip without SILO_TEST_DATABASE_URL and run in CI;
the invariant-1 progress test also runs against the real SQLite backend locally.
Client repos (silo-android, silo-apple) consume the reshaped /downloads/*
contract and the updated_at/?since= progress fields and require coordinated
follow-up.
Implements the maintainer-approved v1 capability proposal for offline sync
(downloads v2).
AI-use disclosure: implemented by Claude (Claude Code) from the approved design
doc under docs/superpowers/specs, with human review.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(downloads): series & season downloads + client-pull monitoring
Build season downloads and a "monitor a series" capability on top of the
downloads v2 (offline sync for mobile) work.
Season downloads:
- POST /downloads accepts season_number (with series:true) to download one
season. CreateSeries/CreateSeason share one body via a listEpisodes closure
and register managed entries under a shared batch_id (original-only). Episode
files are resolved in a single batched query.
Series monitoring (auto-download), client-driven:
- New device-scoped download_subscriptions table with a Sonarr-style mode
(all | future | latest_season | specific_seasons), a client-enforced
delete_watched flag, and a max_storage_bytes cap. The server never deletes
on-device files; retention and the hard cap are the client's, the server
only soft-gates registration.
- The client calls POST /downloads/subscriptions/sync on open / background
refresh; the server registers the in-scope, not-yet-downloaded episodes
(idempotent via the managed-entry unique index) and the device pulls them on
its own schedule. No background worker and no dependency on the notifications
subsystem. latest_season follows new seasons (>= subscribe-time season);
future excludes the back catalog via air date.
- Subscription CRUD + sync are profile+device authorized (device id from the
X-Silo-Device-Id header only) with a per-request content-access re-check. The
capability endpoint advertises season_download / series_monitoring /
monitoring_modes.
Also lands the downloads-v2 work already present in the tree: durable artifact
(remux/transcode) preparation and offline watch-progress reconciliation, plus
the design-spec updates.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* WIP: epitaxy pre-switch from feat/downloads-v2-offline-sync
* test(downloads): fix deterministic ID collision in reconcile test
Artifact IDs are time-sortable, so two artifacts created in the same
moment share their first 8 chars; combined with a captured timestamp the
two preparing-download IDs collided on downloads_pkey. Use the full
artifact ID, which is unique per row.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(downloads): support sqlite userdb backend for managed downloads
With the sqlite userdb backend, profiles live only in per-user SQLite
stores and public.user_profiles stays empty, so user_devices'
profile FK made every managed create/subscription/offline-sync request
fail with an FK violation. Drop the FK (shared Postgres tables must not
FK profile tables — same rule as notifications) and replace the lost
cascade with an app-level purge on profile deletion, wired through
ProfileHandler for both backends. DB-backed regression tests cover the
no-Postgres-profile-row path and the purge cascade.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(downloads): dispatch encode kick asynchronously
triggerDrain invoked the kick inline, and the kick (taskmanager RunTask)
executes the encode task on the caller's goroutine — so a POST
/api/v1/downloads with a bitrate quality blocked the HTTP request on the
entire queue drain, ffmpeg encodes included, delaying the 202 by minutes
on an idle queue. Dispatch the kick on a goroutine; the task manager
already serializes concurrent runs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(downloads): enforce per-user quota on the encode pipeline
Two gaps let a user bypass MaxConcurrentPerUser entirely for prepared
downloads: artifact-backed rows are created in 'preparing' (never
'queued'/'downloading'), which CountActiveByUser didn't count, and
createArtifactDownload enqueued the encode job before limiter.Check, so
even a 429-rejected request left a job the worker would transcode.
Count 'preparing' as active and check the limiter before Ensure; managed
replacements stay quota-exempt since they don't add a row.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(downloads): protect ephemeral artifact links from LRU eviction
HasActiveLink only counted managed (device_id IS NOT NULL) rows, so
under a byte budget Cleanup could delete an artifact still referenced by
a ready-but-unfetched ephemeral web download — permanently 404ing a row
the API kept listing as ready (the artifact row is gone, so recovery
can't re-queue it). Any non-terminal link now protects the artifact;
only artifacts whose links are all cancelled/failed/revoked are
evictable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(downloads): batch manifests skip bad entries instead of failing whole batch
One deleted or access-filtered episode made GET
/downloads/batches/{id}/manifests 404 for the entire season, so a
client could no longer fetch manifests for the still-valid entries.
Report unbuildable entries in a skipped[] array (revoked | not_found |
error) alongside the delivered manifests, mirroring the create path's
skip idiom. Also cut the batch cost: the shared series detail is
resolved once per batch instead of once per episode, and buildSubtitles
reuses the already-loaded media file instead of re-querying it per
manifest.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(migrations): wrap DO block in StatementBegin/End markers
Under NO TRANSACTION goose splits statements on semicolons, so the
dollar-quoted DO block failed every fresh install with 'unterminated
dollar-quoted string' (SQLSTATE 42601). Already-applied databases are
unaffected. Same fix is being applied to main; identical content merges
cleanly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(api): allow season 0 (Specials) in season downloads
season_number was a plain int dispatched with '> 0', so requesting the
Specials season was indistinguishable from omitting the field and
silently broadened to a full-series download. Dispatch on pointer
presence, treat 0 as the Specials season, and reject negatives with 400.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(downloads): capability quality_presets is never JSON null
PresetsFor returned a nil slice when downloads are disabled or the user
lacks the permission, and Capability's []string{} initialization was
immediately overwritten by it — so GET /downloads/capability serialized
"quality_presets": null where the contract documents an array.
Normalize at the source so every caller inherits the guarantee.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(downloads): subscription sync correctness + batched registration
Three subscription fixes:
- A paused subscription no longer syncs: PATCHing scope (or pausing and
changing scope in one request) registered episodes for a monitor the
user had just stopped, inconsistently with SyncSubscriptions' guard.
- SubModeFuture compares calendar days (UTC): air_date is date-only, so
the strict instant comparison permanently excluded episodes airing the
same day the user subscribed; episodes with no air date now fall back
to their ingest time instead of never registering.
- Registration is one batched fetch (GetManagedEntriesByKeys) plus one
batched INSERT ... ON CONFLICT DO NOTHING RETURNING
(CreateManagedEntriesBatch) instead of a SELECT+INSERT per episode —
a 300-episode series cost ~600 sequential round trips per request and
every no-op sync re-walked the full set. RETURNING yields exactly the
new rows, so the sync response's 'registered' count now honestly
reports 0 in the steady state instead of the full in-scope count on
every app open. The now-unused InsertManagedEntryIfAbsent is removed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(userstore): stamp triggers own the event_at LWW key
MarkProgressBatch (jellycompat series mark-played) advanced updated_at
but never event_at, and both stamp triggers only defaulted event_at when
NULL — so a queued offline event with a client time between the row's
old event_at and the mark could win SetProgressIfNewer and resurrect a
stale resume position that then re-synced to every device.
Make the triggers authoritative instead of adding a tenth hand-written
SET clause: whenever an UPDATE changes updated_at without explicitly
changing event_at, the trigger advances the LWW key; writes that do set
event_at (offline sync's clamped client event time) keep their value.
Postgres gets a CREATE OR REPLACE migration; SQLite gets a v12 userdb
migration that drops and reinstalls the trigger bodies (CREATE TRIGGER
IF NOT EXISTS never replaces). Conformance tests cover both batch paths,
the preserved-client-time invariant, and the v11→v12 upgrade.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(downloads): lifecycle hygiene — squash migrations, dead status, stale-row sweeps
Migrations: fold the 20260621 corrective migration back into the base
Downloads V2 migrations (its columns/constraints already exist there)
and fix the reshape Down, which re-added the narrow status CHECK without
collapsing managed-lifecycle rows first — rollback aborted on any DB
with preparing/ready/revoked rows; validated against a live row. Branch
databases that applied the corrective migration need its version row
removed: DELETE FROM goose_db_version WHERE version_id = 20260621020459.
Code: drop the dead 'registered' status (nothing ever wrote it; the
lifecycle is preparing -> ready; 'revoked' stays reserved for the
planned admin revoke flow) along with unused KindDirect and
ErrInvalidFormat.
Sweeps: Cleanup now runs an age-based hygiene pass independent of the
byte budget — cold terminally-failed artifacts (with .part leftovers),
orphaned ready artifacts no download row references, and ephemeral web
rows older than their convenience-record lifetime (also unpinning their
artifacts and bounding GET /downloads growth). The byte budget remains
the disk quota per the limits & restrictions design.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(downloads): sync API doc with v2 fixes; HEAD on file route; Android handoff
Document the contract changes from the review fixes: batch-manifest
skipped[] shape, honest subscription 'registered' semantics, season 0 =
Specials, always-array quality_presets, bytes_sent actual behavior,
ephemeral 7-day retention, header-pairing requirement, progress-delta
deletion caveat, and the ready/failed push event schema (new §9.4).
Add an Android client handoff section (§11) mirroring the Apple one,
register HEAD on /downloads/{id}/file for download stacks that probe
before ranged GETs, and add season_number to the web create-request
type. Flag the /direct-download session-token-in-URL tradeoff; a
short-lived download-scoped URL is a follow-up.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor: consolidate download/progress helpers, prune dead code, gate sweeps
Behavior-preserving consolidation from the Downloads V2 review:
- appendVideoFilterArgs: one home for the burn-in/hwaccel -vf selection,
shared by the HLS builder and the single-file prepare builder (the
drift pattern that already bit tone-mapping once).
- userstore.ResolveProgressState: one home for the min-resume/watched
threshold rule, replacing five identical copies across both store
backends and the offline-sync ingest.
- Download file selection ranks resolutions via access.CompareQuality
(adds 4320p, agrees with playback) instead of a private switch.
- writeSubtitle uses the shared subtitles.SubtitleContentType mapping.
- config.DefaultTranscodeDir replaces three '/tmp/silo-transcode'
literals.
- Read-side quality/revision defaulting helpers removed: insertArgs plus
the NOT NULL/CHECK schema already guarantee the invariant.
- Dead code removed: Repository.ListByUser, SubscriptionRepository.
ListActiveBySeries, and the stale auto-register-worker comments (the
design is client-pull; no worker exists).
- Redundant left-prefix indexes dropped from the base migrations (their
unique indexes serve the same prefixes).
- recover()'s disk-presence sweep and the stale-row hygiene sweep run on
startup then hourly instead of every 30s tick (both are O(cache
size)).
- gofmt/prettier fixes for pre-existing drift in handlers/playback.go
and pages/Profiles.tsx.
Deferred (noted for follow-ups): quality-ladder preset table collides
with the drafted download limits & restrictions design, which specifies
its own ladder helper; Download-literal construction consolidation and
the managed-identity value object remain open.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(downloads): draft download limits & restrictions design
Design input for the follow-up v1 capability proposal (quality ceiling,
batch size cap, per-user quantity/bandwidth overrides). Committed with
downloads v2 because the remediation work explicitly defers the quality
ladder refactor and revocation wiring to this spec.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(progress): reject malformed updated_at; clamp negative progress inputs
Review findings on #258:
- A malformed (non-RFC3339) updated_at in POST /sync/progress previously
parsed to the zero time, which clampEventAt treated as "now" — letting a
stale offline event win LWW as a fresh server-time write. The item is now
rejected with a per-item error instead.
- ResolveProgressState now clamps negative position/duration before
classification so no backend can persist negative progress through
UpdateProgress/SetProgress.
- The online-write event_at invariant test is table-driven over both
SetProgress and UpdateProgress, which share the same contract.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(downloads): close review gaps — permission gates, file-access recheck, artifact-true manifests
Review findings on #258:
- UpdateSubscription now applies the same feature/DownloadAllowed gate as
CreateSubscription and SyncSubscriptions; a PATCH could previously
re-activate or widen a monitor and register managed rows after an admin
disabled downloads or revoked the user.
- Serving download bytes (managed and ephemeral) and /direct-download now
mirror playback's per-file authorization via catalog.FileAllowedByAccess:
library scope and the profile's max playback quality are re-checked at
serve time, with artifact-backed rows checked against the artifact's
resolution (a 720p transcode of a 4K source stays servable under a 1080p
ceiling).
- Offline manifests for remux/transcode entries now describe the prepared
artifact (container, codecs, resolution, single selected audio track)
instead of the catalog source file the client never receives.
- ArtifactRepository.Requeue reports ErrNotFound when the row was
concurrently swept; ArtifactManager.Ensure recreates the job in that case
instead of linking downloads to a dead artifact id.
- "No downloadable episodes" is a sentinel (mapped to 404
no_downloadable_episodes) rather than a bare error that surfaced as 500.
- Subscription season_numbers are bounds-checked (0–9999) before the int32
narrowing in the repo could silently wrap them.
- HandlePatchDownload reuses requireManaged instead of hand-rolling the
same managed-identity checks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The web subtitle handler wrapped the downloaded-subtitle lookup in
`if err == nil { ... }`, so a real DB/backing-store failure from
ListDownloadedSubtitles was silently swallowed and control fell through
to a generic 404 ("Subtitle track not found") with no server-side signal.
This made genuine internal failures look like an intermittent client-side
"subtitles won't render" bug and left nothing in the logs to diagnose.
On error, log at ERROR (with file_id/track/error, matching the sibling
font-extraction path) and return 500 — mirroring the neighbouring error
handling in the same branch (S3 GetObject -> 502, ConvertToVTT -> 500).
The genuine not-found fall-through is preserved for the case where the
listing succeeds but the requested index is out of range.
Closes#248
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
os.ReadDir reports lstat semantics, so a symlink pointing at a directory
has IsDir() == false and was silently dropped by HandleBrowse. Library
folders reached through directory symlinks (e.g. a Docker bind mount whose
entries link into a FUSE mount) never appeared in the folder picker even
though the targets were valid and browsable.
Resolve non-directory entries with os.Stat to follow the link and include
them when the target is a directory; entries whose target cannot be stat'd
(broken or looping links) are skipped rather than erroring the request.
Regular directories keep the lstat fast path and take no extra syscall.
Closes#208
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(watchsync): omit empty Trakt episode ids so series mark-watched targets the right show
omitempty on a struct value is a no-op in encoding/json, so the history
export sent all-zero episode ids ({tmdb:0,tvdb:0}); Trakt matched the
degenerate id to one default show, mis-recording every watched series.
Make episode IDs a *traktIDs pointer and attach it only when a real id
exists, else use the show + season/number fallback (mirroring scrobble).
Adds a debug log on the show-fallback path and payload tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(watchsync): sync TV episodes to Trakt via valid nested shows payload
Episode history exports were dropped by Trakt because they were emitted
into the flat episodes[] array with a bogus sibling show object plus
season/number keys — a shape the Trakt API does not accept, so it
silently discarded them (200/201 with no history recorded). Movies were
unaffected since they always carry their own external IDs.
Two coordinated changes:
- watchstate/identity.go: ResolveHistoryIdentity now carries the
episode's own imdb/tmdb/tvdb IDs (already stored on the episodes
table) so episodes with real IDs export via the flat episodes[].ids
form, matching how movies work.
- watchsync/providers/trakt/provider.go: episodes without their own ID
now serialize into the correct nested shows[].seasons[].episodes[]
structure keyed by the show's IDs, merging episodes by show/season.
Same fix applied to the history-remove payload. Empty-payload guards
account for the new shows[] list.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(watchstate): keep episode identity when episode has its own IDs
Address CodeRabbit review on PR #254: the episode identity builder
dropped the whole identity whenever series IDs were empty, so episodes
with a valid episode IMDb/TMDB/TVDB ID but no series IDs never reached
the flat episodes[].ids Trakt path. Only require series IDs when the
episode has no IDs of its own (nested show fallback).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add push notifications support
* fix(notifications): address push notification review findings
- Gate the capability endpoint's apple_push availability on the admin
delivery toggle, matching web push: Available now means setup will
actually deliver.
- Reject direct admin writes to push_relay_deployment_id/api_key; the
relay issues them as a pair during registration and a lone write
desyncs them (and poisons the next rotation request).
- Purge a device's registrations under other profiles when it
re-registers, so a profile switch on a shared device stops the old
profile's pushes (attempts cascade); adds a DB-backed test.
- Extract the shared channelDispatcher core + retry sweep and rebuild
the webhook/web push/Apple push dispatchers on it instead of keeping
three copies of the worker-pool/retry loop.
- Deduplicate relay URL validation (admin setting + register flow) and
the push outbox attempt-building loops behind shared helpers.
- Cap free-text decline reasons in notification display bodies.
- Fix TestHandleApplePushDisplayDB expectations to match the shared
display copy (test previously failed under SILO_TEST_DATABASE_URL).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(notifications): route push relay URL writes through registration only
Direct writes to notifications.push_relay_url via the admin settings
endpoint bypassed the relay registration flow, letting the stored URL
drift out of sync with the deployment id / API key pair the relay
minted for it. Reject the URL alongside the deployment id and API key
in the settings handler; POST /admin/notifications/push/relay/register
remains the only path that persists all three together.
The admin UI's Relay URL field now edits local draft state and is
applied by the Register/Rotate action instead of the settings save.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
- preferredAudioStreamIndex: when default track is TrueHD/MLP and device
profile doesn't explicitly list that codec, falls back to best compat
track (EAC3 > AC3 > DTS > AAC…), same language preferred. Prevents
Android TV clients receiving undecoded lossless passthrough streams.
- audioTrackDisplayTitle: structured display title (e.g. "English - EAC3 5.1")
replacing the raw firstNonEmpty fallback. Suppresses generic embedded
labels (Stereo, Surround) that add no information.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* perf(jellycompat): fast-paths for slow Jellyfin-compat queries
Rebuilds the jellycompat slow-query fast-paths cleanly on main, without
the hub-sections virtual-library feature (which is intentionally absent
on this branch). Each path is bounded so behavior stays predictable
under load.
Fast-paths:
- Episode catalog: add the load-bearing `si.type = 'series'` hydration
guard plus a matching entry-scan series-parent guard, so the page and
count scans stay aligned with hydration (no under-filled pages, no
inflated TotalRecordCount from podcast/audiobook episode rows).
- /Shows/{id}/Episodes AdjacentTo: serve a bounded prev/self/next
neighbor window with series-membership validation and a cross-series
id guard, falling back to the full listing.
- Resume: serve Continue Watching through the capped native
continue-watching fetcher (next-up suppressed, in-progress only)
instead of loadProgressPage's unbounded scan. Wires only the
sections read-time fetcher (StoreProvider) into ItemsHandler; no
hub-section/virtual-library exposure.
- Played-history: push the type + library filter down into SQL.
- recently_added: fan a no-parentId browse out across libraries with an
app-side dedup merge, clamped to BrowseFilters.MaxLimit, and exclude
compat-hidden (audiobook/podcast) libraries from the fan-out.
- Detail fields: batch per-item GetItemDetail in the Latest/browse
detail-field paths via GetItemDetailsByIDs, with per-item fallback.
- Search: term-length and result-count guards for aux search.
* perf(jellycompat): clamp Resume fast-path fetch to maxResumeItems
A large client-supplied StartIndex on /UserItems/Resume inflated
loadResumeViaSections' fetchLimit (pageSize + StartIndex) and fed it
straight into ResolvedSection.ItemLimit, letting the continue-watching
fetcher scan up to its continueProgressMaxScanned hard cap (1000 rows)
instead of stopping near maxResumeItems. Clamp fetchLimit to
maxResumeItems and short-circuit to an empty page when StartIndex is at
or beyond the cap (no visible row can land there), keeping the capped
fetcher's scan bounded and predictable under load.
Addresses CodeRabbit review finding on PR #247.
* fix(metadata): merge duplicate people on refresh instead of looping on 23505
The background person-refresh worker re-selected the same people every cycle.
When a refresh resolved an external id (tmdb_id/imdb_id) already held by another
people row, the UPDATE violated a partial-unique index and raised SQLSTATE
23505; the tx rolled back so updated_at never advanced and FindCandidates
re-qualified the row forever. The underlying cause is two people rows for the
same human created from credits ingested with disjoint id sets (one tmdb-only,
one imdb-only) that BatchFindOrCreate never reconciled.
PersonRepository.Update now reconciles the collision instead of failing:
- The common no-collision path stays a single plain transaction (no added cost).
- On a 23505, updateResolvingConflicts runs the whole reconciliation in ONE
transaction, retrying the write via savepoints so it commits atomically: it
locks both rows FOR UPDATE in id order, then either merges the partner into
the survivor (repoint item_people skipping duplicate credits, fold the
partner's ids/fields onto the survivor, delete the partner) or, when the rows
are not confidently the same human, drops just the conflicting id.
- canMergePeople requires compatible ids AND matching names, so a provider that
hands the same id to two genuinely different people cannot trigger a
destructive delete; that case falls to the non-destructive drop.
- A row merged away concurrently surfaces as pgx.ErrNoRows, which the refresh
service maps to ErrPersonNotFound.
Existing stuck rows self-heal: they are still re-selected each cycle and now
merge (or drop) instead of looping, draining the warning population to zero.
Adds unit tests for the merge-decision logic (guard, compatibility,
field-folding, constraint mapping).
AI-use disclosure: implemented and adversarially reviewed with AI assistance.
* fix(metadata): preserve survivor's existing id when declining a person merge
The non-mergeable branch of resolveExternalIDConflict blanked the
conflicting external-id field before retrying the write. Because
execPersonUpdate is a full-row UPDATE, the retry persisted an empty
string and returned success, silently dropping a previously-valid
provider id (e.g. the admin PATCH path that mutates an existing id into
a colliding value) instead of leaving the row unchanged as the 23505
did.
Restore the locked survivor's currently-persisted value for the field so
the retried write is a no-op on that column: it commits without looping,
without deleting a possibly-distinct person, and without blanking an id
the survivor already held. Writing a row's own current value back can
never violate the unique index, so the field will not re-trigger the
conflict. The refresh-worker path is unchanged (survivor value is empty).
Convert clearExternalIDField into a general setExternalIDField setter and
extend its unit test to cover set-to-value and set-to-empty.
* fix(jellycompat): guard aux search paths and index-back person search
Short, recursive type-ahead terms (e.g. a single "a") against the
PostgreSQL people index and the in-memory collection/box-set filter
produced multi-second scans that pegged the server when a client fired
one search per keystroke. Every jellycompat search path except the
Meilisearch-backed /Items media search now rejects a provided SearchTerm
shorter than 3 runes without touching any backend, and caps results at
20 regardless of the client-requested Limit:
- /Persons (PostgreSQL people scan)
- /Search/Hints (catalog search)
- /Items BoxSet (in-memory collection filter)
Shared policy + helpers live in search_guard.go (auxSearchMinTermLen=2,
i.e. reject 1-2 runes / allow 3+, and auxSearchMaxResults=20) with unit
coverage. The 3-rune floor is deliberate: 3 runes is the point where a
pg_trgm trigram index becomes usable, so the gate lines up with the index
and still lets legitimate short titles/names ("300", "Saw", 3-letter
actors) and 3-char type-ahead hints through.
PersonRepository.Search filters with `name ILIKE '%'||$1||'%'` rather
than `LOWER(name) LIKE '%'||LOWER($1)||'%'`. The old expression filtered
on LOWER(name), which the trigram GIN index idx_people_name_trgm (built
on name) could not serve, so every search fell back to an ordered index
scan on idx_people_name that walked the whole table for rare terms
(~300-400ms on the 889k-row production people table). ILIKE on name lets
pg_trgm serve rare 3+ char terms from the trigram index, while the planner
still picks the ordered btree scan with early termination for common
terms. ILIKE is case-insensitive, so behavior is preserved (including the
pre-existing treatment of % and _ in the term as LIKE wildcards).
Verified on production silo-postgres (889,302 people), parameterized query
under both custom and generic plan cache modes:
rare 3-char 'qzx': 304ms -> 1-8ms (trgm bitmap index)
rare 4-char 'zzzz': 329ms -> 1-2ms (trgm bitmap index)
common 3-char 'ann': 15ms -> 7-74ms (btree early-stop / trgm bitmap)
New worst case across all terms is ~83ms (generic-plan common 3-char).
* fix(jellycompat): stop gating meilisearch hints, clamp box-set search
Two review follow-ups on the aux-search guards:
- HandleSearchHints is served by the catalog (Meilisearch-backed) search
provider, which already bounds and short-term-handles its own results.
Gating it with auxSearchTermTooShort contradicted the guard's own
"non-Meilisearch paths only" policy and hid valid 1-2 char titles
("Up", "It") from global type-ahead. Drop the too-short gate there;
keep the empty-term check and the result clamp.
- handleBoxSetsList gated short terms but passed query.limit straight to
slicePage, which treats <=0 as no cap, so a box-set *search* was
uncapped unlike Persons/Hints. Clamp the limit when a search term is
present; empty-term browse keeps its client paging window.
Folds in findings from Anthropic's Claude Code guidance and the agentic-SWE
literature:
- Manual-only trigger (disable-model-invocation) for this side-effecting,
PR-creating workflow, per Anthropic's fix-issue template.
- Reproduce-first hard gate: confirm the bug exists on current main (red test
before the fix); stop and report if it doesn't reproduce, rather than patching
correct/stale code.
- Verification is the primary correctness signal; full-suite run for regressions;
forbid reaching green by weakening/skipping tests or relaxing CI.
- Adversarial-review loop: confirm each finding is real/grounded before acting,
fix only correctness/requirement-affecting findings (no gold-plating), and cap
the loop at 2 (max 3) iterations before escalating.
- Guards against hallucinated APIs, plan-proportional-to-scope, and long-run
context-loss (progress + failed-approaches notes, re-assert guardrails after
compaction).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds a committed, shared Claude Code skill that takes a GitHub issue number,
creates an isolated worktree off the latest main, finds the root cause or
scopes the feature, implements and verifies the change, runs a Codex
adversarial review, and opens a ready PR.
Carves a narrow exception into the .gitignore /.claude/ rule so only this
skill directory is tracked; everything else under .claude/ stays ignored.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
media_items had many columns the Go model (models.MediaItem) declares as
non-pointer string/int fields, but migration 001 left them nullable. Five
scanners (catalog item_repo, catalog browse, jellycompat, sections, and
the catalog API handler) read these straight into the non-pointer fields,
so a NULL row panics with "cannot scan NULL into *string" (or *int).
item_repo papers over a subset (poster/backdrop/logo/metadata paths) with
COALESCE in its SELECT, but the other four scanners read the same columns
raw and crash; sort_title/original_title/etc. are not coalesced anywhere.
No writer stores a meaningful NULL (every insert/upsert passes the Go
field, '' or 0 at worst) and all sort/filter SQL already collapses NULL
and '' (e.g. COALESCE(NULLIF(BTRIM(sort_title), ''), title); "poster_path
IS NULL OR poster_path = ''"). Enforce the invariant the code already
assumes at the schema level rather than scattering COALESCE across every
current and future scanner. Mirrors what later migrations already did for
original_language, show_status, default_metadata_language, and the
*_source_path columns (all NOT NULL DEFAULT '').
- Migration: backfill existing NULLs, then NOT NULL DEFAULT '' on 15 text
columns (sort_title, original_title, content_rating, overview, tagline,
imdb_id/tmdb_id/tvdb_id, poster_path, poster_thumbhash, backdrop_path,
backdrop_thumbhash, logo_path, metadata_s3_path, metadata_etag) and
NOT NULL DEFAULT 0 on year/runtime.
- item_repo: SetLocalPoster and UpdateArtworkIfSourceMatches now store ''
for empty thumbhashes (was NULLIF($,'')) — the only deliberate NULL
writers — matching the upsert path.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(watchsync): sync watchlists with Trakt/Simkl/MDBList
Extend the watch-providers feature to sync a user's watchlist, generalizing
the existing favorites pipeline rather than duplicating it.
What changed
- Generalize the favorites sync into one ListKind-parameterized pipeline
(internal/watchsync/lists.go) driving both favorites and watchlist; the
per-favorites service methods are replaced by kind-generic ones. The shadow
table watch_provider_favorite_items becomes watch_provider_list_items with a
list_kind discriminator.
- Providers: Trakt gains watchlist sync (/sync/watchlist, distinct from
favorites); Simkl gains plan-to-watch sync; MDBList is re-mapped from
favorites to watchlist (its only list is a watchlist) — its capabilities now
report import_favorites=false / import_watchlist=true, and the migration
re-binds existing MDBList connections.
- Auto-remove watched items from the watchlist: a standalone, default-on
profile preference (user_profiles.remove_watched_from_watchlist) removes a
movie when watched and a series once every episode is watched. Implemented as
watchstate.CompletionObserver (internal/watchlist.Maintainer), wired into the
manual mark-watched, playback-stop, and jellycompat mark-played paths.
- Optional MDBList sort-order mirroring: an opt-in, capability-gated toggle
mirrors MDBList's watchlist order into Silo via user_watchlist.sort_index;
ListWatchlist orders by sort_index then added_at, so both /api/v1/watchlist
and the catalog watchlist view inherit it.
- Real-time + scheduled: local add/remove pushes to connected providers
immediately (removals gated by the opt-in removals toggle); the hourly job is
the inbound/import + retry/reconcile path.
- Web: watch-provider settings gain watchlist import/export/removals and
"mirror watchlist order" toggles plus watchlist sync stats.
Why
- The favorites and watchlist pipelines are ~90% identical; generalizing keeps
one code path (per CLAUDE.md's anti-duplication guidance) instead of cloning.
API/compat
- All new fields on ConnectionStatus/Capabilities/ConnectionUpdate/SyncRun and
the web types are additive (Silo v1 additive-only rule). No existing field is
renamed, removed, or retyped.
Risks / follow-up
- MDBList capability flip is intentional and client-visible: silo-android /
silo-apple may need to surface MDBList under the watchlist (not favorites) UI.
- MDBList existing users: their MDBList list previously mirrored Silo favorites
and now mirrors Silo watchlist; the first post-migration sync is a union
(removals default off), so nothing is destructively purged.
- Order mirroring reflects the order MDBList returns from /watchlist/items
(couldn't confirm against their docs — Cloudflare-blocked); if it ever
diverges from the UI sort, a sort param is the small follow-up.
Tests: new maintainer (auto-remove) and watchlist-order unit tests; provider +
service tests updated. go build, go test (affected pkgs), migrate-validate,
verify-local-paths, web prettier/eslint/tsc all pass.
AI-use disclosure: implemented with Claude Code (Claude Opus 4.8).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(watchsync): update list shadow table references
* fix(watchsync): address review — retry/progress + error propagation
Addresses CodeRabbit review on #227:
- maintainer: propagate transient catalog lookup errors instead of silently
treating every items.GetByID failure as "maybe an episode".
- exportList: mark every queued item not confirmed sent (not_found, failed, or
omitted) so the pending loop always advances; the next run's upsert clears the
error and re-attempts, so transient failures still retry.
- removePendingListItems + realtime removal: treat Sent and NotFound as
reconciled; leave true failures pending (no last_error, which would strand
them from the removal query) so the scheduled run retries, using in-memory
dedupe to terminate the loop.
- exportLocalListItems: send the normalized items (with computed
ProviderItemKey), not the original event slice.
- UpdateConnection: clear mirrored watchlist order before persisting the disable
and propagate failures, so a failed clear can't report "disabled" while
sort_index ordering is still active.
- web: include favorite + watchlist removal counts in the exported "sent" total.
- test: align serviceFakeRepo list-state with Postgres (clear last_error on
successful transitions); add maintainer error-propagation test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
toggleLevelProvider merged the toggled level into the levelChains state
object, which starts empty. After the first toggle the state collapsed to
a single level, so once chainDirty flipped true activeLevelChains dropped
every untouched level and the dialog rendered them empty.
Seed from the full resolved activeLevelChains map and write it back the
same way reorderLevel already does, so disabling a source on one TV level
no longer blanks the others on the first toggle.
The earlier redesign (b9a9613) only patched the per-level source fallback,
which kept the toggled level's items but did not preserve the rest of the
map; builds containing it still reproduced the issue.
Part of #213
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Resolves 8 open Dependabot alerts in web/pnpm-lock.yaml (all dev-only
transitive dependencies) by pinning patched versions through the existing
pnpm.overrides mechanism:
- undici 7.24.7 -> 7.28.0 (via jsdom; alerts #43-#49, incl. 2 high:
TLS cert validation bypass & cross-origin routing via SOCKS5 ProxyAgent)
- js-yaml 4.1.1 -> 4.2.0 (via @eslint/eslintrc; alert #39, quadratic DoS)
- @babel/core 7.29.0 -> 7.29.6 (via @vitejs/plugin-react; alert #38,
arbitrary file read via sourceMappingURL)
No vulnerable versions remain in the lockfile. Web build and lint pass;
the only failing tests (ServerStorageStep ResizeObserver) are pre-existing
on main and unrelated to these bumps.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(collections): sync matches across all bound libraries
Collection sync resolved item membership against only the legacy
library_collections.library_id, so a collection bound to multiple
libraries (N:N library_collection_libraries) had its items collapsed to a
single library on every sync.
Switch the MDBList/TMDB/Trakt sync matchers to the existing
LibraryItemRepository.GetItemsInFolders over collection.LibraryIDs.
GetByID already populates LibraryIDs from the N:N table (falling back to
[LibraryID]), so single-library collections are unaffected. No schema
change; listing and the catalog open path were already N:N-aware.
Unblocks merging duplicated per-library collections into single
multi-library collections.
* fix(collections): dedupe matched items by media_item_id before ReplaceItems
A collection sync resolves source entries (external IDs) to library
content_ids and appends one item per matched entry without deduping by
media_item_id. When two different source entries resolve to the same
library item, the same content_id is appended twice; ReplaceItems then
hits the library_collection_items composite PK (collection_id,
media_item_id) with a duplicate-key violation, failing the whole sync
with an HTTP 500 and zero items.
The multi-library union added in the previous commit (GetItemsInFolders
over collection.LibraryIDs) widens the matched set and so raises the
collision probability, making this latent bug more likely to fire.
Dedupe at the single choke point in ReplaceItems (first occurrence wins,
preserving rank order) so all five sync paths are covered at once, and
renumber position over the surviving rows so it stays dense.
Also log the underlying error in HandleSyncAdminCollection, which
previously mapped every non-sentinel sync error to a generic 500 with
no log line, making this failure invisible on the server.
* fix(jellycompat): authenticate Android TV token-less direct play
Stock Jellyfin Android TV ignores the api_key-bearing DirectStreamUrl
returned from PlaybackInfo and builds its own direct-play URL with no
auth header, no api_key/ApiKey query param, and no PlaySessionId. The
media request arrives via the player's HTTP stack (okhttp) with
auth_kind=none, so PlaybackSessionAuth 401s it — the client retries,
falls back to a transcode that stalls, and surfaces "player error".
Add a third fallback in PlaybackSessionAuth, scoped strictly to the
direct-play video stream routes (/Videos/{id}/stream[.{container}]) via
the chi route pattern so /Items/{id}/Download stays protected: anchor
auth on the PlaybackSession negotiated during the preceding (already
authenticated) PlaybackInfo, looked up by mediaSourceId when present
(else the route item id), and resolve its CompatToken.
Covered by tests: token-less direct play with a matching session
succeeds, no matching session 401s, and Download is not loosened.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(jellycompat): require session item match + expand direct-play tests
Address CodeRabbit review on PR #200:
- Require the matched PlaybackSession's RouteItemID to equal the requested
route item before authorizing, so a mediaSourceId cannot authorize a
stream for a different item.
- Seed the compat session in the 401 tests so they fail on route/session
scoping rather than a missing session.
- Table-drive the positive test across /Videos/{id}/stream and
/Videos/{id}/stream.{container}, plus the route-item lookup branch when
mediaSourceId is absent; add a cross-item denial test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(jellycompat): honor client subtitle selection in PlaybackInfo
Jellyfin clients send the chosen subtitle as SubtitleStreamIndex on
PlaybackInfo, but the compat layer only ever advertised the media's
embedded default subtitle (or none). Picking a subtitle before playback
had no effect and the video started without it — and an explicit
"subtitles off" was ignored when the media had a default subtitle.
Mirror the existing audio-selection plumbing for subtitles:
- parse SubtitleStreamIndex from the PlaybackInfo body and query
- persist it as SelectedSubtitleStreamIndex on the play-session source
- advertise it via DefaultSubtitleStreamIndex and flip IsDefault onto the
chosen stream (embedded, external, or Silo-downloaded)
- honor a negative index as an explicit "subtitles off"
The selection is validated against streamable tracks (bitmap subs that
require burn-in are excluded, matching delivery) and the downloaded
subtitle range, falling back to the media default when invalid.
Out of scope (follow-ups): bitmap-subtitle burn-in transcode wiring and
mid-playback subtitle changes on progress reports.
Part of #217
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(jellycompat): handle downloaded-subtitle lookup failure in selection
Don't let a transient ListDownloadedSubtitles error masquerade as "no
downloaded subtitles", which silently downgraded a valid subtitle
selection to the media default. The lookup error is now logged, and
resolution honors a requested index it cannot validate (it may be a
downloaded subtitle) instead of discarding the user's choice, while
embedded/external selections continue to resolve normally.
Addresses CodeRabbit review feedback on #222.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
catalogSearchMeilisearchSchemaVersion hashed the embedder, canonical dimensions, and index types but not SemanticEnabled. Because attachDocumentVectors omits _vectors when semantic is off, toggling semantic on without a full rebuild left previously indexed (and incrementally synced) documents without vectors while the Postgres coverage gate still reported the type "ready" — silently degrading hybrid ranking until a manual rebuild.
Folding SemanticEnabled into the schema version makes a toggle diverge from the stored version, so the provider falls back to keyword, sync skips, the admin status surfaces the schema mismatch, and a rebuild is required to restore hybrid — consistent with how embedder and index-type changes already behave.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extract listPodcastShowAudioFiles and record each show's audio paths in
seenPaths during ScanPodcastFolder so reconciliation does not treat
still-present episodes as removed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Completes the search-provider-interface wiring that the catalog hardening
commits already call into:
- Skip the transactional search-index-event write path when Meilisearch is
not the active provider (ItemRepository.WithActiveSearchProvider /
SearchIndexEventRepository.disabledByActiveProvider).
- Dead-letter catalog_search_index_events after 10 attempts instead of
retrying forever.
- Track the rebuild high-water mark (MaxEventID / MarkProcessedThrough) and
persist last_processed_event_id in UpdateStateAfterRebuild so a rebuild
reconciles events enqueued during the rebuild.
- Validate (read-only) the embedding lock when embedding a search query
instead of establishing/mutating it.
- Surface total_exact on the legacy /items browse response.
- Wire the active catalog search provider into the scanner and item repo at
startup.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Render the new `semantic` block from the admin catalog-search status
endpoint in the Search Settings page: semantic readiness (with
disabled reason), overall vector coverage %, coverage-updated
timestamp, embedder capability, and a compact per-type coverage list.
Reuses the existing StatusRow/FieldGroup/Badge patterns and adds a
formatPercent helper; keeps the existing Vectorized Documents row.
The TS `semantic?` field is optional and the whole block is guarded so
a new frontend talking to an older backend during rollout degrades
gracefully.
Part of the hybrid-semantic-search hardening work (Task 8).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
EmbedAll previously ran the full text-staleness CTE on every page, paying
five item_people LATERAL joins per eligible row just to detect whether an
item's canonical text had drifted - even when the real work was embedding
brand-new (missing) items during active backfill.
Restructure EmbedAll into two passes:
- Pass 1 (cheap): drain missing/model-stale items via the repurposed
ItemsNeedingEmbedding query (single LEFT JOIN, no LATERAL), paged by a
content_id cursor so a failed/skipped item is retried next run instead of
stalling the page.
- Pass 2 (expensive): only once Pass 1 fully drains, run one bounded
ListEmbeddingTextCandidates scan (LIMIT embeddingTextStaleQuotaPerRun=200)
to re-embed text-drifted items. Re-embedding refreshes canonical_text, so
handled rows drop out next run - no Pass 2 cursor needed.
Coverage-first tradeoff (documented in EmbedAll): under steady state Pass 1
drains every run so text-stale items stay fresh; only under pathological
continuous heavy ingest does Pass 2 get skipped, deliberately prioritizing
covering new items over re-embedding changed ones.
Supporting changes:
- ItemsNeedingEmbedding gains an afterID cursor + ORDER BY; SQL extracted to
buildItemsNeedingEmbeddingSQL for a cheap-shape unit test (no item_people /
LATERAL).
- Add an `embedder` interface seam (Engine.embClient) so EmbedAll is testable
with a fake; *embeddings.Client still satisfies it.
- Extract embedBatch to DRY both passes, preserving quota/billing early-return,
single-item fallback, ensureEmbeddingLock-before-upsert, and skip-on-store-error.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a rate-limited Meilisearch embedder capability check and surface a
semantic-coverage block in the admin search status.
Capability validation confirms the active index declares the configured
embedder with source="userProvided" and dimensions matching the canonical
embedding dimension, then runs a unit-vector hybrid probe to confirm the
index accepts the request shape. The whole path (GetSettings + evaluate +
probe) is cached for 5m and NEVER trips the circuit, marks a fallback, or
modifies unhealthyUntil, so a misconfigured embedder cannot take keyword
search down. The pure evaluator (evaluateEmbedderSettings) is unit-tested
across missing/wrong-source/wrong-dimensions/ok with distinct reasons.
The admin status gains a Semantic block read entirely from the in-memory
coverage snapshot (Snapshot + CoverageReady) with no new DB query in
Status(); document_count/vector_document_count are unchanged (additive
only). buildSemanticStatus is pure and unit-tested (nil snapshot and a
populated, deterministically-sorted snapshot).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add per-query search observability to /api/v1/catalog. The internal
CatalogSearchResult now carries Mode + SemanticUsed alongside the existing
Provider + FallbackReason; all four are plumbed through CatalogResult (which
previously dropped them in resolveDirectSearchSource) and exposed as an
additive, omitempty search_diagnostics object on the catalog response.
Two correctness points:
- For Meilisearch, Mode/SemanticUsed are derived from the POST-downgrade
request (baseSearchReq.Hybrid != nil) at the end of searchMeilisearch, so a
hybrid->keyword self-downgrade reports mode="keyword", semantic_used=false.
- search_diagnostics is emitted iff a provider search actually ran
(result.Provider != ""), which naturally omits browse, preview,
non-relevance-sort q=, and group=work paths (none set Provider).
API additive-only: existing response fields keep declaration order and stay
byte-stable; only the new omitempty field is added.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire the semantic-coverage building blocks together so the Meilisearch
provider only goes hybrid once the active embedding model covers enough of
the requested item types.
- Add Coverage SemanticCoverageGate to MeilisearchProviderConfig and consult
it in buildMeilisearchSearchRequest after the wordcount check; when
not-ready, record "semantic_not_ready: <reason>" as a diagnostic fallback
and stay keyword-only (shouldUseSemanticSearch's bool contract unchanged).
- Build and own a semanticCoverageTracker in
NewCatalogSearchServiceFromSettings, but only when semantic is enabled and
a real pool is present. Derive the model provider via a panic-safe comma-ok
helper (semanticModelProvider) so a nil or non-implementing vectorizer
degrades to a not-ready gate instead of asserting a nil interface.
- Add StartCoverageRefresh and call it from NewRouter on deps.AppContext so
the refresher runs for the process lifetime.
Coverage == nil preserves today's behavior exactly; existing hybrid and
SkipTotal tests stay green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add the in-memory coverage gate the search hot path will consult to
decide whether semantic results may be served for a set of item types.
The tracker reads an atomic.Pointer snapshot with no DB and no lock on
the read path, refreshes single-flight under a mutex on a 2-minute
ticker, and applies per-type hysteresis (enable 0.90 / disable 0.80,
holding the previous latch inside the band) to avoid flapping. It
collapses to not-ready when the active embedding model changes so a
stale latch cannot leak across models, and fails safe: a not-yet-
computed snapshot reports not-ready (never panics), and a model-lookup
or count-query error retains the last-good snapshot instead of
publishing zeros.
Refresh counts through the existing catalogSemanticCoverageByType via an
injectable fetch seam, so the behavior is unit-testable with canned
counts and a fake model provider (no pgx.Rows faking). Reuses the
Task-1 coverageQuerier and extends catalogTypeCoverage with Ratio/Ready,
filled by the pure computeCoverageSnapshot helper. Adds the
SemanticCoverageGate interface beside the other provider interfaces;
Task 4 wires the gate and starts the refresher.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add recommendations.Engine.ActiveEmbeddingModel, which reads the
embedding lock from server_settings and returns the locked model (or ""
when no lock is established). Define the catalog
CatalogSemanticModelProvider interface so a later task can scope vector
coverage checks to the active model; the recommendations Engine already
satisfies it. No wiring yet (Task 4 injects recEngine as this provider).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Establish how semantic vector "coverage" is counted: current-model
embeddings over embed-eligible items, per media type, from Postgres.
- Move the embed-eligibility predicate to a single source of truth in
embeddingvectors.ItemEligibilityWhereClause; recommendations now
delegates to it (output byte-identical).
- Add catalogSemanticCoverageByType (per-type eligible/vectorized) and
define the coverageQuerier interface in catalog. Both numerator and
denominator apply the eligibility predicate, so vectorized never
exceeds eligible (C1 guarantee) even with a stale embedding on a
now-unmatched item.
- countCatalogSearchVectorDocuments now takes a coverageQuerier and a
model, applies the eligibility + model filter, and is the per-type
numerator summed across types. Update all four callers (pass "").
- Add idx_media_item_embeddings_model (CONCURRENTLY, self-healing
guard) so the model filter does not scan the embeddings table.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
