* perf(catalog): fix audiobook detail N+1 + slow people facets Audiobook detail pages were slow in proportion to track count (up to 433 files/book). Root causes, found by EXPLAIN ANALYZE on the live DB: - effectiveAudioSelection ran 3-4 user-store queries (profile, audio pref, library pref) per file inside buildPlaybackInfo's loop, though the results are invariant across a request. Introduce a request-scoped audioPrefResolver that memoizes the store lookups (library prefs keyed by folder); a 400-file audiobook now issues each query once instead of per file. Selection logic is unchanged (audioPreference returns a copy so the original-language sentinel is still resolved per file). - buildAudiobookExtension ran its four independent related-content queries serially; run them concurrently so latency is the slowest, not the sum. - author/narrator browse facets did a full people-table scan; add a (kind, content_id, person_id) index so the facet resolves from an index-only scan of just that kind's credits (~112ms -> ~49ms on the live library). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(catalog): cache audiobook author/narrator group browse The Authors/Narrators audiobook pages were slow on cold load and slow again after a hard refresh (fast only while the React Query client cache was warm). Root cause (EXPLAIN ANALYZE on live, 31K-audiobook library): the grouped browse query is ~234ms/page, there are ~13K distinct authors, and the client pages through the entire list on every load (sequential 500-row requests). With no server-side cache each of the ~20 pages re-ran the full aggregation (COUNT(*) OVER() forces it), so a cold load was ~20x234ms. The client's 60s staleTime was the only thing making a warm revisit fast; a refresh wiped it. Fix: AudiobookGroupsCache caches the full sorted group list per (library, group_by, sort, viewer) for 60s (matching the client staleTime, so no extra staleness) and serves every page as an in-memory slice — one aggregation per window instead of one per page, and a refresh is a cache hit. Also raise the client page size 500->2000 so fewer sequential round-trips are needed now that a larger page is a cheap slice. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(settings): throttle per-request device last_seen upserts Device-setting reads (HandleGetDeviceSetting, HandleGetEffectiveSettings, HandleGetEffectiveSubtitleAppearance) each registered the request's device — an INSERT ... ON CONFLICT upsert of last_seen_at on a single per-device row. A page that fetches many settings fired hundreds of these concurrently; they serialized on that row's lock (observed 100-237ms each, ~250 per page load in the slow query log), taxing every settings fetch. Throttle device registration to one upsert per (profile, device) per 5 minutes via an in-process TTL cache, marking the device seen before the upsert so a concurrent burst collapses to a single write. last_seen_at stays fresh to within the window. Reads no longer issue a contended write on the hot path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf+fix(audiobooks): probe-repair, resume position, cache storm, groups reveal, hot-row + stats index From the full audiobook code review (EXPLAIN + slow-query trace on live): - #1 (P0, detail-page killer): NeedsCriticalProbeRepair required video codec/ resolution/tracks, which audio-only files never have, so PlaybackProbeEnsurer re-ran ffprobe per file on every detail/watch load (up to N serial spawns for an N-track book) and never converged. Gate video-field checks on the file actually having a video stream. TDD. - #3 (P0): abs session-sync rewound the resume cursor — UpdateProgressPosition did an unconditional SET with no monotonic guard, ignored its error, and no-op'd when no row existed (first-listen resume lost). Now a finish-preserving GREATEST upsert; caller logs failures. - #4 (P0 perf): progress reports fired every ~10s invalidated all of catalogKeys.all → refetched every active browse/detail query incl the 13k audiobook group lists. Scope invalidation to the reported item's detail. - #6 (P1 perf): Authors/Narrators page rendered all ~13k groups + cover images at once (main-thread freeze). Incremental reveal: render a capped window, grow on scroll via IntersectionObserver. - #10: throttle abs TouchToken last_seen upsert (one per token per 5min) — same hot-row contention class as the device fix. - #8: index abs_playback_sessions (user_id, profile_id, started_at) for the listening-stats aggregations. Deferred (need contract/validation): listening-time idempotency (client delta-vs- cumulative), scanner deleted-file reconcile, abs session retention job, abs list- handler batch fetch, scanner-output P2s (need re-backfill). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(audiobooks): batch-fetch abs list/shelf handlers (kill N+1) handleSimilarItems, handleItemsInProgress, and handleGetMyProgress called MediaStore.GetAudiobookByID once per row — up to ~500 single fetches (each a few queries) on app open. Add GetAudiobooksByIDs (one access-scoped fetch + people/series hydrated once for the whole set) and look results up from the returned map, preserving order. Underlying primitives (GetByIDsWithAccess, hydratePeople/Series) were already batch-capable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(audiobooks): reconcile deleted files on scan + prune session history #5: ScanAudiobookFolder only ever upserted — deleted/renamed books leaked media_items/media_files/memberships forever. Mirror the ebook reconcile: collect seenPaths during the walk, MarkMissing files no longer on disk, then reconcileLibraryMemberships. Safety mirrors ebooks/video: an inaccessible root (unmounted source) is skipped entirely, and a walk that saw zero files while the DB has rows only reconciles after operator cleanup confirmation (ebookEmptyCleanupAllowed) — so a flapping mount can't wipe the catalog. Soft mark only; the existing grace-period purge hard-deletes later. Reconcile runs only on a fully-completed (non-cancelled) scan. (#9 coarse case already handled: audiobookFolderShouldSkip skips unchanged folders; per-file reuse deferred.) #8-retention: abs_playback_sessions grew unbounded (one row per play-start, never deleted) and fed every listening-stats scan. Add an hourly sweep in SessionCleaner: close abandoned open sessions (no /close, stopped syncing >24h) and delete closed sessions older than 90 days. Mirrors the recommendation_cache / missing-files prune pattern. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(audiobooks): address max-effort code-review findings From /code-review max on the pre-PR diff: - DATA RACE (P0): SessionCleaner.lastABSSessionPrune is read+written by both the 15s ticker goroutine and the shutdown-path CleanStale call (main.go defers Stop() to after that call). Guard the prune-due gate with a mutex. (CleanStale was stateless before this branch, so concurrent calls were previously safe.) - ScanAudiobookFolder hardcoded fullScan=true into the empty-walk cleanup guard, but it's also called from ScanSubtree (incremental scans). An empty subtree scan would wrongly consume the operator's one-shot empty-cleanup allowance and warn. Thread a real fullScan flag (true from ScanFolder, false from the two subtree call sites), mirroring the ebook path. - Revert UpdateProgressPosition to UPDATE-only (drop the INSERT-on-missing): keep the monotonic GREATEST + finish guard that fixes the resume rewind, but restore the no-op-on-missing contract so a stray sync tick can't resurrect just-cleared progress or create a zero-duration continue-listening row. - Clamp the audiobook-groups handler limit (paging moved into the cache, leaving the old 500/page bound stranded); also gofmt the Scanner struct. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(audiobooks): address review feedback for scanner and stats * fix(audiobooks): address review feedback * fix(audiobooks): retry failed session prune --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Quick <31828688+Quick104@users.noreply.github.com>
181 lines
5.7 KiB
Go
181 lines
5.7 KiB
Go
package catalog
|
|
|
|
import (
|
|
"context"
|
|
"fmt"
|
|
"sync"
|
|
"sync/atomic"
|
|
"testing"
|
|
"time"
|
|
|
|
"github.com/Silo-Server/silo-server/internal/cache"
|
|
)
|
|
|
|
func newTestGroups(n int) []AudiobookGroup {
|
|
groups := make([]AudiobookGroup, n)
|
|
for i := range groups {
|
|
groups[i] = AudiobookGroup{Name: "author-" + string(rune('a'+i)), ItemCount: i + 1}
|
|
}
|
|
return groups
|
|
}
|
|
|
|
// The grouped list is the expensive aggregation; paging through it (the client
|
|
// fetches every page on each load) must compute it once, not once per page.
|
|
func TestAudiobookGroupsCache_ComputesFullListOncePerKey(t *testing.T) {
|
|
full := newTestGroups(5)
|
|
var fetches int
|
|
c := &AudiobookGroupsCache{
|
|
cache: cache.NewTTLCache[*groupsCacheEntry](),
|
|
ttl: time.Minute,
|
|
fetch: func(context.Context, AudiobookGroupsQuery, AccessFilter) ([]AudiobookGroup, int, error) {
|
|
fetches++
|
|
return full, len(full), nil
|
|
},
|
|
}
|
|
defer c.Close()
|
|
|
|
q := AudiobookGroupsQuery{LibraryID: 7, GroupBy: AudiobookGroupByAuthor, Sort: "name"}
|
|
filter := AccessFilter{UserID: 1, ProfileID: "p1"}
|
|
|
|
page1, total, err := c.Page(context.Background(), AudiobookGroupsQuery{LibraryID: q.LibraryID, GroupBy: q.GroupBy, Sort: q.Sort, Limit: 2, Offset: 0}, filter)
|
|
if err != nil {
|
|
t.Fatalf("page1: %v", err)
|
|
}
|
|
page2, _, err := c.Page(context.Background(), AudiobookGroupsQuery{LibraryID: q.LibraryID, GroupBy: q.GroupBy, Sort: q.Sort, Limit: 2, Offset: 2}, filter)
|
|
if err != nil {
|
|
t.Fatalf("page2: %v", err)
|
|
}
|
|
page3, _, err := c.Page(context.Background(), AudiobookGroupsQuery{LibraryID: q.LibraryID, GroupBy: q.GroupBy, Sort: q.Sort, Limit: 2, Offset: 4}, filter)
|
|
if err != nil {
|
|
t.Fatalf("page3: %v", err)
|
|
}
|
|
|
|
if fetches != 1 {
|
|
t.Fatalf("full list computed %d times across 3 pages of one key; want 1", fetches)
|
|
}
|
|
if total != 5 {
|
|
t.Fatalf("total = %d, want 5", total)
|
|
}
|
|
if len(page1) != 2 || len(page2) != 2 || len(page3) != 1 {
|
|
t.Fatalf("page sizes = %d,%d,%d; want 2,2,1", len(page1), len(page2), len(page3))
|
|
}
|
|
if page1[0].Name != "author-a" || page2[0].Name != "author-c" || page3[0].Name != "author-e" {
|
|
t.Fatalf("slice boundaries wrong: %q %q %q", page1[0].Name, page2[0].Name, page3[0].Name)
|
|
}
|
|
}
|
|
|
|
// A different viewer must not be served another profile's cached counts.
|
|
func TestAudiobookGroupsCache_KeyedByViewer(t *testing.T) {
|
|
var fetches int
|
|
c := &AudiobookGroupsCache{
|
|
cache: cache.NewTTLCache[*groupsCacheEntry](),
|
|
ttl: time.Minute,
|
|
fetch: func(context.Context, AudiobookGroupsQuery, AccessFilter) ([]AudiobookGroup, int, error) {
|
|
fetches++
|
|
return newTestGroups(3), 3, nil
|
|
},
|
|
}
|
|
defer c.Close()
|
|
q := AudiobookGroupsQuery{LibraryID: 7, GroupBy: AudiobookGroupByAuthor, Sort: "name", Limit: 10}
|
|
if _, _, err := c.Page(context.Background(), q, AccessFilter{UserID: 1, ProfileID: "p1"}); err != nil {
|
|
t.Fatalf("viewer1: %v", err)
|
|
}
|
|
if _, _, err := c.Page(context.Background(), q, AccessFilter{UserID: 2, ProfileID: "p2"}); err != nil {
|
|
t.Fatalf("viewer2: %v", err)
|
|
}
|
|
if fetches != 2 {
|
|
t.Fatalf("distinct viewers shared a cache entry: fetches=%d, want 2", fetches)
|
|
}
|
|
}
|
|
|
|
func TestAudiobookGroupsCache_DeduplicatesConcurrentMisses(t *testing.T) {
|
|
var fetches atomic.Int32
|
|
started := make(chan struct{})
|
|
release := make(chan struct{})
|
|
var startedOnce sync.Once
|
|
|
|
c := &AudiobookGroupsCache{
|
|
cache: cache.NewTTLCache[*groupsCacheEntry](),
|
|
ttl: time.Minute,
|
|
fetch: func(context.Context, AudiobookGroupsQuery, AccessFilter) ([]AudiobookGroup, int, error) {
|
|
fetches.Add(1)
|
|
startedOnce.Do(func() { close(started) })
|
|
<-release
|
|
return newTestGroups(4), 4, nil
|
|
},
|
|
}
|
|
defer c.Close()
|
|
|
|
q := AudiobookGroupsQuery{LibraryID: 7, GroupBy: AudiobookGroupByAuthor, Sort: "name", Limit: 2}
|
|
filter := AccessFilter{UserID: 1, ProfileID: "p1"}
|
|
const callers = 8
|
|
ready := make(chan struct{}, callers)
|
|
begin := make(chan struct{})
|
|
errs := make(chan error, callers)
|
|
var wg sync.WaitGroup
|
|
wg.Add(callers)
|
|
for range callers {
|
|
go func() {
|
|
defer wg.Done()
|
|
ready <- struct{}{}
|
|
<-begin
|
|
groups, total, err := c.Page(context.Background(), q, filter)
|
|
if err != nil {
|
|
errs <- err
|
|
return
|
|
}
|
|
if total != 4 || len(groups) != 2 {
|
|
errs <- fmt.Errorf("page result = len %d total %d, want len 2 total 4", len(groups), total)
|
|
}
|
|
}()
|
|
}
|
|
for range callers {
|
|
<-ready
|
|
}
|
|
close(begin)
|
|
select {
|
|
case <-started:
|
|
case <-time.After(time.Second):
|
|
t.Fatal("timed out waiting for cache fetch to start")
|
|
}
|
|
time.Sleep(50 * time.Millisecond)
|
|
close(release)
|
|
wg.Wait()
|
|
close(errs)
|
|
for err := range errs {
|
|
if err != nil {
|
|
t.Fatalf("concurrent page call failed: %v", err)
|
|
}
|
|
}
|
|
if got := fetches.Load(); got != 1 {
|
|
t.Fatalf("concurrent miss fetched %d times; want 1", got)
|
|
}
|
|
}
|
|
|
|
func TestAudiobookGroupsCache_KeyIncludesExcludedMediaTypes(t *testing.T) {
|
|
var fetches int
|
|
c := &AudiobookGroupsCache{
|
|
cache: cache.NewTTLCache[*groupsCacheEntry](),
|
|
ttl: time.Minute,
|
|
fetch: func(context.Context, AudiobookGroupsQuery, AccessFilter) ([]AudiobookGroup, int, error) {
|
|
fetches++
|
|
return newTestGroups(3), 3, nil
|
|
},
|
|
}
|
|
defer c.Close()
|
|
|
|
q := AudiobookGroupsQuery{LibraryID: 7, GroupBy: AudiobookGroupByAuthor, Sort: "name", Limit: 10}
|
|
if _, _, err := c.Page(context.Background(), q, AccessFilter{UserID: 1, ProfileID: "p1", ExcludedMediaTypes: []string{"podcast"}}); err != nil {
|
|
t.Fatalf("podcast excluded: %v", err)
|
|
}
|
|
if _, _, err := c.Page(context.Background(), q, AccessFilter{UserID: 1, ProfileID: "p1", ExcludedMediaTypes: []string{"audiobook"}}); err != nil {
|
|
t.Fatalf("audiobook excluded: %v", err)
|
|
}
|
|
if _, _, err := c.Page(context.Background(), q, AccessFilter{UserID: 1, ProfileID: "p1", ExcludedMediaTypes: []string{"audiobook"}}); err != nil {
|
|
t.Fatalf("audiobook excluded cached: %v", err)
|
|
}
|
|
if fetches != 2 {
|
|
t.Fatalf("excluded media type variants shared a cache entry: fetches=%d, want 2", fetches)
|
|
}
|
|
}
|