From c07d40eb46ac7321ff731d9050a5ca8a012c02da Mon Sep 17 00:00:00 2001 From: RXWatcher <14085001+RXWatcher@users.noreply.github.com> Date: Wed, 27 May 2026 13:12:48 +0200 Subject: [PATCH] docs: design for unifying audiobook collections + sections Captures the decision to fold abs_playlists / abs_user_collections / abs_smart_collections into the canonical user_personal_collections with a discriminating collection_type column, lift the smartcoll engine to internal/smartcoll/, and make page_sections recipes media-type-parameterized so audiobooks land in homepage rails. --- ...27-unified-audiobook-collections-design.md | 295 ++++++++++++++++++ 1 file changed, 295 insertions(+) create mode 100644 docs/superpowers/specs/2026-05-27-unified-audiobook-collections-design.md diff --git a/docs/superpowers/specs/2026-05-27-unified-audiobook-collections-design.md b/docs/superpowers/specs/2026-05-27-unified-audiobook-collections-design.md new file mode 100644 index 00000000..a8a8eb75 --- /dev/null +++ b/docs/superpowers/specs/2026-05-27-unified-audiobook-collections-design.md @@ -0,0 +1,295 @@ +# Unified User Collections (Audiobooks + Movies + TV) Design + +**Status:** design — pending implementation plan. + +**Source brainstorm:** in-session, 2026-05-27. Driven by audiobook coverage gaps in `page_sections`, `library_collections`, and `user_personal_collections`, plus a parallel `abs_*` collection/playlist/smart-collection stack that doesn't intersect with the silo-canonical tables. + +**Commands assume the repository root is the cwd.** + +--- + +## 1. Problem + +Audiobooks today live in a parallel collections world from movies and TV: + +| Surface | Audiobook coverage today | +|---|---| +| `page_sections` (homepage rails) | None. Recipes reference only `movie`/`series`. | +| `library_collections` (admin-owned, library-scoped) | Schema supports `MediaAudiobook` kind but nothing surfaces audiobook-typed admin collections in practice. | +| `user_personal_collections` (user-owned) | Zero references to audiobooks anywhere in `internal/usercollections/`. | +| `abs_user_collections` + `abs_collection_items` | Full audiobook-only user-curated collections behind the ABS-compat API. | +| `abs_smart_collections` | Audiobook-only smart collections with a rule DSL (port of continuum-plugin-audiobooks). | +| `abs_playlists` + `abs_playlist_items` | Audiobook + podcast-episode ordered playlists. | + +Net effect: audiobooks don't appear in any of silo's first-party discovery surfaces, the ABS-app's collection/playlist features don't intersect with silo's native ones, and we maintain two parallel storage stacks. + +The data inside the `abs_*` tables today is essentially empty (1 playlist, 0 user collections, 0 smart collections), so unification can land via hard cutover with minimal data migration cost. + +--- + +## 2. Goals + +- One canonical storage for user-owned lists across all media types. +- Audiobook items eligible for `page_sections` recipes, `library_collections`, and `user_personal_collections`. +- ABS-compat endpoints (`/api/collections`, `/api/playlists`, `/api/smart-collections`) continue to work and return identical wire shapes, but their persistence collapses into the canonical store. +- One smart-collection engine evaluated against all media types. +- Podcast-episode-level granularity preserved (ABS playlists can name a specific episode within a library item). + +## 3. Non-goals + +- Admin-curated audiobook collections via the ABS API surface. `library_collections` already supports audiobooks; whether ABS clients should see those admin lists is a separate question deferred for a later design. +- Frontend changes. Audiobook detail/library pages already exist (per `2026-05-24-audiobook-ui-redesign`) and will consume the unified endpoints when they ship. An admin section-builder media-type selector and any UI surfacing of audiobook sections/collections are follow-up plans. +- Cross-user shared collections, public sharing, social features. Out of scope. +- Episode-level granularity for movies/TV. The new `sub_item_id` column is nullable and only populated by podcast playlists. + +## 4. Architecture + +### 4.1 Storage model + +One canonical user-owned-lists table — `user_personal_collections` — extended to discriminate four flavors via `collection_type`: + +| `collection_type` | Semantics | +|---|---| +| `manual` | Existing. Hand-curated list of items. Order optional. | +| `synced` | Existing. External sync (Trakt etc.). Items derived from external source. | +| `playlist` | **New.** Ordered sequence; `position` column on items is meaningful. | +| `smart` | **New.** Rule-based; items materialized at read time from `query_definition` via `internal/smartcoll/`. | + +`user_personal_collection_items` gains one nullable column: + +```sql +ALTER TABLE user_personal_collection_items ADD COLUMN sub_item_id text NOT NULL DEFAULT ''; +``` + +`sub_item_id` is populated only when the entry refers to a sub-item of a library item (today: podcast episode within a podcast library item). All other entries leave it empty. Audiobook entries always use empty `sub_item_id` because an audiobook is a single `media_items` row. + +The CHECK constraint on `collection_type` is widened to admit the new values: + +```sql +ALTER TABLE user_personal_collections + DROP CONSTRAINT IF EXISTS user_personal_collections_type_check; +ALTER TABLE user_personal_collections + ADD CONSTRAINT user_personal_collections_type_check + CHECK (collection_type IN ('manual', 'synced', 'playlist', 'smart')); +``` + +Admin-curated lists (`library_collections`, `library_collection_items`, etc.) are unchanged. They already support audiobooks through the existing `MediaKind` enum. + +### 4.2 Smart-collection engine + +The current audiobook-only smart-collection engine in `internal/audiobooks/smartcoll/` is promoted to `internal/smartcoll/`: + +``` +internal/audiobooks/smartcoll/ → internal/smartcoll/ +``` + +Rationale: smart collections are no longer audiobook-only. The DSL operates on `media_items` regardless of type; audiobook-specific rule kinds (e.g. `narrator`, `series_position`) stay registered but evaluate to no-op predicates on non-audiobook items. + +The existing simple-shape `query_definition` on `user_personal_collections` (mostly `library_ids` filter) is a strict subset of the smartcoll DSL — existing rows continue to evaluate correctly because their library-id filter maps to the new DSL's library-id predicate. + +### 4.3 ABS-compat handler mapping + +The three ABS store adapter files in `internal/audiobooks/` get their bodies rewritten to query the canonical tables. Method signatures stay the same so the ABS HTTP handlers above them don't change. + +| ABS endpoint | Backing query after cutover | +|---|---| +| `GET /api/libraries/{id}/collections` | `SELECT … FROM user_personal_collections WHERE collection_type='manual'` joined to items in this library | +| `POST /api/collections` | INSERT `user_personal_collections` with `collection_type='manual'`, scoping via `query_definition.library_ids` | +| `GET /api/libraries/{id}/playlists` | `SELECT … FROM user_personal_collections WHERE collection_type='playlist'` joined to items in this library | +| `POST /api/playlists` | INSERT `user_personal_collections` with `collection_type='playlist'` | +| `GET /api/libraries/{id}/smart-collections` | `SELECT … FROM user_personal_collections WHERE collection_type='smart'`; items materialized via `internal/smartcoll` | +| `episode_id` in playlist items wire shape | maps to `sub_item_id` column | + +### 4.4 Page sections (homepage rails) + +`page_sections` rows gain a media-types filter. The cleanest carrier is a typed column: + +```sql +ALTER TABLE page_sections + ADD COLUMN media_types text[] NOT NULL DEFAULT ARRAY['movie','series']; +``` + +Default preserves the current "movies and TV" behavior on existing rows so the migration doesn't change any user-visible rails. + +Each recipe declaration in `internal/sections/recipes/` gains a `SupportedMediaTypes` field. Fetchers add `WHERE mi.type = ANY($media_types)` to their queries. Existing recipes set `SupportedMediaTypes = ['movie','series']`; they keep their current behavior. Recipes that work for audiobooks declare `['movie','series','audiobook']` (or `['audiobook']` for audiobook-only ones). + +Two new recipes ship at the same time: + +- **`continue_listening`** — analog of `continue_watching`; audiobook items with non-zero progress and not finished. +- **`by_audiobook_series`** — book series rail, drawing from the `audiobook_series` table; mirrors `by_show`. + +More audiobook-flavored recipes (top narrators, by genre, new from followed authors) are deferred until the baseline two are exercised. + +### 4.5 ABS app contract preservation + +Wire-shape compatibility is part of the contract — the ABS Android/iOS apps cannot break. The store-adapter rewrites must produce byte-for-byte identical JSON for the existing endpoints. Concretely: + +- IDs remain stable for the moved row(s). The migration preserves `abs_playlists.id` as the new `user_personal_collections.id`. +- `episode_id` field is emitted from `sub_item_id` (empty string when null/empty). +- `is_public` field on playlists/smart collections is emitted from `is_shared`. +- `cover_item` on playlists — needs decision (see §6). + +### 4.6 Module structure after the change + +``` +internal/ + smartcoll/ -- NEW (promoted from audiobooks/smartcoll/) + types.go + evaluator.go + rule_registry.go + usercollections/ -- existing; extended for playlist/smart kinds + listing.go -- gains Kind filter + types.go -- adds CollectionKind = playlist|smart + smartfetch.go -- new; materializes smart-collection items via internal/smartcoll + sections/ + recipes/ + *.go -- each gains SupportedMediaTypes + continue_listening.go -- new + by_audiobook_series.go -- new + audiobooks/ + abs_collection_store.go -- rewritten: canonical SQL + abs_playlist_store.go -- rewritten: canonical SQL + abs_smart_collection_store.go -- rewritten: canonical SQL + smartcoll + smartcoll/ -- REMOVED (lifted to internal/smartcoll/) +``` + +--- + +## 5. Migration + +Single pair: `migrations/156_unify_user_collections.up.sql` / `.down.sql`. Hard cutover. + +### 5.1 Up + +```sql +-- 1. Extend canonical items table with sub-item granularity. +ALTER TABLE user_personal_collection_items + ADD COLUMN sub_item_id text NOT NULL DEFAULT ''; + +-- 2. Widen the collection_type enum. +ALTER TABLE user_personal_collections + DROP CONSTRAINT IF EXISTS user_personal_collections_type_check; +ALTER TABLE user_personal_collections + ADD CONSTRAINT user_personal_collections_type_check + CHECK (collection_type IN ('manual', 'synced', 'playlist', 'smart')); + +-- 3. Move the single existing abs_playlists row. +INSERT INTO user_personal_collections + (id, user_id, profile_id, name, description, collection_type, + is_shared, created_at, updated_at, creator_profile_id) +SELECT + id, user_id, COALESCE(profile_id::text, ''), name, description, 'playlist', + is_public, created_at, updated_at, COALESCE(profile_id::text, '') +FROM abs_playlists; + +INSERT INTO user_personal_collection_items + (user_id, collection_id, media_item_id, sub_item_id, position, added_at) +SELECT p.user_id, i.playlist_id, i.library_item_id, i.episode_id, i.position, i.added_at +FROM abs_playlist_items i +JOIN abs_playlists p ON p.id = i.playlist_id; + +-- 4. Drop the abs_* collection tables. +DROP TABLE abs_playlist_items; +DROP TABLE abs_playlists; +DROP TABLE abs_collection_items; +DROP TABLE abs_user_collections; +DROP TABLE abs_smart_collections; + +-- 5. Extend page_sections with media-type filter. +ALTER TABLE page_sections + ADD COLUMN media_types text[] NOT NULL DEFAULT ARRAY['movie','series']; +``` + +### 5.2 Down + +```sql +-- Reverse the page_sections column. +ALTER TABLE page_sections DROP COLUMN media_types; + +-- Recreate the abs_* tables empty by re-applying their CREATE TABLE +-- statements verbatim from the original migration files (no data restored): +-- migrations/149_abs_user_collections.up.sql → abs_user_collections +-- migrations/150_abs_collection_items.up.sql → abs_collection_items +-- migrations/151_abs_playlists.up.sql → abs_playlists +-- migrations/152_abs_playlist_items.up.sql → abs_playlist_items +-- migrations/153_abs_smart_collections.up.sql → abs_smart_collections +-- The implementer copies the CREATE TABLE bodies (and indexes) into this +-- down migration; constraint/index names must match the originals so a +-- subsequent up-down-up cycle is idempotent. + +-- Remove rows we promoted. +DELETE FROM user_personal_collection_items + WHERE collection_id IN ( + SELECT id FROM user_personal_collections WHERE collection_type IN ('playlist','smart') + ); +DELETE FROM user_personal_collections WHERE collection_type IN ('playlist','smart'); + +-- Restore narrow CHECK constraint. +ALTER TABLE user_personal_collections + DROP CONSTRAINT IF EXISTS user_personal_collections_type_check; +ALTER TABLE user_personal_collections + ADD CONSTRAINT user_personal_collections_type_check + CHECK (collection_type IN ('manual', 'synced')); + +-- Drop the sub_item_id column. +ALTER TABLE user_personal_collection_items DROP COLUMN sub_item_id; +``` + +The down migration is symmetrically structured but lossy in reverse: rolling back loses any playlist/smart rows created after the cutover. That's acceptable because the abs_* tables were near-empty before the up. + +### 5.3 Risk + +| Risk | Probability | Mitigation | +|---|---|---| +| Down-migration loses real user data | Low (one row today; grows over time) | Snapshot the table before running down in any prod context | +| ABS app sees wire-shape regressions | Medium | Snapshot tests on JSON output of each ABS endpoint before/after | +| smartcoll DSL gap (silo-native query_definition row that doesn't fit DSL subset) | Low | Pre-migration query enumerates non-conforming rows; spec asserts subset compat | +| `library_collections` UI gains audiobook rows it can't render | Low | UI changes deferred; admin doesn't currently create audiobook-typed library collections | + +--- + +## 6. Open questions + +1. **Playlist `cover_item` field.** `abs_playlists.cover_item` is a FK to `media_items.content_id`. `user_personal_collections.poster_url` is a plain text URL. Two options for the migration: (a) drop `cover_item`, regenerate poster URLs from the first item in the playlist; or (b) add a `cover_content_id` column to `user_personal_collections`. Recommend (a) for simplicity — the existing playlist row has no `cover_item` set. + +2. **`is_pinned` on smart collections.** `abs_smart_collections.is_pinned` has no analog in `user_personal_collections`. Likely needs a new boolean column; defer until smart-collection UI work names a use case. + +3. **`color` on smart collections.** `abs_smart_collections.color` has no analog. Same call as `is_pinned` — defer; the existing 0 rows means no data to preserve. + +4. **Admin-curated audiobook collections in the ABS API.** Out of scope per §3, but flagged for a future design. + +--- + +## 7. Testing + +- **Migration round-trip test.** Run `up.sql` → `down.sql` → `up.sql` on a fresh test DB seeded with one of each `abs_*` row. Assert canonical-table contents are identical before and after. +- **ABS endpoint wire-shape snapshots.** Generate JSON for `GET /api/libraries/{id}/collections|playlists|smart-collections` before the cutover; assert byte-equal after. +- **smartcoll engine portability.** Move the existing `internal/audiobooks/smartcoll/*_test.go` into `internal/smartcoll/` and add three new tests: + - audiobook-specific rule (`narrator` filter) evaluated against an audiobook → expected items. + - audiobook-specific rule evaluated against a movie → empty set (no-op predicate). + - simple library-id filter (silo-native query_definition shape) evaluated under the unified DSL → expected items. +- **Section recipe regression.** Existing recipes with default `media_types=['movie','series']` return identical results to today. New `continue_listening` and `by_audiobook_series` recipes covered with focused unit tests. + +--- + +## 8. Sub-projects + +This design is decomposed for implementation: + +1. **Schema migration + canonical storage extensions** — migration 156, CHECK/column additions, no behavior changes yet. +2. **Smart-collection engine lift** — move `internal/audiobooks/smartcoll/` to `internal/smartcoll/`, update imports, no logic changes. +3. **ABS store adapter rewrites** — collection/playlist/smart-collection stores query canonical tables. +4. **Section recipes for audiobooks** — recipe `SupportedMediaTypes` field, two new recipes, fetcher type filter. +5. **Documentation + follow-up tickets** — open-question call-outs (cover_item, is_pinned, color, admin collections in ABS). + +Each sub-project has a single-MR scope and ships independently of the others. Sub-project 1 lands first; sub-projects 2–4 can land in any order after it. + +--- + +## 9. Source references + +- Existing canonical tables: `internal/usercollections/`, `internal/collections/templates/templates.go` +- Existing smart-collection engine: `internal/audiobooks/smartcoll/` +- Existing section recipes: `internal/sections/recipes/`, `internal/api/handlers/sections_preview.go`, `internal/api/handlers/sections_bulk.go`, `internal/api/handlers/recipes.go` +- Existing ABS store adapters: `internal/audiobooks/abs_collection_store.go`, `internal/audiobooks/abs_playlist_store.go`, `internal/audiobooks/abs_smart_collection_store.go` +- Related shipped specs: `docs/superpowers/specs/2026-05-26-abs-smart-collections-design.md`, `docs/superpowers/specs/2026-05-24-audiobooks-absorption-design.md`