Files
silo-server/internal/jellycompat/userdata_direct.go
T
02e62767a1 feat(watchsync): sync watchlists with Trakt/Simkl/MDBList (#227)
* feat(watchsync): sync watchlists with Trakt/Simkl/MDBList

Extend the watch-providers feature to sync a user's watchlist, generalizing
the existing favorites pipeline rather than duplicating it.

What changed
- Generalize the favorites sync into one ListKind-parameterized pipeline
  (internal/watchsync/lists.go) driving both favorites and watchlist; the
  per-favorites service methods are replaced by kind-generic ones. The shadow
  table watch_provider_favorite_items becomes watch_provider_list_items with a
  list_kind discriminator.
- Providers: Trakt gains watchlist sync (/sync/watchlist, distinct from
  favorites); Simkl gains plan-to-watch sync; MDBList is re-mapped from
  favorites to watchlist (its only list is a watchlist) — its capabilities now
  report import_favorites=false / import_watchlist=true, and the migration
  re-binds existing MDBList connections.
- Auto-remove watched items from the watchlist: a standalone, default-on
  profile preference (user_profiles.remove_watched_from_watchlist) removes a
  movie when watched and a series once every episode is watched. Implemented as
  watchstate.CompletionObserver (internal/watchlist.Maintainer), wired into the
  manual mark-watched, playback-stop, and jellycompat mark-played paths.
- Optional MDBList sort-order mirroring: an opt-in, capability-gated toggle
  mirrors MDBList's watchlist order into Silo via user_watchlist.sort_index;
  ListWatchlist orders by sort_index then added_at, so both /api/v1/watchlist
  and the catalog watchlist view inherit it.
- Real-time + scheduled: local add/remove pushes to connected providers
  immediately (removals gated by the opt-in removals toggle); the hourly job is
  the inbound/import + retry/reconcile path.
- Web: watch-provider settings gain watchlist import/export/removals and
  "mirror watchlist order" toggles plus watchlist sync stats.

Why
- The favorites and watchlist pipelines are ~90% identical; generalizing keeps
  one code path (per CLAUDE.md's anti-duplication guidance) instead of cloning.

API/compat
- All new fields on ConnectionStatus/Capabilities/ConnectionUpdate/SyncRun and
  the web types are additive (Silo v1 additive-only rule). No existing field is
  renamed, removed, or retyped.

Risks / follow-up
- MDBList capability flip is intentional and client-visible: silo-android /
  silo-apple may need to surface MDBList under the watchlist (not favorites) UI.
- MDBList existing users: their MDBList list previously mirrored Silo favorites
  and now mirrors Silo watchlist; the first post-migration sync is a union
  (removals default off), so nothing is destructively purged.
- Order mirroring reflects the order MDBList returns from /watchlist/items
  (couldn't confirm against their docs — Cloudflare-blocked); if it ever
  diverges from the UI sort, a sort param is the small follow-up.

Tests: new maintainer (auto-remove) and watchlist-order unit tests; provider +
service tests updated. go build, go test (affected pkgs), migrate-validate,
verify-local-paths, web prettier/eslint/tsc all pass.

AI-use disclosure: implemented with Claude Code (Claude Opus 4.8).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(watchsync): update list shadow table references

* fix(watchsync): address review — retry/progress + error propagation

Addresses CodeRabbit review on #227:
- maintainer: propagate transient catalog lookup errors instead of silently
  treating every items.GetByID failure as "maybe an episode".
- exportList: mark every queued item not confirmed sent (not_found, failed, or
  omitted) so the pending loop always advances; the next run's upsert clears the
  error and re-attempts, so transient failures still retry.
- removePendingListItems + realtime removal: treat Sent and NotFound as
  reconciled; leave true failures pending (no last_error, which would strand
  them from the removal query) so the scheduled run retries, using in-memory
  dedupe to terminate the loop.
- exportLocalListItems: send the normalized items (with computed
  ProviderItemKey), not the original event slice.
- UpdateConnection: clear mirrored watchlist order before persisting the disable
  and propagate failures, so a failed clear can't report "disabled" while
  sort_index ordering is still active.
- web: include favorite + watchlist removal counts in the exported "sent" total.
- test: align serviceFakeRepo list-state with Postgres (clear last_error on
  successful transitions); add maintainer error-propagation test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 16:05:53 -04:00

327 lines
12 KiB
Go

package jellycompat
import (
"context"
"fmt"
"log/slog"
"time"
"github.com/Silo-Server/silo-server/internal/catalog"
"github.com/Silo-Server/silo-server/internal/userstore"
"github.com/Silo-Server/silo-server/internal/watchstate"
)
// directUserDataService implements UserDataService using the user store directly.
type directUserDataService struct {
storeProvider userstore.UserStoreProvider
itemRepo *catalog.ItemRepository
detailSvc *catalog.DetailService
watchState *watchstate.Service
resumeFilter *catalog.ContinueWatchingProgressFilter
profileStaler profileStaler
profileRefreshRequester profileRefreshRequester
}
func newDirectUserDataService(
storeProvider userstore.UserStoreProvider,
itemRepo *catalog.ItemRepository,
episodeRepo *catalog.EpisodeRepository,
providerIDRepo *catalog.ProviderIDRepository,
detailSvc *catalog.DetailService,
resumeFilter *catalog.ContinueWatchingProgressFilter,
staler profileStaler,
requester profileRefreshRequester,
completionObserver watchstate.CompletionObserver,
) *directUserDataService {
return &directUserDataService{
storeProvider: storeProvider,
itemRepo: itemRepo,
detailSvc: detailSvc,
watchState: watchstate.NewService(storeProvider).
WithStableIdentityResolver(watchstate.NewStableIdentityResolver(itemRepo, episodeRepo, providerIDRepo)).
WithCompletionObserver(completionObserver),
resumeFilter: resumeFilter,
profileStaler: staler,
profileRefreshRequester: requester,
}
}
func (s *directUserDataService) ListFavorites(ctx context.Context, session *Session, limit, offset int) ([]upstreamListItem, error) {
store, err := s.storeProvider.ForUser(ctx, session.StreamAppUserID)
if err != nil {
return nil, fmt.Errorf("open user store: %w", err)
}
// ABS-surface favorites (audiobooks/podcasts) are filtered out below, so
// the limit/offset window must apply to the *filtered* list — a raw
// store-level window would shift or shrink the visible page. Over-fetch
// the raw rows, filter, then window.
scanLimit := min(max((limit+offset)*2, 200), 10000)
favorites, err := store.ListFavorites(ctx, session.ProfileID, scanLimit, 0)
if err != nil {
return nil, fmt.Errorf("list favorites: %w", err)
}
contentIDs := make([]string, 0, len(favorites))
for _, fav := range favorites {
contentIDs = append(contentIDs, fav.MediaItemID)
}
if len(contentIDs) == 0 {
return []upstreamListItem{}, nil
}
items, err := s.itemRepo.GetByIDs(ctx, contentIDs)
if err != nil {
return nil, fmt.Errorf("get favorite items: %w", err)
}
// Build a map for ordering by the original favorites list order
itemMap := make(map[string]*upstreamListItem, len(items))
for _, mi := range items {
// Favorites are shared with the ABS surface; its media types are
// never exposed here (they would 404 on detail/PlaybackInfo).
if isCompatExcludedMediaType(mi.Type) {
continue
}
li := mediaItemToListItem(mi)
itemMap[mi.ContentID] = &li
}
ordered := make([]upstreamListItem, 0, len(contentIDs))
for _, id := range contentIDs {
if li, ok := itemMap[id]; ok {
ordered = append(ordered, *li)
}
}
// Presign artwork only for the page being returned.
result := slicePage(ordered, offset, limit)
for i := range result {
result[i].PosterURL = compatPresignImage(s.detailSvc, ctx, result[i].PosterURL, "poster", compatCardImageSize)
result[i].BackdropURL = compatPresignImage(s.detailSvc, ctx, result[i].BackdropURL, "backdrop", compatCardImageSize)
result[i].LogoURL = compatPresignImage(s.detailSvc, ctx, result[i].LogoURL, "logo", compatCardImageSize)
}
if result == nil {
result = []upstreamListItem{}
}
return result, nil
}
func (s *directUserDataService) ListFavoritesByMediaItems(ctx context.Context, session *Session, mediaItemIDs []string) (map[string]bool, error) {
store, err := s.storeProvider.ForUser(ctx, session.StreamAppUserID)
if err != nil {
return nil, fmt.Errorf("open user store: %w", err)
}
result, err := store.ListFavoritesByMediaItems(ctx, session.ProfileID, mediaItemIDs)
if err != nil {
return nil, fmt.Errorf("list favorites by media items: %w", err)
}
if result == nil {
return map[string]bool{}, nil
}
return result, nil
}
func (s *directUserDataService) IsFavorite(ctx context.Context, session *Session, contentID string) (bool, error) {
store, err := s.storeProvider.ForUser(ctx, session.StreamAppUserID)
if err != nil {
return false, fmt.Errorf("open user store: %w", err)
}
favorite, err := store.IsFavorite(ctx, session.ProfileID, contentID)
if err != nil {
return false, fmt.Errorf("check favorite: %w", err)
}
return favorite, nil
}
func (s *directUserDataService) AddFavorite(ctx context.Context, session *Session, contentID string) error {
store, err := s.storeProvider.ForUser(ctx, session.StreamAppUserID)
if err != nil {
return fmt.Errorf("open user store: %w", err)
}
if err := store.AddFavorite(ctx, session.ProfileID, contentID); err != nil {
return err
}
triggerProfileRefresh(ctx, s.profileStaler, s.profileRefreshRequester, session.StreamAppUserID, session.ProfileID)
return nil
}
func (s *directUserDataService) RemoveFavorite(ctx context.Context, session *Session, contentID string) error {
store, err := s.storeProvider.ForUser(ctx, session.StreamAppUserID)
if err != nil {
return fmt.Errorf("open user store: %w", err)
}
if err := store.RemoveFavorite(ctx, session.ProfileID, contentID); err != nil {
return err
}
triggerProfileRefresh(ctx, s.profileStaler, s.profileRefreshRequester, session.StreamAppUserID, session.ProfileID)
return nil
}
func (s *directUserDataService) ListProgress(ctx context.Context, session *Session, status string, limit, offset int) ([]upstreamProgress, error) {
store, err := s.storeProvider.ForUser(ctx, session.StreamAppUserID)
if err != nil {
return nil, fmt.Errorf("open user store: %w", err)
}
entries, err := store.ListProgress(ctx, session.ProfileID, status, limit, offset)
if err != nil {
return nil, fmt.Errorf("list progress: %w", err)
}
result := make([]upstreamProgress, 0, len(entries))
for _, entry := range entries {
result = append(result, toUpstreamProgress(entry))
}
return result, nil
}
// FilterResumeProgress applies the same hiding rules as the first-party
// Continue Watching fetcher: dismissed entries and episodes superseded by a
// later-completed episode in the same series.
func (s *directUserDataService) FilterResumeProgress(ctx context.Context, session *Session, entries []upstreamProgress) ([]upstreamProgress, error) {
if len(entries) == 0 {
return entries, nil
}
store, err := s.storeProvider.ForUser(ctx, session.StreamAppUserID)
if err != nil {
return nil, fmt.Errorf("open user store: %w", err)
}
progress := make([]userstore.WatchProgress, 0, len(entries))
for _, entry := range entries {
progress = append(progress, fromUpstreamProgress(entry))
}
// Dismissal lookup failures degrade to showing the entries, matching the
// first-party fetcher.
if dismissals, err := store.ListHomeDismissals(ctx, session.ProfileID, userstore.HomeSurfaceContinueWatching); err != nil {
slog.Error("listing continue watching dismissals", "profile_id", session.ProfileID, "error", err)
} else {
progress = catalog.NewHomeDismissalIndex(dismissals).FilterProgress(progress)
}
superseded, err := s.resumeFilter.SupersededEpisodeProgressIDs(ctx, store, session.ProfileID, progress)
if err != nil {
return nil, fmt.Errorf("filter superseded progress: %w", err)
}
progress = catalog.FilterSupersededProgress(progress, superseded)
result := make([]upstreamProgress, 0, len(progress))
for _, entry := range progress {
result = append(result, toUpstreamProgress(entry))
}
return result, nil
}
func (s *directUserDataService) ListProgressByMediaItems(ctx context.Context, session *Session, mediaItemIDs []string) (map[string]*upstreamProgress, error) {
store, err := s.storeProvider.ForUser(ctx, session.StreamAppUserID)
if err != nil {
return nil, fmt.Errorf("open user store: %w", err)
}
mediaItemIDs = normalizeContentIDs(mediaItemIDs)
progressMap, err := userstore.ListProgressWithCompletedHistory(ctx, store, session.ProfileID, mediaItemIDs)
if err != nil {
return nil, fmt.Errorf("list progress by media items: %w", err)
}
result := make(map[string]*upstreamProgress, len(progressMap))
for contentID, progress := range progressMap {
entry := toUpstreamProgress(progress)
result[contentID] = &entry
}
return result, nil
}
func (s *directUserDataService) GetProgress(ctx context.Context, session *Session, contentID string) (*upstreamProgress, error) {
store, err := s.storeProvider.ForUser(ctx, session.StreamAppUserID)
if err != nil {
return nil, fmt.Errorf("open user store: %w", err)
}
progress, err := userstore.GetProgressWithCompletedHistory(ctx, store, session.ProfileID, contentID)
if err != nil {
return nil, fmt.Errorf("get progress: %w", err)
}
if progress == nil {
return nil, nil
}
entry := toUpstreamProgress(*progress)
return &entry, nil
}
func (s *directUserDataService) MarkPlayed(ctx context.Context, session *Session, contentID string) error {
if s.watchState == nil {
return fmt.Errorf("watch state service is not configured")
}
if err := s.watchState.RecordJellycompatMarkPlayed(ctx, session.StreamAppUserID, session.ProfileID, contentID, time.Now().UTC()); err != nil {
return err
}
triggerProfileRefresh(ctx, s.profileStaler, s.profileRefreshRequester, session.StreamAppUserID, session.ProfileID)
return nil
}
func (s *directUserDataService) MarkPlayedBatch(ctx context.Context, session *Session, contentIDs []string) error {
if s.watchState == nil {
return fmt.Errorf("watch state service is not configured")
}
if len(contentIDs) == 0 {
return nil
}
if err := s.watchState.RecordJellycompatMarkPlayedBatch(ctx, session.StreamAppUserID, session.ProfileID, contentIDs, time.Now().UTC()); err != nil {
return err
}
triggerProfileRefresh(ctx, s.profileStaler, s.profileRefreshRequester, session.StreamAppUserID, session.ProfileID)
return nil
}
func (s *directUserDataService) MarkUnplayed(ctx context.Context, session *Session, contentID string) error {
if s.watchState == nil {
return fmt.Errorf("watch state service is not configured")
}
if err := s.watchState.RecordJellycompatMarkUnplayed(ctx, session.StreamAppUserID, session.ProfileID, contentID); err != nil {
return err
}
triggerProfileRefresh(ctx, s.profileStaler, s.profileRefreshRequester, session.StreamAppUserID, session.ProfileID)
return nil
}
func (s *directUserDataService) MarkUnplayedBatch(ctx context.Context, session *Session, contentIDs []string) error {
if s.watchState == nil {
return fmt.Errorf("watch state service is not configured")
}
if len(contentIDs) == 0 {
return nil
}
if err := s.watchState.RecordJellycompatMarkUnplayedBatch(ctx, session.StreamAppUserID, session.ProfileID, contentIDs); err != nil {
return err
}
triggerProfileRefresh(ctx, s.profileStaler, s.profileRefreshRequester, session.StreamAppUserID, session.ProfileID)
return nil
}
func toUpstreamProgress(entry userstore.WatchProgress) upstreamProgress {
return upstreamProgress{
MediaItemID: entry.MediaItemID,
PositionSeconds: entry.PositionSeconds,
DurationSeconds: entry.DurationSeconds,
Completed: entry.Completed,
UpdatedAt: entry.UpdatedAt,
}
}
func fromUpstreamProgress(entry upstreamProgress) userstore.WatchProgress {
return userstore.WatchProgress{
MediaItemID: entry.MediaItemID,
PositionSeconds: entry.PositionSeconds,
DurationSeconds: entry.DurationSeconds,
Completed: entry.Completed,
UpdatedAt: entry.UpdatedAt,
}
}