Files
silo-server/internal/scanner/scanner.go
T
c4cbcddeae feat(manga): manga library type — series grouping, reading loop, AniList/MangaDex metadata + status badge (#138)
* docs: design spec for manga library type (host sub-project)

Forks the ebooks library type into a 'manga' type: series detected from the
folder tree as a first-class type='manga' item, .cbz/.cbr chapters stay
readable ebook items linked via a new manga_chapters table, browse shows series
cards, enrichment targets the series item at content level 'manga'. Hands off to
a follow-on plugin spec for the manga metadata source.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: implementation plan for manga library type (host)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(scanner): manga filename index/volume parser

* feat(scanner): manga series-name-from-folder detection

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* docs(plan): align manga DB/scanner tasks to scanner pure-planner pattern (no test-DB)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(scanner): manga parser corpus regression

Add TestParseMangaIndexCorpus — 36 real-world scanlation filenames
covering bare chapter, decimal chapter, v/vol-prefix volume, and
c/ch-prefix chapter patterns; asserts <5% miss rate.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(db): manga_chapters link table

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(scanner): manga_chapters repository + pure chapter-write mapping

Adds mangaChapterWrite (pure, unit-tested), upsertMangaChapter, and
listMangaChapters following the ebook/audiobook thin-SQL pattern.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(scanner): recognize manga library type

Add isMangaLibraryType helper (unexported, matching the style of
isEbookLibraryType / isAudiobookLibraryType) with a corresponding
TestIsMangaLibraryType unit test.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(api): manga library content level

Map library type "manga" to content level ["manga"] in
metadataContentLevelsForLibraryType so that seedDefaultChain seeds a
manga-level metadata provider chain when a manga library is created.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(scanner): route manga libraries to a manga scan path

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(scanner): group manga chapters under a manga series item

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(scanner): give manga series item a library membership so it browses

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(catalog): browse manga libraries as series

Accept "manga" as a valid media_scope so a manga library browses only its
type='manga' series items; the per-chapter type='ebook' items are naturally
excluded because MediaScopeItemTypes("manga") expands to {"manga"}. Add the
manga default library sections (scoped to media_scope='manga') so the library
feed shows series cards. Refresh the two media_scope validation error messages.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(catalog): manga series detail lists chapters

For a type='manga' item, attach its chapters to the detail response via a new
MangaDetailExtension. fetchMangaChapters joins manga_chapters to media_items on
the chapter content ID, scopes to the series, and orders by chapter_index
(NULLS LAST) then sort_title — matching the scanner's chapter ordering.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(web): manga detail types + library browse scoping

Add MangaChapter/MangaDetailExtension TS types mirroring the host
catalog structs, wire manga? onto ItemDetail, and admit "manga" as a
QueryDefinition.media_scope. Scope manga libraries to media_scope=manga
in browse (host expands it to type=manga series items) while reusing the
ebook sort universe via getLibrarySortRelevanceScope. Add isMangaLibraryType.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(web): manga series detail with volume-grouped chapter list

Add MangaContent detail view: a DetailHero series header plus a chapter
list grouped by volume. groupMangaChapters (pure, unit-tested) buckets
chapters by their volume token, orders chapters within a group by
chapter_index (nulls last) and orders groups by their minimum index;
loose (volume-less) chapters collapse into a trailing "Chapters" group.
Each chapter links to the existing ebook reader by content_id alone
(file_id is optional — the reader resolves the file server-side), reusing
buildMediaPlayHref. Admit "manga" into ItemDetail.type and wire the
detail switch. Continue-reading is deferred (needs per-chapter progress
fan-out / a last-read timestamp not in the current payload).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(web): handle manga in playable-type + collection filter-scope unions

Adding "manga" to the shared ItemDetail["type"] and
QueryDefinition["media_scope"] unions leaked into consumers with narrower
local types, breaking the production tsc build. Fixes:

- mediaNavigation: admit "manga" into PlayableMediaType. Manga series are
  not directly playable (you open the detail page and read a chapter,
  itself an ebook item), so buildMediaPlayHref falls through to the item
  href for them, like series/season.
- FilterRuleEditor: add "manga" to FilterRuleMediaScope and relabel
  "watched" -> "Read" for manga as well as ebook (manga is read).
- CollectionGuidedRulesEditor: add "manga" to GuidedFormState.mediaScope,
  a "Manga" media-type option, ebook-like "Read Status" labels, and map
  manga -> ebook sort-relevance scope (manga has no dedicated sort scope).
- CatalogFilterBar (cascading leak surfaced after the above): add a
  "Manga" scope option and map manga -> ebook sort-relevance scope in both
  scope handlers.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(web): offer manga as a library type in the create dialog

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(scanner): strip scene-release junk from manga series names

Add cleanMangaSeriesName which repeatedly strips trailing parenthetical
groups (year, year-range, Digital, release-group tags) then trims any
dangling dash, so folder names like "404 Demons (Digital) (Oak)" resolve
to "404 Demons". Wire it into mangaSeriesFromPath so both the series
title and the mangaSeriesGroupKey identity key use the cleaned value.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(web): flat volume/chapter manga list; nest only multi-chapter volumes

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(scanner): parse manga index after stripping series-name prefix

Numbers inside a series title (e.g. "404 Demons", "365 Days to the
Wedding") were wrongly grabbed as the chapter number because
parseMangaIndex matched the first bare number in the full filename.
mangaIndexForFile now strips the series-name prefix before delegating
to parseMangaIndex, so only the number that follows the title is used.
reconcileMangaFile in manga_scan.go is updated to call mangaIndexForFile
instead of parseMangaIndex directly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(scanner): stop missing-file reconcile from deleting manga series items

Manga series items are file-less virtual parents; the shared
ReconcileFolderMembership swept them every scan because they have no
media_file. Exclude type='manga' from file-presence membership reconciliation,
and add a manga-scan step that deletes only series with zero remaining chapters.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ebooks): exclude manga chapters from individual ebook enrichment

Manga chapters are type='ebook' parts of a series; the ebook enrichment sweep
was searching each one against book sources (Gutenberg/Anna's/etc.) and failing
in a pointless storm. Exclude items with a manga_chapters link; series-level
enrichment is handled separately.

* docs: design spec for manga metadata plugin + series enrichment (sub-project 2)

New silo-plugin-manga-metadata (AniList, high-confidence matching) + a host
MangaEnricher for type='manga' series; default-enabled metadata source for manga
libraries.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: implementation plan for manga metadata plugin + series enrichment

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(db): manga_enrichment_state table

Mirrors ebook_enrichment_state: dedicated failure counter for the manga
enrichment sweep so it does not contend with media_items.refresh_failures.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(manga): series enricher (claims type='manga', resolves manga chain)

* feat(manga): sync_manga_metadata task + enricher wiring

* feat(catalog): expose manga chapter/volume counts in browse

Add manga_chapter_count and manga_volume_count to browse cards so the
frontend can render a Vols N / Ch N chip on manga series. The counts come
from two index-backed correlated subqueries over manga_chapters in the
browse SELECT (mangaCountColumns), scanned positionally before added_at and
nilled out for non-manga rows. Threaded through models.MediaItem and exposed
on the itemListResponse JSON card.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(sections): scope manga home recent sections to type=manga series

A manga library mixes type='manga' series with type='ebook' chapters, so
the auto-generated home 'Recently Added/Released in <Library>' rows surfaced
the junk chapter filenames. Add GeneratedHomeLibraryRecentConfigScoped which
emits the modern QueryDefinition shape (library_ids + media_scope) so a manga
library's generated home rows filter to type='manga' only. Other library
types are unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(catalog): exclude manga chapters from browse/section/search surfaces

Manga CHAPTER items (type='ebook' rows linked into a type='manga' series
via manga_chapters) were leaking into catalog browse, section resolution,
and search as standalone items showing junk filenames. They are internal
sub-units of the series and only the series should appear.

There is no single shared item-listing chokepoint: browse, the query/preview
executor, and search each build their own WHERE. Add a shared, index-backed
anti-join predicate (manga_chapters.chapter_content_id is the PK) via
mangaChapterExclusionWhere and wire it into all three builders. By-id fetch
paths that legitimately resolve chapters (ebook reader, continue-reading,
series detail chapter list) use separate queries and are unaffected.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(scanner): use #NN as the manga volume for Vol.YYYY #NN releases

mangaVolYearIssue early-return was returning the year token (e.g. "Vol.2003")
as the volume label, which the frontend couldn't prettify to "Volume N".
Now returns "v<issue>" (e.g. "v04") so the existing frontend regex ^v?(\d+)$
renders it as "Volume 4" correctly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(web): manga count chip on posters

Add an optional manga_chapter_count / manga_volume_count to the browse
item type and render a top-right "Vols N" / "Ch N" chip on ItemCard,
strictly gated on type==='manga'. The label prefers "Vols" when the
volume count dominates, "Ch" otherwise; the chip is hidden when the
chapter count is missing or non-positive. No other card type renders it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(web): manga reader back returns to series (no loop)

The ebook reader's back action defaulted to the chapter's own item
detail (/item/<chapter>), whose back returned to the reader — an
infinite loop for manga chapters. The reader now honors an explicit
backTo search param when present, navigating there instead. Absent for
normal ebooks, so their back behavior is unchanged. Only manga chapter
rows pass backTo, keeping the fix manga-only.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(web): manga chapter row actions (read/mark-read/download)

Each manga chapter/volume row now offers Read (the existing reader link,
now carrying a backTo to the series), Mark-read (the shared watched-state
mutation per chapter content_id), and Download (lazily fetches the
chapter's file versions on demand and opens the shared
DownloadVersionPicker, gated on user.download_allowed). The
volume-unit / loose-chapter / section structure from buildMangaList is
unchanged. Scoped to MangaContent only; EbookContent is untouched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(web): validate reader backTo param is a safe in-app relative path

Prevents open-redirect / javascript:-URI XSS from a crafted ?backTo= URL.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(catalog): include per-chapter read state in manga detail

Manga chapters are ebook items, so a chapter is "read" when the viewer's
ebook_reader_progress row crosses the finished threshold. fetchMangaChapters
now LEFT JOINs that table scoped to the AccessFilter's user_id/profile_id and
exposes a per-chapter Read bool on MangaChapter, threaded through
buildMangaExtension. The detail payload previously carried no read state, so
the row toggle always started unread.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(web): manga rows reflect read state on load

MangaChapter now carries an optional read flag from the detail payload, and
MangaRow seeds its mark-read toggle from chapter.read instead of always
starting unread. The optimistic toggle + shared watched mutation are
unchanged; only the initial value is seeded.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(sections): exclude manga chapters from recently-added/released/random + other library-listing sections

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(sections): manga recently-added/released cards show the latest volume's cover

* fix(manga): keep enrichment honest about no-match vs enriched, batch 50->200

- sweep stats now separate enriched / no_match / failed: a stamped no-match
  was counted (and logged) as an enrichment, which masked a collapse of the
  real match rate during the backfill
- batch size 50 -> 200 (SILO_MANGA_ENRICH_BATCH overrides): with the plugin
  serving GetMetadata from its search cache an item costs one rate-limited
  AniList request, so a sweep still fits the 5-minute task interval

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(manga): size enrich batch to the 5-minute interval at AniList's real budget

140 items x ~2.1s/request fits the interval; an overlong sweep makes the task
manager drop the next trigger and the effective rate falls below the AniList
budget.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(catalog): manga count chip data missing from library browse

manga_chapter_count/manga_volume_count were only added to BrowseRepository,
but /library/{id}?tab=library flows through previewQuerySource ->
QueryExecutor.PreviewPage, which selects qualifiedListItemColumns and scans
with scanItems - so manga cards never carried the counts and the Vols/Ch
poster chip stayed hidden.

Append mangaCountColumns to the preview-page SELECT and scan them via a new
scanItemsWithMangaCounts (nil for non-manga rows, mirroring scanBrowseItems).
Extract listItemScanDests so the three scan variants share one destination
list instead of duplicating the 48-column scan.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(web): manga chip reads 'X Volumes · X Chapters', menu verbs say Read

- chip: show distinct-volume and loose-chapter counts side by side instead
  of the single 'Vols N'/'Ch N' heuristic; mangaCountColumns now counts
  DISTINCT volume tokens (rows sharing a volume are one volume) and only
  un-volumed rows as chapters
- watched-state labels: type='manga' fell through to the video default, so
  the card dot menu and detail page said 'Mark Watched' - manga now uses
  the ebook reading verbs (Mark Read / Mark Unread, 'Marked as read' toast)
- format MangaContent.test.tsx (pre-existing prettier miss)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(manga): backdrop enrichment - banner hero art + backdrop-only backfill

- cache remote backdrops like posters (cacheRemoteImages generalizes the
  poster-only path; failures keep the provider URL, which still renders)
- claim arm for enriched items missing a backdrop: fetched by stored
  provider ID (search skipped - no rate spend, no re-match risk) and only
  the backdrop is written; stamping after the attempt keeps banner-less
  series from being re-claimed every sweep
- backfill = one-time SQL clearing last_refreshed for poster-set/
  backdrop-empty manga

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(manga): reading-loop UX - continue CTA, next chapter, series-aware cards, file details

Fixes the four high-priority findings from the manga UX review plus a
file-inspector request:

- H1: series hero gets a Continue / Start Reading / Read Again CTA
  targeting the first unread chapter (firstUnreadChapter over the ordered
  list), plus an overflow menu (View Details, admin Refresh Metadata)
- H2: the reader resolves its owning manga series (chapter detail now
  carries series_id/series_title) and offers next-chapter navigation: a
  header next button and an end-of-book floating CTA at >=99.5% progress;
  back defaults to the series even without a backTo param
- H3: chapter rows show a persistent read check + muted title, and the
  mark-read mutation carries series_id so the series detail cache
  invalidates (read states no longer revert on revisit)
- H4: continue-reading cards for manga chapters present the series:
  sections payload resolves chapter->series linkage, the card heading/image
  link to the series, and meta lines launch the reader
- View Details: manga series menus (card dot menu + detail overflow) open
  a file inspector showing folder paths and per-chapter file names/sizes
  via GET /catalog/items/{id}/manga-files; paths are stripped for viewers
  without file-path visibility (item-versions policy)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(manga): UX mediums - richer detail page, smarter list, manga sort scope

Second batch from the manga UX review (M1-M7):

- M1: multi-chapter volume sections are collapsible (fully read sections
  start collapsed) with sticky headers, and long series get a 'Jump to
  <next unread>' anchor above the list
- M2: the series hero shows the author line (HeroCrewLine learns Author
  credits with person links; DetailHero now renders crewLine and genre
  chips independently) and Volumes/Chapters badges
- M3: browse-card count chip abbreviates to '12 Vol - 3 Ch' so it fits
  narrow cards without occluding covers
- M4: manga gets its own sort scope: Duration/Bitrate (meaningless for
  file-less series rows) disappear, reading labels (Date Read / Reads)
  apply, Author stays
- M5: global search labels manga results 'Manga' instead of the raw type
- M6: chapters carry the viewer's reading fraction; part-read rows show an
  inline progress bar + percent
- M7: chapter rows show the extracted cover thumbnail (presigned
  poster_url on the chapters payload) instead of a generic icon

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(manga): UX lows - volume token dedupe, comic reader chrome, empty-state hint

- buildMangaList buckets volumes by canonical numeric token so mixed
  release naming (v01 + 1) yields one Volume 1 instead of duplicates
- cbz/cbr readers start with the side panel closed and hide prose-only
  chrome (reading ruler, TTS, typography/font controls, hyphenation,
  writing mode) while keeping comic-relevant settings (theme, brightness,
  margin, right-to-left, spread, flow)
- manga empty state mentions chapters appear after the library scan

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(manga): publication status badge via new SDK status field

- vendor the unpublished plugin SDK (adds MetadataItem.status) under
  internal/compat/ with a relative go.mod replace, following the
  zishang520-webtransport-go convention; swap to the published module
  before the upstream PR
- map plugin status into MetadataResult.ShowStatus, persist it during
  manga enrichment, and show it as the hero status badge (show_status was
  already on the detail payload and MetadataBadges)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(manga): generalize backdrop pass to secondary fields (backdrop + status)

The backdrop-only claim arm becomes a secondary-fields pass: enriched items
missing a backdrop and/or publication status are claimed, fetched by stored
provider ID, and only the missing secondary fields are written. Lets the
new status field backfill across the already-enriched library instead of
applying only to future enrichments.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(metadata): merge ShowStatus through MergeMetadata/MergeGlobalMetadata

The new MetadataResult.ShowStatus never reached the accumulated result the
manga enricher persists from - the field-by-field merges didn't know it, so
the status backfill pass obtained nothing. Regression-tested on both paths.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(manga): keep scanner identity IDs out of the metadata flow

filterMangaProviderIDs passed the scanner's manga_series identity row
through, so the search-skip-when-already-matched guard saw provider IDs on
every item and never searched: unmatched items went straight to a by-ID
fetch with no usable ID and were stamped as terminal no-match without a
single provider request (and the MangaDex fallback was never consulted).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: gitignore docker-compose.override.yml (local deployment override)

The override unpublishes the bundled redis/postgres host ports
(ports: !override []). It is a per-deployment, local-only file: ignoring it
keeps a rebase from main and git clean -fd from disturbing it, and keeps it
out of any PR. Its accidental absence once exposed Redis to the internet.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(manga): code-review fixes — no-match guard, sort comparator, volume-count consistency

- enrichWithProviders: set accumulator.HasMetadata after a provider result
  merges (MergeMetadata doesn't propagate it). Without this, a confident
  match carrying only genres/authors/status/year but no cover and no overview
  failed the no-match check and was discarded + terminally stamped.
- byChapterIndex: both un-indexed chapters yield POSITIVE_INFINITY, so the
  subtraction was Infinity-Infinity=NaN (Array.sort treats NaN as 0, leaving
  order undefined). Compare explicitly for a stable order.
- MangaContent volume/chapter badges: derive counts from the rendered
  buildMangaList entries (which canonicalize v01 ≡ 1) instead of raw distinct
  volume tokens, so the badge can no longer say '2 Volumes' over one row.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(manga): clarify the enrichment claim's secondary arm is admin-reset-only

The secondary arm (poster present, backdrop/status missing) requires
last_refreshed IS NULL, so it is only reachable when an operator resets
last_refreshed to backfill a newly-added field — not an automatic periodic
re-check (which would re-fetch banner-less series every sweep). Documents the
intent so it does not read as dead code.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(manga): collapse continue-reading chapters per series; batch provider-id lookup

- Continue Reading now collapses multiple in-progress chapters of the same
  manga into one card (most recently read kept), mirroring the episode→series
  collapse. The reading section resolves chapter→series linkage into itemMeta
  (applyMangaChapterSeriesMeta) and runs the shared
  collapseContinueWatchingSeriesCandidates, which the reading path previously
  skipped.
- claimBatch resolves provider IDs for the whole batch in one query via the
  new ProviderIDRepository.GetByContentIDs (content_id = ANY), replacing the
  per-item GetByContentID N+1.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(web): manga publication-status chip on browse cards + more legible chips

- Color-coded publication status pill (Ongoing/Completed/Hiatus/Cancelled/
  Upcoming) in the manga card's top-left corner, mirroring the vol/chapter
  count chip top-right. Strictly manga-gated; show_status was already on the
  browse payload.
- New .glass-chip (78% surface vs glass-subtle's 40%) for the manga count +
  status pills so the labels stay legible over busy cover art.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* build(manga): depend on published silo-plugin-sdk v0.7.0

Replace the vendored internal/compat/silo-plugin-sdk copy with a normal
dependency on the published SDK module at v0.7.0, which adds
MetadataItem.status (publication/airing status) consumed by the manga
status badge at internal/metadata/plugin_provider.go.

- go.mod: pin v0.7.0, drop the local-path replace directive
- remove the vendored internal/compat/silo-plugin-sdk tree
- Dockerfile: drop the vendored-SDK COPY
- strip the manga design docs/plans from docs/superpowers (internal)

Requires Silo-Server/silo-plugin-sdk#4 merged and tagged v0.7.0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(manga): exclude chapters from the matcher's unmatched-item lister

Manga chapters are type='ebook' items that stay status='pending' by
design - provider metadata lives on the type='manga' series item. The
scan-final RetryUnmatchedItemsByFolderAndPathPrefix listed all of them
and ran a rate-limited ebook-plugin search per chapter: 31,564 chapters
x ~1s = 8h46m appended to a 2-minute manga library scan (observed
live), every one a guaranteed no-match. Earlier runs never survived to
completion, so the library's last_scanned_at stayed NULL forever.

Add the same manga_chapters NOT EXISTS guard the ebook enricher's
claim query already uses. Verified live: the same library now scans in
27s with retried_items=0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(scanner): never probe-repair ebook/comic files (ebook+manga detail-page killer)

NeedsCriticalProbeRepair was always true for BaseType 'ebook' files (epub, pdf,
cbz, cbr — incl. manga chapters): buildEbookMediaFile leaves ProbeUpdatedAt nil
and they have no audio/video, so probeEnsurer.Ensure spawned ffprobe per file on
every detail/watch load and never converged (ffprobe errors on zip/rar, result
never persisted). Short-circuit probe-repair for ebook base type — they're read
directly and never use the transcode/playback probe pipeline.

SHARED fix: benefits both the ebooks and manga library types.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf+fix(ebooks): parallelize detail extension + preserve finished read-state

- buildEbookExtension ran its 3 related-content queries (series, also-by-author,
  similar) sequentially; run them concurrently like buildAudiobookExtension so
  ebook detail latency is the slowest query, not their sum.
- PGEbookReaderProgressStore.Upsert did an unconditional SET progress=EXCLUDED;
  a routine autosave (e.g. reopening a finished book) could drop it below the
  0.9 finished threshold and silently un-mark it read (and clear the manga
  chapter checkmark, which rides on the same row). Guard: once finished,
  progress only moves on an explicit unread (row delete); below threshold it
  tracks freely.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf(manga): batch chapter presign, index volume counts, quiet scan log

- fetchMangaChapters presigned each chapter poster individually; a long-running
  series has hundreds of chapters. Batch them in one PresignImageURLs call, and
  add the missing rows.Err() check (was silently returning partial lists).
- The browse manga count chip's count(DISTINCT volume) subquery wasn't covered
  by manga_chapters_series (series_content_id, chapter_index); add
  idx_manga_chapters_series_volume (series_content_id, volume) so both count
  subqueries are index-only.
- Downgrade the per-chapter "manga scan: indexed" log from Info to Debug (one
  line per .cbz; the 500-file progress log already covers operator visibility).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(manga): address PR #138 code-review findings

Folds PR #142 into the manga branch (already done via fast-forward) and
remediates the issues surfaced in the #138 code review.

Correctness:
- Preserve the scanner's manga_series identity anchor through enrichment.
  ReplaceByContentID's DELETE was unconditional, so the first successful
  enrichment wiped the manga_series provider-id row the scanner relies on
  for idempotency, causing duplicate series + metadata loss on the next
  scan. excludedProviderIDs now also means "not deleted", and the DELETE
  preserves those rows. (internal/catalog/provider_id_repo.go)
- Fall back to the series cover when the latest chapter has no poster.
  Poster columns default to '' (not NULL), so the manga series-card poster
  override blanked cards via a plain COALESCE; wrap operands in NULLIF.
  (internal/sections/fetcher.go)
- Keep backTo a real query param on reader links when libraryId is absent.
  It was string-concatenated with '&', producing a malformed URL on
  deep-links; route it through the query helper instead.
  (web/src/lib/mediaNavigation.ts, EbookReader.tsx, MangaContent.tsx)

Quality:
- Hide manga chapters from favorites/watchlist browse, matching the
  exclusion enforced on every other listing surface.
  (internal/catalog/favorites_browse.go)
- Centralize the manga chapter exclusion predicate into a single exported
  catalog.MangaChapterExclusionWhere, removing four duplicated copies.
  (catalog, sections, ebooks)
- Skip the two manga count subqueries on browse scopes that cannot contain
  manga (non-manga type filters), substituting NULL placeholders.
  (internal/catalog/browse.go)
- Normalize provider publication status (AniList/MangaDex/SDK variants)
  into a stable label set so show_status carries one manga value-domain.
  (internal/manga/enrichment.go)

Adds unit tests for the poster NULLIF contract, browse gating, and status
normalization.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore: regenerate go.sum after rebase onto main

Drops stale silo-plugin-sdk v0.6.0 and other leftover hashes from the
intermediate rebased states; go.mod is now on the published v0.7.0 tag.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(scanner): adapt manga scan to ebookFileShouldSkip 3-value signature

main changed ebookFileShouldSkip to also return the existing content ID;
the manga scan path only needs the unchanged flag, so discard the new
return. Resolves a silent semantic conflict from the rebase onto main.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Silo Server Developer <warmasterx555@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 20:13:10 -04:00

2839 lines
86 KiB
Go

package scanner
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"log/slog"
"os"
"path/filepath"
"reflect"
"strings"
"sync"
"sync/atomic"
"time"
"github.com/Silo-Server/silo-server/internal/catalog"
"github.com/Silo-Server/silo-server/internal/models"
"github.com/Silo-Server/silo-server/internal/naming"
"github.com/Silo-Server/silo-server/internal/s3client"
)
// videoExtensions is the set of file extensions recognized as media files.
var videoExtensions = map[string]bool{
".mkv": true,
".mp4": true,
".avi": true,
".m4v": true,
".ts": true,
".wmv": true,
}
// SupportsVideoFile reports whether the given path uses a recognized media extension.
func SupportsVideoFile(filePath string) bool {
return videoExtensions[strings.ToLower(filepath.Ext(filePath))]
}
// ignoredDirNames is the set of directory names skipped during scanning.
var ignoredDirNames = map[string]bool{
".recyclebin": true,
"@recycle": true,
"@eadir": true,
".trash": true,
"#recycle": true,
"$recycle.bin": true,
".deleted": true,
".inbound": true,
".downloads": true,
}
var ignoredMovieSupplementalDirNames = map[string]bool{
"sample": true,
"samples": true,
"extra": true,
"extras": true,
"featurette": true,
"featurettes": true,
"behind the scenes": true,
"deleted scenes": true,
"trailer": true,
"trailers": true,
"subs": true,
"subtitles": true,
}
func normalizeScannerDirLabel(name string) string {
surface := strings.ToLower(strings.TrimSpace(name))
surface = strings.NewReplacer(".", " ", "_", " ", "-", " ").Replace(surface)
return strings.Join(strings.Fields(surface), " ")
}
func shouldSkipMovieSupplementalDir(path string) bool {
label := normalizeScannerDirLabel(filepath.Base(path))
if label == "" {
return false
}
return ignoredMovieSupplementalDirNames[label]
}
func shouldSkipMovieSupplementalFile(path string) bool {
baseNoExt := strings.TrimSuffix(filepath.Base(path), filepath.Ext(path))
surface := normalizeScannerDirLabel(baseNoExt)
if surface == "" {
return false
}
if surface != "sample" && !strings.HasPrefix(surface, "sample ") && !strings.HasSuffix(surface, " sample") {
return false
}
parentTitle, parentYear, trusted := naming.ParseInferFolderTitleYear(filepath.Base(filepath.Dir(path)))
if !trusted {
return true
}
stem := naming.ParseInferMovieStem(baseNoExt, parentTitle, parentYear)
return stem.Title == "" || !naming.InferTitlesCoherent(parentTitle, stem.Title)
}
// Scanner discovers and indexes media files in media folders.
// scannerImageCacher is the slice of imagecache.Cacher the book scanners use.
// Wired via SetImageCacher; nil-safe when absent.
type scannerImageCacher interface {
audiobookCoverCacher
ebookCoverCacher
}
type Scanner struct {
fileRepo *FileRepository
rootSnapshotRepo *ScannedRootRepository
groupSnapshotRepo *ScannedGroupRepository
rootOverrideRepo *MediaRootOverrideRepository
groupOverrideRepo *MediaGroupOverrideRepository
locationRepo *ObservedLocationRepository
groupLocationRepo *GroupLocationRepository
folderRepo *catalog.FolderRepository
libraryRepo *catalog.LibraryItemRepository
episodeLibraryRepo *catalog.EpisodeLibraryRepository
itemRepo *catalog.ItemRepository
personRepo *catalog.PersonRepository
episodeRepo *catalog.EpisodeRepository
ffprobePath string
s3Client *s3client.Client // public assets bucket (may be nil)
imageCacher scannerImageCacher
// workers is atomic so admin settings changes can resize the per-scan
// worker pool while a scan is running (applies to the next scan).
workers atomic.Int32
emptyTrashAfterScan bool
markerFetcher func(context.Context, string) *IntroCreditsMarkers
metadataQueue MetadataQueueProducer
movieQueueSyncer MovieQueueSyncer
seriesQueueSyncer SeriesQueueSyncer
literaryWorkLinker LiteraryWorkLinker
}
// SetImageCacher installs the imagecache.Cacher used by book scanners to push
// embedded cover art into the public assets bucket. Optional; if unset, local
// covers are not extracted.
func (s *Scanner) SetImageCacher(cacher scannerImageCacher) {
if s == nil {
return
}
s.imageCacher = cacher
}
// SetWorkers updates the scan worker pool size. Safe for concurrent use; the
// next scan run picks the new value up. Values below 1 are ignored.
func (s *Scanner) SetWorkers(workers int) {
if s == nil || workers < 1 {
return
}
s.workers.Store(int32(workers))
}
func (s *Scanner) workerCount() int { return int(s.workers.Load()) }
const scanProgressLogInterval = 10 * time.Second
// MetadataQueueProducer enqueues durable initial metadata work rows after a
// successful scan upsert.
type MetadataQueueProducer interface {
EnqueueMovieFile(ctx context.Context, fileID int) error
EnqueueSeriesRoot(ctx context.Context, folderID int, observedRootPath string) error
}
type LiteraryWorkLinker interface {
AutoLinkContent(ctx context.Context, contentID string) (workID string, linked bool, err error)
}
func (s *Scanner) SetLiteraryWorkLinker(linker LiteraryWorkLinker) {
if s == nil {
return
}
s.literaryWorkLinker = linker
}
// MovieQueueSyncer synchronizes pending movie-file match queue state from the
// scanner's persisted file rows.
type MovieQueueSyncer interface {
SyncForFolder(ctx context.Context, folderID int) error
SyncInScope(ctx context.Context, folderID int, scopePath string) error
}
// SeriesQueueSyncer synchronizes pending series-root match queue state from
// the scanner's persisted snapshot tables.
type SeriesQueueSyncer interface {
SyncForFolder(ctx context.Context, folderID int) error
SyncInScope(ctx context.Context, folderID int, scopePath string) error
}
// NewScanner creates a new Scanner with the given dependencies.
func NewScanner(fileRepo *FileRepository, ffprobePath string, s3Client *s3client.Client, workers int, emptyTrashAfterScan bool) *Scanner {
if workers < 1 {
workers = 8
}
s := &Scanner{
fileRepo: fileRepo,
rootSnapshotRepo: NewScannedRootRepository(fileRepo.Pool()),
groupSnapshotRepo: NewScannedGroupRepository(fileRepo.Pool()),
rootOverrideRepo: NewMediaRootOverrideRepository(fileRepo.Pool()),
groupOverrideRepo: NewMediaGroupOverrideRepository(fileRepo.Pool()),
locationRepo: NewObservedLocationRepository(fileRepo.Pool()),
groupLocationRepo: NewGroupLocationRepository(fileRepo.Pool()),
folderRepo: catalog.NewFolderRepository(fileRepo.Pool()),
libraryRepo: catalog.NewLibraryItemRepository(fileRepo.Pool()),
episodeLibraryRepo: catalog.NewEpisodeLibraryRepository(fileRepo.Pool()),
itemRepo: catalog.NewItemRepository(fileRepo.Pool()),
personRepo: catalog.NewPersonRepository(fileRepo.Pool()),
episodeRepo: catalog.NewEpisodeRepository(fileRepo.Pool()),
ffprobePath: ffprobePath,
s3Client: s3Client,
emptyTrashAfterScan: emptyTrashAfterScan,
markerFetcher: nil,
}
s.SetWorkers(workers)
return s
}
// SetSeriesQueueSyncer installs the optional pending-series root queue synchronizer.
func (s *Scanner) SetSeriesQueueSyncer(syncer SeriesQueueSyncer) {
if s == nil {
return
}
s.seriesQueueSyncer = syncer
}
// SetMovieQueueSyncer installs the optional pending movie-file match queue synchronizer.
func (s *Scanner) SetMovieQueueSyncer(syncer MovieQueueSyncer) {
if s == nil {
return
}
s.movieQueueSyncer = syncer
}
// SetMetadataQueueProducer installs the optional inline metadata queue producer.
func (s *Scanner) SetMetadataQueueProducer(producer MetadataQueueProducer) {
if s == nil {
return
}
s.metadataQueue = producer
}
// ScanFolder walks a media folder's directory tree, discovers media files,
// probes them for technical data, and upserts them into the database.
// Files previously in the DB that no longer exist on disk are marked as missing.
//
// Audiobook libraries are handled by ScanAudiobookFolder and podcast
// libraries by ScanPodcastFolder; both bypass the per-file movie/TV
// pipeline entirely.
func (s *Scanner) ScanFolder(ctx context.Context, folder *models.MediaFolder) (*ScanResult, error) {
watchCtx, stopWatch := s.watchFolderContext(ctx, folder.ID)
defer stopWatch()
if isAudiobookLibraryType(folder.Type) {
if err := s.ScanAudiobookFolder(watchCtx, folder); err != nil {
return nil, err
}
if err := s.syncFolderScopedAudioLibraryState(watchCtx, folder.ID); err != nil {
return nil, err
}
return &ScanResult{}, nil
}
if isPodcastLibraryType(folder.Type) {
if err := s.ScanPodcastFolder(watchCtx, folder); err != nil {
return nil, err
}
if err := s.syncFolderScopedAudioLibraryState(watchCtx, folder.ID); err != nil {
return nil, err
}
return &ScanResult{}, nil
}
if isMangaLibraryType(folder.Type) {
if err := s.ScanMangaFolder(watchCtx, folder); err != nil {
return nil, err
}
return &ScanResult{}, nil
}
if isEbookLibraryType(folder.Type) {
if err := s.ScanEbookFolder(watchCtx, folder); err != nil {
return nil, err
}
return &ScanResult{}, nil
}
return s.scanPaths(watchCtx, folder, folder.Paths, folder.Paths, true)
}
// ScanSubtree walks a single subtree within a media folder and reconciles only
// files that live beneath that subtree.
func (s *Scanner) ScanSubtree(ctx context.Context, folder *models.MediaFolder, subtreePath string) (*ScanResult, error) {
cleanSubtree := filepath.Clean(subtreePath)
watchCtx, stopWatch := s.watchFolderContext(ctx, folder.ID)
defer stopWatch()
if isAudiobookLibraryType(folder.Type) {
scanRoot, err := cleanScopedAudiobookScanRoot(subtreePath)
if err != nil {
return nil, err
}
if err := s.ScanAudiobookFolder(watchCtx, scopedFolderPaths(folder, []string{scanRoot})); err != nil {
return nil, err
}
if err := s.syncFolderScopedAudioLibraryState(watchCtx, folder.ID); err != nil {
return nil, err
}
return &ScanResult{}, nil
}
if isMangaLibraryType(folder.Type) {
if err := s.scanMangaPaths(watchCtx, folder, []string{cleanSubtree}, false); err != nil {
return nil, err
}
return &ScanResult{}, nil
}
if isEbookLibraryType(folder.Type) {
if err := s.scanEbookPaths(watchCtx, folder, []string{cleanSubtree}, false); err != nil {
return nil, err
}
return &ScanResult{}, nil
}
return s.scanPaths(watchCtx, folder, []string{cleanSubtree}, []string{cleanSubtree}, false)
}
func cleanScopedAudiobookScanRoot(path string) (string, error) {
clean := filepath.Clean(path)
if clean == "" || clean == "." || clean == ".." || clean == string(filepath.Separator) ||
strings.HasPrefix(clean, ".."+string(filepath.Separator)) || !filepath.IsAbs(clean) {
return "", fmt.Errorf("invalid audiobook scan root: %s", path)
}
return clean, nil
}
func scopedFolderPaths(folder *models.MediaFolder, paths []string) *models.MediaFolder {
if folder == nil {
return nil
}
clone := *folder
clone.Paths = paths
return &clone
}
func isMovieLibraryType(libraryType string) bool {
switch strings.ToLower(strings.TrimSpace(libraryType)) {
case "movie", "movies":
return true
default:
return false
}
}
func isAudiobookLibraryType(libraryType string) bool {
switch strings.ToLower(strings.TrimSpace(libraryType)) {
case "audiobook", "audiobooks":
return true
default:
return false
}
}
func isPodcastLibraryType(libraryType string) bool {
switch strings.ToLower(strings.TrimSpace(libraryType)) {
case "podcast", "podcasts":
return true
default:
return false
}
}
func isEbookLibraryType(libraryType string) bool {
switch strings.ToLower(strings.TrimSpace(libraryType)) {
case "ebook", "ebooks":
return true
default:
return false
}
}
func isMangaLibraryType(t string) bool {
switch strings.ToLower(strings.TrimSpace(t)) {
case "manga":
return true
default:
return false
}
}
// walkMode tells walkLogicalTree which file extensions to surface and
// which library-specific filename heuristics (sample/extra skipping)
// to apply.
type walkMode int
const (
walkModeVideo walkMode = iota // bare video walk: video extensions, no movie skipping
walkModeMovie // movie library: video extensions + sample/extra skipping
walkModeAudiobook // audiobook library: audio extensions, no skipping
walkModePodcast // podcast library: audio extensions, no skipping
walkModeEbook // ebook library: ebook extensions, no skipping
)
// walkModeFor derives a walkMode from a media_folders.type string.
// Unknown types default to walkModeVideo so existing call sites that
// pass arbitrary types preserve their prior behavior.
func walkModeFor(folderType string) walkMode {
switch {
case isMovieLibraryType(folderType):
return walkModeMovie
case isAudiobookLibraryType(folderType):
return walkModeAudiobook
case isPodcastLibraryType(folderType):
return walkModePodcast
case isEbookLibraryType(folderType):
return walkModeEbook
case isMangaLibraryType(folderType):
// Manga chapters are .cbz/.cbr archives, surfaced by the ebook walk.
return walkModeEbook
default:
return walkModeVideo
}
}
// acceptsExt reports whether the given lowercased extension belongs to
// the file types this walk mode is looking for.
func (m walkMode) acceptsExt(ext string) bool {
switch m {
case walkModeAudiobook, walkModePodcast:
return audioExtensions[ext]
case walkModeEbook:
return ebookExtensions[ext]
default:
return videoExtensions[ext]
}
}
func canonicalWalkPath(path string) (string, error) {
resolved, err := filepath.EvalSymlinks(path)
if err != nil {
return "", err
}
abs, err := filepath.Abs(resolved)
if err == nil {
resolved = abs
}
return filepath.Clean(resolved), nil
}
func isIgnoredDirectoryPath(path string) bool {
return ignoredDirNames[strings.ToLower(filepath.Base(path))]
}
// recordWalkFailure counts an entry the walk could not read or resolve.
// Callers that pass a non-nil counter (the book scanners) use it to exclude
// incompletely walked roots from missing-file reconciliation; the video walk
// passes nil and keeps its historical log-and-continue behavior.
func recordWalkFailure(failures *int) {
if failures != nil {
*failures++
}
}
func walkLogicalTree(
ctx context.Context,
logicalPath string,
physicalPath string,
mode walkMode,
visitedPhysicalDirs map[string]struct{},
filePaths *[]string,
walkFailures *int,
) error {
if ctx != nil {
if err := ctx.Err(); err != nil {
return err
}
}
info, err := os.Lstat(physicalPath)
if err != nil {
slog.Warn("scanner: walk lstat failed", "path", logicalPath, "physical_path", physicalPath, "error", err)
recordWalkFailure(walkFailures)
return nil
}
if info.Mode()&os.ModeSymlink != 0 {
resolved, err := filepath.EvalSymlinks(physicalPath)
if err != nil {
slog.Warn("scanner: symlink resolve failed", "path", logicalPath, "physical_path", physicalPath, "error", err)
recordWalkFailure(walkFailures)
return nil
}
targetInfo, err := os.Stat(resolved)
if err != nil {
slog.Warn("scanner: symlink stat failed", "path", logicalPath, "resolved_path", resolved, "error", err)
recordWalkFailure(walkFailures)
return nil
}
if targetInfo.IsDir() {
return walkLogicalTree(ctx, logicalPath, resolved, mode, visitedPhysicalDirs, filePaths, walkFailures)
}
if mode == walkModeMovie && shouldSkipMovieSupplementalFile(logicalPath) {
return nil
}
if mode.acceptsExt(strings.ToLower(filepath.Ext(logicalPath))) {
*filePaths = append(*filePaths, logicalPath)
}
return nil
}
if !info.IsDir() {
if mode == walkModeMovie && shouldSkipMovieSupplementalFile(logicalPath) {
return nil
}
if mode.acceptsExt(strings.ToLower(filepath.Ext(logicalPath))) {
*filePaths = append(*filePaths, logicalPath)
}
return nil
}
canonicalDir, err := canonicalWalkPath(physicalPath)
if err != nil {
slog.Warn("scanner: canonical path resolution failed", "path", logicalPath, "physical_path", physicalPath, "error", err)
recordWalkFailure(walkFailures)
return nil
}
if _, seen := visitedPhysicalDirs[canonicalDir]; seen {
return nil
}
visitedPhysicalDirs[canonicalDir] = struct{}{}
if isIgnoredDirectoryPath(logicalPath) {
return nil
}
if mode == walkModeMovie && shouldSkipMovieSupplementalDir(logicalPath) {
return nil
}
entries, err := os.ReadDir(physicalPath)
if err != nil {
slog.Warn("scanner: directory read failed", "path", logicalPath, "physical_path", physicalPath, "error", err)
recordWalkFailure(walkFailures)
return nil
}
for _, entry := range entries {
if ctx != nil {
if err := ctx.Err(); err != nil {
return err
}
}
logicalChild := filepath.Join(logicalPath, entry.Name())
physicalChild := filepath.Join(physicalPath, entry.Name())
if entry.Type()&os.ModeSymlink != 0 {
resolved, err := filepath.EvalSymlinks(physicalChild)
if err != nil {
slog.Warn("scanner: symlink resolve failed", "path", logicalChild, "physical_path", physicalChild, "error", err)
recordWalkFailure(walkFailures)
continue
}
targetInfo, err := os.Stat(resolved)
if err != nil {
slog.Warn("scanner: symlink stat failed", "path", logicalChild, "resolved_path", resolved, "error", err)
recordWalkFailure(walkFailures)
continue
}
if targetInfo.IsDir() {
if err := walkLogicalTree(ctx, logicalChild, resolved, mode, visitedPhysicalDirs, filePaths, walkFailures); err != nil {
return err
}
continue
}
if mode == walkModeMovie && shouldSkipMovieSupplementalFile(logicalChild) {
continue
}
if mode.acceptsExt(strings.ToLower(filepath.Ext(entry.Name()))) {
*filePaths = append(*filePaths, logicalChild)
}
continue
}
if entry.IsDir() {
if err := walkLogicalTree(ctx, logicalChild, physicalChild, mode, visitedPhysicalDirs, filePaths, walkFailures); err != nil {
return err
}
continue
}
if mode == walkModeMovie && shouldSkipMovieSupplementalFile(logicalChild) {
continue
}
if mode.acceptsExt(strings.ToLower(filepath.Ext(entry.Name()))) {
*filePaths = append(*filePaths, logicalChild)
}
}
return nil
}
func collectLogicalFilePaths(ctx context.Context, walkRoots []string, libraryType string) ([]string, error) {
filePaths := make([]string, 0)
visitedPhysicalDirs := make(map[string]struct{})
mode := walkModeFor(libraryType)
for _, rootPath := range walkRoots {
if ctx != nil {
if err := ctx.Err(); err != nil {
return nil, err
}
}
cleanRoot := filepath.Clean(rootPath)
if cleanRoot == "" || cleanRoot == "." {
continue
}
if err := walkLogicalTree(ctx, cleanRoot, cleanRoot, mode, visitedPhysicalDirs, &filePaths, nil); err != nil {
return nil, err
}
}
return filePaths, nil
}
func (s *Scanner) scanPaths(
ctx context.Context,
folder *models.MediaFolder,
walkRoots []string,
reconcileRoots []string,
allowEmptyRootGuard bool,
) (*ScanResult, error) {
if err := s.ensureFolderEnabled(ctx, folder.ID); err != nil {
return nil, err
}
if allowEmptyRootGuard {
return s.scanFolderByRoots(ctx, folder, walkRoots, reconcileRoots)
}
result := &ScanResult{}
// Get existing files in this scan scope from the DB.
var (
existingFiles []*scanStateFile
err error
)
if allowEmptyRootGuard {
existingFiles, err = s.fileRepo.GetScanStateByFolder(ctx, folder.ID)
if err != nil {
return nil, fmt.Errorf("getting existing files for folder %d: %w", folder.ID, err)
}
} else {
existingFiles, err = s.fileRepo.GetScanStateByFolderAndPathPrefix(ctx, folder.ID, reconcileRoots[0])
if err != nil {
return nil, fmt.Errorf("getting existing files for folder %d path %q: %w", folder.ID, reconcileRoots[0], err)
}
}
// Build a set of existing file paths for quick lookup.
existingByPath := make(map[string]*scanStateFile, len(existingFiles))
for _, f := range existingFiles {
existingByPath[f.FilePath] = f
}
existingContentStatuses, err := s.itemRepo.GetStatusByIDs(ctx, collectScanStateContentIDs(existingFiles))
if err != nil {
return nil, fmt.Errorf("loading item statuses for folder %d: %w", folder.ID, err)
}
// Phase 1: Collect all media file paths.
reportProgress(ctx, ProgressUpdate{
Phase: "walking",
Message: "Discovering media files",
CurrentScope: firstScope(reconcileRoots),
})
filePaths, walkErr := collectLogicalFilePaths(ctx, walkRoots, folder.Type)
if walkErr != nil {
return nil, fmt.Errorf("walking media roots: %w", walkErr)
}
slog.Info("scanner: discovered files",
"folder_id", folder.ID,
"scope", firstScope(reconcileRoots),
"files", len(filePaths),
)
reportProgress(ctx, ProgressUpdate{
Phase: "processing",
Message: "Processing discovered files",
CurrentScope: firstScope(reconcileRoots),
TotalFiles: len(filePaths),
FilesDiscovered: len(filePaths),
})
// Track which paths we see on disk so we can detect missing files.
seenPaths := make(map[string]bool, len(filePaths))
for _, p := range filePaths {
seenPaths[p] = true
}
rootOverrides, err := s.loadRootOverrides(ctx, folder.ID, reconcileRoots)
if err != nil {
return nil, fmt.Errorf("loading root overrides: %w", err)
}
rootInference := inferRootAssignments(filePaths, folder.Type, folder.ID, rootOverrides)
groupInference := inferGroupAssignments(filePaths, folder.Type, folder.ID, rootInference.Assignments)
groupOverrides, err := s.loadGroupOverrides(ctx, folder.ID)
if err != nil {
return nil, fmt.Errorf("loading group overrides: %w", err)
}
applyGroupOverrides(&groupInference, groupOverrides)
result.RootObservations = rootInference.Observations
s.logRootInferenceDisagreements(rootInference.Assignments)
if err := s.reconcileScannedRoots(
ctx,
folder.ID,
reconcileRoots,
rootInference.Snapshots,
); err != nil {
return nil, fmt.Errorf("reconciling scanned roots: %w", err)
}
if _, err := s.clearLegacyLinksForUnmatchableRoots(ctx, folder.ID, result.RootObservations); err != nil {
return nil, fmt.Errorf("clearing legacy links for unmatchable roots: %w", err)
}
allowEmptyCleanup := false
if allowEmptyRootGuard && len(filePaths) == 0 && len(existingFiles) > 0 {
var err error
allowEmptyCleanup, err = s.folderRepo.ConsumeEmptyCleanupAllowance(ctx, folder.ID)
if err != nil {
return nil, fmt.Errorf("checking empty cleanup confirmation for folder %d: %w", folder.ID, err)
}
if !allowEmptyCleanup {
result.EmptyRootGuarded = true
if err := s.folderRepo.SetScanWarning(ctx, folder.ID,
"empty_root",
"Scan found 0 media files; cleanup was skipped until deletion is confirmed.",
time.Now().UTC(),
); err != nil {
return nil, fmt.Errorf("recording empty-root warning for folder %d: %w", folder.ID, err)
}
return result, nil
}
}
// Phase 2: Process files concurrently with a worker pool.
var wg sync.WaitGroup
pathCh := make(chan string, s.workerCount())
var newCount, updatedCount, unchangedCount, errorCount, processedCount atomic.Int64
subtitleCache := newExternalSubtitleDirCache()
for range s.workerCount() {
wg.Go(func() {
for path := range pathCh {
if ctx.Err() != nil {
continue // drain channel
}
action, updateReasons, processErr := s.processFile(ctx, path, folder, existingByPath, existingContentStatuses, rootInference.Assignments[path], groupInference.Assignments[path], subtitleCache)
if processErr != nil {
slog.Error("scanner: file processing failed", "path", path, "error", processErr)
errorCount.Add(1)
continue
}
switch action {
case actionNew:
newCount.Add(1)
slog.Debug("scanner: new file added", "path", path)
case actionUpdated:
updatedCount.Add(1)
slog.Debug("scanner: file updated", "path", path, "reasons", updateReasons)
case actionUnchanged:
unchangedCount.Add(1)
}
processedCount.Add(1)
}
})
}
stopProgress := make(chan struct{})
progressDone := make(chan struct{})
go func() {
ticker := time.NewTicker(scanProgressLogInterval)
defer ticker.Stop()
defer close(progressDone)
for {
select {
case <-ctx.Done():
return
case <-stopProgress:
return
case <-ticker.C:
processed := int(processedCount.Load())
total := len(filePaths)
slog.Info("scanner: processing progress",
"folder_id", folder.ID,
"scope", firstScope(reconcileRoots),
"processed", processed,
"total", total,
"new", newCount.Load(),
"updated", updatedCount.Load(),
"unchanged", unchangedCount.Load(),
"errors", errorCount.Load(),
)
reportProgress(ctx, ProgressUpdate{
Phase: "processing",
Message: "Processing discovered files",
CurrentScope: firstScope(reconcileRoots),
TotalFiles: total,
FilesDiscovered: total,
FilesProcessed: processed,
New: int(newCount.Load()),
Updated: int(updatedCount.Load()),
Unchanged: int(unchangedCount.Load()),
Errors: int(errorCount.Load()),
})
}
}
}()
for _, p := range filePaths {
pathCh <- p
}
close(pathCh)
wg.Wait()
close(stopProgress)
<-progressDone
result.New = int(newCount.Load())
result.Updated = int(updatedCount.Load())
result.Unchanged = int(unchangedCount.Load())
result.Errors = int(errorCount.Load())
reportProgress(ctx, ProgressUpdate{
Phase: "reconciling",
Message: "Reconciling scan state",
CurrentScope: firstScope(reconcileRoots),
TotalFiles: len(filePaths),
FilesDiscovered: len(filePaths),
FilesProcessed: int(processedCount.Load()),
New: result.New,
Updated: result.Updated,
Unchanged: result.Unchanged,
Errors: result.Errors,
})
if walkErr != nil {
return nil, walkErr
}
// If the scan was cancelled, return partial results without marking
// files as missing or deleting records — that would corrupt state.
if ctx.Err() != nil {
return result, ctx.Err()
}
if err := s.syncPresentLibraryState(ctx, folder.ID); err != nil {
return nil, fmt.Errorf("syncing present library state for folder %d: %w", folder.ID, err)
}
if err := s.reconcileScannedGroups(ctx, folder.ID, allowEmptyRootGuard, reconcileRoots, !allowEmptyRootGuard, groupInference); err != nil {
return nil, fmt.Errorf("reconciling scanned groups: %w", err)
}
// Mark files that were in the DB but not found on disk as missing.
now := time.Now().UTC()
for _, existing := range existingFiles {
if seenPaths[existing.FilePath] {
continue
}
// Only mark as missing if not already marked.
if existing.MissingSince == nil {
if err := s.fileRepo.MarkMissing(ctx, existing.ID, now); err != nil {
slog.Error("scanner: failed to mark file missing",
"path", existing.FilePath,
"error", err,
)
result.Errors++
continue
}
}
result.Missing++
}
// Empty trash: delete all files marked as missing for this folder.
// Safe because the empty-root guard (above) returns early when 0 files
// are found on disk, so we only reach here when the root is populated.
if s.emptyTrashAfterScan {
trashed, err := s.fileRepo.DeleteMissingByFolder(ctx, folder.ID)
if err != nil {
return nil, fmt.Errorf("emptying trash for folder %d: %w", folder.ID, err)
}
if trashed > 0 {
slog.Info("scanner: emptied trash", "folder_id", folder.ID, "deleted", trashed)
}
result.FilesDeleted += trashed
}
staleFileIDs := collectStaleRemovedPathFileIDs(existingFiles, seenPaths, reconcileRoots)
if len(filePaths) == 0 && allowEmptyCleanup {
staleFileIDs = make([]int, 0, len(existingFiles))
for _, existing := range existingFiles {
staleFileIDs = append(staleFileIDs, existing.ID)
}
}
deletedFiles, err := s.fileRepo.DeleteByIDs(ctx, staleFileIDs)
if err != nil {
return nil, fmt.Errorf("deleting stale files for folder %d: %w", folder.ID, err)
}
result.FilesDeleted += deletedFiles
removedMemberships, deletedItems, orphanedImageDirs, err := s.reconcileLibraryMemberships(ctx, folder.ID)
if err != nil {
return nil, fmt.Errorf("reconciling library membership for folder %d: %w", folder.ID, err)
}
result.MembershipsRemoved = removedMemberships
result.ItemsDeleted = deletedItems
if s.seriesQueueSyncer != nil {
if allowEmptyRootGuard {
if err := s.seriesQueueSyncer.SyncForFolder(ctx, folder.ID); err != nil {
return nil, fmt.Errorf("syncing series match queue for folder %d: %w", folder.ID, err)
}
slog.Info("metadata: series root queue sync",
"folder_id", folder.ID,
"scope", "folder",
)
} else if len(reconcileRoots) > 0 {
for _, scopePath := range reconcileRoots {
if err := s.seriesQueueSyncer.SyncInScope(ctx, folder.ID, scopePath); err != nil {
return nil, fmt.Errorf("syncing series match queue for scope %q: %w", scopePath, err)
}
slog.Info("metadata: series root queue sync",
"folder_id", folder.ID,
"scope", scopePath,
)
}
}
}
if s.movieQueueSyncer != nil {
if allowEmptyRootGuard {
if err := s.movieQueueSyncer.SyncForFolder(ctx, folder.ID); err != nil {
return nil, fmt.Errorf("syncing movie match queue for folder %d: %w", folder.ID, err)
}
slog.Info("metadata: movie file queue sync",
"folder_id", folder.ID,
"scope", "folder",
)
} else if len(reconcileRoots) > 0 {
for _, scopePath := range reconcileRoots {
if err := s.movieQueueSyncer.SyncInScope(ctx, folder.ID, scopePath); err != nil {
return nil, fmt.Errorf("syncing movie match queue for scope %q: %w", scopePath, err)
}
slog.Info("metadata: movie file queue sync",
"folder_id", folder.ID,
"scope", scopePath,
)
}
}
}
// Best-effort S3 image cleanup for orphaned items.
if s.s3Client != nil && len(orphanedImageDirs) > 0 {
bucket := s.s3Client.Bucket()
for _, dir := range orphanedImageDirs {
_, _ = s.s3Client.DeletePrefix(ctx, bucket, dir)
}
}
if allowEmptyRootGuard && (len(filePaths) > 0 || allowEmptyCleanup) {
if err := s.folderRepo.ClearScanWarning(ctx, folder.ID); err != nil {
return nil, fmt.Errorf("clearing scan warning for folder %d: %w", folder.ID, err)
}
}
return result, nil
}
type scopedScan struct {
walkRoots []string
reconcileRoots []string
existingFiles []*scanStateFile
filePaths []string
seenPaths map[string]bool
rootInference rootInferenceResult
groupInference groupInferenceResult
result *ScanResult
}
func (s *Scanner) scanFolderByRoots(
ctx context.Context,
folder *models.MediaFolder,
walkRoots []string,
reconcileRoots []string,
) (*ScanResult, error) {
result := &ScanResult{}
roots := compactScanRoots(reconcileRoots)
if len(roots) == 0 {
roots = compactScanRoots(walkRoots)
}
pendingEmptyScopes := make([]*scopedScan, 0)
totalExisting := 0
seenAnyFiles := false
allowEmptyCleanup := false
for _, root := range roots {
reportProgress(ctx, ProgressUpdate{
Phase: "walking",
Message: "Scanning library root",
CurrentScope: root,
})
scope, err := s.scanScope(ctx, folder, []string{root}, []string{root})
if err != nil {
return nil, err
}
totalExisting += len(scope.existingFiles)
mergeScanResult(result, scope.result)
if ctx.Err() != nil {
return result, ctx.Err()
}
if len(scope.filePaths) > 0 {
seenAnyFiles = true
for _, pending := range pendingEmptyScopes {
beforeErrors := pending.result.Errors
if err := s.applyScopedScan(ctx, folder, pending, false); err != nil {
return nil, err
}
mergeCleanupResult(result, pending.result, beforeErrors)
}
pendingEmptyScopes = pendingEmptyScopes[:0]
beforeErrors := scope.result.Errors
if err := s.applyScopedScan(ctx, folder, scope, false); err != nil {
return nil, err
}
mergeCleanupResult(result, scope.result, beforeErrors)
continue
}
if seenAnyFiles {
beforeErrors := scope.result.Errors
if err := s.applyScopedScan(ctx, folder, scope, false); err != nil {
return nil, err
}
mergeCleanupResult(result, scope.result, beforeErrors)
continue
}
pendingEmptyScopes = append(pendingEmptyScopes, scope)
}
if !seenAnyFiles && totalExisting > 0 {
var err error
allowEmptyCleanup, err = s.folderRepo.ConsumeEmptyCleanupAllowance(ctx, folder.ID)
if err != nil {
return nil, fmt.Errorf("checking empty cleanup confirmation for folder %d: %w", folder.ID, err)
}
if !allowEmptyCleanup {
result.EmptyRootGuarded = true
if err := s.folderRepo.SetScanWarning(ctx, folder.ID,
"empty_root",
"Scan found 0 media files; cleanup was skipped until deletion is confirmed.",
time.Now().UTC(),
); err != nil {
return nil, fmt.Errorf("recording empty-root warning for folder %d: %w", folder.ID, err)
}
return result, nil
}
}
for _, pending := range pendingEmptyScopes {
beforeErrors := pending.result.Errors
if err := s.applyScopedScan(ctx, folder, pending, allowEmptyCleanup); err != nil {
return nil, err
}
mergeCleanupResult(result, pending.result, beforeErrors)
}
if ctx.Err() != nil {
return result, ctx.Err()
}
reportProgress(ctx, ProgressUpdate{
Phase: "reconciling",
Message: "Reconciling library state",
CurrentScope: "folder",
})
if err := s.syncPresentLibraryState(ctx, folder.ID); err != nil {
return nil, fmt.Errorf("syncing present library state for folder %d: %w", folder.ID, err)
}
if s.emptyTrashAfterScan {
trashed, err := s.fileRepo.DeleteMissingByFolder(ctx, folder.ID)
if err != nil {
return nil, fmt.Errorf("emptying trash for folder %d: %w", folder.ID, err)
}
if trashed > 0 {
slog.Info("scanner: emptied trash", "folder_id", folder.ID, "deleted", trashed)
}
result.FilesDeleted += trashed
}
staleOutsideRoots, err := s.fileRepo.ListIDsOutsideRoots(ctx, folder.ID, roots)
if err != nil {
return nil, fmt.Errorf("listing stale files outside configured roots for folder %d: %w", folder.ID, err)
}
deletedOutsideRoots, err := s.fileRepo.DeleteByIDs(ctx, staleOutsideRoots)
if err != nil {
return nil, fmt.Errorf("deleting stale files outside configured roots for folder %d: %w", folder.ID, err)
}
result.FilesDeleted += deletedOutsideRoots
removedMemberships, deletedItems, orphanedImageDirs, err := s.reconcileLibraryMemberships(ctx, folder.ID)
if err != nil {
return nil, fmt.Errorf("reconciling library membership for folder %d: %w", folder.ID, err)
}
result.MembershipsRemoved = removedMemberships
result.ItemsDeleted = deletedItems
if s.seriesQueueSyncer != nil {
reportProgress(ctx, ProgressUpdate{
Phase: "queue_sync",
Message: "Syncing series match queue",
CurrentScope: "folder",
})
if err := s.seriesQueueSyncer.SyncForFolder(ctx, folder.ID); err != nil {
return nil, fmt.Errorf("syncing series match queue for folder %d: %w", folder.ID, err)
}
slog.Info("metadata: series root queue sync",
"folder_id", folder.ID,
"scope", "folder",
)
}
if s.movieQueueSyncer != nil {
reportProgress(ctx, ProgressUpdate{
Phase: "queue_sync",
Message: "Syncing movie match queue",
CurrentScope: "folder",
})
if err := s.movieQueueSyncer.SyncForFolder(ctx, folder.ID); err != nil {
return nil, fmt.Errorf("syncing movie match queue for folder %d: %w", folder.ID, err)
}
slog.Info("metadata: movie file queue sync",
"folder_id", folder.ID,
"scope", "folder",
)
}
if s.s3Client != nil && len(orphanedImageDirs) > 0 {
bucket := s.s3Client.Bucket()
for _, dir := range orphanedImageDirs {
_, _ = s.s3Client.DeletePrefix(ctx, bucket, dir)
}
}
if seenAnyFiles || allowEmptyCleanup {
if err := s.folderRepo.ClearScanWarning(ctx, folder.ID); err != nil {
return nil, fmt.Errorf("clearing scan warning for folder %d: %w", folder.ID, err)
}
}
return result, nil
}
func (s *Scanner) scanScope(
ctx context.Context,
folder *models.MediaFolder,
walkRoots []string,
reconcileRoots []string,
) (*scopedScan, error) {
result := &ScanResult{}
existingFiles, err := s.fileRepo.GetScanStateByFolderAndPathPrefix(ctx, folder.ID, reconcileRoots[0])
if err != nil {
return nil, fmt.Errorf("getting existing files for folder %d path %q: %w", folder.ID, reconcileRoots[0], err)
}
existingByPath := make(map[string]*scanStateFile, len(existingFiles))
for _, f := range existingFiles {
existingByPath[f.FilePath] = f
}
existingContentStatuses, err := s.itemRepo.GetStatusByIDs(ctx, collectScanStateContentIDs(existingFiles))
if err != nil {
return nil, fmt.Errorf("loading item statuses for folder %d path %q: %w", folder.ID, reconcileRoots[0], err)
}
filePaths, walkErr := collectLogicalFilePaths(ctx, walkRoots, folder.Type)
if walkErr != nil {
return nil, fmt.Errorf("walking media roots for %q: %w", reconcileRoots[0], walkErr)
}
slog.Info("scanner: discovered files",
"folder_id", folder.ID,
"scope", firstScope(reconcileRoots),
"files", len(filePaths),
)
reportProgress(ctx, ProgressUpdate{
Phase: "processing",
Message: "Processing discovered files",
CurrentScope: firstScope(reconcileRoots),
TotalFiles: len(filePaths),
FilesDiscovered: len(filePaths),
})
seenPaths := make(map[string]bool, len(filePaths))
for _, p := range filePaths {
seenPaths[p] = true
}
rootOverrides, err := s.loadRootOverrides(ctx, folder.ID, reconcileRoots)
if err != nil {
return nil, fmt.Errorf("loading root overrides: %w", err)
}
rootInference := inferRootAssignments(filePaths, folder.Type, folder.ID, rootOverrides)
groupInference := inferGroupAssignments(filePaths, folder.Type, folder.ID, rootInference.Assignments)
groupOverrides, err := s.loadGroupOverrides(ctx, folder.ID)
if err != nil {
return nil, fmt.Errorf("loading group overrides: %w", err)
}
applyGroupOverrides(&groupInference, groupOverrides)
result.RootObservations = append(result.RootObservations, rootInference.Observations...)
s.logRootInferenceDisagreements(rootInference.Assignments)
if _, err := s.clearLegacyLinksForUnmatchableRoots(ctx, folder.ID, result.RootObservations); err != nil {
return nil, fmt.Errorf("clearing legacy links for unmatchable roots: %w", err)
}
var wg sync.WaitGroup
pathCh := make(chan string, s.workerCount())
var newCount, updatedCount, unchangedCount, errorCount, processedCount atomic.Int64
subtitleCache := newExternalSubtitleDirCache()
for range s.workerCount() {
wg.Go(func() {
for path := range pathCh {
if ctx.Err() != nil {
continue
}
action, updateReasons, processErr := s.processFile(ctx, path, folder, existingByPath, existingContentStatuses, rootInference.Assignments[path], groupInference.Assignments[path], subtitleCache)
if processErr != nil {
slog.Error("scanner: file processing failed", "path", path, "error", processErr)
errorCount.Add(1)
continue
}
switch action {
case actionNew:
newCount.Add(1)
slog.Debug("scanner: new file added", "path", path)
case actionUpdated:
updatedCount.Add(1)
slog.Debug("scanner: file updated", "path", path, "reasons", updateReasons)
case actionUnchanged:
unchangedCount.Add(1)
}
processedCount.Add(1)
}
})
}
stopProgress := make(chan struct{})
progressDone := make(chan struct{})
go func() {
ticker := time.NewTicker(scanProgressLogInterval)
defer ticker.Stop()
defer close(progressDone)
for {
select {
case <-ctx.Done():
return
case <-stopProgress:
return
case <-ticker.C:
processed := int(processedCount.Load())
total := len(filePaths)
slog.Info("scanner: processing progress",
"folder_id", folder.ID,
"scope", firstScope(reconcileRoots),
"processed", processed,
"total", total,
"new", newCount.Load(),
"updated", updatedCount.Load(),
"unchanged", unchangedCount.Load(),
"errors", errorCount.Load(),
)
reportProgress(ctx, ProgressUpdate{
Phase: "processing",
Message: "Processing discovered files",
CurrentScope: firstScope(reconcileRoots),
TotalFiles: total,
FilesDiscovered: total,
FilesProcessed: processed,
New: int(newCount.Load()),
Updated: int(updatedCount.Load()),
Unchanged: int(unchangedCount.Load()),
Errors: int(errorCount.Load()),
})
}
}
}()
for _, p := range filePaths {
pathCh <- p
}
close(pathCh)
wg.Wait()
close(stopProgress)
<-progressDone
result.New = int(newCount.Load())
result.Updated = int(updatedCount.Load())
result.Unchanged = int(unchangedCount.Load())
result.Errors = int(errorCount.Load())
reportProgress(ctx, ProgressUpdate{
Phase: "reconciling",
Message: "Reconciling scan state",
CurrentScope: firstScope(reconcileRoots),
TotalFiles: len(filePaths),
FilesDiscovered: len(filePaths),
FilesProcessed: int(processedCount.Load()),
New: result.New,
Updated: result.Updated,
Unchanged: result.Unchanged,
Errors: result.Errors,
})
return &scopedScan{
walkRoots: append([]string(nil), walkRoots...),
reconcileRoots: append([]string(nil), reconcileRoots...),
existingFiles: existingFiles,
filePaths: filePaths,
seenPaths: seenPaths,
rootInference: rootInference,
groupInference: groupInference,
result: result,
}, nil
}
func (s *Scanner) applyScopedScan(
ctx context.Context,
folder *models.MediaFolder,
scope *scopedScan,
forceDeleteAll bool,
) error {
if scope == nil || scope.result == nil {
return nil
}
if err := s.reconcileScannedRoots(
ctx,
folder.ID,
scope.reconcileRoots,
scope.rootInference.Snapshots,
); err != nil {
return fmt.Errorf("reconciling scanned roots for scope %q: %w", scope.reconcileRoots[0], err)
}
if err := s.reconcileScannedGroups(ctx, folder.ID, false, scope.reconcileRoots, true, scope.groupInference); err != nil {
return fmt.Errorf("reconciling scanned groups for scope %q: %w", scope.reconcileRoots[0], err)
}
now := time.Now().UTC()
for _, existing := range scope.existingFiles {
if scope.seenPaths[existing.FilePath] {
continue
}
if existing.MissingSince == nil {
if err := s.fileRepo.MarkMissing(ctx, existing.ID, now); err != nil {
slog.Error("scanner: failed to mark file missing",
"path", existing.FilePath,
"error", err,
)
scope.result.Errors++
continue
}
}
scope.result.Missing++
}
staleFileIDs := collectStaleRemovedPathFileIDs(scope.existingFiles, scope.seenPaths, scope.reconcileRoots)
if forceDeleteAll && len(scope.filePaths) == 0 {
staleFileIDs = make([]int, 0, len(scope.existingFiles))
for _, existing := range scope.existingFiles {
staleFileIDs = append(staleFileIDs, existing.ID)
}
}
deletedFiles, err := s.fileRepo.DeleteByIDs(ctx, staleFileIDs)
if err != nil {
return fmt.Errorf("deleting stale files for scope %q: %w", scope.reconcileRoots[0], err)
}
scope.result.FilesDeleted += deletedFiles
return nil
}
func mergeScanResult(dst *ScanResult, src *ScanResult) {
if dst == nil || src == nil {
return
}
dst.New += src.New
dst.Updated += src.Updated
dst.Unchanged += src.Unchanged
dst.Errors += src.Errors
dst.RootObservations = append(dst.RootObservations, src.RootObservations...)
dst.EmptyRootGuarded = dst.EmptyRootGuarded || src.EmptyRootGuarded
}
func firstScope(scopes []string) string {
if len(scopes) == 0 {
return ""
}
return scopes[0]
}
func mergeCleanupResult(dst *ScanResult, src *ScanResult, priorErrors int) {
if dst == nil || src == nil {
return
}
dst.Missing += src.Missing
dst.FilesDeleted += src.FilesDeleted
if src.Errors > priorErrors {
dst.Errors += src.Errors - priorErrors
}
}
func compactScanRoots(paths []string) []string {
out := make([]string, 0, len(paths))
for _, rawPath := range paths {
if strings.TrimSpace(rawPath) == "" {
continue
}
path := filepath.Clean(rawPath)
if path == "" || path == "." {
continue
}
if pathWithinAnyRoot(path, out) {
continue
}
filtered := out[:0]
for _, existing := range out {
if pathWithinAnyRoot(existing, []string{path}) {
continue
}
filtered = append(filtered, existing)
}
out = append(filtered, path)
}
return out
}
func (s *Scanner) syncPresentLibraryState(ctx context.Context, folderID int) error {
if _, err := s.fileRepo.Pool().Exec(ctx, `
UPDATE media_files mf
SET content_id = NULL,
updated_at = NOW()
WHERE mf.media_folder_id = $1
AND mf.missing_since IS NULL
AND mf.content_id IS NOT NULL
AND NOT EXISTS (
SELECT 1
FROM media_items mi
WHERE mi.content_id = mf.content_id
)
`, folderID); err != nil {
return fmt.Errorf("clearing dangling content links: %w", err)
}
if _, err := s.fileRepo.Pool().Exec(ctx, `
UPDATE media_files mf
SET episode_id = NULL,
updated_at = NOW()
WHERE mf.media_folder_id = $1
AND mf.missing_since IS NULL
AND mf.episode_id IS NOT NULL
AND NOT EXISTS (
SELECT 1
FROM episodes e
WHERE e.content_id = mf.episode_id
)
`, folderID); err != nil {
return fmt.Errorf("clearing dangling episode links: %w", err)
}
if _, err := s.fileRepo.Pool().Exec(ctx, `
INSERT INTO media_item_libraries (content_id, media_folder_id, first_seen_at)
SELECT DISTINCT mf.content_id, mf.media_folder_id, NOW()
FROM media_files mf
JOIN media_items mi ON mi.content_id = mf.content_id
WHERE mf.media_folder_id = $1
AND mf.missing_since IS NULL
AND mf.content_id IS NOT NULL
ON CONFLICT (content_id, media_folder_id) DO NOTHING
`, folderID); err != nil {
return fmt.Errorf("restoring folder memberships: %w", err)
}
if _, err := s.fileRepo.Pool().Exec(ctx, `
INSERT INTO episode_libraries (episode_id, media_folder_id, first_seen_at)
SELECT mf.episode_id, mf.media_folder_id, MIN(mf.created_at)
FROM media_files mf
JOIN episodes e ON e.content_id = mf.episode_id
WHERE mf.media_folder_id = $1
AND mf.missing_since IS NULL
AND mf.episode_id IS NOT NULL
GROUP BY mf.episode_id, mf.media_folder_id
ON CONFLICT (episode_id, media_folder_id) DO NOTHING
`, folderID); err != nil {
return fmt.Errorf("restoring episode folder memberships: %w", err)
}
return nil
}
func (s *Scanner) syncFolderScopedAudioLibraryState(ctx context.Context, folderID int) error {
if err := s.syncPresentLibraryState(ctx, folderID); err != nil {
return err
}
if _, err := s.fileRepo.Pool().Exec(ctx, `
INSERT INTO media_item_roots (media_folder_id, canonical_root_path, content_id)
SELECT DISTINCT mf.media_folder_id, mf.canonical_root_path, mf.content_id
FROM media_files mf
JOIN media_items mi ON mi.content_id = mf.content_id
WHERE mf.media_folder_id = $1
AND mf.missing_since IS NULL
AND mf.content_id IS NOT NULL
AND COALESCE(mf.canonical_root_path, '') <> ''
AND mi.type IN ('audiobook', 'podcast')
ON CONFLICT (media_folder_id, canonical_root_path)
DO UPDATE SET content_id = EXCLUDED.content_id,
last_seen_at = NOW()
`, folderID); err != nil {
return fmt.Errorf("restoring folder-scoped audio roots: %w", err)
}
return nil
}
func (s *Scanner) reconcileLibraryMemberships(ctx context.Context, folderID int) (int, int, []string, error) {
if s.episodeLibraryRepo != nil {
if _, err := s.episodeLibraryRepo.ReconcileFolderMembership(ctx, folderID); err != nil {
return 0, 0, nil, err
}
}
return s.libraryRepo.ReconcileFolderMembership(ctx, folderID)
}
func collectStaleRemovedPathFileIDs(existingFiles []*scanStateFile, seenPaths map[string]bool, roots []string) []int {
ids := make([]int, 0)
for _, existing := range existingFiles {
if seenPaths[existing.FilePath] {
continue
}
if pathWithinAnyRoot(existing.FilePath, roots) {
continue
}
ids = append(ids, existing.ID)
}
return ids
}
func collectScanStateContentIDs(files []*scanStateFile) []string {
contentIDs := make([]string, 0, len(files))
seen := make(map[string]struct{}, len(files))
for _, file := range files {
if file == nil || strings.TrimSpace(file.ContentID) == "" {
continue
}
if _, ok := seen[file.ContentID]; ok {
continue
}
seen[file.ContentID] = struct{}{}
contentIDs = append(contentIDs, file.ContentID)
}
return contentIDs
}
func pathWithinAnyRoot(path string, roots []string) bool {
cleanPath := filepath.Clean(path)
for _, root := range roots {
cleanRoot := filepath.Clean(root)
rel, err := filepath.Rel(cleanRoot, cleanPath)
if err != nil {
continue
}
if rel == "." || rel == "" {
return true
}
if rel != ".." && !strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
return true
}
}
return false
}
// ScanFile scans a single file and upserts it into the database.
func (s *Scanner) ScanFile(ctx context.Context, filePath string, folder *models.MediaFolder) error {
var stopWatch context.CancelFunc
ctx, stopWatch = s.watchFolderContext(ctx, folder.ID)
defer stopWatch()
if err := s.ensureFolderEnabled(ctx, folder.ID); err != nil {
return err
}
cleanFile := filepath.Clean(filePath)
if isAudiobookLibraryType(folder.Type) {
if !SupportsAudioFile(cleanFile) {
return fmt.Errorf("unrecognized audio extension: %s", strings.ToLower(filepath.Ext(cleanFile)))
}
scanRoot, err := cleanScopedAudiobookScanRoot(filepath.Dir(cleanFile))
if err != nil {
return err
}
if err := s.ScanAudiobookFolder(ctx, scopedFolderPaths(folder, []string{scanRoot})); err != nil {
return err
}
return s.syncFolderScopedAudioLibraryState(ctx, folder.ID)
}
// Verify the file extension is recognized.
ext := strings.ToLower(filepath.Ext(cleanFile))
if !videoExtensions[ext] {
return fmt.Errorf("unrecognized video extension: %s", ext)
}
// Look up only this specific file instead of loading the entire folder.
existingByPath := make(map[string]*scanStateFile, 1)
existing, err := s.fileRepo.GetByPath(ctx, filePath)
if err == nil {
existingByPath[filePath] = scanStateFromMediaFile(existing)
}
existingContentStatuses, err := s.itemRepo.GetStatusByIDs(ctx, collectScanStateContentIDs([]*scanStateFile{existingByPath[filePath]}))
if err != nil {
return fmt.Errorf("loading item statuses for file: %w", err)
}
observation, ok := ObserveRoot(filePath, folder.Type)
if ok {
cleared, clearErr := s.clearLegacyLinksForUnmatchableRoots(ctx, folder.ID, []RootObservation{observation})
if clearErr != nil {
return clearErr
}
if cleared > 0 {
if _, _, _, reconcileErr := s.reconcileLibraryMemberships(ctx, folder.ID); reconcileErr != nil {
return fmt.Errorf("reconciling folder membership after clearing legacy links: %w", reconcileErr)
}
}
}
rootOverrides, err := s.loadRootOverrides(ctx, folder.ID, []string{filepath.Dir(filePath)})
if err != nil {
return fmt.Errorf("loading root overrides for file: %w", err)
}
rootInference := inferRootAssignments([]string{filePath}, folder.Type, folder.ID, rootOverrides)
s.logRootInferenceDisagreements(rootInference.Assignments)
groupInference := inferGroupAssignments([]string{filePath}, folder.Type, folder.ID, rootInference.Assignments)
groupOverrides, err := s.loadGroupOverrides(ctx, folder.ID)
if err != nil {
return fmt.Errorf("loading group overrides for file: %w", err)
}
applyGroupOverrides(&groupInference, groupOverrides)
_, _, err = s.processFile(ctx, filePath, folder, existingByPath, existingContentStatuses, rootInference.Assignments[filepath.Clean(filePath)], groupInference.Assignments[filepath.Clean(filePath)], newExternalSubtitleDirCache())
if err != nil {
return err
}
if err := s.reconcileScannedRoots(
ctx,
folder.ID,
nil,
rootInference.Snapshots,
); err != nil {
return fmt.Errorf("reconciling scanned root for file: %w", err)
}
scopePath := filepath.Dir(filePath)
if err := s.reconcileScannedGroups(ctx, folder.ID, false, []string{scopePath}, false, groupInference); err != nil {
return fmt.Errorf("reconciling scanned groups for file: %w", err)
}
if err := s.syncPresentLibraryState(ctx, folder.ID); err != nil {
return fmt.Errorf("syncing present library state for file: %w", err)
}
if s.seriesQueueSyncer != nil {
if err := s.seriesQueueSyncer.SyncInScope(ctx, folder.ID, scopePath); err != nil {
return fmt.Errorf("syncing series match queue for file: %w", err)
}
slog.Info("metadata: series root queue sync",
"folder_id", folder.ID,
"scope", scopePath,
)
}
if s.movieQueueSyncer != nil {
if err := s.movieQueueSyncer.SyncInScope(ctx, folder.ID, filepath.Clean(filePath)); err != nil {
return fmt.Errorf("syncing movie match queue for file: %w", err)
}
slog.Info("metadata: movie file queue sync",
"folder_id", folder.ID,
"scope", filepath.Clean(filePath),
)
}
return nil
}
func (s *Scanner) watchFolderContext(ctx context.Context, folderID int) (context.Context, context.CancelFunc) {
cancel := func() {}
if ctx == nil || folderID <= 0 || s == nil || s.folderRepo == nil {
return ctx, cancel
}
watchCtx, cancel := context.WithCancel(ctx)
enabled, err := s.folderEnabledState(watchCtx, folderID)
if err != nil {
slog.Warn("scanner: failed to load folder state", "folder_id", folderID, "error", err)
cancel()
return watchCtx, cancel
}
if !enabled {
cancel()
return watchCtx, cancel
}
go func() {
ticker := time.NewTicker(time.Second)
defer ticker.Stop()
for {
select {
case <-watchCtx.Done():
return
case <-ticker.C:
enabled, err := s.folderEnabledState(watchCtx, folderID)
if err != nil {
slog.Warn("scanner: failed to refresh folder state", "folder_id", folderID, "error", err)
continue
}
if !enabled {
cancel()
return
}
}
}
}()
return watchCtx, cancel
}
func (s *Scanner) ensureFolderEnabled(ctx context.Context, folderID int) error {
if ctx != nil {
if err := ctx.Err(); err != nil {
return err
}
}
if s == nil || s.folderRepo == nil || folderID <= 0 {
return nil
}
enabled, err := s.folderEnabledState(ctx, folderID)
switch {
case errors.Is(err, catalog.ErrFolderNotFound):
return context.Canceled
case err != nil:
return fmt.Errorf("loading folder state: %w", err)
case !enabled:
return context.Canceled
default:
return nil
}
}
func (s *Scanner) folderEnabledState(ctx context.Context, folderID int) (bool, error) {
if s == nil || s.folderRepo == nil || folderID <= 0 {
return true, nil
}
folder, err := s.folderRepo.GetByID(ctx, folderID)
switch {
case errors.Is(err, catalog.ErrFolderNotFound):
return false, err
case err != nil:
return false, err
case folder == nil:
return false, catalog.ErrFolderNotFound
default:
return folder.Enabled, nil
}
}
// fileAction represents what happened when processing a file.
type fileAction int
const (
actionNew fileAction = iota
actionUpdated
actionUnchanged
)
// processFile handles a single file: checks if it changed, gathers hints,
// probes it, and upserts it.
func (s *Scanner) processFile(
ctx context.Context,
filePath string,
folder *models.MediaFolder,
existingByPath map[string]*scanStateFile,
existingContentStatuses map[string]string,
assignment fileRootAssignment,
groupAssignment fileGroupAssignment,
subtitleCache *externalSubtitleDirCache,
) (fileAction, []string, error) {
// Stat the file.
info, err := os.Stat(filePath)
if err != nil {
return 0, nil, fmt.Errorf("stat %s: %w", filePath, err)
}
fileSize := info.Size()
fileModifiedAt := normalizeFileModifiedAt(info.ModTime())
var externalSubs []ExternalSubtitleInfo
externalSubsLoaded := false
externalSubsChecked := false
loadExternalSubs := func() []ExternalSubtitleInfo {
if externalSubsLoaded {
return externalSubs
}
externalSubsLoaded = true
var subErr error
externalSubs, subErr = subtitleCache.Detect(filePath)
if subErr != nil {
slog.Warn("scanner: subtitle detection failed", "path", filePath, "error", subErr)
externalSubs = nil
externalSubsChecked = false
return externalSubs
}
externalSubsChecked = true
return externalSubs
}
// Check if unchanged (same size) only when playback-critical probe data
// is already present or this scanner cannot repair it on this node. This
// lets rescans repair legacy rows when local ffprobe metadata is available
// without forcing endless rewrites on nodes that cannot probe.
if existing, ok := existingByPath[filePath]; ok {
currentExternalSubtitlePaths := externalSubtitleInfoPaths(loadExternalSubs())
updateReasons := scanStateUpdateReasons(existing, fileSize, fileModifiedAt, currentExternalSubtitlePaths, externalSubsChecked, assignment, groupAssignment, folder.Type, s.ffprobePath != "")
if shouldSkipStableConfirmedScanState(existing, existingContentStatuses[existing.ContentID], fileSize, fileModifiedAt, updateReasons, s.ffprobePath != "") {
return actionUnchanged, nil, nil
}
if len(updateReasons) == 0 {
return actionUnchanged, nil, nil
}
action := actionUpdated
// Gather hints (OSHash only).
hints := s.gatherHints(filePath)
fileHash := hints.FileHash
// Try to get probe data.
probe, probeSource := s.probeFile(ctx, filePath)
// Detect external subtitles.
externalSubs = loadExternalSubs()
// Check for intro/credits markers from S3.
markerFetcher := s.markerFetcher
if markerFetcher == nil {
markerFetcher = s.fetchMarkers
}
markers := markerFetcher(ctx, fileHash)
// Build the media file for upsert.
mf := models.MediaFile{
MediaFolderID: folder.ID,
FilePath: filePath,
FileSize: fileSize,
FileModifiedAt: &fileModifiedAt,
FileHash: fileHash,
}
if assignment.RootPath != "" {
mf.CanonicalRootPath = filepath.Clean(assignment.RootPath)
} else if root, ok := naming.DetectCanonicalRoot(filePath, folder.Type); ok {
mf.CanonicalRootPath = filepath.Clean(root.RootPath)
}
mf.ObservedRootPath = filepath.Clean(groupAssignment.ObservedRootPath)
mf.ContentGroupKey = groupAssignment.ContentGroupKey
mf.GroupKeyVersion = groupAssignment.GroupKeyVersion
mf.BaseTitle = groupAssignment.BaseTitle
mf.BaseYear = groupAssignment.BaseYear
mf.BaseType = groupAssignment.BaseType
mf.IdentityConfidence = groupAssignment.Confidence
mf.IdentityJSON = append([]byte(nil), groupAssignment.EvidenceJSON...)
if filenameHints := naming.ParseFilename(filePath, folder.Type); filenameHints != nil &&
filenameHints.Type == "series" && filenameHints.EpisodeNum > 0 {
mf.SeasonNumber = filenameHints.SeasonNum
mf.EpisodeNumber = filenameHints.EpisodeNum
}
variantHints := naming.ParseVariantHints(filePath, folder.Type)
if existing != nil && existing.EditionSource == "import" && existing.EditionKey != "" {
variantHints = &naming.VariantHints{
EditionRaw: existing.EditionRaw,
EditionKey: existing.EditionKey,
EditionSource: existing.EditionSource,
EditionConfidence: existing.EditionConfidence,
PresentationKind: existing.PresentationKind,
PresentationGroupKey: existing.PresentationGroupKey,
PresentationPartIndex: existing.PresentationPartIndex,
MultiEpisodeStart: existing.MultiEpisodeStart,
MultiEpisodeEnd: existing.MultiEpisodeEnd,
}
}
if variantHints != nil {
mf.EditionRaw = variantHints.EditionRaw
mf.EditionKey = variantHints.EditionKey
mf.EditionConfidence = variantHints.EditionConfidence
mf.EditionSource = variantHints.EditionSource
mf.PresentationKind = variantHints.PresentationKind
mf.PresentationGroupKey = variantHints.PresentationGroupKey
mf.PresentationPartIndex = variantHints.PresentationPartIndex
mf.MultiEpisodeStart = variantHints.MultiEpisodeStart
mf.MultiEpisodeEnd = variantHints.MultiEpisodeEnd
}
// Apply probe data if available.
if probe != nil {
applyProbeData(&mf, probe, probeSource)
}
if mf.SubtitleTracks == nil {
mf.SubtitleTracks = []models.SubtitleTrack{}
}
modelExternalSubs := make([]models.ExternalSubtitle, len(externalSubs))
for i, es := range externalSubs {
modelExternalSubs[i] = models.ExternalSubtitle{
Path: es.Path,
Language: es.Language,
Format: es.Format,
Title: es.Title,
Forced: es.Forced,
}
}
mf.ExternalSubtitles = modelExternalSubs
if mf.ExternalSubtitles == nil {
mf.ExternalSubtitles = []models.ExternalSubtitle{}
}
upserted, upsertErr := s.fileRepo.Upsert(ctx, mf)
if upsertErr != nil {
return 0, nil, fmt.Errorf("upserting file %s: %w", filePath, upsertErr)
}
if err := s.enqueueMetadataWork(ctx, folder, upserted); err != nil {
return 0, nil, fmt.Errorf("enqueueing metadata work for file %s: %w", filePath, err)
}
if markers != nil {
applied, markerErr := s.fileRepo.UpsertMarkers(ctx, upserted.ID, MarkerUpdate{
IntroStart: markers.IntroStart,
IntroEnd: markers.IntroEnd,
CreditsStart: markers.CreditsStart,
CreditsEnd: markers.CreditsEnd,
MarkersSource: models.MarkerSourceS3,
})
if markerErr != nil {
return 0, nil, fmt.Errorf("upserting markers for file %s: %w", filePath, markerErr)
}
if !applied {
slog.Debug("scanner: skipped lower-priority s3 markers", "path", filePath, "hash", fileHash)
}
}
return action, updateReasons, nil
}
action := actionNew
// Gather hints (OSHash only).
hints := s.gatherHints(filePath)
fileHash := hints.FileHash
// Try to get probe data.
probe, probeSource := s.probeFile(ctx, filePath)
// Detect external subtitles.
externalSubs = loadExternalSubs()
// Check for intro/credits markers from S3.
markerFetcher := s.markerFetcher
if markerFetcher == nil {
markerFetcher = s.fetchMarkers
}
markers := markerFetcher(ctx, fileHash)
// Build the media file for upsert.
mf := models.MediaFile{
MediaFolderID: folder.ID,
FilePath: filePath,
FileSize: fileSize,
FileModifiedAt: &fileModifiedAt,
FileHash: fileHash,
}
if assignment.RootPath != "" {
mf.CanonicalRootPath = filepath.Clean(assignment.RootPath)
} else if root, ok := naming.DetectCanonicalRoot(filePath, folder.Type); ok {
mf.CanonicalRootPath = filepath.Clean(root.RootPath)
}
mf.ObservedRootPath = filepath.Clean(groupAssignment.ObservedRootPath)
mf.ContentGroupKey = groupAssignment.ContentGroupKey
mf.GroupKeyVersion = groupAssignment.GroupKeyVersion
mf.BaseTitle = groupAssignment.BaseTitle
mf.BaseYear = groupAssignment.BaseYear
mf.BaseType = groupAssignment.BaseType
mf.IdentityConfidence = groupAssignment.Confidence
mf.IdentityJSON = append([]byte(nil), groupAssignment.EvidenceJSON...)
if filenameHints := naming.ParseFilename(filePath, folder.Type); filenameHints != nil &&
filenameHints.Type == "series" && filenameHints.EpisodeNum > 0 {
mf.SeasonNumber = filenameHints.SeasonNum
mf.EpisodeNumber = filenameHints.EpisodeNum
}
variantHints := naming.ParseVariantHints(filePath, folder.Type)
if existing, ok := existingByPath[filePath]; ok && existing != nil && existing.EditionSource == "import" && existing.EditionKey != "" {
variantHints = &naming.VariantHints{
EditionRaw: existing.EditionRaw,
EditionKey: existing.EditionKey,
EditionSource: existing.EditionSource,
EditionConfidence: existing.EditionConfidence,
PresentationKind: existing.PresentationKind,
PresentationGroupKey: existing.PresentationGroupKey,
PresentationPartIndex: existing.PresentationPartIndex,
MultiEpisodeStart: existing.MultiEpisodeStart,
MultiEpisodeEnd: existing.MultiEpisodeEnd,
}
}
if variantHints != nil {
mf.EditionRaw = variantHints.EditionRaw
mf.EditionKey = variantHints.EditionKey
mf.EditionConfidence = variantHints.EditionConfidence
mf.EditionSource = variantHints.EditionSource
mf.PresentationKind = variantHints.PresentationKind
mf.PresentationGroupKey = variantHints.PresentationGroupKey
mf.PresentationPartIndex = variantHints.PresentationPartIndex
mf.MultiEpisodeStart = variantHints.MultiEpisodeStart
mf.MultiEpisodeEnd = variantHints.MultiEpisodeEnd
}
// Apply probe data if available.
if probe != nil {
applyProbeData(&mf, probe, probeSource)
}
if mf.SubtitleTracks == nil {
mf.SubtitleTracks = []models.SubtitleTrack{}
}
// Convert external subtitles to model type.
modelExternalSubs := make([]models.ExternalSubtitle, len(externalSubs))
for i, es := range externalSubs {
modelExternalSubs[i] = models.ExternalSubtitle{
Path: es.Path,
Language: es.Language,
Format: es.Format,
Title: es.Title,
Forced: es.Forced,
}
}
mf.ExternalSubtitles = modelExternalSubs
if mf.ExternalSubtitles == nil {
mf.ExternalSubtitles = []models.ExternalSubtitle{}
}
// Upsert into DB.
upserted, upsertErr := s.fileRepo.Upsert(ctx, mf)
if upsertErr != nil {
return 0, nil, fmt.Errorf("upserting file %s: %w", filePath, upsertErr)
}
if err := s.enqueueMetadataWork(ctx, folder, upserted); err != nil {
return 0, nil, fmt.Errorf("enqueueing metadata work for file %s: %w", filePath, err)
}
if markers != nil {
applied, markerErr := s.fileRepo.UpsertMarkers(ctx, upserted.ID, MarkerUpdate{
IntroStart: markers.IntroStart,
IntroEnd: markers.IntroEnd,
CreditsStart: markers.CreditsStart,
CreditsEnd: markers.CreditsEnd,
MarkersSource: models.MarkerSourceS3,
})
if markerErr != nil {
return 0, nil, fmt.Errorf("upserting markers for file %s: %w", filePath, markerErr)
}
if !applied {
slog.Debug("scanner: skipped lower-priority s3 markers", "path", filePath, "hash", fileHash)
}
}
return action, nil, nil
}
func scanStateUpdateReasons(
existing *scanStateFile,
fileSize int64,
fileModifiedAt time.Time,
currentExternalSubtitlePaths []string,
externalSubtitlesChecked bool,
assignment fileRootAssignment,
groupAssignment fileGroupAssignment,
libraryType string,
canRepairProbe bool,
) []string {
if existing == nil {
return nil
}
reasons := make([]string, 0, 7)
if existing.FileSize != fileSize {
reasons = append(reasons, "size_changed")
}
if !sameFileModifiedAt(existing.FileModifiedAt, fileModifiedAt) {
reasons = append(reasons, "mtime_changed")
}
if existing.MissingSince != nil {
reasons = append(reasons, "was_missing")
}
if canRepairProbe && needsCriticalProbeRepairScanState(existing) {
reasons = append(reasons, "probe_repair")
}
if externalSubtitlesChecked {
if !sameStringSet(existing.ExternalSubtitlePaths, currentExternalSubtitlePaths) {
reasons = append(reasons, "external_subtitle_changed")
}
} else if hasMissingExternalSubtitlePath(existing.ExternalSubtitlePaths) {
reasons = append(reasons, "external_subtitle_missing")
}
if scanStateRootAssignmentChanged(existing, assignment, libraryType) {
reasons = append(reasons, "root_assignment_changed")
}
if scanStateGroupAssignmentChanged(existing, groupAssignment) {
reasons = append(reasons, "group_assignment_changed")
}
return reasons
}
func externalSubtitleInfoPaths(subtitles []ExternalSubtitleInfo) []string {
paths := make([]string, 0, len(subtitles))
for _, subtitle := range subtitles {
path := strings.TrimSpace(subtitle.Path)
if path == "" {
continue
}
paths = append(paths, filepath.Clean(path))
}
return paths
}
func sameStringSet(left, right []string) bool {
seen := make(map[string]int, len(left))
for _, value := range left {
value = filepath.Clean(strings.TrimSpace(value))
if value == "." || value == "" {
continue
}
seen[value]++
}
for _, value := range right {
value = filepath.Clean(strings.TrimSpace(value))
if value == "." || value == "" {
continue
}
if seen[value] == 0 {
return false
}
seen[value]--
}
for _, count := range seen {
if count != 0 {
return false
}
}
return true
}
func hasMissingExternalSubtitlePath(paths []string) bool {
for _, path := range paths {
if strings.TrimSpace(path) == "" {
continue
}
if _, err := os.Stat(path); errors.Is(err, os.ErrNotExist) {
return true
}
}
return false
}
func shouldSkipStableConfirmedScanState(
existing *scanStateFile,
itemStatus string,
fileSize int64,
fileModifiedAt time.Time,
updateReasons []string,
canRepairProbe bool,
) bool {
if existing == nil || existing.ContentID == "" {
return false
}
if !strings.EqualFold(strings.TrimSpace(itemStatus), "matched") {
return false
}
if existing.FileSize != fileSize {
return false
}
if !sameFileModifiedAt(existing.FileModifiedAt, fileModifiedAt) {
return false
}
if existing.MissingSince != nil {
return false
}
if canRepairProbe && needsCriticalProbeRepairScanState(existing) {
return false
}
if len(updateReasons) > 0 {
return false
}
return true
}
func scannerUpdateReasons(
existing *models.MediaFile,
fileSize int64,
fileModifiedAt time.Time,
assignment fileRootAssignment,
groupAssignment fileGroupAssignment,
libraryType string,
canRepairProbe bool,
) []string {
if existing == nil {
return nil
}
reasons := make([]string, 0, 6)
if existing.FileSize != fileSize {
reasons = append(reasons, "size_changed")
}
if !sameFileModifiedAt(existing.FileModifiedAt, fileModifiedAt) {
reasons = append(reasons, "mtime_changed")
}
if existing.MissingSince != nil {
reasons = append(reasons, "was_missing")
}
if canRepairProbe && NeedsCriticalProbeRepair(existing) {
reasons = append(reasons, "probe_repair")
}
if rootAssignmentChanged(existing, assignment, libraryType) {
reasons = append(reasons, "root_assignment_changed")
}
if groupAssignmentChanged(existing, groupAssignment) {
reasons = append(reasons, "group_assignment_changed")
}
return reasons
}
func shouldSkipStableConfirmedFile(
existing *models.MediaFile,
itemStatus string,
fileSize int64,
fileModifiedAt time.Time,
updateReasons []string,
canRepairProbe bool,
) bool {
if existing == nil || existing.ContentID == "" {
return false
}
if !strings.EqualFold(strings.TrimSpace(itemStatus), "matched") {
return false
}
if existing.FileSize != fileSize {
return false
}
if !sameFileModifiedAt(existing.FileModifiedAt, fileModifiedAt) {
return false
}
if existing.MissingSince != nil {
return false
}
if canRepairProbe && NeedsCriticalProbeRepair(existing) {
return false
}
if len(updateReasons) > 0 {
return false
}
return true
}
func sameFileModifiedAt(existing *time.Time, current time.Time) bool {
if existing == nil {
return false
}
return normalizeFileModifiedAt(*existing).Equal(normalizeFileModifiedAt(current))
}
func normalizeFileModifiedAt(ts time.Time) time.Time {
return ts.UTC().Truncate(time.Microsecond)
}
func needsCriticalProbeRepairScanState(file *scanStateFile) bool {
if file == nil {
return true
}
if strings.TrimSpace(file.ProbeSource) == "" || file.ProbeUpdatedAt == nil {
return true
}
if file.Duration <= 0 {
return true
}
if strings.TrimSpace(file.Container) == "" {
return true
}
if strings.TrimSpace(file.CodecVideo) == "" {
return true
}
if strings.TrimSpace(file.CodecAudio) == "" {
return true
}
if strings.TrimSpace(file.Resolution) == "" {
return true
}
if !file.HasVideoTracks {
return true
}
if !file.HasAudioTracks {
return true
}
if !file.HasChapters {
return true
}
return false
}
func (s *Scanner) enqueueMetadataWork(ctx context.Context, folder *models.MediaFolder, file *models.MediaFile) error {
if s == nil || s.metadataQueue == nil || folder == nil || file == nil {
return nil
}
switch strings.ToLower(strings.TrimSpace(folder.Type)) {
case "movie", "movies":
if err := s.metadataQueue.EnqueueMovieFile(ctx, file.ID); err != nil {
return err
}
slog.Debug("metadata queue: movie file enqueued",
"folder_id", folder.ID,
"file_id", file.ID,
"path", file.FilePath,
)
case "series", "tv", "show", "tvshows":
if strings.TrimSpace(file.ObservedRootPath) == "" {
return nil
}
if err := s.metadataQueue.EnqueueSeriesRoot(ctx, folder.ID, file.ObservedRootPath); err != nil {
return err
}
slog.Debug("metadata queue: series root touched",
"folder_id", folder.ID,
"observed_root_path", file.ObservedRootPath,
"file_id", file.ID,
)
}
return nil
}
func (s *Scanner) reconcileScannedGroups(
ctx context.Context,
folderID int,
fullFolderScan bool,
scopeRoots []string,
deleteMissingInScope bool,
groups groupInferenceResult,
) error {
if s == nil {
return nil
}
if s.groupSnapshotRepo != nil {
if fullFolderScan {
if err := s.groupSnapshotRepo.ReplaceForFolder(ctx, folderID, groups.ScannedGroups); err != nil {
return err
}
} else if deleteMissingInScope {
for _, scope := range scopeRoots {
if err := s.groupSnapshotRepo.ReplaceInScope(ctx, folderID, scope, filterScannedGroupsByScope(groups.ScannedGroups, scope)); err != nil {
return err
}
}
} else {
if err := s.groupSnapshotRepo.UpsertMany(ctx, groups.ScannedGroups); err != nil {
return err
}
}
}
if s.locationRepo != nil {
if fullFolderScan {
if err := s.locationRepo.ReplaceForFolder(ctx, folderID, groups.Locations); err != nil {
return err
}
} else if deleteMissingInScope {
for _, scope := range scopeRoots {
if err := s.locationRepo.ReplaceInScope(ctx, folderID, scope, filterObservedLocationsByScope(groups.Locations, scope)); err != nil {
return err
}
}
} else {
if err := s.locationRepo.UpsertMany(ctx, groups.Locations); err != nil {
return err
}
}
}
if s.groupLocationRepo != nil {
if fullFolderScan {
if err := s.groupLocationRepo.ReplaceForFolder(ctx, folderID, groups.GroupLocations); err != nil {
return err
}
} else if deleteMissingInScope {
for _, scope := range scopeRoots {
if err := s.groupLocationRepo.ReplaceInScope(ctx, folderID, scope, filterGroupLocationsByScope(groups.GroupLocations, scope)); err != nil {
return err
}
}
} else {
if err := s.groupLocationRepo.UpsertMany(ctx, groups.GroupLocations); err != nil {
return err
}
}
}
return nil
}
func filterScannedGroupsByScope(groups []models.ScannedMediaGroup, scopePath string) []models.ScannedMediaGroup {
filtered := make([]models.ScannedMediaGroup, 0, len(groups))
for _, group := range groups {
if pathWithinAnyRoot(group.SampleObservedRootPath, []string{scopePath}) {
filtered = append(filtered, group)
}
}
return filtered
}
func filterObservedLocationsByScope(locations []models.ObservedMediaLocation, scopePath string) []models.ObservedMediaLocation {
filtered := make([]models.ObservedMediaLocation, 0, len(locations))
for _, location := range locations {
if pathWithinAnyRoot(location.ObservedRootPath, []string{scopePath}) {
filtered = append(filtered, location)
}
}
return filtered
}
func filterGroupLocationsByScope(locations []models.MediaGroupLocation, scopePath string) []models.MediaGroupLocation {
filtered := make([]models.MediaGroupLocation, 0, len(locations))
for _, location := range locations {
if pathWithinAnyRoot(location.ObservedRootPath, []string{scopePath}) {
filtered = append(filtered, location)
}
}
return filtered
}
func (s *Scanner) reconcileScannedRoots(
ctx context.Context,
folderID int,
scopeRoots []string,
roots []models.ScannedMediaRoot,
) error {
if s == nil || s.rootSnapshotRepo == nil {
return nil
}
seenByScope := make(map[string][]string, len(scopeRoots))
for _, scope := range scopeRoots {
seenByScope[scope] = []string{}
}
for _, root := range roots {
for scope := range seenByScope {
if pathWithinAnyRoot(root.RootPath, []string{scope}) {
seenByScope[scope] = append(seenByScope[scope], root.RootPath)
}
}
}
if err := s.rootSnapshotRepo.UpsertMany(ctx, roots); err != nil {
return err
}
for scope, seenRoots := range seenByScope {
if err := s.rootSnapshotRepo.DeleteMissingInScope(ctx, folderID, scope, seenRoots); err != nil {
return err
}
}
return nil
}
func (s *Scanner) loadRootOverrides(
ctx context.Context,
folderID int,
scopeRoots []string,
) (map[string]models.MediaRootOverride, error) {
overridesByRoot := map[string]models.MediaRootOverride{}
if s == nil || s.rootOverrideRepo == nil || folderID <= 0 {
return overridesByRoot, nil
}
overrides, err := s.rootOverrideRepo.ListByFolder(ctx, folderID)
if err != nil {
return nil, err
}
for _, override := range overrides {
if len(scopeRoots) > 0 && !pathWithinAnyRoot(override.RootPath, scopeRoots) {
continue
}
overridesByRoot[filepath.Clean(override.RootPath)] = override
}
return overridesByRoot, nil
}
func (s *Scanner) loadGroupOverrides(
ctx context.Context,
folderID int,
) (map[string]models.MediaGroupOverride, error) {
overridesByKey := map[string]models.MediaGroupOverride{}
if s == nil || s.groupOverrideRepo == nil || folderID <= 0 {
return overridesByKey, nil
}
overrides, err := s.groupOverrideRepo.ListByFolder(ctx, folderID)
if err != nil {
return nil, err
}
for _, override := range overrides {
overridesByKey[groupOverrideKey(override.GroupKeyVersion, override.ContentGroupKey)] = override
}
return overridesByKey, nil
}
func (s *Scanner) logRootInferenceDisagreements(assignments map[string]fileRootAssignment) {
for _, assignment := range assignments {
if assignment.LegacyRootPath == "" {
continue
}
if assignment.LegacyRootPath == assignment.RootPath && assignment.LegacyType == assignment.InferredType {
continue
}
slog.Info("scanner: root inference disagreement",
"file_path", assignment.FilePath,
"legacy_root_path", assignment.LegacyRootPath,
"inferred_root_path", assignment.RootPath,
"legacy_type", assignment.LegacyType,
"inferred_type", assignment.InferredType,
"wrapper_collapsed", assignment.WrapperCollapsed,
"promoted_ancestor", assignment.PromotedAncestor,
)
}
}
func scanStateRootAssignmentChanged(existing *scanStateFile, assignment fileRootAssignment, libraryType string) bool {
if existing == nil {
return true
}
expectedRoot := assignment.RootPath
if expectedRoot == "" {
if root, ok := naming.DetectCanonicalRoot(existing.FilePath, libraryType); ok {
expectedRoot = filepath.Clean(root.RootPath)
}
}
if filepath.Clean(existing.CanonicalRootPath) != filepath.Clean(expectedRoot) {
return true
}
hints := naming.ParseVariantHints(existing.FilePath, libraryType)
if existing.EditionSource == "import" && existing.EditionKey != "" {
hints = &naming.VariantHints{
EditionRaw: existing.EditionRaw,
EditionKey: existing.EditionKey,
EditionSource: existing.EditionSource,
EditionConfidence: existing.EditionConfidence,
PresentationKind: existing.PresentationKind,
PresentationGroupKey: existing.PresentationGroupKey,
PresentationPartIndex: existing.PresentationPartIndex,
MultiEpisodeStart: existing.MultiEpisodeStart,
MultiEpisodeEnd: existing.MultiEpisodeEnd,
}
}
if hints == nil {
hints = &naming.VariantHints{}
}
if existing.EditionRaw != hints.EditionRaw ||
existing.EditionKey != hints.EditionKey ||
existing.EditionSource != hints.EditionSource ||
existing.PresentationKind != hints.PresentationKind ||
existing.PresentationGroupKey != hints.PresentationGroupKey ||
existing.PresentationPartIndex != hints.PresentationPartIndex ||
existing.MultiEpisodeStart != hints.MultiEpisodeStart ||
existing.MultiEpisodeEnd != hints.MultiEpisodeEnd {
return true
}
switch {
case existing.EditionConfidence == nil && hints.EditionConfidence == nil:
return false
case existing.EditionConfidence == nil || hints.EditionConfidence == nil:
return true
default:
return *existing.EditionConfidence != *hints.EditionConfidence
}
}
func scanStateGroupAssignmentChanged(existing *scanStateFile, assignment fileGroupAssignment) bool {
if existing == nil {
return true
}
if filepath.Clean(existing.ObservedRootPath) != filepath.Clean(assignment.ObservedRootPath) {
return true
}
if existing.ContentGroupKey != assignment.ContentGroupKey ||
existing.GroupKeyVersion != assignment.GroupKeyVersion ||
existing.BaseTitle != assignment.BaseTitle ||
existing.BaseYear != assignment.BaseYear ||
existing.BaseType != assignment.BaseType ||
existing.IdentityConfidence != assignment.Confidence {
return true
}
return !identityEvidenceEqual(existing.IdentityJSON, assignment.EvidenceJSON)
}
func rootAssignmentChanged(existing *models.MediaFile, assignment fileRootAssignment, libraryType string) bool {
if existing == nil {
return true
}
expectedRoot := assignment.RootPath
if expectedRoot == "" {
if root, ok := naming.DetectCanonicalRoot(existing.FilePath, libraryType); ok {
expectedRoot = filepath.Clean(root.RootPath)
}
}
if filepath.Clean(existing.CanonicalRootPath) != filepath.Clean(expectedRoot) {
return true
}
hints := naming.ParseVariantHints(existing.FilePath, libraryType)
if existing.EditionSource == "import" && existing.EditionKey != "" {
hints = &naming.VariantHints{
EditionRaw: existing.EditionRaw,
EditionKey: existing.EditionKey,
EditionSource: existing.EditionSource,
EditionConfidence: existing.EditionConfidence,
PresentationKind: existing.PresentationKind,
PresentationGroupKey: existing.PresentationGroupKey,
PresentationPartIndex: existing.PresentationPartIndex,
MultiEpisodeStart: existing.MultiEpisodeStart,
MultiEpisodeEnd: existing.MultiEpisodeEnd,
}
}
if hints == nil {
hints = &naming.VariantHints{}
}
if existing.EditionRaw != hints.EditionRaw ||
existing.EditionKey != hints.EditionKey ||
existing.EditionSource != hints.EditionSource ||
existing.PresentationKind != hints.PresentationKind ||
existing.PresentationGroupKey != hints.PresentationGroupKey ||
existing.PresentationPartIndex != hints.PresentationPartIndex ||
existing.MultiEpisodeStart != hints.MultiEpisodeStart ||
existing.MultiEpisodeEnd != hints.MultiEpisodeEnd {
return true
}
switch {
case existing.EditionConfidence == nil && hints.EditionConfidence == nil:
return false
case existing.EditionConfidence == nil || hints.EditionConfidence == nil:
return true
default:
return *existing.EditionConfidence != *hints.EditionConfidence
}
}
func groupAssignmentChanged(existing *models.MediaFile, assignment fileGroupAssignment) bool {
if existing == nil {
return true
}
if filepath.Clean(existing.ObservedRootPath) != filepath.Clean(assignment.ObservedRootPath) {
return true
}
if existing.ContentGroupKey != assignment.ContentGroupKey ||
existing.GroupKeyVersion != assignment.GroupKeyVersion ||
existing.BaseTitle != assignment.BaseTitle ||
existing.BaseYear != assignment.BaseYear ||
existing.BaseType != assignment.BaseType ||
existing.IdentityConfidence != assignment.Confidence {
return true
}
return !identityEvidenceEqual(existing.IdentityJSON, assignment.EvidenceJSON)
}
func identityEvidenceEqual(existing, expected []byte) bool {
if bytes.Equal(existing, expected) {
return true
}
if len(bytes.TrimSpace(existing)) == 0 || len(bytes.TrimSpace(expected)) == 0 {
return len(bytes.TrimSpace(existing)) == 0 && len(bytes.TrimSpace(expected)) == 0
}
var existingValue any
if err := json.Unmarshal(existing, &existingValue); err != nil {
return false
}
var expectedValue any
if err := json.Unmarshal(expected, &expectedValue); err != nil {
return false
}
return reflect.DeepEqual(existingValue, expectedValue)
}
func applyProbeData(mf *models.MediaFile, probe *ProbeData, probeSource string) {
mf.CodecVideo = probe.CodecVideo
mf.CodecAudio = probe.CodecAudio
mf.Resolution = probe.Resolution
mf.AudioChannels = probe.AudioChannels
mf.HDR = probe.HDR
mf.Container = probe.Container
mf.Duration = probe.Duration
mf.Bitrate = probe.Bitrate
mf.ProbeSource = probeSource
now := time.Now().UTC()
mf.ProbeUpdatedAt = &now
videoTracks := make([]models.VideoTrack, len(probe.VideoTracks))
for i, vt := range probe.VideoTracks {
videoTracks[i] = models.VideoTrack{
Title: vt.Title,
Codec: vt.Codec,
DolbyVision: vt.DolbyVision,
Profile: vt.Profile,
Level: vt.Level,
Width: vt.Width,
Height: vt.Height,
AspectRatio: vt.AspectRatio,
Interlaced: vt.Interlaced,
FrameRate: vt.FrameRate,
Bitrate: vt.Bitrate,
VideoRange: vt.VideoRange,
ColorPrimaries: vt.ColorPrimaries,
ColorSpace: vt.ColorSpace,
ColorTransfer: vt.ColorTransfer,
BitDepth: vt.BitDepth,
PixelFormat: vt.PixelFormat,
ReferenceFrames: vt.ReferenceFrames,
}
}
mf.VideoTracks = videoTracks
audioTracks := make([]models.AudioTrack, len(probe.AudioTracks))
for i, at := range probe.AudioTracks {
audioTracks[i] = models.AudioTrack{
Title: at.Title,
EmbeddedTitle: at.EmbeddedTitle,
Language: at.Language,
Codec: at.Codec,
Layout: at.Layout,
Channels: at.Channels,
Bitrate: at.Bitrate,
SampleRate: at.SampleRate,
BitDepth: at.BitDepth,
Default: at.Default,
}
}
mf.AudioTracks = audioTracks
subtitleTracks := make([]models.SubtitleTrack, len(probe.SubtitleTracks))
for i, st := range probe.SubtitleTracks {
subtitleTracks[i] = models.SubtitleTrack{
Index: st.Index,
Language: st.Language,
Codec: st.Codec,
Title: st.Title,
EmbeddedTitle: st.EmbeddedTitle,
Resolution: st.Resolution,
Forced: st.Forced,
Default: st.Default,
HearingImpaired: st.HearingImpaired,
}
}
mf.SubtitleTracks = subtitleTracks
chapters := make([]models.MediaChapter, len(probe.Chapters))
for i, chapter := range probe.Chapters {
chapters[i] = models.MediaChapter{
Index: chapter.Index,
Title: chapter.Title,
StartSeconds: chapter.StartSeconds,
EndSeconds: chapter.EndSeconds,
Source: chapter.Source,
}
}
mf.Chapters = chapters
}
// clearLegacyLinksForUnmatchableRoots was previously used to clear content
// links for files under roots without embedded folder IDs. With the library
// matching redesign, roots without folder IDs are now valid and create
// pending items, so this function is a no-op.
// TODO: remove callers and delete this function
func (s *Scanner) clearLegacyLinksForUnmatchableRoots(_ context.Context, _ int, _ []RootObservation) (int, error) {
return 0, nil
}
// gatherHints computes the OSHash for a media file.
func (s *Scanner) gatherHints(filePath string) FileHints {
hints := FileHints{}
hash, err := ComputeOSHash(filePath)
if err != nil {
slog.Warn("scanner: OSHash computation failed", "path", filePath, "error", err)
} else {
hints.FileHash = hash
}
return hints
}
// probeFile attempts to get probe data by running local ffprobe.
func (s *Scanner) probeFile(ctx context.Context, filePath string) (*ProbeData, string) {
if s.ffprobePath != "" {
probe, err := ProbeFile(ctx, s.ffprobePath, filePath)
if err != nil {
slog.Warn("scanner: ffprobe failed", "path", filePath, "error", err)
return nil, "local"
}
return probe, "local"
}
return nil, "local"
}
// fetchMarkers checks S3 for intro/credits markers for the given file hash.
func (s *Scanner) fetchMarkers(ctx context.Context, fileHash string) *IntroCreditsMarkers {
if fileHash == "" || s.s3Client == nil {
return nil
}
key := fmt.Sprintf("markers/%s.json", fileHash)
data, err := s.s3Client.GetObject(ctx, s.s3Client.Bucket(), key)
if err != nil {
// Not found is expected; don't log it.
return nil
}
var markers IntroCreditsMarkers
if err := json.Unmarshal(data, &markers); err != nil {
slog.Warn("scanner: markers JSON parse failed",
"hash", fileHash,
"error", err,
)
return nil
}
return &markers
}