* fix(scanner): stop hiding media when a library root is offline
A scan that cannot read a library root found no files there, so every
cataloged file under it was marked missing. Catalog reads all filter on
missing_since IS NULL, so marking is equivalent to deletion from a user's
point of view: the title leaves browse, search and next-up, and playback
answers "Source media file is missing" for media that is intact on disk.
The dead-root protection already existed but only guarded the destructive
operations. protectedConfiguredRoots was computed *after* the marking loop
in scanPaths, and applyScopedScan received the protected set but applied it
only to its force-delete branch. So an unreachable root could not lose its
rows, but could still have its entire catalog hidden until the next
successful scan.
On a CephFS deployment whose per-library subvolume mounts flap, this marked
190 present files missing in a single day — 15% of all missing-flagged rows
were files sitting untouched on disk, some flagged more than 20 hours after
their last write.
Hoist the probe above the marking loop and skip files under an unreachable
or suspect-empty root in both the folder and scoped paths. Pass the
unreachable set to the walked-scope call too: a nested child mount can die
under a healthy parent, and its rows are inside the parent's scope.
An offline root tells us nothing about whether its files exist. The only
safe reading is to leave them alone and let the next good scan decide.
Genuine deletions under a reachable root are unaffected and still marked
and swept on the same schedule as before.
Report the count as ScanResult.MissingSkippedProtected and log it, so an
operator can tell "my library shrank" from "my mount dropped".
Known gap: a suspect-empty *nested child* root under a healthy parent is
still marked missing, because the suspect set is not resolved until after
the walk loop. Unreachable roots — the case observed in production — are
covered.
* fix(scanner): protect suspect-empty and partially-walked roots too
Addresses review findings on #472. The original change guarded missing-marking
against probe-unreachable roots, but left three ways for a storage fault to
still hide a healthy library.
Suspect-empty detection was reactive. suspectEmptyRoots asked
ListRootsWithOnlyMissingFiles, which returns a root only once it has NO live
rows left. On the first scan after a mount drops — the moment that matters —
the rows are still live, so the root was not classified suspect and the scan
marked everything missing. The protection then engaged on the next scan, in
time to protect the wreckage. Ask ListRootsWithCatalogedFiles instead: any
cataloged row under an empty-but-reachable root is the lost-mount signature.
Intentional emptying is still reachable through the operator's one-time
cleanup allowance, which is the deliberate path for it.
Nested suspect-empty children were unprotected. Root compaction sends only the
populated parent through the walked-scope branch, which received only
unreachableRoots, so an empty child mountpoint had its rows marked missing on
its parent scanning cleanly. Pass the suspect set as well.
Partial walks were treated as authoritative. walkLogicalTree deliberately
swallows per-entry Lstat/ReadDir failures so one bad file cannot abort a scan
of a million, and collectLogicalFilePaths passed nil for the failure counter —
so the video path had no signal at all. A mount dying partway through
traversal produced a short file list indistinguishable from a large deletion.
Thread the counter through, and exclude a scope whose walk came back
incomplete from missing reconciliation, mirroring what the ebook scanner
already does via ebookRootScan.failed.
Also extract the duplicated mark-missing loop into markMissingExcludingProtected
so the folder and scoped paths cannot drift, and correct two comments that
still described the pre-fix "files are marked missing" behaviour — the exact
text a future reader would have trusted when reintroducing this bug.
TestScanFolderNestedSuspectEmptyChildRootProtection asserted the old
behaviour and is updated accordingly.
* fix(scanner): scope walk-failure protection and stop pruning on partial walks
Addresses the second Codex review round on #472. The previous commit's
incomplete-walk protection was too blunt in one direction and applied too late
in another.
Walk failures were counted, not located, and any non-zero count protected the
whole library root. A dangling symlink is both common and permanent, so that
would have suppressed missing-file reconciliation for its entire root on every
future scan — genuinely deleted titles would stay live indefinitely. That is
the same class of bug as the one this PR fixes, pointing the other way.
recordWalkFailure now records the logical path of each unreadable entry, and
only those paths are protected. Per-entry failures record the child path, so a
dangling symlink protects itself and nothing else, while a directory that
cannot be read protects its subtree.
Snapshot and group pruning ran before the protection. reconcileScannedRoots
and reconcileScannedGroups delete whatever the walk did not see, and both run
ahead of the missing-file guard, so a partial walk still dropped root
snapshots, observed locations and group locations for the unread portion —
corrupting later metadata matching even though the media_files rows survived.
Upserting what was seen is always safe; pruning now waits for a scan that read
the whole tree.
The confirmed-cleanup allowance was consumed to no effect for nested suspect
children. The walked-parent branch protected them unconditionally and runs
before the allowance is consumed, and an already-reconciled scope cannot be
revisited — so arming the allowance burned the confirmation while the child's
rows stayed live forever. Read the allowance without consuming it before the
walk loop, and honour it there. Unreachable roots stay protected either way:
an outage is never a confirmation to erase a catalog.
Two new regression tests, plus signature updates in the ebook pipeline, which
already tracked walk failures and now shares the path-based representation.
* fix(scanner): re-probe nested roots and gate group pruning on walk completeness
Third Codex review round on #472; both findings confirmed.
Group pruning ignored walk completeness in the subtree path. scanPaths passed
the completeness decision to reconcileScannedRoots but left
reconcileScannedGroups on !allowEmptyRootGuard, which is always true for
ScanSubtree — so a subtree scan that hit an unreadable directory still replaced
group snapshots and locations from a partial inventory. Same rule now applies
to both.
Nested roots were not re-probed before their parent was reconciled. Root
compaction folds a child mount into its parent for traversal, so a child that
is healthy at the initial probe but drops before the parent is walked leaves no
scope of its own, and the post-walk re-probe only revisits scopes that walked
empty. The parent walks files, looks healthy, and the child's rows are marked
missing on its success. reprobeNestedRoots re-checks this root's configured
children immediately before reconciling, protecting any that have since become
unreachable — or suspect-empty, unless the operator has confirmed cleanup.
Also guard suspectEmptyRoots against a nil file repository, matching
emptyCleanupArmed: without a catalog there is nothing to protect.
* fix(scanner): keep re-probed outages protected through folder-wide cleanup
Fourth Codex review round on #472; both findings confirmed. The first could
destroy data.
reprobeNestedRoots protected a root it found offline only for the scope being
reconciled, then discarded the result. The folder-wide membership reconcile and
the trash sweep afterwards rebuilt their protected set from the initial probe
alone, so rows under a child that dropped mid-scan — already marked missing and
past the removal grace — were hard-deleted by the very scan that noticed the
outage. Accumulate those roots in reprobedRoots, fold them into
protectedScanRoots, and reuse that set for the membership reconcile and sweep
instead of rebuilding. They now also land in ScanResult.UnreachableRoots so the
folder warning reflects the outage rather than presenting a partial scan as
clean.
Snapshot and group pruning was enabled for scopes that were never walked. The
gate was len(walkFailures) == 0, but an unreachable root gets nil walkRoots, so
it has no walk and therefore no failures — and pruning then deleted its
snapshots, observed locations and group locations even though its media rows
were protected. The same held for a suspect-empty child compacted into a
populated parent. Pruning now additionally requires that the scope was actually
walked and contains no protected path.
The new test pins that the sweep honours the protected set it is given. It does
not reproduce the mid-scan race itself: staging that needs the drop to land
between the probe and the walk, which a test cannot reach without hooks. That
path is covered by inspection, and the test comment says so rather than
implying coverage it does not have.
* fix(scanner): route every protection source through one folder-wide set
Fifth Codex review round on #472. Two P1s, one of them the second data-loss
path in this area — and the direct sibling of the one fixed in 35326adc, which
is the reason this commit changes the structure rather than patching another
edge.
Rows beneath a directory the walk could not read were protected only inside
applyScopedScan. The folder-wide protected set was rebuilt from the probe
results alone, so DeleteMissingByFolder could permanently delete rows past the
removal grace under a subtree this scan never managed to read — deleting on the
strength of an observation that was never made.
The recurring defect is structural: protection is discovered in several places
(initial probe, mid-loop re-probe, per-scope walk failures) and consumed in
several more (scoped reconcile, membership reconcile, trash sweep), and each
fix so far has wired up one edge and missed another. Every source now
accumulates folder-wide and every consumer reads the combined set, so a new
source has one place to register instead of several to remember.
reprobeNestedRoots classified from two probe batches. It called
probeUnreachableRoots, then suspectEmptyRoots probed the same paths again; a
child dropping between the samples was reachable to the first and discarded by
the second, which only returns reachable-and-empty roots. It now classifies
both states from one batch, so the disconnect it exists to catch cannot fall
between its own probes.
Re-probed roots kept their classification instead of being collapsed into
unreachableRoots, which had been reporting a suspect-empty child as
unreachable and giving operators contradictory failure information.
The new regression test is verified to fail with the propagation disabled and
pass with it, rather than assumed to cover the path.
Not addressed: the cleanup allowance is read without being reserved, so two
overlapping full scans of one folder can both observe it armed. Narrow, needs
a transactional reserve in the scan-claim query, and is left for follow-up
rather than bundled here.
* fix(scanner): resolve root protection before scoped metadata pruning
Sixth Codex review round on #472.
scanPaths pruned before it knew what was protected. reconcileScannedRoots and
reconcileScannedGroups ran roughly 160 lines ahead of protectedConfiguredRoots,
so a ScanSubtree of a mount that dropped but left a reachable empty mountpoint
walked clean, reported no failures, and pruned root snapshots and observed and
group locations against that empty inventory — preserving the media rows while
deleting the metadata describing them. Protection is now resolved before any
reconciliation, and both prunes share one decision, matching applyScopedScan.
Pending empty scopes never re-probed their nested children. A parent whose only
media lives in a child walks empty when that child drops, so it lands in
pendingEmptyScopes rather than the populated-scope branch where
reprobeNestedRoots ran. Probing the parent alone proves nothing: it still holds
the child's bare mountpoint directory, so it reads present and non-empty. With
a healthy sibling keeping the folder-wide empty guard quiet, nothing protected
the child. Both branches now re-probe.
MissingSkippedProtected never left the scanner. Both ingest-to-result
conversions copied every other cleanup count but not this one, and
events.ScanRunResult had no field, so scan history, completion events and API
responses reported an all-zero no-op for a scan that skipped files because
storage was offline. Added as a new field, which is additive under the v1 API
rules.
Test honesty: the new test does NOT exercise the pending-scope re-probe. It
empties the child before the scan, so the initial probe classifies it and
protection arrives by that path — verified by confirming the test still passes
with the re-probe disabled. It is named and commented for what it does cover.
The mid-scan race behind both re-probe fixes needs the drop to land between the
probe and the walk, which is not reachable from a test without hooks; those
fixes rest on inspection.
1582 lines
50 KiB
Go
1582 lines
50 KiB
Go
package scanner
|
|
|
|
import (
|
|
"context"
|
|
"errors"
|
|
"fmt"
|
|
"io"
|
|
"log/slog"
|
|
"os"
|
|
"path/filepath"
|
|
"sort"
|
|
"strconv"
|
|
"strings"
|
|
"sync"
|
|
"sync/atomic"
|
|
"time"
|
|
"unicode"
|
|
|
|
"github.com/Silo-Server/silo-server/internal/idgen"
|
|
"github.com/Silo-Server/silo-server/internal/imageutil"
|
|
"github.com/Silo-Server/silo-server/internal/models"
|
|
"github.com/Silo-Server/silo-server/internal/rootcheck"
|
|
"github.com/Silo-Server/silo-server/internal/titleutil"
|
|
"github.com/jackc/pgx/v5"
|
|
"github.com/jackc/pgx/v5/pgconn"
|
|
)
|
|
|
|
type ebookSQLExecutor interface {
|
|
Exec(ctx context.Context, sql string, arguments ...any) (pgconn.CommandTag, error)
|
|
}
|
|
|
|
type filesystemMediaItemReader interface {
|
|
GetByIDs(ctx context.Context, ids []string) ([]*models.MediaItem, error)
|
|
}
|
|
|
|
func ebookScanWorkers() int {
|
|
if v := os.Getenv("SILO_EBOOK_SCAN_WORKERS"); v != "" {
|
|
if n, err := strconv.Atoi(v); err == nil && n > 0 {
|
|
return n
|
|
}
|
|
}
|
|
return audiobookScanWorkers()
|
|
}
|
|
|
|
func (s *Scanner) ScanEbookFolder(ctx context.Context, folder *models.MediaFolder) error {
|
|
if s == nil || folder == nil {
|
|
return fmt.Errorf("ScanEbookFolder: nil scanner or folder")
|
|
}
|
|
return s.scanEbookPaths(ctx, folder, folder.Paths, true)
|
|
}
|
|
|
|
// ebookRootScan captures the walk outcome for one configured library root.
|
|
type ebookRootScan struct {
|
|
root string
|
|
files []string
|
|
fileOnly bool // explicit single-file scan; index it but never reconcile a subtree
|
|
rootErr error // root stat failed, or the root is not a directory
|
|
walkFailures []string // logical paths within the subtree the walk could not read or resolve
|
|
}
|
|
|
|
// failed reports whether the walk under this root is known to be incomplete.
|
|
// Files found under a failed root are still indexed, but the root is excluded
|
|
// from missing-file reconciliation: a transient mount/permission problem (or
|
|
// a mid-walk subtree error) must not cascade into marking — and, with
|
|
// empty_trash_after_scan, deleting — everything under it.
|
|
func (r *ebookRootScan) failed() bool {
|
|
return r.rootErr != nil || len(r.walkFailures) > 0
|
|
}
|
|
|
|
// collectEbookRootScans walks every configured root with the shared
|
|
// logical-tree walker (so symlinked roots and symlinked subdirectories
|
|
// resolve and scan exactly like the video pipeline) and records per-root walk
|
|
// failures. The only error returned is context cancellation.
|
|
func collectEbookRootScans(ctx context.Context, folderID int, roots []string) ([]ebookRootScan, error) {
|
|
scans := make([]ebookRootScan, 0, len(roots))
|
|
visitedPhysicalDirs := make(map[string]struct{})
|
|
for _, root := range roots {
|
|
if err := ctx.Err(); err != nil {
|
|
return nil, err
|
|
}
|
|
cleanRoot := filepath.Clean(strings.TrimSpace(root))
|
|
if cleanRoot == "" || cleanRoot == "." {
|
|
continue
|
|
}
|
|
scan := ebookRootScan{root: cleanRoot}
|
|
info, statErr := os.Stat(cleanRoot)
|
|
switch {
|
|
case statErr != nil:
|
|
// Unmounted/missing/permission-broken root: failed, not empty.
|
|
scan.rootErr = fmt.Errorf("stat root: %w", statErr)
|
|
case !info.IsDir():
|
|
if info.Mode().IsRegular() && SupportsEbookFile(cleanRoot) {
|
|
scan.fileOnly = true
|
|
} else {
|
|
scan.rootErr = fmt.Errorf("root is not a directory after symlink resolution")
|
|
}
|
|
}
|
|
if statErr == nil {
|
|
if err := walkLogicalTree(ctx, cleanRoot, cleanRoot, walkModeEbook, visitedPhysicalDirs, &scan.files, &scan.walkFailures); err != nil {
|
|
return nil, err
|
|
}
|
|
}
|
|
if scan.failed() {
|
|
slog.WarnContext(ctx, "ebook scan: root walk incomplete; root excluded from missing-file reconciliation", "component", "scanner",
|
|
"folder_id", folderID,
|
|
"root", cleanRoot,
|
|
"walk_failures", len(scan.walkFailures),
|
|
"error", scan.rootErr,
|
|
)
|
|
}
|
|
scans = append(scans, scan)
|
|
}
|
|
return scans, nil
|
|
}
|
|
|
|
// splitEbookReconcileRoots partitions walked roots into the set that may take
|
|
// part in missing-file reconciliation and reports whether any reconcilable
|
|
// root saw at least one ebook on disk.
|
|
func splitEbookReconcileRoots(scans []ebookRootScan) (reconcileRoots []string, sawFiles bool) {
|
|
reconcileRoots = make([]string, 0, len(scans))
|
|
for i := range scans {
|
|
scan := &scans[i]
|
|
if scan.failed() || scan.fileOnly {
|
|
continue
|
|
}
|
|
if len(scan.files) > 0 {
|
|
sawFiles = true
|
|
}
|
|
reconcileRoots = append(reconcileRoots, scan.root)
|
|
}
|
|
return reconcileRoots, sawFiles
|
|
}
|
|
|
|
func (s *Scanner) scanEbookPaths(ctx context.Context, folder *models.MediaFolder, roots []string, fullScan bool) error {
|
|
if s == nil || folder == nil {
|
|
return fmt.Errorf("scanEbookPaths: nil scanner or folder")
|
|
}
|
|
scans, err := collectEbookRootScans(ctx, folder.ID, roots)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
// Every discovered file is indexed, including files under roots whose
|
|
// walk partially failed: indexing is additive and safe, only the
|
|
// destructive reconciliation below is restricted to cleanly walked roots.
|
|
var candidates []string
|
|
for i := range scans {
|
|
candidates = append(candidates, scans[i].files...)
|
|
}
|
|
|
|
if len(candidates) == 0 {
|
|
return s.reconcileEbookScan(ctx, folder, scans, nil, fullScan)
|
|
}
|
|
|
|
workers := ebookScanWorkers()
|
|
slog.InfoContext(ctx, "ebook scan: starting", "component", "scanner",
|
|
"folder_id", folder.ID,
|
|
"candidates", len(candidates),
|
|
"workers", workers,
|
|
)
|
|
reportEbookScanProgress(ctx, folder.ID, len(candidates), 0, 0, 0)
|
|
|
|
ch := make(chan string, workers*2)
|
|
groupLocks := newEbookGroupLocks()
|
|
var (
|
|
wg sync.WaitGroup
|
|
processed int64
|
|
failed int64
|
|
skipped int64
|
|
failMu sync.Mutex
|
|
failures []error
|
|
cancelErr error
|
|
)
|
|
start := time.Now()
|
|
for i := 0; i < workers; i++ {
|
|
wg.Add(1)
|
|
go func() {
|
|
defer wg.Done()
|
|
for path := range ch {
|
|
if ctx.Err() != nil {
|
|
return
|
|
}
|
|
if err := s.reconcileEbookFile(ctx, folder, path, &skipped, groupLocks); err != nil {
|
|
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
|
|
failMu.Lock()
|
|
if cancelErr == nil {
|
|
cancelErr = err
|
|
}
|
|
failMu.Unlock()
|
|
return
|
|
}
|
|
atomic.AddInt64(&failed, 1)
|
|
failMu.Lock()
|
|
failures = append(failures, fmt.Errorf("%s: %w", path, err))
|
|
failMu.Unlock()
|
|
slog.WarnContext(ctx, "ebook scan: file failed", "component", "scanner",
|
|
"folder_id", folder.ID,
|
|
"path", path,
|
|
"error", err,
|
|
)
|
|
}
|
|
n := atomic.AddInt64(&processed, 1)
|
|
if n%500 == 0 || n == int64(len(candidates)) {
|
|
failedCount := atomic.LoadInt64(&failed)
|
|
skippedCount := atomic.LoadInt64(&skipped)
|
|
slog.InfoContext(ctx, "ebook scan: progress", "component", "scanner",
|
|
"folder_id", folder.ID,
|
|
"processed", n,
|
|
"failed", failedCount,
|
|
"skipped", skippedCount,
|
|
"total", len(candidates),
|
|
"elapsed_sec", int(time.Since(start).Seconds()),
|
|
)
|
|
reportEbookScanProgress(ctx, folder.ID, len(candidates), int(n), int(failedCount), int(skippedCount))
|
|
}
|
|
}
|
|
}()
|
|
}
|
|
|
|
for _, p := range candidates {
|
|
select {
|
|
case ch <- p:
|
|
case <-ctx.Done():
|
|
close(ch)
|
|
wg.Wait()
|
|
return ctx.Err()
|
|
}
|
|
}
|
|
close(ch)
|
|
wg.Wait()
|
|
if err := ctx.Err(); err != nil {
|
|
return err
|
|
}
|
|
if cancelErr != nil {
|
|
return cancelErr
|
|
}
|
|
|
|
slog.InfoContext(ctx, "ebook scan: completed", "component", "scanner",
|
|
"folder_id", folder.ID,
|
|
"processed", atomic.LoadInt64(&processed),
|
|
"failed", atomic.LoadInt64(&failed),
|
|
"skipped", atomic.LoadInt64(&skipped),
|
|
"elapsed_sec", int(time.Since(start).Seconds()),
|
|
)
|
|
if processedCount := atomic.LoadInt64(&processed); processedCount > 0 {
|
|
failedCount := atomic.LoadInt64(&failed)
|
|
skippedCount := atomic.LoadInt64(&skipped)
|
|
if failedCount > 0 && skippedCount == 0 && failedCount == processedCount {
|
|
return fmt.Errorf("ebook scan failed for every attempted folder_id=%d: %w", folder.ID, errors.Join(failures...))
|
|
}
|
|
}
|
|
|
|
seenPaths := make(map[string]bool, len(candidates))
|
|
for _, p := range candidates {
|
|
seenPaths[p] = true
|
|
}
|
|
return s.reconcileEbookScan(ctx, folder, scans, seenPaths, fullScan)
|
|
}
|
|
|
|
// ebookCleanupGuardRepo is the slice of catalog.FolderRepository the
|
|
// empty-root cleanup guard needs.
|
|
type ebookCleanupGuardRepo interface {
|
|
ConsumeEmptyCleanupAllowance(ctx context.Context, id int) (bool, error)
|
|
SetScanWarning(ctx context.Context, id int, code, message string, warnedAt time.Time) error
|
|
}
|
|
|
|
// ebookEmptyCleanupAllowed mirrors the video pipeline's empty-root guard: a
|
|
// scan that saw zero ebooks while the DB still has rows under the scanned
|
|
// roots may only reconcile (and therefore delete) when the operator has
|
|
// explicitly confirmed the cleanup. Subtree scans never consume the
|
|
// folder-level allowance; the next full scan converges them.
|
|
func ebookEmptyCleanupAllowed(ctx context.Context, repo ebookCleanupGuardRepo, folderID int, fullScan bool) (bool, error) {
|
|
if !fullScan || repo == nil {
|
|
return false, nil
|
|
}
|
|
allowed, err := repo.ConsumeEmptyCleanupAllowance(ctx, folderID)
|
|
if err != nil {
|
|
return false, fmt.Errorf("checking empty cleanup confirmation for folder %d: %w", folderID, err)
|
|
}
|
|
if !allowed {
|
|
if err := repo.SetScanWarning(ctx, folderID,
|
|
"empty_root",
|
|
"Scan found 0 media files; cleanup was skipped until deletion is confirmed.",
|
|
time.Now().UTC(),
|
|
); err != nil {
|
|
return false, fmt.Errorf("recording empty-root warning for folder %d: %w", folderID, err)
|
|
}
|
|
}
|
|
return allowed, nil
|
|
}
|
|
|
|
// emptyCleanupDecision determines whether an audio/ebook scan may clean roots
|
|
// that contain catalog rows but produced no files. A completely empty scan is
|
|
// blocked until confirmation; in a mixed multi-root scan only literally empty
|
|
// roots need confirmation, while healthy roots continue reconciling normally.
|
|
func (s *Scanner) emptyCleanupDecision(
|
|
ctx context.Context,
|
|
folder *models.MediaFolder,
|
|
roots []string,
|
|
seenPaths map[string]bool,
|
|
fullScan bool,
|
|
mediaKind string,
|
|
) (confirmed, blockAll bool, err error) {
|
|
if s.fileRepo == nil || len(roots) == 0 {
|
|
return false, false, nil
|
|
}
|
|
wholeScanEmpty := len(seenPaths) == 0
|
|
probes := rootcheck.ProbeManyWithTimeout(ctx, roots, rootcheck.DefaultProbeTimeout)
|
|
needsConfirmation := false
|
|
for i, root := range roots {
|
|
if !wholeScanEmpty {
|
|
seenUnderRoot := false
|
|
for path := range seenPaths {
|
|
if pathWithinAnyRoot(path, []string{root}) {
|
|
seenUnderRoot = true
|
|
break
|
|
}
|
|
}
|
|
if seenUnderRoot || !probes[i].Reachable || !probes[i].Empty {
|
|
continue
|
|
}
|
|
} else if !probes[i].Reachable {
|
|
continue
|
|
}
|
|
existing, listErr := s.fileRepo.GetByFolderAndPathPrefix(ctx, folder.ID, root)
|
|
if listErr != nil {
|
|
return false, false, fmt.Errorf("listing existing %s files for %q: %w", mediaKind, root, listErr)
|
|
}
|
|
if len(existing) > 0 {
|
|
needsConfirmation = true
|
|
break
|
|
}
|
|
}
|
|
if !needsConfirmation {
|
|
return false, false, nil
|
|
}
|
|
var guard ebookCleanupGuardRepo
|
|
if s.folderRepo != nil {
|
|
guard = s.folderRepo
|
|
}
|
|
allowed := false
|
|
if wholeScanEmpty {
|
|
var allowErr error
|
|
allowed, allowErr = ebookEmptyCleanupAllowed(ctx, guard, folder.ID, fullScan)
|
|
if allowErr != nil {
|
|
return false, false, allowErr
|
|
}
|
|
} else if fullScan && guard != nil {
|
|
var allowErr error
|
|
allowed, allowErr = guard.ConsumeEmptyCleanupAllowance(ctx, folder.ID)
|
|
if allowErr != nil {
|
|
return false, false, fmt.Errorf("checking partial-root cleanup confirmation for folder %d: %w", folder.ID, allowErr)
|
|
}
|
|
if !allowed {
|
|
if warnErr := guard.SetScanWarning(ctx, folder.ID,
|
|
"dead_root",
|
|
"One or more roots are reachable but empty while the library still has cataloged files; cleanup was skipped until deletion is confirmed.",
|
|
time.Now().UTC(),
|
|
); warnErr != nil {
|
|
return false, false, fmt.Errorf("recording partial-root warning for folder %d: %w", folder.ID, warnErr)
|
|
}
|
|
}
|
|
}
|
|
if !allowed {
|
|
slog.WarnContext(ctx, mediaKind+" scan: empty roots still have cataloged files; cleanup requires confirmation",
|
|
"component", "scanner", "folder_id", folder.ID, "full_scan", fullScan)
|
|
}
|
|
return allowed, wholeScanEmpty && !allowed, nil
|
|
}
|
|
|
|
// reconcileEbookScan applies the post-walk safety policy and then performs
|
|
// missing-file reconciliation for the roots that walked cleanly.
|
|
func (s *Scanner) reconcileEbookScan(ctx context.Context, folder *models.MediaFolder, scans []ebookRootScan, seenPaths map[string]bool, fullScan bool) error {
|
|
if folder != nil {
|
|
defer s.reconcileMissingEbookEnrichment(ctx, folder.ID)
|
|
}
|
|
reconcileRoots, _ := splitEbookReconcileRoots(scans)
|
|
if len(reconcileRoots) == 0 {
|
|
allFailed := len(scans) > 0
|
|
for i := range scans {
|
|
if !scans[i].failed() {
|
|
allFailed = false
|
|
break
|
|
}
|
|
}
|
|
if allFailed {
|
|
slog.WarnContext(ctx, "ebook scan: every root walk failed; skipping missing-file reconciliation", "component", "scanner",
|
|
"folder_id", folder.ID,
|
|
)
|
|
}
|
|
return nil
|
|
}
|
|
if s.fileRepo == nil || s.libraryRepo == nil {
|
|
return nil
|
|
}
|
|
|
|
confirmedCleanup, blockAll, err := s.emptyCleanupDecision(
|
|
ctx, folder, reconcileRoots, seenPaths, fullScan, "ebook",
|
|
)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
if blockAll {
|
|
return nil
|
|
}
|
|
|
|
if err := s.reconcileMissingEbookFiles(ctx, folder, reconcileRoots, seenPaths, confirmedCleanup); err != nil {
|
|
return err
|
|
}
|
|
if fullScan && s.folderRepo != nil {
|
|
// The cleanup either ran with files present or was explicitly
|
|
// confirmed; any prior empty-root warning is stale now.
|
|
if err := s.folderRepo.ClearScanWarning(ctx, folder.ID); err != nil {
|
|
return fmt.Errorf("clearing scan warning for folder %d: %w", folder.ID, err)
|
|
}
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// reconcileMissingEbookFiles mirrors the video/audio scan cleanup: DB files
|
|
// under the scanned roots that were not seen on disk are marked missing, the
|
|
// folder trash is optionally emptied, and library memberships are reconciled
|
|
// so items with no remaining files are removed (renames therefore converge on
|
|
// the newly indexed path instead of leaving a stale duplicate item).
|
|
func (s *Scanner) reconcileMissingEbookFiles(ctx context.Context, folder *models.MediaFolder, roots []string, seenPaths map[string]bool, confirmedCleanup bool) error {
|
|
if s.fileRepo == nil || s.libraryRepo == nil || len(roots) == 0 {
|
|
return nil
|
|
}
|
|
|
|
now := time.Now().UTC()
|
|
missing := 0
|
|
for _, root := range roots {
|
|
existing, err := s.fileRepo.GetByFolderAndPathPrefix(ctx, folder.ID, root)
|
|
if err != nil {
|
|
return fmt.Errorf("listing existing ebook files for %q: %w", root, err)
|
|
}
|
|
for _, mf := range existing {
|
|
if mf == nil || seenPaths[mf.FilePath] {
|
|
continue
|
|
}
|
|
if mf.MissingSince == nil {
|
|
if err := s.fileRepo.MarkMissing(ctx, mf.ID, now); err != nil {
|
|
slog.ErrorContext(ctx, "ebook scan: failed to mark file missing", "component", "scanner",
|
|
"folder_id", folder.ID,
|
|
"path", mf.FilePath,
|
|
"error", err,
|
|
)
|
|
continue
|
|
}
|
|
}
|
|
missing++
|
|
}
|
|
}
|
|
|
|
trashed, removedMemberships, deletedItems, err := s.sweepMissingAndReconcile(ctx, folder, confirmedCleanup)
|
|
if trashed > 0 {
|
|
slog.InfoContext(ctx, "ebook scan: emptied trash", "component", "scanner", "folder_id", folder.ID, "deleted", trashed)
|
|
}
|
|
if err != nil {
|
|
return err
|
|
}
|
|
if missing > 0 || removedMemberships > 0 || deletedItems > 0 {
|
|
slog.InfoContext(ctx, "ebook scan: reconciled missing files", "component", "scanner",
|
|
"folder_id", folder.ID, "missing", missing,
|
|
"memberships_removed", removedMemberships, "items_deleted", deletedItems)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
func reportEbookScanProgress(ctx context.Context, folderID int, total, processed, failed, skipped int) {
|
|
reportProgress(ctx, ProgressUpdate{
|
|
Phase: "ebook_scan",
|
|
Message: fmt.Sprintf("Scanning ebooks in folder %d", folderID),
|
|
CurrentScope: strconv.Itoa(folderID),
|
|
TotalFiles: total,
|
|
FilesDiscovered: total,
|
|
FilesProcessed: processed,
|
|
Errors: failed,
|
|
Unchanged: skipped,
|
|
})
|
|
}
|
|
|
|
func (s *Scanner) reconcileEbookFile(ctx context.Context, folder *models.MediaFolder, filePath string, skipped *int64, groupLocks *ebookGroupLocks) error {
|
|
info, err := os.Stat(filePath)
|
|
if err != nil {
|
|
if errors.Is(err, os.ErrNotExist) {
|
|
return nil
|
|
}
|
|
return fmt.Errorf("stat ebook file %s: %w", filePath, err)
|
|
}
|
|
size := info.Size()
|
|
modifiedAt := normalizeFileModifiedAt(info.ModTime())
|
|
if sidecarPath := ebookOPFSidecarPath(filePath); sidecarPath != "" {
|
|
if sidecarInfo, statErr := os.Stat(sidecarPath); statErr == nil {
|
|
sidecarModifiedAt := normalizeFileModifiedAt(sidecarInfo.ModTime())
|
|
if sidecarModifiedAt.After(modifiedAt) {
|
|
modifiedAt = sidecarModifiedAt
|
|
}
|
|
}
|
|
}
|
|
|
|
existingContentID, isUnchanged, skipErr := s.ebookFileShouldSkip(ctx, folder, filePath, size, modifiedAt)
|
|
if skipErr != nil {
|
|
slog.WarnContext(ctx, "ebook scan: skip-check failed, falling through", "component", "scanner",
|
|
"folder_id", folder.ID,
|
|
"path", filePath,
|
|
"error", skipErr,
|
|
)
|
|
} else if isUnchanged {
|
|
s.autoLinkLiteraryWork(ctx, existingContentID)
|
|
atomic.AddInt64(skipped, 1)
|
|
return nil
|
|
}
|
|
|
|
parsed, err := parseEbookFile(filePath)
|
|
if err != nil {
|
|
return fmt.Errorf("parse ebook file %s: %w", filePath, err)
|
|
}
|
|
if parsed.Title == "" {
|
|
parsed.Title = ebookTitleFromPath(filePath)
|
|
}
|
|
if len(parsed.Authors) == 0 {
|
|
if author := ebookAuthorFromPath(filePath); author != "" {
|
|
parsed.Authors = []string{author}
|
|
// A path-derived title carries the same " - Author" suffix; drop it
|
|
// so the title, group key, and enrichment query stay clean. Compare
|
|
// normalized so case/spacing variants (e.g. "a. f. carter") match
|
|
// the recovered author and don't leave a duplicated suffix.
|
|
if idx := strings.LastIndex(parsed.Title, " - "); idx >= 0 {
|
|
suffixAuthor := strings.TrimSpace(parsed.Title[idx+len(" - "):])
|
|
if normalizeEbookIdentityPart(suffixAuthor) == normalizeEbookIdentityPart(author) {
|
|
parsed.Title = strings.TrimSpace(parsed.Title[:idx])
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
groupKey := ebookContentGroupKey(&parsed, filePath)
|
|
unlock := groupLocks.lock(groupKey)
|
|
defer unlock()
|
|
|
|
contentID, curated, err := s.upsertEbookMediaItem(ctx, folder.ID, filePath, &parsed, groupKey)
|
|
if err != nil {
|
|
return fmt.Errorf("upsert ebook item: %w", err)
|
|
}
|
|
if err := s.upsertEbookMediaFile(ctx, folder, contentID, filePath, size, modifiedAt, &parsed, groupKey); err != nil {
|
|
return fmt.Errorf("upsert ebook file: %w", err)
|
|
}
|
|
if err := applyEbookLocalCover(ctx, s.itemRepo, s.imageCacher, contentID, filePath, &parsed); err != nil {
|
|
slog.WarnContext(ctx, "ebook scan: local cover upload failed", "component", "scanner",
|
|
"folder_id", folder.ID,
|
|
"content_id", contentID,
|
|
"path", filePath,
|
|
"error", err,
|
|
)
|
|
}
|
|
if err := s.upsertEbookPeople(ctx, contentID, &parsed, curated); err != nil {
|
|
return fmt.Errorf("upsert ebook people: %w", err)
|
|
}
|
|
if err := s.upsertEbookSeries(ctx, contentID, &parsed, curated); err != nil {
|
|
return fmt.Errorf("upsert ebook series: %w", err)
|
|
}
|
|
if err := insertEbookLibraryMembership(ctx, s.fileRepo.Pool(), contentID, folder.ID); err != nil {
|
|
return fmt.Errorf("upsert ebook library membership: %w", err)
|
|
}
|
|
if parsed.ISBN != "" {
|
|
if err := insertEbookISBNProviderID(ctx, s.fileRepo.Pool(), contentID, parsed.ISBN); err != nil {
|
|
return fmt.Errorf("upsert ebook ISBN provider id: %w", err)
|
|
}
|
|
}
|
|
if err := s.enqueueEbookEnrichment(ctx, contentID); err != nil {
|
|
return fmt.Errorf("enqueue ebook enrichment: %w", err)
|
|
}
|
|
s.autoLinkLiteraryWork(ctx, contentID)
|
|
slog.InfoContext(ctx, "ebook scan: indexed", "component", "scanner",
|
|
"folder_id", folder.ID,
|
|
"content_id", contentID,
|
|
"title", parsed.Title,
|
|
"authors", parsed.Authors,
|
|
"path", filePath,
|
|
)
|
|
return nil
|
|
}
|
|
|
|
const ebookEnrichmentPriority = 100
|
|
const ebookEnrichmentReconcileLimit = 500
|
|
|
|
func (s *Scanner) enqueueEbookEnrichment(ctx context.Context, contentID string) error {
|
|
if s == nil || s.ebookEnrichmentQueue == nil || strings.TrimSpace(contentID) == "" {
|
|
return nil
|
|
}
|
|
if err := s.ebookEnrichmentQueue.Enqueue(ctx, contentID, ebookEnrichmentPriority); err != nil {
|
|
slog.WarnContext(ctx, "ebook scan: metadata enrichment enqueue failed",
|
|
"component", "scanner",
|
|
"content_id", contentID,
|
|
"error", err,
|
|
)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
func (s *Scanner) reconcileMissingEbookEnrichment(ctx context.Context, folderID int) {
|
|
if s == nil || s.ebookEnrichmentQueue == nil || folderID <= 0 {
|
|
return
|
|
}
|
|
reconciled, inspected, wrapped, err := s.ebookEnrichmentQueue.ReconcileMissing(
|
|
ctx,
|
|
folderID,
|
|
ebookEnrichmentPriority,
|
|
ebookEnrichmentReconcileLimit,
|
|
)
|
|
if err != nil {
|
|
slog.WarnContext(ctx, "ebook scan: metadata enrichment reconciliation failed",
|
|
"component", "scanner",
|
|
"folder_id", folderID,
|
|
"limit", ebookEnrichmentReconcileLimit,
|
|
"error", err,
|
|
)
|
|
return
|
|
}
|
|
if reconciled > 0 {
|
|
slog.InfoContext(ctx, "ebook scan: repaired missing metadata enrichment jobs",
|
|
"component", "scanner",
|
|
"folder_id", folderID,
|
|
"reconciled", reconciled,
|
|
)
|
|
}
|
|
slog.DebugContext(ctx, "ebook scan: metadata enrichment reconciliation window complete",
|
|
"component", "scanner",
|
|
"folder_id", folderID,
|
|
"inspected", inspected,
|
|
"reconciled", reconciled,
|
|
"wrapped", wrapped,
|
|
)
|
|
}
|
|
|
|
func (s *Scanner) autoLinkLiteraryWork(ctx context.Context, contentID string) {
|
|
if s == nil || s.literaryWorkLinker == nil || strings.TrimSpace(contentID) == "" {
|
|
return
|
|
}
|
|
workID, linked, err := s.literaryWorkLinker.AutoLinkContent(ctx, contentID)
|
|
if err != nil {
|
|
slog.WarnContext(ctx, "literary work auto-link failed", "component", "scanner", "content_id", contentID, "error", err)
|
|
return
|
|
}
|
|
if linked {
|
|
slog.InfoContext(ctx, "literary work auto-linked", "component", "scanner", "content_id", contentID, "work_id", workID)
|
|
}
|
|
}
|
|
|
|
func (s *Scanner) ebookFileShouldSkip(ctx context.Context, folder *models.MediaFolder, filePath string, size int64, modifiedAt time.Time) (string, bool, error) {
|
|
if s.fileRepo == nil || s.itemRepo == nil {
|
|
return "", false, nil
|
|
}
|
|
existing, err := s.fileRepo.ListByObservedRootPath(ctx, folder.ID, filePath)
|
|
if err != nil {
|
|
return "", false, fmt.Errorf("list existing files: %w", err)
|
|
}
|
|
if len(existing) != 1 {
|
|
return "", false, nil
|
|
}
|
|
mf := existing[0]
|
|
if mf.FilePath != filePath || mf.FileSize != size || mf.FileModifiedAt == nil || !sameFileModifiedAt(mf.FileModifiedAt, modifiedAt) {
|
|
return "", false, nil
|
|
}
|
|
if mf.ContentID == "" {
|
|
return "", false, nil
|
|
}
|
|
if mf.GroupKeyVersion != ebookGroupKeyVersion {
|
|
// The grouping scheme changed since this row was written; reprocess
|
|
// once so the stored key is rewritten under the current scheme and
|
|
// sibling-format lookups can find it again.
|
|
return "", false, nil
|
|
}
|
|
statuses, err := s.itemRepo.GetStatusByIDs(ctx, []string{mf.ContentID})
|
|
if err != nil {
|
|
return "", false, fmt.Errorf("get item status: %w", err)
|
|
}
|
|
if strings.EqualFold(strings.TrimSpace(statuses[mf.ContentID]), "unmatched") {
|
|
return "", false, nil
|
|
}
|
|
return mf.ContentID, true, nil
|
|
}
|
|
|
|
// upsertEbookMediaItem resolves or creates the media item for the file and
|
|
// reports whether the item's metadata is curated (provider-matched), in which
|
|
// case dependent writes (people, series) must be fill-empty only.
|
|
func (s *Scanner) upsertEbookMediaItem(ctx context.Context, folderID int, filePath string, book *parsedEbook, groupKey string) (string, bool, error) {
|
|
if s.itemRepo == nil {
|
|
return "", false, fmt.Errorf("itemRepo not configured on Scanner")
|
|
}
|
|
if s.fileRepo == nil {
|
|
return "", false, fmt.Errorf("fileRepo not configured on Scanner")
|
|
}
|
|
|
|
existingID, err := s.fileRepo.FindContentIDByRootPath(ctx, folderID, filePath, "ebook")
|
|
if err != nil {
|
|
return "", false, fmt.Errorf("find ebook by root path: %w", err)
|
|
}
|
|
if existingID != "" {
|
|
curated, err := updateExistingEbookMediaItem(ctx, s.itemRepo, s.itemRepo, existingID, book)
|
|
if err != nil {
|
|
return "", false, err
|
|
}
|
|
return existingID, curated, nil
|
|
}
|
|
if existing := s.findEbookByFilePath(ctx, filePath); existing != nil {
|
|
if ebookItemHasCuratedMetadata(existing) {
|
|
return existing.ContentID, true, nil
|
|
}
|
|
applyEbookToMediaItem(existing, book)
|
|
if existing.SortTitle == "" {
|
|
existing.SortTitle = titleutil.DeriveDefaultSortTitle(existing.Title)
|
|
}
|
|
if err := s.itemRepo.Upsert(ctx, existing); err != nil {
|
|
return "", false, err
|
|
}
|
|
return existing.ContentID, false, nil
|
|
}
|
|
if existing := s.findEbookByContentGroupKey(ctx, folderID, groupKey, book.Format, filePath); existing != nil {
|
|
// The sibling file joins the group either way, but curated metadata
|
|
// on a provider-matched item must not be clobbered by whatever this
|
|
// file happens to embed.
|
|
if ebookItemHasCuratedMetadata(existing) {
|
|
return existing.ContentID, true, nil
|
|
}
|
|
applyEbookToMediaItem(existing, book)
|
|
if existing.SortTitle == "" {
|
|
existing.SortTitle = titleutil.DeriveDefaultSortTitle(existing.Title)
|
|
}
|
|
if err := s.itemRepo.Upsert(ctx, existing); err != nil {
|
|
return "", false, err
|
|
}
|
|
return existing.ContentID, false, nil
|
|
}
|
|
contentID, err := resolveEbookMediaItem(ctx, s.fileRepo, s.itemRepo, folderID, filePath, book)
|
|
if err != nil {
|
|
return "", false, err
|
|
}
|
|
return contentID, false, nil
|
|
}
|
|
|
|
func resolveEbookMediaItem(
|
|
ctx context.Context,
|
|
rootFinder filesystemRootContentFinder,
|
|
itemWriter filesystemMediaItemWriter,
|
|
folderID int,
|
|
filePath string,
|
|
book *parsedEbook,
|
|
) (string, error) {
|
|
if rootFinder == nil {
|
|
return "", fmt.Errorf("root content finder not configured")
|
|
}
|
|
if itemWriter == nil {
|
|
return "", fmt.Errorf("media item writer not configured")
|
|
}
|
|
existingID, err := rootFinder.FindContentIDByRootPath(ctx, folderID, filePath, "ebook")
|
|
if err != nil {
|
|
return "", fmt.Errorf("find ebook by root path: %w", err)
|
|
}
|
|
if existingID != "" {
|
|
return existingID, nil
|
|
}
|
|
|
|
cleanTitle := strings.TrimSpace(book.Title)
|
|
if cleanTitle == "" {
|
|
cleanTitle = ebookTitleFromPath(filePath)
|
|
}
|
|
return createEbookMediaItem(ctx, itemWriter, book, cleanTitle)
|
|
}
|
|
|
|
func createEbookMediaItem(ctx context.Context, itemWriter filesystemMediaItemWriter, book *parsedEbook, cleanTitle string) (string, error) {
|
|
id, err := idgen.NextID()
|
|
if err != nil {
|
|
return "", fmt.Errorf("generate content_id: %w", err)
|
|
}
|
|
item := &models.MediaItem{
|
|
ContentID: id,
|
|
Type: "ebook",
|
|
// Explicit "pending" (the item upsert writes the literal status, so the
|
|
// DB default never applies): enrichment promotes it to "matched", which
|
|
// arms ebookItemHasCuratedMetadata against file-metadata clobbering.
|
|
Status: "pending",
|
|
Title: cleanTitle,
|
|
SortTitle: titleutil.DeriveDefaultSortTitle(cleanTitle),
|
|
Year: book.Year,
|
|
}
|
|
applyEbookToMediaItem(item, book)
|
|
if item.SortTitle == "" {
|
|
item.SortTitle = titleutil.DeriveDefaultSortTitle(item.Title)
|
|
}
|
|
if err := itemWriter.Upsert(ctx, item); err != nil {
|
|
return "", err
|
|
}
|
|
return id, nil
|
|
}
|
|
|
|
// updateExistingEbookMediaItem refreshes a known item from file metadata and
|
|
// reports whether the item's metadata is curated (and was therefore left
|
|
// untouched).
|
|
func updateExistingEbookMediaItem(ctx context.Context, itemReader filesystemMediaItemReader, itemWriter filesystemMediaItemWriter, contentID string, book *parsedEbook) (bool, error) {
|
|
if itemReader == nil {
|
|
return false, fmt.Errorf("media item reader not configured")
|
|
}
|
|
if itemWriter == nil {
|
|
return false, fmt.Errorf("media item writer not configured")
|
|
}
|
|
items, err := itemReader.GetByIDs(ctx, []string{contentID})
|
|
if err != nil {
|
|
return false, fmt.Errorf("get ebook media item %s: %w", contentID, err)
|
|
}
|
|
if len(items) == 0 || items[0] == nil {
|
|
return false, fmt.Errorf("ebook media item %s not found", contentID)
|
|
}
|
|
item := items[0]
|
|
if ebookItemHasCuratedMetadata(item) {
|
|
// A matched item's metadata was curated by a provider or a person;
|
|
// re-parsing the file (mtime change, group-key version bump) must not
|
|
// clobber it with embedded file metadata.
|
|
return true, nil
|
|
}
|
|
applyEbookToMediaItem(item, book)
|
|
if item.SortTitle == "" {
|
|
item.SortTitle = titleutil.DeriveDefaultSortTitle(item.Title)
|
|
}
|
|
return false, itemWriter.Upsert(ctx, item)
|
|
}
|
|
|
|
// ebookItemHasCuratedMetadata reports whether the item's metadata is owned by
|
|
// a provider match (or manual curation) and therefore must not be overwritten
|
|
// by metadata embedded in scanned files.
|
|
func ebookItemHasCuratedMetadata(item *models.MediaItem) bool {
|
|
return item != nil && strings.EqualFold(strings.TrimSpace(item.Status), "matched")
|
|
}
|
|
|
|
func applyEbookToMediaItem(item *models.MediaItem, book *parsedEbook) {
|
|
item.Type = "ebook"
|
|
if title := strings.TrimSpace(book.Title); title != "" {
|
|
item.Title = title
|
|
}
|
|
if book.Year > 0 {
|
|
item.Year = book.Year
|
|
}
|
|
if book.Description != "" && (item.Overview == "" || looksLikeHTML(item.Overview)) {
|
|
item.Overview = book.Description
|
|
}
|
|
if book.Publisher != "" {
|
|
item.Studios = mergeUniqueStrings(item.Studios, []string{book.Publisher})
|
|
}
|
|
if len(book.Genres) > 0 {
|
|
item.Genres = mergeUniqueStrings(item.Genres, book.Genres)
|
|
}
|
|
if !book.PublishedAt.IsZero() && item.ReleaseDate == nil {
|
|
rd := book.PublishedAt.UTC().Format("2006-01-02")
|
|
item.ReleaseDate = &rd
|
|
}
|
|
if book.Language != "" && item.OriginalLanguage == "" {
|
|
item.OriginalLanguage = book.Language
|
|
}
|
|
}
|
|
|
|
func (s *Scanner) findEbookByFilePath(ctx context.Context, filePath string) *models.MediaItem {
|
|
if s.fileRepo == nil {
|
|
return nil
|
|
}
|
|
var existingID string
|
|
err := s.fileRepo.Pool().QueryRow(ctx, `
|
|
SELECT mf.content_id
|
|
FROM media_files mf
|
|
JOIN media_items mi ON mi.content_id = mf.content_id
|
|
WHERE mf.file_path = $1
|
|
AND mi.type = 'ebook'
|
|
LIMIT 1
|
|
`, filePath).Scan(&existingID)
|
|
if err != nil || existingID == "" {
|
|
return nil
|
|
}
|
|
items, err := s.itemRepo.GetByIDs(ctx, []string{existingID})
|
|
if err != nil || len(items) == 0 {
|
|
return nil
|
|
}
|
|
return items[0]
|
|
}
|
|
|
|
// findEbookByContentGroupKey locates an existing item that a sibling format of
|
|
// the same book should join. Grouping merges *different* formats only: an item
|
|
// that already owns another file of the same format is excluded, so two
|
|
// distinct books whose sparse metadata collides on the same key (e.g. every
|
|
// volume carrying the series name as its title) stay separate items instead of
|
|
// silently merging.
|
|
func (s *Scanner) findEbookByContentGroupKey(ctx context.Context, folderID int, groupKey string, format string, filePath string) *models.MediaItem {
|
|
if s.fileRepo == nil || s.itemRepo == nil || strings.TrimSpace(groupKey) == "" {
|
|
return nil
|
|
}
|
|
var existingID string
|
|
err := s.fileRepo.Pool().QueryRow(ctx, `
|
|
SELECT mf.content_id
|
|
FROM media_files mf
|
|
JOIN media_items mi ON mi.content_id = mf.content_id
|
|
WHERE mf.media_folder_id = $1
|
|
AND mf.group_key_version = $2
|
|
AND mf.content_group_key = $3
|
|
AND mf.missing_since IS NULL
|
|
AND mi.type = 'ebook'
|
|
AND NOT EXISTS (
|
|
SELECT 1 FROM media_files dup
|
|
WHERE dup.content_id = mf.content_id
|
|
AND dup.missing_since IS NULL
|
|
AND lower(dup.container) = lower($4)
|
|
AND dup.file_path <> $5
|
|
)
|
|
ORDER BY CASE WHEN lower(trim(mi.status)) = 'matched' THEN 0 ELSE 1 END,
|
|
mf.id ASC
|
|
LIMIT 1
|
|
`, folderID, ebookGroupKeyVersion, groupKey, format, filePath).Scan(&existingID)
|
|
if err != nil || existingID == "" {
|
|
return nil
|
|
}
|
|
items, err := s.itemRepo.GetByIDs(ctx, []string{existingID})
|
|
if err != nil || len(items) == 0 {
|
|
return nil
|
|
}
|
|
return items[0]
|
|
}
|
|
|
|
// ebookGroupKeyVersion versions the ebookContentGroupKey scheme. Bump it
|
|
// whenever the key shape changes so rows keyed under an older scheme are
|
|
// reprocessed and re-keyed (see ebookFileShouldSkip) instead of silently never
|
|
// matching sibling-format lookups again.
|
|
const ebookGroupKeyVersion = 2
|
|
|
|
func (s *Scanner) upsertEbookMediaFile(ctx context.Context, folder *models.MediaFolder, contentID string, filePath string, size int64, modifiedAt time.Time, book *parsedEbook, groupKey string) error {
|
|
mf := buildEbookMediaFile(folder, contentID, filePath, size, modifiedAt, book, groupKey)
|
|
if _, err := s.fileRepo.Upsert(ctx, mf); err != nil {
|
|
return fmt.Errorf("upsert media file %s: %w", filePath, err)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
func buildEbookMediaFile(folder *models.MediaFolder, contentID string, filePath string, size int64, modifiedAt time.Time, book *parsedEbook, groupKey string) models.MediaFile {
|
|
return models.MediaFile{
|
|
ContentID: contentID,
|
|
MediaFolderID: folder.ID,
|
|
CanonicalRootPath: filePath,
|
|
ObservedRootPath: filePath,
|
|
ContentGroupKey: groupKey,
|
|
GroupKeyVersion: ebookGroupKeyVersion,
|
|
BaseTitle: book.Title,
|
|
BaseYear: book.Year,
|
|
BaseType: "ebook",
|
|
IdentityConfidence: ebookIdentityConfidence(book),
|
|
FilePath: filePath,
|
|
FileSize: size,
|
|
FileModifiedAt: &modifiedAt,
|
|
Container: book.Format,
|
|
Duration: book.PageCount,
|
|
ProbeSource: "local",
|
|
}
|
|
}
|
|
|
|
func ebookContentGroupKey(book *parsedEbook, filePath string) string {
|
|
if book != nil {
|
|
if isbn := normalizeEbookISBN(book.ISBN); isbn != "" {
|
|
return "ebook:isbn:" + isbn
|
|
}
|
|
}
|
|
|
|
title := ""
|
|
authors := []string(nil)
|
|
if book != nil {
|
|
title = strings.TrimSpace(book.Title)
|
|
authors = book.Authors
|
|
}
|
|
if title == "" {
|
|
title = ebookTitleFromPath(filePath)
|
|
}
|
|
normalizedTitle := normalizeEbookIdentityPart(title)
|
|
if normalizedTitle == "" {
|
|
cleanPath := strings.TrimSpace(filepath.Clean(filePath))
|
|
if cleanPath == "." {
|
|
return ""
|
|
}
|
|
return "ebook:path:" + cleanPath
|
|
}
|
|
|
|
normalizedAuthors := make([]string, 0, len(authors))
|
|
seenAuthors := make(map[string]struct{}, len(authors))
|
|
for _, author := range authors {
|
|
normalized := normalizeEbookIdentityPart(author)
|
|
if normalized == "" {
|
|
continue
|
|
}
|
|
if _, ok := seenAuthors[normalized]; ok {
|
|
continue
|
|
}
|
|
seenAuthors[normalized] = struct{}{}
|
|
normalizedAuthors = append(normalizedAuthors, normalized)
|
|
}
|
|
sort.Strings(normalizedAuthors)
|
|
if len(normalizedAuthors) > 0 {
|
|
return "ebook:title_author:" + normalizedTitle + "|" + strings.Join(normalizedAuthors, ",")
|
|
}
|
|
|
|
dir := strings.TrimSpace(filepath.Clean(filepath.Dir(filePath)))
|
|
if dir == "." {
|
|
dir = ""
|
|
}
|
|
return "ebook:title:" + normalizedTitle + "|dir:" + normalizeEbookIdentityPart(dir)
|
|
}
|
|
|
|
func ebookTitleFromPath(filePath string) string {
|
|
base := filepath.Base(filePath)
|
|
if base == "." || base == string(filepath.Separator) {
|
|
return ""
|
|
}
|
|
// ".fb2.zip" is a double extension that filepath.Ext (and the
|
|
// normalized ".fbz" format token) would leave half-stripped.
|
|
if strings.HasSuffix(strings.ToLower(base), ".fb2.zip") {
|
|
return base[:len(base)-len(".fb2.zip")]
|
|
}
|
|
return strings.TrimSuffix(base, filepath.Ext(base))
|
|
}
|
|
|
|
// ebookAuthorFromPath recovers an author for libraries that shelve books as
|
|
// ".../<Author>/<Title>/<Title> - <Author>.ext" but embed no author in the file
|
|
// (common for PDF/MOBI/AZW3). It returns a value only when two independent path
|
|
// signals agree: the grandparent directory name and the filename's trailing
|
|
// " - X" segment. Authorless layouts — magazines, language courses, flat dumps —
|
|
// satisfy neither or only one signal, so they never get a junk author that would
|
|
// poison the enrichment search.
|
|
func ebookAuthorFromPath(filePath string) string {
|
|
base := ebookTitleFromPath(filePath)
|
|
idx := strings.LastIndex(base, " - ")
|
|
if idx < 0 {
|
|
return ""
|
|
}
|
|
fromName := strings.TrimSpace(base[idx+len(" - "):])
|
|
if fromName == "" {
|
|
return ""
|
|
}
|
|
grandparent := strings.TrimSpace(filepath.Base(filepath.Dir(filepath.Dir(filePath))))
|
|
switch grandparent {
|
|
case "", ".", string(filepath.Separator):
|
|
return ""
|
|
}
|
|
if normalizeEbookIdentityPart(fromName) != normalizeEbookIdentityPart(grandparent) {
|
|
return ""
|
|
}
|
|
// Position alone cannot tell author from title: some libraries shelve
|
|
// books as "<Title>/<Author>/<Author> - <Title>" (inverted) which also
|
|
// satisfies the grandparent==suffix check and would assign the title as
|
|
// the author. Require the candidate to look like a person name; series and
|
|
// title folders ("De legenden van de Alfen") fail this and are rejected.
|
|
if !looksLikePersonName(grandparent) {
|
|
return ""
|
|
}
|
|
// Return the directory form, which carries canonical casing ("A. F. Carter"
|
|
// rather than a lowercased filename suffix).
|
|
return grandparent
|
|
}
|
|
|
|
// looksLikePersonName reports whether value is shaped like an author name. A
|
|
// "Last, First" comma form is accepted outright; otherwise every token must be
|
|
// capitalized or a known name particle (van, de, von, ...). Title/series
|
|
// strings contain lowercase content words and so are rejected.
|
|
func looksLikePersonName(value string) bool {
|
|
value = strings.TrimSpace(value)
|
|
if value == "" {
|
|
return false
|
|
}
|
|
if strings.ContainsAny(value, "0123456789") {
|
|
return false
|
|
}
|
|
if strings.Contains(value, ",") {
|
|
return true
|
|
}
|
|
particles := map[string]struct{}{
|
|
"van": {}, "von": {}, "de": {}, "der": {}, "den": {}, "het": {}, "di": {},
|
|
"da": {}, "del": {}, "della": {}, "la": {}, "le": {}, "el": {}, "du": {},
|
|
"dos": {}, "das": {}, "bin": {}, "al": {}, "ter": {}, "te": {}, "ten": {},
|
|
"op": {}, "'t": {},
|
|
}
|
|
hasUpper := false
|
|
for _, token := range strings.Fields(value) {
|
|
r := []rune(token)[0]
|
|
if unicode.IsUpper(r) {
|
|
hasUpper = true
|
|
continue
|
|
}
|
|
if _, ok := particles[strings.ToLower(token)]; ok {
|
|
continue
|
|
}
|
|
return false
|
|
}
|
|
return hasUpper
|
|
}
|
|
|
|
func normalizeEbookIdentityPart(value string) string {
|
|
value = strings.ToLower(strings.TrimSpace(value))
|
|
var b strings.Builder
|
|
previousSpace := false
|
|
for _, r := range value {
|
|
if unicode.IsLetter(r) || unicode.IsDigit(r) {
|
|
b.WriteRune(r)
|
|
previousSpace = false
|
|
continue
|
|
}
|
|
if unicode.IsSpace(r) || r == '&' || r == '-' || r == '_' || r == ':' || r == '/' || r == '\\' || r == ',' || r == '.' {
|
|
if b.Len() > 0 && !previousSpace {
|
|
b.WriteByte(' ')
|
|
previousSpace = true
|
|
}
|
|
}
|
|
}
|
|
return strings.TrimSpace(b.String())
|
|
}
|
|
|
|
type ebookGroupLocks struct {
|
|
mu sync.Mutex
|
|
locks map[string]*sync.Mutex
|
|
}
|
|
|
|
func newEbookGroupLocks() *ebookGroupLocks {
|
|
return &ebookGroupLocks{locks: make(map[string]*sync.Mutex)}
|
|
}
|
|
|
|
func (l *ebookGroupLocks) lock(key string) func() {
|
|
if l == nil || strings.TrimSpace(key) == "" {
|
|
return func() {}
|
|
}
|
|
l.mu.Lock()
|
|
groupLock := l.locks[key]
|
|
if groupLock == nil {
|
|
groupLock = &sync.Mutex{}
|
|
l.locks[key] = groupLock
|
|
}
|
|
l.mu.Unlock()
|
|
|
|
groupLock.Lock()
|
|
return groupLock.Unlock
|
|
}
|
|
|
|
// localEbookPosterPrefix mirrors the storage layout produced by
|
|
// imagecache.CacheEbookCover ("local/ebooks/{contentID}/poster/..."); it marks
|
|
// posters this pipeline owns and is therefore allowed to refresh.
|
|
const localEbookPosterPrefix = "local/ebooks/"
|
|
|
|
type ebookCoverMetadataStore interface {
|
|
GetPoster(ctx context.Context, contentID string) (posterPath string, posterThumbhash string, err error)
|
|
SetLocalPoster(ctx context.Context, contentID, posterPath, thumbhash, localPrefix string) (bool, error)
|
|
}
|
|
|
|
// applyEbookLocalCover records the best locally available cover for the file:
|
|
// a sidecar image that belongs to this book wins over the embedded cover, and
|
|
// exactly one cover is applied per reconcile. A sidecar discovery error does
|
|
// not block the embedded fallback.
|
|
func applyEbookLocalCover(ctx context.Context, store ebookCoverMetadataStore, cacher ebookCoverCacher, contentID string, ebookFilePath string, book *parsedEbook) error {
|
|
if store == nil || cacher == nil || contentID == "" {
|
|
return nil
|
|
}
|
|
var data []byte
|
|
var sidecarErr error
|
|
if ebookFilePath != "" {
|
|
cover, _, err := findSidecarBookCover(ebookFilePath)
|
|
if err != nil {
|
|
sidecarErr = err
|
|
} else if cover != nil {
|
|
data = cover.Bytes
|
|
}
|
|
}
|
|
if len(data) == 0 && book != nil && book.Cover != nil {
|
|
data = book.Cover.Bytes
|
|
}
|
|
if len(data) == 0 {
|
|
return sidecarErr
|
|
}
|
|
return errors.Join(sidecarErr, cacheEbookCoverBytes(ctx, store, cacher, contentID, data))
|
|
}
|
|
|
|
func cacheEbookCoverBytes(ctx context.Context, store ebookCoverMetadataStore, cacher ebookCoverCacher, contentID string, data []byte) error {
|
|
if len(data) == 0 {
|
|
return nil
|
|
}
|
|
existingPath, existingThumbhash, err := store.GetPoster(ctx, contentID)
|
|
if err != nil {
|
|
return fmt.Errorf("get ebook poster for cover: %w", err)
|
|
}
|
|
existingPath = strings.TrimSpace(existingPath)
|
|
if existingPath != "" {
|
|
// Provider or manually applied artwork always wins over scan covers.
|
|
if !strings.HasPrefix(existingPath, localEbookPosterPrefix) {
|
|
return nil
|
|
}
|
|
// Re-extracted bytes of an unchanged cover hash identically; skip
|
|
// the variant regeneration and upload churn. A replaced cover hashes
|
|
// differently and falls through to refresh the stale poster.
|
|
if thumbhash, err := imageutil.Thumbhash(data); err == nil && thumbhash == existingThumbhash {
|
|
return nil
|
|
}
|
|
}
|
|
posterPath, thumbhash, err := cacher.CacheEbookCover(ctx, data, contentID)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
if _, err := store.SetLocalPoster(ctx, contentID, posterPath, thumbhash, localEbookPosterPrefix); err != nil {
|
|
return fmt.Errorf("set ebook local poster: %w", err)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
var sidecarCoverNames = []string{"cover", "folder", "front", "poster", "thumbnail"}
|
|
var sidecarCoverExtensions = []string{".jpg", ".jpeg", ".png", ".webp", ".avif", ".gif", ".bmp"}
|
|
|
|
// findSidecarBookCover looks for cover art next to the ebook file. An image
|
|
// named after the book file always belongs to it; the generic artwork names
|
|
// (cover.jpg, folder.png, ...) are trusted only when this is the directory's
|
|
// sole ebook, so one cover.jpg in a flat multi-book folder is not applied to
|
|
// every book in it.
|
|
func findSidecarBookCover(ebookFilePath string) (*parsedEbookCover, string, error) {
|
|
if ebookFilePath == "" {
|
|
return nil, "", nil
|
|
}
|
|
entries, err := os.ReadDir(filepath.Dir(ebookFilePath))
|
|
if err != nil {
|
|
return nil, "", err
|
|
}
|
|
byName := make(map[string]string, len(entries))
|
|
ebookCount := 0
|
|
for _, entry := range entries {
|
|
if entry.IsDir() {
|
|
continue
|
|
}
|
|
if SupportsEbookFile(entry.Name()) {
|
|
ebookCount++
|
|
}
|
|
byName[strings.ToLower(entry.Name())] = filepath.Join(filepath.Dir(ebookFilePath), entry.Name())
|
|
}
|
|
|
|
var candidates []string
|
|
if base := strings.ToLower(ebookTitleFromPath(ebookFilePath)); base != "" {
|
|
for _, ext := range sidecarCoverExtensions {
|
|
candidates = append(candidates, base+ext)
|
|
}
|
|
}
|
|
if ebookCount <= 1 {
|
|
for _, name := range sidecarCoverNames {
|
|
for _, ext := range sidecarCoverExtensions {
|
|
candidates = append(candidates, name+ext)
|
|
}
|
|
}
|
|
}
|
|
for _, candidate := range candidates {
|
|
path := byName[candidate]
|
|
if path == "" {
|
|
continue
|
|
}
|
|
data, err := readSidecarCover(path)
|
|
if err != nil {
|
|
return nil, path, err
|
|
}
|
|
return &parsedEbookCover{
|
|
ContentType: ebookImageContentType(path),
|
|
Bytes: data,
|
|
}, path, nil
|
|
}
|
|
return nil, "", nil
|
|
}
|
|
|
|
func readSidecarCover(path string) ([]byte, error) {
|
|
file, err := os.Open(path)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
defer file.Close()
|
|
info, err := file.Stat()
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
if info.Size() > maxEPUBMetadataEntrySize {
|
|
return nil, fmt.Errorf("sidecar cover too large: %s", path)
|
|
}
|
|
data, err := io.ReadAll(io.LimitReader(file, maxEPUBMetadataEntrySize+1))
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
if len(data) > maxEPUBMetadataEntrySize {
|
|
return nil, fmt.Errorf("sidecar cover too large: %s", path)
|
|
}
|
|
if len(data) == 0 {
|
|
return nil, fmt.Errorf("sidecar cover empty: %s", path)
|
|
}
|
|
return data, nil
|
|
}
|
|
|
|
func insertEbookLibraryMembership(ctx context.Context, exec ebookSQLExecutor, contentID string, folderID int) error {
|
|
_, err := exec.Exec(ctx, `
|
|
INSERT INTO media_item_libraries (content_id, media_folder_id, first_seen_at)
|
|
VALUES ($1, $2, NOW())
|
|
ON CONFLICT (content_id, media_folder_id) DO NOTHING
|
|
`, contentID, folderID)
|
|
return err
|
|
}
|
|
|
|
func insertEbookISBNProviderID(ctx context.Context, exec ebookSQLExecutor, contentID string, isbn string) error {
|
|
_, err := exec.Exec(ctx, `
|
|
INSERT INTO media_item_provider_ids (content_id, provider, provider_id, item_type)
|
|
VALUES ($1, 'isbn', $2, 'ebook')
|
|
ON CONFLICT (content_id, provider) DO UPDATE SET
|
|
provider_id = EXCLUDED.provider_id,
|
|
updated_at = NOW()
|
|
WHERE media_item_provider_ids.provider_id IS DISTINCT FROM EXCLUDED.provider_id
|
|
`, contentID, isbn)
|
|
// A different item may already own this ISBN under the (provider, provider_id)
|
|
// unique constraint; duplicate copies must not fail the scan.
|
|
var pgErr *pgconn.PgError
|
|
if errors.As(err, &pgErr) && pgErr.Code == "23505" {
|
|
return nil
|
|
}
|
|
return err
|
|
}
|
|
|
|
type ebookCredit struct {
|
|
Name string
|
|
Kind models.PersonKind
|
|
}
|
|
|
|
func ebookPeopleCreditsEqual(existing []models.ItemPerson, desired []ebookCredit) bool {
|
|
existingAuthors := make([]models.ItemPerson, 0, len(existing))
|
|
for _, p := range existing {
|
|
if p.Kind == models.PersonKindNarrator {
|
|
return false
|
|
}
|
|
if p.Kind == models.PersonKindAuthor {
|
|
existingAuthors = append(existingAuthors, p)
|
|
}
|
|
}
|
|
if len(existingAuthors) != len(desired) {
|
|
return false
|
|
}
|
|
type key struct {
|
|
name string
|
|
kind models.PersonKind
|
|
}
|
|
have := make(map[key]struct{}, len(existingAuthors))
|
|
for _, p := range existingAuthors {
|
|
have[key{strings.ToLower(strings.TrimSpace(p.Person.Name)), p.Kind}] = struct{}{}
|
|
}
|
|
for _, d := range desired {
|
|
k := key{strings.ToLower(strings.TrimSpace(d.Name)), d.Kind}
|
|
if _, ok := have[k]; !ok {
|
|
return false
|
|
}
|
|
}
|
|
return true
|
|
}
|
|
|
|
type ebookResolvedAuthor struct {
|
|
ID int64
|
|
Name string
|
|
}
|
|
|
|
func mergeEbookPeople(existing []models.ItemPerson, authors []ebookResolvedAuthor) []models.ItemPerson {
|
|
people := make([]models.ItemPerson, 0, len(existing)+len(authors))
|
|
for _, p := range existing {
|
|
if p.Kind == models.PersonKindAuthor || p.Kind == models.PersonKindNarrator {
|
|
continue
|
|
}
|
|
p.SortOrder = len(people)
|
|
people = append(people, p)
|
|
}
|
|
for _, author := range authors {
|
|
people = append(people, models.ItemPerson{
|
|
Person: models.Person{ID: author.ID, Name: author.Name},
|
|
Kind: models.PersonKindAuthor,
|
|
SortOrder: len(people),
|
|
})
|
|
}
|
|
return people
|
|
}
|
|
|
|
func ebookPeopleForReplace(existing []models.ItemPerson, getErr error, authors []ebookResolvedAuthor) ([]models.ItemPerson, error) {
|
|
if getErr != nil {
|
|
return nil, fmt.Errorf("get ebook people: %w", getErr)
|
|
}
|
|
return mergeEbookPeople(existing, authors), nil
|
|
}
|
|
|
|
// ebookPeopleWriteAllowed reports whether file-embedded authors may be written
|
|
// for the item. Curated (provider-matched) items are fill-empty only: file
|
|
// metadata may supply authors when the item has none, but must never replace
|
|
// provider-enriched author credits.
|
|
func ebookPeopleWriteAllowed(curated bool, existing []models.ItemPerson) bool {
|
|
if !curated {
|
|
return true
|
|
}
|
|
for _, p := range existing {
|
|
if p.Kind == models.PersonKindAuthor {
|
|
return false
|
|
}
|
|
}
|
|
return true
|
|
}
|
|
|
|
func (s *Scanner) upsertEbookPeople(ctx context.Context, contentID string, book *parsedEbook, curated bool) error {
|
|
if s.personRepo == nil {
|
|
return fmt.Errorf("personRepo not configured on Scanner")
|
|
}
|
|
if s.itemRepo == nil {
|
|
return fmt.Errorf("itemRepo not configured on Scanner")
|
|
}
|
|
|
|
var desired []ebookCredit
|
|
for _, author := range book.Authors {
|
|
if name := strings.TrimSpace(author); name != "" {
|
|
desired = append(desired, ebookCredit{Name: name, Kind: models.PersonKindAuthor})
|
|
}
|
|
}
|
|
if len(desired) == 0 {
|
|
return nil
|
|
}
|
|
|
|
existing, err := s.itemRepo.GetPeople(ctx, contentID)
|
|
if err != nil {
|
|
return fmt.Errorf("get ebook people: %w", err)
|
|
}
|
|
if !ebookPeopleWriteAllowed(curated, existing) {
|
|
return nil
|
|
}
|
|
if ebookPeopleCreditsEqual(existing, desired) {
|
|
return nil
|
|
}
|
|
|
|
authors := make([]ebookResolvedAuthor, 0, len(desired))
|
|
for _, c := range desired {
|
|
personID, err := s.personRepo.FindOrCreate(ctx, models.Person{Name: c.Name})
|
|
if err != nil {
|
|
return fmt.Errorf("find-or-create person %q: %w", c.Name, err)
|
|
}
|
|
authors = append(authors, ebookResolvedAuthor{ID: personID, Name: c.Name})
|
|
}
|
|
people, err := ebookPeopleForReplace(existing, nil, authors)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
return s.itemRepo.ReplacePeople(ctx, contentID, people)
|
|
}
|
|
|
|
func (s *Scanner) upsertEbookSeries(ctx context.Context, contentID string, book *parsedEbook, curated bool) error {
|
|
if s == nil {
|
|
return fmt.Errorf("Scanner not configured")
|
|
}
|
|
if s.fileRepo == nil {
|
|
return fmt.Errorf("fileRepo not configured on Scanner")
|
|
}
|
|
|
|
var currentName *string
|
|
var currentIdx *float64
|
|
err := s.fileRepo.Pool().QueryRow(ctx, `
|
|
SELECT series_name, series_index FROM ebook_series WHERE content_id = $1
|
|
`, contentID).Scan(¤tName, ¤tIdx)
|
|
|
|
plan, err := planEbookSeriesWrite(book, currentName, currentIdx, err, curated)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
|
|
switch plan.Kind {
|
|
case ebookSeriesWriteNone:
|
|
return nil
|
|
case ebookSeriesWriteDelete:
|
|
if _, delErr := s.fileRepo.Pool().Exec(ctx,
|
|
`DELETE FROM ebook_series WHERE content_id = $1`, contentID); delErr != nil {
|
|
return fmt.Errorf("delete ebook_series row: %w", delErr)
|
|
}
|
|
return nil
|
|
case ebookSeriesWriteUpsert:
|
|
var idx any
|
|
if plan.Index != nil {
|
|
idx = *plan.Index
|
|
}
|
|
if _, err := s.fileRepo.Pool().Exec(ctx, `
|
|
INSERT INTO ebook_series (content_id, series_name, series_index, updated_at)
|
|
VALUES ($1, $2, $3, NOW())
|
|
ON CONFLICT (content_id) DO UPDATE SET
|
|
series_name = EXCLUDED.series_name,
|
|
series_index = EXCLUDED.series_index,
|
|
updated_at = NOW()
|
|
`, contentID, plan.Name, idx); err != nil {
|
|
return fmt.Errorf("upsert ebook_series row: %w", err)
|
|
}
|
|
return nil
|
|
default:
|
|
return fmt.Errorf("unknown ebook_series write kind: %d", plan.Kind)
|
|
}
|
|
}
|
|
|
|
func ebookSeriesDesired(book *parsedEbook) (string, *float64) {
|
|
if book == nil {
|
|
return "", nil
|
|
}
|
|
return strings.TrimSpace(book.Series), parseSeriesIndex(book.SeriesIndex)
|
|
}
|
|
|
|
type ebookSeriesWriteKind int
|
|
|
|
const (
|
|
ebookSeriesWriteNone ebookSeriesWriteKind = iota
|
|
ebookSeriesWriteDelete
|
|
ebookSeriesWriteUpsert
|
|
)
|
|
|
|
type ebookSeriesWritePlan struct {
|
|
Kind ebookSeriesWriteKind
|
|
Name string
|
|
Index *float64
|
|
}
|
|
|
|
// planEbookSeriesWrite decides how the ebook_series row should change for
|
|
// file-embedded series metadata. fillOnly applies to curated
|
|
// (provider-matched) items: the file may supply a series when the item has
|
|
// none, but must never replace or delete an existing (provider-enriched) row.
|
|
func planEbookSeriesWrite(book *parsedEbook, currentName *string, currentIdx *float64, queryErr error, fillOnly bool) (ebookSeriesWritePlan, error) {
|
|
if queryErr != nil {
|
|
if !errors.Is(queryErr, pgx.ErrNoRows) {
|
|
return ebookSeriesWritePlan{}, fmt.Errorf("query ebook_series: %w", queryErr)
|
|
}
|
|
currentName = nil
|
|
currentIdx = nil
|
|
}
|
|
|
|
desiredName, desiredIdx := ebookSeriesDesired(book)
|
|
if desiredName == "" {
|
|
if currentName == nil || fillOnly {
|
|
return ebookSeriesWritePlan{Kind: ebookSeriesWriteNone}, nil
|
|
}
|
|
return ebookSeriesWritePlan{Kind: ebookSeriesWriteDelete}, nil
|
|
}
|
|
|
|
if currentName != nil {
|
|
if fillOnly {
|
|
return ebookSeriesWritePlan{Kind: ebookSeriesWriteNone}, nil
|
|
}
|
|
if *currentName == desiredName && floatPtrEqual(currentIdx, desiredIdx) {
|
|
return ebookSeriesWritePlan{Kind: ebookSeriesWriteNone}, nil
|
|
}
|
|
}
|
|
|
|
return ebookSeriesWritePlan{
|
|
Kind: ebookSeriesWriteUpsert,
|
|
Name: desiredName,
|
|
Index: desiredIdx,
|
|
}, nil
|
|
}
|
|
|
|
func ebookIdentityConfidence(book *parsedEbook) string {
|
|
if book == nil {
|
|
return "low"
|
|
}
|
|
|
|
score := 0
|
|
if book.Title != "" {
|
|
score++
|
|
}
|
|
if len(book.Authors) > 0 {
|
|
score++
|
|
}
|
|
if book.Year > 0 {
|
|
score++
|
|
}
|
|
if book.ISBN != "" {
|
|
score++
|
|
}
|
|
|
|
switch {
|
|
case score >= 4:
|
|
return "high"
|
|
case score > 0:
|
|
return "medium"
|
|
default:
|
|
return "low"
|
|
}
|
|
}
|