Files
silo-server/internal/scanner/types.go
T
8fc054c15d fix(scanner): never purge files under unreachable library roots (#372)
* fix(scanner): never purge files under unreachable library roots

An unreachable root is not a removed root. When one root of a multi-root
library dies (unmounted share, dead drive) while another root still has
files, the whole-library empty-root guard does not fire — the surviving
root produced files — so the scan marks everything under the dead root
missing_since (desired: hides it from browse/playback) and then, with the
default scanner.empty_trash_after_scan=true + 24h file_removal_grace, the
next scan after the grace hard-deletes every row under the dead root. A
week-long drive outage silently destroys the root's entire catalog state:
probe data, intro/credits markers, file hashes. Worse, membership
reconciliation immediately purges media_items whose only files lived on
the dead root, cascading user collections (library_collection_items has
ON DELETE CASCADE) and deleting cached artwork.

This change makes "temporarily offline" survivable:

- Probe each configured root at scan start (os.Stat + IsDir + ReadDir,
  factored into the new internal/rootcheck package and shared with the
  admin mount-check endpoint). Unreachable roots are skipped by the walk
  but their scopes still reconcile, so files are still marked missing.
- The trash sweep (DeleteMissingByFolder) now excludes rows whose path
  sits under an unreachable root, using the same exact-path + escaped
  prefix-LIKE matching as ListIDsOutsideRoots (a sibling root that merely
  shares a string prefix is never protected). With all roots reachable
  the emitted SQL is unchanged.
- Membership removal still happens — browse/home hide items via
  media_item_libraries, so removal is what keeps a dead-root-only title
  out of the catalog — but the orphan media_items purge exempts items
  whose files sit under an unreachable root. Their metadata, artwork,
  and collection links survive; when the root returns, the upsert clears
  missing_since and syncPresentLibraryState re-inserts the membership,
  restoring the item with zero re-probing or re-matching.
- The folder surfaces scan_warning_code='dead_root' with a message naming
  the unreachable roots; a fully healthy scan or a successful mount check
  clears it, mirroring empty_root. The admin UI shows a badge and banner.
- Deliberate deletion is untouched: removing a path from the library
  config still purges via ListIDsOutsideRoots, files under reachable
  roots keep the exact 24h-grace purge, the empty-root guard and the
  autoscan dead-mount guard are unchanged.

The audiobook/podcast/ebook reconcile paths share the same folder-wide
sweep and orphan purge, so they get the same guard.

Covered by tests: an end-to-end two-root scan (root dies -> rows survive
a zero-grace sweep and warning is set; root returns -> rows resurrect
with their original ids and the warning clears; deleting a file under a
reachable root still purges), repo-level sweep-protection and
sibling-prefix tests, orphan-purge exemption, and rootcheck unit tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(scanner): probe uncompacted roots and take dead-root path on full outage

Review follow-ups: (1) probe every configured path instead of the compacted
traversal roots, so a nested child mount that dies under a reachable parent
is still protected from the sweep; (2) when every configured root is
unreachable, bypass the empty-root confirm flow (without consuming the
one-time cleanup allowance), mark files missing, and raise dead_root instead
of empty_root; (3) dead_root warning banner no longer shows empty-root
confirm-deletion guidance as its fallback hint.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor(scanner): simplify dead-root protection plumbing

- extract pathscope.CoverageClauses as the single builder for the
  exact-path + escaped prefix-LIKE root predicate; scanner's
  rootCoverageClauses delegates to it and catalog's
  excludeOrphansUnderProtectedPrefixes reuses it instead of hand-rolling
  the same clause loop
- extract Scanner.sweepMissingAndReconcile to replace the identical
  trash-sweep + membership-reconcile + S3-image-cleanup block that was
  triplicated across the audiobook, ebook, and podcast scans (callers
  keep their flavor-specific log lines so messages stay constant)
- add unreachableConfiguredRoots helper for the repeated
  probeUnreachableRoots(ctx, folder.ID, cleanScanRoots(folder.Paths))
  expression in scanPaths and ScanFile
- drop the unread Path field from rootcheck.Result
- move the dead/empty-root warning text constants in AdminLibraries.tsx
  out of the middle of the import block

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(scanner): close dead-root protection gaps found in review

Remediates the confirmed findings from the deep review of this PR:

- Scoped audiobook scans (autoscan file events, subtree scans) ran the
  folder-wide sweep while probing only the scoped clone's Paths, so a
  healthy-subtree event could hard-delete a dead sibling root's rows.
  sweepMissingAndReconcile now reloads the folder's configured roots
  from the DB and probes them uncompacted, which also protects nested
  child mounts in the audiobook/ebook/podcast reconcilers.

- A lost mount that leaves an empty, stat-able mountpoint probed as
  reachable and kept the historical purge timeline. A reachable root
  that is a literally empty directory while cataloged rows remain under
  it is now treated as suspect: rows are only marked missing, the sweep
  and orphan purge exempt it, dead_root is raised, and the mount-check
  endpoint reports it (additive suspect_empty field) instead of
  clearing the warning. Arming the one-time empty-cleanup allowance
  completes the deletion, including in the mixed case where other
  roots are healthy. Roots that still have directory entries keep the
  historical grace-then-purge path.

- Confirmed empty cleanup (allow_empty_cleanup_once) no longer
  force-deletes rows under probe-dead roots: an outage is not a
  confirmation, so a dead sibling root's catalog survives a confirmed
  cleanout of a reachable empty root.

- Root probes are now bounded (rootcheck.ProbeWithTimeout, 5s): a hung
  network mount degrades into the protected unreachable path with a
  probe_timeout error code instead of stalling every scan of the
  folder indefinitely.

- Documented the cross-library limitation of the orphan-purge
  exemption next to the query it applies to.

All behavior is pinned by new DB-backed tests (suspect-empty
protection + confirmed completion, confirmed-cleanup dead-root
survival, scoped/nested-root sweep protection, suspect-root query,
probe timeout).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(scanner): address dead-root review findings

---------

Co-authored-by: rxwatcher <rxwatcher@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Quick104 <31828688+Quick104@users.noreply.github.com>
2026-07-16 13:58:44 -04:00

176 lines
5.1 KiB
Go

package scanner
// ScanResult contains the outcome of scanning a media folder.
type ScanResult struct {
New int
Updated int
Unchanged int
Missing int
FilesDeleted int
MembershipsRemoved int
ItemsDeleted int
Errors int
EmptyRootGuarded bool
// UnreachableRoots lists configured library roots that failed the
// reachability probe at scan start. Files under them were marked missing
// (hidden) but were exempted from trash emptying and item purging.
UnreachableRoots []string
// SuspectEmptyRoots lists roots that probed reachable but were literally
// empty directories while the library still holds cataloged files under
// them — the signature of a lost mount exposing its bare mountpoint
// directory. They receive the same cleanup exemptions as
// UnreachableRoots until the operator confirms cleanup or the files
// return.
SuspectEmptyRoots []string
RootObservations []RootObservation
}
// FileHints contains the OSHash gathered during scanning.
type FileHints struct {
FileHash string // OSHash from xattr or computed
}
// ProbeData contains media file technical information from ffprobe.
type ProbeData struct {
CodecVideo string
CodecAudio string
Resolution string // 1080p, 2160p, etc.
AudioChannels int
HDR bool
Container string
Duration int // seconds
Bitrate int // kbps
VideoTracks []VideoTrackInfo
AudioTracks []AudioTrackInfo
SubtitleTracks []SubtitleTrackInfo
Chapters []ChapterInfo
FormatTags map[string]string // format-level tags from ffprobe (title, artist, album, date, etc.)
}
// VideoTrackInfo describes a probed video track.
type VideoTrackInfo struct {
Title string
Codec string
DolbyVision string
DVProfile int
DVBLCompatID int
DVELPresent bool
DVEnhancementLayer string
HDR10Plus bool
Profile string
Level int
Width int
Height int
AspectRatio string
Interlaced bool
FrameRate string
Bitrate int
VideoRange string
VideoRangeType string
ColorPrimaries string
ColorSpace string
ColorTransfer string
BitDepth int
PixelFormat string
ReferenceFrames int
}
// AudioTrackInfo describes a probed audio track.
type AudioTrackInfo struct {
Title string
EmbeddedTitle string
Language string
Codec string
Profile string
Layout string
Channels int
Bitrate int
SampleRate int
BitDepth int
Default bool
}
// SubtitleTrackInfo describes an embedded subtitle track from probing.
type SubtitleTrackInfo struct {
Index int
Language string
Codec string
Title string
EmbeddedTitle string
Resolution string
Forced bool
Default bool
HearingImpaired bool
}
// ExternalSubtitleInfo describes a discovered sidecar subtitle file.
type ExternalSubtitleInfo struct {
Path string
Language string
Format string // srt, vtt, ass, ssa, sub
Title string
Forced bool
}
// ChapterInfo describes an embedded chapter extracted from ffprobe.
type ChapterInfo struct {
Index int
Title string
StartSeconds float64
EndSeconds float64
Source string
}
// IntroCreditsMarkers contains intro/credits timing from S3 markers.
type IntroCreditsMarkers struct {
IntroStart *float64
IntroEnd *float64
CreditsStart *float64
CreditsEnd *float64
}
// MarkerUpdate is the narrow marker-only update payload shared by scanner and analyzers.
type MarkerUpdate struct {
IntroStart *float64
IntroEnd *float64
CreditsStart *float64
CreditsEnd *float64
RecapStart *float64
RecapEnd *float64
PreviewStart *float64
PreviewEnd *float64
MarkersSource string
MarkersProvider *string
MarkersConfidence *float64
MarkersAlgorithm string
// Optional per-segment provenance overrides. When set for a segment,
// UpsertMarkers writes these source/provider/confidence/algorithm values
// for that segment instead of the shared Markers* fields above. Used by
// merged multi-provider results where each segment may come from a
// different provider/source class; left nil for single-source writes
// (scanner/s3/local).
IntroProvenance *SegmentProvenance
CreditsProvenance *SegmentProvenance
RecapProvenance *SegmentProvenance
PreviewProvenance *SegmentProvenance
}
// SegmentProvenance is a per-segment attribution override for MarkerUpdate.
// When Source is empty, the shared MarkerUpdate.MarkersSource applies.
type SegmentProvenance struct {
Source string
Provider *string
Confidence *float64
Algorithm string
}
// HasAnySegment reports whether the update would write at least one segment.
// An update with no segment bounds set is a no-op and skipped by UpsertMarkers.
func (u MarkerUpdate) HasAnySegment() bool {
return u.IntroStart != nil || u.IntroEnd != nil ||
u.CreditsStart != nil || u.CreditsEnd != nil ||
u.RecapStart != nil || u.RecapEnd != nil ||
u.PreviewStart != nil || u.PreviewEnd != nil
}