* fix(scanner): never purge files under unreachable library roots An unreachable root is not a removed root. When one root of a multi-root library dies (unmounted share, dead drive) while another root still has files, the whole-library empty-root guard does not fire — the surviving root produced files — so the scan marks everything under the dead root missing_since (desired: hides it from browse/playback) and then, with the default scanner.empty_trash_after_scan=true + 24h file_removal_grace, the next scan after the grace hard-deletes every row under the dead root. A week-long drive outage silently destroys the root's entire catalog state: probe data, intro/credits markers, file hashes. Worse, membership reconciliation immediately purges media_items whose only files lived on the dead root, cascading user collections (library_collection_items has ON DELETE CASCADE) and deleting cached artwork. This change makes "temporarily offline" survivable: - Probe each configured root at scan start (os.Stat + IsDir + ReadDir, factored into the new internal/rootcheck package and shared with the admin mount-check endpoint). Unreachable roots are skipped by the walk but their scopes still reconcile, so files are still marked missing. - The trash sweep (DeleteMissingByFolder) now excludes rows whose path sits under an unreachable root, using the same exact-path + escaped prefix-LIKE matching as ListIDsOutsideRoots (a sibling root that merely shares a string prefix is never protected). With all roots reachable the emitted SQL is unchanged. - Membership removal still happens — browse/home hide items via media_item_libraries, so removal is what keeps a dead-root-only title out of the catalog — but the orphan media_items purge exempts items whose files sit under an unreachable root. Their metadata, artwork, and collection links survive; when the root returns, the upsert clears missing_since and syncPresentLibraryState re-inserts the membership, restoring the item with zero re-probing or re-matching. - The folder surfaces scan_warning_code='dead_root' with a message naming the unreachable roots; a fully healthy scan or a successful mount check clears it, mirroring empty_root. The admin UI shows a badge and banner. - Deliberate deletion is untouched: removing a path from the library config still purges via ListIDsOutsideRoots, files under reachable roots keep the exact 24h-grace purge, the empty-root guard and the autoscan dead-mount guard are unchanged. The audiobook/podcast/ebook reconcile paths share the same folder-wide sweep and orphan purge, so they get the same guard. Covered by tests: an end-to-end two-root scan (root dies -> rows survive a zero-grace sweep and warning is set; root returns -> rows resurrect with their original ids and the warning clears; deleting a file under a reachable root still purges), repo-level sweep-protection and sibling-prefix tests, orphan-purge exemption, and rootcheck unit tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(scanner): probe uncompacted roots and take dead-root path on full outage Review follow-ups: (1) probe every configured path instead of the compacted traversal roots, so a nested child mount that dies under a reachable parent is still protected from the sweep; (2) when every configured root is unreachable, bypass the empty-root confirm flow (without consuming the one-time cleanup allowance), mark files missing, and raise dead_root instead of empty_root; (3) dead_root warning banner no longer shows empty-root confirm-deletion guidance as its fallback hint. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(scanner): simplify dead-root protection plumbing - extract pathscope.CoverageClauses as the single builder for the exact-path + escaped prefix-LIKE root predicate; scanner's rootCoverageClauses delegates to it and catalog's excludeOrphansUnderProtectedPrefixes reuses it instead of hand-rolling the same clause loop - extract Scanner.sweepMissingAndReconcile to replace the identical trash-sweep + membership-reconcile + S3-image-cleanup block that was triplicated across the audiobook, ebook, and podcast scans (callers keep their flavor-specific log lines so messages stay constant) - add unreachableConfiguredRoots helper for the repeated probeUnreachableRoots(ctx, folder.ID, cleanScanRoots(folder.Paths)) expression in scanPaths and ScanFile - drop the unread Path field from rootcheck.Result - move the dead/empty-root warning text constants in AdminLibraries.tsx out of the middle of the import block Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(scanner): close dead-root protection gaps found in review Remediates the confirmed findings from the deep review of this PR: - Scoped audiobook scans (autoscan file events, subtree scans) ran the folder-wide sweep while probing only the scoped clone's Paths, so a healthy-subtree event could hard-delete a dead sibling root's rows. sweepMissingAndReconcile now reloads the folder's configured roots from the DB and probes them uncompacted, which also protects nested child mounts in the audiobook/ebook/podcast reconcilers. - A lost mount that leaves an empty, stat-able mountpoint probed as reachable and kept the historical purge timeline. A reachable root that is a literally empty directory while cataloged rows remain under it is now treated as suspect: rows are only marked missing, the sweep and orphan purge exempt it, dead_root is raised, and the mount-check endpoint reports it (additive suspect_empty field) instead of clearing the warning. Arming the one-time empty-cleanup allowance completes the deletion, including in the mixed case where other roots are healthy. Roots that still have directory entries keep the historical grace-then-purge path. - Confirmed empty cleanup (allow_empty_cleanup_once) no longer force-deletes rows under probe-dead roots: an outage is not a confirmation, so a dead sibling root's catalog survives a confirmed cleanout of a reachable empty root. - Root probes are now bounded (rootcheck.ProbeWithTimeout, 5s): a hung network mount degrades into the protected unreachable path with a probe_timeout error code instead of stalling every scan of the folder indefinitely. - Documented the cross-library limitation of the orphan-purge exemption next to the query it applies to. All behavior is pinned by new DB-backed tests (suspect-empty protection + confirmed completion, confirmed-cleanup dead-root survival, scoped/nested-root sweep protection, suspect-root query, probe timeout). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(scanner): address dead-root review findings --------- Co-authored-by: rxwatcher <rxwatcher@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Quick104 <31828688+Quick104@users.noreply.github.com>
176 lines
5.1 KiB
Go
176 lines
5.1 KiB
Go
package scanner
|
|
|
|
// ScanResult contains the outcome of scanning a media folder.
|
|
type ScanResult struct {
|
|
New int
|
|
Updated int
|
|
Unchanged int
|
|
Missing int
|
|
FilesDeleted int
|
|
MembershipsRemoved int
|
|
ItemsDeleted int
|
|
Errors int
|
|
EmptyRootGuarded bool
|
|
// UnreachableRoots lists configured library roots that failed the
|
|
// reachability probe at scan start. Files under them were marked missing
|
|
// (hidden) but were exempted from trash emptying and item purging.
|
|
UnreachableRoots []string
|
|
// SuspectEmptyRoots lists roots that probed reachable but were literally
|
|
// empty directories while the library still holds cataloged files under
|
|
// them — the signature of a lost mount exposing its bare mountpoint
|
|
// directory. They receive the same cleanup exemptions as
|
|
// UnreachableRoots until the operator confirms cleanup or the files
|
|
// return.
|
|
SuspectEmptyRoots []string
|
|
RootObservations []RootObservation
|
|
}
|
|
|
|
// FileHints contains the OSHash gathered during scanning.
|
|
type FileHints struct {
|
|
FileHash string // OSHash from xattr or computed
|
|
}
|
|
|
|
// ProbeData contains media file technical information from ffprobe.
|
|
type ProbeData struct {
|
|
CodecVideo string
|
|
CodecAudio string
|
|
Resolution string // 1080p, 2160p, etc.
|
|
AudioChannels int
|
|
HDR bool
|
|
Container string
|
|
Duration int // seconds
|
|
Bitrate int // kbps
|
|
VideoTracks []VideoTrackInfo
|
|
AudioTracks []AudioTrackInfo
|
|
SubtitleTracks []SubtitleTrackInfo
|
|
Chapters []ChapterInfo
|
|
FormatTags map[string]string // format-level tags from ffprobe (title, artist, album, date, etc.)
|
|
}
|
|
|
|
// VideoTrackInfo describes a probed video track.
|
|
type VideoTrackInfo struct {
|
|
Title string
|
|
Codec string
|
|
DolbyVision string
|
|
DVProfile int
|
|
DVBLCompatID int
|
|
DVELPresent bool
|
|
DVEnhancementLayer string
|
|
HDR10Plus bool
|
|
Profile string
|
|
Level int
|
|
Width int
|
|
Height int
|
|
AspectRatio string
|
|
Interlaced bool
|
|
FrameRate string
|
|
Bitrate int
|
|
VideoRange string
|
|
VideoRangeType string
|
|
ColorPrimaries string
|
|
ColorSpace string
|
|
ColorTransfer string
|
|
BitDepth int
|
|
PixelFormat string
|
|
ReferenceFrames int
|
|
}
|
|
|
|
// AudioTrackInfo describes a probed audio track.
|
|
type AudioTrackInfo struct {
|
|
Title string
|
|
EmbeddedTitle string
|
|
Language string
|
|
Codec string
|
|
Profile string
|
|
Layout string
|
|
Channels int
|
|
Bitrate int
|
|
SampleRate int
|
|
BitDepth int
|
|
Default bool
|
|
}
|
|
|
|
// SubtitleTrackInfo describes an embedded subtitle track from probing.
|
|
type SubtitleTrackInfo struct {
|
|
Index int
|
|
Language string
|
|
Codec string
|
|
Title string
|
|
EmbeddedTitle string
|
|
Resolution string
|
|
Forced bool
|
|
Default bool
|
|
HearingImpaired bool
|
|
}
|
|
|
|
// ExternalSubtitleInfo describes a discovered sidecar subtitle file.
|
|
type ExternalSubtitleInfo struct {
|
|
Path string
|
|
Language string
|
|
Format string // srt, vtt, ass, ssa, sub
|
|
Title string
|
|
Forced bool
|
|
}
|
|
|
|
// ChapterInfo describes an embedded chapter extracted from ffprobe.
|
|
type ChapterInfo struct {
|
|
Index int
|
|
Title string
|
|
StartSeconds float64
|
|
EndSeconds float64
|
|
Source string
|
|
}
|
|
|
|
// IntroCreditsMarkers contains intro/credits timing from S3 markers.
|
|
type IntroCreditsMarkers struct {
|
|
IntroStart *float64
|
|
IntroEnd *float64
|
|
CreditsStart *float64
|
|
CreditsEnd *float64
|
|
}
|
|
|
|
// MarkerUpdate is the narrow marker-only update payload shared by scanner and analyzers.
|
|
type MarkerUpdate struct {
|
|
IntroStart *float64
|
|
IntroEnd *float64
|
|
CreditsStart *float64
|
|
CreditsEnd *float64
|
|
RecapStart *float64
|
|
RecapEnd *float64
|
|
PreviewStart *float64
|
|
PreviewEnd *float64
|
|
MarkersSource string
|
|
MarkersProvider *string
|
|
MarkersConfidence *float64
|
|
MarkersAlgorithm string
|
|
|
|
// Optional per-segment provenance overrides. When set for a segment,
|
|
// UpsertMarkers writes these source/provider/confidence/algorithm values
|
|
// for that segment instead of the shared Markers* fields above. Used by
|
|
// merged multi-provider results where each segment may come from a
|
|
// different provider/source class; left nil for single-source writes
|
|
// (scanner/s3/local).
|
|
IntroProvenance *SegmentProvenance
|
|
CreditsProvenance *SegmentProvenance
|
|
RecapProvenance *SegmentProvenance
|
|
PreviewProvenance *SegmentProvenance
|
|
}
|
|
|
|
// SegmentProvenance is a per-segment attribution override for MarkerUpdate.
|
|
// When Source is empty, the shared MarkerUpdate.MarkersSource applies.
|
|
type SegmentProvenance struct {
|
|
Source string
|
|
Provider *string
|
|
Confidence *float64
|
|
Algorithm string
|
|
}
|
|
|
|
// HasAnySegment reports whether the update would write at least one segment.
|
|
// An update with no segment bounds set is a no-op and skipped by UpsertMarkers.
|
|
func (u MarkerUpdate) HasAnySegment() bool {
|
|
return u.IntroStart != nil || u.IntroEnd != nil ||
|
|
u.CreditsStart != nil || u.CreditsEnd != nil ||
|
|
u.RecapStart != nil || u.RecapEnd != nil ||
|
|
u.PreviewStart != nil || u.PreviewEnd != nil
|
|
}
|