Files
silo-server/internal/rootcheck/rootcheck_test.go
T
8fc054c15d fix(scanner): never purge files under unreachable library roots (#372)
* fix(scanner): never purge files under unreachable library roots

An unreachable root is not a removed root. When one root of a multi-root
library dies (unmounted share, dead drive) while another root still has
files, the whole-library empty-root guard does not fire — the surviving
root produced files — so the scan marks everything under the dead root
missing_since (desired: hides it from browse/playback) and then, with the
default scanner.empty_trash_after_scan=true + 24h file_removal_grace, the
next scan after the grace hard-deletes every row under the dead root. A
week-long drive outage silently destroys the root's entire catalog state:
probe data, intro/credits markers, file hashes. Worse, membership
reconciliation immediately purges media_items whose only files lived on
the dead root, cascading user collections (library_collection_items has
ON DELETE CASCADE) and deleting cached artwork.

This change makes "temporarily offline" survivable:

- Probe each configured root at scan start (os.Stat + IsDir + ReadDir,
  factored into the new internal/rootcheck package and shared with the
  admin mount-check endpoint). Unreachable roots are skipped by the walk
  but their scopes still reconcile, so files are still marked missing.
- The trash sweep (DeleteMissingByFolder) now excludes rows whose path
  sits under an unreachable root, using the same exact-path + escaped
  prefix-LIKE matching as ListIDsOutsideRoots (a sibling root that merely
  shares a string prefix is never protected). With all roots reachable
  the emitted SQL is unchanged.
- Membership removal still happens — browse/home hide items via
  media_item_libraries, so removal is what keeps a dead-root-only title
  out of the catalog — but the orphan media_items purge exempts items
  whose files sit under an unreachable root. Their metadata, artwork,
  and collection links survive; when the root returns, the upsert clears
  missing_since and syncPresentLibraryState re-inserts the membership,
  restoring the item with zero re-probing or re-matching.
- The folder surfaces scan_warning_code='dead_root' with a message naming
  the unreachable roots; a fully healthy scan or a successful mount check
  clears it, mirroring empty_root. The admin UI shows a badge and banner.
- Deliberate deletion is untouched: removing a path from the library
  config still purges via ListIDsOutsideRoots, files under reachable
  roots keep the exact 24h-grace purge, the empty-root guard and the
  autoscan dead-mount guard are unchanged.

The audiobook/podcast/ebook reconcile paths share the same folder-wide
sweep and orphan purge, so they get the same guard.

Covered by tests: an end-to-end two-root scan (root dies -> rows survive
a zero-grace sweep and warning is set; root returns -> rows resurrect
with their original ids and the warning clears; deleting a file under a
reachable root still purges), repo-level sweep-protection and
sibling-prefix tests, orphan-purge exemption, and rootcheck unit tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(scanner): probe uncompacted roots and take dead-root path on full outage

Review follow-ups: (1) probe every configured path instead of the compacted
traversal roots, so a nested child mount that dies under a reachable parent
is still protected from the sweep; (2) when every configured root is
unreachable, bypass the empty-root confirm flow (without consuming the
one-time cleanup allowance), mark files missing, and raise dead_root instead
of empty_root; (3) dead_root warning banner no longer shows empty-root
confirm-deletion guidance as its fallback hint.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor(scanner): simplify dead-root protection plumbing

- extract pathscope.CoverageClauses as the single builder for the
  exact-path + escaped prefix-LIKE root predicate; scanner's
  rootCoverageClauses delegates to it and catalog's
  excludeOrphansUnderProtectedPrefixes reuses it instead of hand-rolling
  the same clause loop
- extract Scanner.sweepMissingAndReconcile to replace the identical
  trash-sweep + membership-reconcile + S3-image-cleanup block that was
  triplicated across the audiobook, ebook, and podcast scans (callers
  keep their flavor-specific log lines so messages stay constant)
- add unreachableConfiguredRoots helper for the repeated
  probeUnreachableRoots(ctx, folder.ID, cleanScanRoots(folder.Paths))
  expression in scanPaths and ScanFile
- drop the unread Path field from rootcheck.Result
- move the dead/empty-root warning text constants in AdminLibraries.tsx
  out of the middle of the import block

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(scanner): close dead-root protection gaps found in review

Remediates the confirmed findings from the deep review of this PR:

- Scoped audiobook scans (autoscan file events, subtree scans) ran the
  folder-wide sweep while probing only the scoped clone's Paths, so a
  healthy-subtree event could hard-delete a dead sibling root's rows.
  sweepMissingAndReconcile now reloads the folder's configured roots
  from the DB and probes them uncompacted, which also protects nested
  child mounts in the audiobook/ebook/podcast reconcilers.

- A lost mount that leaves an empty, stat-able mountpoint probed as
  reachable and kept the historical purge timeline. A reachable root
  that is a literally empty directory while cataloged rows remain under
  it is now treated as suspect: rows are only marked missing, the sweep
  and orphan purge exempt it, dead_root is raised, and the mount-check
  endpoint reports it (additive suspect_empty field) instead of
  clearing the warning. Arming the one-time empty-cleanup allowance
  completes the deletion, including in the mixed case where other
  roots are healthy. Roots that still have directory entries keep the
  historical grace-then-purge path.

- Confirmed empty cleanup (allow_empty_cleanup_once) no longer
  force-deletes rows under probe-dead roots: an outage is not a
  confirmation, so a dead sibling root's catalog survives a confirmed
  cleanout of a reachable empty root.

- Root probes are now bounded (rootcheck.ProbeWithTimeout, 5s): a hung
  network mount degrades into the protected unreachable path with a
  probe_timeout error code instead of stalling every scan of the
  folder indefinitely.

- Documented the cross-library limitation of the orphan-purge
  exemption next to the query it applies to.

All behavior is pinned by new DB-backed tests (suspect-empty
protection + confirmed completion, confirmed-cleanup dead-root
survival, scoped/nested-root sweep protection, suspect-root query,
probe timeout).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(scanner): address dead-root review findings

---------

Co-authored-by: rxwatcher <rxwatcher@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Quick104 <31828688+Quick104@users.noreply.github.com>
2026-07-16 13:58:44 -04:00

203 lines
5.2 KiB
Go

package rootcheck
import (
"context"
"os"
"path/filepath"
"sync/atomic"
"testing"
"time"
)
func TestProbeReachableDirectory(t *testing.T) {
t.Parallel()
res := Probe(t.TempDir())
if !res.Reachable {
t.Fatalf("Probe(temp dir) = %+v, want reachable", res)
}
if res.ErrorCode != "" || res.ErrorMessage != "" {
t.Fatalf("Probe(temp dir) error fields = %q/%q, want empty", res.ErrorCode, res.ErrorMessage)
}
}
func TestProbeMissingPath(t *testing.T) {
t.Parallel()
res := Probe(filepath.Join(t.TempDir(), "does-not-exist"))
if res.Reachable {
t.Fatal("Probe(missing path) reported reachable")
}
if res.ErrorCode != ErrCodeNotFound {
t.Fatalf("ErrorCode = %q, want %q", res.ErrorCode, ErrCodeNotFound)
}
}
func TestProbeRegularFile(t *testing.T) {
t.Parallel()
path := filepath.Join(t.TempDir(), "file.txt")
if err := os.WriteFile(path, []byte("x"), 0o644); err != nil {
t.Fatalf("write file: %v", err)
}
res := Probe(path)
if res.Reachable {
t.Fatal("Probe(regular file) reported reachable")
}
if res.ErrorCode != ErrCodeNotDirectory {
t.Fatalf("ErrorCode = %q, want %q", res.ErrorCode, ErrCodeNotDirectory)
}
}
func TestProbeUnreadableDirectory(t *testing.T) {
t.Parallel()
if os.Getuid() == 0 {
t.Skip("running as root; permission bits are not enforced")
}
path := filepath.Join(t.TempDir(), "locked")
if err := os.Mkdir(path, 0o000); err != nil {
t.Fatalf("mkdir: %v", err)
}
t.Cleanup(func() { _ = os.Chmod(path, 0o755) })
res := Probe(path)
if res.Reachable {
t.Fatal("Probe(unreadable dir) reported reachable")
}
if res.ErrorCode != ErrCodePermissionDenied {
t.Fatalf("ErrorCode = %q, want %q", res.ErrorCode, ErrCodePermissionDenied)
}
}
func TestProbeWithTimeoutReachableDirectory(t *testing.T) {
t.Parallel()
res := ProbeWithTimeout(context.Background(), t.TempDir(), DefaultProbeTimeout)
if !res.Reachable {
t.Fatalf("ProbeWithTimeout(temp dir) = %+v, want reachable", res)
}
}
func TestProbeBoundedTimesOutOnHungProbe(t *testing.T) {
t.Parallel()
release := make(chan struct{})
t.Cleanup(func() { close(release) })
res := probeBounded(context.Background(), 10*time.Millisecond, func() Result {
<-release // simulate a stat/readdir blocked on a hung mount
return Result{Reachable: true}
})
if res.Reachable {
t.Fatal("hung probe reported reachable")
}
if res.ErrorCode != ErrCodeTimeout {
t.Fatalf("ErrorCode = %q, want %q", res.ErrorCode, ErrCodeTimeout)
}
}
func TestProbeBoundedHonorsContextCancellation(t *testing.T) {
t.Parallel()
ctx, cancel := context.WithCancel(context.Background())
cancel()
release := make(chan struct{})
t.Cleanup(func() { close(release) })
res := probeBounded(ctx, time.Minute, func() Result {
<-release
return Result{Reachable: true}
})
if res.Reachable || res.ErrorCode != ErrCodeTimeout {
t.Fatalf("canceled probe = %+v, want unreachable/%s", res, ErrCodeTimeout)
}
}
func TestProbeBoundedReturnsFastResult(t *testing.T) {
t.Parallel()
res := probeBounded(context.Background(), time.Minute, func() Result {
return Result{Reachable: false, ErrorCode: ErrCodeNotFound, ErrorMessage: "Path does not exist"}
})
if res.Reachable || res.ErrorCode != ErrCodeNotFound {
t.Fatalf("probeBounded passthrough = %+v, want the probe's own result", res)
}
}
func TestProbeCoordinatorCoalescesBlockedPath(t *testing.T) {
t.Parallel()
var calls atomic.Int32
release := make(chan struct{})
t.Cleanup(func() { close(release) })
coordinator := newProbeCoordinator()
probe := func(string) Result {
calls.Add(1)
<-release
return Result{Reachable: true}
}
results := make(chan Result, 2)
for range 2 {
go func() {
results <- coordinator.probe(context.Background(), "/hung", 20*time.Millisecond, probe)
}()
}
for range 2 {
if result := <-results; result.Reachable || result.ErrorCode != ErrCodeTimeout {
t.Fatalf("coalesced blocked probe = %+v, want timeout", result)
}
}
if got := calls.Load(); got != 1 {
t.Fatalf("underlying probe calls = %d, want 1", got)
}
}
func TestProbeManyPreservesOrderAndBoundsConcurrency(t *testing.T) {
t.Parallel()
var active atomic.Int32
var peak atomic.Int32
probe := func(_ context.Context, path string, _ time.Duration) Result {
current := active.Add(1)
for {
observed := peak.Load()
if current <= observed || peak.CompareAndSwap(observed, current) {
break
}
}
time.Sleep(10 * time.Millisecond)
active.Add(-1)
return Result{Reachable: true, ErrorMessage: path}
}
paths := []string{"one", "two", "three", "four"}
results := probeMany(context.Background(), paths, time.Second, 2, probe)
if got := peak.Load(); got > 2 {
t.Fatalf("peak concurrent probes = %d, want at most 2", got)
}
for i, result := range results {
if result.ErrorMessage != paths[i] {
t.Fatalf("result[%d] = %q, want %q", i, result.ErrorMessage, paths[i])
}
}
}
func TestProbeReportsEmptyDirectory(t *testing.T) {
t.Parallel()
dir := t.TempDir()
res := Probe(dir)
if !res.Reachable || !res.Empty {
t.Fatalf("Probe(empty dir) = %+v, want reachable and empty", res)
}
if err := os.Mkdir(filepath.Join(dir, "sub"), 0o755); err != nil {
t.Fatalf("mkdir: %v", err)
}
res = Probe(dir)
if !res.Reachable || res.Empty {
t.Fatalf("Probe(non-empty dir) = %+v, want reachable and not empty", res)
}
}