Files
silo-server/internal/playback/transcode_cleanup.go
T
CoffeeKnyteandGitHub b3722dac58 fix(playback): open the listener before sweeping stale transcode dirs (#413)
* fix(playback): run orphaned-transcode cleanup in the background at startup

The native and Jellyfin-compat routers swept stale per-session transcode
dirs synchronously during NewRouter, before the listener bound. On a slow
network filesystem this blocked startup for 80+s (64 leftover dirs on the
last deploy), so restart-reconnect clients were turned away and the health
check reported the server unhealthy the whole time.

Move both sweeps into a background goroutine (StartBackgroundOrphanCleanup)
so the listener comes up immediately and the cleanup runs concurrently. The
delete logic is unchanged: same active-session snapshot and MaxTokenTTL
age-sparing, only later. A package-level mutex serializes concurrent sweeps
of the shared transcode root so the two background sweeps can't race on
os.RemoveAll.

Part of #412

* fix(transcode): background the node boot-time transcode-dir sweep

A dedicated transcode node swept leftover transcode dirs synchronously in
NewServer, before startStandaloneServer bound its listener. On a slow
network filesystem that delete blocked the node from coming online at boot,
the same startup-stall class as the main server.

Move the sweep into the shared StartBackgroundOrphanCleanup goroutine so the
node's listener binds immediately. Backgrounding required an age guard: the
sweep previously ran as a full wipe (minAge=0) with an empty active-set,
which was only safe because it completed before any request could arrive.
Run concurrently that would race a token-carried reconstruct writing into
TranscodeDir/<sessionID>, deleting segments a fresh ffmpeg is producing.
Passing MaxTokenTTL spares any dir younger than the max token lifetime —
exactly the ones a still-valid reconnect could reconstruct — while dirs
older than any surviving token (never reconstructable) are still reclaimed.

Part of #412

* feat(playback): reclaim orphaned transcode dirs periodically, not just at boot

The orphaned-transcode sweep only ran at startup on both the central server
and transcode nodes, so it only ever reclaimed dirs left by an ungraceful
prior shutdown. During a long uptime the in-memory session reapers delete the
dirs of sessions they still track, but a dir whose owning session was dropped
without its RemoveAll succeeding becomes an "untracked orphan" with no runtime
GC — on a box that runs for weeks these accumulate until the next restart.

Add StartPeriodicOrphanCleanup: an immediate background sweep followed by an
hourly re-run bound to a lifecycle context. Wire it on all three surfaces —
native API and Jellyfin-compat (via deps.AppContext) and the transcode node
(via a new Server.StartOrphanSweeper(appCtx), replacing its boot-only sweep).
When no context is supplied (tests) it degrades to a single boot-time sweep so
no ticker goroutine outlives the caller. The sweep stays age-guarded at
MaxTokenTTL, so nothing reconstructable is ever reaped.

Because the node sweep now runs during live traffic, it snapshots the live
job set (Server.activeSessionIDs) and spares those dirs by id rather than by
age alone — a long-lived session that only re-serves already-written segments
stops advancing its dir mtime, which age could otherwise misclassify. In
integrated mode the native and compat sweeps share one TranscodeDir but each
snapshots only its own manager's live set; the resulting cross-manager reap of
a >24h idle dir is bounded (rebuilds from token/recipe) and documented at both
call sites.

Part of #412
2026-07-16 14:48:00 -04:00

153 lines
5.2 KiB
Go

package playback
import (
"context"
"fmt"
"log/slog"
"os"
"path/filepath"
"runtime/debug"
"strings"
"sync"
"time"
)
var orphanCleanupMu sync.Mutex
// CleanupOrphanedTranscodeDirs removes per-session transcode directories that
// are not associated with any currently active session IDs.
//
// minAge spares a dir whose most recent modification is younger than the given
// age even when it is absent from activeSessionIDs. Under token-carried
// reconstruction there is no durable card index, so an in-memory miss does not
// prove a session is dead — a client holding a still-valid token may yet
// reconstruct it. Sparing dirs younger than the maximum token lifetime closes
// that race; once a dir is older than any surviving token, it is safe to reap.
// Pass 0 to disable age-sparing (e.g. a dedicated node's boot-time full wipe,
// where node restart is an accepted session loss).
func CleanupOrphanedTranscodeDirs(root string, activeSessionIDs map[string]struct{}, minAge time.Duration) (int, error) {
// Serialize concurrent orphan sweeps of the shared transcode root so two
// rare startup sweeps cannot race on os.RemoveAll; one global mutex is fine.
orphanCleanupMu.Lock()
defer orphanCleanupMu.Unlock()
if root == "" {
return 0, nil
}
entries, err := os.ReadDir(root)
if err != nil {
if os.IsNotExist(err) {
return 0, nil
}
return 0, fmt.Errorf("read transcode dir %q: %w", root, err)
}
removed := 0
cutoff := time.Now().Add(-minAge)
for _, entry := range entries {
if !entry.IsDir() {
continue
}
if transcodeDirBelongsToActiveSession(entry.Name(), activeSessionIDs) {
continue
}
// The subtitle cache is not session state; it manages its own eviction.
if entry.Name() == subtitleCacheDirName {
continue
}
dir := filepath.Join(root, entry.Name())
if minAge > 0 {
info, statErr := entry.Info()
if statErr != nil {
// Can't verify age: conservatively retain rather than risk
// reaping a dir a surviving token could still reconstruct.
continue
}
if info.ModTime().After(cutoff) {
// Recently active: a surviving token could still reconstruct it.
continue
}
}
if err := os.RemoveAll(dir); err != nil {
return removed, fmt.Errorf("remove orphaned transcode dir %q: %w", dir, err)
}
removed++
}
return removed, nil
}
// OrphanCleanupInterval is how often the periodic orphan sweep re-runs. A dir is
// only reapable once it is older than MaxTokenTTL (24h), so sweeping much more
// often than hourly buys nothing; hourly bounds how long an untracked orphan (a
// dir whose owning session vanished without its RemoveAll succeeding) lingers on
// a process that is never restarted, without adding meaningful load.
const OrphanCleanupInterval = time.Hour
// runOrphanCleanup executes one sweep, recovering from a panic (a background
// goroutine's unrecovered panic would crash the process) and logging the result.
func runOrphanCleanup(component, dir string, cleanup func() (int, error)) {
defer func() {
if r := recover(); r != nil {
slog.Error("transcode cleanup panicked", "component", component, "dir", dir, "panic", r, "stack", string(debug.Stack()))
}
}()
if cleaned, err := cleanup(); err != nil {
slog.Warn("transcode cleanup failed", "component", component, "dir", dir, "error", err)
} else if cleaned > 0 {
slog.Info("transcode cleanup removed orphaned dirs", "component", component, "dir", dir, "count", cleaned)
}
}
// StartBackgroundOrphanCleanup runs a single orphaned-transcode sweep in its own
// goroutine so a slow network-filesystem delete never blocks server startup.
// The sweep is already safe to run concurrently with request handling: it
// spares live/in-flight sessions and any dir younger than MaxTokenTTL, and
// CleanupOrphanedTranscodeDirs serializes concurrent sweeps of the same root.
func StartBackgroundOrphanCleanup(component, dir string, cleanup func() (int, error)) {
go runOrphanCleanup(component, dir, cleanup)
}
// StartPeriodicOrphanCleanup runs an immediate background sweep and then repeats
// it every interval until ctx is cancelled. The startup sweep only reclaims dirs
// orphaned by an ungraceful prior shutdown; the periodic re-run additionally
// bounds "untracked orphan" accumulation (a dir whose owning session was dropped
// without its RemoveAll succeeding) on a process that stays up for weeks. When
// ctx is nil or interval is non-positive it degrades to a single boot-time sweep
// so no ticker goroutine outlives a caller with no lifecycle handle (e.g. tests).
func StartPeriodicOrphanCleanup(ctx context.Context, component, dir string, cleanup func() (int, error), interval time.Duration) {
if ctx == nil || interval <= 0 {
StartBackgroundOrphanCleanup(component, dir, cleanup)
return
}
go func() {
runOrphanCleanup(component, dir, cleanup)
ticker := time.NewTicker(interval)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
runOrphanCleanup(component, dir, cleanup)
}
}
}()
}
func transcodeDirBelongsToActiveSession(name string, activeSessionIDs map[string]struct{}) bool {
if _, ok := activeSessionIDs[name]; ok {
return true
}
for sessionID := range activeSessionIDs {
if sessionID != "" && strings.HasPrefix(name, sessionID+"-") {
return true
}
}
return false
}