* fix(playback): run orphaned-transcode cleanup in the background at startup The native and Jellyfin-compat routers swept stale per-session transcode dirs synchronously during NewRouter, before the listener bound. On a slow network filesystem this blocked startup for 80+s (64 leftover dirs on the last deploy), so restart-reconnect clients were turned away and the health check reported the server unhealthy the whole time. Move both sweeps into a background goroutine (StartBackgroundOrphanCleanup) so the listener comes up immediately and the cleanup runs concurrently. The delete logic is unchanged: same active-session snapshot and MaxTokenTTL age-sparing, only later. A package-level mutex serializes concurrent sweeps of the shared transcode root so the two background sweeps can't race on os.RemoveAll. Part of #412 * fix(transcode): background the node boot-time transcode-dir sweep A dedicated transcode node swept leftover transcode dirs synchronously in NewServer, before startStandaloneServer bound its listener. On a slow network filesystem that delete blocked the node from coming online at boot, the same startup-stall class as the main server. Move the sweep into the shared StartBackgroundOrphanCleanup goroutine so the node's listener binds immediately. Backgrounding required an age guard: the sweep previously ran as a full wipe (minAge=0) with an empty active-set, which was only safe because it completed before any request could arrive. Run concurrently that would race a token-carried reconstruct writing into TranscodeDir/<sessionID>, deleting segments a fresh ffmpeg is producing. Passing MaxTokenTTL spares any dir younger than the max token lifetime — exactly the ones a still-valid reconnect could reconstruct — while dirs older than any surviving token (never reconstructable) are still reclaimed. Part of #412 * feat(playback): reclaim orphaned transcode dirs periodically, not just at boot The orphaned-transcode sweep only ran at startup on both the central server and transcode nodes, so it only ever reclaimed dirs left by an ungraceful prior shutdown. During a long uptime the in-memory session reapers delete the dirs of sessions they still track, but a dir whose owning session was dropped without its RemoveAll succeeding becomes an "untracked orphan" with no runtime GC — on a box that runs for weeks these accumulate until the next restart. Add StartPeriodicOrphanCleanup: an immediate background sweep followed by an hourly re-run bound to a lifecycle context. Wire it on all three surfaces — native API and Jellyfin-compat (via deps.AppContext) and the transcode node (via a new Server.StartOrphanSweeper(appCtx), replacing its boot-only sweep). When no context is supplied (tests) it degrades to a single boot-time sweep so no ticker goroutine outlives the caller. The sweep stays age-guarded at MaxTokenTTL, so nothing reconstructable is ever reaped. Because the node sweep now runs during live traffic, it snapshots the live job set (Server.activeSessionIDs) and spares those dirs by id rather than by age alone — a long-lived session that only re-serves already-written segments stops advancing its dir mtime, which age could otherwise misclassify. In integrated mode the native and compat sweeps share one TranscodeDir but each snapshots only its own manager's live set; the resulting cross-manager reap of a >24h idle dir is bounded (rebuilds from token/recipe) and documented at both call sites. Part of #412
153 lines
5.2 KiB
Go
153 lines
5.2 KiB
Go
package playback
|
|
|
|
import (
|
|
"context"
|
|
"fmt"
|
|
"log/slog"
|
|
"os"
|
|
"path/filepath"
|
|
"runtime/debug"
|
|
"strings"
|
|
"sync"
|
|
"time"
|
|
)
|
|
|
|
var orphanCleanupMu sync.Mutex
|
|
|
|
// CleanupOrphanedTranscodeDirs removes per-session transcode directories that
|
|
// are not associated with any currently active session IDs.
|
|
//
|
|
// minAge spares a dir whose most recent modification is younger than the given
|
|
// age even when it is absent from activeSessionIDs. Under token-carried
|
|
// reconstruction there is no durable card index, so an in-memory miss does not
|
|
// prove a session is dead — a client holding a still-valid token may yet
|
|
// reconstruct it. Sparing dirs younger than the maximum token lifetime closes
|
|
// that race; once a dir is older than any surviving token, it is safe to reap.
|
|
// Pass 0 to disable age-sparing (e.g. a dedicated node's boot-time full wipe,
|
|
// where node restart is an accepted session loss).
|
|
func CleanupOrphanedTranscodeDirs(root string, activeSessionIDs map[string]struct{}, minAge time.Duration) (int, error) {
|
|
// Serialize concurrent orphan sweeps of the shared transcode root so two
|
|
// rare startup sweeps cannot race on os.RemoveAll; one global mutex is fine.
|
|
orphanCleanupMu.Lock()
|
|
defer orphanCleanupMu.Unlock()
|
|
|
|
if root == "" {
|
|
return 0, nil
|
|
}
|
|
|
|
entries, err := os.ReadDir(root)
|
|
if err != nil {
|
|
if os.IsNotExist(err) {
|
|
return 0, nil
|
|
}
|
|
return 0, fmt.Errorf("read transcode dir %q: %w", root, err)
|
|
}
|
|
|
|
removed := 0
|
|
cutoff := time.Now().Add(-minAge)
|
|
for _, entry := range entries {
|
|
if !entry.IsDir() {
|
|
continue
|
|
}
|
|
|
|
if transcodeDirBelongsToActiveSession(entry.Name(), activeSessionIDs) {
|
|
continue
|
|
}
|
|
|
|
// The subtitle cache is not session state; it manages its own eviction.
|
|
if entry.Name() == subtitleCacheDirName {
|
|
continue
|
|
}
|
|
|
|
dir := filepath.Join(root, entry.Name())
|
|
if minAge > 0 {
|
|
info, statErr := entry.Info()
|
|
if statErr != nil {
|
|
// Can't verify age: conservatively retain rather than risk
|
|
// reaping a dir a surviving token could still reconstruct.
|
|
continue
|
|
}
|
|
if info.ModTime().After(cutoff) {
|
|
// Recently active: a surviving token could still reconstruct it.
|
|
continue
|
|
}
|
|
}
|
|
if err := os.RemoveAll(dir); err != nil {
|
|
return removed, fmt.Errorf("remove orphaned transcode dir %q: %w", dir, err)
|
|
}
|
|
removed++
|
|
}
|
|
|
|
return removed, nil
|
|
}
|
|
|
|
// OrphanCleanupInterval is how often the periodic orphan sweep re-runs. A dir is
|
|
// only reapable once it is older than MaxTokenTTL (24h), so sweeping much more
|
|
// often than hourly buys nothing; hourly bounds how long an untracked orphan (a
|
|
// dir whose owning session vanished without its RemoveAll succeeding) lingers on
|
|
// a process that is never restarted, without adding meaningful load.
|
|
const OrphanCleanupInterval = time.Hour
|
|
|
|
// runOrphanCleanup executes one sweep, recovering from a panic (a background
|
|
// goroutine's unrecovered panic would crash the process) and logging the result.
|
|
func runOrphanCleanup(component, dir string, cleanup func() (int, error)) {
|
|
defer func() {
|
|
if r := recover(); r != nil {
|
|
slog.Error("transcode cleanup panicked", "component", component, "dir", dir, "panic", r, "stack", string(debug.Stack()))
|
|
}
|
|
}()
|
|
if cleaned, err := cleanup(); err != nil {
|
|
slog.Warn("transcode cleanup failed", "component", component, "dir", dir, "error", err)
|
|
} else if cleaned > 0 {
|
|
slog.Info("transcode cleanup removed orphaned dirs", "component", component, "dir", dir, "count", cleaned)
|
|
}
|
|
}
|
|
|
|
// StartBackgroundOrphanCleanup runs a single orphaned-transcode sweep in its own
|
|
// goroutine so a slow network-filesystem delete never blocks server startup.
|
|
// The sweep is already safe to run concurrently with request handling: it
|
|
// spares live/in-flight sessions and any dir younger than MaxTokenTTL, and
|
|
// CleanupOrphanedTranscodeDirs serializes concurrent sweeps of the same root.
|
|
func StartBackgroundOrphanCleanup(component, dir string, cleanup func() (int, error)) {
|
|
go runOrphanCleanup(component, dir, cleanup)
|
|
}
|
|
|
|
// StartPeriodicOrphanCleanup runs an immediate background sweep and then repeats
|
|
// it every interval until ctx is cancelled. The startup sweep only reclaims dirs
|
|
// orphaned by an ungraceful prior shutdown; the periodic re-run additionally
|
|
// bounds "untracked orphan" accumulation (a dir whose owning session was dropped
|
|
// without its RemoveAll succeeding) on a process that stays up for weeks. When
|
|
// ctx is nil or interval is non-positive it degrades to a single boot-time sweep
|
|
// so no ticker goroutine outlives a caller with no lifecycle handle (e.g. tests).
|
|
func StartPeriodicOrphanCleanup(ctx context.Context, component, dir string, cleanup func() (int, error), interval time.Duration) {
|
|
if ctx == nil || interval <= 0 {
|
|
StartBackgroundOrphanCleanup(component, dir, cleanup)
|
|
return
|
|
}
|
|
go func() {
|
|
runOrphanCleanup(component, dir, cleanup)
|
|
ticker := time.NewTicker(interval)
|
|
defer ticker.Stop()
|
|
for {
|
|
select {
|
|
case <-ctx.Done():
|
|
return
|
|
case <-ticker.C:
|
|
runOrphanCleanup(component, dir, cleanup)
|
|
}
|
|
}
|
|
}()
|
|
}
|
|
|
|
func transcodeDirBelongsToActiveSession(name string, activeSessionIDs map[string]struct{}) bool {
|
|
if _, ok := activeSessionIDs[name]; ok {
|
|
return true
|
|
}
|
|
for sessionID := range activeSessionIDs {
|
|
if sessionID != "" && strings.HasPrefix(name, sessionID+"-") {
|
|
return true
|
|
}
|
|
}
|
|
return false
|
|
}
|