Files
silo-server/internal/playback/prepare_file.go
567cdd1a15 feat(playback): balance transcode sessions across multiple GPUs (#425)
* feat(playback): balance transcode sessions across multiple GPUs

playback.hw_device now accepts a comma-separated render-device list (e.g.
"/dev/dri/renderD128,/dev/dri/renderD129"). Each transcode session resolves
the list to one concrete device at spawn — the present device with the
fewest active GPU sessions, ties keeping list order — and holds that device
for its whole lifetime (seek/audio restarts reuse it); the reservation
releases on session shutdown, idempotently, including early spawn-failure
paths. Software-accel sessions never reserve, so they cannot skew the
balance.

A single configured value keeps the historical pass-through contract and an
empty value still auto-detects, so existing deployments are unaffected.
PickRenderDevice is list-aware too, picking least-loaded without reserving,
which lets the non-session consumers (chapter thumbnails, download
artifacts, transcode nodes) spread load best-effort when given a list.

Motivation: hosts with two identical media GPUs (e.g. dual Arc A310)
previously pinned every session to one device while the second sat idle.

* feat(admin): GPU device picker for playback hw_device

The hw-accel detection endpoint now reports render_device_details — each
render device with a human label derived from its sysfs PCI vendor/device
ids ("Intel GPU (0x56a6)") — and the Playback settings page renders them as
per-device toggles instead of requiring a hand-typed device path. No
selection means auto (first available device); one selection pins every
session; multiple selections balance least-loaded. The stored
playback.hw_device value stays the comma-separated list, written in stable
detection order regardless of click order, and a configured-but-undetected
device stays visible so a temporarily missing GPU is not silently dropped
on save.

* fix(playback): make GPU selection and reservation atomic

Review follow-up: resolveSessionHWDevice previously selected the
least-loaded device and incremented its count in two separate critical
sections, so concurrent session starts could all pick the same device
before any reservation landed. Device presence checks now happen outside
the lock and selection + reservation share one critical section; a
concurrency test asserts an exact split across two devices for eight
simultaneous starts, which the two-step version cannot guarantee.

* refactor(playback): one typed GPU acquisition boundary, release on process exit

Replace the CSV-handling spread across resolveSessionHWDevice and
PickRenderDevice with HWDeviceSet + AcquireHWDevice in hwdevice.go: every
GPU workload resolves exactly one device immediately before spawn.
Balancing is explicitly QSV/VAAPI-only — NVENC addresses GPUs by CUDA
index/UUID, so a multi-entry list warns and uses the first entry instead
of collapsing through the path-presence filter. Sessions now release
their reservation only after ffmpeg has been reaped (shutdown waits on
done first), closing the window where a new start could pick a device
the old process still occupied. Render-device sysfs descriptions move to
gpudetect.go so the allocator file owns only selection/reservation.

* fix(downloads): prepared downloads acquire a GPU through the shared pool

PrepareFile resolves the configured hw_device list to one concrete
device via AcquireHWDevice and holds the reservation until ffmpeg exits
(Run is synchronous, so the deferred release is the process-exit
boundary). Download encodes now participate in the same active-load
accounting as streaming sessions instead of best-effort spreading.

* fix(chapterthumbs): resolve hw_device list per extraction via the shared pool

ExtractFrame acquires one concrete device from AcquireHWDevice for the
hardware attempt (released when the attempt finishes) instead of passing
the raw comma-separated value to ffmpeg as a single device path. The
service stops pre-resolving and caching a device at first use — the raw
configured value flows through and each extraction resolves it.

* fix(transcodenode): fresh starts use this node's configured hw_device

/transcode/start constructed TranscodeOpts with an empty HWDevice, so
fresh sessions auto-detected the first GPU and bypassed the configured
list while reconstructed sessions honored it. Both paths now feed the
node-local config value into StartTranscode's shared resolution.

* feat(admin): node-aware GPU inventory on /admin/system/hw-accel

playback.hw_device is one cluster-wide value consumed by every transcode
node, but the endpoint probed only whichever healthy node had the fewest
jobs — an admin could configure devices that don't exist on the other
nodes. The endpoint now probes every healthy node concurrently and
returns a nodes array (URL, name, resolved accel, devices, or probe
error) alongside the backward-compatible flat fields, and the config doc
states the homogeneous-path contract explicitly.

* feat(admin): GPU picker survives empty detection, warns on node divergence

The picker rows are now the union of detected devices and configured
entries, so configured-but-missing devices stay visible (and
deselectable) when detection returns nothing or an older node omits
render_device_details (plain render_devices paths fall back to a generic
label). Per-node inventories from the hw-accel endpoint drive two
warnings: a banner when responding nodes report different device sets,
and a per-row note listing nodes missing that device. The multi-select
is hidden for NVENC — balancing is QSV/VA-API only — with a notice when
a multi-device value is already stored.

* style: gofmt touched files

* fix(playback): release GPU reservations on process exit

---------

Co-authored-by: rxwatcher <rxwatcher@users.noreply.github.com>
Co-authored-by: Quick <31828688+Quick104@users.noreply.github.com>
2026-08-04 11:33:15 -04:00

146 lines
4.9 KiB
Go

package playback
import (
"context"
"fmt"
"os"
"os/exec"
"path/filepath"
"time"
"github.com/Silo-Server/silo-server/internal/models"
)
// PrepareTarget describes the concrete encode target for a prepared download
// artifact (remux or transcode-to-file).
type PrepareTarget struct {
Container string
CodecVideo string // "copy" for remux, else an encoder codec (e.g. "h264")
CodecAudio string // "copy" or "aac"
Resolution string // "" = keep source resolution (no scale)
AudioTrackIndex int
TargetBitrateKbps int // 0 = encoder default/CRF; >0 caps video bitrate
}
// ResolvePrepareTarget computes the encode target for a remux/transcode download
// of file, reusing Resolve so a download's encoding matches the streaming
// decision for the same client (no duplicated codec logic).
//
// - remux: copy video; copy audio unless the client can't decode it (then AAC);
// keep source resolution.
// - transcode: H.264/AAC, downscaled to the client's max resolution when the
// source exceeds it.
func ResolvePrepareTarget(file *models.MediaFile, format string, caps ClientCapabilities, settings AdminSettings) PrepareTarget {
t := PrepareTarget{Container: "mp4", AudioTrackIndex: -1}
decision := Resolve(file, caps, settings)
if format == "remux" {
t.CodecVideo = "copy"
if decision.TranscodeAudio {
t.CodecAudio = "aac"
} else {
t.CodecAudio = "copy"
}
return t
}
// transcode
t.CodecVideo = "h264"
t.CodecAudio = "aac"
if caps.MaxResolution != "" && resolutionOrder(file.Resolution) > resolutionOrder(caps.MaxResolution) {
t.Resolution = caps.MaxResolution
}
return t
}
// PrepareFile encodes a single finalized MP4 (with a relocated moov atom via
// -movflags +faststart, enabling clean seek/resume) from opts.InputPath. It
// writes to outputPath+".part" and atomically renames on success so a partial
// file is never observable at outputPath. The call blocks until ffmpeg exits.
func PrepareFile(ctx context.Context, opts TranscodeOpts, outputPath string) error {
if outputPath == "" {
return fmt.Errorf("prepare-file: empty output path")
}
if err := os.MkdirAll(filepath.Dir(outputPath), 0o755); err != nil {
return fmt.Errorf("prepare-file: create output dir: %w", err)
}
partPath := outputPath + ".part"
// A reclaimed job overwrites its own .part; ffmpeg -y handles that, but remove
// any stale partial first so a failed prior attempt can't be mistaken for output.
_ = os.Remove(partPath)
// Resolve a multi-device hw_device list to one concrete GPU for this
// encode. Run blocks until ffmpeg exits, so the deferred release fires at
// exactly the process-exit boundary.
opts.HWAccel = resolveEffectiveTranscodeHWAccel(opts)
hwDevice, releaseHWDevice := AcquireHWDevice(opts.HWDevice, opts.HWAccel)
opts.HWDevice = hwDevice
defer releaseHWDevice()
args := buildPrepareFileArgs(opts, partPath)
bin := opts.FFmpegPath
if bin == "" {
bin = ffmpegBinary()
}
cmd := exec.CommandContext(ctx, bin, args...)
stderr := newBoundedTailBuffer(stderrTailMaxBytes)
cmd.Stderr = stderr
cmd.WaitDelay = 3 * time.Second
if err := cmd.Run(); err != nil {
_ = os.Remove(partPath)
if tail := truncateStderr(stderr.String()); tail != "" {
return fmt.Errorf("%w: %w (stderr: %s)", ErrTranscodeFailed, err, tail)
}
return fmt.Errorf("%w: %w", ErrTranscodeFailed, err)
}
if err := os.Rename(partPath, outputPath); err != nil {
_ = os.Remove(partPath)
return fmt.Errorf("prepare-file: finalize artifact: %w", err)
}
return nil
}
// buildPrepareFileArgs constructs single-file ffmpeg args. It mirrors
// buildFFmpegArgs' input/stream/codec/audio/subtitle handling but emits one
// faststart MP4 instead of HLS segments. Full-file output needs no seek or
// segment-boundary keyframes.
func buildPrepareFileArgs(opts TranscodeOpts, outputPath string) []string {
opts.HWAccel = resolveEffectiveTranscodeHWAccel(opts)
isVideoCopy := opts.TargetCodecVideo == "copy"
isAudioCopy := opts.TargetCodecAudio == "copy"
args := []string{"-nostdin", "-hide_banner", "-loglevel", "error"}
if !isVideoCopy {
args = appendHWAccelArgs(args, opts)
}
args = append(args,
"-fflags", "+genpts+fastseek",
"-analyzeduration", "3000000",
"-probesize", "5000000",
)
args = append(args, "-i", opts.InputPath)
args = append(args, "-map_metadata", "-1", "-map_chapters", "-1")
args = appendStreamSelectionArgs(args, opts)
if isVideoCopy {
args = append(args, "-c:v", "copy")
} else {
args = appendVideoArgs(args, opts)
}
if isVideoCopy && !isAudioCopy {
args = append(args, "-threads", "1", "-filter_threads", "1", "-filter_complex_threads", "1")
}
args = appendAudioArgs(args, opts)
if !isVideoCopy {
args = appendVideoFilterArgs(args, opts)
}
// One finalized MP4. +faststart relocates the moov atom in a finalization pass
// (impossible over a pure pipe) so the file is cleanly seekable and resumable.
args = append(args, "-movflags", "+faststart", "-f", "mp4", "-y", outputPath)
return args
}