feat(playback): stream kill switch + async over-cap enforcer

Add the enforcement layer on top of server-observed monitoring: a revocation
kill switch that stops any stream within ~120s and keeps it dead, plus an async
over-cap enforcer that drives kills off the live monitoring picture — entirely
off the per-segment hot path and with no client-protocol change.

- internal/streamrevoke: the central kill list. IsRevoked is a pure in-memory
  lookup safe on the request hot path; a Redis pub/sub + poll mirror keeps edge
  caches current, and a Postgres durable mirror lets kills survive a server
  restart AND a Redis flush so a restart-resilient stream cannot be reconstructed
  and re-served after being killed. A user revocation is a cutoff (kills tokens
  minted before it, spares post-reauth tokens), not a 24h ban.
- internal/streamenforcer: async over-cap brain — reads the monitoring snapshot
  and per-user limits, selects victims, and collapses every reason (exceeded
  limit, admin terminate, abuse) to the same action: write a revocation.
- Edge + native + jellycompat enforcement: proxy refuses revoked sessions on
  every request and cuts long direct-play/remux pours mid-stream; the transcode
  node guards both serve and the reconstruct path so a killed session is never
  re-spawned after a node restart; jellycompat serve surfaces close their
  kill-switch coverage holes.
- streamtoken.IssuedTime exposes the token iat the user-kill cutoff compares
  against; token IssuedTime + revocation guards wire through router, downloads,
  and admin terminate-by-id (with admin-list dedupe).
- Restore sendfile zero-copy on direct-play/remux byte counting so the monitor's
  served-byte accounting does not cost the sendfile fast path.
- migrations/sql: stream_revocations durable table.

Part of the stream monitoring & kill-switch epic.
This commit is contained in:
CoffeeKnyte
2026-07-29 12:26:03 +00:00
parent 22ffaad911
commit 0217cce2df
23 changed files with 2272 additions and 33 deletions
+39
View File
@@ -98,6 +98,7 @@ type Server struct {
reaperOnce sync.Once
mu sync.RWMutex
activeJobs atomic.Int32
revocation revocationStore
// reconstructGroup single-flights node-side session reconstruction per session
// id so a post-restart wave of concurrent manifest/segment requests for the same
@@ -378,6 +379,31 @@ func (s *Server) SetRecipeStore(store recipeStore) {
s.recipeStore = store
}
// revocationStore is the subset of *streamrevoke.Store the node consults to
// refuse serving (and rebuilding) killed sessions. IsRevoked is a pure in-memory
// lookup. Nil disables enforcement. Defense-in-depth: the proxy in front already
// enforces, but this guards the reconstruct path so a killed session is never
// re-spawned after a node restart.
type revocationStore interface {
IsRevoked(sessionID string, userID int, startedAt time.Time) bool
Refuse(w http.ResponseWriter, sessionID string, userID int, startedAt time.Time) bool
}
// SetRevocationStore wires the kill switch this node consults per request.
func (s *Server) SetRevocationStore(store revocationStore) {
s.revocation = store
}
// refuseIfRevoked refuses (403) a serve request for a revoked session. It checks
// the session key only (no per-segment token parse): the node is always fronted
// by the proxy/central, which enforce per-user revocations, and the enforcer +
// admin terminate revoke by session id. The reconstruct guard uses the verified
// token's user id directly, so per-user kills still block a rebuild after a
// node restart.
func (s *Server) refuseIfRevoked(w http.ResponseWriter, sessionID string) bool {
return s.revocation != nil && s.revocation.Refuse(w, sessionID, 0, time.Time{})
}
// Handler returns the chi.Router with all transcode routes.
func (s *Server) Handler() http.Handler {
s.startIdleReaper()
@@ -623,6 +649,11 @@ func (s *Server) reconstructFromToken(r *http.Request, sessionID string, request
"session", sessionID, "playback_session_id", sessionID)
return nil
}
// Never rebuild a killed session: without this guard a node restart would let
// reconstruction re-spawn ffmpeg for a session the kill switch already revoked.
if s.revocation != nil && s.revocation.IsRevoked(claims.SessionID, claims.UserID, claims.IssuedTime()) {
return nil
}
card := playback.RecipeCardFromClaims(claims)
// The token's recipe must be a transcode card for the session id in the URL: a
// mismatch is a forged or stale request, and direct/remux cards carry no encode
@@ -841,6 +872,10 @@ func (s *Server) handleStop(w http.ResponseWriter, r *http.Request) {
func (s *Server) handleManifest(w http.ResponseWriter, r *http.Request) {
sessionID := chi.URLParam(r, "session_id")
if s.refuseIfRevoked(w, sessionID) {
return
}
// Lookup and liveness refresh happen atomically so the idle reaper can
// never unregister the job between them and tear down a session this
// request is about to serve from.
@@ -880,6 +915,10 @@ func (s *Server) handleSegment(w http.ResponseWriter, r *http.Request) {
sessionID := chi.URLParam(r, "session_id")
name := chi.URLParam(r, "name")
if s.refuseIfRevoked(w, sessionID) {
return
}
// Lookup and liveness refresh happen atomically so the idle reaper can
// never unregister the job between them and tear down a session this
// request is about to serve from.