feat(playback): stream kill switch + async over-cap enforcer
Add the enforcement layer on top of server-observed monitoring: a revocation kill switch that stops any stream within ~120s and keeps it dead, plus an async over-cap enforcer that drives kills off the live monitoring picture — entirely off the per-segment hot path and with no client-protocol change. - internal/streamrevoke: the central kill list. IsRevoked is a pure in-memory lookup safe on the request hot path; a Redis pub/sub + poll mirror keeps edge caches current, and a Postgres durable mirror lets kills survive a server restart AND a Redis flush so a restart-resilient stream cannot be reconstructed and re-served after being killed. A user revocation is a cutoff (kills tokens minted before it, spares post-reauth tokens), not a 24h ban. - internal/streamenforcer: async over-cap brain — reads the monitoring snapshot and per-user limits, selects victims, and collapses every reason (exceeded limit, admin terminate, abuse) to the same action: write a revocation. - Edge + native + jellycompat enforcement: proxy refuses revoked sessions on every request and cuts long direct-play/remux pours mid-stream; the transcode node guards both serve and the reconstruct path so a killed session is never re-spawned after a node restart; jellycompat serve surfaces close their kill-switch coverage holes. - streamtoken.IssuedTime exposes the token iat the user-kill cutoff compares against; token IssuedTime + revocation guards wire through router, downloads, and admin terminate-by-id (with admin-list dedupe). - Restore sendfile zero-copy on direct-play/remux byte counting so the monitor's served-byte accounting does not cost the sendfile fast path. - migrations/sql: stream_revocations durable table. Part of the stream monitoring & kill-switch epic.
This commit is contained in:
@@ -98,6 +98,7 @@ type Server struct {
|
||||
reaperOnce sync.Once
|
||||
mu sync.RWMutex
|
||||
activeJobs atomic.Int32
|
||||
revocation revocationStore
|
||||
|
||||
// reconstructGroup single-flights node-side session reconstruction per session
|
||||
// id so a post-restart wave of concurrent manifest/segment requests for the same
|
||||
@@ -378,6 +379,31 @@ func (s *Server) SetRecipeStore(store recipeStore) {
|
||||
s.recipeStore = store
|
||||
}
|
||||
|
||||
// revocationStore is the subset of *streamrevoke.Store the node consults to
|
||||
// refuse serving (and rebuilding) killed sessions. IsRevoked is a pure in-memory
|
||||
// lookup. Nil disables enforcement. Defense-in-depth: the proxy in front already
|
||||
// enforces, but this guards the reconstruct path so a killed session is never
|
||||
// re-spawned after a node restart.
|
||||
type revocationStore interface {
|
||||
IsRevoked(sessionID string, userID int, startedAt time.Time) bool
|
||||
Refuse(w http.ResponseWriter, sessionID string, userID int, startedAt time.Time) bool
|
||||
}
|
||||
|
||||
// SetRevocationStore wires the kill switch this node consults per request.
|
||||
func (s *Server) SetRevocationStore(store revocationStore) {
|
||||
s.revocation = store
|
||||
}
|
||||
|
||||
// refuseIfRevoked refuses (403) a serve request for a revoked session. It checks
|
||||
// the session key only (no per-segment token parse): the node is always fronted
|
||||
// by the proxy/central, which enforce per-user revocations, and the enforcer +
|
||||
// admin terminate revoke by session id. The reconstruct guard uses the verified
|
||||
// token's user id directly, so per-user kills still block a rebuild after a
|
||||
// node restart.
|
||||
func (s *Server) refuseIfRevoked(w http.ResponseWriter, sessionID string) bool {
|
||||
return s.revocation != nil && s.revocation.Refuse(w, sessionID, 0, time.Time{})
|
||||
}
|
||||
|
||||
// Handler returns the chi.Router with all transcode routes.
|
||||
func (s *Server) Handler() http.Handler {
|
||||
s.startIdleReaper()
|
||||
@@ -623,6 +649,11 @@ func (s *Server) reconstructFromToken(r *http.Request, sessionID string, request
|
||||
"session", sessionID, "playback_session_id", sessionID)
|
||||
return nil
|
||||
}
|
||||
// Never rebuild a killed session: without this guard a node restart would let
|
||||
// reconstruction re-spawn ffmpeg for a session the kill switch already revoked.
|
||||
if s.revocation != nil && s.revocation.IsRevoked(claims.SessionID, claims.UserID, claims.IssuedTime()) {
|
||||
return nil
|
||||
}
|
||||
card := playback.RecipeCardFromClaims(claims)
|
||||
// The token's recipe must be a transcode card for the session id in the URL: a
|
||||
// mismatch is a forged or stale request, and direct/remux cards carry no encode
|
||||
@@ -841,6 +872,10 @@ func (s *Server) handleStop(w http.ResponseWriter, r *http.Request) {
|
||||
func (s *Server) handleManifest(w http.ResponseWriter, r *http.Request) {
|
||||
sessionID := chi.URLParam(r, "session_id")
|
||||
|
||||
if s.refuseIfRevoked(w, sessionID) {
|
||||
return
|
||||
}
|
||||
|
||||
// Lookup and liveness refresh happen atomically so the idle reaper can
|
||||
// never unregister the job between them and tear down a session this
|
||||
// request is about to serve from.
|
||||
@@ -880,6 +915,10 @@ func (s *Server) handleSegment(w http.ResponseWriter, r *http.Request) {
|
||||
sessionID := chi.URLParam(r, "session_id")
|
||||
name := chi.URLParam(r, "name")
|
||||
|
||||
if s.refuseIfRevoked(w, sessionID) {
|
||||
return
|
||||
}
|
||||
|
||||
// Lookup and liveness refresh happen atomically so the idle reaper can
|
||||
// never unregister the job between them and tear down a session this
|
||||
// request is about to serve from.
|
||||
|
||||
Reference in New Issue
Block a user