fix(playback): make edge tracking lifecycle-safe and session identity canonical
Five defects that all produced a WRONG over-cap count, which is why they land before the revocation batch: decision A1 raises the over-cap revocation TTL from 5m to ~24h, removing the self-healing that currently limits the damage of a miscount. A false positive after A1 blocks a legitimate stream for a day, so the count has to be trustworthy first. #1 -- overlapping edge requests deleted a live stream. Tracker.sessions was a set and Remove tore down all state plus the Redis key, while both proxy pour handlers deferred removal unconditionally. Two overlapping Range GETs on one session id -- ordinary seek behaviour -- meant the first to finish deleted the record while the second was still pouring, and later AddBytes calls were then dropped because AddBytes ignores bytes for a session with no live record. The stream went invisible to authoritative monitoring while still serving. Track now returns a Lease that the request-scoped caller releases exactly once; teardown happens when the last live lease is released. A plain refcount would have been wrong: Track(A) -> Remove -> Track(B) -> Release(A) decrements B, and clamping at zero does not help because the count legitimately belongs to B. That is not hypothetical -- the transcode node deliberately replaces sessions under the same id so a quality switch does not orphan ffmpeg, and it calls unconditional Remove from its reaper and stop paths. So each generation carries an epoch, Remove and Cleanup bump it, and a release from a superseded generation is a logged no-op. Lease identity is a set rather than a counter, which makes a duplicate release detectable instead of silently destructive. The transcode node keeps using Remove: its Track calls are not request-scoped and are correctly owned by session lifecycle. "Every Track needs a paired Release" is true only of the request-scoped callers. #8 -- async transcode tracking could leave a permanent ghost. The tracking write ran as a bare goroutine with a WithoutCancel context, so if stop won the race the delayed Track recreated the record after cleanup -- and because it landed in sessions, Snapshot treated it as live until Remove and it NEVER idle-expired. A permanent phantom inflating its owner's count, able to trigger false over-cap kills of that user's real streams. The write now takes the per-session lifecycle lock that stop and reap already hold, and re-checks session pointer identity before writing, so a stopped or replaced generation cannot resurrect a record. Pointer identity rather than id equality is what makes same-id replacement safe. The write stays off the request path -- the API server and the playback client are blocked on the 202. #9 + M3 -- protocol-v3 counted one stream twice. The stream token carries a transport id distinct from the logical session id, and the node tracked under the transport id while the API/proxy record used the logical one, so mergeStreams saw two streams. M3 was the reason this had not yet bitten: the v3 fresh-start caller sent no owner attribution at all, so the transport record landed under user 0, which the enforcer skips -- silently exempting the stream from the cap entirely. Fresh v3 starts now carry the logical session id and full owner attribution (both were already in scope at the call site), and merging is keyed on logical identity where present via one shared helper used by both merge functions, which had already drifted apart once. The enforcer view resolves SessionID to the logical id so a kill targets the real session rather than a replaceable transport generation. The raw admin view keeps the transport id and exposes logical_session_id as an additive omitempty field, advertised on the node-sessions capability endpoint, so the v1 response shape is unchanged. GAP-15 -- edge transcode liveness was request-observed. touchTranscodeSession fired before proxying, so hammering dead segment URLs advanced LastServedAt with zero bytes served. Visibility and liveness are now separate operations: EnsureEphemeral makes a session visible without claiming bytes were served, and served-byte liveness advances only from a 2xx/206 upstream response. Previously the proxy metered every upstream body regardless of status, so a node 404's error body counted as served bytes -- moving the touch later would not have fixed it. S4 -- LiveLocalSessions moved from the HTTP handlers package to streammonitor, which owns monitoring. A background enforcer importing api/handlers was backwards. Pure move; its existing mapping assertions moved with it. The LastActivityAt fallback inside it is left as-is -- decision A5 removes it in the liveness batch. Verified with go test -race across nodesessions, proxy and transcodenode; the overlap regression test was confirmed to fail under the old unconditional teardown. Part of #305.
This commit is contained in:
CoffeeKnyte
@@ -23,43 +23,6 @@ import (
|
||||
"github.com/redis/go-redis/v9"
|
||||
)
|
||||
|
||||
// LiveLocalSessions maps the in-process playback sessions into monitoring
|
||||
// records (the same SessionInfo shape edge nodes write to Redis), so integrated
|
||||
// single-node streams — which never touch Redis — are still visible to the
|
||||
// monitor and the admin active-streams view. Shared by the enforcer's FuncSource
|
||||
// and the admin session list so there is exactly one Session→record mapping.
|
||||
func LiveLocalSessions(sm *playback.SessionManager, nodeName string) []nodesessions.SessionInfo {
|
||||
if sm == nil {
|
||||
return nil
|
||||
}
|
||||
live := sm.AllSessions()
|
||||
out := make([]nodesessions.SessionInfo, 0, len(live))
|
||||
for _, s := range live {
|
||||
lastServedAt := s.LastServedAt
|
||||
if lastServedAt.IsZero() {
|
||||
lastServedAt = s.LastActivityAt
|
||||
}
|
||||
out = append(out, nodesessions.SessionInfo{
|
||||
SessionID: s.ID,
|
||||
NodeName: nodeName,
|
||||
AuthUserID: s.UserID,
|
||||
ProfileID: s.ProfileID,
|
||||
Type: string(s.PlayMethod),
|
||||
Route: s.Origin(),
|
||||
MediaFileID: s.MediaFileID,
|
||||
ClientIP: s.ClientIP,
|
||||
ClientName: s.ClientName,
|
||||
Position: s.Position,
|
||||
Resolution: s.TargetResolution,
|
||||
HWAccel: s.TranscodeHWAccel,
|
||||
StartedAt: s.StartedAt.UTC().Format(time.RFC3339),
|
||||
LastServedAt: lastServedAt.UTC().Format(time.RFC3339),
|
||||
BytesServed: s.BytesServed,
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// NodeRepository defines the operations the NodeHandler needs on the node store.
|
||||
type NodeRepository interface {
|
||||
List(ctx context.Context) ([]*nodepool.Node, error)
|
||||
@@ -422,7 +385,7 @@ func (h *NodeHandler) HandleListSessions(w http.ResponseWriter, r *http.Request)
|
||||
// Integrated single-node streams live only in the in-process session manager.
|
||||
// Include them in the unfiltered listing (a node_id filter targets an edge).
|
||||
if h.sessionMgr != nil && nodeFilter == "" {
|
||||
infos = append(infos, LiveLocalSessions(h.sessionMgr, h.localNodeName)...)
|
||||
infos = append(infos, streammonitor.LiveLocalSessions(h.sessionMgr, h.localNodeName)...)
|
||||
}
|
||||
|
||||
sessions := []json.RawMessage{}
|
||||
@@ -453,6 +416,9 @@ func (h *NodeHandler) HandleListSessions(w http.ResponseWriter, r *http.Request)
|
||||
// deployment can serve the field as an empty list forever. Collapsing the two
|
||||
// would advertise download monitoring that is not actually running.
|
||||
type nodeSessionsCapabilitiesResponse struct {
|
||||
// LogicalSessionID reports that session records may carry the stable logical
|
||||
// identity associated with a replaceable transport generation.
|
||||
LogicalSessionID bool `json:"logical_session_id"`
|
||||
// Transfers reports that the node-sessions payload carries a transfers array.
|
||||
Transfers bool `json:"transfers"`
|
||||
// TransfersActive reports that a transfer registry is wired on this server,
|
||||
@@ -466,8 +432,9 @@ type nodeSessionsCapabilitiesResponse struct {
|
||||
// endpoint's availability rather than being advertised from an unrelated route.
|
||||
func (h *NodeHandler) HandleGetNodeSessionsCapabilities(w http.ResponseWriter, _ *http.Request) {
|
||||
writeJSON(w, http.StatusOK, nodeSessionsCapabilitiesResponse{
|
||||
Transfers: true,
|
||||
TransfersActive: h.transfers != nil,
|
||||
LogicalSessionID: true,
|
||||
Transfers: true,
|
||||
TransfersActive: h.transfers != nil,
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
@@ -118,8 +118,8 @@ func TestNodeSessionsCapabilitiesSeparatesSchemaFromRuntimeWiring(t *testing.T)
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &body); err != nil {
|
||||
t.Fatalf("decode response: %v", err)
|
||||
}
|
||||
if !body.Transfers || !body.TransfersActive {
|
||||
t.Fatalf("capabilities = %+v, want both transfers and transfers_active true", body)
|
||||
if !body.LogicalSessionID || !body.Transfers || !body.TransfersActive {
|
||||
t.Fatalf("capabilities = %+v, want schema flags and transfers_active true", body)
|
||||
}
|
||||
})
|
||||
|
||||
@@ -135,6 +135,9 @@ func TestNodeSessionsCapabilitiesSeparatesSchemaFromRuntimeWiring(t *testing.T)
|
||||
if !body.Transfers {
|
||||
t.Error("transfers = false; the response shape carries the key regardless of wiring")
|
||||
}
|
||||
if !body.LogicalSessionID {
|
||||
t.Error("logical_session_id = false; SessionInfo schema supports the field regardless of wiring")
|
||||
}
|
||||
if body.TransfersActive {
|
||||
t.Error("transfers_active = true with no registry wired; that advertises monitoring that is not running")
|
||||
}
|
||||
|
||||
@@ -2589,6 +2589,7 @@ func (h *PlaybackHandler) HandleChangeAudioTrack(w http.ResponseWriter, r *http.
|
||||
}
|
||||
nodeReq := transcodenode.TranscodeStartRequest{
|
||||
SessionID: restartTransportID,
|
||||
LogicalSessionID: updatedSession.ID,
|
||||
InputPath: file.FilePath,
|
||||
SourceVideoCodec: file.CodecVideo,
|
||||
SeekSeconds: restartSeekSeconds,
|
||||
@@ -3265,6 +3266,7 @@ func (h *PlaybackHandler) HandleStartTranscode(w http.ResponseWriter, r *http.Re
|
||||
}
|
||||
nodeReq := transcodenode.TranscodeStartRequest{
|
||||
SessionID: replacementTransportID,
|
||||
LogicalSessionID: session.ID,
|
||||
InputPath: file.FilePath,
|
||||
SourceVideoCodec: file.CodecVideo,
|
||||
VideoBitstreamFilter: videoBitstreamFilter,
|
||||
|
||||
@@ -766,7 +766,33 @@ func (h *PlaybackHandler) prepareRemoteTransportV3(r *http.Request, session *pla
|
||||
videoCodec = "copy"
|
||||
}
|
||||
seekSeconds, startSegment := configureHLSTimelineV3(result.Plan, videoCodec, 2, float64(file.Duration))
|
||||
req := transcodenode.TranscodeStartRequest{SessionID: transportID, InputPath: file.FilePath, SourceVideoCodec: file.CodecVideo, VideoBitstreamFilter: videoBitstreamFilterForPlanV3(result.Plan), SeekSeconds: seekSeconds, StartSegmentNumber: startSegment, TargetResolution: result.TargetResolution, TargetCodecVideo: videoCodec, TargetCodecAudio: result.TargetAudioCodec, TargetAudioChannels: result.TargetAudioChannels, TargetBitrateKbps: result.TargetBitrateKbps, SegmentDuration: 2, HWAccel: h.playbackConfig().HWAccel, AudioTrackIndex: plannedAudioTrackIndexV3(result, session.AudioTrackIndex), SubtitleTrackIndex: result.SubtitleTransportTrackIndex, SubtitleBurnIn: result.SubtitleBurnIn, SubtitleCodec: result.SubtitleCodec, TotalDuration: float64(file.Duration), RequireReady: true}
|
||||
req := transcodenode.TranscodeStartRequest{
|
||||
SessionID: transportID,
|
||||
LogicalSessionID: session.ID,
|
||||
InputPath: file.FilePath,
|
||||
SourceVideoCodec: file.CodecVideo,
|
||||
VideoBitstreamFilter: videoBitstreamFilterForPlanV3(result.Plan),
|
||||
SeekSeconds: seekSeconds,
|
||||
StartSegmentNumber: startSegment,
|
||||
TargetResolution: result.TargetResolution,
|
||||
TargetCodecVideo: videoCodec,
|
||||
TargetCodecAudio: result.TargetAudioCodec,
|
||||
TargetAudioChannels: result.TargetAudioChannels,
|
||||
TargetBitrateKbps: result.TargetBitrateKbps,
|
||||
SegmentDuration: 2,
|
||||
HWAccel: h.playbackConfig().HWAccel,
|
||||
AudioTrackIndex: plannedAudioTrackIndexV3(result, session.AudioTrackIndex),
|
||||
SubtitleTrackIndex: result.SubtitleTransportTrackIndex,
|
||||
SubtitleBurnIn: result.SubtitleBurnIn,
|
||||
SubtitleCodec: result.SubtitleCodec,
|
||||
TotalDuration: float64(file.Duration),
|
||||
RequireReady: true,
|
||||
AuthUserID: session.UserID,
|
||||
ProfileID: session.ProfileID,
|
||||
MediaFileID: file.ID,
|
||||
Route: session.Origin(),
|
||||
ClientName: session.ClientName,
|
||||
}
|
||||
nodeResp, status, err := h.startRemotePlaybackTransport(r.Context(), node.URL, req)
|
||||
if err != nil {
|
||||
// A timeout can fire after the node actually started the job; the
|
||||
|
||||
@@ -1156,6 +1156,14 @@ func TestPrepareTransportV3RequiresRemoteManifestReadiness(t *testing.T) {
|
||||
if !startRequest.RequireReady {
|
||||
t.Fatal("protocol-v3 remote start did not require manifest readiness")
|
||||
}
|
||||
file := v3HandlerFixtureFile(t)
|
||||
if startRequest.LogicalSessionID != "session-ready" ||
|
||||
startRequest.AuthUserID != 7 ||
|
||||
startRequest.ProfileID != "profile-1" ||
|
||||
startRequest.MediaFileID != file.ID ||
|
||||
startRequest.Route != playback.OriginNative {
|
||||
t.Fatalf("protocol-v3 remote monitoring attribution = %+v", startRequest)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleStartPlaybackUnknownProtocolUsesLegacyBranch(t *testing.T) {
|
||||
|
||||
Reference in New Issue
Block a user