Feature/failover (#924)

* feature stream failover
This commit is contained in:
euzu
2026-10-16 21:10:07 +02:00
committed by GitHub
parent 86d19f955c
commit 0e4e01409c
203 changed files with 15664 additions and 820 deletions
+22
View File
@@ -186,6 +186,16 @@
## 🌟 New Features
- **Channel groups with stream failover.** A target with `options.channel_groups.enabled` collapses equal live
channels from several inputs (matched by caption, name, title or EPG id) into one channel. A start picks the best
member by QoS history, probed quality and free provider capacity and moves on to the next member when one fails.
While a client watches an MPEG-TS stream, a member that ends, fails, stops sending data or is too slow is replaced
underneath the open response, spliced at PAT/PMT and a keyframe; this works for direct and shared streams. Failing
members and inputs are paused for a while, and an outage limit and a switch limit bound the switching. The settings
live once in `config.yml` (`channel_groups`, Web UI: Settings → Channel groups); a target turns the feature on and
can override matching, source order, grace and failover on its own Channel groups page. Off by default. See
[Channel Groups & Stream Failover](docs/src/configuration/channel-groups.md).
- **Maintenance mode.** Inputs can be put into maintenance from the new dashboard card or through
`/api/v1/maintenance` (`*` addresses all inputs). New streams of an input in maintenance get the new
`maintenance.ts` custom video; playlist and EPG updates keep the last persisted state, and probes, metadata
@@ -1599,6 +1609,18 @@
## ⚙️ New Settings
- **config.yml (`channel_groups`)**:
- Added the global channel group settings: `match_by`, `match_as_ascii`, `match_normalize`, `epg_conflict`,
`name_strip_quality`, `policy`, `allow_provider_grace`, `start` (`timeout_secs`, `max_attempts`), `failover`
(`enabled`, `no_data_timeout_secs`, `slow_source_pct`, `slow_source_secs`, `max_outage_secs`,
`same_source_retries`, `stable_after_secs`, `max_switches`, `max_switches_window_secs`, `audio_language`) and
`circuit` (`source_failures`, `cooldown_secs`, `cooldown_max_secs`, `input_failures`, `input_window_secs`). See
[Channel Groups & Stream Failover](docs/src/configuration/channel-groups.md).
- **source.yml (target `options`)**:
- Added `channel_groups` (default off): `enabled` plus the optional overrides `match_by`, `match_as_ascii`,
`epg_conflict`, `name_strip_quality`, `policy`, `allow_provider_grace` and `failover`.
- **config.yml (`reverse_proxy.stream`)**:
- `hls_wrap_media_playlist` (default `true`): answers a proxied live HLS entry request whose provider returns a media
playlist with a single-variant master playlist, so playlist refreshes keep their provider account across client
Generated
+2
View File
@@ -4917,6 +4917,8 @@ dependencies = [
"lru",
"rand 0.9.5",
"reqwest",
"serde",
"serde_json",
"shared",
"tempfile",
"thiserror 2.0.20",
+4
View File
@@ -138,6 +138,7 @@ Tuliprox is not just a metadata processor — it actively manages stream deliver
- **Redirect Mode** — avoid proxy overhead where direct delivery is preferred
- **Shared MPEG-TS Streams** — several viewers can reuse one upstream provider stream
- **Shared HLS Sessions** — reuse one server-side HLS session while retaining individual user leases and accounting
- **Channel Groups** — one channel per live program across inputs, with source selection and runtime failover
- **HLS Segment Cache & Prefetch** — fetch manifests, initialization maps, and segments once and serve them locally
- **HLS Recovery** — optional manifest recovery, bounded segment repair, stale-origin recovery, and diagnostics for difficult upstream streams
- **Universal Catchup** — expose provider archive/catchup through regular M3U-compatible `.ts` URLs
@@ -166,6 +167,9 @@ reconnects, probes, and background jobs.
Upstream providers are not always reliable. Tuliprox is designed to preserve usable service when they fail partially or temporarily.
- Automatic provider URL failover on timeouts and server errors
- **Channel groups with stream failover** collapse the same live channel from several providers or accounts into one
channel; playback starts on the best reachable source and, for MPEG-TS, switches to another source while the client
keeps watching
- Choose whether a provider resumes from the last working URL or restarts from the first URL on future requests
- `provider://` references resolve to the currently active provider URL
- Asynchronous DNS resolution and caching
+2
View File
@@ -109,6 +109,8 @@ tuliprox-btree = { workspace = true, features = ["test-support"] }
tuliprox-mpegts = { workspace = true, features = ["test-support"] }
# API tests build a GeoIp that answers a fixed country.
tuliprox-repository = { workspace = true, features = ["test-support"] }
# Channel group selection tests seed pending provider releases.
tuliprox-session = { workspace = true, features = ["test-support"] }
tokio = { workspace = true, features = ["test-util", "macros"] }
rcgen = "0.14.9"
rustls = "0.23.45"
@@ -0,0 +1,761 @@
use super::{create_stream_response_details, resolve_initial_stalker_playback_url};
use crate::{
api::model::{AppState, BoxedProviderStream, ProviderStreamCustomReason, StreamDetails, StreamError},
auth::Fingerprint,
model::{ConfigInput, ConfigTarget, ProxyUserCredentials},
};
use axum::http::HeaderMap;
use bytes::Bytes;
use futures::StreamExt;
use shared::model::{
ChannelGroupPlayback, ChannelGroupPolicy, ChannelGroupStartConfig, PlaylistItemType, StreamChannel,
UserConnectionPermission, VirtualId,
};
use std::{collections::VecDeque, future::Future, net::SocketAddr, sync::Arc, time::Duration};
use tokio::sync::Notify;
use tuliprox_repository::channel_group_repository::{ChannelGroupRecord, GroupMember};
pub(crate) use tuliprox_session::channel_group::{member_key, CircuitAdmission, CircuitView, SelectionMode};
use tuliprox_session::{
active_provider_manager::{CapacityTier, ConnectionKind, ManagedProviderHandle, PlaybackLeaseRef},
channel_group::{rank_members, FailureClass, HalfOpenGuard, MemberHealthIndex, RankedCandidate},
stream_options::StreamOptions,
};
pub(crate) enum PlaybackSource<'a> {
Single { input: &'a Arc<ConfigInput>, stream_url: &'a str },
Group { group: Arc<ChannelGroupRecord>, mode: SelectionMode, prefer: Option<usize> },
}
pub(crate) struct ResolvedGroupPlayback {
pub acquired: AcquiredMember,
pub member: GroupMember,
pub input: Arc<ConfigInput>,
pub stream_url: Arc<str>,
}
pub(crate) fn group_shared_key(target_id: u16, group_vid: u32) -> String { format!("group:{target_id}:{group_vid}") }
pub(crate) struct GroupSelectionRequest<'a> {
pub target_name: &'a Arc<str>,
pub group: &'a ChannelGroupRecord,
pub policy: ChannelGroupPolicy,
pub mode: SelectionMode,
pub start: ChannelGroupStartConfig,
pub addr: &'a SocketAddr,
pub username: &'a str,
pub priority: i8,
pub kind: ConnectionKind,
pub lease: Option<PlaybackLeaseRef<'a>>,
pub allow_provider_grace: bool,
pub exclude: &'a [usize],
pub demoted_inputs: &'a [Arc<str>],
pub prefer: Option<usize>,
/// Recording proxy users play inputs in maintenance, as for single-source playback.
pub bypass_maintenance: bool,
/// Runtime switches must not evict other playbacks; only a start may preempt.
pub allow_preempt: bool,
}
impl<'a> GroupSelectionRequest<'a> {
/// A start selection: live mode, grace and preemption allowed, nothing excluded.
pub(crate) fn new(
target_name: &'a Arc<str>,
group: &'a ChannelGroupRecord,
start: ChannelGroupStartConfig,
addr: &'a SocketAddr,
username: &'a str,
priority: i8,
kind: ConnectionKind,
) -> Self {
Self {
target_name,
group,
policy: group.policy,
mode: SelectionMode::Live,
start,
addr,
username,
priority,
kind,
lease: None,
allow_provider_grace: true,
exclude: &[],
demoted_inputs: &[],
prefer: None,
bypass_maintenance: false,
allow_preempt: true,
}
}
}
pub(crate) struct AcquiredMember {
pub index: usize,
pub handle: ManagedProviderHandle,
pub tier: CapacityTier,
pub half_open: Option<HalfOpenGuard>,
}
#[derive(Debug, PartialEq, Eq)]
pub(crate) enum GroupSelectionFailure {
Exhausted,
NoCandidate,
/// Every remaining candidate's input is in maintenance.
Maintenance,
}
pub(crate) async fn acquire_group_member(
app_state: &Arc<AppState>,
req: GroupSelectionRequest<'_>,
) -> Result<AcquiredMember, GroupSelectionFailure> {
acquire_group_member_inner(app_state, req, None::<fn(&Arc<AppState>)>).await
}
#[cfg(test)]
pub(crate) async fn acquire_group_member_with_race<F: FnOnce(&Arc<AppState>)>(
app_state: &Arc<AppState>,
req: GroupSelectionRequest<'_>,
after_probe: F,
) -> Result<AcquiredMember, GroupSelectionFailure> {
acquire_group_member_inner(app_state, req, Some(after_probe)).await
}
async fn acquire_group_member_inner<F: FnOnce(&Arc<AppState>)>(
app_state: &Arc<AppState>,
req: GroupSelectionRequest<'_>,
after_probe: Option<F>,
) -> Result<AcquiredMember, GroupSelectionFailure> {
let budget_secs = req.start.timeout_secs;
if budget_secs > 0 {
match tokio::time::timeout(Duration::from_secs(budget_secs), select_member_loop(app_state, req, after_probe))
.await
{
Ok(result) => result,
Err(_) => Err(GroupSelectionFailure::Exhausted),
}
} else {
select_member_loop(app_state, req, after_probe).await
}
}
fn check_member_circuit(
app_state: &Arc<AppState>,
req: &GroupSelectionRequest<'_>,
cand_idx: usize,
first_remaining_idx: usize,
all_circuits_open: bool,
) -> (bool, Option<HalfOpenGuard>) {
let key = member_key(&req.group.members[cand_idx]);
let admission = app_state.member_health.admit(&key, std::time::Instant::now());
match admission {
CircuitAdmission::Closed => (true, None),
CircuitAdmission::HalfOpen(guard) => (true, Some(guard)),
CircuitAdmission::Open => (all_circuits_open && cand_idx == first_remaining_idx, None),
}
}
fn select_tier_candidates<F>(
app_state: &Arc<AppState>,
req: &GroupSelectionRequest<'_>,
remaining: &mut Vec<RankedCandidate>,
probed_tiers: &mut Vec<(usize, CapacityTier)>,
all_circuits_open: bool,
filter_tier: F,
probe_tier: &dyn Fn(usize) -> CapacityTier,
) -> Option<AcquiredMember>
where
F: Fn(CapacityTier) -> Option<bool>,
{
loop {
let mut tried_candidate = false;
let candidate_indices: Vec<(usize, bool)> = probed_tiers
.iter()
.filter_map(|(idx, tier)| filter_tier(*tier).map(|allow_grace| (*idx, allow_grace)))
.collect();
for (cand_idx, allow_grace) in candidate_indices {
let Some((_, tier)) = probed_tiers.iter().find(|(idx, _)| *idx == cand_idx).copied() else {
continue;
};
let (can_try, half_open_guard) =
check_member_circuit(app_state, req, cand_idx, remaining[0].index, all_circuits_open);
if !can_try {
continue;
}
tried_candidate = true;
let handle_opt = app_state.active_provider.try_acquire_connection_no_preempt(
&req.group.members[cand_idx].input_name,
allow_grace,
req.addr,
req.priority,
req.kind,
req.lease,
);
if let Some(handle) = handle_opt {
return Some(AcquiredMember {
index: cand_idx,
handle: ManagedProviderHandle::new(Arc::clone(&app_state.active_provider), handle),
tier,
half_open: half_open_guard,
});
}
remaining.retain(|c| c.index != cand_idx);
*probed_tiers = remaining.iter().map(|c| (c.index, probe_tier(c.index))).collect();
break;
}
if !tried_candidate {
break;
}
}
None
}
async fn try_acquire_preempt(
app_state: &Arc<AppState>,
req: &GroupSelectionRequest<'_>,
remaining: &[RankedCandidate],
all_circuits_open: bool,
) -> Option<AcquiredMember> {
for c in remaining {
let (can_try, half_open_guard) =
check_member_circuit(app_state, req, c.index, remaining[0].index, all_circuits_open);
if !can_try {
continue;
}
let handle_opt = app_state
.active_provider
.acquire_connection_with_lease_for_session_await(
&req.group.members[c.index].input_name,
req.addr,
req.allow_provider_grace,
req.priority,
req.kind,
req.lease,
)
.await;
if let Some(handle) = handle_opt {
return Some(AcquiredMember {
index: c.index,
handle: ManagedProviderHandle::new(Arc::clone(&app_state.active_provider), handle),
tier: CapacityTier::Preempt,
half_open: half_open_guard,
});
}
break;
}
None
}
async fn select_member_loop<F: FnOnce(&Arc<AppState>)>(
app_state: &Arc<AppState>,
req: GroupSelectionRequest<'_>,
after_probe: Option<F>,
) -> Result<AcquiredMember, GroupSelectionFailure> {
let now = std::time::Instant::now();
let mut ranked =
rank_members(req.target_name, &req.group.members, req.policy, &app_state.member_health, &req.mode, now);
ranked.retain(|c| !req.exclude.contains(&c.index));
if ranked.is_empty() {
return Err(GroupSelectionFailure::NoCandidate);
}
if !req.bypass_maintenance {
ranked.retain(|c| !app_state.maintenance.is_in_maintenance(&req.group.members[c.index].input_name));
if ranked.is_empty() {
return Err(GroupSelectionFailure::Maintenance);
}
}
ranked.sort_by_key(|c| req.demoted_inputs.contains(&req.group.members[c.index].input_name));
if let Some(prefer_idx) = req.prefer {
if let Some(pos) = ranked.iter().position(|c| c.index == prefer_idx) {
let cand = ranked.remove(pos);
ranked.insert(0, cand);
}
}
let all_circuits_open = ranked.iter().all(|c| {
let key = member_key(&req.group.members[c.index]);
matches!(app_state.member_health.circuit_view(&key, now), CircuitView::Open { remaining } if remaining > Duration::ZERO)
});
let own_releasing = app_state.active_users.pending_release_allocations(req.username).await;
let probe_tier = |index: usize| {
app_state.active_provider.capacity_probe(
&req.group.members[index].input_name,
req.priority,
req.kind,
req.lease.map(|l| l.provider_owner()),
&own_releasing,
)
};
let mut remaining = ranked;
let mut probed_tiers: Vec<(usize, CapacityTier)> =
remaining.iter().map(|c| (c.index, probe_tier(c.index))).collect();
if let Some(hook) = after_probe {
hook(app_state);
}
// Tier 1: Free ∪ OwnRelease
// The own slot being released is reused through grace, unless grace is not allowed at all.
let own_release_grace = req.allow_provider_grace;
let tier1_match = |tier| match tier {
CapacityTier::OwnRelease => Some(own_release_grace),
CapacityTier::Free => Some(false),
_ => None,
};
if let Some(acquired) = select_tier_candidates(
app_state,
&req,
&mut remaining,
&mut probed_tiers,
all_circuits_open,
tier1_match,
&probe_tier,
) {
return Ok(acquired);
}
// Tier 2: Grace (only if allow_provider_grace)
if req.allow_provider_grace {
let tier2_match = |tier| match tier {
CapacityTier::Grace => Some(true),
CapacityTier::Free => Some(false),
_ => None,
};
if let Some(acquired) = select_tier_candidates(
app_state,
&req,
&mut remaining,
&mut probed_tiers,
all_circuits_open,
tier2_match,
&probe_tier,
) {
return Ok(acquired);
}
}
// Tier 3: Preempt on the best-ranked candidate
if !req.allow_preempt {
return Err(GroupSelectionFailure::Exhausted);
}
if let Some(acquired) = try_acquire_preempt(app_state, &req, &remaining, all_circuits_open).await {
return Ok(acquired);
}
Err(GroupSelectionFailure::Exhausted)
}
pub(crate) async fn resolve_hls_group_entry(
app_state: &Arc<AppState>,
group: Arc<ChannelGroupRecord>,
req: GroupSelectionRequest<'_>,
) -> Result<ResolvedGroupPlayback, GroupSelectionFailure> {
let max_attempts = u32::from(req.start.max_attempts.max(1));
let mut attempts = 0;
let start_time = std::time::Instant::now();
let budget = Duration::from_secs(req.start.timeout_secs);
let mut exclude: Vec<usize> = req.exclude.to_vec();
let demoted_inputs: Vec<Arc<str>> = req.demoted_inputs.to_vec();
while attempts < max_attempts && (req.start.timeout_secs == 0 || start_time.elapsed() < budget) {
attempts += 1;
let attempt_req = GroupSelectionRequest {
mode: req.mode.clone(),
exclude: &exclude,
demoted_inputs: &demoted_inputs,
prefer: if attempts == 1 { req.prefer } else { None },
..req
};
let acquired = acquire_group_member(app_state, attempt_req).await?;
let member = group.members[acquired.index].clone();
let Some(member_input) = app_state.app_config.get_input_by_name(&member.input_name) else {
record_member_failure(&app_state.member_health, &member, acquired.half_open, FailureClass::Stream);
drop(acquired.handle);
exclude.push(acquired.index);
continue;
};
let stream_url = Arc::clone(&member.url);
return Ok(ResolvedGroupPlayback { acquired, member, input: member_input, stream_url });
}
Err(GroupSelectionFailure::Exhausted)
}
pub(crate) fn record_member_failure(
health: &MemberHealthIndex,
member: &GroupMember,
guard: Option<HalfOpenGuard>,
class: FailureClass,
) {
let now = std::time::Instant::now();
match guard {
Some(guard) => health.record_classified_failure_with(guard, class, now),
None => health.record_classified_failure(&member_key(member), class, now),
}
}
pub(crate) fn record_member_success(health: &MemberHealthIndex, member: &GroupMember, guard: Option<HalfOpenGuard>) {
match guard {
Some(guard) => health.record_success_with(guard),
None => health.record_success(&member_key(member)),
}
}
/// Selection settings of `target`, defaults when the target has no `channel_groups` block.
/// The `config.yml` channel group settings with the overrides of `target`.
pub(crate) fn group_playback_settings(app_state: &AppState, target: &ConfigTarget) -> ChannelGroupPlayback {
let target_cfg = target.options.as_ref().and_then(|options| options.channel_groups.as_ref());
app_state.app_config.config.load().channel_groups.playback(target_cfg)
}
/// First member whose input is configured, with its URL; it stands in for the group until a
/// member is selected.
pub(crate) fn group_representative(
app_state: &AppState,
group: &ChannelGroupRecord,
) -> Option<(Arc<ConfigInput>, Arc<str>)> {
group.members.iter().find_map(|member| {
app_state.app_config.get_input_by_name(&member.input_name).map(|input| (input, Arc::clone(&member.url)))
})
}
/// The client channel of a group playback carrying the identity of `member`.
pub(crate) fn member_stream_channel(
base: &StreamChannel,
group: &ChannelGroupRecord,
member: &GroupMember,
) -> StreamChannel {
let mut channel = base.clone();
channel.virtual_id = member.member_vid.get();
channel.group_virtual_id = Some(group.group_vid.get());
channel.input_name = Arc::clone(&member.input_name);
channel.item_type = member.item_type;
channel.cluster = member.cluster;
channel.title = Arc::clone(&member.title);
channel.url = Arc::clone(&member.url);
channel.provider_id = member.provider_id;
channel
}
pub(crate) const PROBE_MIN_BYTES: usize = 3 * tuliprox_mpegts::packet::TS_PACKET_SIZE;
const PROBE_MAX_EMPTY_CHUNKS: u32 = 32;
/// Upper bound for the provider response of one member; the start budget can only shorten it.
pub(crate) const PROVIDER_OPEN_TIMEOUT: Duration = Duration::from_secs(4);
/// Upper bound for the first bytes of one member; the start budget can only shorten it.
pub(crate) const PROBE_FIRST_BYTES_TIMEOUT: Duration = Duration::from_secs(3);
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum ProbedPayload {
Ts,
Playlist,
}
fn starts_with_playlist_tag(head: &[u8]) -> bool {
let head = head.strip_prefix(b"\xEF\xBB\xBF").unwrap_or(head);
let start = head.iter().position(|byte| !byte.is_ascii_whitespace()).unwrap_or(head.len());
head[start..].starts_with(b"#EXTM3U")
}
/// Reads the head of a member body until it shows MPEG-TS sync or an HLS/DASH playlist, bounded by
/// `deadline`. Returns the body with the read chunks put back in front.
pub(crate) async fn probe_member_payload(
mut stream: BoxedProviderStream,
adaptive_hint: bool,
deadline: tokio::time::Instant,
) -> Result<(BoxedProviderStream, ProbedPayload), ()> {
if adaptive_hint {
return Ok((stream, ProbedPayload::Playlist));
}
let mut chunks: Vec<Bytes> = Vec::new();
let mut head: Vec<u8> = Vec::with_capacity(PROBE_MIN_BYTES);
let mut empty_chunks = 0u32;
while head.len() < PROBE_MIN_BYTES {
match tokio::time::timeout_at(deadline, stream.next()).await {
Ok(Some(Ok(chunk))) if chunk.is_empty() => {
empty_chunks += 1;
if empty_chunks > PROBE_MAX_EMPTY_CHUNKS {
return Err(());
}
}
Ok(Some(Ok(chunk))) => {
let take = chunk.len().min(PROBE_MIN_BYTES - head.len());
head.extend_from_slice(&chunk[..take]);
chunks.push(chunk);
}
Ok(None) => break,
Ok(Some(Err(_))) | Err(_) => return Err(()),
}
}
let payload = if starts_with_playlist_tag(&head) {
ProbedPayload::Playlist
} else if tuliprox_mpegts::packet::ts_sync_offset(&head)
.is_some_and(|offset| head.len() > offset + tuliprox_mpegts::packet::TS_PACKET_SIZE)
{
// Two sync bytes one packet apart: a single 0x47 in a short text body proves nothing.
ProbedPayload::Ts
} else {
return Err(());
};
let stream = if chunks.is_empty() { stream } else { PrefixedStream::new(chunks, stream).boxed() };
Ok((stream, payload))
}
/// A provider body with the chunks a probe already read put back in front.
pub(crate) struct PrefixedStream {
head: VecDeque<Bytes>,
inner: BoxedProviderStream,
}
impl PrefixedStream {
pub(crate) fn new(head: Vec<Bytes>, inner: BoxedProviderStream) -> Self { Self { head: head.into(), inner } }
}
impl futures::Stream for PrefixedStream {
type Item = Result<Bytes, StreamError>;
fn poll_next(
self: std::pin::Pin<&mut Self>,
cx: &mut std::task::Context<'_>,
) -> std::task::Poll<Option<Self::Item>> {
let this = self.get_mut();
match this.head.pop_front() {
Some(chunk) => std::task::Poll::Ready(Some(Ok(chunk))),
None => this.inner.as_mut().poll_next(cx),
}
}
}
/// Waits until `has_capacity` holds or `deadline` passes, re-checking on every `notify` wake-up
/// instead of polling.
pub(crate) async fn wait_for_capacity<F, Fut>(
notify: &Notify,
deadline: tokio::time::Instant,
mut has_capacity: F,
) -> bool
where
F: FnMut() -> Fut,
Fut: Future<Output = bool>,
{
loop {
let notified = notify.notified();
tokio::pin!(notified);
notified.as_mut().enable();
if has_capacity().await {
return true;
}
tokio::select! {
() = notified => {}
() = tokio::time::sleep_until(deadline) => return has_capacity().await,
}
}
}
/// Waits until `user` is below its connection limit or `deadline` passes.
pub(crate) async fn wait_for_user_capacity(
app_state: &AppState,
user: &ProxyUserCredentials,
deadline: tokio::time::Instant,
) -> bool {
let notify = app_state.connection_manager.capacity_notified();
wait_for_capacity(&notify, deadline, || async {
app_state.active_users.user_connections(&user.username).await < user.max_connections
})
.await
}
pub(crate) struct GroupOpenRequest<'a> {
pub app_state: &'a Arc<AppState>,
pub user: &'a ProxyUserCredentials,
pub fingerprint: &'a Fingerprint,
pub req_headers: &'a HeaderMap,
pub group: &'a ChannelGroupRecord,
pub base_channel: &'a StreamChannel,
pub stream_options: &'a StreamOptions,
pub share_stream: bool,
pub connection_permission: UserConnectionPermission,
pub connection_kind: ConnectionKind,
pub priority: i8,
pub session_token: &'a str,
pub is_reopen: bool,
pub deadline: tokio::time::Instant,
}
pub(crate) struct OpenedGroupMember {
pub index: usize,
pub tier: CapacityTier,
pub member: GroupMember,
pub input: Arc<ConfigInput>,
pub url: Arc<str>,
pub channel: StreamChannel,
/// `details.stream` carries the probed head in front of the provider body.
pub details: StreamDetails,
pub payload: ProbedPayload,
}
pub(crate) struct MemberOpenFailure {
pub index: usize,
pub input_name: Arc<str>,
pub class: FailureClass,
}
fn is_adaptive_content_type(details: &StreamDetails) -> bool {
details.stream_info.as_ref().is_some_and(|(headers, _, _, _)| {
headers.iter().any(|(name, value)| {
name.as_str().eq_ignore_ascii_case("content-type") && {
let value = value.to_ascii_lowercase();
value.contains("mpegurl") || value.contains("dash+xml")
}
})
})
}
/// The URL to request for `member`; a Stalker member needs its playback link resolved first.
async fn member_playback_url(
req: &GroupOpenRequest<'_>,
input: &Arc<ConfigInput>,
member: &GroupMember,
) -> Option<Arc<str>> {
if !input.input_type.is_stalker() {
return Some(Arc::clone(&member.url));
}
resolve_initial_stalker_playback_url(
req.app_state,
input,
req.user,
member.provider_id,
member.cluster,
member.item_type,
&member.url,
)
.await
.ok()
}
/// Why an open produced no usable provider body, `None` when it did. A custom video in place of
/// the body counts as a failure.
fn open_failure_class(details: &StreamDetails) -> Option<FailureClass> {
let exhausted_class = || {
if details.custom_reason == Some(ProviderStreamCustomReason::ProviderExhausted) {
FailureClass::Capacity
} else {
FailureClass::Stream
}
};
match &details.stream_info {
Some((_, _, _, Some(_))) => Some(exhausted_class()),
_ if details.stream.is_some() => None,
Some((_, status, _, None)) => Some(FailureClass::from_status(*status)),
None => Some(exhausted_class()),
}
}
/// Opens the acquired member: input lookup, Stalker link, provider open, custom-video and status
/// classification, payload probe. Records the outcome in the health index; on failure the provider
/// slot is released.
pub(crate) async fn open_group_member(
req: &GroupOpenRequest<'_>,
acquired: AcquiredMember,
) -> Result<OpenedGroupMember, MemberOpenFailure> {
let app_state = req.app_state;
let AcquiredMember { index, handle, tier, half_open } = acquired;
let member = req.group.members[index].clone();
let fail = |class: FailureClass, guard: Option<HalfOpenGuard>| {
record_member_failure(&app_state.member_health, &member, guard, class);
MemberOpenFailure { index, input_name: Arc::clone(&member.input_name), class }
};
let Some(input) = app_state.app_config.get_input_by_name(&member.input_name) else {
drop(handle);
return Err(fail(FailureClass::Stream, half_open));
};
let channel = member_stream_channel(req.base_channel, req.group, &member);
// Neither the Stalker link nor the provider response may outlive the budget: a provider that
// accepts the connection and never answers must not hold the start or the switch.
let open_deadline = req.deadline.min(tokio::time::Instant::now() + PROVIDER_OPEN_TIMEOUT);
let Ok(Some(url)) = tokio::time::timeout_at(open_deadline, member_playback_url(req, &input, &member)).await else {
drop(handle);
return Err(fail(FailureClass::Transport, half_open));
};
let content_representation = if member.item_type == PlaylistItemType::Catchup {
crate::api::model::ProviderContentRepresentationMode::Identity
} else {
crate::api::model::ProviderContentRepresentationMode::PreserveOrigin
};
let opened = tokio::time::timeout_at(
open_deadline,
create_stream_response_details(
app_state,
req.stream_options,
url.as_ref(),
&req.user.username,
req.fingerprint,
req.req_headers,
&input,
&channel,
member.item_type,
content_representation,
req.share_stream,
req.connection_permission,
None,
true,
true,
VirtualId::new(member.member_vid.get()),
req.priority,
req.connection_kind,
req.is_reopen,
Some(req.session_token),
None,
true,
None,
None,
None,
Some(handle),
),
)
.await;
let mut details = match opened {
Ok(Ok(details)) => details,
Ok(Err(_)) => return Err(fail(FailureClass::Stream, half_open)),
// The dropped open released the provider slot it held.
Err(_) => return Err(fail(FailureClass::Transport, half_open)),
};
let release =
|details: StreamDetails| app_state.connection_manager.release_managed_provider_handle(details.provider_handle);
if let Some(class) = open_failure_class(&details) {
release(details);
return Err(fail(class, half_open));
}
let Some(provider_stream) = details.stream.take() else {
release(details);
return Err(fail(FailureClass::Stream, half_open));
};
let adaptive_hint = member.item_type.is_live_adaptive() || is_adaptive_content_type(&details);
let probe_deadline = req.deadline.min(tokio::time::Instant::now() + PROBE_FIRST_BYTES_TIMEOUT);
let Ok((provider_stream, payload)) = probe_member_payload(provider_stream, adaptive_hint, probe_deadline).await
else {
release(details);
return Err(fail(FailureClass::Stream, half_open));
};
details.stream = Some(provider_stream);
record_member_success(&app_state.member_health, &member, half_open);
Ok(OpenedGroupMember { index, tier, member, input, url, channel, details, payload })
}
@@ -0,0 +1,277 @@
use shared::model::ChannelGroupFailoverConfig;
use std::{
collections::VecDeque,
time::{Duration, Instant},
};
/// Playback a member needs before its failure opens a fresh switch budget.
const FRESH_BUDGET_AFTER: Duration = Duration::from_secs(5);
#[derive(Debug, PartialEq, Eq)]
pub(crate) enum BudgetDecision {
/// Reopen the current member, it played long enough to deserve another try.
RetrySame,
/// Select another member, excluding the ones already tried in this incident.
Next { exclude: Vec<usize> },
/// Stop switching: the incident budget or the flap guard is exhausted.
GiveUp,
}
/// Decides how a running group stream reacts to a failing member. An incident starts at the
/// first trigger and ends after `stable_after_secs` of playback without a new one. Each stretch
/// without playback is bounded by `max_outage_secs`, and `max_switches` bounds the switches within
/// `max_switches_window_secs` across incidents.
pub(crate) struct FailoverBudget {
budget: Duration,
stable_reset: Duration,
flap_window: Duration,
flap_max_switches: u8,
same_member_retries: u8,
incident_started: Option<Instant>,
last_trigger: Option<Instant>,
tried: Vec<usize>,
switches: VecDeque<Instant>,
same_retries_left: u8,
member_first_data: Option<Instant>,
}
impl FailoverBudget {
pub(crate) fn new(failover: &ChannelGroupFailoverConfig) -> Self {
Self {
budget: Duration::from_secs(failover.max_outage_secs),
stable_reset: Duration::from_secs(failover.stable_after_secs),
flap_window: Duration::from_secs(failover.max_switches_window_secs),
flap_max_switches: failover.max_switches,
same_member_retries: failover.same_source_retries,
incident_started: None,
last_trigger: None,
tried: Vec::new(),
switches: VecDeque::new(),
same_retries_left: failover.same_source_retries,
member_first_data: None,
}
}
pub(crate) fn on_trigger(&mut self, current: usize, member_count: usize, now: Instant) -> BudgetDecision {
self.last_trigger = Some(now);
let started = *self.incident_started.get_or_insert(now);
if now.saturating_duration_since(started) >= self.budget {
return BudgetDecision::GiveUp;
}
if self.flap_max_switches > 0 {
while self.switches.front().is_some_and(|at| now.saturating_duration_since(*at) > self.flap_window) {
self.switches.pop_front();
}
if self.switches.len() >= usize::from(self.flap_max_switches) {
return BudgetDecision::GiveUp;
}
}
let stable =
self.member_first_data.is_some_and(|first| now.saturating_duration_since(first) >= self.stable_reset);
if stable && self.same_retries_left > 0 {
self.same_retries_left -= 1;
return BudgetDecision::RetrySame;
}
if !self.tried.contains(&current) {
self.tried.push(current);
}
if self.tried.len() >= member_count {
self.tried.retain(|index| *index == current);
}
BudgetDecision::Next { exclude: self.tried.clone() }
}
/// Whether a trigger now would give up, without starting or changing an incident.
pub(crate) fn would_give_up(&self, now: Instant) -> bool {
let budget_spent =
self.incident_started.is_some_and(|started| now.saturating_duration_since(started) >= self.budget);
let flapping = self.flap_max_switches > 0
&& self.switches.iter().filter(|at| now.saturating_duration_since(**at) <= self.flap_window).count()
>= usize::from(self.flap_max_switches);
budget_spent || flapping
}
pub(crate) fn on_switched(&mut self, index: usize, previous: usize, now: Instant) {
if self.flap_max_switches > 0 {
self.switches.push_back(now);
}
if index != previous {
self.same_retries_left = self.same_member_retries;
}
self.member_first_data = None;
}
/// Called for packets sent to the client.
pub(crate) fn on_data(&mut self, now: Instant) {
let first_data = *self.member_first_data.get_or_insert(now);
// The budget bounds the time without playback: a member that really plays restarts it,
// while the members tried in this incident stay excluded until it ends. A member that
// sends a few packets and fails (an error clip) does not.
if self.incident_started.is_some()
&& now.saturating_duration_since(first_data) >= FRESH_BUDGET_AFTER.min(self.budget / 2)
{
self.incident_started = None;
}
if self.last_trigger.is_some_and(|last| now.saturating_duration_since(last) >= self.stable_reset) {
self.incident_started = None;
self.last_trigger = None;
self.tried.clear();
self.same_retries_left = self.same_member_retries;
}
}
/// End of the current incident's switch budget.
pub(crate) fn deadline(&self) -> Option<Instant> { self.incident_started.map(|started| started + self.budget) }
}
#[cfg(test)]
mod tests {
use super::*;
use shared::model::ChannelGroupFailoverConfig;
use std::time::{Duration, Instant};
fn config(max_switches: u8) -> ChannelGroupFailoverConfig {
ChannelGroupFailoverConfig {
max_outage_secs: 20,
stable_after_secs: 30,
max_switches_window_secs: 600,
max_switches,
same_source_retries: 1,
..ChannelGroupFailoverConfig::default()
}
}
fn secs(start: Instant, secs: u64) -> Instant { start + Duration::from_secs(secs) }
#[test]
fn budget_exceeded_gives_up() {
let t0 = Instant::now();
let mut budget = FailoverBudget::new(&config(6));
assert_eq!(budget.on_trigger(0, 3, t0), BudgetDecision::Next { exclude: vec![0] });
budget.on_switched(1, 0, secs(t0, 1));
assert_eq!(budget.on_trigger(1, 3, secs(t0, 21)), BudgetDecision::GiveUp);
}
#[test]
fn budget_resets_after_stable_playback() {
let t0 = Instant::now();
let mut budget = FailoverBudget::new(&config(6));
budget.on_trigger(0, 3, t0);
budget.on_switched(1, 0, secs(t0, 1));
budget.on_data(secs(t0, 2));
budget.on_data(secs(t0, 31));
assert_eq!(budget.deadline(), None);
assert_eq!(budget.on_trigger(1, 3, secs(t0, 32)), BudgetDecision::RetrySame);
assert_eq!(budget.deadline(), Some(secs(t0, 52)));
}
#[test]
fn budget_is_per_incident_not_per_stream() {
let t0 = Instant::now();
let mut budget = FailoverBudget::new(&config(0));
for incident in 0..5u64 {
let start = secs(t0, incident * 100);
let current = usize::try_from(incident % 2).expect("index");
let next = 1 - current;
assert_ne!(budget.on_trigger(current, 2, start), BudgetDecision::GiveUp);
budget.on_switched(next, current, start);
budget.on_data(start);
budget.on_data(start + Duration::from_secs(31));
}
}
#[test]
fn walk_restarts_after_full_round_within_budget() {
let t0 = Instant::now();
let mut budget = FailoverBudget::new(&config(6));
assert_eq!(budget.on_trigger(0, 3, t0), BudgetDecision::Next { exclude: vec![0] });
budget.on_switched(1, 0, t0);
assert_eq!(budget.on_trigger(1, 3, secs(t0, 1)), BudgetDecision::Next { exclude: vec![0, 1] });
budget.on_switched(2, 1, secs(t0, 1));
assert_eq!(budget.on_trigger(2, 3, secs(t0, 2)), BudgetDecision::Next { exclude: vec![2] });
}
#[test]
fn flap_guard_gives_up() {
let t0 = Instant::now();
let mut budget = FailoverBudget::new(&config(2));
budget.on_trigger(0, 3, t0);
budget.on_switched(1, 0, t0);
budget.on_trigger(1, 3, secs(t0, 1));
budget.on_switched(2, 1, secs(t0, 1));
assert_eq!(budget.on_trigger(2, 3, secs(t0, 2)), BudgetDecision::GiveUp);
}
#[test]
fn flap_guard_disabled_with_zero() {
let t0 = Instant::now();
let mut budget = FailoverBudget::new(&config(0));
for step in 0..10u64 {
let current = usize::try_from(step % 3).expect("index");
assert!(matches!(budget.on_trigger(current, 3, secs(t0, step)), BudgetDecision::Next { .. }));
budget.on_switched((current + 1) % 3, current, secs(t0, step));
}
}
#[test]
fn fresh_member_failure_skips_same_member_retry() {
let t0 = Instant::now();
let mut budget = FailoverBudget::new(&config(6));
budget.on_data(t0);
assert_eq!(budget.on_trigger(0, 2, secs(t0, 5)), BudgetDecision::Next { exclude: vec![0] });
}
#[test]
fn stable_member_failure_retries_same_member_first() {
let t0 = Instant::now();
let mut budget = FailoverBudget::new(&config(6));
budget.on_data(t0);
assert_eq!(budget.on_trigger(0, 2, secs(t0, 31)), BudgetDecision::RetrySame);
budget.on_switched(0, 0, secs(t0, 31));
budget.on_data(secs(t0, 32));
assert_eq!(budget.on_trigger(0, 2, secs(t0, 33)), BudgetDecision::Next { exclude: vec![0] });
}
#[test]
fn failure_after_good_playback_gets_a_fresh_budget() {
let t0 = Instant::now();
let mut budget = FailoverBudget::new(&config(6));
assert_eq!(budget.on_trigger(0, 3, t0), BudgetDecision::Next { exclude: vec![0] });
budget.on_switched(1, 0, secs(t0, 1));
budget.on_data(secs(t0, 1));
budget.on_data(secs(t0, 25));
assert_eq!(budget.on_trigger(1, 3, secs(t0, 26)), BudgetDecision::Next { exclude: vec![0, 1] });
}
#[test]
fn flap_history_is_not_kept_without_guard() {
let t0 = Instant::now();
let mut budget = FailoverBudget::new(&config(0));
for step in 0..100u64 {
budget.on_switched(1, 0, secs(t0, step));
}
assert!(budget.switches.is_empty());
}
#[test]
fn brief_data_does_not_restart_budget() {
let t0 = Instant::now();
let mut budget = FailoverBudget::new(&config(0));
assert_eq!(budget.on_trigger(0, 2, t0), BudgetDecision::Next { exclude: vec![0] });
budget.on_switched(1, 0, secs(t0, 1));
budget.on_data(secs(t0, 1));
budget.on_data(secs(t0, 3));
assert_eq!(budget.on_trigger(1, 2, secs(t0, 21)), BudgetDecision::GiveUp);
}
#[test]
fn would_give_up_leaves_the_incident_untouched() {
let t0 = Instant::now();
let mut budget = FailoverBudget::new(&config(6));
assert!(!budget.would_give_up(t0));
assert_eq!(budget.deadline(), None, "asking starts no incident");
budget.on_trigger(0, 2, t0);
assert!(budget.would_give_up(secs(t0, 21)));
assert!(!budget.would_give_up(secs(t0, 5)));
}
}
@@ -0,0 +1,634 @@
//! Runtime failover of channel group playback: the provider body behind an open client response
//! is replaced by another member when it fails, stalls or delivers slower than real time.
mod budget;
#[cfg(test)]
mod tests;
use super::{
channel_group::{
acquire_group_member, open_group_member, record_member_failure, GroupOpenRequest, GroupSelectionFailure,
GroupSelectionRequest, OpenedGroupMember, ProbedPayload,
},
connection_priority_for_kind, get_stream_options, StreamResponseMode,
};
use crate::{
api::model::{AppState, BoxedProviderStream, StreamError},
auth::Fingerprint,
model::{ConfigTarget, ProxyUserCredentials},
utils::debug_if_enabled,
};
use axum::http::HeaderMap;
pub(crate) use budget::{BudgetDecision, FailoverBudget};
use bytes::Bytes;
use futures::Future;
use shared::model::{ChannelGroupPlayback, EventMessage, StreamChannel, SwitchReason, UserConnectionPermission};
use std::{
pin::Pin,
sync::Arc,
task::{Context, Poll},
time::{Duration, Instant},
};
use tokio::time::Sleep;
use tokio_util::sync::CancellationToken;
use tuliprox_core::model::AllocationId;
use tuliprox_mpegts::{
packet::ts_null_packet_bytes,
splice::{RealtimeMeter, SpliceOutput, TsSpliceAligner},
};
use tuliprox_repository::channel_group_repository::ChannelGroupRecord;
use tuliprox_session::{
active_provider_manager::{ConnectionKind, ManagedProviderHandle},
channel_group::{AudioSignature, FailureClass, SelectionMode},
MemberBinding,
};
/// While a switch runs, a null packet this often keeps players and proxies from timing out.
const KEEPALIVE_INTERVAL: Duration = Duration::from_millis(500);
/// Longest stretch of a new member held back while waiting for its splice point (PAT/PMT and a
/// keyframe); beyond it the member starts at its first PAT.
const SPLICE_BUDGET_BYTES: usize = 8 * 1024 * 1024;
const SPLICE_BUDGET: Duration = Duration::from_secs(4);
/// Re-check interval while every candidate is busy during a switch.
const EXHAUSTED_RETRY_INTERVAL: Duration = Duration::from_millis(100);
/// Pause before a new round over members that all failed within one switch; it doubles per
/// round up to the maximum.
const NO_CANDIDATE_RETRY_INTERVAL: Duration = Duration::from_millis(500);
const NO_CANDIDATE_RETRY_MAX_INTERVAL: Duration = Duration::from_secs(4);
/// Upper bound for the first bytes of a member opened at runtime.
const RUNTIME_OPEN_TIMEOUT: Duration = Duration::from_secs(3);
/// Everything a running group stream needs to open another member for its client.
pub(crate) struct GroupFailoverContext {
pub app_state: Arc<AppState>,
pub target: Arc<ConfigTarget>,
/// `target.name` as the health and selection key, allocated once per stream.
pub target_name: Arc<str>,
pub group: Arc<ChannelGroupRecord>,
pub playback: ChannelGroupPlayback,
pub user: ProxyUserCredentials,
pub fingerprint: Fingerprint,
pub req_headers: HeaderMap,
pub session_token: Arc<str>,
pub connection_kind: ConnectionKind,
pub connection_permission: UserConnectionPermission,
pub share_stream: bool,
pub base_channel: StreamChannel,
/// Key of the shared stream this group stream feeds; its provider allocation moves with the
/// member.
pub shared_key: Option<Arc<str>>,
}
/// The member a switch opened: its provider body and the slot it holds.
pub(crate) struct RuntimeMember {
pub index: usize,
pub stream: BoxedProviderStream,
pub handle: Option<ManagedProviderHandle>,
/// Cancelled when the provider manager preempts the member's slot.
pub cancel: Option<CancellationToken>,
/// The member's provider allocation; a shared stream owns it now.
pub allocation_id: Option<AllocationId>,
}
pub(crate) type SwitchFuture = Pin<Box<dyn Future<Output = Option<RuntimeMember>> + Send>>;
/// Decides about and prepares member switches of one running group stream. The owner of the
/// provider handle (direct client stream or shared producer) executes them.
pub(crate) struct GroupFailoverDriver {
ctx: Arc<GroupFailoverContext>,
current: usize,
budget: FailoverBudget,
aligner: TsSpliceAligner,
meter: RealtimeMeter,
stall_timeout: Option<Duration>,
stall: Pin<Box<Sleep>>,
stall_armed: bool,
keepalive: Pin<Box<Sleep>>,
pending_since: Option<Instant>,
underrun: bool,
/// Second piece of a split chunk, sent before the provider body is polled again.
pending_output: Option<Bytes>,
/// Provider allocation of the current member while a shared stream owns it.
shared_allocation: Option<AllocationId>,
}
impl GroupFailoverDriver {
pub(crate) fn new(ctx: Arc<GroupFailoverContext>, current: usize) -> Self {
let failover = ctx.playback.failover;
Self {
budget: FailoverBudget::new(&failover),
aligner: TsSpliceAligner::new(SPLICE_BUDGET_BYTES, SPLICE_BUDGET),
meter: RealtimeMeter::new(failover.slow_source_pct, Duration::from_secs(failover.slow_source_secs)),
stall_timeout: (failover.no_data_timeout_secs > 0)
.then(|| Duration::from_secs(failover.no_data_timeout_secs)),
stall: Box::pin(tokio::time::sleep(Duration::ZERO)),
stall_armed: false,
keepalive: Box::pin(tokio::time::sleep(Duration::ZERO)),
pending_since: None,
underrun: false,
pending_output: None,
shared_allocation: None,
ctx,
current,
}
}
pub(crate) fn group_vid(&self) -> u32 { self.ctx.group.group_vid.get() }
/// Hot path: aligns the chunk and does the per-chunk bookkeeping. `None` while the aligner
/// holds packets back.
pub(crate) fn process_chunk(&mut self, chunk: Bytes) -> Option<Bytes> {
let now = Instant::now();
let waited = self.pending_since.take().map_or(Duration::ZERO, |since| now.saturating_duration_since(since));
let (first, second) = match self.aligner.push(chunk, now) {
SpliceOutput::Hold => return None,
SpliceOutput::Emit(packets) => (packets, None),
SpliceOutput::EmitTwo(first, second) => (first, Some(second)),
};
// Only packets that reach the client count as playback and hold off the stall check.
self.stall_armed = false;
self.budget.on_data(now);
let mut slow = self.meter.observe(&first, waited, now);
if let Some(second) = &second {
slow |= self.meter.observe(second, Duration::ZERO, now);
}
if slow {
self.underrun = true;
self.meter.reset(now);
}
self.pending_output = second;
Some(first)
}
/// The second piece of the last split chunk, to send before polling the provider again.
pub(crate) fn take_pending_output(&mut self) -> Option<Bytes> { self.pending_output.take() }
/// Called while the provider body is pending; true once it stayed silent for the stall timeout.
pub(crate) fn poll_stalled(&mut self, cx: &mut Context<'_>) -> bool {
let now = Instant::now();
// The wait also feeds the underrun meter, with or without a stall timeout.
self.pending_since.get_or_insert(now);
let Some(timeout) = self.stall_timeout else {
return false;
};
if !self.stall_armed {
self.stall.as_mut().reset(tokio::time::Instant::from_std(now + timeout));
self.stall_armed = true;
}
self.stall.as_mut().poll(cx).is_ready()
}
/// True once the realtime meter detected an underrun; consumes the signal.
pub(crate) fn take_underrun(&mut self) -> bool { std::mem::take(&mut self.underrun) }
/// Records the failure of the current member and returns the future that opens the next one,
/// or `None` when the budget is spent. The caller releases the current provider slot before
/// it polls the future, so a retry on the same account finds its slot free.
pub(crate) fn begin_switch(&mut self, reason: SwitchReason, status: Option<u16>) -> Option<SwitchFuture> {
let now = Instant::now();
// A slow member still plays: when no switch is allowed it keeps playing, and the check
// leaves the incident untouched.
if reason == SwitchReason::Underrun && self.budget.would_give_up(now) {
return None;
}
let group = &self.ctx.group;
// The member failed whether or not another one may take over.
record_member_failure(
&self.ctx.app_state.member_health,
&group.members[self.current],
None,
failure_class_for(reason, status),
);
let decision = self.budget.on_trigger(self.current, group.members.len(), now);
let (exclude, prefer) = match decision {
BudgetDecision::GiveUp => return None,
BudgetDecision::RetrySame => (Vec::new(), Some(self.current)),
BudgetDecision::Next { exclude } => (exclude, None),
};
let deadline = self.budget.deadline().unwrap_or(now);
self.stall_armed = false;
self.pending_since = None;
self.keepalive.as_mut().reset(tokio::time::Instant::now() + KEEPALIVE_INTERVAL);
Some(Box::pin(switch_member(
Arc::clone(&self.ctx),
SwitchRequest {
shared_allocation: self.shared_allocation,
previous: self.current,
exclude,
prefer,
deadline: tokio::time::Instant::from_std(deadline),
reason,
},
)))
}
/// Continues on the member a switch opened.
pub(crate) fn on_switched(&mut self, index: usize) {
let now = Instant::now();
self.budget.on_switched(index, self.current, now);
self.current = index;
self.aligner.begin_member(now);
self.meter.begin_member(now);
self.stall_armed = false;
self.pending_since = None;
self.underrun = false;
}
/// While a switch runs: a null packet every `KEEPALIVE_INTERVAL`.
pub(crate) fn poll_keepalive(&mut self, cx: &mut Context<'_>) -> Poll<Bytes> {
if self.keepalive.as_mut().poll(cx).is_pending() {
return Poll::Pending;
}
self.keepalive.as_mut().reset(tokio::time::Instant::now() + KEEPALIVE_INTERVAL);
Poll::Ready(ts_null_packet_bytes())
}
}
/// How a member failure counts in the health index: an upstream error by its HTTP status, a
/// transport error without one; an ended, stalled or slow body as a stream failure.
pub(crate) fn failure_class_for(reason: SwitchReason, status: Option<u16>) -> FailureClass {
match (reason, status.and_then(|status| axum::http::StatusCode::from_u16(status).ok())) {
(SwitchReason::UpstreamError, Some(status)) => FailureClass::from_status(status),
(SwitchReason::UpstreamError, None) => FailureClass::Transport,
(SwitchReason::UpstreamEof | SwitchReason::Stall | SwitchReason::Underrun, _) => FailureClass::Stream,
}
}
/// The token the provider manager cancels when it preempts the slot of `handle`.
pub(crate) fn member_cancel_token(handle: Option<&ManagedProviderHandle>) -> Option<CancellationToken> {
handle.and_then(ManagedProviderHandle::handle).and_then(|handle| handle.cancel_token.clone())
}
struct SwitchRequest {
/// Provider allocation of the replaced member, when a shared stream owns it.
shared_allocation: Option<AllocationId>,
previous: usize,
exclude: Vec<usize>,
prefer: Option<usize>,
deadline: tokio::time::Instant,
reason: SwitchReason,
}
/// Selection mode of a runtime switch: prefer members like the one that played.
fn runtime_mode(ctx: &GroupFailoverContext, previous: usize) -> SelectionMode {
let probe = ctx
.app_state
.member_health
.health(&ctx.target_name, &ctx.group.members[previous], std::time::Instant::now())
.probe;
SelectionMode::Runtime {
current_codec: probe.as_ref().map(|p| p.codec.clone()).unwrap_or_default(),
current_resolution: probe.as_ref().map(|p| p.resolution.clone()).unwrap_or_default(),
current_audio: probe.as_ref().map(|p| AudioSignature {
codec: p.audio_codec.clone(),
channels: p.audio_channels.clone(),
language: p.audio_language.clone(),
}),
audio_language: ctx.playback.failover.audio_language,
}
}
/// Takes over the opened member: a shared stream receives its provider slot, the session is bound
/// to it and the switch is announced.
async fn adopt_member(
ctx: &GroupFailoverContext,
detached: Option<AllocationId>,
previous: usize,
reason: SwitchReason,
opened: OpenedGroupMember,
stream: BoxedProviderStream,
) -> Option<RuntimeMember> {
let app_state = &ctx.app_state;
let mut handle = opened.details.provider_handle;
let cancel = member_cancel_token(handle.as_ref());
let slot = handle.as_ref().and_then(ManagedProviderHandle::handle);
let allocation_id = slot.map(|handle| handle.allocation_id);
let provider = slot
.and_then(|handle| handle.allocation.get_provider_name())
.unwrap_or_else(|| Arc::clone(&opened.member.input_name));
if let (Some(key), Some(detached)) = (&ctx.shared_key, detached) {
let attached = match handle.take() {
Some(handle) => app_state.shared_stream_manager.attach_provider(key, detached, handle).await,
None => true,
};
if !attached {
// The shared stream ended meanwhile; the dropped handle released its slot.
return None;
}
}
let member_vid = opened.member.member_vid.get();
app_state
.active_users
.bind_member(
&ctx.user.username,
&ctx.session_token,
MemberBinding {
group_vid: ctx.group.group_vid.get(),
member_vid,
member_index: opened.index,
input_name: Arc::clone(&opened.member.input_name),
},
)
.await;
if ctx.shared_key.is_some() {
// Every viewer of the shared stream plays the new member, not only the one that started it.
app_state
.active_users
.update_shared_group_provider(ctx.target.id, ctx.group.group_vid.get(), Arc::clone(&provider))
.await;
}
app_state
.active_users
.update_session_provider_binding(&ctx.user.username, &ctx.session_token, provider, Arc::clone(&opened.url))
.await;
app_state.event_manager.send_event(EventMessage::ChannelGroupMemberSwitched {
target_id: ctx.target.id,
group_vid: ctx.group.group_vid.get(),
from_member_vid: ctx.group.members[previous].member_vid.get(),
to_member_vid: member_vid,
reason,
});
Some(RuntimeMember { index: opened.index, stream, handle, cancel, allocation_id })
}
async fn switch_member(ctx: Arc<GroupFailoverContext>, request: SwitchRequest) -> Option<RuntimeMember> {
let SwitchRequest { previous, mut exclude, mut prefer, deadline, reason, shared_allocation } = request;
let app_state = &ctx.app_state;
let priority = connection_priority_for_kind(&ctx.user, ctx.connection_kind);
let stream_options = get_stream_options(&app_state.app_config, StreamResponseMode::Stream);
let mode = runtime_mode(&ctx, previous);
let mut demoted_inputs: Vec<Arc<str>> = Vec::new();
// Members a new round must not try again within this switch: their provider refused the
// account (auth, capacity) or delivers nothing that can be spliced.
let mut hard_exclude: Vec<usize> = Vec::new();
let mut round_pause = NO_CANDIDATE_RETRY_INTERVAL;
if let Some(key) = &ctx.shared_key {
// Subscribers stay while the failed member's slot is given back. Without the expected
// allocation the shared stream is no longer this one's and nothing may be switched.
let detached = match shared_allocation {
Some(allocation_id) => app_state.shared_stream_manager.detach_provider(key, allocation_id).await,
None => false,
};
if !detached {
return None;
}
}
loop {
if tokio::time::Instant::now() >= deadline {
return None;
}
let wanted = prefer.take();
let req = GroupSelectionRequest {
mode: mode.clone(),
allow_provider_grace: false,
exclude: &exclude,
demoted_inputs: &demoted_inputs,
prefer: wanted,
bypass_maintenance: ctx.user.is_recording_proxy_user(),
allow_preempt: false,
..GroupSelectionRequest::new(
&ctx.target_name,
&ctx.group,
ctx.playback.start,
&ctx.fingerprint.addr,
&ctx.user.username,
priority,
ctx.connection_kind,
)
};
let acquired = match tokio::time::timeout_at(deadline, acquire_group_member(app_state, req)).await {
Ok(Ok(acquired)) => acquired,
Ok(Err(GroupSelectionFailure::Exhausted)) => {
// A released slot frees once its provider body has ended, which happens
// asynchronously; the slot this stream just gave back is worth waiting for.
prefer = wanted;
tokio::time::sleep_until(deadline.min(tokio::time::Instant::now() + EXHAUSTED_RETRY_INTERVAL)).await;
continue;
}
Ok(Err(GroupSelectionFailure::NoCandidate)) => {
// Every other member failed in this switch: start a new round over them after a
// pause, still bounded by the budget. The replaced member stays out.
exclude.retain(|index| *index == previous || hard_exclude.contains(index));
tokio::time::sleep_until(deadline.min(tokio::time::Instant::now() + round_pause)).await;
round_pause = (round_pause * 2).min(NO_CANDIDATE_RETRY_MAX_INTERVAL);
continue;
}
Ok(Err(_)) | Err(_) => return None,
};
let open_req = GroupOpenRequest {
app_state,
user: &ctx.user,
fingerprint: &ctx.fingerprint,
req_headers: &ctx.req_headers,
group: &ctx.group,
base_channel: &ctx.base_channel,
stream_options: &stream_options,
share_stream: ctx.share_stream,
connection_permission: ctx.connection_permission,
connection_kind: ctx.connection_kind,
priority,
session_token: &ctx.session_token,
is_reopen: true,
deadline: deadline.min(tokio::time::Instant::now() + RUNTIME_OPEN_TIMEOUT),
};
match open_group_member(&open_req, acquired).await {
Ok(mut opened) => {
let stream = opened.details.stream.take();
let (Some(stream), ProbedPayload::Ts) = (stream, opened.payload) else {
// Only MPEG-TS can be spliced into the running response.
app_state.connection_manager.release_managed_provider_handle(opened.details.provider_handle);
exclude.push(opened.index);
hard_exclude.push(opened.index);
continue;
};
return adopt_member(&ctx, shared_allocation, previous, reason, opened, stream).await;
}
Err(failure) => {
if matches!(failure.class, FailureClass::Auth | FailureClass::Capacity) {
if !demoted_inputs.contains(&failure.input_name) {
demoted_inputs.push(failure.input_name);
}
hard_exclude.push(failure.index);
}
exclude.push(failure.index);
}
}
}
}
/// How a group playback's member body ended for good: no member could take over.
pub(crate) enum GroupEnd {
Eof,
Error(StreamError),
Stalled,
/// A switch was started for `reason` but no member could be opened within the budget.
SwitchFailed {
reason: SwitchReason,
},
}
pub(crate) enum GroupPoll {
Data(Bytes),
/// A null packet sent while a switch runs: keeps the connection alive, carries no media.
Keepalive(Bytes),
Pending,
Ended(GroupEnd),
}
/// Owner of a group playback's provider slot and body: the direct client stream or the producer
/// of a shared stream.
pub(crate) trait MemberHost {
fn member_body(&mut self) -> Option<&mut BoxedProviderStream>;
/// True when the provider manager preempted the current member's slot: no failover then.
fn member_preempted(&self) -> bool;
/// Gives up the failed member's body and slot before the next member is acquired, so the
/// same account can be reused.
fn release_member(&mut self);
fn install_member(&mut self, member: RuntimeMember);
}
/// The switch state machine of a group playback, shared by its direct and shared hosts.
pub(crate) struct GroupSwitcher {
driver: GroupFailoverDriver,
/// The switch in flight and the failure that started it.
switch: Option<(SwitchFuture, SwitchReason)>,
}
impl GroupSwitcher {
pub(crate) fn new(driver: GroupFailoverDriver) -> Self { Self { driver, switch: None } }
/// True while another member is being opened.
pub(crate) fn is_switching(&self) -> bool { self.switch.is_some() }
/// Polls the member body of `host`, switching members underneath it when it fails.
pub(crate) fn poll_member(&mut self, host: &mut impl MemberHost, cx: &mut Context<'_>) -> GroupPoll {
loop {
if let Some(packets) = self.driver.take_pending_output() {
return GroupPoll::Data(packets);
}
if let Some((switch, reason)) = self.switch.as_mut() {
let reason = *reason;
match switch.as_mut().poll(cx) {
Poll::Ready(Some(member)) => {
self.switch = None;
self.driver.on_switched(member.index);
self.driver.shared_allocation = member.allocation_id;
host.install_member(member);
continue;
}
Poll::Ready(None) => {
self.switch = None;
return GroupPoll::Ended(GroupEnd::SwitchFailed { reason });
}
Poll::Pending => {
return match self.driver.poll_keepalive(cx) {
Poll::Ready(keepalive) => GroupPoll::Keepalive(keepalive),
Poll::Pending => GroupPoll::Pending,
};
}
}
}
let Some(body) = host.member_body() else {
return GroupPoll::Ended(GroupEnd::Eof);
};
match body.as_mut().poll_next(cx) {
Poll::Ready(Some(Ok(chunk))) => {
let packets = self.driver.process_chunk(chunk);
if self.driver.take_underrun() {
// The packets already read still go out; the next poll switches.
self.start_switch(host, SwitchReason::Underrun, None);
}
if let Some(packets) = packets {
return GroupPoll::Data(packets);
}
}
Poll::Ready(Some(Err(err))) => {
if !self.start_switch(host, SwitchReason::UpstreamError, err.provider_http_status()) {
return GroupPoll::Ended(GroupEnd::Error(err));
}
}
Poll::Ready(None) => {
if !self.start_switch(host, SwitchReason::UpstreamEof, None) {
return GroupPoll::Ended(GroupEnd::Eof);
}
}
Poll::Pending => {
if !self.driver.poll_stalled(cx) {
return GroupPoll::Pending;
}
if !self.start_switch(host, SwitchReason::Stall, None) {
return GroupPoll::Ended(GroupEnd::Stalled);
}
}
}
}
}
fn start_switch(&mut self, host: &mut impl MemberHost, reason: SwitchReason, status: Option<u16>) -> bool {
if host.member_preempted() {
return false;
}
let Some(switch) = self.driver.begin_switch(reason, status) else {
return false;
};
debug_if_enabled!("Group stream {} lost its member ({reason:?}); switching", self.driver.group_vid());
host.release_member();
self.switch = Some((switch, reason));
true
}
}
/// The producer of a shared group stream: forwards the member body to the shared stream and
/// replaces the member underneath it when it fails. The shared stream owns the provider slot.
pub(crate) struct GroupFailoverStream {
switcher: GroupSwitcher,
host: SharedMemberHost,
}
struct SharedMemberHost {
body: Option<BoxedProviderStream>,
/// Preemption of the current member's slot.
member_cancel: Option<CancellationToken>,
}
impl MemberHost for SharedMemberHost {
fn member_body(&mut self) -> Option<&mut BoxedProviderStream> { self.body.as_mut() }
fn member_preempted(&self) -> bool { self.member_cancel.as_ref().is_some_and(CancellationToken::is_cancelled) }
fn release_member(&mut self) { self.body = None; }
fn install_member(&mut self, member: RuntimeMember) {
self.member_cancel = member.cancel;
self.body = Some(member.stream);
}
}
impl GroupFailoverStream {
/// `slot` is the provider handle of the first member, which the shared stream takes over.
pub(crate) fn new(
mut driver: GroupFailoverDriver,
inner: BoxedProviderStream,
slot: Option<&ManagedProviderHandle>,
) -> Self {
driver.shared_allocation = slot.and_then(ManagedProviderHandle::handle).map(|handle| handle.allocation_id);
Self {
switcher: GroupSwitcher::new(driver),
host: SharedMemberHost { body: Some(inner), member_cancel: member_cancel_token(slot) },
}
}
}
impl futures::Stream for GroupFailoverStream {
type Item = Result<Bytes, StreamError>;
fn poll_next(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
let this = self.get_mut();
match this.switcher.poll_member(&mut this.host, cx) {
GroupPoll::Data(packets) | GroupPoll::Keepalive(packets) => Poll::Ready(Some(Ok(packets))),
GroupPoll::Pending => Poll::Pending,
GroupPoll::Ended(GroupEnd::Error(err)) => Poll::Ready(Some(Err(err))),
GroupPoll::Ended(GroupEnd::Eof | GroupEnd::Stalled | GroupEnd::SwitchFailed { .. }) => Poll::Ready(None),
}
}
}
@@ -0,0 +1,207 @@
use super::failure_class_for;
use shared::model::SwitchReason;
use tuliprox_session::channel_group::FailureClass;
#[test]
fn upstream_error_class_follows_http_status() {
assert_eq!(failure_class_for(SwitchReason::UpstreamError, Some(401)), FailureClass::Auth);
assert_eq!(failure_class_for(SwitchReason::UpstreamError, Some(429)), FailureClass::Capacity);
assert_eq!(failure_class_for(SwitchReason::UpstreamError, Some(502)), FailureClass::Stream);
assert_eq!(failure_class_for(SwitchReason::UpstreamError, None), FailureClass::Transport);
for reason in [SwitchReason::UpstreamEof, SwitchReason::Stall, SwitchReason::Underrun] {
assert_eq!(failure_class_for(reason, Some(401)), FailureClass::Stream);
}
}
mod switcher {
use super::super::{
GroupEnd, GroupFailoverContext, GroupFailoverDriver, GroupPoll, GroupSwitcher, MemberHost, RuntimeMember,
};
use crate::api::{
api_utils::tests::group_playback_support::{member_ts, GroupPlaybackEnv},
model::{BoxedProviderStream, StreamError},
};
use bytes::Bytes;
use futures::StreamExt;
use shared::model::ChannelGroupFailoverConfig;
use std::{
sync::Arc,
task::{Context, Poll},
time::Duration,
};
struct FakeHost {
body: Option<BoxedProviderStream>,
preempted: bool,
released: u32,
installed: u32,
}
impl FakeHost {
fn new(body: BoxedProviderStream, preempted: bool) -> Self {
Self { body: Some(body), preempted, released: 0, installed: 0 }
}
}
impl MemberHost for FakeHost {
fn member_body(&mut self) -> Option<&mut BoxedProviderStream> { self.body.as_mut() }
fn member_preempted(&self) -> bool { self.preempted }
fn release_member(&mut self) {
self.released += 1;
self.body = None;
}
fn install_member(&mut self, member: RuntimeMember) {
self.installed += 1;
self.body = Some(member.stream);
}
}
async fn switcher(failover: ChannelGroupFailoverConfig) -> (GroupPlaybackEnv, GroupSwitcher) {
let env = GroupPlaybackEnv::builder(&["solis", "korath"]).build().await;
let user = env.app_state.app_config.get_user_credentials("viewer").expect("viewer");
let ctx = GroupFailoverContext {
app_state: Arc::clone(&env.app_state),
target: Arc::clone(&env.target),
target_name: env.target.name.as_str().into(),
group: Arc::clone(&env.group_record),
playback: shared::model::ChannelGroupPlayback {
failover,
..shared::model::ChannelGroupsConfig::default().playback(None)
},
user: (*user).clone(),
fingerprint: crate::auth::Fingerprint::new(
"fp".to_string(),
"127.0.0.1".to_string(),
std::net::SocketAddr::from(([127, 0, 0, 1], 1)),
),
req_headers: axum::http::HeaderMap::new(),
session_token: "switcher-session".into(),
connection_kind: tuliprox_session::active_provider_manager::ConnectionKind::Normal,
connection_permission: shared::model::UserConnectionPermission::Allowed,
share_stream: false,
base_channel: shared::model::StreamChannel::default(),
shared_key: None,
};
let switcher = GroupSwitcher::new(GroupFailoverDriver::new(Arc::new(ctx), 0));
(env, switcher)
}
fn no_budget() -> ChannelGroupFailoverConfig {
ChannelGroupFailoverConfig { max_outage_secs: 0, ..ChannelGroupFailoverConfig::default() }
}
async fn poll(switcher: &mut GroupSwitcher, host: &mut FakeHost) -> GroupPoll {
std::future::poll_fn(|cx: &mut Context<'_>| match switcher.poll_member(host, cx) {
GroupPoll::Pending => Poll::Pending,
ready => Poll::Ready(ready),
})
.await
}
#[tokio::test]
async fn preempted_member_ends_without_switch() {
let (_env, mut switcher) = switcher(ChannelGroupFailoverConfig::default()).await;
let mut host = FakeHost::new(futures::stream::empty().boxed(), true);
assert!(matches!(poll(&mut switcher, &mut host).await, GroupPoll::Ended(GroupEnd::Eof)));
assert_eq!((host.released, host.installed), (0, 0));
assert!(!switcher.is_switching());
}
#[tokio::test]
async fn spent_budget_ends_with_the_body_end() {
let (_env, mut switcher) = switcher(no_budget()).await;
let error = futures::stream::iter([Err(StreamError::Stream("provider reset".to_string()))]).boxed();
let mut host = FakeHost::new(error, false);
assert!(matches!(poll(&mut switcher, &mut host).await, GroupPoll::Ended(GroupEnd::Error(_))));
assert_eq!(host.released, 0, "nothing is released when no switch follows");
}
#[tokio::test]
async fn split_chunks_reach_the_client_in_order_before_the_end() {
let (_env, mut switcher) = switcher(no_budget()).await;
let ts = member_ts(0x100, 10);
let chunks = vec![Ok(Bytes::copy_from_slice(&ts[..300])), Ok(Bytes::copy_from_slice(&ts[300..]))];
let mut host = FakeHost::new(futures::stream::iter(chunks).boxed(), false);
let mut sent = Vec::new();
loop {
match poll(&mut switcher, &mut host).await {
GroupPoll::Data(bytes) => sent.extend_from_slice(&bytes),
GroupPoll::Ended(GroupEnd::Eof) => break,
_ => panic!("unexpected poll result"),
}
}
assert_eq!(sent, ts[..ts.len() / 188 * 188]);
}
#[tokio::test]
async fn stall_without_budget_ends_stalled() {
let selection = ChannelGroupFailoverConfig { no_data_timeout_secs: 1, ..no_budget() };
let (_env, mut switcher) = switcher(selection).await;
let mut host = FakeHost::new(futures::stream::pending().boxed(), false);
let ended =
tokio::time::timeout(Duration::from_secs(3), poll(&mut switcher, &mut host)).await.expect("bounded");
assert!(matches!(ended, GroupPoll::Ended(GroupEnd::Stalled)));
}
/// A packet on PID 0x100 whose adaptation field carries `pcr_27mhz`.
fn pcr_packet(pcr_27mhz: u64) -> Vec<u8> {
let base = pcr_27mhz / 300;
let ext = pcr_27mhz % 300;
let byte = |value: u64| u8::try_from(value & 0xFF).expect("byte");
let mut packet = vec![0x47, 0x01, 0x00, 0x20, 183, 0x10];
packet.extend([
byte(base >> 25),
byte(base >> 17),
byte(base >> 9),
byte(base >> 1),
byte((base & 1) << 7) | 0x7E | byte(ext >> 8),
byte(ext),
]);
packet.resize(188, 0xFF);
packet
}
#[tokio::test]
async fn slow_member_triggers_underrun_switch() {
let selection = ChannelGroupFailoverConfig {
slow_source_pct: 90,
slow_source_secs: 2,
no_data_timeout_secs: 0,
..ChannelGroupFailoverConfig::default()
};
let (_env, mut switcher) = switcher(selection).await;
// Every 100 ms of wall time the provider delivers 40 ms of media.
let slow = futures::stream::unfold(27_000_000_u64, |pcr| async move {
tokio::time::sleep(Duration::from_millis(100)).await;
let next = pcr + 27_000 * 40;
Some((Ok::<Bytes, StreamError>(Bytes::from(pcr_packet(next))), next))
})
.boxed();
let mut host = FakeHost::new(slow, false);
let underrun_switch = tokio::time::timeout(Duration::from_secs(8), async {
while host.released == 0 {
let _ = poll(&mut switcher, &mut host).await;
}
})
.await;
assert!(underrun_switch.is_ok(), "the underrun starts a switch");
}
#[tokio::test]
async fn bytes_that_never_reach_the_client_do_not_hold_off_the_stall() {
let selection = ChannelGroupFailoverConfig { no_data_timeout_secs: 1, ..no_budget() };
let (_env, mut switcher) = switcher(selection).await;
// Text instead of MPEG-TS every 100 ms: the aligner drops it all.
let garbage = futures::stream::unfold((), |()| async {
tokio::time::sleep(Duration::from_millis(100)).await;
Some((Ok::<Bytes, StreamError>(Bytes::from_static(b"<html>offline</html>")), ()))
})
.boxed();
let mut host = FakeHost::new(garbage, false);
let ended = tokio::time::timeout(Duration::from_secs(4), poll(&mut switcher, &mut host)).await;
assert!(matches!(ended, Ok(GroupPoll::Ended(GroupEnd::Stalled))), "no media within the stall timeout");
}
}
@@ -345,6 +345,7 @@ pub(crate) async fn local_stream_response(
req_headers,
meter_uid: metering.meter_uid,
meter_stream: metering.meter_stream,
group_failover: None,
})
.await
{
+1 -1
View File
@@ -1,4 +1,4 @@
//! Gate 1: a new playback request for an input in maintenance gets the
//! Maintenance check: a new playback request for an input in maintenance gets the
//! maintenance clip instead of a provider connection.
//!
//! Requests that continue something already admitted pass: the DVR's internal
+5 -2
View File
@@ -233,11 +233,13 @@ pub enum ResourceFetchPolicy {
const API_STREAM_CHUNK_SIZE: usize = 64 * 1024;
#[cfg(test)]
mod tests;
pub(crate) mod tests;
mod activation;
mod auth;
mod catchup;
pub(crate) mod channel_group;
pub(crate) mod channel_group_failover;
mod clock;
mod local_files;
mod maintenance;
@@ -318,12 +320,13 @@ pub use self::{
strategy::redirect_response,
streaming::{
force_provider_stream_response, is_hls_stream_share_enabled, is_seek_request, is_seekable_media_request,
is_stream_share_enabled,
is_stream_share_enabled, sticky_playback_context,
},
};
pub(crate) use self::{
auth::{input_for_user, with_recording_headers},
catchup::get_catchup_session_ttl_secs,
channel_group::{group_playback_settings, resolve_hls_group_entry, GroupSelectionRequest, PlaybackSource},
local_files::local_stream_response,
maintenance::{blocks_new_playback, is_running_playback, maintenance_rejection_response},
provider_acquire::{acquire_exact_provider_handle, stream_response_with_provider_handle},
+413 -80
View File
@@ -1,5 +1,11 @@
use super::{
activate_session_before_stream_open, admission_failure_response, blocks_new_playback,
channel_group::{
acquire_group_member, group_playback_settings, group_representative, group_shared_key, open_group_member,
wait_for_user_capacity, GroupOpenRequest, GroupSelectionFailure, GroupSelectionRequest, OpenedGroupMember,
PlaybackSource, ProbedPayload,
},
channel_group_failover::{GroupFailoverContext, GroupFailoverDriver, GroupFailoverStream},
cleanup_failed_detected_catchup_hls, connection_priority_for_kind, create_stream_response_details,
detected_catchup_hls_response, get_session_reservation_ttl_secs, get_stream_options,
is_socket_bound_playback_session, is_stream_share_enabled, mark_response_as_uncompressed,
@@ -25,13 +31,17 @@ use axum::{
http::{HeaderMap, StatusCode},
response::IntoResponse,
};
use futures::StreamExt;
use log::{debug, error, log_enabled, trace};
use shared::{
model::{ConnectFailureReason, FailureStage, PlaylistItemType, StreamChannel, UserConnectionPermission, VirtualId},
model::{
ConnectFailureReason, EventMessage, FailureStage, PlaylistItemType, StreamChannel, UserConnectionPermission,
VirtualId,
},
utils::{extract_extension_from_url, sanitize_sensitive_info, Internable},
};
use std::{borrow::Cow, sync::Arc, time::Duration};
use tuliprox_session::ProviderSessionHeaders;
use tuliprox_session::{channel_group::FailureClass, ProviderSessionHeaders};
pub(super) fn create_unmapped_provider_stream(app_config: &AppConfig) -> ProviderStreamState {
ProviderStreamState::Custom {
@@ -293,10 +303,9 @@ pub(crate) async fn stream_response_with_provider_handle(
session_token: &str,
request_class: Option<PlaybackRequestClass>,
mut stream_channel: StreamChannel,
stream_url: &str,
source: PlaybackSource<'_>,
pinned_provider: Option<&Arc<str>>,
req_headers: &HeaderMap,
input: &Arc<ConfigInput>,
target: &Arc<ConfigTarget>,
user: &ProxyUserCredentials,
connection_permission: UserConnectionPermission,
@@ -305,26 +314,53 @@ pub(crate) async fn stream_response_with_provider_handle(
grace_mode: Option<crate::api::model::GraceMode>,
preacquired_provider_handle: Option<tuliprox_session::ManagedProviderHandle>,
) -> impl IntoResponse + Send {
if blocks_new_playback(app_state, input, user, preacquired_provider_handle.is_some()) {
return admission_failure_response(
app_state,
fingerprint,
user,
stream_channel,
Arc::clone(&input.name),
req_headers,
ConnectFailureReason::Maintenance,
);
if let PlaybackSource::Single { input, .. } = &source {
if blocks_new_playback(app_state, input, user, preacquired_provider_handle.is_some()) {
return admission_failure_response(
app_state,
fingerprint,
user,
stream_channel,
Arc::clone(&input.name),
req_headers,
ConnectFailureReason::Maintenance,
);
}
}
let _transition_guard = app_state.active_users.acquire_playback_transition(&user.username, session_token).await;
let request_log_stream_url = resolve_request_url_for_logging(input, stream_url);
if let PlaybackSource::Group { group, .. } = &source {
stream_channel.group_virtual_id = Some(group.group_vid.get());
}
// A group stands in with its first resolvable member until the selection below picks one.
let (initial_input, initial_stream_url): (Arc<ConfigInput>, Cow<'_, str>) = match &source {
PlaybackSource::Single { input, stream_url } => (Arc::clone(input), Cow::Borrowed(*stream_url)),
PlaybackSource::Group { group, .. } => {
let Some((input, url)) = group_representative(app_state, group) else {
app_state.event_manager.send_event(EventMessage::ChannelGroupExhausted {
target_id: target.id,
group_vid: group.group_vid.get(),
});
return create_custom_video_stream_response(
&app_state.provider_stream_ctx(),
&fingerprint.addr,
CustomVideoStreamType::ChannelUnavailable,
)
.into_response();
};
(input, Cow::Owned(url.to_string()))
}
};
let request_log_stream_url = resolve_request_url_for_logging(&initial_input, &initial_stream_url);
if log_enabled!(log::Level::Trace) {
trace!("Try to open stream {}", sanitize_sensitive_info(request_log_stream_url.as_ref()));
}
let virtual_id = stream_channel.virtual_id;
let virtual_id = stream_channel.client_virtual_id();
let item_type = stream_channel.item_type;
let playback_extension = extract_extension_from_url(stream_url);
let playback_extension = extract_extension_from_url(&initial_stream_url);
let socket_bound = is_socket_bound_playback_session(item_type, playback_extension);
let mut connection_permission = connection_permission;
let mut connection_kind = connection_kind;
@@ -332,13 +368,13 @@ pub(crate) async fn stream_response_with_provider_handle(
app_state,
SessionActivationRequest {
fingerprint,
input,
input: &initial_input,
user,
session_token,
request_class,
virtual_id: VirtualId::new(virtual_id),
item_type,
stream_url,
stream_url: &initial_stream_url,
connection_permission,
connection_kind,
granted_grace_mode: grace_mode,
@@ -353,14 +389,19 @@ pub(crate) async fn stream_response_with_provider_handle(
let allow_shared_reuse =
connection_permission != UserConnectionPermission::Exhausted || allow_exhausted_shared_reconnect;
let shared_lookup_key: Cow<'_, str> = match &source {
PlaybackSource::Group { group, .. } => Cow::Owned(group_shared_key(target.id, group.group_vid.get())),
PlaybackSource::Single { stream_url, .. } => Cow::Borrowed(*stream_url),
};
let share_stream = is_stream_share_enabled(item_type, target);
let _shared_lock = if share_stream {
let write_lock = app_state.app_config.file_locks.write_lock_str(stream_url).await;
let mut shared_lock = if share_stream {
let write_lock = app_state.app_config.file_locks.write_lock_str(shared_lookup_key.as_ref()).await;
if allow_shared_reuse {
if let Some(value) = try_shared_stream_response_if_any(
app_state,
stream_url,
shared_lookup_key.as_ref(),
fingerprint,
user,
connection_permission,
@@ -382,7 +423,7 @@ pub(crate) async fn stream_response_with_provider_handle(
if item_type == PlaylistItemType::Live && allow_shared_reuse {
if let Some(value) = try_shared_stream_response_if_any(
app_state,
stream_url,
shared_lookup_key.as_ref(),
fingerprint,
user,
connection_permission,
@@ -394,7 +435,10 @@ pub(crate) async fn stream_response_with_provider_handle(
)
.await
{
debug_if_enabled!("Opportunistic shared stream reuse for {}", sanitize_sensitive_info(stream_url));
debug_if_enabled!(
"Opportunistic shared stream reuse for {}",
sanitize_sensitive_info(shared_lookup_key.as_ref())
);
return value.into_response();
}
}
@@ -419,7 +463,7 @@ pub(crate) async fn stream_response_with_provider_handle(
fingerprint,
user,
stream_channel: stream_channel.clone(),
provider_name: input.name.clone(),
provider_name: initial_input.name.clone(),
req_headers,
reason: ConnectFailureReason::UserConnectionsExhausted,
failure_stage: FailureStage::Admission,
@@ -432,53 +476,52 @@ pub(crate) async fn stream_response_with_provider_handle(
.into_response();
}
let stream_options = get_stream_options(&app_state.app_config, StreamResponseMode::Stream);
let session_state = app_state.active_users.get_and_update_user_session(&user.username, session_token).await;
let pinned_provider = pinned_provider.filter(|provider| {
item_type.is_live()
|| session_state
.as_ref()
.is_some_and(|session| session.media_started.load(std::sync::atomic::Ordering::Acquire))
|| app_state.active_provider.should_reuse_playback_provider(session_token, provider)
});
let provider_session_headers =
session_state.as_ref().and_then(|session| session.provider_session_headers_for(stream_url));
let mut stream_details = match create_stream_response_details(
app_state,
&stream_options,
stream_url,
&user.username,
fingerprint,
req_headers,
input,
&stream_channel,
item_type,
if item_type == PlaylistItemType::Catchup {
crate::api::model::ProviderContentRepresentationMode::Identity
} else {
crate::api::model::ProviderContentRepresentationMode::PreserveOrigin
},
share_stream,
connection_permission,
pinned_provider,
pinned_provider.is_none() || allows_provider_pool_failover(item_type),
true,
VirtualId::new(stream_channel.virtual_id),
connection_priority_for_kind(user, connection_kind),
connection_kind,
false,
Some(session_token),
provider_session_headers.as_deref(),
pinned_provider.is_some(),
grace_mode.map(|m| matches!(m, crate::api::model::GraceMode::Hold)),
activation.grace_context.clone(),
None,
preacquired_provider_handle,
)
.await
if matches!(&source, PlaybackSource::Group { .. })
&& connection_permission == UserConnectionPermission::GracePeriod
&& matches!(grace_mode, Some(crate::api::model::GraceMode::Hold))
{
Ok(stream_details) => stream_details,
Err(err) => {
// A hold can last the whole grace period: viewers joining this group's shared stream
// must not wait behind it, so the share lock is given up while holding.
drop(shared_lock.take());
let hold_duration = Duration::from_millis(app_state.get_grace_options().period_millis);
let mut passed = wait_for_user_capacity(app_state, user, tokio::time::Instant::now() + hold_duration).await;
if !passed {
if let Some(ref ctx) = activation.grace_context {
let eviction_guard = if socket_bound {
tuliprox_session::admission::EvictionReentryGuard::SocketPlayback {
virtual_id: shared::model::VirtualId(virtual_id),
}
} else {
tuliprox_session::admission::EvictionReentryGuard::Session(session_token)
};
let remaining_result = tuliprox_session::admission::evaluate_remaining_strategies_after_grace(
&app_state.admission_ctx(),
tuliprox_session::admission::AdmissionRequest {
username: &user.username,
max_connections: user.max_connections,
soft_connections: user.soft_connections,
client_ip: &fingerprint.client_ip,
request_addr: &fingerprint.addr,
use_session_admission: true,
session_token: Some(session_token),
activate_unbound_session: true,
eviction_reentry_guard: eviction_guard,
},
ctx,
activation.admission.kind(),
)
.await;
if matches!(
remaining_result.admission.permission(),
UserConnectionPermission::Allowed | UserConnectionPermission::GracePeriod
) {
passed = true;
}
}
}
if !passed {
app_state
.active_users
.release_unbound_session_reservation(
@@ -488,8 +531,262 @@ pub(crate) async fn stream_response_with_provider_handle(
activation.placeholder_transition_version.is_some(),
)
.await;
error!("Failed to stream: {err}");
return StatusCode::INTERNAL_SERVER_ERROR.into_response();
record_connect_failed_attempt(ConnectFailedAttempt {
app_state,
fingerprint,
user,
stream_channel: stream_channel.clone(),
provider_name: initial_input.name.clone(),
req_headers,
reason: ConnectFailureReason::UserConnectionsExhausted,
failure_stage: FailureStage::Admission,
});
return create_custom_video_stream_response(
&app_state.provider_stream_ctx(),
&fingerprint.addr,
CustomVideoStreamType::UserConnectionsExhausted,
)
.into_response();
}
connection_permission = UserConnectionPermission::Allowed;
if share_stream {
let write_lock = app_state.app_config.file_locks.write_lock_str(shared_lookup_key.as_ref()).await;
if let Some(value) = try_shared_stream_response_if_any(
app_state,
shared_lookup_key.as_ref(),
fingerprint,
user,
connection_permission,
connection_kind,
stream_channel.clone(),
session_token,
req_headers,
activation.placeholder_transition_version,
)
.await
{
return value.into_response();
}
let _ = shared_lock.insert(write_lock);
}
}
let stream_options = get_stream_options(&app_state.app_config, StreamResponseMode::Stream);
// A group started on an MPEG-TS member can fail over at runtime.
let (mut stream_details, input, stream_url, mut stream_channel, group_runtime) = match source {
PlaybackSource::Single { input, stream_url } => {
let session_state = app_state.active_users.get_and_update_user_session(&user.username, session_token).await;
let pinned_provider = pinned_provider.filter(|provider| {
item_type.is_live()
|| session_state
.as_ref()
.is_some_and(|session| session.media_started.load(std::sync::atomic::Ordering::Acquire))
|| app_state.active_provider.should_reuse_playback_provider(session_token, provider)
});
let provider_session_headers =
session_state.as_ref().and_then(|session| session.provider_session_headers_for(stream_url));
let details = match create_stream_response_details(
app_state,
&stream_options,
stream_url,
&user.username,
fingerprint,
req_headers,
input,
&stream_channel,
item_type,
if item_type == PlaylistItemType::Catchup {
crate::api::model::ProviderContentRepresentationMode::Identity
} else {
crate::api::model::ProviderContentRepresentationMode::PreserveOrigin
},
share_stream,
connection_permission,
pinned_provider,
pinned_provider.is_none() || allows_provider_pool_failover(item_type),
true,
VirtualId::new(stream_channel.virtual_id),
connection_priority_for_kind(user, connection_kind),
connection_kind,
false,
Some(session_token),
provider_session_headers.as_deref(),
pinned_provider.is_some(),
grace_mode.map(|m| matches!(m, crate::api::model::GraceMode::Hold)),
activation.grace_context.clone(),
None,
preacquired_provider_handle,
)
.await
{
Ok(stream_details) => stream_details,
Err(err) => {
app_state
.active_users
.release_unbound_session_reservation(
&user.username,
session_token,
activation.placeholder_transition_version,
activation.placeholder_transition_version.is_some(),
)
.await;
error!("Failed to stream: {err}");
return StatusCode::INTERNAL_SERVER_ERROR.into_response();
}
};
(details, Arc::clone(input), Arc::<str>::from(stream_url), stream_channel, None)
}
PlaybackSource::Group { group, mode, prefer } => {
let playback = group_playback_settings(app_state, target);
let target_name: Arc<str> = Arc::from(target.name.as_str());
let priority = connection_priority_for_kind(user, connection_kind);
// The start budget bounds selection, provider open and payload probe of all attempts.
let budget_secs = if playback.start.timeout_secs == 0 { 86_400 } else { playback.start.timeout_secs };
let start_deadline = tokio::time::Instant::now() + Duration::from_secs(budget_secs);
let open_req = GroupOpenRequest {
app_state,
user,
fingerprint,
req_headers,
group: &group,
base_channel: &stream_channel,
stream_options: &stream_options,
share_stream,
connection_permission,
connection_kind,
priority,
session_token,
is_reopen: false,
deadline: start_deadline,
};
let mut exclude: Vec<usize> = Vec::new();
let mut demoted_inputs: Vec<Arc<str>> = Vec::new();
let mut opened: Option<OpenedGroupMember> = None;
let mut maintenance_only = false;
for attempt in 0..playback.start.max_attempts.max(1) {
let remaining = start_deadline.saturating_duration_since(tokio::time::Instant::now());
if remaining.is_zero() {
break;
}
let req = GroupSelectionRequest {
mode: mode.clone(),
allow_provider_grace: playback.allow_provider_grace,
exclude: &exclude,
demoted_inputs: &demoted_inputs,
prefer: if attempt == 0 { prefer } else { None },
bypass_maintenance: user.is_recording_proxy_user(),
..GroupSelectionRequest::new(
&target_name,
&group,
playback.start,
&fingerprint.addr,
&user.username,
priority,
connection_kind,
)
};
let acquired = match tokio::time::timeout(remaining, acquire_group_member(app_state, req)).await {
Ok(Ok(acquired)) => acquired,
Ok(Err(GroupSelectionFailure::Maintenance)) => {
maintenance_only = exclude.is_empty();
break;
}
Ok(Err(GroupSelectionFailure::Exhausted | GroupSelectionFailure::NoCandidate)) | Err(_) => break,
};
match open_group_member(&open_req, acquired).await {
Ok(member) => {
opened = Some(member);
break;
}
Err(failure) => {
if matches!(failure.class, FailureClass::Auth | FailureClass::Capacity) {
demoted_inputs.push(failure.input_name);
}
exclude.push(failure.index);
}
}
}
let Some(opened) = opened else {
app_state
.active_users
.release_unbound_session_reservation(
&user.username,
session_token,
activation.placeholder_transition_version,
activation.placeholder_transition_version.is_some(),
)
.await;
if maintenance_only {
return admission_failure_response(
app_state,
fingerprint,
user,
stream_channel,
Arc::clone(&initial_input.name),
req_headers,
ConnectFailureReason::Maintenance,
);
}
app_state.event_manager.send_event(EventMessage::ChannelGroupExhausted {
target_id: target.id,
group_vid: group.group_vid.get(),
});
record_connect_failed_attempt(ConnectFailedAttempt {
app_state,
fingerprint,
user,
stream_channel: stream_channel.clone(),
provider_name: initial_input.name.clone(),
req_headers,
reason: ConnectFailureReason::ProviderConnectionsExhausted,
failure_stage: FailureStage::ProviderOpen,
});
return create_custom_video_stream_response(
&app_state.provider_stream_ctx(),
&fingerprint.addr,
CustomVideoStreamType::ProviderConnectionsExhausted,
)
.into_response();
};
app_state.event_manager.send_event(EventMessage::ChannelGroupMemberSelected {
target_id: target.id,
group_vid: group.group_vid.get(),
member_vid: opened.member.member_vid.get(),
tier: opened.tier,
});
app_state
.active_users
.bind_member(
&user.username,
session_token,
tuliprox_session::MemberBinding {
group_vid: group.group_vid.get(),
member_vid: opened.member.member_vid.get(),
member_index: opened.index,
input_name: Arc::clone(&opened.member.input_name),
},
)
.await;
// The session plays the member's provider now, not the representative's.
let member_provider =
opened.details.provider_name.clone().unwrap_or_else(|| Arc::clone(&opened.member.input_name));
app_state
.active_users
.update_session_provider_binding(
&user.username,
session_token,
member_provider,
Arc::clone(&opened.url),
)
.await;
let OpenedGroupMember { index, details, input, url, channel, payload, .. } = opened;
let runtime =
(playback.failover.enabled && payload == ProbedPayload::Ts && item_type == PlaylistItemType::Live)
.then(|| (Arc::clone(&group), index, playback));
(details, input, url, channel, runtime)
}
};
@@ -505,13 +802,13 @@ pub(crate) async fn stream_response_with_provider_handle(
manifest,
user,
target,
input,
input: &input,
fingerprint,
session_token,
virtual_id: VirtualId::new(virtual_id),
connection_permission,
connection_kind,
fallback_stream_url: stream_url,
fallback_stream_url: &stream_url,
})
.await;
}
@@ -539,8 +836,9 @@ pub(crate) async fn stream_response_with_provider_handle(
.as_ref()
.map(|(h, sc, response_url, cvt)| (h.clone(), *sc, response_url.clone(), *cvt));
let provider_name = stream_details.provider_name.clone();
let actual_request_url = stream_details.request_url.clone().unwrap_or_else(|| Arc::<str>::from(stream_url));
let log_actual_request_url = resolve_request_url_for_logging(input, actual_request_url.as_ref());
let actual_request_url =
stream_details.request_url.clone().unwrap_or_else(|| Arc::<str>::from(stream_url.as_ref()));
let log_actual_request_url = resolve_request_url_for_logging(&input, actual_request_url.as_ref());
debug_if_enabled!(
"Provider request mapping: allocated_provider={} actual_request_url={}",
@@ -603,7 +901,7 @@ pub(crate) async fn stream_response_with_provider_handle(
};
let mut metering = prepare_stream_metering(
app_state,
stream_url,
if is_stream_shared { shared_lookup_key.as_ref() } else { stream_url.as_ref() },
is_stream_shared,
stream_details.stream.is_some(),
stream_details.has_deferred_provider_open(),
@@ -670,6 +968,40 @@ pub(crate) async fn stream_response_with_provider_handle(
} else {
false
};
let has_provider_stream = stream_details.stream.is_some() && stream_details.custom_reason.is_none();
let group_driver = group_runtime.filter(|_| has_provider_stream).map(|(group, index, playback)| {
GroupFailoverDriver::new(
Arc::new(GroupFailoverContext {
app_state: Arc::clone(app_state),
target: Arc::clone(target),
target_name: target.name.as_str().into(),
group,
playback,
user: user.clone(),
fingerprint: fingerprint.clone(),
req_headers: req_headers.clone(),
session_token: Arc::from(session_token),
connection_kind,
connection_permission,
share_stream,
base_channel: stream_channel.clone(),
shared_key: is_stream_shared.then(|| Arc::from(shared_lookup_key.as_ref())),
}),
index,
)
});
// A shared producer fails over inside the stream it feeds; a direct response through its
// client stream, which owns the provider slot.
let group_failover = match group_driver {
Some(driver) if is_stream_shared => {
// The shared stream takes over this handle below.
let slot = provider_handle.as_ref();
stream_details.stream =
stream_details.stream.take().map(|stream| GroupFailoverStream::new(driver, stream, slot).boxed());
None
}
driver => driver,
};
let stream = match create_active_client_stream(crate::api::model::ActiveClientStreamParams {
stream_details,
app_state,
@@ -683,6 +1015,7 @@ pub(crate) async fn stream_response_with_provider_handle(
req_headers,
meter_uid: metering.meter_uid,
meter_stream: metering.meter_stream,
group_failover,
})
.await
{
@@ -722,7 +1055,7 @@ pub(crate) async fn stream_response_with_provider_handle(
active_provider: &app_state.active_provider,
connection_manager: &app_state.connection_manager,
},
stream_url,
shared_lookup_key.as_ref(),
stream,
&fingerprint.addr,
shared_subscriber_id,
@@ -774,7 +1107,7 @@ pub(crate) async fn stream_response_with_provider_handle(
.and_then(|(_, _, u, _)| u.as_ref())
.map_or_else(|| Cow::Owned(actual_request_url.to_string()), |url| Cow::Owned(url.to_string()))
};
let log_session_url = resolve_request_url_for_logging(input, session_url.as_ref());
let log_session_url = resolve_request_url_for_logging(&input, session_url.as_ref());
if log_enabled!(log::Level::Debug) {
if log_session_url.eq(log_actual_request_url.as_ref()) {
debug!(
+30 -9
View File
@@ -7,8 +7,9 @@ use super::{
resolve_streaming_strategy_with_provider_handle, session_reacquire_cleanup_addrs,
should_defer_provider_open_for_grace_hold, should_refresh_stalker_playback, split_provider_stream_open,
stalker_stream_kind, stream_admission_rejected_response, stream_response_with_provider_handle, try_unwrap_body,
ConnectFailedAttempt, CurrentSessionGuard, ForceStreamRequestContext, PlaybackRequestClass, RedirectParams,
ResourceFetchPolicy, StreamMeteringConfig, StreamOptions, StreamResponseMode, HLS_MEDIA_CAPACITY_WAIT,
ConnectFailedAttempt, CurrentSessionGuard, ForceStreamRequestContext, PlaybackRequestClass, PlaybackSource,
RedirectParams, ResourceFetchPolicy, StreamMeteringConfig, StreamOptions, StreamResponseMode,
HLS_MEDIA_CAPACITY_WAIT,
};
use crate::{
api::{
@@ -608,6 +609,24 @@ pub(super) fn no_custom_video_fallback_status(app_config: &AppConfig) -> StatusC
}
}
pub fn sticky_playback_context(
app_state: &Arc<AppState>,
session: &UserSession,
fallback_input: &Arc<ConfigInput>,
fallback_channel: StreamChannel,
) -> (Arc<ConfigInput>, StreamChannel) {
if let Some(binding) = &session.member_binding {
if let Some(member_input) = app_state.app_config.get_input_by_name(&binding.input_name) {
let mut channel = fallback_channel;
channel.virtual_id = binding.member_vid;
channel.group_virtual_id = Some(binding.group_vid);
channel.input_name = Arc::clone(&binding.input_name);
return (member_input, channel);
}
}
(Arc::clone(fallback_input), fallback_channel)
}
pub async fn force_provider_stream_response(
fingerprint: &Fingerprint,
app_state: &Arc<AppState>,
@@ -657,11 +676,13 @@ pub(super) async fn force_stream_response(
fingerprint: &Fingerprint,
app_state: &Arc<AppState>,
user_session: &UserSession,
mut stream_channel: StreamChannel,
stream_channel: StreamChannel,
ctx: ForceStreamRequestContext<'_>,
grace_mode: Option<crate::api::model::GraceMode>,
response_mode: StreamResponseMode,
) -> axum::response::Response {
let (sticky_input, mut stream_channel) =
sticky_playback_context(app_state, user_session, ctx.input, stream_channel);
let hls_resource = response_mode == StreamResponseMode::HlsResource;
// Finite HLS resources stay pinned to the session account and never rebind or clean up
// other sockets, so they skip the session transition gate. Holding it across the
@@ -739,7 +760,7 @@ pub(super) async fn force_stream_response(
&ctx.user.username,
fingerprint,
ctx.req_headers,
ctx.input,
&sticky_input,
&stream_channel,
item_type,
ctx.content_representation,
@@ -898,6 +919,7 @@ pub(super) async fn force_stream_response(
req_headers: ctx.req_headers,
meter_uid: metering.meter_uid,
meter_stream: metering.meter_stream,
group_failover: None,
})
.await
{
@@ -952,7 +974,7 @@ pub(super) async fn force_stream_response(
debug_if_enabled!(
"Streaming provider forced stream request from {}",
sanitize_sensitive_info(
resolve_request_url_for_logging(ctx.input, user_session.stream_url.as_ref()).as_ref()
resolve_request_url_for_logging(&sticky_input, user_session.stream_url.as_ref()).as_ref()
)
);
let mut response = try_unwrap_body!(response.body(body_stream));
@@ -990,10 +1012,9 @@ pub(crate) async fn stream_response(
session_token: &str,
request_class: Option<PlaybackRequestClass>,
stream_channel: StreamChannel,
stream_url: &str,
source: PlaybackSource<'_>,
pinned_provider: Option<&Arc<str>>,
req_headers: &HeaderMap,
input: &Arc<ConfigInput>,
target: &Arc<ConfigTarget>,
user: &ProxyUserCredentials,
connection_permission: UserConnectionPermission,
@@ -1007,10 +1028,9 @@ pub(crate) async fn stream_response(
session_token,
request_class,
stream_channel,
stream_url,
source,
pinned_provider,
req_headers,
input,
target,
user,
connection_permission,
@@ -1139,6 +1159,7 @@ pub(super) async fn try_shared_stream_response_if_any(
req_headers,
meter_uid: metering.meter_uid,
meter_stream: metering.meter_stream,
group_failover: None,
})
.await
{
+7
View File
@@ -37,3 +37,10 @@ mod stream_responses_provider_capacity;
mod stream_responses_provider_fallback;
mod stream_responses_response_headers;
mod stream_responses_session_activation;
mod channel_group_playback;
mod channel_group_probe;
mod channel_group_runtime;
mod channel_group_selection;
mod channel_group_sticky;
pub(crate) mod group_playback_support;
@@ -4,7 +4,7 @@ use super::{
create_test_provider_app_config, create_test_shared_target, load_test_channel, load_test_user,
resolve_admission_with_strategies, should_allow_exhausted_shared_reconnect,
should_defer_provider_open_for_grace_hold, spawn_load_test_origin, stream_admission_rejected_response,
stream_response, AdmissionRequest, EvictionReentryGuard, SessionActivationRequest,
stream_response, AdmissionRequest, EvictionReentryGuard, PlaybackSource, SessionActivationRequest,
};
use crate::{
api::model::{CustomVideoStreamType, UserSession},
@@ -99,10 +99,9 @@ async fn shared_admission_rejection_does_not_leak_meter_registration() {
"shared-session-adm",
None,
channel,
&stream_url,
PlaybackSource::Single { input: &input, stream_url: &stream_url },
None,
&HeaderMap::default(),
&input,
&target,
&user,
UserConnectionPermission::Allowed,
@@ -0,0 +1,178 @@
use super::{
group_playback_support::{ts_fixture, GroupPlaybackEnv},
support::FakeOriginMode,
};
use shared::model::AdmissionStrategy;
use std::sync::Arc;
use tuliprox_session::channel_group::CircuitView;
#[tokio::test]
async fn xtream_group_playback_uses_member_input() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"])
.with_qos("korath", 2, 95)
.with_qos("solis", 1, 20)
.build()
.await;
let response = env.get_xtream_live(env.group_vid).await;
assert_eq!(response.status(), 200);
assert_eq!(env.origin_hits("korath"), 1);
assert_eq!(env.origin_hits("solis"), 0);
}
#[tokio::test]
async fn start_failover_moves_to_next_member_on_open_failure() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"]).origin_status("solis", 503).build().await;
let response = env.get_xtream_live(env.group_vid).await;
assert_eq!(response.status(), 200);
assert_eq!(env.origin_hits("korath"), 1);
assert!(matches!(env.circuit("solis"), CircuitView::Counting(1)));
}
#[tokio::test]
async fn capacity_rejection_moves_to_other_input_without_circuit() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"]).origin_status("solis", 429).build().await;
let response = env.get_xtream_live(env.group_vid).await;
assert_eq!(response.status(), 200);
assert_eq!(env.origin_hits("korath"), 1);
assert!(matches!(env.circuit("solis"), CircuitView::Closed));
}
#[tokio::test]
async fn non_ts_first_chunk_fails_over() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"])
.origin_body("solis", "video/mp2t", b"Channel offline, please try later".to_vec())
.build()
.await;
assert_eq!(env.get_xtream_live(env.group_vid).await.status(), 200);
assert_eq!(env.origin_hits("korath"), 1);
assert!(matches!(env.circuit("solis"), CircuitView::Counting(1)));
}
#[tokio::test]
async fn ts_with_leading_garbage_is_accepted() {
let mut body = vec![0u8; 17];
body.extend(ts_fixture());
let env = GroupPlaybackEnv::builder(&["solis", "korath"]).origin_body("solis", "video/mp2t", body).build().await;
assert_eq!(env.get_xtream_live(env.group_vid).await.status(), 200);
assert_eq!(env.origin_hits("korath"), 0);
}
#[tokio::test]
async fn auth_failure_counts_for_member_and_input() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"]).origin_status("solis", 401).build().await;
assert_eq!(env.get_xtream_live(env.group_vid).await.status(), 200);
assert_eq!(env.origin_hits("korath"), 1);
assert!(matches!(env.circuit("solis"), CircuitView::Counting(1)));
assert!(matches!(env.input_circuit("solis"), CircuitView::Closed)); // one member only, input stays closed
}
#[tokio::test]
async fn member_vid_request_plays_group_preferring_that_member() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"]).with_qos("korath", 2, 95).build().await;
let response = env.get_xtream_live(env.member_vid("solis")).await;
assert_eq!(response.status(), 200);
assert_eq!((env.origin_hits("solis"), env.origin_hits("korath")), (1, 0)); // preferred despite lower QoS
}
#[tokio::test]
async fn favourite_copy_plays_as_group() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"]).with_favourite().with_qos("korath", 2, 95).build().await;
assert_eq!(env.get_xtream_live(env.favourite_vid()).await.status(), 200);
assert_eq!(env.origin_hits("korath"), 1);
}
#[tokio::test]
async fn non_group_playback_is_unchanged() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"]).build().await;
assert_eq!(env.get_xtream_live(env.plain_vid).await.status(), 200);
assert_eq!(env.origin_hits("solis"), 1);
}
#[tokio::test]
async fn m3u_group_playback_uses_member_input() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"]).with_qos("korath", 2, 95).build().await;
assert_eq!(env.get_m3u_live(env.group_vid).await.status(), 200);
assert_eq!(env.origin_hits("solis"), 0);
}
#[tokio::test]
async fn stalker_member_url_resolved_after_selection() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"]).stalker("solis").with_qos("korath", 2, 95).build().await;
assert_eq!(env.get_xtream_live(env.group_vid).await.status(), 200);
assert_eq!(env.origin_hits("solis"), 0); // no Stalker create_link against the representative
}
#[tokio::test]
async fn admission_uses_group_vid_for_eviction() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"])
.user_limit(1)
.admission(vec![AdmissionStrategy::EvictUserSameIpOldest])
.build()
.await;
let first = env.get_xtream_live(env.group_vid).await;
assert_eq!(first.status(), 200);
let second = env.get_xtream_live(env.plain_vid).await;
assert_eq!(second.status(), 200);
assert_eq!(env.active_session_vids("viewer").await, vec![env.plain_vid]); // group session evicted by its group vid
}
#[tokio::test]
async fn grace_hold_selects_member_only_after_hold_passes() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"])
.user_limit(1)
.admission(vec![AdmissionStrategy::GraceHoldStream])
.build()
.await;
let _first = env.get_xtream_live(env.plain_vid).await;
let slots_before = env.slots_in_use();
let rejected = env.get_xtream_live(env.group_vid).await; // first stream still active → hold fails
assert_ne!(rejected.status(), 200);
assert_eq!(env.slots_in_use(), slots_before); // no member slot was taken during the hold
}
#[tokio::test]
async fn group_skips_member_in_maintenance() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"]).with_qos("solis", 1, 95).build().await;
env.app_state.maintenance.add("solis", None, 1).await.expect("maintenance");
assert_eq!(env.get_xtream_live(env.group_vid).await.status(), 200);
assert_eq!((env.origin_hits("solis"), env.origin_hits("korath")), (0, 1));
}
#[tokio::test]
async fn group_with_all_members_in_maintenance_returns_maintenance() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"]).build().await;
env.app_state.maintenance.add("solis", None, 1).await.expect("maintenance");
env.app_state.maintenance.add("korath", None, 1).await.expect("maintenance");
let response = env.get_xtream_live(env.group_vid).await;
assert_ne!(response.status(), 200);
assert_eq!((env.origin_hits("solis"), env.origin_hits("korath")), (0, 0));
assert_eq!(env.slots_in_use(), 0);
}
#[tokio::test]
async fn start_budget_bounds_all_attempts() {
let blocked = Arc::new(tokio::sync::Notify::new());
let channel_groups = shared::model::ChannelGroupsConfig {
start: shared::model::ChannelGroupStartConfig { timeout_secs: 2, ..Default::default() },
..Default::default()
};
let env = GroupPlaybackEnv::builder(&["solis", "korath"])
.origin_mode("solis", FakeOriginMode::BlockFirstChunk(Arc::clone(&blocked)))
.origin_mode("korath", FakeOriginMode::BlockFirstChunk(Arc::clone(&blocked)))
.with_channel_groups(channel_groups)
.build()
.await;
let started = std::time::Instant::now();
let response = env.get_xtream_live(env.group_vid).await;
let elapsed = started.elapsed();
blocked.notify_waiters();
assert!(elapsed < std::time::Duration::from_millis(2_500), "start took {elapsed:?}");
drop(response);
// A failed open releases its slot once the dropped provider body completes.
let released = async {
while env.slots_in_use() != 0 {
tokio::time::sleep(std::time::Duration::from_millis(20)).await;
}
};
assert!(tokio::time::timeout(std::time::Duration::from_secs(2), released).await.is_ok());
}
@@ -0,0 +1,213 @@
use super::{
super::channel_group::{
probe_member_payload, wait_for_capacity, wait_for_user_capacity, ProbedPayload, PROBE_MIN_BYTES,
},
group_playback_support::ts_fixture,
support::{create_test_app_config, create_test_app_state_for_config},
};
use crate::{
api::model::{BoxedProviderStream, StreamError},
model::ProxyUserCredentials,
};
use bytes::Bytes;
use futures::StreamExt;
use std::{sync::Arc, time::Duration};
use tokio::time::Instant;
fn stream_of(chunks: Vec<Bytes>) -> BoxedProviderStream {
futures::stream::iter(chunks.into_iter().map(Ok::<Bytes, StreamError>)).boxed()
}
async fn collect(stream: BoxedProviderStream) -> Vec<u8> {
stream
.fold(Vec::new(), |mut acc, chunk| async move {
acc.extend_from_slice(&chunk.expect("chunk"));
acc
})
.await
}
fn deadline_in(secs: u64) -> Instant { Instant::now() + Duration::from_secs(secs) }
#[tokio::test]
async fn probe_accepts_ts_split_over_chunks() {
let ts = ts_fixture();
let chunks = ts.chunks(100).map(Bytes::copy_from_slice).collect();
let (stream, payload) = probe_member_payload(stream_of(chunks), false, deadline_in(3)).await.expect("ts accepted");
assert!(matches!(payload, ProbedPayload::Ts));
assert_eq!(collect(stream).await, ts);
}
#[tokio::test]
async fn probe_accepts_extm3u_with_text_plain() {
let body = Bytes::from_static(b"\xEF\xBB\xBF #EXTM3U\n#EXT-X-VERSION:3\nsegment.ts\n");
let (stream, payload) =
probe_member_payload(stream_of(vec![body.clone()]), false, deadline_in(3)).await.expect("playlist accepted");
assert!(matches!(payload, ProbedPayload::Playlist));
assert_eq!(collect(stream).await, body.to_vec());
}
#[tokio::test]
async fn probe_trusts_adaptive_hint_without_reading() {
let pending: BoxedProviderStream = futures::stream::pending().boxed();
let (_stream, payload) = probe_member_payload(pending, true, deadline_in(3)).await.expect("hint accepted");
assert!(matches!(payload, ProbedPayload::Playlist));
}
#[tokio::test]
async fn probe_rejects_garbage() {
let body = Bytes::from(vec![b'x'; PROBE_MIN_BYTES * 2]);
assert!(probe_member_payload(stream_of(vec![body]), false, deadline_in(3)).await.is_err());
}
#[tokio::test(start_paused = true)]
async fn probe_rejects_endless_empty_chunks_within_deadline() {
let started = Instant::now();
let deadline = started + Duration::from_secs(3);
let empty: BoxedProviderStream = futures::stream::repeat_with(|| Ok::<Bytes, StreamError>(Bytes::new())).boxed();
assert!(probe_member_payload(empty, false, deadline).await.is_err());
assert!(Instant::now() <= deadline);
}
#[tokio::test(start_paused = true)]
async fn probe_times_out_on_silent_stream() {
let deadline = deadline_in(3);
let silent: BoxedProviderStream = futures::stream::pending().boxed();
assert!(probe_member_payload(silent, false, deadline).await.is_err());
assert!(Instant::now() >= deadline);
}
#[tokio::test]
async fn probe_puts_read_chunks_back_in_order() {
let ts = ts_fixture();
let mut tail = ts.clone();
tail.extend_from_slice(&ts);
let chunks = vec![
Bytes::copy_from_slice(&ts[..10]),
Bytes::new(),
Bytes::copy_from_slice(&ts[10..]),
Bytes::from(ts.clone()),
];
let (stream, _) = probe_member_payload(stream_of(chunks), false, deadline_in(3)).await.expect("ts accepted");
assert_eq!(collect(stream).await, tail);
}
fn viewer(max_connections: u32) -> ProxyUserCredentials {
let mut user = ProxyUserCredentials::default();
user.username = "viewer".to_string();
user.max_connections = max_connections;
user
}
#[tokio::test]
async fn wait_for_user_capacity_returns_immediately_below_limit() {
let app_state = create_test_app_state_for_config(Arc::new(create_test_app_config()));
assert!(wait_for_user_capacity(&app_state, &viewer(1), deadline_in(2)).await);
}
#[tokio::test]
async fn wait_for_user_capacity_times_out() {
let app_state = create_test_app_state_for_config(Arc::new(create_test_app_config()));
let started = Instant::now();
assert!(!wait_for_user_capacity(&app_state, &viewer(0), started + Duration::from_millis(300)).await);
assert!(started.elapsed() >= Duration::from_millis(300));
}
#[tokio::test]
async fn wait_for_capacity_rechecks_on_notification() {
let notify = Arc::new(tokio::sync::Notify::new());
let checks = Arc::new(std::sync::atomic::AtomicUsize::new(0));
let started = Instant::now();
let waiter = {
let notify = Arc::clone(&notify);
let checks = Arc::clone(&checks);
tokio::spawn(async move {
wait_for_capacity(&notify, deadline_in(5), || {
let checks = Arc::clone(&checks);
async move { checks.fetch_add(1, std::sync::atomic::Ordering::SeqCst) >= 1 }
})
.await
})
};
tokio::time::sleep(Duration::from_millis(100)).await;
notify.notify_waiters();
assert!(waiter.await.expect("waiter"));
assert!(started.elapsed() < Duration::from_secs(1));
}
#[tokio::test]
async fn refresh_member_keys_keeps_only_group_member_probes() {
let env = super::group_playback_support::GroupPlaybackEnv::builder(&["solis", "korath"]).build().await;
let member = env.group_record.members[0].clone();
let mut stranger = member.clone();
stranger.input_stream_id = "not-a-member".into();
assert_eq!(crate::api::tasks::refresh_member_keys(&env.app_state).await, env.group_record.members.len());
let video = Some(r#"{"codec_name":"h264","width":1920,"height":1080}"#);
let health = &env.app_state.member_health;
health.patch_probe(&tuliprox_session::channel_group::member_key(&stranger), video, 4_000);
health.patch_probe(&tuliprox_session::channel_group::member_key(&member), video, 4_000);
let target: Arc<str> = env.target.name.as_str().into();
let now = std::time::Instant::now();
assert!(health.health(&target, &stranger, now).probe.is_none());
assert!(health.health(&target, &member, now).probe.is_some());
}
#[tokio::test]
async fn probe_rejects_short_text_body_with_sync_byte() {
let body = Bytes::from_static(b"Gone. Please try again later.");
assert!(probe_member_payload(stream_of(vec![body]), false, deadline_in(3)).await.is_err());
}
#[tokio::test]
async fn prefixed_stream_passes_inner_after_head() {
use super::super::channel_group::PrefixedStream;
let head = vec![Bytes::from_static(b"ab"), Bytes::from_static(b"c")];
let inner = stream_of(vec![Bytes::from_static(b"de")]);
let stream = PrefixedStream::new(head, inner).boxed();
assert_eq!(collect(stream).await, b"abcde");
}
async fn wait_for(timeout: Duration, mut condition: impl FnMut() -> bool) -> bool {
let deadline = Instant::now() + timeout;
while !condition() {
if Instant::now() >= deadline {
return false;
}
tokio::time::sleep(Duration::from_millis(20)).await;
}
true
}
#[tokio::test]
async fn member_key_task_refreshes_on_sources_change_and_stops_on_cancel() {
let env = super::group_playback_support::GroupPlaybackEnv::builder(&["solis", "korath"]).build().await;
let member = env.group_record.members[0].clone();
let key = tuliprox_session::channel_group::member_key(&member);
let target: Arc<str> = env.target.name.as_str().into();
let health = Arc::clone(&env.app_state.member_health);
let video = Some(r#"{"codec_name":"h264","width":1920,"height":1080}"#);
let cancel = tokio_util::sync::CancellationToken::new();
let task = crate::api::tasks::exec_channel_group_member_keys(&env.app_state, &cancel);
// The startup refresh registers the members.
assert!(
wait_for(Duration::from_secs(2), || {
health.patch_probe(&key, video, 1_000);
health.health(&target, &member, std::time::Instant::now()).probe.is_some()
})
.await
);
health.set_member_keys(std::collections::HashSet::new());
env.app_state
.event_manager
.send_event(shared::model::EventMessage::ConfigChange(shared::model::ConfigType::Sources));
assert!(
wait_for(Duration::from_secs(2), || {
health.patch_probe(&key, video, 1_000);
health.health(&target, &member, std::time::Instant::now()).probe.is_some()
})
.await,
"a sources change refreshes the member keys"
);
cancel.cancel();
assert!(tokio::time::timeout(Duration::from_secs(2), task).await.is_ok(), "the task stops on cancel");
}
@@ -0,0 +1,502 @@
use super::{
group_playback_support::{has_pid, member_ts, read_body_until, GroupPlaybackEnv},
support::FakeOriginMode,
};
use shared::model::{ChannelGroupFailoverConfig, EventMessage, SwitchReason};
use std::time::Duration;
const SOLIS_PID: u16 = 0x100;
const KORATH_PID: u16 = 0x200;
fn runtime_selection(no_data_timeout_secs: u64, max_outage_secs: u64) -> ChannelGroupFailoverConfig {
ChannelGroupFailoverConfig { no_data_timeout_secs, max_outage_secs, ..ChannelGroupFailoverConfig::default() }
}
async fn wait_until(timeout: Duration, condition: impl Fn() -> bool) -> bool {
let deadline = tokio::time::Instant::now() + timeout;
while !condition() {
if tokio::time::Instant::now() >= deadline {
return false;
}
tokio::time::sleep(Duration::from_millis(20)).await;
}
true
}
fn input_slots(env: &GroupPlaybackEnv, input: &str) -> usize {
env.app_state.active_provider.active_connections().unwrap_or_default().get(input).copied().unwrap_or_default()
}
#[tokio::test]
async fn switches_member_on_upstream_eof_without_client_reconnect() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"])
.with_qos("solis", 1, 95)
.origin_mode("solis", FakeOriginMode::FixedBytes(member_ts(SOLIS_PID, 40)))
.origin_mode("korath", FakeOriginMode::TsStream(member_ts(KORATH_PID, 40)))
.build()
.await;
let response = env.get_xtream_live(env.group_vid).await;
assert_eq!(response.status(), 200);
let (body, ended) = read_body_until(response, Duration::from_secs(5), |read| has_pid(read, KORATH_PID)).await;
assert!(!ended, "the client body stays open across the switch");
assert!(has_pid(&body, SOLIS_PID) && has_pid(&body, KORATH_PID));
assert_eq!((env.origin_hits("solis"), env.origin_hits("korath")), (1, 1));
}
#[tokio::test]
async fn switches_member_on_stall() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"])
.with_qos("solis", 1, 95)
.with_failover(runtime_selection(1, 20))
.origin_mode("solis", FakeOriginMode::TsThenStall(member_ts(SOLIS_PID, 40)))
.origin_mode("korath", FakeOriginMode::TsStream(member_ts(KORATH_PID, 40)))
.build()
.await;
let response = env.get_xtream_live(env.group_vid).await;
let (body, ended) = read_body_until(response, Duration::from_secs(5), |read| has_pid(read, KORATH_PID)).await;
assert!(!ended);
assert!(has_pid(&body, KORATH_PID));
}
#[tokio::test]
async fn provider_slot_released_on_switch() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"])
.with_qos("solis", 1, 95)
.origin_mode("solis", FakeOriginMode::FixedBytes(member_ts(SOLIS_PID, 40)))
.origin_mode("korath", FakeOriginMode::TsStream(member_ts(KORATH_PID, 40)))
.build()
.await;
let response = env.get_xtream_live(env.group_vid).await;
let (_body, ended) = read_body_until(response, Duration::from_secs(5), |read| has_pid(read, KORATH_PID)).await;
assert!(!ended);
assert!(
wait_until(Duration::from_secs(2), || input_slots(&env, "solis") == 0 && input_slots(&env, "korath") == 1)
.await
);
}
#[tokio::test]
async fn runtime_budget_exhausted_ends_with_channel_unavailable() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"])
.with_failover(runtime_selection(8, 2))
.origin_mode("solis", FakeOriginMode::FixedBytes(member_ts(SOLIS_PID, 10)))
.origin_mode("korath", FakeOriginMode::FixedBytes(member_ts(KORATH_PID, 10)))
.build()
.await;
let response = env.get_xtream_live(env.group_vid).await;
let (_body, ended) = read_body_until(response, Duration::from_secs(8), |_| false).await;
assert!(ended, "the body ends once the switch budget is spent");
assert!(wait_until(Duration::from_secs(2), || env.slots_in_use() == 0).await);
}
#[tokio::test]
async fn runtime_switch_does_not_use_grace() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"])
.with_qos("solis", 1, 95)
.with_failover(runtime_selection(8, 2))
.origin_mode("solis", FakeOriginMode::FixedBytes(member_ts(SOLIS_PID, 40)))
.origin_mode("korath", FakeOriginMode::TsStream(member_ts(KORATH_PID, 40)))
.build()
.await;
let _korath_full = env.hold("korath", 0);
let response = env.get_xtream_live(env.group_vid).await;
let (body, ended) = read_body_until(response, Duration::from_secs(6), |_| false).await;
assert!(ended);
assert!(!has_pid(&body, KORATH_PID));
assert_eq!(env.origin_hits("korath"), 0);
}
#[tokio::test]
async fn switch_emits_member_switched_event() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"])
.with_qos("solis", 1, 95)
.origin_mode("solis", FakeOriginMode::FixedBytes(member_ts(SOLIS_PID, 40)))
.origin_mode("korath", FakeOriginMode::TsStream(member_ts(KORATH_PID, 40)))
.build()
.await;
let mut events = env.app_state.event_manager.get_event_channel();
let response = env.get_xtream_live(env.group_vid).await;
let (_body, _) = read_body_until(response, Duration::from_secs(5), |read| has_pid(read, KORATH_PID)).await;
let switched = tokio::time::timeout(Duration::from_secs(2), async {
loop {
if let Ok(EventMessage::ChannelGroupMemberSwitched { from_member_vid, to_member_vid, reason, .. }) =
events.recv().await
{
return (from_member_vid, to_member_vid, reason);
}
}
})
.await
.expect("switch event");
assert_eq!(switched, (env.member_vid("solis"), env.member_vid("korath"), SwitchReason::UpstreamEof));
}
#[tokio::test]
async fn non_group_stream_eof_unchanged() {
let plain = member_ts(SOLIS_PID, 10);
let env = GroupPlaybackEnv::builder(&["solis", "korath"])
.origin_mode("solis", FakeOriginMode::FixedBytes(plain.clone()))
.build()
.await;
let response = env.get_xtream_live(env.plain_vid).await;
let (body, ended) = read_body_until(response, Duration::from_secs(5), |_| false).await;
assert!(ended);
assert_eq!(body, plain);
assert_eq!(env.origin_hits("korath"), 0);
}
#[tokio::test]
async fn switch_continues_at_pat_with_new_version() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"])
.with_qos("solis", 1, 95)
.origin_mode("solis", FakeOriginMode::FixedBytes(member_ts(SOLIS_PID, 40)))
.origin_mode("korath", FakeOriginMode::TsStream(member_ts(KORATH_PID, 40)))
.build()
.await;
let response = env.get_xtream_live(env.group_vid).await;
let (body, _) = read_body_until(response, Duration::from_secs(5), |read| has_pid(read, KORATH_PID)).await;
let packets: Vec<&[u8]> = body.as_chunks::<188>().0.iter().map(<[u8; 188]>::as_slice).collect();
let pid = |packet: &[u8]| (u16::from(packet[1] & 0x1F) << 8) | u16::from(packet[2]);
let last_solis = packets.iter().rposition(|packet| pid(packet) == SOLIS_PID).expect("solis packets");
let first_korath = packets.iter().position(|packet| pid(packet) == KORATH_PID).expect("korath packets");
let between: Vec<u16> = packets[last_solis + 1..first_korath].iter().map(|packet| pid(packet)).collect();
let splice = between.iter().position(|pid| *pid == 0).expect("PAT before the new member");
assert!(between[..splice].iter().all(|pid| *pid == 0x1FFF), "only keep-alive null packets before the splice");
assert_eq!(between[splice + 1], 0x1000, "PMT follows the PAT");
let pat = packets[last_solis + 1 + splice];
assert_eq!((pat[10] >> 1) & 0x1F, 1, "PAT version continues after the first member's version 0");
}
#[tokio::test]
async fn shared_group_stream_switches_member_and_moves_allocation() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"])
.share_ts()
.with_qos("solis", 1, 95)
.origin_mode("solis", FakeOriginMode::FixedBytes(member_ts(SOLIS_PID, 40)))
.origin_mode("korath", FakeOriginMode::TsStream(member_ts(KORATH_PID, 40)))
.build()
.await;
let response = env.get_xtream_live(env.group_vid).await;
assert_eq!(response.status(), 200);
let (body, ended) = read_body_until(response, Duration::from_secs(5), |read| has_pid(read, KORATH_PID)).await;
assert!(!ended, "the shared subscriber keeps its response across the switch");
assert!(has_pid(&body, KORATH_PID));
assert!(
wait_until(Duration::from_secs(2), || input_slots(&env, "solis") == 0 && input_slots(&env, "korath") == 1)
.await
);
}
#[tokio::test]
async fn group_stream_sleep_timer_ends_without_switch() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"])
.with_qos("solis", 1, 95)
// Only an end of the member body may trigger a switch; the paused clock would fire the
// stall and underrun checks.
.with_failover(ChannelGroupFailoverConfig {
no_data_timeout_secs: 0,
slow_source_pct: 0,
..ChannelGroupFailoverConfig::default()
})
.sleep_timer_mins(1)
.origin_mode("solis", FakeOriginMode::TsStream(member_ts(SOLIS_PID, 40)))
.origin_mode("korath", FakeOriginMode::TsStream(member_ts(KORATH_PID, 40)))
.build()
.await;
let response = env.get_xtream_live(env.group_vid).await;
tokio::time::pause();
let (_body, ended) = read_body_until(response, Duration::from_secs(120), |_| false).await;
assert!(ended, "the sleep timer ends the response");
assert_eq!(env.origin_hits("korath"), 0, "the sleep timer triggers no switch");
assert!(
matches!(env.circuit("solis"), tuliprox_session::channel_group::CircuitView::Closed),
"the sleep timer is no member failure"
);
}
#[tokio::test]
async fn start_skips_member_whose_open_hangs() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"])
.with_qos("solis", 1, 95)
.origin_mode("solis", FakeOriginMode::Silent)
.origin_mode("korath", FakeOriginMode::TsStream(member_ts(KORATH_PID, 40)))
.build()
.await;
let response = tokio::time::timeout(Duration::from_secs(6), env.get_xtream_live(env.group_vid))
.await
.expect("the hanging open is bounded");
let (body, _) = read_body_until(response, Duration::from_secs(3), |read| has_pid(read, KORATH_PID)).await;
assert!(has_pid(&body, KORATH_PID));
}
#[tokio::test]
async fn runtime_switch_to_silent_member_respects_budget() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"])
.with_qos("solis", 1, 95)
.with_failover(runtime_selection(8, 2))
.origin_mode("solis", FakeOriginMode::FixedBytes(member_ts(SOLIS_PID, 10)))
.origin_mode("korath", FakeOriginMode::Silent)
.build()
.await;
let response = env.get_xtream_live(env.group_vid).await;
let (_body, ended) = read_body_until(response, Duration::from_secs(6), |_| false).await;
assert!(ended, "the switch budget bounds a member that never answers");
assert!(wait_until(Duration::from_secs(2), || env.slots_in_use() == 0).await);
}
#[tokio::test]
async fn stall_after_spent_budget_ends_stream() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"])
.with_failover(runtime_selection(1, 2))
.origin_mode("solis", FakeOriginMode::TsThenStall(member_ts(SOLIS_PID, 10)))
.origin_mode("korath", FakeOriginMode::TsThenStall(member_ts(KORATH_PID, 10)))
.build()
.await;
let response = env.get_xtream_live(env.group_vid).await;
let (_body, ended) = read_body_until(response, Duration::from_secs(8), |_| false).await;
assert!(ended, "a stall after the spent budget ends the response");
assert!(wait_until(Duration::from_secs(2), || env.slots_in_use() == 0).await);
}
#[tokio::test]
async fn retry_same_member_on_single_connection_account() {
let selection = ChannelGroupFailoverConfig {
no_data_timeout_secs: 1,
stable_after_secs: 1,
same_source_retries: 1,
max_outage_secs: 10,
..ChannelGroupFailoverConfig::default()
};
let env = GroupPlaybackEnv::builder(&["solis", "korath"])
.with_qos("solis", 1, 95)
.with_failover(selection)
.max_connections("solis", 1)
.origin_mode("solis", FakeOriginMode::TsThenStall(member_ts(SOLIS_PID, 40)))
.build()
.await;
let _korath_full = env.hold("korath", 0);
let response = env.get_xtream_live(env.group_vid).await;
let (_body, ended) = read_body_until(response, Duration::from_secs(6), |_| env.origin_hits("solis") >= 2).await;
assert!(!ended, "the stream waits for its own released slot instead of ending");
assert_eq!(env.origin_hits("solis"), 2);
}
#[tokio::test]
async fn preempted_shared_group_member_does_not_fail_over() {
use super::super::channel_group_failover::{GroupFailoverContext, GroupFailoverDriver, GroupFailoverStream};
use futures::StreamExt;
let env = GroupPlaybackEnv::builder(&["solis", "korath"]).build().await;
let user = env.app_state.app_config.get_user_credentials("viewer").expect("viewer");
let ctx = GroupFailoverContext {
app_state: std::sync::Arc::clone(&env.app_state),
target: std::sync::Arc::clone(&env.target),
target_name: env.target.name.as_str().into(),
group: std::sync::Arc::clone(&env.group_record),
playback: shared::model::ChannelGroupsConfig::default().playback(None),
user: (*user).clone(),
fingerprint: super::support::create_test_fingerprint(std::net::SocketAddr::from(([127, 0, 0, 1], 12345))),
req_headers: axum::http::HeaderMap::new(),
session_token: "preempted-session".into(),
connection_kind: tuliprox_session::active_provider_manager::ConnectionKind::Normal,
connection_permission: shared::model::UserConnectionPermission::Allowed,
share_stream: true,
base_channel: shared::model::StreamChannel::default(),
shared_key: Some("group:1:preempted".into()),
};
let slot = env
.app_state
.active_provider
.acquire_connection(
&"solis".into(),
&std::net::SocketAddr::from(([127, 0, 0, 1], 1)),
0,
tuliprox_session::active_provider_manager::ConnectionKind::Normal,
)
.expect("solis slot");
let slot =
tuliprox_session::ManagedProviderHandle::new(std::sync::Arc::clone(&env.app_state.active_provider), slot);
// The provider manager cancels this token when it preempts the slot.
slot.handle().and_then(|handle| handle.cancel_token.as_ref()).expect("cancel token").cancel();
let ended_body = futures::stream::empty().boxed();
let mut stream =
GroupFailoverStream::new(GroupFailoverDriver::new(std::sync::Arc::new(ctx), 0), ended_body, Some(&slot));
let next = tokio::time::timeout(Duration::from_secs(2), stream.next()).await.expect("ends without a switch");
assert!(next.is_none());
assert_eq!((env.origin_hits("solis"), env.origin_hits("korath")), (0, 0));
}
async fn session_provider(env: &GroupPlaybackEnv) -> String {
env.app_state.active_users.bound_member_session_provider("viewer").await.map(|p| p.to_string()).unwrap_or_default()
}
#[tokio::test]
async fn group_session_provider_follows_selected_member() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"])
.with_qos("korath", 2, 95)
.origin_mode("korath", FakeOriginMode::TsStream(member_ts(KORATH_PID, 40)))
.build()
.await;
let response = env.get_xtream_live(env.group_vid).await;
let (_body, _) = read_body_until(response, Duration::from_secs(3), |read| has_pid(read, KORATH_PID)).await;
assert_eq!(session_provider(&env).await, "korath");
}
#[tokio::test]
async fn group_session_provider_follows_runtime_switch() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"])
.with_qos("solis", 1, 95)
.origin_mode("solis", FakeOriginMode::FixedBytes(member_ts(SOLIS_PID, 40)))
.origin_mode("korath", FakeOriginMode::TsStream(member_ts(KORATH_PID, 40)))
.build()
.await;
let response = env.get_xtream_live(env.group_vid).await;
let (_body, _) = read_body_until(response, Duration::from_secs(5), |read| has_pid(read, KORATH_PID)).await;
assert_eq!(session_provider(&env).await, "korath");
}
fn no_flap_guard(no_data_timeout_secs: u64, max_outage_secs: u64) -> ChannelGroupFailoverConfig {
ChannelGroupFailoverConfig { max_switches: 0, ..runtime_selection(no_data_timeout_secs, max_outage_secs) }
}
#[tokio::test]
async fn all_members_ending_quickly_are_bounded_by_budget_without_flap_guard() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"])
.with_failover(no_flap_guard(8, 2))
.origin_mode("solis", FakeOriginMode::FixedBytes(member_ts(SOLIS_PID, 10)))
.origin_mode("korath", FakeOriginMode::FixedBytes(member_ts(KORATH_PID, 10)))
.build()
.await;
let started = std::time::Instant::now();
let response = env.get_xtream_live(env.group_vid).await;
let (_body, ended) = read_body_until(response, Duration::from_secs(8), |_| false).await;
assert!(ended, "the switch budget alone ends the response");
assert!(started.elapsed() < Duration::from_secs(6), "ended after {:?}", started.elapsed());
}
#[tokio::test]
async fn all_members_stalling_are_bounded_by_budget_without_flap_guard() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"])
.with_failover(no_flap_guard(1, 3))
.origin_mode("solis", FakeOriginMode::TsThenStall(member_ts(SOLIS_PID, 10)))
.origin_mode("korath", FakeOriginMode::TsThenStall(member_ts(KORATH_PID, 10)))
.build()
.await;
let started = std::time::Instant::now();
let response = env.get_xtream_live(env.group_vid).await;
let (_body, ended) = read_body_until(response, Duration::from_secs(12), |_| false).await;
assert!(ended, "the switch budget alone ends the response");
assert!(started.elapsed() < Duration::from_secs(9), "ended after {:?}", started.elapsed());
}
#[tokio::test]
async fn switch_retries_failed_member_until_budget() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"])
.with_qos("solis", 1, 95)
.with_failover(no_flap_guard(8, 8))
.origin_mode("solis", FakeOriginMode::FixedBytes(member_ts(SOLIS_PID, 10)))
.origin_mode("korath", FakeOriginMode::Silent)
.build()
.await;
let response = env.get_xtream_live(env.group_vid).await;
let (_body, ended) = read_body_until(response, Duration::from_secs(14), |_| false).await;
assert!(ended);
assert!(env.origin_hits("korath") >= 2, "korath tried {} times within the budget", env.origin_hits("korath"));
}
#[tokio::test]
async fn give_up_still_records_member_failure() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"])
.with_qos("solis", 1, 95)
.with_failover(no_flap_guard(8, 0))
.origin_mode("solis", FakeOriginMode::FixedBytes(member_ts(SOLIS_PID, 10)))
.build()
.await;
let response = env.get_xtream_live(env.group_vid).await;
let (_body, ended) = read_body_until(response, Duration::from_secs(5), |_| false).await;
assert!(ended, "a spent budget ends the response at once");
assert!(
matches!(env.circuit("solis"), tuliprox_session::channel_group::CircuitView::Counting(1)),
"the failure that ended the stream counts for the member"
);
}
#[tokio::test]
async fn failed_switch_shows_channel_unavailable_video() {
const MEMBER_PID: u16 = 0x150;
let env = GroupPlaybackEnv::builder(&["solis", "korath"])
.with_qos("solis", 1, 95)
.channel_unavailable_video()
.with_failover(no_flap_guard(8, 2))
.origin_mode("solis", FakeOriginMode::FixedBytes(member_ts(MEMBER_PID, 10)))
.origin_mode("korath", FakeOriginMode::Status(503))
.build()
.await;
let response = env.get_xtream_live(env.group_vid).await;
// The fixture's video PID 0x100 shows the channel-unavailable clip.
let (body, _) = read_body_until(response, Duration::from_secs(8), |read| has_pid(read, 0x100)).await;
assert!(has_pid(&body, MEMBER_PID));
assert!(has_pid(&body, 0x100), "the failed switch ends with the channel-unavailable video");
}
#[tokio::test]
async fn shared_switch_keeps_both_subscribers_and_rebinds_their_provider() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"])
.share_ts()
.with_qos("solis", 1, 95)
.origin_mode("solis", FakeOriginMode::TsThenStall(member_ts(SOLIS_PID, 40)))
.origin_mode("korath", FakeOriginMode::TsStream(member_ts(KORATH_PID, 40)))
.with_failover(runtime_selection(1, 20))
.build()
.await;
let first = env.get_xtream_live_from(env.group_vid, 12345).await;
let second = env.get_xtream_live_from(env.group_vid, 12346).await;
assert_eq!((first.status().as_u16(), second.status().as_u16()), (200, 200));
let (one, two) = tokio::join!(
read_body_until(first, Duration::from_secs(6), |read| has_pid(read, KORATH_PID)),
read_body_until(second, Duration::from_secs(6), |read| has_pid(read, KORATH_PID))
);
assert!(has_pid(&one.0, KORATH_PID) && has_pid(&two.0, KORATH_PID), "both subscribers continue on korath");
assert_eq!(env.origin_hits("solis"), 1, "one shared producer");
let providers = env.app_state.active_users.stream_providers("viewer").await;
assert_eq!(providers.len(), 2);
assert!(providers.iter().all(|provider| provider.as_ref() == "korath"), "providers {providers:?}");
}
#[tokio::test]
async fn capacity_rejecting_member_is_opened_once_per_switch() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"])
.with_qos("solis", 1, 95)
.with_failover(no_flap_guard(8, 4))
.origin_mode("solis", FakeOriginMode::FixedBytes(member_ts(SOLIS_PID, 10)))
.origin_mode("korath", FakeOriginMode::Status(429))
.build()
.await;
let response = env.get_xtream_live(env.group_vid).await;
let (_body, ended) = read_body_until(response, Duration::from_secs(10), |_| false).await;
assert!(ended);
// One open repeats the HTTP request itself (request retry, up to 3 requests); further rounds
// of the switch must not open the member again.
assert!(
env.origin_hits("korath") <= 3,
"a provider that reports no capacity is not asked again in the same switch ({} requests)",
env.origin_hits("korath")
);
}
#[tokio::test]
async fn disabled_failover_ends_stream_with_its_member() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"])
.with_qos("solis", 1, 95)
.with_failover(ChannelGroupFailoverConfig { enabled: false, ..ChannelGroupFailoverConfig::default() })
.origin_mode("solis", FakeOriginMode::FixedBytes(member_ts(SOLIS_PID, 10)))
.origin_mode("korath", FakeOriginMode::TsStream(member_ts(KORATH_PID, 40)))
.build()
.await;
let response = env.get_xtream_live(env.group_vid).await;
let (body, ended) = read_body_until(response, Duration::from_secs(5), |_| false).await;
assert!(ended, "the response ends with its member");
assert!(has_pid(&body, SOLIS_PID));
assert!(!has_pid(&body, KORATH_PID), "no switch without failover");
assert_eq!(env.origin_hits("korath"), 0);
}
@@ -0,0 +1,329 @@
use super::{
super::channel_group::{
acquire_group_member, acquire_group_member_with_race, group_playback_settings, group_representative,
member_stream_channel, GroupSelectionFailure, GroupSelectionRequest,
},
support::{create_test_app_config, create_test_app_state_for_config},
};
use crate::api::model::AppState;
use arc_swap::ArcSwap;
use shared::{
model::{
ChannelGroupPolicy, ChannelGroupsConfig, InputType, PlaylistItemType, ReverseProxyConfigDto, StreamConfigDto,
VirtualId, XtreamCluster,
},
utils::Internable,
};
use std::{net::SocketAddr, sync::Arc, time::Duration};
use tuliprox_core::model::{Config, ConfigInput, ProviderHandle, ReverseProxyConfig, SourcesConfig};
use tuliprox_repository::{
channel_group_repository::{ChannelGroupRecord, GroupMember},
ChannelGroupOrigin,
};
use tuliprox_session::{
active_provider_manager::{CapacityTier, ConnectionKind},
channel_group::SelectionMode,
};
static CLIENT_ADDR: std::sync::LazyLock<SocketAddr> =
std::sync::LazyLock::new(|| "127.0.0.1:40000".parse().expect("valid client addr"));
fn addr(port: u16) -> SocketAddr { SocketAddr::from(([127, 0, 0, 1], port)) }
fn xtream_input(id: u16, name: &str, max_connections: u16) -> Arc<ConfigInput> {
Arc::new(ConfigInput {
id,
name: name.intern(),
input_type: InputType::Xtream,
url: format!("http://{name}.invalid"),
username: Some("user".into()),
password: Some("pass".into()),
enabled: true,
max_connections,
..ConfigInput::default()
})
}
fn group_member(input: &str, stream_id: &str, provider_id: u32, member_vid: u32) -> GroupMember {
GroupMember {
member_vid: VirtualId::new(member_vid),
input_name: input.into(),
provider_id,
input_stream_id: stream_id.into(),
url: format!("http://{input}.invalid/{stream_id}.ts").into(),
item_type: PlaylistItemType::Live,
cluster: XtreamCluster::Live,
title: format!("{input} stream").into(),
archive: None,
epg_channel_id: None,
additional_properties: None,
}
}
fn app_config_with_sources(sources: SourcesConfig) -> tuliprox_core::model::AppConfig {
app_config_with_sources_and_config(sources, Config::default())
}
fn app_config_with_sources_and_config(sources: SourcesConfig, mut config: Config) -> tuliprox_core::model::AppConfig {
let dto = ReverseProxyConfigDto {
stream: Some(StreamConfigDto { grace_period_millis: 2_000, ..StreamConfigDto::default() }),
..ReverseProxyConfigDto::default()
};
config.reverse_proxy = Some(ReverseProxyConfig::from(&dto));
let mut app_config = create_test_app_config();
app_config.config = Arc::new(ArcSwap::from_pointee(config));
app_config.sources = Arc::new(ArcSwap::from_pointee(sources));
app_config
}
struct SelectionEnv {
app_state: Arc<AppState>,
group: ChannelGroupRecord,
config: ChannelGroupsConfig,
target: Arc<str>,
}
impl SelectionEnv {
fn new(members: &[(&str, u16)]) -> Self {
let sources = SourcesConfig {
inputs: members
.iter()
.enumerate()
.map(|(i, (name, max))| xtream_input(u16::try_from(i + 1).expect("id"), name, *max))
.collect(),
..SourcesConfig::default()
};
let app_state = create_test_app_state_for_config(Arc::new(app_config_with_sources(sources)));
let group = ChannelGroupRecord {
group_vid: VirtualId::new(1),
key: "zdf".into(),
origin: ChannelGroupOrigin::Auto,
policy: ChannelGroupPolicy::Priority,
orphaned: Vec::new(),
members: members
.iter()
.enumerate()
.map(|(i, (name, _))| {
group_member(
name,
&(i + 1).to_string(),
u32::try_from(i + 1).expect("provider_id"),
100 + u32::try_from(i).expect("member_vid"),
)
})
.collect(),
};
Self { app_state, group, config: ChannelGroupsConfig::default(), target: "Main".into() }
}
fn hold(&self, input: &str, priority: i8) -> ProviderHandle {
self.app_state
.active_provider
.acquire_connection(&input.into(), &addr(200), priority, ConnectionKind::Normal)
.expect("hold slot")
}
fn request<'a>(&'a self, exclude: &'a [usize], allow_grace: bool) -> GroupSelectionRequest<'a> {
GroupSelectionRequest {
allow_provider_grace: allow_grace,
exclude,
..GroupSelectionRequest::new(
&self.target,
&self.group,
self.config.start,
&CLIENT_ADDR,
"viewer",
0,
ConnectionKind::Normal,
)
}
}
fn slots_in_use(&self) -> usize {
self.app_state.active_provider.active_connections().unwrap_or_default().values().sum()
}
}
#[tokio::test]
async fn free_member_is_preferred_over_better_member_needing_grace() {
let env = SelectionEnv::new(&[("korath", 1), ("solis", 1)]);
let _blocker = env.hold("korath", 0);
let acquired = acquire_group_member(&env.app_state, env.request(&[], true)).await.expect("acquired");
assert_eq!((acquired.index, acquired.tier), (1, CapacityTier::Free));
}
#[tokio::test]
async fn lost_race_on_free_reprobes_free_before_grace() {
let env = SelectionEnv::new(&[("solis", 1), ("korath", 1), ("veyra", 1)]);
let _veyra_full = env.hold("veyra", 0);
let acquired = acquire_group_member_with_race(&env.app_state, env.request(&[], true), |state| {
let handle = state.active_provider.acquire_connection(&"solis".into(), &addr(201), 0, ConnectionKind::Normal);
std::mem::forget(handle);
})
.await
.expect("acquired");
assert_eq!((acquired.index, acquired.tier), (1, CapacityTier::Free));
}
#[tokio::test]
async fn own_release_keeps_best_member_when_zapping() {
let env = SelectionEnv::new(&[("korath", 1), ("solis", 1)]);
let own = env.hold("korath", 0);
env.app_state.active_users.set_pending_release_for_test("viewer", &[own.allocation_id]).await;
let acquired = acquire_group_member(&env.app_state, env.request(&[], true)).await.expect("acquired");
assert_eq!((acquired.index, acquired.tier), (0, CapacityTier::OwnRelease));
}
#[tokio::test]
async fn preemption_only_on_best_ranked_when_nothing_free() {
let env = SelectionEnv::new(&[("korath", 1), ("solis", 1)]);
let _a = env.hold("korath", 10);
let _b = env.hold("solis", 10);
let mut req = env.request(&[], false);
req.priority = -5;
let acquired = acquire_group_member(&env.app_state, req).await.expect("acquired");
assert_eq!((acquired.index, acquired.tier), (0, CapacityTier::Preempt));
}
#[tokio::test]
async fn demoted_input_members_are_tried_last() {
let env = SelectionEnv::new(&[("korath", 1), ("solis", 1)]);
let demoted: [Arc<str>; 1] = ["korath".into()];
let mut req = env.request(&[], true);
req.demoted_inputs = &demoted;
let acquired = acquire_group_member(&env.app_state, req).await.expect("acquired");
assert_eq!((acquired.index, acquired.tier), (1, CapacityTier::Free));
}
#[tokio::test]
async fn grace_tier_skipped_when_disabled() {
let env = SelectionEnv::new(&[("korath", 1)]);
let _full = env.hold("korath", 0);
let result = acquire_group_member(&env.app_state, env.request(&[], false)).await;
assert!(matches!(result, Err(GroupSelectionFailure::Exhausted)));
}
#[tokio::test]
async fn all_members_exhausted_returns_exhausted_within_budget() {
let mut env = SelectionEnv::new(&[("korath", 1), ("solis", 1)]);
env.config.start.timeout_secs = 1;
let _a = env.hold("korath", 10);
let _a_grace = env
.app_state
.active_provider
.try_acquire_connection_no_preempt(&"korath".into(), true, &addr(201), 10, ConnectionKind::Normal, None)
.expect("grace");
let _b = env.hold("solis", 10);
let _b_grace = env
.app_state
.active_provider
.try_acquire_connection_no_preempt(&"solis".into(), true, &addr(202), 10, ConnectionKind::Normal, None)
.expect("grace");
let before = env.slots_in_use();
let started = tokio::time::Instant::now();
let mut req = env.request(&[], true);
req.priority = 10;
let result = acquire_group_member(&env.app_state, req).await;
assert!(matches!(result, Err(GroupSelectionFailure::Exhausted)));
assert!(started.elapsed() < Duration::from_secs(2));
assert_eq!(env.slots_in_use(), before);
}
#[tokio::test]
async fn representative_skips_member_without_configured_input() {
let mut env = SelectionEnv::new(&[("korath", 1)]);
env.group.members.insert(0, group_member("orun", "9", 9, 99));
let (input, url) = group_representative(&env.app_state, &env.group).expect("representative");
assert_eq!(input.name.as_ref(), "korath");
assert_eq!(url.as_ref(), env.group.members[1].url.as_ref());
}
#[tokio::test]
async fn representative_of_group_without_resolvable_member_is_none() {
let mut env = SelectionEnv::new(&[("korath", 1)]);
env.group.members.clear();
assert!(group_representative(&env.app_state, &env.group).is_none());
}
#[test]
fn member_stream_channel_carries_member_identity() {
let env_group = ChannelGroupRecord {
group_vid: VirtualId::new(7),
key: "zdf".into(),
origin: ChannelGroupOrigin::Auto,
policy: ChannelGroupPolicy::Priority,
orphaned: Vec::new(),
members: vec![group_member("korath", "42", 42, 300)],
};
let base = shared::model::StreamChannel {
target_id: 3,
virtual_id: 7,
input_name: "solis".intern(),
..shared::model::StreamChannel::default()
};
let channel = member_stream_channel(&base, &env_group, &env_group.members[0]);
assert_eq!((channel.target_id, channel.virtual_id, channel.group_virtual_id), (3, 300, Some(7)));
assert_eq!((channel.input_name.as_ref(), channel.provider_id), ("korath", 42));
assert_eq!(channel.url.as_ref(), env_group.members[0].url.as_ref());
}
#[tokio::test]
async fn group_playback_settings_apply_target_overrides_to_global_config() {
let global = ChannelGroupsConfig {
start: shared::model::ChannelGroupStartConfig { timeout_secs: 7, max_attempts: 2 },
..ChannelGroupsConfig::default()
};
let config = Config { channel_groups: global.clone(), ..Config::default() };
let app_state = create_test_app_state_for_config(Arc::new(app_config_with_sources_and_config(
SourcesConfig::default(),
config,
)));
let plain = tuliprox_core::model::ConfigTarget::from(&shared::model::ConfigTargetDto::default());
assert_eq!(group_playback_settings(&app_state, &plain), global.playback(None));
let dto = shared::model::ConfigTargetDto {
options: Some(shared::model::ConfigTargetOptions {
channel_groups: Some(shared::model::ChannelGroupsTargetConfig {
enabled: true,
allow_provider_grace: Some(false),
failover: Some(false),
..Default::default()
}),
..Default::default()
}),
..Default::default()
};
let target = tuliprox_core::model::ConfigTarget::from(&dto);
let playback = group_playback_settings(&app_state, &target);
assert_eq!(playback.start, global.start);
assert!(!playback.allow_provider_grace);
assert!(!playback.failover.enabled);
}
#[tokio::test]
async fn own_release_tier_respects_disabled_grace() {
let env = SelectionEnv::new(&[("korath", 1), ("solis", 1)]);
let own = env.hold("korath", 0);
env.app_state.active_users.set_pending_release_for_test("viewer", &[own.allocation_id]).await;
let acquired = acquire_group_member(&env.app_state, env.request(&[], false)).await.expect("acquired");
assert_eq!((acquired.index, acquired.tier), (1, CapacityTier::Free));
}
#[tokio::test]
async fn selection_request_defaults_match_start_path() {
let env = SelectionEnv::new(&[("korath", 1)]);
let req = GroupSelectionRequest::new(
&env.target,
&env.group,
env.config.start,
&CLIENT_ADDR,
"viewer",
3,
ConnectionKind::Normal,
);
assert_eq!(req.policy, env.group.policy);
assert_eq!(req.mode, SelectionMode::Live);
assert!(req.lease.is_none() && req.exclude.is_empty() && req.demoted_inputs.is_empty() && req.prefer.is_none());
assert!(req.allow_provider_grace && req.allow_preempt && !req.bypass_maintenance);
assert_eq!((req.priority, req.username), (3, "viewer"));
}
@@ -0,0 +1,70 @@
use super::group_playback_support::GroupPlaybackEnv;
use axum::http::StatusCode;
use futures::StreamExt;
#[tokio::test]
async fn range_request_uses_bound_member_input_not_representative() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"]).with_qos("korath", 2, 95).build().await;
let initial = env.get_xtream_live(env.group_vid).await;
assert_eq!(initial.status(), StatusCode::OK);
assert_eq!(env.bound_member_input("viewer").await, "korath");
assert_eq!(env.origin_hits("korath"), 1);
assert_eq!(env.origin_hits("solis"), 0);
let seek = env.get_xtream_live_with_range(env.group_vid, "bytes=1000-").await;
assert!(seek.status().is_success());
assert_eq!(env.bound_member_input("viewer").await, "korath");
assert_eq!(env.origin_hits("korath"), 2);
assert_eq!(env.origin_hits("solis"), 0);
}
#[tokio::test]
async fn ts_reconnect_stays_on_member_unless_circuit_open() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"]).with_qos("korath", 2, 95).build().await;
let first = env.get_xtream_live(env.group_vid).await;
assert_eq!(first.status(), StatusCode::OK);
drop(first);
// reconnect within session TTL stays on korath
let second = env.get_xtream_live(env.group_vid).await;
assert_eq!(second.status(), StatusCode::OK);
drop(second);
assert_eq!((env.origin_hits("korath"), env.origin_hits("solis")), (2, 0));
assert_eq!(env.bound_member_input("viewer").await, "korath");
// Open circuit on korath -> sticky binding is broken by circuit breaker, fails over to solis
env.open_circuit("korath");
let third = env.get_xtream_live(env.group_vid).await;
assert_eq!(third.status(), StatusCode::OK);
drop(third);
assert_eq!(env.origin_hits("korath"), 2);
assert_eq!(env.origin_hits("solis"), 1);
assert_eq!(env.bound_member_input("viewer").await, "solis");
}
#[tokio::test]
async fn binding_survives_group_removal_until_stream_end() {
let env = GroupPlaybackEnv::builder(&["solis", "korath"]).build().await;
let response = env.get_xtream_live(env.group_vid).await;
assert_eq!(response.status(), StatusCode::OK);
assert_eq!(env.bound_member_input("viewer").await, "solis");
assert!(env.active_session_vids("viewer").await.contains(&env.group_vid));
let mut body = response.into_body().into_data_stream();
// Republish target storage with groups removed while stream is actively running
env.republish_without_groups().await;
// Running stream body continues delivering bytes despite group removal from storage
let chunk = body.next().await.expect("stream continues").expect("bytes");
assert!(!chunk.is_empty());
assert_eq!(env.bound_member_input("viewer").await, "solis");
// New request for the removed group VID sees new state and does not panic (returns 502 Bad Gateway / channel unavailable)
let after = env.get_xtream_live(env.group_vid).await;
assert_ne!(after.status(), StatusCode::INTERNAL_SERVER_ERROR);
assert_eq!(after.status(), StatusCode::BAD_GATEWAY);
// After body finishes/is dropped, active session lease cleans up
drop(body);
}
@@ -5,7 +5,7 @@ use super::{
create_test_provider_app_state, create_test_shared_target, force_provider_stream_response, get_stream_options,
load_test_channel, load_test_session, load_test_user, session_reacquire_cleanup_addrs,
spawn_controlled_fake_origin, spawn_range_aware_test_origin, stream_response, FakeOriginMode,
ForceStreamRequestContext, PlaybackRequestClass, SessionActivationRequest, StreamResponseMode,
ForceStreamRequestContext, PlaybackRequestClass, PlaybackSource, SessionActivationRequest, StreamResponseMode,
};
use crate::{
api::model::{create_channel_unavailable_stream, UserSession},
@@ -1025,10 +1025,9 @@ async fn shared_proxy_clients_keep_independent_bodies_and_cleanup() {
"shared-session-1",
None,
channel1,
&stream_url,
PlaybackSource::Single { input: &input, stream_url: &stream_url },
None,
&HeaderMap::default(),
&input,
&target,
&user1,
UserConnectionPermission::Allowed,
@@ -1046,10 +1045,9 @@ async fn shared_proxy_clients_keep_independent_bodies_and_cleanup() {
"shared-session-2",
None,
channel2,
&stream_url,
PlaybackSource::Single { input: &input, stream_url: &stream_url },
None,
&HeaderMap::default(),
&input,
&target,
&user2,
UserConnectionPermission::Allowed,
@@ -1136,10 +1134,9 @@ async fn shared_start_abort_does_not_leak_meter_registration() {
"shared-session-abort",
None,
channel,
&stream_url,
PlaybackSource::Single { input: &input, stream_url: &stream_url },
None,
&HeaderMap::default(),
&input,
&target,
&user,
UserConnectionPermission::Allowed,
File diff suppressed because it is too large Load Diff
@@ -1,6 +1,6 @@
use super::{
create_test_app_state_for_config, create_test_fingerprint, create_test_live_channel,
create_test_provider_app_config, create_test_provider_app_state, stream_response,
create_test_provider_app_config, create_test_provider_app_state, stream_response, PlaybackSource,
};
use crate::{
api::{api_utils::blocks_new_playback, model::AppState},
@@ -102,7 +102,7 @@ fn gate_runs_before_shared_stream_join() {
#[test]
fn redirect_defers_maintenance_inputs_to_the_gated_proxy_path() {
// Redirect mode would send the client straight to the provider; for an input in
// maintenance it must fall through to the proxied path, where gate 1 answers.
// maintenance it must fall through to the proxied path, where the maintenance check answers.
let body = function_body(include_str!("../strategy.rs"), "pub async fn redirect_response<");
let gate = body.find("if blocks_new_playback(").expect("gate present");
let redirect = body.find("Some(redirect(").expect("redirect response present");
@@ -172,10 +172,9 @@ async fn new_ts_stream_for_input_in_maintenance_gets_the_clip_without_provider_s
"maintenance-session",
None,
create_test_live_channel(stream_url),
stream_url,
PlaybackSource::Single { input: &input, stream_url },
None,
&HeaderMap::default(),
&input,
&maintenance_clip_target(),
&viewer(),
shared::model::UserConnectionPermission::Allowed,
@@ -872,6 +872,7 @@ async fn xtream_vod_m3u_reverse_proxies_external_sources_and_alias_redirects_wit
Some("mp4"),
None,
None,
None,
)
.await
.into_response();
@@ -4,7 +4,7 @@ use super::{
is_media_server_playback_url, is_media_server_stream_ref_url, is_seek_request, is_seekable_media_request,
mark_response_as_uncompressed, resolve_request_url_for_logging, resolve_stream_config_u64, resource_proxy_response,
spawn_legacy_hls_test_origin, stream_response, stream_responses::forced_legacy_hls_test_response,
user_with_network_access, NetworkAccessDecision,
user_with_network_access, NetworkAccessDecision, PlaybackSource,
};
use crate::{
api::model::{SharedStreamCtx, SharedStreamManager},
@@ -365,10 +365,9 @@ async fn stream_response_preserves_soft_kind_for_shared_reuse() {
"soft-session",
None,
create_test_live_channel(stream_url),
stream_url,
PlaybackSource::Single { input: &input, stream_url },
None,
&HeaderMap::default(),
&input,
&target,
&user,
admission.permission(),
@@ -2,7 +2,7 @@ use super::{
activate_session_before_stream_open, create_test_app_config, create_test_app_state,
create_test_app_state_for_config, create_test_app_state_with_stream_config, create_test_fingerprint,
create_test_live_channel, create_test_provider_app_config, resolve_admission_with_strategies, stream_response,
AdmissionRequest, EvictionReentryGuard, PlaybackRequestClass, SessionActivationRequest,
AdmissionRequest, EvictionReentryGuard, PlaybackRequestClass, PlaybackSource, SessionActivationRequest,
};
use crate::model::{Config, ConfigTarget, ProxyUserCredentials};
use arc_swap::{ArcSwap, ArcSwapOption};
@@ -573,10 +573,9 @@ async fn stream_response_rolls_back_provisional_user_activation_when_provider_op
"rollback-session",
None,
channel,
stream_url,
PlaybackSource::Single { input: &input, stream_url },
None,
&HeaderMap::default(),
&input,
&target,
&user,
UserConnectionPermission::Allowed,
+55 -1
View File
@@ -104,14 +104,27 @@ pub(in crate::api::api_utils::tests) async fn spawn_range_aware_test_origin(
#[allow(dead_code)]
#[derive(Clone)]
pub(in crate::api::api_utils::tests) enum FakeOriginMode {
pub(crate) enum FakeOriginMode {
EmptyBody,
BlockFirstChunk(Arc<tokio::sync::Notify>),
ErrorBeforeFirstChunk,
ErrorAfterFirstChunk(Vec<u8>),
FixedBytes(Vec<u8>),
Status(u16),
Body {
content_type: String,
bytes: Vec<u8>,
},
Hls,
/// MPEG-TS bytes, then an open connection without data.
TsThenStall(Vec<u8>),
/// The bytes repeated every 50 ms until the client disconnects.
TsStream(Vec<u8>),
/// Accepts the request and never answers.
Silent,
}
#[allow(clippy::too_many_lines)]
pub(in crate::api::api_utils::tests) async fn spawn_controlled_fake_origin(
mode: FakeOriginMode,
max_requests: usize,
@@ -178,6 +191,44 @@ pub(in crate::api::api_utils::tests) async fn spawn_controlled_fake_origin(
let _ = socket.write_all(bytes).await;
let _ = socket.shutdown().await;
}
FakeOriginMode::TsThenStall(bytes) => {
let head = "HTTP/1.1 200 OK\r\nContent-Type: video/mp2t\r\nConnection: close\r\n\r\n";
let _ = socket.write_all(head.as_bytes()).await;
let _ = socket.write_all(bytes).await;
// Keeps the connection open without data until the client goes away.
let mut probe = [0_u8; 1];
let _ = socket.read(&mut probe).await;
}
FakeOriginMode::TsStream(bytes) => {
let head = "HTTP/1.1 200 OK\r\nContent-Type: video/mp2t\r\nConnection: close\r\n\r\n";
let _ = socket.write_all(head.as_bytes()).await;
while socket.write_all(bytes).await.is_ok() {
tokio::time::sleep(std::time::Duration::from_millis(50)).await;
}
}
FakeOriginMode::Silent => {
let mut probe = [0_u8; 1];
let _ = socket.read(&mut probe).await;
}
FakeOriginMode::Status(status) => {
let head = format!("HTTP/1.1 {status}\r\nContent-Length: 0\r\nConnection: close\r\n\r\n");
let _ = socket.write_all(head.as_bytes()).await;
let _ = socket.shutdown().await;
}
FakeOriginMode::Body { content_type, bytes } => {
let head = format!(
"HTTP/1.1 200 OK\r\nContent-Type: {content_type}\r\nContent-Length: {}\r\nConnection: close\r\n\r\n",
bytes.len()
);
let _ = socket.write_all(head.as_bytes()).await;
let _ = socket.write_all(bytes).await;
let _ = socket.shutdown().await;
}
FakeOriginMode::Hls => {
let head = "HTTP/1.1 200 OK\r\nContent-Type: application/vnd.apple.mpegurl\r\nContent-Length: 0\r\nConnection: close\r\n\r\n";
let _ = socket.write_all(head.as_bytes()).await;
let _ = socket.shutdown().await;
}
}
}
requests
@@ -470,6 +521,7 @@ pub(in crate::api::api_utils::tests) fn create_test_app_state_for_config(app_cfg
auth: crate::api::model::AuthState::for_tests(),
playlist_updates: crate::api::model::PlaylistUpdateControl::for_tests(),
maintenance: crate::api::model::create_test_maintenance(""),
member_health: Arc::new(tuliprox_session::MemberHealthIndex::default()),
})
}
@@ -507,6 +559,7 @@ pub(in crate::api::api_utils::tests) fn create_test_local_channel(url: &str) ->
StreamChannel {
target_id: 1,
virtual_id: 41,
group_virtual_id: None,
provider_id: 0,
input_name: "library".intern(),
item_type: PlaylistItemType::LocalVideo,
@@ -528,6 +581,7 @@ pub(in crate::api::api_utils::tests) fn create_test_live_channel(url: &str) -> S
StreamChannel {
target_id: 1,
virtual_id: 42,
group_virtual_id: None,
provider_id: 1,
input_name: "provider_1".intern(),
item_type: PlaylistItemType::Live,
@@ -711,6 +711,7 @@ mod tests {
qos_aggregation: CancellationToken::new(),
recordings: CancellationToken::new(),
hls_cache: CancellationToken::new(),
channel_groups: tokio_util::sync::CancellationToken::new(),
};
let metadata_manager = Arc::new(MetadataUpdateManager::new(tokens.metadata.clone()));
@@ -738,6 +739,7 @@ mod tests {
auth: crate::api::model::AuthState::for_tests(),
playlist_updates: crate::api::model::PlaylistUpdateControl::for_tests(),
maintenance: crate::api::model::create_test_maintenance(""),
member_health: Arc::new(tuliprox_session::MemberHealthIndex::default()),
})
}
@@ -155,10 +155,16 @@ pub(in crate::api::endpoints::hls_api) fn spawn_hls_runtime_bandwidth_persistenc
let app_config = Arc::clone(&app_state.app_config);
let hls_proxy = Arc::clone(&app_state.hls.proxy);
let session = Arc::clone(session);
let member_health = Arc::clone(&app_state.member_health);
Some(tokio::spawn(async move {
let outcome = match persist_input_live_bitrate_bps(&app_config, &input, &stream_ref, bitrate_bps).await {
Ok(repository_outcome) => hls_bandwidth_persistence_outcome(repository_outcome, &proxy_session_id),
Ok(repository_outcome) => {
let kbps = bitrate_bps / 1000;
let key = (Arc::clone(&input.name), Arc::from(stream_ref.as_ref()));
member_health.patch_probe(&key, None, kbps);
hls_bandwidth_persistence_outcome(repository_outcome, &proxy_session_id)
}
Err(err) => {
error!(
"HLS runtime bandwidth persistence failed: proxy_session={} error={}",
@@ -113,7 +113,7 @@ pub(in crate::api::endpoints::hls_api) async fn try_reserve_hls_entry_origin_acc
})
}
#[allow(clippy::too_many_arguments)]
#[allow(clippy::too_many_arguments, clippy::too_many_lines)]
pub(in crate::api::endpoints::hls_api) async fn try_reserve_hls_virtual_entry_origin_account_for_redirect(
app_state: &Arc<AppState>,
fingerprint: &Fingerprint,
@@ -168,6 +168,40 @@ pub(in crate::api::endpoints::hls_api) async fn try_reserve_hls_virtual_entry_or
};
let session_owner = shared_hls_session_owner.as_deref().unwrap_or(session_token.as_str());
let classified = tuliprox_repository::target_stream_item::classify_virtual_id(
&app_state.app_config,
&app_state.playlists,
target,
VirtualId::new(virtual_id),
)
.await;
if let tuliprox_repository::target_stream_item::ResolvedStreamTarget::Group(group) = classified {
let playback = crate::api::api_utils::group_playback_settings(app_state, target);
let target_name: Arc<str> = Arc::from(target.name.as_str());
let req = crate::api::api_utils::GroupSelectionRequest {
lease: Some(PlaybackLeaseRef::new(session_owner, PlaybackKind::LiveHls)),
allow_provider_grace: playback.allow_provider_grace,
bypass_maintenance: user.is_recording_proxy_user(),
..crate::api::api_utils::GroupSelectionRequest::new(
&target_name,
&group,
playback.start,
&fingerprint.addr,
&user.username,
connection_priority_for_kind(user, connection_kind),
connection_kind,
)
};
let Ok(resolved) = crate::api::api_utils::resolve_hls_group_entry(app_state, Arc::clone(&group), req).await
else {
return false;
};
let mut acquired_handle = resolved.acquired.handle;
let provider_handle = acquired_handle.take();
app_state.connection_manager.release_provider_handle(provider_handle);
return true;
}
let Some(reservation) = try_reserve_hls_entry_origin_account_for_redirect(
app_state,
fingerprint,
@@ -185,9 +185,33 @@ async fn hls_recreate_content_allowed(
return Some(user_allows_entry_content(user, scope, &item));
}
}
let item =
m3u_get_item_for_stream_id(virtual_id, &app_state.app_config, &app_state.playlists, target).await.ok()?;
Some(user_allows_entry_content(user, HlsEntryOutputScope::ItemType(item.item_type), &item))
if let Ok(item) = m3u_get_item_for_stream_id(virtual_id, &app_state.app_config, &app_state.playlists, target).await
{
return Some(user_allows_entry_content(user, HlsEntryOutputScope::ItemType(item.item_type), &item));
}
let classified = tuliprox_repository::target_stream_item::classify_virtual_id(
&app_state.app_config,
&app_state.playlists,
target,
shared::model::VirtualId::new(virtual_id),
)
.await;
if let tuliprox_repository::target_stream_item::ResolvedStreamTarget::Member { group, index } = classified {
if target.has_output(TargetType::Xtream) {
let item = tuliprox_repository::target_stream_item::member_to_xtream_item(&group, index, target);
let scope = if is_catchup {
HlsEntryOutputScope::LiveCluster
} else {
HlsEntryOutputScope::ItemType(item.item_type)
};
return Some(user_allows_entry_content(user, scope, &item));
}
let item = tuliprox_repository::target_stream_item::member_to_m3u_item(&group, index, target);
let scope =
if is_catchup { HlsEntryOutputScope::LiveCluster } else { HlsEntryOutputScope::ItemType(item.item_type) };
return Some(user_allows_entry_content(user, scope, &item));
}
None
}
/// Recreates a missing user session from a sealed `Manifest(Entry)` token.
@@ -195,7 +219,7 @@ async fn hls_recreate_content_allowed(
/// With the wrapper the player refreshes only the variant URI, so after a session expiry the
/// token route has to do what an entry refresh did before: identity, access, admission, source,
/// session, fetch. The session keeps the sealed session token and therefore its playback owner.
#[allow(clippy::too_many_arguments)]
#[allow(clippy::too_many_arguments, clippy::too_many_lines)]
pub(super) async fn recreate_hls_session_from_token(
fingerprint: &Fingerprint,
req_headers: &HeaderMap,
@@ -211,9 +235,21 @@ pub(super) async fn recreate_hls_session_from_token(
else {
return StatusCode::BAD_REQUEST.into_response();
};
if hls_cache_enabled_for_user(app_state, target, user)
|| !hls_session_hint_matches_requester(fingerprint, &user.username, virtual_id, hint)
{
let classified = tuliprox_repository::target_stream_item::classify_virtual_id(
&app_state.app_config,
&app_state.playlists,
target,
shared::model::VirtualId::new(virtual_id),
)
.await;
let hint_matches = hls_session_hint_matches_requester(fingerprint, &user.username, virtual_id, hint)
|| match &classified {
tuliprox_repository::target_stream_item::ResolvedStreamTarget::Member { group, .. } => {
hls_session_hint_matches_requester(fingerprint, &user.username, group.group_vid.get(), hint)
}
_ => false,
};
if hls_cache_enabled_for_user(app_state, target, user) || !hint_matches {
return StatusCode::BAD_REQUEST.into_response();
}
// Only an expired session comes back; an evicted, kicked or terminated one stays ended,
@@ -5,15 +5,16 @@ use super::{
hls_custom_video_manifest_response, hls_entry_user_session_token, hls_media_playlist_wrap_enabled,
hls_panel_provisioning_or_status_response, hls_playback_kind, hls_response, m3u_archive_epg_reference_ts,
m3u_catchup_epg_reference_from_session_token, normalize_xtream_live_hls_url, release_prepared_hls_manifest_session,
terminate_failed_hls_manifest_session, try_reserve_hls_entry_origin_account_for_redirect, wrap_media_playlist,
HlsEntryStreamContext, HlsMediaPlaylistWrap, HlsOriginEntryUrl, HlsRequestStage,
resolve_hls_origin_playlist_url, terminate_failed_hls_manifest_session,
try_reserve_hls_entry_origin_account_for_redirect, wrap_media_playlist, HlsEntryStreamContext,
HlsMediaPlaylistWrap, HlsOriginEntryUrl, HlsRequestStage,
};
use crate::{
api::{
api_utils::{
acquire_exact_provider_handle, blocks_new_playback, connection_priority_for_kind,
get_hls_playback_ttl_secs, is_running_playback, select_provider_stream_url, ExactProviderAcquire,
HLS_MANIFEST_CAPACITY_WAIT,
get_hls_playback_ttl_secs, group_playback_settings, is_running_playback, resolve_hls_group_entry,
select_provider_stream_url, ExactProviderAcquire, GroupSelectionRequest, HLS_MANIFEST_CAPACITY_WAIT,
},
model::{AppState, CustomVideoStreamType, PlaybackLeaseRef, ProviderAllocation, UserSession},
},
@@ -40,6 +41,10 @@ use tuliprox_hls::api::{
build_hls_origin_session_owner, build_proxy_session_id, extract_hls_provider_session_headers, HlsOriginSource,
HlsSessionKey,
};
use tuliprox_session::{
channel_group::{FailureClass, HalfOpenGuard},
MemberBinding,
};
pub(in crate::api::endpoints::hls_api) struct HlsCacheManifestOrigin<'a> {
pub(in crate::api::endpoints::hls_api) raw_request_url: &'a str,
@@ -88,9 +93,26 @@ pub(in crate::api) async fn handle_hls_stream_request(
if app_state.active_users.is_user_blocked_for_stream(&user.username, VirtualId::new(virtual_id)).await {
return axum::http::StatusCode::BAD_REQUEST.into_response();
}
let entry_group = if matches!(stage, HlsRequestStage::Entry) {
match tuliprox_repository::target_stream_item::classify_virtual_id(
&app_state.app_config,
&app_state.playlists,
target,
VirtualId::new(virtual_id),
)
.await
{
tuliprox_repository::target_stream_item::ResolvedStreamTarget::Group(group) => Some(group),
_ => None,
}
} else {
None
};
// Only a fresh entry is gated. Variant refreshes, segments and entry re-polls of a running
// session (unwrapped media playlists) belong to a running playback.
// session (unwrapped media playlists) belong to a running playback. A group checks
// maintenance per member during selection.
if matches!(stage, HlsRequestStage::Entry)
&& entry_group.is_none()
&& !is_running_playback(user_session)
&& blocks_new_playback(app_state, input, user, false)
{
@@ -120,7 +142,7 @@ pub(in crate::api) async fn handle_hls_stream_request(
};
// The wrapper seals the normalized entry URL, not the provider-resolved one, so every
// refresh can resolve it to whichever account is selected then.
let entry_url = url.clone();
let mut entry_url = url.clone();
// Recover archive context when callers (esp. Xtream timeshift) pass None but the
// resolved provider URL / catchup session still carries Flussonic archive markers.
let archive_reference = archive_reference.or_else(|| m3u_archive_epg_reference_ts(&url)).or_else(|| {
@@ -149,6 +171,79 @@ pub(in crate::api) async fn handle_hls_stream_request(
);
if hls_cache_enabled_for_user(app_state, target, user) {
if let Some(group) = entry_group {
// The shared origin belongs to the selected member, so viewers of the same member share
// it. The cache path reserves its own provider account; the selection only picks.
let playback = group_playback_settings(app_state, target);
let target_name: Arc<str> = Arc::from(target.name.as_str());
let kind = connection_kind.unwrap_or(crate::api::model::ConnectionKind::Normal);
let req = GroupSelectionRequest {
allow_provider_grace: playback.allow_provider_grace,
bypass_maintenance: user.is_recording_proxy_user(),
allow_preempt: false,
..GroupSelectionRequest::new(
&target_name,
&group,
playback.start,
&fingerprint.addr,
&user.username,
connection_priority_for_kind(user, kind),
kind,
)
};
let Ok(resolved) = resolve_hls_group_entry(app_state, Arc::clone(&group), req).await else {
return hls_panel_provisioning_or_status_response(
app_state,
user,
input,
virtual_id,
original_hls_entry_path,
server_info.path.as_deref(),
StatusCode::SERVICE_UNAVAILABLE,
)
.await;
};
let mut selection_handle = resolved.acquired.handle;
app_state.connection_manager.release_provider_handle(selection_handle.take());
let member_url = resolve_hls_origin_playlist_url(
app_state,
target,
&resolved.input,
resolved.member.member_vid.get(),
resolved.stream_url.as_ref(),
)
.await
.unwrap_or_else(|_| resolved.stream_url.to_string());
let member_url =
ensure_hls_manifest_extension(&normalize_xtream_live_hls_url(&member_url, &resolved.input));
let member_origin = build_hls_origin_resolution(&resolved.input, &member_url);
let member_source = build_hls_origin_source_for_playback(
&resolved.input,
resolved.member.input_stream_id.to_string(),
archive_reference,
Some(member_url.as_str()),
);
return create_hls_cache_entry_master_playlist_response(
app_state,
fingerprint,
user,
member_source,
virtual_id,
user_session,
stream_context.known_bitrate_bps(),
session_token_hint,
if archive_reference.is_some() {
member_url.as_str()
} else {
member_origin.as_ref().map_or(member_url.as_str(), |origin| origin.session_entry_url.as_str())
},
&resolved.input,
connection_permission,
connection_kind,
server_info.path.as_deref(),
)
.await;
}
let Some(origin_source) = hls_origin_source.clone() else {
return axum::http::StatusCode::SERVICE_UNAVAILABLE.into_response();
};
@@ -175,8 +270,21 @@ pub(in crate::api) async fn handle_hls_stream_request(
}
let fallback_connection_kind = connection_kind.unwrap_or(crate::api::model::ConnectionKind::Normal);
let mut bound_member_info: Option<(u32, Arc<ConfigInput>, String, Option<HalfOpenGuard>)> = None;
let (request_url, session_token, provider_handle, selected_provider_config) = if let Some(session) = user_session {
let pinned_provider = if session.provider.is_empty() { &input.name } else { &session.provider };
let bound_inp_opt = session.member_binding.as_ref().and_then(|binding| {
app_state.app_config.get_input_by_name(&binding.input_name).map(|inp| (binding.member_vid, inp))
});
if let Some((member_vid, ref bound_inp)) = bound_inp_opt {
bound_member_info = Some((member_vid, Arc::clone(bound_inp), String::new(), None));
}
let effective_lookup_input = bound_inp_opt.as_ref().map_or(input, |(_, inp)| inp.as_ref());
let pinned_provider = if session.provider.is_empty() {
effective_lookup_input.name.clone()
} else {
Arc::clone(&session.provider)
};
let pinned_kind = hls_playback_kind(archive_reference, &url, Some(session.token.as_str()));
let session_kind =
session.connection_kind.or(connection_kind).unwrap_or(crate::api::model::ConnectionKind::Normal);
@@ -185,7 +293,7 @@ pub(in crate::api) async fn handle_hls_stream_request(
let acquired = acquire_exact_provider_handle(
app_state,
&ExactProviderAcquire {
provider: pinned_provider,
provider: &pinned_provider,
addr: &fingerprint.addr,
allow_grace: false,
priority: connection_priority_for_kind(user, session_kind),
@@ -200,7 +308,7 @@ pub(in crate::api) async fn handle_hls_stream_request(
} else {
debug_if_enabled!(
"HLS pinned provider {} unavailable for {}; aborting allocation to prevent mid-session migration",
sanitize_sensitive_info(pinned_provider),
sanitize_sensitive_info(&pinned_provider),
sanitize_sensitive_info(&fingerprint.addr.to_string())
);
None
@@ -216,7 +324,7 @@ pub(in crate::api) async fn handle_hls_stream_request(
return hls_panel_provisioning_or_status_response(
app_state,
user,
input,
effective_lookup_input,
virtual_id,
original_hls_entry_path,
server_info.path.as_deref(),
@@ -250,7 +358,7 @@ pub(in crate::api) async fn handle_hls_stream_request(
};
let Some((_provider_name, stream_url)) = select_provider_stream_url(
&url,
input,
effective_lookup_input,
&selected_provider_config,
accept_requested_stream_url,
&app_state.app_config,
@@ -277,7 +385,7 @@ pub(in crate::api) async fn handle_hls_stream_request(
debug_if_enabled!(
"HLS session {} changed while waiting for pinned provider {}",
sanitize_sensitive_info(&session.token),
sanitize_sensitive_info(pinned_provider)
sanitize_sensitive_info(&pinned_provider)
);
return axum::http::StatusCode::SERVICE_UNAVAILABLE.into_response();
}
@@ -291,7 +399,7 @@ pub(in crate::api) async fn handle_hls_stream_request(
.create_user_session(crate::api::model::CreateUserSessionParams {
user,
session_token: &session.token,
virtual_id,
virtual_id: session.virtual_id,
provider: &selected_provider_config.name,
stream_url: &stream_url,
addr: &fingerprint.addr,
@@ -312,67 +420,199 @@ pub(in crate::api) async fn handle_hls_stream_request(
None => (url, None, None, None),
}
} else {
// Append/shift catchup must keep an m3u-catchup session token even when shared HLS
// cache is off; otherwise rewritten segments register as LiveHls in the panel.
let user_session_token = hls_entry_user_session_token(
fingerprint,
&user.username,
virtual_id,
session_token_hint,
archive_reference,
);
let hls_session_owner = if hls_cache_enabled_for_user(app_state, target, user) {
let session_key = HlsSessionKey::new(input.id, stream_context.stream_ref());
let proxy_session_id = build_proxy_session_id(&session_key, &app_state.get_encrypt_secret());
Some(build_hls_origin_session_owner(&proxy_session_id))
} else {
None
};
let session_owner = hls_session_owner.as_deref().unwrap_or(user_session_token.as_str());
// Archive playback keeps its own reconnect window semantics, so the lease must
// be classified as catchup rather than plain live HLS.
let playback_kind = hls_playback_kind(archive_reference, &url, Some(user_session_token.as_str()));
let hls_session_ttl_secs = get_hls_playback_ttl_secs(app_state, playback_kind);
let Some(reservation) = try_reserve_hls_entry_origin_account_for_redirect(
app_state,
fingerprint,
user,
input,
virtual_id,
&url,
&user_session_token,
session_owner,
playback_kind,
hls_session_ttl_secs,
connection_permission,
fallback_connection_kind,
true,
let classified = tuliprox_repository::target_stream_item::classify_virtual_id(
&app_state.app_config,
&app_state.playlists,
target,
VirtualId::new(virtual_id),
)
.await
else {
return hls_panel_provisioning_or_status_response(
.await;
if let tuliprox_repository::target_stream_item::ResolvedStreamTarget::Group(group) = classified {
let user_session_token = hls_entry_user_session_token(
fingerprint,
&user.username,
virtual_id,
session_token_hint,
archive_reference,
);
let playback_kind = hls_playback_kind(archive_reference, &url, Some(user_session_token.as_str()));
let hls_session_ttl_secs = get_hls_playback_ttl_secs(app_state, playback_kind);
let playback = group_playback_settings(app_state, target);
let target_name: Arc<str> = Arc::from(target.name.as_str());
let req = GroupSelectionRequest {
lease: Some(PlaybackLeaseRef::new(user_session_token.as_str(), playback_kind)),
allow_provider_grace: playback.allow_provider_grace,
bypass_maintenance: user.is_recording_proxy_user(),
..GroupSelectionRequest::new(
&target_name,
&group,
playback.start,
&fingerprint.addr,
&user.username,
connection_priority_for_kind(user, fallback_connection_kind),
fallback_connection_kind,
)
};
let Ok(resolved) = resolve_hls_group_entry(app_state, Arc::clone(&group), req).await else {
return hls_panel_provisioning_or_status_response(
app_state,
user,
input,
virtual_id,
original_hls_entry_path,
server_info.path.as_deref(),
StatusCode::SERVICE_UNAVAILABLE,
)
.await;
};
let mut acquired_handle = resolved.acquired.handle;
let provider_handle = acquired_handle.take();
let selected_provider_config =
provider_handle.as_ref().and_then(|handle| handle.allocation.get_provider_config());
let origin_member_url = resolve_hls_origin_playlist_url(
app_state,
target,
&resolved.input,
resolved.member.member_vid.get(),
resolved.stream_url.as_ref(),
)
.await
.unwrap_or_else(|_| resolved.stream_url.to_string());
let member_url = if let Some(cfg) = &selected_provider_config {
select_provider_stream_url(&origin_member_url, &resolved.input, cfg, false, &app_state.app_config)
.await
.map_or_else(|| origin_member_url.clone(), |(_, u)| u)
} else {
origin_member_url.clone()
};
let provider_name = selected_provider_config
.as_ref()
.map_or_else(|| Arc::clone(&resolved.input.name), |cfg| Arc::clone(&cfg.name));
let session_token = app_state
.active_users
.create_user_session(crate::api::model::CreateUserSessionParams {
user,
session_token: &user_session_token,
virtual_id,
provider: &provider_name,
stream_url: &member_url,
addr: &fingerprint.addr,
connection_permission,
connection_kind: Some(fallback_connection_kind),
socket_bound: PlaylistItemType::LiveHls.uses_socket_bound_session(),
})
.await;
app_state
.active_users
.bind_member(
&user.username,
&session_token,
MemberBinding {
group_vid: group.group_vid.get(),
member_vid: resolved.member.member_vid.get(),
member_index: resolved.acquired.index,
input_name: Arc::clone(&resolved.member.input_name),
},
)
.await;
app_state.active_provider.refresh_adaptive_playback_lease(
&provider_name,
&user_session_token,
playback_kind,
hls_session_ttl_secs,
);
bound_member_info = Some((
resolved.member.member_vid.get(),
resolved.input,
resolved.member.input_stream_id.to_string(),
resolved.acquired.half_open,
));
debug_if_enabled!(
"API endpoint [HLS] create_session_fingerprint user={} virtual_id={virtual_id} (group) member_vid={} provider={} stream_url={}",
sanitize_sensitive_info(&user.username),
resolved.member.member_vid.get(),
selected_provider_config.as_ref().map_or("<unknown>", |provider| provider.name.as_ref()),
sanitize_sensitive_info(&member_url)
);
entry_url = member_url.clone();
(member_url, Some(session_token), provider_handle, selected_provider_config)
} else {
// Append/shift catchup must keep an m3u-catchup session token even when shared HLS
// cache is off; otherwise rewritten segments register as LiveHls in the panel.
let user_session_token = hls_entry_user_session_token(
fingerprint,
&user.username,
virtual_id,
session_token_hint,
archive_reference,
);
let hls_session_owner = if hls_cache_enabled_for_user(app_state, target, user) {
let session_key = HlsSessionKey::new(input.id, stream_context.stream_ref());
let proxy_session_id = build_proxy_session_id(&session_key, &app_state.get_encrypt_secret());
Some(build_hls_origin_session_owner(&proxy_session_id))
} else {
None
};
let session_owner = hls_session_owner.as_deref().unwrap_or(user_session_token.as_str());
// Archive playback keeps its own reconnect window semantics, so the lease must
// be classified as catchup rather than plain live HLS.
let playback_kind = hls_playback_kind(archive_reference, &url, Some(user_session_token.as_str()));
let hls_session_ttl_secs = get_hls_playback_ttl_secs(app_state, playback_kind);
let Some(reservation) = try_reserve_hls_entry_origin_account_for_redirect(
app_state,
fingerprint,
user,
input,
virtual_id,
original_hls_entry_path,
server_info.path.as_deref(),
StatusCode::SERVICE_UNAVAILABLE,
&url,
&user_session_token,
session_owner,
playback_kind,
hls_session_ttl_secs,
connection_permission,
fallback_connection_kind,
true,
)
.await;
};
debug_if_enabled!(
"API endpoint [HLS] create_session_fingerprint user={} virtual_id={virtual_id} provider={} stream_url={}",
sanitize_sensitive_info(&user.username),
reservation.selected_provider_config.as_ref().map_or("<unknown>", |provider| provider.name.as_ref()),
sanitize_sensitive_info(&reservation.request_url)
);
(
reservation.request_url,
Some(reservation.session_token),
reservation.provider_handle,
reservation.selected_provider_config,
)
.await
else {
return hls_panel_provisioning_or_status_response(
app_state,
user,
input,
virtual_id,
original_hls_entry_path,
server_info.path.as_deref(),
StatusCode::SERVICE_UNAVAILABLE,
)
.await;
};
debug_if_enabled!(
"API endpoint [HLS] create_session_fingerprint user={} virtual_id={virtual_id} provider={} stream_url={}",
sanitize_sensitive_info(&user.username),
reservation.selected_provider_config.as_ref().map_or("<unknown>", |provider| provider.name.as_ref()),
sanitize_sensitive_info(&reservation.request_url)
);
(
reservation.request_url,
Some(reservation.session_token),
reservation.provider_handle,
reservation.selected_provider_config,
)
}
};
// The session as created or reserved above; manifest response cookies may only enter this
@@ -403,9 +643,16 @@ pub(in crate::api) async fn handle_hls_stream_request(
}
}
let (effective_virtual_id, effective_input, effective_stream_ref, half_open_guard) =
if let Some((m_vid, ref m_input, ref m_sref, guard)) = bound_member_info {
(m_vid, m_input.as_ref(), if m_sref.is_empty() { stream_ref.as_str() } else { m_sref.as_str() }, guard)
} else {
(virtual_id, input, stream_ref.as_str(), None)
};
let user_agent_stream_index = crate::api::api_utils::resolve_stream_user_agent_index(
app_state,
input,
effective_input,
provider_handle.is_some(),
&user.username,
session_token.as_deref(),
@@ -426,7 +673,7 @@ pub(in crate::api) async fn handle_hls_stream_request(
// breaks forced same-account reuse on the next HLS/Catchup stream request.
app_state.connection_manager.release_provider_handle(provider_handle);
let input_source = InputSource::from(input).with_url(request_url);
let input_source = InputSource::from(effective_input).with_url(request_url);
let download_result = match download_legacy_hls_manifest(app_state, &input_source, &headers).await {
Ok((content, response_url, response_headers)) => match classify_hls_playlist(&content) {
// A refresh keeps its session on a malformed body; the player retries the playlist.
@@ -447,6 +694,9 @@ pub(in crate::api) async fn handle_hls_stream_request(
};
match download_result {
Ok((content, response_url, response_headers, playlist_kind)) => {
if let Some(guard) = half_open_guard {
app_state.member_health.record_success_with(guard);
}
let encrypt_secret = app_state.get_encrypt_secret();
let base_url = server_info.get_base_url();
let rewrite_hls_props = RewriteHlsProps {
@@ -455,8 +705,8 @@ pub(in crate::api) async fn handle_hls_stream_request(
content: &content,
hls_url: response_url,
target_id: target.id,
virtual_id,
input_id: input.id,
virtual_id: effective_virtual_id,
input_id: effective_input.id,
user_token: session_token.as_deref(),
origin_provider: selected_provider_name.as_deref(),
playlist_kind: Some(playlist_kind),
@@ -489,14 +739,14 @@ pub(in crate::api) async fn handle_hls_stream_request(
user,
base_url: &base_url,
target_id: target.id,
input,
virtual_id,
input: effective_input,
virtual_id: effective_virtual_id,
session_token,
sealed_url: &entry_url,
provider,
binding_tag: provider_binding_tag,
known_bitrate_bps: stream_context.known_bitrate_bps(),
stream_ref: Some(&stream_ref),
stream_ref: Some(effective_stream_ref),
},
hls_content,
)
@@ -507,6 +757,13 @@ pub(in crate::api) async fn handle_hls_stream_request(
hls_response(hls_content).into_response()
}
Err(err) => {
if let Some(guard) = half_open_guard {
app_state.member_health.record_classified_failure_with(
guard,
FailureClass::Stream,
std::time::Instant::now(),
);
}
error!("Failed to download m3u8: {}", request::text_response_error_log_label(&err));
if let Some(session_token) = session_token.as_deref() {
terminate_failed_hls_manifest_session(
@@ -544,8 +801,28 @@ pub(in crate::api::endpoints::hls_api) async fn get_stream_channel(
}
}
let target_id = target.id;
m3u_get_item_for_stream_id(virtual_id, &app_state.app_config, &app_state.playlists, target)
.await
.ok()
.map(|pli| pli.to_stream_channel(target_id))
if let Ok(pli) = m3u_get_item_for_stream_id(virtual_id, &app_state.app_config, &app_state.playlists, target).await {
return Some(pli.to_stream_channel(target_id));
}
let classified = tuliprox_repository::target_stream_item::classify_virtual_id(
&app_state.app_config,
&app_state.playlists,
target,
VirtualId::new(virtual_id),
)
.await;
match classified {
tuliprox_repository::target_stream_item::ResolvedStreamTarget::Member { group, index } => {
let mut channel = if target.has_output(TargetType::Xtream) {
tuliprox_repository::target_stream_item::member_to_xtream_item(&group, index, target)
.to_stream_channel(target_id)
} else {
tuliprox_repository::target_stream_item::member_to_m3u_item(&group, index, target)
.to_stream_channel(target_id)
};
channel.group_virtual_id = Some(group.group_vid.get());
Some(channel)
}
_ => None,
}
}
@@ -181,6 +181,7 @@ pub(in crate::api::endpoints::hls_api) fn fallback_hls_cache_stream_channel(
StreamChannel {
target_id,
virtual_id,
group_virtual_id: None,
provider_id: 0,
input_name: Arc::clone(&origin_source.input_name),
item_type: PlaylistItemType::LiveHls,
@@ -31,7 +31,7 @@ use shared::{
defaults::HLS_EXT,
model::{
ConnectFailureReason, PlaylistEntry, PlaylistItemType, StreamChannel, TargetType, UserConnectionPermission,
XtreamCluster,
VirtualId, XtreamCluster,
},
utils::{generate_random_string, is_m3u_catchup_session_token, sanitize_sensitive_info, Internable},
};
@@ -58,19 +58,47 @@ pub(in crate::api) async fn resolve_hls_virtual_source_for_target(
{
let stream_context = hls_stream_context_or_unavailable(&item, virtual_id)?;
(Arc::clone(&item.input_name), stream_context)
} else {
let item = m3u_get_item_for_stream_id(virtual_id, &app_state.app_config, &app_state.playlists, target)
.await
.map_err(|_| StatusCode::NOT_FOUND)?;
} else if let Ok(item) =
m3u_get_item_for_stream_id(virtual_id, &app_state.app_config, &app_state.playlists, target).await
{
let stream_context = hls_stream_context_or_unavailable(&item, virtual_id)?;
(Arc::clone(&item.input_name), stream_context)
} else {
let classified = tuliprox_repository::target_stream_item::classify_virtual_id(
&app_state.app_config,
&app_state.playlists,
target,
VirtualId::new(virtual_id),
)
.await;
if let tuliprox_repository::target_stream_item::ResolvedStreamTarget::Member { group, index } = classified {
let item = tuliprox_repository::target_stream_item::member_to_xtream_item(&group, index, target);
let stream_context = hls_stream_context_or_unavailable(&item, virtual_id)?;
(Arc::clone(&item.input_name), stream_context)
} else {
return Err(StatusCode::NOT_FOUND);
}
}
} else {
let item = m3u_get_item_for_stream_id(virtual_id, &app_state.app_config, &app_state.playlists, target)
.await
.map_err(|_| StatusCode::NOT_FOUND)?;
} else if let Ok(item) =
m3u_get_item_for_stream_id(virtual_id, &app_state.app_config, &app_state.playlists, target).await
{
let stream_context = hls_stream_context_or_unavailable(&item, virtual_id)?;
(Arc::clone(&item.input_name), stream_context)
} else {
let classified = tuliprox_repository::target_stream_item::classify_virtual_id(
&app_state.app_config,
&app_state.playlists,
target,
VirtualId::new(virtual_id),
)
.await;
if let tuliprox_repository::target_stream_item::ResolvedStreamTarget::Member { group, index } = classified {
let item = tuliprox_repository::target_stream_item::member_to_m3u_item(&group, index, target);
let stream_context = hls_stream_context_or_unavailable(&item, virtual_id)?;
(Arc::clone(&item.input_name), stream_context)
} else {
return Err(StatusCode::NOT_FOUND);
}
};
let input = app_state.app_config.get_input_by_name(&input_name).ok_or(StatusCode::NOT_FOUND)?;
Ok(HlsResolvedVirtualSource { input, stream_context })
@@ -78,15 +106,30 @@ pub(in crate::api) async fn resolve_hls_virtual_source_for_target(
pub(in crate::api::endpoints::hls_api) async fn resolve_hls_origin_playlist_url(
app_state: &Arc<AppState>,
target: &Arc<ConfigTarget>,
target: &ConfigTarget,
input: &ConfigInput,
virtual_id: u32,
fallback_url: &str,
) -> Result<String, StatusCode> {
if input.input_type.is_xtream() && target.has_output(TargetType::Xtream) {
let pli = xtream_get_item_for_stream_id(virtual_id, &app_state.app_config, &app_state.playlists, target, None)
.await
.map_err(|_| StatusCode::NOT_FOUND)?;
let pli = if let Ok(item) =
xtream_get_item_for_stream_id(virtual_id, &app_state.app_config, &app_state.playlists, target, None).await
{
item
} else {
let classified = tuliprox_repository::target_stream_item::classify_virtual_id(
&app_state.app_config,
&app_state.playlists,
target,
VirtualId::new(virtual_id),
)
.await;
if let tuliprox_repository::target_stream_item::ResolvedStreamTarget::Member { group, index } = classified {
tuliprox_repository::target_stream_item::member_to_xtream_item(&group, index, target)
} else {
return Err(StatusCode::NOT_FOUND);
}
};
let hls_extension = format!(".{HLS_EXT}");
let (query_path, _) = get_query_path("", Some(&hls_extension), &pli, app_state);
return get_xtream_player_api_stream_url(input, ApiStreamContext::Live, &query_path, &pli.url)
@@ -133,6 +176,7 @@ pub(in crate::api::endpoints::hls_api) async fn resolve_stream_channel(
None => StreamChannel {
target_id: target.id,
virtual_id,
group_virtual_id: None,
provider_id: 0,
input_name: Arc::clone(&input.name),
item_type: PlaylistItemType::LiveHls,
@@ -417,7 +461,9 @@ pub(in crate::api::endpoints::hls_api) async fn hls_api_stream_resolved(
let archive_reference = resolve_m3u_archive_reference(&hls_url, Some(session.token.as_str()))
.or_else(|| m3u_archive_epg_reference_ts(session.stream_url.as_ref()));
session.stream_url = hls_url.clone();
if session.virtual_id == virtual_id {
if session.virtual_id == virtual_id
|| session.member_binding.as_ref().is_some_and(|b| b.member_vid == virtual_id)
{
app_state.connection_manager.touch_http_activity(&user.username, &session.token, &fingerprint.addr).await;
} else {
return axum::http::StatusCode::BAD_REQUEST.into_response();
@@ -138,3 +138,5 @@ mod terminal_tail_recovery_before_cutover;
mod terminal_tail_terminal_cutover;
mod terminal_tail_terminal_fallback;
mod terminal_tail_terminal_preflight;
mod channel_group;
@@ -0,0 +1,50 @@
use crate::api::api_utils::tests::group_playback_support::{first_segment_url, GroupPlaybackEnv};
use axum::http::StatusCode;
#[tokio::test]
async fn hls_entry_selects_member_with_tiers() {
let env = GroupPlaybackEnv::new(&["solis", "korath"]).hls().build().await;
let _blocker = env.hold("solis", 0);
let manifest = env.get_hls_manifest(env.group_vid).await;
assert_eq!(manifest.status(), StatusCode::OK);
assert_eq!(env.origin_hits("solis"), 0);
assert_eq!(env.origin_hits("korath"), 1);
assert_eq!(env.bound_member_input("viewer").await, "korath");
}
#[tokio::test]
async fn segment_routing_resolves_member_vid_to_member_input() {
let env = GroupPlaybackEnv::new(&["solis", "korath"]).hls().with_qos("korath", 2, 95).build().await;
let manifest = env.get_hls_manifest_text(env.group_vid).await;
let segment_url = first_segment_url(&manifest);
assert!(segment_url.contains(&env.member_vid("korath").to_string()));
let segment = env.get(&segment_url).await;
assert_eq!(segment.status(), StatusCode::OK);
assert_eq!(env.origin_hits("solis"), 0);
assert_eq!(env.origin_hits("korath"), 1);
assert_eq!(env.bound_member_input("viewer").await, "korath");
}
#[tokio::test]
async fn manifest_refresh_never_reselects_even_with_open_circuit() {
let env = GroupPlaybackEnv::new(&["solis", "korath"]).hls().build().await;
let first = env.get_hls_manifest_text(env.group_vid).await;
let bound = env.bound_member_input("viewer").await;
assert!(!bound.is_empty());
env.open_circuit(&bound);
let second = env.refresh_hls_manifest(&first).await;
assert_eq!(second.status(), StatusCode::OK);
assert_eq!(env.bound_member_input("viewer").await, bound);
}
#[tokio::test]
async fn hls_cache_entry_uses_selected_member_input() {
let env = GroupPlaybackEnv::new(&["solis", "korath"]).hls_cache().with_qos("korath", 2, 95).build().await;
let manifest = env.get_hls_manifest(env.group_vid).await;
assert!(manifest.status().is_success() || manifest.status().is_redirection(), "status {}", manifest.status());
let body = axum::body::to_bytes(manifest.into_body(), 1024 * 1024).await.expect("body");
// The variant request makes the shared origin fetch its upstream playlist.
let _variant = env.get(&first_segment_url(&String::from_utf8_lossy(&body))).await;
assert_eq!(env.origin_hits("solis"), 0);
assert!(env.origin_hits("korath") > 0);
}
@@ -285,6 +285,7 @@ pub(in crate::api::endpoints::hls_api::tests) fn test_app_state_with_hls_proxy_a
auth: crate::api::model::AuthState::for_tests(),
playlist_updates: crate::api::model::PlaylistUpdateControl::for_tests(),
maintenance: crate::api::model::create_test_maintenance(""),
member_health: Arc::new(tuliprox_session::MemberHealthIndex::default()),
})
}
+124 -28
View File
@@ -9,7 +9,7 @@ use crate::{
resolve_initial_stalker_playback_url, resource_proxy_response, resource_redirect_or_proxy,
separate_number_and_remainder, should_allow_exhausted_shared_reconnect,
stream_response_with_provider_handle, try_option_bad_request, try_result_bad_request, try_result_not_found,
try_unwrap_body, RedirectParams,
try_unwrap_body, PlaybackSource, RedirectParams,
},
endpoints::{
hls_api::{
@@ -155,6 +155,7 @@ pub(in crate::api) async fn m3u_api_stream_loaded(
stream_ext: Option<&str>,
archive_discriminator: Option<&str>,
provider_allocation_id: Option<u64>,
source: Option<PlaybackSource<'_>>,
) -> impl IntoResponse + Send {
let target_name = &target.name;
if !target.has_output(TargetType::M3u) {
@@ -183,7 +184,30 @@ pub(in crate::api) async fn m3u_api_stream_loaded(
}
let virtual_id = pli.virtual_id;
if app_state.active_users.is_user_blocked_for_stream(&user.username, virtual_id).await {
let resolved_target = tuliprox_repository::target_stream_item::classify_virtual_id(
&app_state.app_config,
&app_state.playlists,
&target,
virtual_id,
)
.await;
let is_group = source.as_ref().map_or_else(
|| {
matches!(
resolved_target,
tuliprox_repository::target_stream_item::ResolvedStreamTarget::Group(_)
| tuliprox_repository::target_stream_item::ResolvedStreamTarget::Member { .. }
)
},
|s| matches!(s, PlaybackSource::Group { .. }),
);
let check_vid = match &resolved_target {
tuliprox_repository::target_stream_item::ResolvedStreamTarget::Group(group)
| tuliprox_repository::target_stream_item::ResolvedStreamTarget::Member { group, .. } => group.group_vid,
tuliprox_repository::target_stream_item::ResolvedStreamTarget::Item => virtual_id,
};
if app_state.active_users.is_user_blocked_for_stream(&user.username, check_vid).await {
return axum::http::StatusCode::BAD_REQUEST.into_response();
}
@@ -249,23 +273,25 @@ pub(in crate::api) async fn m3u_api_stream_loaded(
}
let cluster = pli.item_type.cluster();
pli.url = match resolve_initial_stalker_playback_url(
app_state,
&input,
&user,
pli.get_provider_id().unwrap_or_default(),
cluster,
pli.item_type,
&pli.url,
)
.await
{
Ok(url) => url,
Err(err) => {
error!("Failed to resolve initial Stalker playback URL: {}", sanitize_sensitive_info(&err.to_string()));
return axum::http::StatusCode::BAD_GATEWAY.into_response();
}
};
if !is_group {
pli.url = match resolve_initial_stalker_playback_url(
app_state,
&input,
&user,
pli.get_provider_id().unwrap_or_default(),
cluster,
pli.item_type,
&pli.url,
)
.await
{
Ok(url) => url,
Err(err) => {
error!("Failed to resolve initial Stalker playback URL: {}", sanitize_sensitive_info(&err.to_string()));
return axum::http::StatusCode::BAD_GATEWAY.into_response();
}
};
}
let stream_channel = || {
let mut channel = pli.to_stream_channel(target.id);
@@ -355,11 +381,13 @@ pub(in crate::api) async fn m3u_api_stream_loaded(
}
if session.virtual_id == virtual_id.get() && is_seekable_media_request(cluster, req_headers, Some(extension)) {
// partial request means we are in reverse proxy mode, seek happened
let (input, stream_channel) =
crate::api::api_utils::sticky_playback_context(app_state, session, &input, stream_channel());
return force_provider_stream_response(
fingerprint,
app_state,
session,
stream_channel(),
stream_channel,
crate::api::api_utils::ForceStreamRequestContext {
req_headers,
input: &input,
@@ -382,7 +410,9 @@ pub(in crate::api) async fn m3u_api_stream_loaded(
pli.url.clone()
};
if !is_running_playback(user_session.as_ref()) && blocks_new_playback(app_state, &input, &user, false) {
// A group checks maintenance per member during selection; its representative input says nothing.
if !is_group && !is_running_playback(user_session.as_ref()) && blocks_new_playback(app_state, &input, &user, false)
{
return maintenance_rejection_response(
app_state,
fingerprint,
@@ -527,16 +557,47 @@ pub(in crate::api) async fn m3u_api_stream_loaded(
None
};
let bound_prefer = user_session
.as_ref()
.and_then(|session| session.member_binding.as_ref())
.map(|b| (b.group_vid, b.member_index));
let mut source = source.unwrap_or_else(|| match resolved_target {
tuliprox_repository::target_stream_item::ResolvedStreamTarget::Group(group) => {
let prefer = bound_prefer.filter(|(gvid, _)| *gvid == group.group_vid.get()).map(|(_, idx)| idx);
PlaybackSource::Group { group, mode: tuliprox_session::channel_group::SelectionMode::Live, prefer }
}
tuliprox_repository::target_stream_item::ResolvedStreamTarget::Member { group, index } => {
PlaybackSource::Group {
group,
mode: tuliprox_session::channel_group::SelectionMode::Live,
prefer: Some(index),
}
}
tuliprox_repository::target_stream_item::ResolvedStreamTarget::Item => {
PlaybackSource::Single { input: &input, stream_url: &session_url }
}
});
if let PlaybackSource::Group { group, prefer, .. } = &mut source {
if prefer.is_none() {
if let Some((gvid, idx)) = bound_prefer {
if gvid == group.group_vid.get() {
*prefer = Some(idx);
}
}
}
}
stream_response_with_provider_handle(
fingerprint,
app_state,
&session_key,
Some(request_class),
stream_channel().with_epg_reference_ts(archive_reference),
&session_url,
source,
pinned_provider,
req_headers,
&input,
&target,
&user,
connection_permission,
@@ -716,6 +777,7 @@ async fn m3u_api_stream(
Some(archive.extension()),
Some(&discriminator),
None,
None,
)
.await
.into_response();
@@ -723,10 +785,25 @@ async fn m3u_api_stream(
let (action_stream_id, stream_ext) = separate_number_and_remainder(stream_req.stream_id);
let req_virtual_id: u32 = try_result_bad_request!(action_stream_id.trim().parse());
let pli =
match m3u_get_item_for_stream_id(req_virtual_id, &app_state.app_config, &app_state.playlists, &target).await {
Ok(pli) => pli,
Err(err) => {
let resolved_target = tuliprox_repository::target_stream_item::classify_virtual_id(
&app_state.app_config,
&app_state.playlists,
&target,
shared::model::VirtualId::new(req_virtual_id),
)
.await;
let pli_res =
m3u_get_item_for_stream_id(req_virtual_id, &app_state.app_config, &app_state.playlists, &target).await;
let pli = match pli_res {
Ok(pli) => pli,
Err(err) => match &resolved_target {
tuliprox_repository::target_stream_item::ResolvedStreamTarget::Group(group) => {
tuliprox_repository::target_stream_item::member_to_m3u_item(group, 0, &target)
}
tuliprox_repository::target_stream_item::ResolvedStreamTarget::Member { group, index } => {
tuliprox_repository::target_stream_item::member_to_m3u_item(group, *index, &target)
}
tuliprox_repository::target_stream_item::ResolvedStreamTarget::Item => {
error!("Failed to read m3u item for stream id {req_virtual_id}: {err}");
if stream_ext == Some(HLS_EXT) {
return axum::http::StatusCode::NOT_FOUND.into_response();
@@ -738,7 +815,8 @@ async fn m3u_api_stream(
)
.into_response();
}
};
},
};
let input = try_option_bad_request!(
app_state.app_config.get_input_by_name(&pli.input_name),
@@ -756,6 +834,21 @@ async fn m3u_api_stream(
} else {
(pli, None)
};
let source = match resolved_target {
tuliprox_repository::target_stream_item::ResolvedStreamTarget::Group(group) => Some(PlaybackSource::Group {
group,
mode: tuliprox_session::channel_group::SelectionMode::Live,
prefer: None,
}),
tuliprox_repository::target_stream_item::ResolvedStreamTarget::Member { group, index } => {
Some(PlaybackSource::Group {
group,
mode: tuliprox_session::channel_group::SelectionMode::Live,
prefer: Some(index),
})
}
tuliprox_repository::target_stream_item::ResolvedStreamTarget::Item => None,
};
m3u_api_stream_loaded(
user,
target,
@@ -767,6 +860,7 @@ async fn m3u_api_stream(
stream_ext,
archive_discriminator.as_deref(),
None,
source,
)
.await
.into_response()
@@ -843,6 +937,7 @@ async fn m3u_api_stream_nested(
stream_ext,
archive_discriminator.as_deref(),
None,
None,
)
.await
.into_response()
@@ -911,6 +1006,7 @@ async fn m3u_api_catchup(
None,
Some(&archive_discriminator),
None,
None,
)
.await
.into_response()
+1
View File
@@ -430,6 +430,7 @@ mod tests {
let channel = StreamChannel {
target_id: 1,
virtual_id: 70,
group_virtual_id: None,
provider_id: 1,
input_name: "input".intern(),
item_type: PlaylistItemType::Series,
@@ -313,6 +313,7 @@ pub(super) async fn playlist_recording_stream(
None,
None,
query.provider_allocation_id,
None,
)
.await
.into_response()
@@ -17,3 +17,4 @@ mod behavior_epg_preview;
mod behavior_manual_update;
mod behavior_provider_resolution;
mod behavior_recording_routes;
mod channel_groups;
@@ -0,0 +1,101 @@
use crate::{api::api_utils::tests::group_playback_support::GroupPlaybackEnv, model::Epg};
use shared::{
model::{EpgChannel, EpgProgramme},
utils::Internable,
};
use std::sync::Arc;
fn test_epg(channel_id: &str, title: &str, programme_title: &str) -> Epg {
let mut programme = EpgProgramme::new(3600, 7200, channel_id.intern());
programme.title = Some(programme_title.intern());
Epg {
priority: 0,
logo_override: false,
attributes: None,
children: vec![Arc::new(EpgChannel {
id: channel_id.intern(),
title: Some(title.intern()),
icon: None,
programmes: vec![programme],
})],
}
}
#[tokio::test]
async fn channel_groups_outputs_and_epg_sanity() {
let epg = test_epg("zdf.de", "ZDF", "Heute Journal");
let env = GroupPlaybackEnv::builder(&["solis", "korath"])
.without_plain_channel()
.with_member_archives(&[Some(3), Some(7)])
.with_epg(epg, "zdf.de")
.with_hdhomerun()
.build()
.await;
let member_solis = env.member_vid("solis");
let member_korath = env.member_vid("korath");
let group_vid = env.group_vid;
// 1. Xtream get_live_streams lists the group vid, no member vid, and tv_archive/tv_archive_duration equals member maximum
let res = env.get_xtream_live_streams().await;
assert_eq!(res.status(), 200);
let body_text = GroupPlaybackEnv::response_text(res).await;
let streams: serde_json::Value = serde_json::from_str(&body_text).expect("valid JSON array");
let list = streams.as_array().expect("array of streams");
assert_eq!(list.len(), 1, "Only group stream should be listed: {body_text}");
assert_eq!(list[0]["stream_id"], group_vid);
assert_eq!(list[0]["name"], "ZDF HD");
assert_eq!(list[0]["epg_channel_id"], "zdf.de");
assert_eq!(list[0]["tv_archive"], 1);
assert_eq!(list[0]["tv_archive_duration"], 7); // maximum of 3 and 7
assert!(!body_text.contains(&member_solis.to_string()), "must not contain member solis vid");
assert!(!body_text.contains(&member_korath.to_string()), "must not contain member korath vid");
// 2. M3U output lists one entry with the group EPG id
let res = env.get_m3u_playlist().await;
assert_eq!(res.status(), 200);
let m3u_text = GroupPlaybackEnv::response_text(res).await;
assert_eq!(m3u_text.matches("#EXTINF:").count(), 1, "M3U must contain exactly one stream: {m3u_text}");
assert!(m3u_text.contains(r#"tvg-id="zdf.de""#), "M3U must have group EPG id: {m3u_text}");
assert!(m3u_text.contains(&format!("/{group_vid}")), "M3U stream URL must reference group vid: {m3u_text}");
assert!(!m3u_text.contains(&format!("/{member_solis}")), "M3U must not contain member solis vid");
assert!(!m3u_text.contains(&format!("/{member_korath}")), "M3U must not contain member korath vid");
// 3. HDHomeRun lineup lists one entry
let res = env.get_hdhomerun_lineup().await;
assert_eq!(res.status(), 200);
let lineup_text = GroupPlaybackEnv::response_text(res).await;
let lineup: serde_json::Value = serde_json::from_str(&lineup_text).expect("valid JSON array");
let lineup_list = lineup.as_array().expect("array of lineup items");
assert_eq!(lineup_list.len(), 1, "Lineup must contain exactly one item: {lineup_text}");
assert_eq!(lineup_list[0]["GuideNumber"], "zdf.de");
assert_eq!(lineup_list[0]["GuideName"], "ZDF HD");
let lineup_url = lineup_list[0]["URL"].as_str().unwrap_or("");
assert!(
lineup_url.contains(&group_vid.to_string()),
"Lineup URL must reference group vid: {lineup_url} vs group {group_vid}"
);
assert!(
!lineup_url.ends_with(&format!("/{member_solis}")) && !lineup_url.ends_with(&format!("/{member_solis}.ts")),
"Lineup must not end with member solis vid: {lineup_url} vs solis {member_solis}"
);
assert!(
!lineup_url.ends_with(&format!("/{member_korath}")) && !lineup_url.ends_with(&format!("/{member_korath}.ts")),
"Lineup must not end with member korath vid: {lineup_url} vs korath {member_korath}"
);
// 4. XMLTV contains the group EPG channel
let res = env.get_xmltv_epg().await;
assert_eq!(res.status(), 200);
let xmltv_text = GroupPlaybackEnv::response_text(res).await;
assert!(xmltv_text.contains(r#"<channel id="zdf.de">"#), "XMLTV must contain group EPG channel: {xmltv_text}");
assert!(
xmltv_text.contains("<display-name>ZDF HD</display-name>"),
"XMLTV must contain channel display name: {xmltv_text}"
);
assert!(
xmltv_text.contains(r#"channel="zdf.de""#),
"XMLTV must have programme referencing group EPG channel: {xmltv_text}"
);
assert!(xmltv_text.contains("<title>Heute Journal</title>"), "XMLTV must contain programme title: {xmltv_text}");
}
@@ -145,6 +145,7 @@ pub(in crate::api::endpoints::v1_api_playlist::tests) fn test_app_state_with_man
qos_aggregation: CancellationToken::new(),
recordings: CancellationToken::new(),
hls_cache: CancellationToken::new(),
channel_groups: tokio_util::sync::CancellationToken::new(),
};
let metadata_manager = Arc::new(MetadataUpdateManager::new(tokens.metadata.clone()));
@@ -172,5 +173,6 @@ pub(in crate::api::endpoints::v1_api_playlist::tests) fn test_app_state_with_man
auth: crate::api::model::AuthState::for_tests(),
playlist_updates: crate::api::model::PlaylistUpdateControl::for_tests_with_sender(manual_update_sender),
maintenance: crate::api::model::create_test_maintenance(""),
member_health: Arc::new(tuliprox_session::MemberHealthIndex::default()),
})
}
@@ -87,7 +87,7 @@ pub(super) async fn playlist_webplayer_stream(
);
let user = Arc::new(create_api_proxy_user(&app_state));
m3u_api_stream_loaded(user, target, &fingerprint, &req_headers, &app_state, pli, input, None, None, None)
m3u_api_stream_loaded(user, target, &fingerprint, &req_headers, &app_state, pli, input, None, None, None, None)
.await
.into_response()
}
+29 -2
View File
@@ -540,7 +540,11 @@ fn to_protocol_message(event: EventMessage) -> Option<(ProtocolMessage, &'static
// Auth decisions reach notification channels and plugins, not the
// Web UI socket: there is no panel that renders them, and pushing
// every sign-in to every connected admin is noise.
| EventMessage::AuthAudit(_) => return None,
| EventMessage::AuthAudit(_)
| EventMessage::ChannelGroupMemberSelected { .. }
| EventMessage::ChannelGroupExhausted { .. }
| EventMessage::ChannelGroupCircuitChanged { .. }
| EventMessage::ChannelGroupMemberSwitched { .. } => return None,
})
}
@@ -748,7 +752,7 @@ mod tests {
// every notification-only kind against `expected = true` and the
// test failed on `DiskAlert` - it has been red since the lifecycle
// events joined the bus.
const NOT_ON_THE_WIRE: [EventKind; 28] = [
const NOT_ON_THE_WIRE: [EventKind; 32] = [
EventKind::ConnectionDenied,
EventKind::ScheduledTaskFailed,
EventKind::ProviderFetchFailed,
@@ -777,6 +781,10 @@ mod tests {
EventKind::AuthSignInFailed,
EventKind::AuthSignInThrottled,
EventKind::AuthPermissionDenied,
EventKind::ChannelGroupMemberSelected,
EventKind::ChannelGroupExhausted,
EventKind::ChannelGroupCircuitChanged,
EventKind::ChannelGroupMemberSwitched,
];
for (event, kind) in sample_event_of_every_kind() {
@@ -941,6 +949,25 @@ mod tests {
auth_audit(AuthAuditOutcome::SignInThrottled),
auth_audit(AuthAuditOutcome::PermissionDenied),
EventMessage::MaintenanceChanged(Arc::default()),
EventMessage::ChannelGroupMemberSelected {
target_id: 1,
group_vid: 10,
member_vid: 11,
tier: shared::model::channel_group::CapacityTier::Free,
},
EventMessage::ChannelGroupExhausted { target_id: 1, group_vid: 10 },
EventMessage::ChannelGroupMemberSwitched {
target_id: 1,
group_vid: 10,
from_member_vid: 11,
to_member_vid: 12,
reason: shared::model::channel_group::SwitchReason::UpstreamEof,
},
EventMessage::ChannelGroupCircuitChanged {
input_name: "test".into(),
input_stream_id: "1".into(),
state: shared::model::channel_group::CircuitState::Closed,
},
];
assert_eq!(samples.len(), EventKind::ALL.len(), "add the new variant to this list");
samples
+103 -44
View File
@@ -14,7 +14,7 @@ use crate::{
redirect, redirect_response, reentry_suppressed_response, resolve_initial_stalker_playback_url,
resource_proxy_response, resource_redirect_or_proxy, separate_number_and_remainder,
should_allow_exhausted_shared_reconnect, stream_response, stream_response_with_provider_handle,
try_option_bad_request, try_result_bad_request, try_unwrap_body, RedirectParams,
try_option_bad_request, try_result_bad_request, try_unwrap_body, PlaybackSource, RedirectParams,
},
endpoints::{
hls_api::{
@@ -295,27 +295,53 @@ async fn xtream_player_api_stream(
}
let req_virtual_id: u32 = try_result_bad_request!(action_stream_id.trim().parse());
let Ok(mut pli) =
xtream_get_item_for_stream_id(req_virtual_id, &app_state.app_config, &app_state.playlists, &target, None).await
else {
error!("Failed to read xtream item for stream id {req_virtual_id}");
if is_hls_manifest_request {
return hls_custom_video_manifest_response(
app_state,
&user,
CustomVideoStreamType::ChannelUnavailable,
axum::http::StatusCode::NOT_FOUND,
)
.await;
}
return create_custom_video_stream_response(
&app_state.provider_stream_ctx(),
&fingerprint.addr,
CustomVideoStreamType::ChannelUnavailable,
)
.into_response();
let resolved_target = tuliprox_repository::target_stream_item::classify_virtual_id(
&app_state.app_config,
&app_state.playlists,
&target,
VirtualId::new(req_virtual_id),
)
.await;
let is_group = matches!(
resolved_target,
tuliprox_repository::target_stream_item::ResolvedStreamTarget::Group(_)
| tuliprox_repository::target_stream_item::ResolvedStreamTarget::Member { .. }
);
let pli_res =
xtream_get_item_for_stream_id(req_virtual_id, &app_state.app_config, &app_state.playlists, &target, None).await;
let mut pli = match pli_res {
Ok(pli) => pli,
Err(err) => match &resolved_target {
tuliprox_repository::target_stream_item::ResolvedStreamTarget::Group(group) => {
tuliprox_repository::target_stream_item::member_to_xtream_item(group, 0, &target)
}
tuliprox_repository::target_stream_item::ResolvedStreamTarget::Member { group, index } => {
tuliprox_repository::target_stream_item::member_to_xtream_item(group, *index, &target)
}
tuliprox_repository::target_stream_item::ResolvedStreamTarget::Item => {
error!("Failed to read xtream item for stream id {req_virtual_id}: {err}");
if is_hls_manifest_request {
return hls_custom_video_manifest_response(
app_state,
&user,
CustomVideoStreamType::ChannelUnavailable,
axum::http::StatusCode::NOT_FOUND,
)
.await;
}
return create_custom_video_stream_response(
&app_state.provider_stream_ctx(),
&fingerprint.addr,
CustomVideoStreamType::ChannelUnavailable,
)
.into_response();
}
},
};
let virtual_id = pli.virtual_id;
let input_option = app_state.app_config.get_input_by_name(&pli.input_name);
let stream_ext = input_option
.as_deref()
@@ -346,8 +372,12 @@ async fn xtream_player_api_stream(
.into_response();
}
let virtual_id = pli.virtual_id;
if app_state.active_users.is_user_blocked_for_stream(&user.username, virtual_id).await {
let check_vid = match &resolved_target {
tuliprox_repository::target_stream_item::ResolvedStreamTarget::Group(group)
| tuliprox_repository::target_stream_item::ResolvedStreamTarget::Member { group, .. } => group.group_vid,
tuliprox_repository::target_stream_item::ResolvedStreamTarget::Item => virtual_id,
};
if app_state.active_users.is_user_blocked_for_stream(&user.username, check_vid).await {
return axum::http::StatusCode::BAD_REQUEST.into_response();
}
@@ -441,23 +471,25 @@ async fn xtream_player_api_stream(
(pli.xtream_cluster, pli.item_type)
};
pli.url = match resolve_initial_stalker_playback_url(
app_state,
&input,
&user,
pli.provider_id,
pli.xtream_cluster,
item_type,
&pli.url,
)
.await
{
Ok(url) => url,
Err(err) => {
error!("Failed to resolve initial Stalker playback URL: {}", sanitize_sensitive_info(&err.to_string()));
return axum::http::StatusCode::BAD_GATEWAY.into_response();
}
};
if !is_group {
pli.url = match resolve_initial_stalker_playback_url(
app_state,
&input,
&user,
pli.provider_id,
pli.xtream_cluster,
item_type,
&pli.url,
)
.await
{
Ok(url) => url,
Err(err) => {
error!("Failed to resolve initial Stalker playback URL: {}", sanitize_sensitive_info(&err.to_string()));
return axum::http::StatusCode::BAD_GATEWAY.into_response();
}
};
}
let requested_extension = if m3u_timeshift.is_some() {
extract_extension_from_url(&pli.url).map(|ext| concat_string!(".", ext))
@@ -555,6 +587,8 @@ async fn xtream_player_api_stream(
if session.virtual_id == virtual_id.get() && is_seekable_media_request(cluster, req_headers, Some(playback_ext))
{
// partial request means we are in reverse proxy mode, seek happened
let (input, stream_channel) =
crate::api::api_utils::sticky_playback_context(app_state, session, &input, stream_channel);
return force_provider_stream_response(
fingerprint,
app_state,
@@ -579,7 +613,9 @@ async fn xtream_player_api_stream(
pli.url.clone()
};
if !is_running_playback(user_session.as_ref()) && blocks_new_playback(app_state, &input, &user, false) {
// A group checks maintenance per member during selection; its representative input says nothing.
if !is_group && !is_running_playback(user_session.as_ref()) && blocks_new_playback(app_state, &input, &user, false)
{
let stream_channel = create_stream_channel_with_type(target.id, &pli, item_type);
return maintenance_rejection_response(
app_state,
@@ -715,16 +751,40 @@ async fn xtream_player_api_stream(
let pinned_provider =
user_session.as_ref().filter(|_| item_type.requires_provider_affinity()).map(|session| &session.provider);
let source = match resolved_target {
tuliprox_repository::target_stream_item::ResolvedStreamTarget::Group(group) => {
let bound_prefer = user_session
.as_ref()
.and_then(|session| session.member_binding.as_ref())
.filter(|b| b.group_vid == group.group_vid.get())
.map(|b| b.member_index);
PlaybackSource::Group {
group,
mode: tuliprox_session::channel_group::SelectionMode::Live,
prefer: bound_prefer,
}
}
tuliprox_repository::target_stream_item::ResolvedStreamTarget::Member { group, index } => {
PlaybackSource::Group {
group,
mode: tuliprox_session::channel_group::SelectionMode::Live,
prefer: Some(index),
}
}
tuliprox_repository::target_stream_item::ResolvedStreamTarget::Item => {
PlaybackSource::Single { input: &input, stream_url: &stream_url }
}
};
stream_response(
fingerprint,
app_state,
session_key.as_str(),
Some(request_class),
stream_channel,
&stream_url,
source,
pinned_provider,
req_headers,
&input,
&target,
&user,
connection_permission,
@@ -1077,10 +1137,9 @@ pub(in crate::api) async fn xtream_player_api_stream_with_resolved_target(
session_key.as_str(),
None,
pli.to_stream_channel(target.id),
&stream_url,
PlaybackSource::Single { input: &input, stream_url: &stream_url },
None,
req_headers,
&input,
&target,
&user,
UserConnectionPermission::Allowed,
@@ -912,6 +912,7 @@ async fn occupy_the_only_connection(app_state: &Arc<AppState>, user: &ProxyUserC
let channel = shared::model::StreamChannel {
target_id: 1,
virtual_id: 7,
group_virtual_id: None,
provider_id: 7,
input_name: "provider-b".intern(),
item_type: PlaylistItemType::Live,
+68 -19
View File
@@ -19,18 +19,19 @@ use crate::{
model::{
create_cache, create_http_client, create_http_client_no_redirect, create_public_http_client_no_redirect,
create_resource_http_client_no_redirect, create_resource_public_http_client_no_redirect, exec_provider_dns,
exec_qos_aggregation, load_playlists_into_memory_cache,
exec_qos_aggregation_with_listener, load_playlists_into_memory_cache,
recording_rule_scheduler::spawn_recording_rule_scheduler,
recording_supervisor::start_recording_supervisors, ActiveProviderManager, ActiveUserManager, AppState,
AuthState, CancelTokens, ConnectionManager, EventManager, EventMessage, HdHomerunAppState, HlsState,
ManualPlaylistUpdateRequest, MetadataUpdateManager, PlaylistStorageState, PlaylistUpdateControl,
RecordingQueue, SharedStreamManager,
recording_supervisor::start_recording_supervisors, ActiveProviderManager, ActiveUserManager,
AppQosAggregationListener, AppState, AuthState, CancelTokens, ConnectionManager, EventManager,
EventMessage, HdHomerunAppState, HlsState, ManualPlaylistUpdateRequest, MetadataUpdateManager,
PlaylistStorageState, PlaylistUpdateControl, RecordingQueue, SharedStreamManager,
},
panel_api::{sync_panel_api_exp_dates, sync_panel_api_exp_dates_on_boot},
serve::serve,
sys_usage::exec_system_usage,
tasks::{
exec_config_watch, exec_interner_prune, exec_scheduler, exec_xtream_expiry_sync, spawn_ssdp_discover_task,
exec_channel_group_member_keys, exec_config_watch, exec_interner_prune, exec_scheduler,
exec_xtream_expiry_sync, spawn_ssdp_discover_task,
},
},
model::{
@@ -291,6 +292,49 @@ pub(crate) async fn load_maintenance(app_config: &AppConfig) -> Arc<MaintenanceR
Arc::new(MaintenanceRegistry::load(path, &known).await)
}
/// Create all HTTP clients required by the server state.
fn create_all_http_clients(app_config: &Arc<AppConfig>) -> Result<Arc<HttpClients>, TuliproxError> {
Ok(Arc::new(HttpClients::new(
create_http_client(app_config)?,
create_http_client_no_redirect(app_config)?,
create_public_http_client_no_redirect(app_config)?,
create_resource_http_client_no_redirect(app_config)?,
create_resource_public_http_client_no_redirect(app_config)?,
)))
}
/// Populate initial `QoS` snapshots from repository into the health index.
fn populate_initial_member_health(storage_dir: &std::path::Path, member_health: &tuliprox_session::MemberHealthIndex) {
let mut initial_snapshots = Vec::new();
let _ = tuliprox_repository::QosSnapshotRepository::for_each_snapshot_read_only(storage_dir, |record| {
initial_snapshots.push(record.clone());
});
if initial_snapshots.is_empty() {
return;
}
let entries = initial_snapshots.iter().map(|record| {
let key = if record.provider_id > 0 {
tuliprox_session::QosKey::Provider {
input: Arc::clone(&record.input_name),
provider_id: record.provider_id,
}
} else {
tuliprox_session::QosKey::TargetMember {
target: Arc::clone(&record.target_name),
member_vid: record.virtual_id,
}
};
let part = tuliprox_session::QosPart {
score_24h: record.window_24h.score,
score_7d: record.window_7d.score,
confidence: record.window_24h.confidence,
failure_streak: record.window_24h.successive_failure_streak,
};
(key, part)
});
member_health.replace_qos(entries);
}
async fn create_shared_data(
app_config: &Arc<AppConfig>,
forced_targets: &Arc<ProcessTargets>,
@@ -352,11 +396,7 @@ async fn create_shared_data(
cleanup_capacity,
));
let client = create_http_client(app_config)?;
let client_no_redirect = create_http_client_no_redirect(app_config)?;
let public_client_no_redirect = create_public_http_client_no_redirect(app_config)?;
let resource_client_no_redirect = create_resource_http_client_no_redirect(app_config)?;
let resource_public_client_no_redirect = create_resource_public_http_client_no_redirect(app_config)?;
let http_clients = create_all_http_clients(app_config)?;
let tokens = CancelTokens::default();
let metadata_manager = Arc::new(MetadataUpdateManager::new(tokens.metadata.clone()));
@@ -372,16 +412,16 @@ async fn create_shared_data(
})?;
let maintenance = load_maintenance(app_config).await;
let member_health = Arc::new(tuliprox_session::MemberHealthIndex::new(
tuliprox_session::channel_group::CircuitPolicy::from(&config.channel_groups.circuit),
));
event_manager.bind_member_health(&member_health);
member_health.set_event_sink(Arc::clone(&event_manager) as Arc<dyn shared::model::EventSink>);
populate_initial_member_health(std::path::Path::new(&config.storage_dir), &member_health);
let app_state = AppState {
app_config: Arc::clone(app_config),
http_clients: Arc::new(HttpClients::new(
client,
client_no_redirect,
public_client_no_redirect,
resource_client_no_redirect,
resource_public_client_no_redirect,
)),
http_clients,
recordings: Arc::new(recordings),
cache: Arc::new(ArcSwapOption::from(cache)),
shared_stream_manager,
@@ -403,6 +443,7 @@ async fn create_shared_data(
auth: AuthState::new(token_revocations),
playlist_updates: PlaylistUpdateControl::new(Arc::clone(forced_targets), manual_update_sender),
maintenance,
member_health,
};
Ok((app_state, manual_update_rx))
@@ -459,6 +500,7 @@ async fn cancel_all_service_tokens(app_state: &Arc<AppState>) {
cancel_tokens.qos_aggregation.cancel();
cancel_tokens.recordings.cancel();
cancel_tokens.hls_cache.cancel();
cancel_tokens.channel_groups.cancel();
app_state.active_users.shutdown();
// Use the manager's shutdown() rather than cancelling the token directly so
// the is_shutdown flag is set and workers do not attempt to restart after cancellation.
@@ -709,7 +751,12 @@ pub async fn start_server(app_config: Arc<AppConfig>, targets: Arc<ProcessTarget
exec_interner_prune(&app_state);
exec_config_watch(&app_state, &cancel_token_file_watch);
exec_provider_dns(&app_state.app_config, &cancel_token_provider_dns);
exec_qos_aggregation(&app_state.app_config, &cancel_token_qos_aggregation);
exec_qos_aggregation_with_listener(
&app_state.app_config,
&cancel_token_qos_aggregation,
Some(Arc::new(AppQosAggregationListener { member_health: Arc::clone(&app_state.member_health) })),
);
exec_channel_group_member_keys(&app_state, &app_state.cancel_tokens.load().channel_groups);
exec_hls_lifecycle(&app_state.hls_ctx(), &app_state.cancel_tokens.load().hls_cache);
exec_hls_cache_gc(&app_state.hls_ctx(), &app_state.cancel_tokens.load().hls_cache);
@@ -1152,6 +1199,7 @@ mod tests {
qos_aggregation: CancellationToken::new(),
recordings: CancellationToken::new(),
hls_cache: CancellationToken::new(),
channel_groups: tokio_util::sync::CancellationToken::new(),
};
let metadata_manager = Arc::new(MetadataUpdateManager::new(tokens.metadata.clone()));
Arc::new(AppState {
@@ -1178,6 +1226,7 @@ mod tests {
auth: crate::api::model::AuthState::for_tests(),
playlist_updates: crate::api::model::PlaylistUpdateControl::for_tests(),
maintenance: crate::api::model::create_test_maintenance(""),
member_health: Arc::new(tuliprox_session::MemberHealthIndex::default()),
})
}
@@ -9,6 +9,8 @@ pub struct CancelTokens {
pub(crate) qos_aggregation: CancellationToken,
pub(crate) recordings: CancellationToken,
pub(crate) hls_cache: CancellationToken,
/// Member key refresh of channel groups; lives across config reloads.
pub(crate) channel_groups: CancellationToken,
}
impl Default for CancelTokens {
fn default() -> Self {
@@ -21,6 +23,7 @@ impl Default for CancelTokens {
qos_aggregation: CancellationToken::new(),
recordings: CancellationToken::new(),
hls_cache: CancellationToken::new(),
channel_groups: CancellationToken::new(),
}
}
}
@@ -63,6 +63,7 @@ pub struct AppState {
pub playlist_updates: PlaylistUpdateControl,
/// Inputs taken out of service; read on every new stream request.
pub maintenance: Arc<MaintenanceRegistry>,
pub member_health: Arc<tuliprox_session::MemberHealthIndex>,
}
impl AppState {
@@ -121,3 +122,33 @@ pub struct HdHomerunAppState {
pub device: Arc<HdHomeRunDeviceConfig>,
pub hd_scan_state: Arc<AtomicI8>,
}
pub struct AppQosAggregationListener {
pub member_health: Arc<tuliprox_session::MemberHealthIndex>,
}
impl tuliprox_session::qos_aggregation_manager::QosAggregationListener for AppQosAggregationListener {
fn on_snapshots(&self, snapshots: &[tuliprox_repository::QosSnapshotRecord]) {
let entries = snapshots.iter().map(|record| {
let key = if record.provider_id > 0 {
tuliprox_session::QosKey::Provider {
input: Arc::clone(&record.input_name),
provider_id: record.provider_id,
}
} else {
tuliprox_session::QosKey::TargetMember {
target: Arc::clone(&record.target_name),
member_vid: record.virtual_id,
}
};
let part = tuliprox_session::QosPart {
score_24h: record.window_24h.score,
score_7d: record.window_7d.score,
confidence: record.window_24h.confidence,
failure_streak: record.window_24h.successive_failure_streak,
};
(key, part)
});
self.member_health.replace_qos(entries);
}
}
+11 -2
View File
@@ -17,7 +17,9 @@ use shared::{error::TuliproxError, model::WebAuthConfigDto};
use std::{collections::HashMap, sync::Arc, time::Duration};
use tokio_util::sync::CancellationToken;
use tuliprox_dvr::recording::recording_transfer::{resume_recording_worker_if_needed, spawn_recording_services};
use tuliprox_session::{provider_dns_manager::exec_provider_dns, qos_aggregation_manager::exec_qos_aggregation};
use tuliprox_session::{
provider_dns_manager::exec_provider_dns, qos_aggregation_manager::exec_qos_aggregation_with_listener,
};
macro_rules! cancel_service {
($field: ident, $flag:expr, $changes:expr, $cancel_tokens:expr) => {
@@ -131,6 +133,7 @@ fn cancel_services(app_state: &Arc<AppState>, changes: &UpdateChanges) {
qos_aggregation,
recordings,
hls_cache: cancel_tokens.hls_cache.clone(),
channel_groups: cancel_tokens.channel_groups.clone(),
};
app_state.cancel_tokens.store(Arc::new(tokens));
@@ -166,7 +169,11 @@ fn start_services(app_state: &Arc<AppState>, changes: &UpdateChanges) {
exec_provider_dns(&app_state.app_config, &app_state.cancel_tokens.load().provider_dns);
}
if changes.flags.contains(UpdateChangesFlags::QosAggregation) {
exec_qos_aggregation(&app_state.app_config, &app_state.cancel_tokens.load().qos_aggregation);
exec_qos_aggregation_with_listener(
&app_state.app_config,
&app_state.cancel_tokens.load().qos_aggregation,
Some(Arc::new(super::AppQosAggregationListener { member_health: Arc::clone(&app_state.member_health) })),
);
let history_cfg =
app_state.app_config.config.load().reverse_proxy.as_ref().and_then(|rp| rp.stream_history.clone());
let connection_manager = Arc::clone(&app_state.connection_manager);
@@ -214,6 +221,8 @@ impl AppState {
let storage_dir = config.storage_dir.clone();
self.active_users.update_config(&config);
self.member_health
.set_policy(tuliprox_session::channel_group::CircuitPolicy::from(&config.channel_groups.circuit));
self.app_config.set_config(config)?;
reload_logger(config_log_level.as_deref());
self.active_provider.update_config(&self.app_config);
@@ -88,6 +88,7 @@ pub fn create_test_app_state(config: Config) -> Arc<AppState> {
auth: AuthState::for_tests(),
playlist_updates: PlaylistUpdateControl::for_tests(),
maintenance,
member_health: Arc::new(tuliprox_session::MemberHealthIndex::default()),
})
}
@@ -276,3 +276,19 @@ async fn contactable_input_hides_inputs_in_maintenance() {
app_state.maintenance.add("provider_a", None, 1).await.unwrap();
assert!(app_state.contactable_input(&name).is_none(), "an input in maintenance is not contacted");
}
#[tokio::test]
async fn config_reload_applies_channel_group_circuit_settings() {
let state = super::create_test_app_state(Config::default());
let circuit = shared::model::ChannelGroupCircuitConfig {
source_failures: 5,
..shared::model::ChannelGroupCircuitConfig::default()
};
let config = Config {
channel_groups: shared::model::ChannelGroupsConfig { circuit, ..shared::model::ChannelGroupsConfig::default() },
..Config::default()
};
assert!(state.set_config(config).await.is_ok());
assert_eq!(*state.member_health.policy(), tuliprox_session::channel_group::CircuitPolicy::from(&circuit));
}
@@ -24,6 +24,8 @@ pub(crate) struct ActiveClientStreamParams<'a> {
pub req_headers: &'a HeaderMap,
pub meter_uid: u32,
pub meter_stream: bool,
/// Replaces a failing member of a direct group playback underneath this response.
pub group_failover: Option<crate::api::api_utils::channel_group_failover::GroupFailoverDriver>,
}
pub(in crate::api) struct ActiveClientStream {
@@ -5,9 +5,12 @@ use super::{
GraceProvisioningInfo, StreamAdmissionError, StreamMode, TimedStreamContext,
};
use crate::{
api::model::{
connection_manager::PROVIDER_END_NOT_SET, AppState, BoxedProviderStream, ConnectionManager, EventManager,
MeteringStream, StreamDetails, StreamMeterHandle,
api::{
api_utils::channel_group_failover::GroupSwitcher,
model::{
connection_manager::PROVIDER_END_NOT_SET, AppState, BoxedProviderStream, ConnectionManager, EventManager,
MeteringStream, StreamDetails, StreamMeterHandle,
},
},
auth::Fingerprint,
model::ProxyUserCredentials,
@@ -107,6 +110,11 @@ pub(super) struct ActiveClientStreamState {
/// finishes it exactly once with the real provider outcome, so a full cleanup queue can
/// never leak the user request or provider lease.
pub(super) request_cleanup: Option<tuliprox_session::OwnedRequestCleanup>,
/// Runtime member failover of a direct group playback. Boxed: most streams are not group
/// streams and should not carry it inline.
pub(super) group_switcher: Option<Box<GroupSwitcher>>,
/// Lease owner re-armed for the confirmation of every member a switch opens.
pub(super) group_lease_owner: Option<Arc<str>>,
}
#[allow(clippy::too_many_lines)]
@@ -126,6 +134,7 @@ pub(crate) async fn create_active_client_stream(
req_headers,
meter_uid,
meter_stream,
group_failover,
} = request;
if connection_permission == UserConnectionPermission::Exhausted {
error!("Something is wrong this should not happen");
@@ -341,13 +350,19 @@ pub(crate) async fn create_active_client_stream(
} else {
stream
};
Some(wrap_timed_client_stream_if_needed(
app_state,
stream,
fingerprint.addr,
VirtualId::new(virtual_id),
display_stream_uid,
))
// A group stream replaces its member body on failure, so the sleep timer goes around
// the whole response instead; its end would otherwise count as a member failure.
if group_failover.is_some() {
Some(stream)
} else {
Some(wrap_timed_client_stream_if_needed(
app_state,
stream,
fingerprint.addr,
VirtualId::new(virtual_id),
display_stream_uid,
))
}
}
};
@@ -392,6 +407,8 @@ pub(crate) async fn create_active_client_stream(
provider_error_class: None,
provider_http_status: None,
provider_reconnect_count: AtomicU8::new(0),
group_lease_owner: group_failover.as_ref().and(lease_owner.clone()),
group_switcher: group_failover.map(|driver| Box::new(GroupSwitcher::new(driver))),
lease_owner,
media_started,
lease_confirmed: false,
@@ -399,5 +416,16 @@ pub(crate) async fn create_active_client_stream(
request_cleanup: registered_request.into_body_cleanup(),
};
Ok(ActiveClientStream { state }.boxed())
let timed_whole_response = state.group_switcher.is_some();
let stream = ActiveClientStream { state }.boxed();
if timed_whole_response {
return Ok(wrap_timed_client_stream_if_needed(
app_state,
stream,
fingerprint.addr,
VirtualId::new(virtual_id),
display_stream_uid,
));
}
Ok(stream)
}
@@ -71,6 +71,8 @@ impl ActiveClientStreamState {
self.terminate_quietly();
return false;
}
// A preempted slot belongs to another playback now: no failover to another member.
self.group_switcher = None;
self.provider_stopped = true;
self.preempt_cancelled = None;
self.stop_grace_task();
@@ -134,53 +136,60 @@ impl ActiveClientStreamState {
self.provider_stopped = true;
self.preempt_cancelled = None;
self.stop_grace_task();
// A member switch in flight is abandoned with the provider stream; while it runs no
// provider handle is held, but the stop is still published.
let switch_abandoned = self.group_switcher.take().is_some_and(|switcher| switcher.is_switching());
if self.provider_handle.is_some() {
// Synchronous provider-slot release; the custom-video detail update is a
// separate, best-effort UI effect.
self.release_provider_handle_and_detach_body();
self.publish_provider_stop(mode);
} else if switch_abandoned {
self.publish_provider_stop(mode);
}
}
if mode == StreamMode::ChannelUnavailable {
if let Some(flag) = &self.send_custom_stream_flag {
let _ = flag.compare_exchange(
StreamMode::Inner as u8,
StreamMode::ChannelUnavailable as u8,
Ordering::AcqRel,
Ordering::Relaxed,
);
}
/// Publishes the end of the provider stream: custom-video mode, waker, panel detail or the
/// release of the stream claim.
pub(super) fn publish_provider_stop(&mut self, mode: StreamMode) {
if mode == StreamMode::ChannelUnavailable {
if let Some(flag) = &self.send_custom_stream_flag {
let _ = flag.compare_exchange(
StreamMode::Inner as u8,
StreamMode::ChannelUnavailable as u8,
Ordering::AcqRel,
Ordering::Relaxed,
);
}
}
if let Some(waker) = &self.waker {
waker.wake();
}
if let Some(waker) = &self.waker {
waker.wake();
}
let addr = self.fingerprint.addr;
let reason = match mode {
StreamMode::ChannelUnavailable => "unavailable provider channel",
StreamMode::UserExhausted => "user grace period exhaustion",
StreamMode::ProviderExhausted => "provider grace period exhaustion",
StreamMode::Provisioning => "provider grace period provisioning",
StreamMode::LowPriorityPreempted => "low-priority preemption",
StreamMode::ReentrySuppressed => "reentry suppression",
StreamMode::Inner | StreamMode::GracePending => "stream mode transition",
};
debug_if_enabled!(
"Provider stream stopped due to {reason} for {}",
sanitize_sensitive_info(&addr.to_string())
);
if let Some(video_type) = Self::custom_video_type_for_mode(mode) {
self.connection_manager.send_cleanup(CleanupEvent::UpdateDetailAndReleaseProvider {
addr,
stream_uid: self.stream_uid,
video_type,
handle: None,
});
} else {
// No custom video for this mode: release the request without publishing
// a detail update so no error clip is shown.
self.release_user_stream();
}
let addr = self.fingerprint.addr;
let reason = match mode {
StreamMode::ChannelUnavailable => "unavailable provider channel",
StreamMode::UserExhausted => "user grace period exhaustion",
StreamMode::ProviderExhausted => "provider grace period exhaustion",
StreamMode::Provisioning => "provider grace period provisioning",
StreamMode::LowPriorityPreempted => "low-priority preemption",
StreamMode::ReentrySuppressed => "reentry suppression",
StreamMode::Inner | StreamMode::GracePending => "stream mode transition",
};
debug_if_enabled!("Provider stream stopped due to {reason} for {}", sanitize_sensitive_info(&addr.to_string()));
if let Some(video_type) = Self::custom_video_type_for_mode(mode) {
self.connection_manager.send_cleanup(CleanupEvent::UpdateDetailAndReleaseProvider {
addr,
stream_uid: self.stream_uid,
video_type,
handle: None,
});
} else {
// No custom video for this mode: release the request without publishing
// a detail update so no error clip is shown.
self.release_user_stream();
}
}
}
@@ -2,21 +2,31 @@ use super::{
ActiveClientStream, ActiveClientStreamState, DeferredProviderOpenOutcome, DeferredProviderOpenState, StreamMode,
TimedStreamContext,
};
use crate::api::model::{
connection_manager::{PROVIDER_END_CLOSED, PROVIDER_END_ERROR, PROVIDER_END_NOT_SET},
open_provider_stream_with_lifecycle, AppState, BoxedProviderStream, ProviderStreamOpen,
ProviderStreamOpenLifecycle, StreamError, TimedClientStream,
use crate::api::{
api_utils::channel_group_failover::{GroupEnd, GroupPoll, MemberHost, RuntimeMember},
model::{
connection_manager::{PROVIDER_END_CLOSED, PROVIDER_END_ERROR, PROVIDER_END_NOT_SET},
open_provider_stream_with_lifecycle, AppState, BoxedProviderStream, ProviderStreamOpen,
ProviderStreamOpenLifecycle, StreamError, TimedClientStream,
},
};
use bytes::Bytes;
use futures::{Future, Stream, StreamExt};
use log::{error, info};
use shared::{model::VirtualId, utils::sanitize_sensitive_info};
use shared::{
model::{SwitchReason, VirtualId},
utils::sanitize_sensitive_info,
};
use std::{
net::SocketAddr,
pin::Pin,
sync::{atomic::Ordering, Arc},
sync::{
atomic::{AtomicU8, Ordering},
Arc,
},
task::{Context, Poll},
};
use tuliprox_session::stream_options::StreamResponseMode;
impl ActiveClientStreamState {
/// Drops the provider handle at most once (releasing the provider slot
@@ -74,6 +84,30 @@ impl ActiveClientStreamState {
}
}
/// No member could take over within the budget: the response ends like a provider stream
/// that ended without failover.
/// Returns true when the channel-unavailable video follows instead of the end of the body.
pub(super) fn end_failed_group_switch(&mut self, reason: SwitchReason) -> bool {
// The provider ended cleanly only when its body ended; otherwise it failed.
let end_reason = if reason == SwitchReason::UpstreamEof { PROVIDER_END_CLOSED } else { PROVIDER_END_ERROR };
let _ = self.provider_end_reason.compare_exchange(
PROVIDER_END_NOT_SET,
end_reason,
Ordering::Relaxed,
Ordering::Relaxed,
);
self.preempt_cancelled = None;
self.stop_grace_task();
self.provider_stopped = true;
let shows_video = self.response_mode == StreamResponseMode::Stream && self.custom_video.unavailable.is_some();
if shows_video {
// The mode flag only exists for streams admitted with grace or preemption video.
self.send_custom_stream_flag.get_or_insert_with(|| Arc::new(AtomicU8::new(StreamMode::Inner as u8)));
}
self.publish_provider_stop(StreamMode::ChannelUnavailable);
shows_video
}
pub(super) fn release_stream_and_provider_handle_once(&mut self) {
self.stop_grace_task();
self.release_provider_handle_and_detach_body();
@@ -98,6 +132,40 @@ impl ActiveClientStreamState {
}
}
impl MemberHost for ActiveClientStreamState {
fn member_body(&mut self) -> Option<&mut BoxedProviderStream> { self.inner.as_mut() }
fn member_preempted(&self) -> bool {
self.provider_handle
.as_ref()
.and_then(|managed| managed.handle())
.and_then(|handle| handle.cancel_token.as_ref())
.is_some_and(tokio_util::sync::CancellationToken::is_cancelled)
}
fn release_member(&mut self) {
self.preempt_cancelled = None;
self.release_provider_handle_and_detach_body();
}
fn install_member(&mut self, member: RuntimeMember) {
self.preempt_cancelled = member
.handle
.as_ref()
.and_then(|managed| managed.handle())
.and_then(|handle| handle.cancel_token.as_ref())
.map(|token| Box::pin(token.clone().cancelled_owned()));
self.provider_handle = member.handle;
self.provider_handle_released = false;
self.inner = Some(self.wrap_provider_stream(member.stream));
self.provider_reconnect_count.fetch_add(1, Ordering::Relaxed);
if self.group_lease_owner.is_some() {
self.lease_owner.clone_from(&self.group_lease_owner);
self.lease_confirmed = false;
}
}
}
pub(super) fn wrap_timed_client_stream_if_needed(
app_state: &Arc<AppState>,
stream: BoxedProviderStream,
@@ -185,100 +253,125 @@ impl Stream for ActiveClientStream {
StreamMode::Inner => {
self.state.reset_custom_video_timeout();
if self.state.inner.is_none() {
if let Some(deferred_provider_open) = self.state.deferred_provider_open.take() {
match deferred_provider_open {
DeferredProviderOpenState::Pending(context) => {
let app_state = Arc::clone(&context.app_state);
let client = {
let http_client = app_state.http_clients.default.load();
http_client.as_ref().clone()
};
let lifecycle = self
.state
.provider_handle
.as_ref()
.and_then(ProviderStreamOpenLifecycle::from_managed);
let future = Box::pin(async move {
match open_provider_stream_with_lifecycle(
&app_state.provider_stream_ctx(),
&client,
context.provider_stream_factory_options,
lifecycle,
)
.await
{
Some(ProviderStreamOpen::Stream(response))
if matches!(response.info, Some((_, _, _, Some(_)))) =>
let polled = if let Some(mut switcher) = self.state.group_switcher.take() {
let polled = switcher.poll_member(&mut self.state, cx);
if !matches!(polled, GroupPoll::Ended(_)) {
self.state.group_switcher = Some(switcher);
}
polled
} else {
if self.state.inner.is_none() {
if let Some(deferred_provider_open) = self.state.deferred_provider_open.take() {
match deferred_provider_open {
DeferredProviderOpenState::Pending(context) => {
let app_state = Arc::clone(&context.app_state);
let client = {
let http_client = app_state.http_clients.default.load();
http_client.as_ref().clone()
};
let lifecycle = self
.state
.provider_handle
.as_ref()
.and_then(ProviderStreamOpenLifecycle::from_managed);
let future = Box::pin(async move {
match open_provider_stream_with_lifecycle(
&app_state.provider_stream_ctx(),
&client,
context.provider_stream_factory_options,
lifecycle,
)
.await
{
let Some((_headers, _status, _response_url, Some(custom_video_type))) =
response.info
else {
return DeferredProviderOpenOutcome::Failed;
};
ActiveClientStreamState::mode_for_custom_video_type(custom_video_type)
Some(ProviderStreamOpen::Stream(response))
if matches!(response.info, Some((_, _, _, Some(_)))) =>
{
let Some((
_headers,
_status,
_response_url,
Some(custom_video_type),
)) = response.info
else {
return DeferredProviderOpenOutcome::Failed;
};
ActiveClientStreamState::mode_for_custom_video_type(
custom_video_type,
)
.map_or(
DeferredProviderOpenOutcome::Failed,
DeferredProviderOpenOutcome::Mode,
)
}
Some(ProviderStreamOpen::Stream(response)) => {
DeferredProviderOpenOutcome::Stream(response.stream)
}
Some(ProviderStreamOpen::UpstreamStatus { .. }) | None => {
DeferredProviderOpenOutcome::Failed
}
}
Some(ProviderStreamOpen::Stream(response)) => {
DeferredProviderOpenOutcome::Stream(response.stream)
}
Some(ProviderStreamOpen::UpstreamStatus { .. }) | None => {
DeferredProviderOpenOutcome::Failed
}
}
});
self.state.deferred_provider_open =
Some(DeferredProviderOpenState::Opening(future));
continue;
}
DeferredProviderOpenState::Opening(mut future) => match future.as_mut().poll(cx) {
Poll::Pending => {
});
self.state.deferred_provider_open =
Some(DeferredProviderOpenState::Opening(future));
return Poll::Pending;
}
Poll::Ready(DeferredProviderOpenOutcome::Stream(stream)) => {
self.state.provider_reconnect_count.fetch_add(1, Ordering::Relaxed);
self.state.inner = Some(self.state.wrap_provider_stream(stream));
self.state.mark_direct_body_progress();
continue;
}
Poll::Ready(DeferredProviderOpenOutcome::Mode(mode)) => {
if !self.state.enter_custom_mode(mode) {
return Poll::Ready(None);
DeferredProviderOpenState::Opening(mut future) => match future.as_mut().poll(cx) {
Poll::Pending => {
self.state.deferred_provider_open =
Some(DeferredProviderOpenState::Opening(future));
return Poll::Pending;
}
continue;
}
Poll::Ready(DeferredProviderOpenOutcome::Failed) => {
if !self.state.enter_custom_mode(StreamMode::ChannelUnavailable) {
return Poll::Ready(None);
Poll::Ready(DeferredProviderOpenOutcome::Stream(stream)) => {
self.state.provider_reconnect_count.fetch_add(1, Ordering::Relaxed);
self.state.inner = Some(self.state.wrap_provider_stream(stream));
self.state.mark_direct_body_progress();
continue;
}
continue;
}
},
Poll::Ready(DeferredProviderOpenOutcome::Mode(mode)) => {
if !self.state.enter_custom_mode(mode) {
return Poll::Ready(None);
}
continue;
}
Poll::Ready(DeferredProviderOpenOutcome::Failed) => {
if !self.state.enter_custom_mode(StreamMode::ChannelUnavailable) {
return Poll::Ready(None);
}
continue;
}
},
}
}
if self.state.grace_task_handle.is_none() {
self.state.stop_provider_stream(StreamMode::ChannelUnavailable);
return Poll::Ready(None);
}
return Poll::Pending;
}
if self.state.grace_task_handle.is_none() {
self.state.stop_provider_stream(StreamMode::ChannelUnavailable);
return Poll::Ready(None);
match self.state.inner.as_mut().map(|inner| Pin::new(inner).poll_next(cx)) {
Some(Poll::Ready(Some(Ok(bytes)))) => GroupPoll::Data(bytes),
Some(Poll::Ready(Some(Err(e)))) => GroupPoll::Ended(GroupEnd::Error(e)),
Some(Poll::Ready(None)) | None => GroupPoll::Ended(GroupEnd::Eof),
Some(Poll::Pending) => GroupPoll::Pending,
}
};
return Poll::Pending;
}
match self.state.inner.as_mut().map(|inner| Pin::new(inner).poll_next(cx)) {
Some(Poll::Ready(Some(Ok(bytes)))) => {
match polled {
GroupPoll::Keepalive(packets) => {
self.state.mark_direct_body_progress();
return Poll::Ready(Some(Ok(packets)));
}
GroupPoll::Data(bytes) => {
if !bytes.is_empty() {
self.state.confirm_lease_once();
}
self.state.mark_direct_body_progress();
return Poll::Ready(Some(Ok(bytes)));
}
Some(Poll::Ready(Some(Err(e)))) => {
GroupPoll::Ended(GroupEnd::Error(e)) => {
error!("Inner stream error: {e:?}");
self.state.provider_error_class = Some(e.provider_error_class());
self.state.provider_http_status = e.provider_http_status();
@@ -295,7 +388,7 @@ impl Stream for ActiveClientStream {
return Poll::Pending;
}
Some(Poll::Ready(None)) | None => {
GroupPoll::Ended(GroupEnd::Eof) => {
let _ = self.state.provider_end_reason.compare_exchange(
PROVIDER_END_NOT_SET,
PROVIDER_END_CLOSED,
@@ -309,7 +402,24 @@ impl Stream for ActiveClientStream {
return Poll::Pending;
}
Some(Poll::Pending) => {
// A stalled group member that no other member could replace.
GroupPoll::Ended(GroupEnd::Stalled) => {
let _ = self.state.provider_end_reason.compare_exchange(
PROVIDER_END_NOT_SET,
PROVIDER_END_ERROR,
Ordering::Relaxed,
Ordering::Relaxed,
);
self.state.stop_provider_stream(StreamMode::ChannelUnavailable);
return Poll::Ready(None);
}
GroupPoll::Ended(GroupEnd::SwitchFailed { reason }) => {
if self.state.end_failed_group_switch(reason) {
continue;
}
return Poll::Ready(None);
}
GroupPoll::Pending => {
if self.state.direct_body_idle_timeout.poll_expired(cx) {
self.state.stop_direct_body_idle_timeout();
return Poll::Ready(None);
@@ -90,6 +90,7 @@ pub(in crate::api::model::streams::active_client_stream::tests) fn create_test_a
auth: crate::api::model::AuthState::for_tests(),
playlist_updates: crate::api::model::PlaylistUpdateControl::for_tests(),
maintenance: crate::api::model::create_test_maintenance(""),
member_health: Arc::new(tuliprox_session::MemberHealthIndex::default()),
})
}
@@ -422,6 +423,7 @@ async fn test_active_client_stream_deferred_provider_grace_retains_provider_hand
req_headers: &HeaderMap::default(),
meter_uid: 0,
meter_stream: false,
group_failover: None,
})
.await
.expect("deferred test stream admission should succeed");
@@ -499,6 +501,7 @@ async fn test_active_client_stream_shared_deferred_provider_grace_retains_provid
req_headers: &HeaderMap::default(),
meter_uid: 0,
meter_stream: false,
group_failover: None,
})
.await
.expect("shared deferred test stream admission should succeed");
@@ -185,6 +185,8 @@ async fn test_active_client_stream_deferred_provider_open_applies_sleep_timer_ti
provider_error_class: None,
provider_http_status: None,
provider_reconnect_count: AtomicU8::new(0),
group_switcher: None,
group_lease_owner: None,
lease_owner: None,
media_started: None,
lease_confirmed: false,
@@ -238,6 +240,7 @@ async fn test_active_client_stream_immediate_provider_stream_emits_meter_batches
req_headers: &HeaderMap::default(),
meter_uid: 55,
meter_stream: true,
group_failover: None,
})
.await
.expect("metered test stream admission should succeed");
@@ -141,6 +141,7 @@ pub(in crate::api::model::streams::active_client_stream::tests) async fn create_
req_headers: &HeaderMap::default(),
meter_uid: 0,
meter_stream: false,
group_failover: None,
})
.await
.expect("direct test stream admission should succeed");
@@ -122,6 +122,7 @@ pub(in crate::api::model::streams::active_client_stream::tests) fn create_test_a
auth: crate::api::model::AuthState::for_tests(),
playlist_updates: crate::api::model::PlaylistUpdateControl::for_tests(),
maintenance: crate::api::model::create_test_maintenance(""),
member_health: Arc::new(tuliprox_session::MemberHealthIndex::default()),
})
}
@@ -147,6 +148,7 @@ pub(in crate::api::model::streams::active_client_stream::tests) fn create_test_s
StreamChannel {
target_id: 1,
virtual_id,
group_virtual_id: None,
provider_id: 1,
input_name: "input".intern(),
item_type: PlaylistItemType::Live,
@@ -232,6 +234,8 @@ pub(in crate::api::model::streams::active_client_stream::tests) fn custom_video_
provider_error_class: None,
provider_http_status: None,
provider_reconnect_count: AtomicU8::new(0),
group_switcher: None,
group_lease_owner: None,
lease_owner: None,
media_started: None,
lease_confirmed: false,
@@ -44,7 +44,8 @@ create_bitset!(
RangeRequested,
PublicDestinationRequired,
HlsResource,
FlussonicAudioTracks
FlussonicAudioTracks,
GroupMember
);
#[derive(Debug, Clone)]
@@ -101,7 +102,9 @@ mod response;
pub use opening::{create_provider_stream, open_provider_stream_with_lifecycle, ProviderStreamOpen};
pub(crate) use options::ProviderStreamFactoryParams;
#[cfg(test)]
use request::{prepare_client, provider_headers_require_manual_redirects, send_with_manual_redirects};
use request::{
initial_open_retry_exhausted, prepare_client, provider_headers_require_manual_redirects, send_with_manual_redirects,
};
#[cfg(test)]
use response::normalize_hls_resource_content_type;
#[cfg(test)]
@@ -124,7 +124,9 @@ pub async fn create_provider_stream(
Some(status),
Some(failure.provider_error_class()),
);
if stream_options.flags.contains(ProviderStreamFactoryFlags::HlsResource) {
if stream_options.flags.contains(ProviderStreamFactoryFlags::HlsResource)
|| stream_options.flags.contains(ProviderStreamFactoryFlags::GroupMember)
{
let headers = match failure {
ProviderStreamRequestFailure::HlsStatus { headers, .. } => headers,
ProviderStreamRequestFailure::Status { .. } => Vec::new(),
@@ -42,6 +42,41 @@ pub(crate) struct ProviderStreamFactoryParams<'a> {
pub content_representation: ProviderContentRepresentationMode,
}
fn compute_flags(
stream_options: &StreamOptions,
item_type: PlaylistItemType,
share_stream: bool,
range_requested: bool,
is_group_member: bool,
) -> ProviderStreamFactoryFlagsSet {
let mut flags = ProviderStreamFactoryFlagsSet::new();
if stream_options.stream_retry {
flags.set(ProviderStreamFactoryFlags::RetryEnabled);
if !item_type.is_live_adaptive() && stream_options.response_mode != StreamResponseMode::HlsResource {
flags.set(ProviderStreamFactoryFlags::InitialRetryLoopEnabled);
}
}
if stream_options.pipe_provider_stream {
flags.set(ProviderStreamFactoryFlags::PipeStream);
}
if stream_options.buffer_enabled {
flags.set(ProviderStreamFactoryFlags::BufferEnabled);
}
if stream_options.response_mode == StreamResponseMode::HlsResource {
flags.set(ProviderStreamFactoryFlags::HlsResource);
}
if share_stream {
flags.set(ProviderStreamFactoryFlags::ShareStream);
}
if range_requested {
flags.set(ProviderStreamFactoryFlags::RangeRequested);
}
if is_group_member {
flags.set(ProviderStreamFactoryFlags::GroupMember);
}
flags
}
impl ProviderStreamFactoryOptions {
pub(crate) fn new(request: &ProviderStreamFactoryParams<'_>) -> Self {
let ProviderStreamFactoryParams {
@@ -97,28 +132,13 @@ impl ProviderStreamFactoryOptions {
})
.and_then(|ua| axum::http::header::HeaderValue::from_str(ua).ok());
let url = (*stream_url).clone();
let mut flags = ProviderStreamFactoryFlagsSet::new();
if stream_options.stream_retry {
flags.set(ProviderStreamFactoryFlags::RetryEnabled);
if !item_type.is_live_adaptive() && stream_options.response_mode != StreamResponseMode::HlsResource {
flags.set(ProviderStreamFactoryFlags::InitialRetryLoopEnabled);
}
}
if stream_options.pipe_provider_stream {
flags.set(ProviderStreamFactoryFlags::PipeStream);
}
if stream_options.buffer_enabled {
flags.set(ProviderStreamFactoryFlags::BufferEnabled);
}
if stream_options.response_mode == StreamResponseMode::HlsResource {
flags.set(ProviderStreamFactoryFlags::HlsResource);
}
if *share_stream {
flags.set(ProviderStreamFactoryFlags::ShareStream);
}
if requested_range.is_some() {
flags.set(ProviderStreamFactoryFlags::RangeRequested);
}
let flags = compute_flags(
stream_options,
*item_type,
*share_stream,
requested_range.is_some(),
stream_channel.as_ref().and_then(|ch| ch.group_virtual_id).is_some(),
);
Self {
item_type: *item_type,
@@ -183,8 +183,10 @@ pub(super) async fn send_with_manual_redirects(
provider.as_ref(),
true,
stream_options.should_retry_provider_request(),
RequestFetchOptions::default()
.with_http_error_responses(stream_options.flags.contains(ProviderStreamFactoryFlags::HlsResource)),
RequestFetchOptions::default().with_http_error_responses(
stream_options.flags.contains(ProviderStreamFactoryFlags::HlsResource)
|| stream_options.flags.contains(ProviderStreamFactoryFlags::GroupMember),
),
|resolved_url| prepare_client(request_client, stream_options, Some(resolved_url), credential_state).0,
)
.await;
@@ -263,8 +265,10 @@ async fn provider_stream_request(
provider.as_ref(),
false,
stream_options.should_retry_provider_request(),
RequestFetchOptions::default()
.with_http_error_responses(stream_options.flags.contains(ProviderStreamFactoryFlags::HlsResource)),
RequestFetchOptions::default().with_http_error_responses(
stream_options.flags.contains(ProviderStreamFactoryFlags::HlsResource)
|| stream_options.flags.contains(ProviderStreamFactoryFlags::GroupMember),
),
|resolved_url| {
let (client, _partial_content) = prepare_client(
request_client,
@@ -426,6 +430,16 @@ fn request_provider_account_probe(ctx: &ProviderStreamCtx, options: &ProviderStr
}
}
/// `connect_err` counts the attempt that just ran. A group member never retries itself: the
/// group selection moves on to the next member instead.
pub(super) fn initial_open_retry_exhausted(is_group_member: bool, connect_err: u32) -> bool {
if is_group_member {
connect_err >= 1
} else {
connect_err > ERR_MAX_RETRY_COUNT
}
}
#[allow(clippy::too_many_lines)]
pub(super) async fn get_provider_stream(
ctx: &ProviderStreamCtx,
@@ -436,6 +450,8 @@ pub(super) async fn get_provider_stream(
debug_if_enabled!("stream provider {}", sanitize_sensitive_info(log_url.as_ref()));
let start = Instant::now();
let mut connect_err: u32 = 1;
let is_group_member = stream_options.flags.contains(ProviderStreamFactoryFlags::GroupMember);
let mut last_failure: Option<ProviderStreamRequestFailure> = None;
let handle_cancel = stream_options.get_cancel_token();
while stream_options.should_continue() {
@@ -464,7 +480,7 @@ pub(super) async fn get_provider_stream(
return Ok(Some(stream_response));
}
Ok(None) => {
if connect_err > ERR_MAX_RETRY_COUNT {
if initial_open_retry_exhausted(is_group_member, connect_err) {
warn!(
"The stream could be unavailable. {}",
sanitize_sensitive_info(stream_options.get_log_url().as_ref())
@@ -495,18 +511,21 @@ pub(super) async fn get_provider_stream(
"The stream could be unavailable. ({status}) {}",
sanitize_sensitive_info(stream_options.get_log_url().as_ref())
);
last_failure = Some(failure);
break;
}
if connect_err > ERR_MAX_RETRY_COUNT {
if initial_open_retry_exhausted(is_group_member, connect_err) {
warn!(
"The stream could be unavailable. ({status}) {}",
sanitize_sensitive_info(stream_options.get_log_url().as_ref())
);
last_failure = Some(failure);
break;
}
last_failure = Some(failure);
}
}
if !stream_options.should_continue() || connect_err > ERR_MAX_RETRY_COUNT {
if !stream_options.should_continue() || initial_open_retry_exhausted(is_group_member, connect_err) {
break;
}
if !stream_options.should_retry_initial_open_loop() {
@@ -533,6 +552,9 @@ pub(super) async fn get_provider_stream(
}
debug_if_enabled!("Stopped reconnecting stream {}", sanitize_sensitive_info(stream_options.get_log_url().as_ref()));
stream_options.cancel_reconnect();
if let Some(failure) = last_failure {
return Err(failure);
}
Err(ProviderStreamRequestFailure::Status {
status: StatusCode::SERVICE_UNAVAILABLE,
provider_error_class: "service_unavailable",
@@ -74,3 +74,17 @@ async fn legacy_hls_stream_decoder_failure_aborts_without_retry() {
assert!(matches!(error, StreamError::ContentDecoding(_)));
assert_eq!(requests.load(Ordering::SeqCst), 1, "a body-stream failure cannot trigger a new request");
}
#[test]
fn non_group_open_retries_until_retry_limit() {
// The attempt counter starts at 1, so a single source gets ERR_MAX_RETRY_COUNT + 1 attempts.
for attempt in 1..=ERR_MAX_RETRY_COUNT {
assert!(!initial_open_retry_exhausted(false, attempt), "attempt {attempt} must allow a retry");
}
assert!(initial_open_retry_exhausted(false, ERR_MAX_RETRY_COUNT + 1));
}
#[test]
fn group_member_open_does_not_retry() {
assert!(initial_open_retry_exhausted(true, 1));
}
@@ -102,6 +102,7 @@ fn test_provider_stream_factory_options_builds_connect_failed_stream_info_from_h
stream_channel: Some(&StreamChannel {
target_id: 1,
virtual_id: 77,
group_virtual_id: None,
provider_id: 3,
input_name: "input-a".intern(),
item_type: PlaylistItemType::Live,
+2
View File
@@ -293,6 +293,7 @@ mod tests {
let stream_channel = StreamChannel {
target_id: 1,
virtual_id: 1,
group_virtual_id: None,
provider_id: 1,
input_name: "input".intern(),
item_type: PlaylistItemType::Series,
@@ -337,6 +338,7 @@ mod tests {
req_headers: &HeaderMap::new(),
meter_uid: 0,
meter_stream: false,
group_failover: None,
})
.await
.expect("test stream admission should succeed");
@@ -0,0 +1,65 @@
use crate::api::model::AppState;
use log::warn;
use shared::model::{ConfigType, EventMessage, PlaylistUpdateState};
use std::{collections::HashSet, sync::Arc};
use tokio::sync::broadcast::error::RecvError;
use tokio_util::sync::CancellationToken;
use tuliprox_repository::target_stream_item::channel_groups_enabled;
/// Restricts the member health probe store to the members of all channel groups.
/// Returns the number of member keys.
pub(in crate::api) async fn refresh_member_keys(app_state: &AppState) -> usize {
let targets: Vec<_> = app_state
.app_config
.sources
.load()
.sources
.iter()
.flat_map(|source| source.targets.iter())
.filter(|target| channel_groups_enabled(target))
.map(Arc::clone)
.collect();
let mut keys = HashSet::new();
for target in targets {
let data =
app_state.playlists.channel_groups.get_or_load(&app_state.app_config, &app_state.playlists, &target).await;
keys.extend(data.member_keys());
}
let count = keys.len();
app_state.member_health.set_member_keys(keys);
count
}
/// Refreshes the member keys at startup, after every playlist update and after a sources or
/// config reload, until `cancel` fires.
pub(in crate::api) fn exec_channel_group_member_keys(
app_state: &Arc<AppState>,
cancel: &CancellationToken,
) -> tokio::task::JoinHandle<()> {
let app_state = Arc::clone(app_state);
let cancel = cancel.clone();
tokio::spawn(async move {
let mut rx = app_state.event_manager.get_event_channel();
refresh_member_keys(&app_state).await;
loop {
let event = tokio::select! {
() = cancel.cancelled() => break,
event = rx.recv() => event,
};
match event {
Ok(EventMessage::PlaylistUpdate(summary)) if summary.state != PlaylistUpdateState::Failure => {
refresh_member_keys(&app_state).await;
}
Ok(EventMessage::ConfigChange(ConfigType::Sources | ConfigType::Config)) => {
refresh_member_keys(&app_state).await;
}
Ok(_) => {}
Err(RecvError::Lagged(skipped)) => {
warn!("Channel group member key refresh lagged by {skipped} events");
refresh_member_keys(&app_state).await;
}
Err(RecvError::Closed) => break,
}
}
})
}
+2
View File
@@ -15,6 +15,7 @@
//! would gain `session` for an event publish - and 194 lines is not worth
//! degrading a leaf crate or minting a package to hold.
mod channel_group_members;
mod config_watch;
mod hdhomerun_ssdp;
mod library_scan;
@@ -22,6 +23,7 @@ mod notification_bridge;
mod scheduler;
mod xtream_expiry;
pub(in crate::api) use channel_group_members::*;
pub(in crate::api) use config_watch::*;
pub(in crate::api) use hdhomerun_ssdp::*;
pub(in crate::api) use library_scan::*;
@@ -287,7 +287,11 @@ pub fn to_notification(message: &EventMessage) -> Option<NotificationEvent> {
// which is the single place that decision is made.
EventMessage::PlaylistUpdateProgress(_)
| EventMessage::SystemInfoUpdate(_)
| EventMessage::RecordingProgress => None,
| EventMessage::RecordingProgress
| EventMessage::ChannelGroupMemberSelected { .. }
| EventMessage::ChannelGroupExhausted { .. }
| EventMessage::ChannelGroupCircuitChanged { .. }
| EventMessage::ChannelGroupMemberSwitched { .. } => None,
}
}
@@ -631,6 +635,25 @@ mod tests {
auth_audit(AuthAuditOutcome::SignInFailed),
auth_audit(AuthAuditOutcome::SignInThrottled),
auth_audit(AuthAuditOutcome::PermissionDenied),
EventMessage::ChannelGroupMemberSelected {
target_id: 1,
group_vid: 10,
member_vid: 11,
tier: shared::model::channel_group::CapacityTier::Free,
},
EventMessage::ChannelGroupExhausted { target_id: 1, group_vid: 10 },
EventMessage::ChannelGroupMemberSwitched {
target_id: 1,
group_vid: 10,
from_member_vid: 11,
to_member_vid: 12,
reason: shared::model::channel_group::SwitchReason::UpstreamEof,
},
EventMessage::ChannelGroupCircuitChanged {
input_name: "test".into(),
input_stream_id: "1".into(),
state: shared::model::channel_group::CircuitState::Closed,
},
];
assert_eq!(samples.len(), EventKind::ALL.len(), "add the new variant to this list");
samples
@@ -103,6 +103,7 @@ fn runtime_config_to_dto(config: &Config) -> ConfigDto {
proxy: config.proxy.as_ref().map(shared::model::ProxyConfigDto::from),
ipcheck: config.ipcheck.as_ref().map(shared::model::IpCheckConfigDto::from),
library: config.library.as_ref().map(library_config_to_dto),
channel_groups: (!config.channel_groups.is_default()).then(|| config.channel_groups.clone()),
}
}
+3 -1
View File
@@ -15,7 +15,7 @@ use shared::{
DEFAULT_RECORDING_DIR, DEFAULT_STORAGE_DIR, DEFAULT_STORAGE_TEMP_DIR, DEFAULT_USER_CONFIG_DIR,
},
error::TuliproxError,
model::{ConfigDto, GeoIpUnavailablePolicy, HdHomeRunDeviceOverview},
model::{ChannelGroupsConfig, ConfigDto, GeoIpUnavailablePolicy, HdHomeRunDeviceOverview},
utils::set_sanitize_sensitive_info,
};
use std::{
@@ -115,6 +115,7 @@ pub struct Config {
pub proxy: Option<ProxyConfig>,
pub ipcheck: Option<IpCheckConfig>,
pub library: Option<LibraryConfig>,
pub channel_groups: ChannelGroupsConfig,
}
/// Resolve a configured recording directory the way `Config::prepare` does:
@@ -367,6 +368,7 @@ impl From<&ConfigDto> for Config {
proxy: dto.proxy.as_ref().map(Into::into),
ipcheck: dto.ipcheck.as_ref().map(Into::into),
library: dto.library.as_ref().map(Into::into),
channel_groups: dto.channel_groups.clone().unwrap_or_default(),
}
}
}
+7 -3
View File
@@ -9,9 +9,9 @@ use shared::{
apply_flags, create_bitset,
foundation::{Filter, ValueProvider},
model::{
ConfigTargetDto, ConfigTargetOptions, DeduplicateConfig, HdHomeRunTargetOutputDto, M3uTargetOutputDto,
MappingStage, PlaylistItemType, ProcessingOrder, StrmExportStyle, StrmTargetOutputDto, TargetOutputDto,
TargetType, TraktConfigDto, WatchContext, WatchContextSet, XtreamTargetOutputDto,
ChannelGroupsTargetConfig, ConfigTargetDto, ConfigTargetOptions, DeduplicateConfig, HdHomeRunTargetOutputDto,
M3uTargetOutputDto, MappingStage, PlaylistItemType, ProcessingOrder, StrmExportStyle, StrmTargetOutputDto,
TargetOutputDto, TargetType, TraktConfigDto, WatchContext, WatchContextSet, XtreamTargetOutputDto,
},
};
use std::{borrow::Cow, sync::Arc};
@@ -31,6 +31,8 @@ pub struct TargetExecutionPlan {
pub transform_stages: Vec<TransformStage>,
pub pre_transform_identity_dedup: bool,
pub post_merge_content_dedup: Option<DeduplicateConfig>,
/// The target's channel group switch and overrides; resolved against `config.yml` when used.
pub channel_groups: Option<ChannelGroupsTargetConfig>,
}
impl Default for TargetExecutionPlan {
@@ -39,6 +41,7 @@ impl Default for TargetExecutionPlan {
transform_stages: vec![TransformStage::Filter, TransformStage::Rename, TransformStage::Map],
pre_transform_identity_dedup: false,
post_merge_content_dedup: None,
channel_groups: None,
}
}
}
@@ -57,6 +60,7 @@ impl TargetExecutionPlan {
transform_stages,
pre_transform_identity_dedup: dto.options.as_ref().is_some_and(|options| options.remove_duplicates),
post_merge_content_dedup: dto.options.as_ref().and_then(|options| options.deduplicate),
channel_groups: dto.options.as_ref().and_then(|options| options.channel_groups).filter(|cfg| cfg.enabled),
}
}
}
+32
View File
@@ -534,6 +534,30 @@ impl ProviderConfig {
true
}
pub fn probe_capacity(&self, grace: bool, grace_period_timeout_secs: u64) -> ProviderConfigAllocation {
let guard = self.read_connection();
if self.is_account_blocked_with_connection(&guard) {
return ProviderConfigAllocation::Exhausted;
}
if self.max_connections == 0 {
return ProviderConfigAllocation::Available;
}
let connections = guard.current_connections;
if connections < self.max_connections {
return ProviderConfigAllocation::Available;
}
if !grace || connections > self.max_connections {
return ProviderConfigAllocation::Exhausted;
}
let now = Instant::now();
if Self::grace_is_active(&guard, now, grace_period_timeout_secs) {
return ProviderConfigAllocation::Exhausted;
}
ProviderConfigAllocation::GracePeriod
}
pub fn release(&self) {
let mut guard = self.write_connection();
// Releasing while over capacity ends the single outstanding grace grant.
@@ -575,6 +599,14 @@ impl ProviderConfigWrapper {
}
}
pub fn probe_capacity(&self, grace: bool, grace_period_timeout_secs: u64) -> ProviderAllocation {
match self.inner.probe_capacity(grace, grace_period_timeout_secs) {
ProviderConfigAllocation::Available => ProviderAllocation::new_available(Arc::clone(&self.inner)),
ProviderConfigAllocation::GracePeriod => ProviderAllocation::new_grace_period(Arc::clone(&self.inner)),
ProviderConfigAllocation::Exhausted => ProviderAllocation::Exhausted,
}
}
pub fn get_next(&self, grace: bool, grace_period_timeout_secs: u64) -> Option<Arc<ProviderConfig>> {
if self.inner.get_next(grace, grace_period_timeout_secs) {
return Some(Arc::clone(&self.inner));
+1
View File
@@ -103,6 +103,7 @@ impl ProviderErrorKind {
| ErrorKind::ConfigTarget
| ErrorKind::ConfigWebUi
| ErrorKind::TargetBouquet
| ErrorKind::ChannelGroup
| ErrorKind::UrlParse
| ErrorKind::FilterParse
| ErrorKind::RegexCompile => Self::Config,
@@ -538,6 +538,8 @@ impl InputWorker {
Self::apply_pending_generic_probe_base(collector, cluster, provider_id, &mut item);
let raw_video = metadata.raw_video.clone();
let raw_audio = metadata.raw_audio.clone();
update_properties(
&mut item.additional_properties,
*item_type,
@@ -549,7 +551,17 @@ impl InputWorker {
);
match item.additional_properties {
Some(StreamProperties::Live(props)) => collector.add_live(provider_id.into(), *props),
Some(StreamProperties::Live(props)) => {
let bitrate_kbps = props.bitrate / 1000;
ctx.events.observe_stream_probe(
Arc::clone(&input.name),
Arc::from(provider_id.to_string()),
raw_video.as_deref(),
raw_audio.as_deref(),
bitrate_kbps,
);
collector.add_live(provider_id.into(), *props);
}
Some(StreamProperties::Video(props)) => collector.add_vod(provider_id.into(), *props),
_ => {}
}
+4
View File
@@ -26,3 +26,7 @@ sha2.workspace = true
thiserror.workspace = true
tokio = { workspace = true, features = ["io-util"] }
zeroize.workspace = true
[dev-dependencies]
tokio = { workspace = true, features = ["macros", "rt-multi-thread"] }
+2
View File
@@ -10,5 +10,7 @@
//! layers parse live ones. Nothing here may name `api`, `AppState` or a
//! repository.
pub mod packet;
pub mod splice;
pub mod transport_stream_buffer;
pub mod ts_inspector;
+255
View File
@@ -0,0 +1,255 @@
//! MPEG-TS packet basics shared by the buffer, the inspector and the splicer: sizes, sync search,
//! header fields, PSI sections and keyframe detection.
use bytes::Bytes;
pub const TS_PACKET_SIZE: usize = 188;
pub const TS_SYNC_BYTE: u8 = 0x47;
pub(crate) const NULL_PID: u16 = 0x1FFF;
/// PCR flag in the adaptation field flags.
pub(crate) const ADAPTATION_FIELD_FLAG_PCR: u8 = 0x10;
const NULL_PID_HIGH: u8 = 0x1F;
const NULL_PID_LOW: u8 = 0xFF;
const fn null_packet() -> [u8; TS_PACKET_SIZE] {
let mut packet = [0xFF; TS_PACKET_SIZE];
packet[0] = TS_SYNC_BYTE;
packet[1] = NULL_PID_HIGH;
packet[2] = NULL_PID_LOW;
packet[3] = 0x10;
packet
}
/// A null packet (PID 0x1FFF): decoders discard it, players and proxies see bytes flowing.
pub static TS_NULL_PACKET: [u8; TS_PACKET_SIZE] = null_packet();
pub fn ts_null_packet_bytes() -> Bytes { Bytes::from_static(&TS_NULL_PACKET) }
/// First offset in `offsets` with a sync byte at it and at the next `confirmations - 1` packet
/// starts. Packet starts past the end of `buf` count as matching only with `partial_tail`.
#[must_use]
pub(crate) fn find_sync(
buf: &[u8],
offsets: impl IntoIterator<Item = usize>,
confirmations: usize,
partial_tail: bool,
) -> Option<usize> {
offsets.into_iter().take_while(|offset| *offset < buf.len()).find(|offset| {
(0..confirmations).all(|index| match buf.get(offset + index * TS_PACKET_SIZE) {
Some(byte) => *byte == TS_SYNC_BYTE,
None => partial_tail,
})
})
}
/// Finds the first offset `o < 188` where `0x47` appears at `o`, `o + 188`, `o + 376`.
/// With fewer than 564 bytes at EOF, all available positions must match.
#[must_use]
pub fn ts_sync_offset(buf: &[u8]) -> Option<usize> { find_sync(buf, 0..TS_PACKET_SIZE, 3, true) }
/// A 13-bit PID from the two bytes that carry it (packet header, PAT and PMT entries).
#[must_use]
pub(crate) fn pid_from_bytes(high: u8, low: u8) -> u16 { (u16::from(high & 0x1F) << 8) | u16::from(low) }
#[must_use]
pub(crate) fn ts_packet_pid(packet: &[u8]) -> u16 { pid_from_bytes(packet[1], packet[2]) }
/// CRC-32/MPEG-2 of PSI sections: polynomial 0x04C11DB7, initial value 0xFFFFFFFF, no reflection,
/// no final xor. A section including its CRC yields 0.
#[must_use]
pub fn mpeg2_crc32(data: &[u8]) -> u32 {
let mut crc = 0xFFFF_FFFF_u32;
for &byte in data {
crc ^= u32::from(byte) << 24;
for _ in 0..8 {
crc = if crc & 0x8000_0000 == 0 { crc << 1 } else { (crc << 1) ^ 0x04C1_1DB7 };
}
}
crc
}
/// Read access to the header fields of one 188-byte packet.
#[derive(Clone, Copy)]
pub(crate) struct TsPacket<'a>(pub(crate) &'a [u8]);
impl<'a> TsPacket<'a> {
pub(crate) fn pid(self) -> u16 { ts_packet_pid(self.0) }
pub(crate) fn payload_unit_start(self) -> bool { self.0[1] & 0x40 != 0 }
/// `adaptation_field_control`: 0b01 payload only, 0b10 adaptation field only, 0b11 both.
pub(crate) fn adaptation_field_control(self) -> u8 { (self.0[3] >> 4) & 0b11 }
pub(crate) fn has_payload(self) -> bool { self.adaptation_field_control() & 0b01 != 0 }
/// Length of the adaptation field (after its length byte), `None` without one.
pub(crate) fn adaptation_len(self) -> Option<usize> { (self.0[3] & 0x20 != 0).then(|| usize::from(self.0[4])) }
/// Whether the adaptation field carries a PCR (flag set and room for its 6 bytes).
pub(crate) fn has_pcr(self) -> bool {
self.adaptation_len().is_some_and(|len| len >= 7 && self.0[5] & ADAPTATION_FIELD_FLAG_PCR != 0)
}
pub(crate) fn random_access(self) -> bool {
self.adaptation_len().is_some_and(|len| len >= 1 && self.0[5] & 0x40 != 0)
}
pub(crate) fn payload(self) -> &'a [u8] {
if self.0[3] & 0x10 == 0 {
return &[];
}
let start = self.adaptation_len().map_or(4, |len| 5 + len);
self.0.get(start..).unwrap_or(&[])
}
pub(crate) fn payload_offset(self) -> Option<usize> {
(self.0[3] & 0x10 != 0)
.then(|| self.adaptation_len().map_or(4, |len| 5 + len))
.filter(|start| *start < TS_PACKET_SIZE)
}
}
/// Bounds of the PSI section that starts in `payload`, when it lies completely inside it.
pub(crate) fn single_packet_section(payload: &[u8]) -> Option<std::ops::Range<usize>> {
let pointer = usize::from(*payload.first()?);
let start = 1 + pointer;
let header = payload.get(start..start + 3)?;
let length = 3 + ((usize::from(header[1] & 0x0F) << 8) | usize::from(header[2]));
(length >= 12 && start + length <= payload.len()).then_some(start..start + length)
}
/// `version_number` of a long-form section.
pub(crate) fn section_version(section: &[u8]) -> u8 { (section[5] >> 1) & 0x1F }
/// Sets `version_number` of a long-form section and recomputes its CRC.
pub(crate) fn rewrite_section_version(section: &mut [u8], version: u8) {
section[5] = (section[5] & 0xC1) | ((version & 0x1F) << 1);
let crc_at = section.len() - 4;
let crc = mpeg2_crc32(&section[..crc_at]);
section[crc_at..].copy_from_slice(&crc.to_be_bytes());
}
/// Most elementary-stream bytes of a PES searched for a keyframe.
pub(crate) const KEYFRAME_SCAN_BYTES: usize = 512;
/// Whether the PES payload starting in a packet begins an intra-coded picture, judged from the
/// NAL units (H.264, HEVC) or sequence header (MPEG-2 video) in this packet's elementary data.
#[must_use]
pub fn is_idr_start(stream_type: u8, pes_payload: &[u8]) -> bool {
pes_elementary_data(pes_payload).is_some_and(|data| es_has_keyframe(stream_type, data))
}
/// The elementary data after the PES header in the payload of a PES start packet.
pub(crate) fn pes_elementary_data(pes_payload: &[u8]) -> Option<&[u8]> {
if pes_payload.len() < 9 || pes_payload[..3] != [0, 0, 1] {
return None;
}
pes_payload.get(9 + usize::from(pes_payload[8])..)
}
/// Whether `data`, the start of a PES's elementary data, opens an intra-coded picture: an IDR
/// slice or the SPS that opens an IDR access unit (H.264), IDR/CRA pictures or the VPS/SPS that
/// open them (HEVC), a sequence header (MPEG-2).
pub(crate) fn es_has_keyframe(stream_type: u8, data: &[u8]) -> bool {
let data = &data[..data.len().min(KEYFRAME_SCAN_BYTES)];
let mut index = 0;
while index + 3 < data.len() {
if data[index..index + 3] == [0, 0, 1] {
let unit = data[index + 3];
let keyframe = match stream_type {
0x1B => matches!(unit & 0x1F, 5 | 7),
0x24 => matches!((unit >> 1) & 0x3F, 19..=21 | 32 | 33),
0x01 | 0x02 => unit == 0xB3,
_ => false,
};
if keyframe {
return true;
}
index += 3;
} else {
index += 1;
}
}
false
}
pub(crate) fn is_video_stream_type(stream_type: u8) -> bool { matches!(stream_type, 0x01 | 0x02 | 0x1B | 0x24) }
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn ts_sync_offset_empty() {
assert_eq!(ts_sync_offset(&[]), None);
}
#[test]
fn ts_sync_offset_aligned() {
let mut buf = vec![0u8; 564];
buf[0] = 0x47;
buf[188] = 0x47;
buf[376] = 0x47;
assert_eq!(ts_sync_offset(&buf), Some(0));
}
#[test]
fn ts_sync_offset_with_garbage_prefix() {
let mut buf = vec![0u8; 17];
let mut ts = vec![0u8; 564];
ts[0] = 0x47;
ts[188] = 0x47;
ts[376] = 0x47;
buf.extend(ts);
assert_eq!(ts_sync_offset(&buf), Some(17));
}
#[test]
fn ts_sync_offset_short_buffer_at_eof() {
let mut buf = vec![0u8; 200];
buf[10] = 0x47;
buf[198] = 0x47;
assert_eq!(ts_sync_offset(&buf), Some(10));
}
#[test]
fn ts_sync_offset_corrupt_packet() {
let mut buf = vec![0u8; 564];
buf[0] = 0x47;
buf[188] = 0x48; // corrupt
buf[376] = 0x47;
assert_eq!(ts_sync_offset(&buf), None);
}
fn with_sync_at(len: usize, positions: &[usize]) -> Vec<u8> {
let mut buf = vec![0u8; len];
for position in positions {
buf[*position] = 0x47;
}
buf
}
#[test]
fn find_sync_without_partial_tail_needs_every_confirmation() {
let buf = with_sync_at(300, &[0, 188]);
assert_eq!(find_sync(&buf, 0..=187, 3, false), None);
assert_eq!(find_sync(&buf, 0..=187, 3, true), Some(0));
}
#[test]
fn find_sync_searches_the_given_offsets() {
let buf = with_sync_at(400 + 3 * TS_PACKET_SIZE, &[400, 588, 776]);
assert_eq!(find_sync(&buf, 0..=500, 3, false), Some(400));
assert_eq!(find_sync(&buf, 0..=399, 3, false), None);
}
#[test]
fn packet_view_reads_pid_and_pcr_flag() {
let mut packet = vec![0x47, 0x41, 0x00, 0x30, 7, 0x10];
packet.resize(TS_PACKET_SIZE, 0);
assert_eq!(ts_packet_pid(&packet), 0x100);
assert!(TsPacket(&packet).has_pcr());
packet[4] = 6;
assert!(!TsPacket(&packet).has_pcr(), "a PCR needs 7 adaptation bytes");
}
}
+429
View File
@@ -0,0 +1,429 @@
use crate::packet::{
es_has_keyframe, is_idr_start, is_video_stream_type, pes_elementary_data, pid_from_bytes, rewrite_section_version,
section_version, single_packet_section, ts_sync_offset, TsPacket, KEYFRAME_SCAN_BYTES, TS_PACKET_SIZE,
TS_SYNC_BYTE,
};
use bytes::{Buf, Bytes, BytesMut};
use std::{
collections::HashMap,
time::{Duration, Instant},
};
const PAT_PID: u16 = 0;
const PAT_TABLE_ID: u8 = 0x00;
const PMT_TABLE_ID: u8 = 0x02;
/// A change to an emitted packet, applied after the chunk was observed.
enum PacketEdit {
Discontinuity { offset: usize },
Version { payload: std::ops::Range<usize>, version: u8 },
}
pub enum SpliceOutput {
/// Nothing to send yet.
Hold,
/// Whole TS packets to send.
Emit(Bytes),
/// Whole TS packets in two pieces, sent in order: the packet completed from carried-over
/// bytes and the aligned rest of the chunk, which is passed on without copying.
EmitTwo(Bytes, Bytes),
}
impl SpliceOutput {
fn of(first: Option<Bytes>, second: Option<Bytes>) -> Self {
match (first.filter(|bytes| !bytes.is_empty()), second.filter(|bytes| !bytes.is_empty())) {
(Some(first), Some(second)) => Self::EmitTwo(first, second),
(Some(single), None) | (None, Some(single)) => Self::Emit(single),
(None, None) => Self::Hold,
}
}
}
/// Set of 13-bit PIDs.
struct PidSet(Box<[u64; 128]>);
impl PidSet {
fn new() -> Self { Self(Box::new([0; 128])) }
fn clear(&mut self) { self.0.fill(0); }
/// Adds `pid`; true when it was not in the set.
fn insert(&mut self, pid: u16) -> bool {
let (word, bit) = (usize::from(pid >> 6) & 127, 1u64 << (pid & 63));
let added = self.0[word] & bit == 0;
self.0[word] |= bit;
added
}
}
/// Program structure of the member currently read, learned from its PAT and PMT.
#[derive(Default)]
struct Programs {
/// PMT PID → program number.
pmt_pids: HashMap<u16, u16>,
/// First video elementary stream: PID and stream type.
video: Option<(u16, u8)>,
/// CRC of the PAT the program map was built from.
pat_crc: Option<u32>,
}
/// Version rewrite of one member: per table, the offset added to the source version.
#[derive(Default)]
struct VersionOffsets {
pat: Option<u8>,
pmt: HashMap<u16, u8>,
}
struct Aligning {
started: Instant,
buffer: BytesMut,
first_pat: Option<usize>,
last_pat: Option<usize>,
pmt_after_last_pat: bool,
/// Video PES whose start packet showed no keyframe yet: the PAT before its start and the
/// elementary data read so far, searched on as its next packets arrive.
open_pes: Option<(usize, Vec<u8>)>,
}
/// Aligns provider bytes to whole TS packets and splices a new member into the output: after
/// `begin_member` the new member is held back until it can start at a PAT/PMT followed by a
/// keyframe, its PSI versions continue the ones already emitted, and the first packet per PID
/// announces the discontinuity.
pub struct TsSpliceAligner {
carry: BytesMut,
budget_bytes: usize,
budget: Duration,
programs: Programs,
aligning: Option<Aligning>,
/// Number of members started; the first one is never rewritten.
members: u32,
offsets: VersionOffsets,
emitted_pat_version: Option<u8>,
emitted_pmt_versions: HashMap<u16, u8>,
discontinuity_seen: PidSet,
}
impl TsSpliceAligner {
#[must_use]
pub fn new(budget_bytes: usize, budget: Duration) -> Self {
Self {
carry: BytesMut::new(),
budget_bytes,
budget,
programs: Programs::default(),
aligning: None,
members: 1,
offsets: VersionOffsets::default(),
emitted_pat_version: None,
emitted_pmt_versions: HashMap::new(),
discontinuity_seen: PidSet::new(),
}
}
/// Starts the next member: its bytes are held until the splice point.
pub fn begin_member(&mut self, now: Instant) {
self.carry.clear();
self.programs = Programs::default();
self.offsets = VersionOffsets::default();
self.discontinuity_seen.clear();
self.members += 1;
self.aligning = Some(Aligning {
started: now,
buffer: BytesMut::new(),
first_pat: None,
last_pat: None,
pmt_after_last_pat: false,
open_pes: None,
});
}
#[must_use]
pub fn is_aligning(&self) -> bool { self.aligning.is_some() }
pub fn push(&mut self, chunk: Bytes, now: Instant) -> SpliceOutput {
let (head, body) = self.whole_packets(chunk);
let head = head.and_then(|head| self.emit_packets(head, now));
let body = self.emit_packets(body, now);
SpliceOutput::of(head, body)
}
fn emit_packets(&mut self, packets: Bytes, now: Instant) -> Option<Bytes> {
if self.aligning.is_some() {
return self.push_aligning(&packets, now);
}
(!packets.is_empty()).then(|| self.process_emitted(packets))
}
/// Whole packets of `chunk` plus what was carried over: the packet completed from the carry
/// (copied, at most 187 bytes from the chunk) and the aligned rest of the chunk (not copied).
fn whole_packets(&mut self, mut chunk: Bytes) -> (Option<Bytes>, Bytes) {
let mut head = None;
if !self.carry.is_empty() {
if self.carry[0] != TS_SYNC_BYTE {
return (None, self.resync(&chunk));
}
let need = TS_PACKET_SIZE - self.carry.len();
if chunk.len() < need {
self.carry.extend_from_slice(&chunk);
return (None, Bytes::new());
}
self.carry.extend_from_slice(&chunk[..need]);
chunk.advance(need);
head = Some(self.carry.split().freeze());
}
if chunk.first().is_some_and(|byte| *byte != TS_SYNC_BYTE) {
return (head, self.resync(&chunk));
}
let whole = chunk.len() / TS_PACKET_SIZE * TS_PACKET_SIZE;
self.carry.extend_from_slice(&chunk[whole..]);
chunk.truncate(whole);
(head, chunk)
}
/// Copying path for input without packet sync at its start: finds the sync and returns the
/// whole packets after it.
fn resync(&mut self, chunk: &[u8]) -> Bytes {
self.carry.extend_from_slice(chunk);
if self.carry.first() != Some(&TS_SYNC_BYTE) {
let Some(offset) = ts_sync_offset(&self.carry) else {
let keep = self.carry.len().min(TS_PACKET_SIZE - 1);
let drop = self.carry.len() - keep;
self.carry.advance(drop);
return Bytes::new();
};
self.carry.advance(offset);
}
let whole = self.carry.len() / TS_PACKET_SIZE * TS_PACKET_SIZE;
self.carry.split_to(whole).freeze()
}
fn push_aligning(&mut self, packets: &[u8], now: Instant) -> Option<Bytes> {
let mut aligning = self.aligning.take()?;
let mut start = None;
for packet in packets.as_chunks::<TS_PACKET_SIZE>().0 {
let position = aligning.buffer.len();
aligning.buffer.extend_from_slice(packet);
if start.is_some() {
continue;
}
let view = TsPacket(packet);
if view.pid() == PAT_PID && view.payload_unit_start() {
aligning.first_pat.get_or_insert(position);
aligning.last_pat = Some(position);
aligning.pmt_after_last_pat = false;
}
if self.observe(view) && aligning.last_pat.is_some() {
aligning.pmt_after_last_pat = true;
if self.programs.video.is_none() {
start = aligning.last_pat;
continue;
}
}
if !aligning.pmt_after_last_pat {
continue;
}
if self.is_keyframe(view) {
start = aligning.last_pat;
} else if let Some(pat) = self.continue_open_pes(&mut aligning, view) {
start = Some(pat);
}
}
let over_budget =
aligning.buffer.len() > self.budget_bytes || now.saturating_duration_since(aligning.started) >= self.budget;
let start = match start {
Some(start) => start,
None if over_budget => aligning.first_pat.unwrap_or(0),
None => {
self.aligning = Some(aligning);
return None;
}
};
// Dropping the skipped head keeps the buffer the only owner, so the PSI rewrite below
// edits it in place instead of copying it.
let mut buffer = aligning.buffer;
buffer.advance(start);
let output = buffer.freeze();
// The emitted part is observed again from its PAT on for versions and discontinuity.
self.programs = Programs::default();
Some(self.process_emitted(output))
}
/// Follows the elementary data of a video PES across its packets; returns the PAT before the
/// PES start once a keyframe shows up within its first bytes.
fn continue_open_pes(&self, aligning: &mut Aligning, view: TsPacket<'_>) -> Option<usize> {
let (pid, stream_type) = self.programs.video?;
if view.pid() != pid {
return None;
}
if view.payload_unit_start() {
// A new PES without keyframe in its start packet: its next packets may hold it.
aligning.open_pes = aligning.last_pat.zip(pes_elementary_data(view.payload()).map(<[u8]>::to_vec));
return None;
}
let (pat, data) = aligning.open_pes.as_mut()?;
let room = KEYFRAME_SCAN_BYTES.saturating_sub(data.len());
let payload = view.payload();
data.extend_from_slice(&payload[..payload.len().min(room)]);
if es_has_keyframe(stream_type, data) {
let pat = *pat;
aligning.open_pes = None;
return Some(pat);
}
if data.len() >= KEYFRAME_SCAN_BYTES {
aligning.open_pes = None;
}
None
}
fn is_keyframe(&self, view: TsPacket<'_>) -> bool {
let Some((pid, stream_type)) = self.programs.video else {
return false;
};
view.pid() == pid
&& view.payload_unit_start()
&& (view.random_access() || is_idr_start(stream_type, view.payload()))
}
/// Learns PAT/PMT structure from `view`; returns true when it was a PMT of a known program.
fn observe(&mut self, view: TsPacket<'_>) -> bool {
if !view.payload_unit_start() {
return false;
}
let pid = view.pid();
let is_pmt = self.programs.pmt_pids.contains_key(&pid);
if pid != PAT_PID && !is_pmt {
return false;
}
let payload = view.payload();
let Some(range) = single_packet_section(payload) else {
return false;
};
let section = &payload[range];
let entries = &section[8..section.len() - 4];
if pid == PAT_PID && section[0] == PAT_TABLE_ID {
// A repeated PAT (several per second) leaves the program map as it is.
let crc = u32::from_be_bytes([
section[section.len() - 4],
section[section.len() - 3],
section[section.len() - 2],
section[section.len() - 1],
]);
if self.programs.pat_crc == Some(crc) {
return false;
}
self.programs.pat_crc = Some(crc);
self.programs.pmt_pids = entries
.as_chunks::<4>()
.0
.iter()
.filter_map(|entry| {
let program = u16::from_be_bytes([entry[0], entry[1]]);
let pmt_pid = pid_from_bytes(entry[2], entry[3]);
(program != 0).then_some((pmt_pid, program))
})
.collect();
return false;
}
if is_pmt && section[0] == PMT_TABLE_ID && entries.len() >= 4 {
let info_len = (usize::from(entries[2] & 0x0F) << 8) | usize::from(entries[3]);
let mut streams = entries.get(4 + info_len..).unwrap_or(&[]);
self.programs.video = None;
while streams.len() >= 5 {
let stream_type = streams[0];
let es_pid = pid_from_bytes(streams[1], streams[2]);
let es_info_len = (usize::from(streams[3] & 0x0F) << 8) | usize::from(streams[4]);
if self.programs.video.is_none() && is_video_stream_type(stream_type) {
self.programs.video = Some((es_pid, stream_type));
}
streams = streams.get(5 + es_info_len..).unwrap_or(&[]);
}
return true;
}
false
}
/// Observes packets about to be emitted and applies the version rewrite and discontinuity
/// flags of a member after the first. Copies only when a packet changes.
fn process_emitted(&mut self, packets: Bytes) -> Bytes {
let rewrite = self.members > 1;
let mut edits: Vec<PacketEdit> = Vec::new();
for (index, packet) in packets.as_chunks::<TS_PACKET_SIZE>().0.iter().enumerate() {
let view = TsPacket(packet);
self.observe(view);
let offset = index * TS_PACKET_SIZE;
let pid = view.pid();
if rewrite && self.discontinuity_seen.insert(pid) && view.adaptation_len().is_some_and(|len| len >= 1) {
edits.push(PacketEdit::Discontinuity { offset });
}
if let Some(version) = self.emitted_version(view, rewrite) {
if let Some(payload_offset) = view.payload_offset() {
edits.push(PacketEdit::Version {
payload: offset + payload_offset..offset + TS_PACKET_SIZE,
version,
});
}
}
}
if edits.is_empty() {
return packets;
}
// In place when the chunk is ours alone, a copy only when it is shared.
let mut buffer = packets.try_into_mut().unwrap_or_else(|shared| BytesMut::from(&shared[..]));
for edit in edits {
match edit {
PacketEdit::Discontinuity { offset } => buffer[offset + 5] |= 0x80,
PacketEdit::Version { payload, version } => {
let payload = &mut buffer[payload];
if let Some(range) = single_packet_section(payload) {
rewrite_section_version(&mut payload[range], version);
}
}
}
}
buffer.freeze()
}
/// Records the version a PSI section in `view` is emitted with; returns it when it differs from
/// the source version and the packet has to be rewritten.
fn emitted_version(&mut self, view: TsPacket<'_>, rewrite: bool) -> Option<u8> {
if !view.payload_unit_start() {
return None;
}
let pid = view.pid();
let program = if pid == PAT_PID { None } else { Some(*self.programs.pmt_pids.get(&pid)?) };
let payload = view.payload();
let range = single_packet_section(payload)?;
let section = &payload[range];
let expected_table = if program.is_some() { PMT_TABLE_ID } else { PAT_TABLE_ID };
if section[0] != expected_table {
return None;
}
let source = section_version(section);
let (offset, last_emitted) = match program {
None => (&mut self.offsets.pat, &mut self.emitted_pat_version),
Some(program) => {
let last = self.emitted_pmt_versions.get(&program).copied();
let offset = self.offsets.pmt.entry(program).or_insert_with(|| {
if rewrite {
last.map_or(0, |last| (last + 1 + 32 - source) % 32)
} else {
0
}
});
let version = (source + *offset) % 32;
self.emitted_pmt_versions.insert(program, version);
return (version != source).then_some(version);
}
};
let offset = *offset.get_or_insert_with(|| {
if rewrite {
last_emitted.map_or(0, |last| (last + 1 + 32 - source) % 32)
} else {
0
}
});
let version = (source + offset) % 32;
*last_emitted = Some(version);
(version != source).then_some(version)
}
}
+12
View File
@@ -0,0 +1,12 @@
//! Splicing of MPEG-TS from several sources into one continuous output.
mod aligner;
mod realtime;
pub use aligner::{SpliceOutput, TsSpliceAligner};
pub use realtime::RealtimeMeter;
#[cfg(test)]
mod realtime_tests;
#[cfg(test)]
mod tests;
+105
View File
@@ -0,0 +1,105 @@
use crate::{
packet::{TsPacket, TS_PACKET_SIZE},
transport_stream_buffer::decode_pcr,
};
use std::time::{Duration, Instant};
const PCR_HZ: u64 = 27_000_000;
/// PCR wraps at 2^33 * 300.
const PCR_MODULUS: u64 = (1 << 33) * 300;
const BUCKET: Duration = Duration::from_secs(1);
/// A bucket in which the consumer waited less than this never counts as an underrun: the
/// provider delivered faster than the consumer read.
const MIN_BUCKET_WAIT: Duration = Duration::from_millis(100);
/// Detects a provider that delivers media slower than real time, from the PCR advance against the
/// time the consumer waited for data.
pub struct RealtimeMeter {
ratio_pct: u8,
window: Duration,
pcr_pid: Option<u16>,
last_pcr: Option<u64>,
bucket_start: Option<Instant>,
bucket_media: Duration,
bucket_waited: Duration,
streak_start: Option<Instant>,
}
impl RealtimeMeter {
#[must_use]
pub fn new(ratio_pct: u8, window: Duration) -> Self {
Self {
ratio_pct,
window,
pcr_pid: None,
last_pcr: None,
bucket_start: None,
bucket_media: Duration::ZERO,
bucket_waited: Duration::ZERO,
streak_start: None,
}
}
/// `packets` are whole TS packets just delivered, `waited` the time the consumer waited for
/// them. Returns true once media arrived slower than `ratio_pct` of real time for `window`.
pub fn observe(&mut self, packets: &[u8], waited: Duration, now: Instant) -> bool {
if self.ratio_pct == 0 {
return false;
}
for packet in packets.as_chunks::<TS_PACKET_SIZE>().0 {
self.observe_pcr(TsPacket(packet), now);
}
if self.pcr_pid.is_none() {
return false;
}
self.bucket_waited += waited;
let bucket_start = *self.bucket_start.get_or_insert(now);
if now.saturating_duration_since(bucket_start) >= BUCKET {
let slow = self.bucket_waited >= MIN_BUCKET_WAIT
&& self.bucket_media.as_micros() * 100 < u128::from(self.ratio_pct) * self.bucket_waited.as_micros();
if slow {
self.streak_start.get_or_insert(bucket_start);
} else {
self.streak_start = None;
}
self.bucket_start = Some(now);
self.bucket_media = Duration::ZERO;
self.bucket_waited = Duration::ZERO;
}
self.streak_start.is_some_and(|start| now.saturating_duration_since(start) >= self.window)
}
/// Starts measuring a new member: its PCR may run on another PID than the previous one's.
pub fn begin_member(&mut self, now: Instant) {
self.pcr_pid = None;
self.reset(now);
}
pub fn reset(&mut self, now: Instant) {
self.last_pcr = None;
self.bucket_start = Some(now);
self.bucket_media = Duration::ZERO;
self.bucket_waited = Duration::ZERO;
self.streak_start = None;
}
fn observe_pcr(&mut self, packet: TsPacket<'_>, now: Instant) {
if !packet.has_pcr() {
return;
}
let pid = packet.pid();
if *self.pcr_pid.get_or_insert(pid) != pid {
return;
}
let pcr = decode_pcr(&packet.0[6..12]);
let Some(last) = self.last_pcr.replace(pcr) else { return };
let delta = (pcr + PCR_MODULUS - last) % PCR_MODULUS;
if delta > PCR_HZ {
// A jump forward or backwards is a new timeline, not delivery speed.
self.reset(now);
self.last_pcr = Some(pcr);
return;
}
self.bucket_media += Duration::from_nanos(delta * 1_000 / 27);
}
}
+115
View File
@@ -0,0 +1,115 @@
#![allow(clippy::cast_possible_truncation, reason = "test builders write PCR fields into single bytes")]
use super::RealtimeMeter;
use crate::packet::TS_PACKET_SIZE;
use std::time::{Duration, Instant};
const PCR_PID: u16 = 0x100;
const STEP: Duration = Duration::from_millis(100);
/// One packet on `pid` with a PCR of `pcr_27mhz` in its adaptation field.
fn pcr_packet(pid: u16, pcr_27mhz: u64) -> Vec<u8> {
let base = pcr_27mhz / 300;
let ext = pcr_27mhz % 300;
let mut packet = vec![0x47, (pid >> 8) as u8 & 0x1F, pid as u8, 0x20, 183, 0x10];
packet.extend([
(base >> 25) as u8,
(base >> 17) as u8,
(base >> 9) as u8,
(base >> 1) as u8,
(((base & 1) << 7) as u8) | 0x7E | ((ext >> 8) as u8),
ext as u8,
]);
packet.resize(TS_PACKET_SIZE, 0xFF);
packet
}
fn plain_packet(pid: u16) -> Vec<u8> {
let mut packet = vec![0x47, (pid >> 8) as u8 & 0x1F, pid as u8, 0x10];
packet.resize(TS_PACKET_SIZE, 0);
packet
}
/// Feeds `steps` steps of 100 ms wall time, each advancing the PCR by `media` and reporting
/// `waited`; returns the first step that reported an underrun.
fn run(
meter: &mut RealtimeMeter,
start: Instant,
pcr: &mut u64,
steps: u32,
media: Duration,
waited: Duration,
) -> Option<u32> {
let mut detected = None;
for step in 0..steps {
*pcr += u64::try_from(media.as_micros()).expect("micros") * 27;
let now = start + STEP * step;
if meter.observe(&pcr_packet(PCR_PID, *pcr), waited, now) && detected.is_none() {
detected = Some(step);
}
}
detected
}
#[test]
fn realtime_meter_detects_slow_provider() {
let mut meter = RealtimeMeter::new(90, Duration::from_secs(15));
let start = Instant::now();
let mut pcr = 27_000_000;
let detected = run(&mut meter, start, &mut pcr, 200, Duration::from_millis(70), STEP).expect("underrun detected");
assert!((150..=170).contains(&detected), "detected at step {detected}");
}
#[test]
fn realtime_meter_ignores_slow_consumer() {
let mut meter = RealtimeMeter::new(90, Duration::from_secs(15));
let mut pcr = 27_000_000;
assert_eq!(run(&mut meter, Instant::now(), &mut pcr, 300, STEP, Duration::from_millis(5)), None);
}
#[test]
fn realtime_meter_inactive_without_pcr() {
let mut meter = RealtimeMeter::new(90, Duration::from_secs(2));
let start = Instant::now();
for step in 0..100 {
assert!(!meter.observe(&plain_packet(PCR_PID), STEP, start + STEP * step));
}
}
#[test]
fn realtime_meter_resets_on_pcr_jump() {
let mut meter = RealtimeMeter::new(90, Duration::from_secs(15));
let start = Instant::now();
let mut pcr = 27_000_000;
assert_eq!(run(&mut meter, start, &mut pcr, 100, Duration::from_millis(70), STEP), None);
pcr += 27_000_000 * 5;
let resumed = start + STEP * 100;
assert_eq!(run(&mut meter, resumed, &mut pcr, 100, Duration::from_millis(70), STEP), None);
}
#[test]
fn realtime_meter_disabled_with_zero_ratio() {
let mut meter = RealtimeMeter::new(0, Duration::from_secs(1));
let mut pcr = 27_000_000;
assert_eq!(run(&mut meter, Instant::now(), &mut pcr, 100, Duration::from_millis(10), STEP), None);
}
#[test]
fn realtime_meter_follows_the_pcr_pid_of_a_new_member() {
let mut meter = RealtimeMeter::new(90, Duration::from_secs(5));
let start = Instant::now();
let mut pcr = 27_000_000;
assert_eq!(run(&mut meter, start, &mut pcr, 20, STEP, STEP), None);
// The next member carries its PCR on another PID and plays in real time.
meter.begin_member(start + STEP * 20);
let mut detected = None;
let mut member_pcr = 54_000_000_u64;
for step in 20..120 {
member_pcr += 27_000 * 100;
if meter.observe(&pcr_packet(0x200, member_pcr), STEP, start + STEP * step) {
detected = Some(step);
break;
}
}
assert_eq!(detected, None, "a real-time member is no underrun");
}
+415
View File
@@ -0,0 +1,415 @@
#![allow(clippy::cast_possible_truncation, reason = "test builders write PIDs and lengths into single bytes")]
use super::{SpliceOutput, TsSpliceAligner};
use crate::packet::{is_idr_start, mpeg2_crc32, ts_null_packet_bytes, TS_NULL_PACKET, TS_PACKET_SIZE};
use bytes::Bytes;
use std::time::{Duration, Instant};
const PMT_PID: u16 = 0x1000;
const VIDEO_PID: u16 = 0x100;
const AUDIO_PID: u16 = 0x101;
const H264: u8 = 0x1B;
const HEVC: u8 = 0x24;
const AAC: u8 = 0x0F;
/// One TS packet. With `af_flags` the adaptation field fills the room the payload leaves;
/// without, the payload is padded with 0xFF to 184 bytes.
fn ts(pid: u16, pusi: bool, af_flags: Option<u8>, payload: &[u8]) -> Vec<u8> {
let mut packet = vec![0x47, (if pusi { 0x40 } else { 0 }) | ((pid >> 8) as u8 & 0x1F), pid as u8];
if let Some(flags) = af_flags {
let af_len = 184 - 1 - payload.len();
assert!(af_len >= 1);
packet.push(0x30);
packet.push(af_len as u8);
packet.push(flags);
packet.extend(std::iter::repeat_n(0xFF, af_len - 1));
packet.extend_from_slice(payload);
} else {
packet.push(0x10);
packet.extend_from_slice(payload);
packet.resize(TS_PACKET_SIZE, 0xFF);
}
assert_eq!(packet.len(), TS_PACKET_SIZE);
packet
}
fn finish_section(mut section: Vec<u8>) -> Vec<u8> {
let length = section.len() - 3 + 4;
section[1] = 0xB0 | ((length >> 8) as u8 & 0x0F);
section[2] = length as u8;
let crc = mpeg2_crc32(&section);
section.extend_from_slice(&crc.to_be_bytes());
section
}
fn pat_section(version: u8, program: u16, pmt_pid: u16) -> Vec<u8> {
let section = vec![
0x00,
0,
0,
0x00,
0x01,
0xC1 | (version << 1),
0x00,
0x00,
(program >> 8) as u8,
program as u8,
0xE0 | (pmt_pid >> 8) as u8,
pmt_pid as u8,
];
finish_section(section)
}
fn pmt_section(version: u8, program: u16, streams: &[(u8, u16)]) -> Vec<u8> {
let pcr_pid = streams.first().map_or(0x1FFF, |(_, pid)| *pid);
let mut section = vec![
0x02,
0,
0,
(program >> 8) as u8,
program as u8,
0xC1 | (version << 1),
0x00,
0x00,
0xE0 | (pcr_pid >> 8) as u8,
pcr_pid as u8,
0xF0,
0x00,
];
for (stream_type, pid) in streams {
section.extend([*stream_type, 0xE0 | (pid >> 8) as u8, *pid as u8, 0xF0, 0x00]);
}
finish_section(section)
}
fn psi(pid: u16, section: &[u8]) -> Vec<u8> {
let mut payload = vec![0x00];
payload.extend_from_slice(section);
ts(pid, true, None, &payload)
}
fn pat(version: u8) -> Vec<u8> { psi(0, &pat_section(version, 1, PMT_PID)) }
fn pmt(version: u8, streams: &[(u8, u16)]) -> Vec<u8> { psi(PMT_PID, &pmt_section(version, 1, streams)) }
fn video_pmt(version: u8) -> Vec<u8> { pmt(version, &[(H264, VIDEO_PID), (AAC, AUDIO_PID)]) }
/// PES start on `pid` whose elementary data holds NAL units of the given header bytes.
fn pes(pid: u16, rai: bool, af: bool, nal_headers: &[&[u8]]) -> Vec<u8> {
let mut payload = vec![0x00, 0x00, 0x01, 0xE0, 0x00, 0x00, 0x80, 0x80, 0x05, 0x21, 0x00, 0x01, 0x00, 0x01];
for header in nal_headers {
payload.extend_from_slice(&[0x00, 0x00, 0x00, 0x01]);
payload.extend_from_slice(header);
payload.extend_from_slice(&[0x88, 0x84]);
}
let flags = if rai {
Some(0x40)
} else if af {
Some(0x00)
} else {
None
};
ts(pid, true, flags, &payload)
}
fn h264(nal_type: u8) -> Vec<u8> { vec![0x60 | nal_type] }
fn hevc(nal_type: u8) -> Vec<u8> { vec![nal_type << 1, 0x01] }
fn video_cont(af: bool) -> Vec<u8> { ts(VIDEO_PID, false, af.then_some(0x00), &[0x11; 100]) }
fn join(packets: &[Vec<u8>]) -> Bytes { Bytes::from(packets.concat()) }
fn emitted(output: SpliceOutput) -> Bytes {
match output {
SpliceOutput::Emit(bytes) => bytes,
SpliceOutput::EmitTwo(first, second) => Bytes::from([first, second].concat()),
SpliceOutput::Hold => panic!("expected emitted packets"),
}
}
fn packets(bytes: &[u8]) -> Vec<&[u8]> {
assert_eq!(bytes.len() % TS_PACKET_SIZE, 0, "output is whole packets");
bytes.chunks(TS_PACKET_SIZE).collect()
}
fn pid_of(packet: &[u8]) -> u16 { (u16::from(packet[1] & 0x1F) << 8) | u16::from(packet[2]) }
/// Version of the single-packet section in `packet`, after checking its CRC.
fn section_version(packet: &[u8]) -> u8 {
let payload_start = if packet[3] & 0x20 != 0 { 5 + usize::from(packet[4]) } else { 4 };
let section = &packet[payload_start + 1 + usize::from(packet[payload_start])..];
let length = 3 + ((usize::from(section[1] & 0x0F) << 8) | usize::from(section[2]));
assert_eq!(mpeg2_crc32(&section[..length]), 0, "section CRC verifies");
(section[5] >> 1) & 0x1F
}
fn discontinuity(packet: &[u8]) -> bool { packet[3] & 0x20 != 0 && packet[4] >= 1 && packet[5] & 0x80 != 0 }
fn aligner() -> TsSpliceAligner { TsSpliceAligner::new(1024 * 1024, Duration::from_secs(5)) }
/// An aligner that already streamed a first member with PAT/PMT `version` and is switching.
fn switched_aligner(version: u8, now: Instant) -> TsSpliceAligner {
let mut aligner = aligner();
emitted(aligner.push(join(&[pat(version), video_pmt(version), pes(VIDEO_PID, true, false, &[&h264(5)])]), now));
aligner.begin_member(now);
aligner
}
#[test]
fn null_packet_is_valid_ts() {
assert_eq!(TS_NULL_PACKET.len(), TS_PACKET_SIZE);
assert_eq!(TS_NULL_PACKET[0], 0x47);
assert_eq!(pid_of(&TS_NULL_PACKET), 0x1FFF);
assert_eq!(ts_null_packet_bytes().as_ptr(), TS_NULL_PACKET.as_ptr());
}
#[test]
fn crc32_matches_known_pat() {
let section = [0x00, 0xB0, 0x0D, 0x00, 0x01, 0xC1, 0x00, 0x00, 0x00, 0x01, 0xF0, 0x00];
assert_eq!(mpeg2_crc32(&section), 0x2AB1_04B2);
}
#[test]
fn passthrough_emits_aligned_chunk_zero_copy() {
let mut aligner = aligner();
let chunk = join(&[pat(0), video_pmt(0), video_cont(false)]);
let out = emitted(aligner.push(chunk.clone(), Instant::now()));
assert_eq!(out.as_ptr(), chunk.as_ptr());
assert_eq!(out, chunk);
}
#[test]
fn realigns_unaligned_input_to_188() {
let mut aligner = aligner();
let stream = join(&[pat(0), video_pmt(0), video_cont(false)]);
let now = Instant::now();
let first = emitted(aligner.push(stream.slice(..200), now));
assert_eq!(&first[..], &stream[..188]);
assert!(matches!(aligner.push(stream.slice(200..300), now), SpliceOutput::Hold));
let rest = emitted(aligner.push(stream.slice(300..), now));
assert_eq!(&rest[..], &stream[188..]);
}
#[test]
fn drops_garbage_before_sync() {
let mut aligner = aligner();
let packets_in = [pat(0), video_pmt(0), video_cont(false), video_cont(false)].concat();
let mut input = vec![0x00; 17];
input.extend_from_slice(&packets_in);
let out = emitted(aligner.push(Bytes::from(input), Instant::now()));
assert_eq!(&out[..], &packets_in[..]);
}
#[test]
fn waits_for_keyframe_with_rai() {
let now = Instant::now();
let mut aligner = switched_aligner(0, now);
assert!(matches!(aligner.push(join(&[pat(0), video_pmt(0), video_cont(false)]), now), SpliceOutput::Hold));
let out = emitted(aligner.push(join(&[pes(VIDEO_PID, true, false, &[&h264(1)])]), now));
let out = packets(&out);
assert_eq!(out.len(), 4);
assert_eq!((pid_of(out[0]), pid_of(out[1])), (0, PMT_PID));
}
#[test]
fn detects_h264_idr_without_rai() {
assert!(is_idr_start(H264, &pes(VIDEO_PID, false, false, &[&h264(9), &h264(7), &h264(8), &h264(5)])[4..]));
let now = Instant::now();
let mut aligner = switched_aligner(0, now);
let out = aligner.push(join(&[pat(0), video_pmt(0), pes(VIDEO_PID, false, false, &[&h264(7), &h264(5)])]), now);
assert_eq!(packets(&emitted(out)).len(), 3);
}
#[test]
fn detects_hevc_idr_without_rai() {
for nal_type in [19, 20, 21] {
assert!(is_idr_start(HEVC, &pes(VIDEO_PID, false, false, &[&hevc(32), &hevc(nal_type)])[4..]));
}
let now = Instant::now();
let mut aligner = switched_aligner(0, now);
let pmt = pmt(0, &[(HEVC, VIDEO_PID)]);
let out = aligner.push(join(&[pat(0), pmt, pes(VIDEO_PID, false, false, &[&hevc(19)])]), now);
assert_eq!(packets(&emitted(out)).len(), 3);
}
#[test]
fn ignores_non_idr_slices() {
assert!(!is_idr_start(H264, &pes(VIDEO_PID, false, false, &[&h264(9), &h264(1)])[4..]));
assert!(!is_idr_start(HEVC, &pes(VIDEO_PID, false, false, &[&hevc(1)])[4..]));
let now = Instant::now();
let mut aligner = switched_aligner(0, now);
let out = aligner.push(join(&[pat(0), video_pmt(0), pes(VIDEO_PID, false, false, &[&h264(1)])]), now);
assert!(matches!(out, SpliceOutput::Hold));
}
#[test]
fn emits_from_pat_before_keyframe() {
let now = Instant::now();
let mut aligner = switched_aligner(0, now);
let input = [
video_cont(false),
pat(0),
video_pmt(0),
video_cont(false),
pat(0),
video_pmt(0),
pes(VIDEO_PID, true, false, &[&h264(5)]),
video_cont(false),
];
let out = emitted(aligner.push(join(&input), now));
let out = packets(&out);
assert_eq!(out.len(), 4);
assert_eq!(pid_of(out[0]), 0);
assert_eq!(pid_of(out[1]), PMT_PID);
assert_eq!(pid_of(out[2]), VIDEO_PID);
}
#[test]
fn audio_only_member_starts_at_pmt() {
let now = Instant::now();
let mut aligner = switched_aligner(0, now);
let audio_pmt = pmt(0, &[(AAC, AUDIO_PID)]);
let out = aligner.push(join(&[ts(AUDIO_PID, false, None, &[1; 10]), pat(0), audio_pmt]), now);
let out = emitted(out);
let out = packets(&out);
assert_eq!(out.len(), 2);
assert_eq!(pid_of(out[0]), 0);
}
#[test]
fn falls_back_after_byte_budget_from_first_pat() {
let now = Instant::now();
let mut aligner = TsSpliceAligner::new(5 * TS_PACKET_SIZE, Duration::from_secs(5));
emitted(aligner.push(join(&[pat(0), video_pmt(0)]), now));
aligner.begin_member(now);
let input = [video_cont(false), pat(0), video_pmt(0), video_cont(false), video_cont(false), video_cont(false)];
let out = emitted(aligner.push(join(&input), now));
let out = packets(&out);
assert_eq!(out.len(), 5);
assert_eq!(pid_of(out[0]), 0);
}
#[test]
fn falls_back_after_time_budget() {
let now = Instant::now();
let mut aligner = switched_aligner(0, now);
assert!(matches!(aligner.push(join(&[video_cont(false), pat(0)]), now), SpliceOutput::Hold));
let later = now + Duration::from_secs(6);
let out = emitted(aligner.push(join(&[video_cont(false)]), later));
let out = packets(&out);
assert_eq!(out.len(), 2);
assert_eq!(pid_of(out[0]), 0);
}
#[test]
fn sets_discontinuity_only_with_existing_adaptation_field() {
let now = Instant::now();
let mut aligner = switched_aligner(0, now);
let input = [
pat(0),
video_pmt(0),
pes(VIDEO_PID, true, false, &[&h264(5)]),
ts(AUDIO_PID, true, None, &[1; 20]),
ts(AUDIO_PID, false, Some(0x00), &[1; 20]),
video_cont(true),
];
let out = emitted(aligner.push(join(&input), now));
let out = packets(&out);
assert!(discontinuity(out[2]), "first video packet has an adaptation field");
assert!(!discontinuity(out[3]), "first audio packet has none");
assert!(!discontinuity(out[4]), "audio was already seen");
assert!(!discontinuity(out[5]), "video was already seen");
}
#[test]
fn second_member_pat_and_pmt_get_new_version_and_valid_crc() {
let now = Instant::now();
let mut aligner = switched_aligner(0, now);
let out = emitted(aligner.push(join(&[pat(0), video_pmt(0), pes(VIDEO_PID, true, false, &[&h264(5)])]), now));
let out = packets(&out);
assert_eq!(section_version(out[0]), 1);
assert_eq!(section_version(out[1]), 1);
let later = emitted(aligner.push(join(&[pat(0), video_pmt(0)]), now));
let later = packets(&later);
assert_eq!((section_version(later[0]), section_version(later[1])), (1, 1));
}
#[test]
fn member_version_change_is_preserved_with_offset() {
let now = Instant::now();
let mut aligner = switched_aligner(4, now);
emitted(aligner.push(join(&[pat(31), video_pmt(31), pes(VIDEO_PID, true, false, &[&h264(5)])]), now));
let out = emitted(aligner.push(join(&[pat(0), video_pmt(0)]), now));
let out = packets(&out);
assert_eq!((section_version(out[0]), section_version(out[1])), (6, 6));
}
#[test]
fn first_member_is_not_rewritten() {
let mut aligner = aligner();
let input = join(&[pat(7), video_pmt(9), pes(VIDEO_PID, true, false, &[&h264(5)])]);
let out = emitted(aligner.push(input.clone(), Instant::now()));
assert_eq!(out, input);
}
#[test]
fn multi_packet_section_passes_unchanged() {
let now = Instant::now();
let mut aligner = switched_aligner(0, now);
let mut long_pmt = video_pmt(0);
long_pmt[7] = 0xBF; // section_length claims more bytes than the packet holds
emitted(aligner.push(join(&[pat(0), video_pmt(0), pes(VIDEO_PID, true, false, &[&h264(5)])]), now));
let out = emitted(aligner.push(Bytes::from(long_pmt.clone()), now));
assert_eq!(&out[..], &long_pmt[..]);
}
#[test]
fn passthrough_splits_unaligned_chunk_without_copying_body() {
let mut aligner = aligner();
let stream = join(&[pat(0), video_pmt(0), video_cont(false), video_cont(false), video_cont(false)]);
let now = Instant::now();
let first = emitted(aligner.push(stream.slice(..300), now));
assert_eq!(&first[..], &stream[..188]);
let second = stream.slice(300..);
let SpliceOutput::EmitTwo(head, body) = aligner.push(second.clone(), now) else {
panic!("expected the completed packet and the aligned rest")
};
assert_eq!(&head[..], &stream[188..376]);
assert_eq!(&body[..], &stream[376..]);
assert_eq!(body.as_ptr(), second[76..].as_ptr(), "the aligned rest is not copied");
}
#[test]
fn h264_sps_counts_as_keyframe_start() {
assert!(is_idr_start(H264, &pes(VIDEO_PID, false, false, &[&h264(9), &h264(7)])[4..]));
assert!(is_idr_start(HEVC, &pes(VIDEO_PID, false, false, &[&hevc(32)])[4..]));
}
#[test]
fn detects_idr_in_second_packet_of_pes() {
let now = Instant::now();
let mut aligner = switched_aligner(0, now);
// AUD and a large SEI fill the PES start packet; the IDR slice follows in the next packet.
let sei = [vec![0x06], vec![0x05; 150]].concat();
let pes_start = pes(VIDEO_PID, false, false, &[&h264(9), &sei]);
let idr_continuation = ts(VIDEO_PID, false, None, &[0x00, 0x00, 0x00, 0x01, 0x65, 0x88, 0x84]);
let input = [pat(0), video_pmt(0), pes_start, idr_continuation, video_cont(false)];
let out = emitted(aligner.push(join(&input), now));
let out = packets(&out);
assert_eq!(out.len(), 5, "the splice starts at the PAT before the PES start");
assert_eq!(pid_of(out[0]), 0);
}
#[test]
fn psi_rewrite_of_unique_chunk_does_not_copy() {
let now = Instant::now();
let mut aligner = switched_aligner(0, now);
emitted(aligner.push(join(&[pat(0), video_pmt(0), pes(VIDEO_PID, true, false, &[&h264(5)])]), now));
let chunk = join(&[pat(0), video_pmt(0), video_cont(false)]);
let input_ptr = chunk.as_ptr();
let out = emitted(aligner.push(chunk, now));
assert_eq!(out.as_ptr(), input_ptr, "the versions are rewritten in place");
let out = packets(&out);
assert_eq!((section_version(out[0]), section_version(out[1])), (1, 1));
}
@@ -109,6 +109,7 @@ mod tests;
mod buffer;
mod clock;
pub(crate) use clock::decode_pcr;
mod layout;
mod pes;
mod profile;
@@ -1,12 +1,10 @@
use super::{
clock::{
add_pts_dts_offset, decode_pcr_at_location, ticks_90khz_to_rounded_millis, ts_chunk_packet_count,
TS_PACKET_SIZE,
},
clock::{add_pts_dts_offset, decode_pcr_at_location, ticks_90khz_to_rounded_millis, ts_chunk_packet_count},
layout::{build_hls_finite_ts_layout, find_ts_alignment, finite_hls_presentation_duration},
profile::timestamp_profile_from_finite_layout,
HlsTsTimestampProfile, TransportStreamBuffer,
};
use crate::packet::TS_PACKET_SIZE;
use bytes::{Bytes, BytesMut};
use futures::task::AtomicWaker;
use sha2::{Digest, Sha256};
@@ -200,10 +198,10 @@ impl TransportStreamBuffer {
let packet_layout =
self.finite_hls_layout.as_ref().ok().and_then(|layout| layout.packets.get(current_pos)).copied();
let packet_has_payload = packet_layout
.map_or_else(|| matches!((packet[3] >> 4) & 0b11, 0b01 | 0b11), |layout| layout.has_payload);
.map_or_else(|| crate::packet::TsPacket(packet).has_payload(), |layout| layout.has_payload);
// O(1) PID lookup — PID is at most 13 bits (0–8191).
let pid = (u16::from(packet[1] & 0x1F) << 8) | u16::from(packet[2]);
let pid = crate::packet::ts_packet_pid(packet);
let entry = &mut self.cc_entries[pid as usize];
// Normalize payload continuity per PID to a clean local sequence.
let (counter, discontinuity_sent) = entry.get_or_insert((0, false));
@@ -36,9 +36,7 @@ pub(super) fn add_pcr_offset_27mhz(timestamp_27mhz: u64, offset_27mhz: u64) -> u
((u128::from(timestamp_27mhz) + u128::from(offset_27mhz)) % u128::from(MAX_PCR)) as u64
}
pub(super) const TS_PACKET_SIZE: usize = 188;
pub(super) const SYNC_BYTE: u8 = 0x47;
use crate::packet::{TS_PACKET_SIZE, TS_SYNC_BYTE};
pub(super) const PACKET_COUNT: usize = 7;
@@ -58,11 +56,6 @@ pub(super) fn ts_chunk_packet_count() -> usize {
*PACKET_COUNT_OVERRIDE
}
pub(super) const ADAPTATION_FIELD_FLAG_PCR: u8 = 0x10;
// PCR flag bit in adaptation field flags
pub(super) const NULL_PID: u16 = 0x1FFF;
pub(super) const HLS_TS_MAX_PENDING_PES_HEADERS: usize = 64;
pub(super) const HLS_TS_TIMESTAMP_HEADER_BYTES: usize = 19;
@@ -97,7 +90,7 @@ pub(super) fn encode_timestamp(ts: u64) -> [u8; 5] {
/// Decode PCR from 6 bytes (adaptation field) into 42-bit PCR base + 9-bit extension as u64
#[inline]
pub(super) fn decode_pcr(pcr_bytes: &[u8]) -> u64 {
pub(crate) fn decode_pcr(pcr_bytes: &[u8]) -> u64 {
let pcr_base = (u64::from(pcr_bytes[0]) << 25)
| ((u64::from(pcr_bytes[1])) << 17)
| ((u64::from(pcr_bytes[2])) << 9)
@@ -113,21 +106,18 @@ pub(super) enum HlsTsTimestampKind {
Pcr,
}
#[inline]
pub(super) fn ts_packet_pid(packet: &[u8]) -> u16 { (u16::from(packet[1] & 0x1F) << 8) | u16::from(packet[2]) }
pub(super) fn same_finite_ts_packet_layout(source: &[u8], prepared: &[u8]) -> bool {
if source.len() != TS_PACKET_SIZE
|| prepared.len() != TS_PACKET_SIZE
|| source[0] != SYNC_BYTE
|| prepared[0] != SYNC_BYTE
|| source[0] != TS_SYNC_BYTE
|| prepared[0] != TS_SYNC_BYTE
|| source[1] & 0x7F != prepared[1] & 0x7F
|| source[2] != prepared[2]
|| source[3] & 0xF0 != prepared[3] & 0xF0
{
return false;
}
let adaptation_field_control = (source[3] >> 4) & 0b11;
let adaptation_field_control = crate::packet::TsPacket(source).adaptation_field_control();
!matches!(adaptation_field_control, 0b10 | 0b11) || source[4] == prepared[4]
}
@@ -137,7 +127,7 @@ pub(super) fn append_finite_discontinuity_packet(following_packet: &[u8], output
let start = output.len();
output.resize(start.saturating_add(TS_PACKET_SIZE), 0xFF);
let marker = &mut output[start..start + TS_PACKET_SIZE];
marker[0] = SYNC_BYTE;
marker[0] = TS_SYNC_BYTE;
marker[1] = following_packet[1] & 0x1F;
marker[2] = following_packet[2];
// FFmpeg retains this adaptation-only marker as the per-PID CC baseline.
@@ -1,28 +1,17 @@
use super::{
clock::{decode_pcr_at_location, decode_timestamp_at_location, MAX_PTS_DTS, SYNC_BYTE, TS_PACKET_SIZE},
clock::{decode_pcr_at_location, decode_timestamp_at_location, MAX_PTS_DTS},
pes::{inspect_hls_ts_packet, HlsFiniteTsLayoutError},
HlsFiniteTsLayout, HlsFiniteTsPacketLayout, HlsPesHeaderAssembler, HlsTsTimestampFieldKind,
};
use crate::packet::TS_PACKET_SIZE;
use std::{
collections::{HashMap, HashSet},
sync::Arc,
};
/// Finds TS alignment by checking for 0x47 sync byte every 188 bytes
/// Finds TS alignment by checking for 0x47 sync byte every 188 bytes, five packets in a row.
pub(super) fn find_ts_alignment(buf: &[u8]) -> Option<usize> {
for offset in 0..TS_PACKET_SIZE {
let mut valid = true;
for i in 0..5 {
if buf.get(offset + i * TS_PACKET_SIZE) != Some(&SYNC_BYTE) {
valid = false;
break;
}
}
if valid {
return Some(offset);
}
}
None
crate::packet::find_sync(buf, 0..TS_PACKET_SIZE, 5, false)
}
pub(super) fn build_hls_finite_ts_layout(buffer: &[u8]) -> Result<HlsFiniteTsLayout, HlsFiniteTsLayoutError> {
@@ -1,11 +1,9 @@
use super::{
clock::{
ts_packet_pid, ADAPTATION_FIELD_FLAG_PCR, HLS_TS_MAX_PENDING_PES_HEADERS, HLS_TS_TIMESTAMP_HEADER_BYTES,
PES_DTS_OFFSET, PES_PTS_OFFSET, SYNC_BYTE, TS_PACKET_SIZE,
},
clock::{HLS_TS_MAX_PENDING_PES_HEADERS, HLS_TS_TIMESTAMP_HEADER_BYTES, PES_DTS_OFFSET, PES_PTS_OFFSET},
HlsCompletedPesTimestamps, HlsCompletedTimestampField, HlsPendingPesHeader, HlsPesHeaderAssembler,
HlsTsPacketEvidence,
};
use crate::packet::{ts_packet_pid, ADAPTATION_FIELD_FLAG_PCR, TS_PACKET_SIZE, TS_SYNC_BYTE};
use std::collections::HashMap;
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
@@ -77,11 +75,11 @@ pub(super) fn inspect_hls_ts_packet(
packet: &[u8],
packet_start: usize,
) -> Result<HlsTsPacketEvidence, HlsFiniteTsLayoutError> {
if packet.len() != TS_PACKET_SIZE || packet[0] != SYNC_BYTE || packet[1] & 0x80 != 0 || packet[3] & 0xC0 != 0 {
if packet.len() != TS_PACKET_SIZE || packet[0] != TS_SYNC_BYTE || packet[1] & 0x80 != 0 || packet[3] & 0xC0 != 0 {
return Err(HlsFiniteTsLayoutError::InvalidTransportPacket);
}
let pid = ts_packet_pid(packet);
let adaptation_field_control = (packet[3] >> 4) & 0b11;
let adaptation_field_control = crate::packet::TsPacket(packet).adaptation_field_control();
let continuity_counter = packet[3] & 0x0F;
let (payload_offset, adaptation_length) = match adaptation_field_control {
0b01 => (Some(4), None),
@@ -2,11 +2,12 @@ use super::{
clock::{
add_pts_dts_offset, decode_pcr, decode_pcr_at_location, decode_timestamp, decode_timestamp_at_location,
forward_clock_distance_90khz, HlsTsTimestampKind, HLS_TS_PROFILE_MIN_TOLERANCE_TICKS_90KHZ,
HLS_TS_SPLICE_MIN_GAP_TICKS_90KHZ, MAX_PTS_DTS, TS_PACKET_SIZE,
HLS_TS_SPLICE_MIN_GAP_TICKS_90KHZ, MAX_PTS_DTS,
},
pes::inspect_hls_ts_packet,
HlsFiniteTsLayout, HlsPesHeaderAssembler, HlsTsTimestampProfileAccumulator, HlsTsTimestampProfileScanner,
};
use crate::packet::TS_PACKET_SIZE;
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub struct HlsTsTimestampProfile {
@@ -2,11 +2,12 @@ use super::{
clock::{
add_pcr_offset_27mhz, add_pts_dts_offset, append_finite_discontinuity_packet, decode_pcr_at_location,
decode_timestamp, encode_pcr, encode_timestamp, gather_timestamp_bytes, pcr_offset_27mhz,
same_finite_ts_packet_layout, ts_packet_pid, ADAPTATION_FIELD_FLAG_PCR, NULL_PID, SYNC_BYTE, TS_PACKET_SIZE,
same_finite_ts_packet_layout,
},
pes::HlsFiniteTsLayoutError,
HlsFiniteTsLayout, HlsFiniteTsPacketLayout, TransportStreamBuffer,
};
use crate::packet::{ts_packet_pid, NULL_PID, TS_PACKET_SIZE, TS_SYNC_BYTE};
use bytes::{Bytes, BytesMut};
use std::task::Waker;
@@ -179,20 +180,12 @@ impl TransportStreamBuffer {
out.resize(start + TS_PACKET_SIZE, 0xFF);
let pkt = &mut out[start..start + TS_PACKET_SIZE];
pkt[0] = SYNC_BYTE;
pkt[0] = TS_SYNC_BYTE;
pkt[1] = new_packet[1] & 0x1F;
pkt[2] = new_packet[2];
// Check if the current packet has a PCR (need at least 7 adaptation bytes: 1 flag + 6 PCR).
let new_pkt_has_pcr = {
let afc = (new_packet[3] >> 4) & 0b11;
if afc == 2 || afc == 3 {
let adaptation_len = new_packet[4] as usize;
adaptation_len >= 7 && (new_packet[5] & ADAPTATION_FIELD_FLAG_PCR) != 0
} else {
false
}
};
let new_pkt_has_pcr = crate::packet::TsPacket(new_packet).has_pcr();
// AFC=2 (Adaptation Only), Scrambling=00 (Unscrambled), CC=cc
pkt[3] = 0x20 | (cc & 0x0F);
@@ -2,18 +2,18 @@ use super::{
clock::{
append_finite_discontinuity_packet, decode_pcr, decode_pcr_at_location, decode_timestamp,
decode_timestamp_at_location, encode_pcr, encode_timestamp, forward_clock_distance_90khz,
gather_timestamp_bytes, pcr_offset_27mhz, ts_packet_pid, ADAPTATION_FIELD_FLAG_PCR,
HLS_TS_SPLICE_MIN_GAP_TICKS_90KHZ, MAX_PTS_DTS, NULL_PID, SYNC_BYTE, TS_PACKET_SIZE,
gather_timestamp_bytes, pcr_offset_27mhz, HLS_TS_SPLICE_MIN_GAP_TICKS_90KHZ, MAX_PTS_DTS,
},
layout::build_hls_finite_ts_layout,
pes::inspect_hls_ts_packet,
*,
};
use crate::packet::{ts_packet_pid, ADAPTATION_FIELD_FLAG_PCR, NULL_PID, TS_PACKET_SIZE, TS_SYNC_BYTE};
use bytes::BytesMut;
fn build_pts_dts_payload_packet(pid: u16, cc: u8, pts: u64, dts: u64) -> [u8; TS_PACKET_SIZE] {
let mut packet = [0xFF_u8; TS_PACKET_SIZE];
packet[0] = SYNC_BYTE;
packet[0] = TS_SYNC_BYTE;
packet[1] = 0x40 | ((pid >> 8) as u8 & 0x1F); // PUSI + PID high bits
packet[2] = (pid & 0xFF) as u8;
packet[3] = 0x10 | (cc & 0x0F); // payload only
@@ -42,7 +42,7 @@ fn build_pts_dts_payload_packet(pid: u16, cc: u8, pts: u64, dts: u64) -> [u8; TS
fn build_pts_only_payload_packet(pid: u16, cc: u8, pts: u64) -> [u8; TS_PACKET_SIZE] {
let mut packet = [0xFF_u8; TS_PACKET_SIZE];
packet[0] = SYNC_BYTE;
packet[0] = TS_SYNC_BYTE;
packet[1] = 0x40 | ((pid >> 8) as u8 & 0x1F);
packet[2] = (pid & 0xFF) as u8;
packet[3] = 0x10 | (cc & 0x0F);
@@ -86,7 +86,7 @@ fn build_split_pes_header_packets(
assert!((1..=184).contains(&payload_length));
let mut packet = [0xFF_u8; TS_PACKET_SIZE];
let packet_index = u8::try_from(packet_index).expect("test packet index fits");
packet[0] = SYNC_BYTE;
packet[0] = TS_SYNC_BYTE;
packet[1] = ((pid >> 8) as u8 & 0x1F) | (u8::from(packet_index == 0) * 0x40);
packet[2] = (pid & 0xFF) as u8;
packet[3] = if payload_length == 184 {
@@ -114,7 +114,7 @@ fn build_split_pes_header_packets(
fn build_pts_dts_pcr_packet(pid: u16, cc: u8, pts: u64, dts: u64, pcr_90khz: u64) -> [u8; TS_PACKET_SIZE] {
let mut packet = [0xFF_u8; TS_PACKET_SIZE];
packet[0] = SYNC_BYTE;
packet[0] = TS_SYNC_BYTE;
packet[1] = 0x40 | ((pid >> 8) as u8 & 0x1F);
packet[2] = (pid & 0xFF) as u8;
packet[3] = 0x30 | (cc & 0x0F);
@@ -170,7 +170,7 @@ fn packet_timestamps(packet: &[u8]) -> (u64, u64, u64) {
fn build_adaptation_only_packet(pid: u16, cc: u8) -> [u8; TS_PACKET_SIZE] {
let mut packet = [0xFF_u8; TS_PACKET_SIZE];
packet[0] = SYNC_BYTE;
packet[0] = TS_SYNC_BYTE;
packet[1] = (pid >> 8) as u8 & 0x1F;
packet[2] = (pid & 0xFF) as u8;
packet[3] = 0x20 | (cc & 0x0F); // adaptation only
@@ -188,7 +188,7 @@ fn build_pcr_only_packet(pid: u16, cc: u8, pcr_90khz: u64) -> [u8; TS_PACKET_SIZ
fn build_payload_packet(pid: u16, cc: u8) -> [u8; TS_PACKET_SIZE] {
let mut packet = [0xFF_u8; TS_PACKET_SIZE];
packet[0] = SYNC_BYTE;
packet[0] = TS_SYNC_BYTE;
packet[1] = (pid >> 8) as u8 & 0x1F;
packet[2] = (pid & 0xFF) as u8;
packet[3] = 0x10 | (cc & 0x0F);
+3 -3
View File
@@ -2,7 +2,7 @@ use crate::transport_stream_buffer::HlsTsTimestampProfileScanner;
use aes::Aes128;
use mpeg2ts_reader::{
demultiplex::{self, FilterChangeset},
packet::{Packet, Pid},
packet::Pid,
psi::{self, BufferSectionSyntaxParser, SectionPacketConsumer, SectionProcessor, SectionSyntaxSectionProcessor},
};
use std::{
@@ -13,7 +13,7 @@ use zeroize::Zeroizing;
const AES_128_BLOCK_BYTES: usize = 16;
const TS_PACKET_BYTES: usize = Packet::SIZE;
use crate::packet::TS_PACKET_SIZE;
const TS_ALIGNMENT_CONFIRMATION_PACKETS: usize = 2;
@@ -23,7 +23,7 @@ const HLS_TS_PROBE_MAX_PACKETS: u64 = 8_192;
const HLS_TS_PROBE_READ_CHUNK_BYTES: usize = 64 * 1024;
const HLS_TS_PROBE_MAX_RESYNC_BYTES: usize = TS_PACKET_BYTES * 4;
const HLS_TS_PROBE_MAX_RESYNC_BYTES: usize = TS_PACKET_SIZE * 4;
const PSI_SYNTAX_HEADER_BYTES: usize = psi::SectionCommonHeader::SIZE + psi::TableSyntaxHeader::SIZE;
+21 -27
View File
@@ -2,10 +2,10 @@ use super::{
protection::HlsTsSourceDecoder, HlsTsDemuxContext, HlsTsInspector, HlsTsMalformedReason, HlsTsMediaEvidence,
HlsTsMediaStreamInspector, HlsTsProbeBudget, HlsTsProbeError, HlsTsProbeOutcome, HlsTsProbeProtection,
HlsTsProtectionReason, HlsTsSpliceEvidence, HlsTsSpliceIncompatibility, HlsTsTrackSignature,
HlsTsTransportStreamInspector, AES_128_BLOCK_BYTES, TS_ALIGNMENT_CONFIRMATION_PACKETS, TS_PACKET_BYTES,
HlsTsTransportStreamInspector, AES_128_BLOCK_BYTES, TS_ALIGNMENT_CONFIRMATION_PACKETS, TS_PACKET_SIZE,
};
use crate::transport_stream_buffer::{HlsTsTimestampProfile, HlsTsTimestampProfileScanner};
use mpeg2ts_reader::{demultiplex, packet::Packet};
use mpeg2ts_reader::demultiplex;
use std::io::Read;
use tokio::io::{AsyncRead, AsyncReadExt};
use zeroize::Zeroizing;
@@ -22,7 +22,7 @@ impl HlsTsMediaStreamInspector {
budget,
timestamp_scanner: HlsTsTimestampProfileScanner::new(expected_duration_ticks_90khz),
transport_scanner: HlsTsTransportStreamInspector::new(),
pending: Vec::with_capacity(budget.bounded_read_chunk_bytes().saturating_add(TS_PACKET_BYTES)),
pending: Vec::with_capacity(budget.bounded_read_chunk_bytes().saturating_add(TS_PACKET_SIZE)),
aligned: false,
packets_examined: 0,
invalid: false,
@@ -35,17 +35,14 @@ impl HlsTsMediaStreamInspector {
}
self.pending.extend_from_slice(bytes);
if !self.aligned {
let confirmation_bytes = TS_PACKET_BYTES.saturating_mul(TS_ALIGNMENT_CONFIRMATION_PACKETS);
let confirmation_bytes = TS_PACKET_SIZE.saturating_mul(TS_ALIGNMENT_CONFIRMATION_PACKETS);
if self.pending.len() < confirmation_bytes {
return;
}
let available_search = self.pending.len().saturating_sub(confirmation_bytes);
let search_end = available_search.min(self.budget.max_resync_bytes);
let alignment = (0..=search_end).find(|offset| {
(0..TS_ALIGNMENT_CONFIRMATION_PACKETS).all(|index| {
self.pending[offset.saturating_add(index.saturating_mul(TS_PACKET_BYTES))] == Packet::SYNC_BYTE
})
});
let alignment =
crate::packet::find_sync(&self.pending, 0..=search_end, TS_ALIGNMENT_CONFIRMATION_PACKETS, false);
let Some(alignment) = alignment else {
if available_search >= self.budget.max_resync_bytes {
self.invalid = true;
@@ -57,12 +54,12 @@ impl HlsTsMediaStreamInspector {
}
let mut consumed = 0usize;
while self.pending.len().saturating_sub(consumed) >= TS_PACKET_BYTES {
while self.pending.len().saturating_sub(consumed) >= TS_PACKET_SIZE {
if self.packets_examined >= self.budget.max_packets {
self.invalid = true;
break;
}
let packet_end = consumed.saturating_add(TS_PACKET_BYTES);
let packet_end = consumed.saturating_add(TS_PACKET_SIZE);
let packet = &self.pending[consumed..packet_end];
self.timestamp_scanner.push_aligned_packet(packet);
self.transport_scanner.push_packet(packet, self.packets_examined);
@@ -122,7 +119,7 @@ impl HlsTsInspector {
budget,
demux,
context,
pending: Vec::with_capacity(budget.bounded_read_chunk_bytes().saturating_add(TS_PACKET_BYTES)),
pending: Vec::with_capacity(budget.bounded_read_chunk_bytes().saturating_add(TS_PACKET_SIZE)),
aligned: false,
source_bytes_examined: 0,
packets_examined: 0,
@@ -156,17 +153,14 @@ impl HlsTsInspector {
) -> Option<HlsTsProbeOutcome> {
self.pending.extend_from_slice(bytes);
if !self.aligned {
let confirmation_bytes = TS_PACKET_BYTES.saturating_mul(TS_ALIGNMENT_CONFIRMATION_PACKETS);
let confirmation_bytes = TS_PACKET_SIZE.saturating_mul(TS_ALIGNMENT_CONFIRMATION_PACKETS);
if self.pending.len() < confirmation_bytes {
return None;
}
let available_search = self.pending.len().saturating_sub(confirmation_bytes);
let search_end = available_search.min(self.budget.max_resync_bytes);
let alignment = (0..=search_end).find(|offset| {
(0..TS_ALIGNMENT_CONFIRMATION_PACKETS).all(|index| {
self.pending[offset.saturating_add(index.saturating_mul(TS_PACKET_BYTES))] == Packet::SYNC_BYTE
})
});
let alignment =
crate::packet::find_sync(&self.pending, 0..=search_end, TS_ALIGNMENT_CONFIRMATION_PACKETS, false);
let Some(alignment) = alignment else {
if available_search >= self.budget.max_resync_bytes {
return Some(HlsTsProbeOutcome::Malformed(HlsTsMalformedReason::InvalidSynchronization));
@@ -178,14 +172,14 @@ impl HlsTsInspector {
}
let mut consumed = 0usize;
while self.pending.len().saturating_sub(consumed) >= TS_PACKET_BYTES {
while self.pending.len().saturating_sub(consumed) >= TS_PACKET_SIZE {
if self.packets_examined >= self.budget.max_packets {
self.pending.drain(..consumed);
return Some(self.budget_exhausted());
}
let mut packet_bytes = [0_u8; TS_PACKET_BYTES];
packet_bytes.copy_from_slice(&self.pending[consumed..consumed.saturating_add(TS_PACKET_BYTES)]);
consumed = consumed.saturating_add(TS_PACKET_BYTES);
let mut packet_bytes = [0_u8; TS_PACKET_SIZE];
packet_bytes.copy_from_slice(&self.pending[consumed..consumed.saturating_add(TS_PACKET_SIZE)]);
consumed = consumed.saturating_add(TS_PACKET_SIZE);
self.packets_examined = self.packets_examined.saturating_add(1);
if let Some(outcome) = self.validate_packet(&packet_bytes) {
self.pending.drain(..consumed);
@@ -207,8 +201,8 @@ impl HlsTsInspector {
None
}
pub(super) fn validate_packet(&self, bytes: &[u8; TS_PACKET_BYTES]) -> Option<HlsTsProbeOutcome> {
if bytes[0] != Packet::SYNC_BYTE {
pub(super) fn validate_packet(&self, bytes: &[u8; TS_PACKET_SIZE]) -> Option<HlsTsProbeOutcome> {
if bytes[0] != crate::packet::TS_SYNC_BYTE {
return Some(HlsTsProbeOutcome::Malformed(HlsTsMalformedReason::InvalidSynchronization));
}
if bytes[1] & 0x80 != 0 {
@@ -217,17 +211,17 @@ impl HlsTsInspector {
if bytes[3] & 0xC0 != 0 {
return Some(HlsTsProbeOutcome::UnsupportedProtection(HlsTsProtectionReason::TransportScrambling));
}
let adaptation_control = (bytes[3] >> 4) & 0b11;
let adaptation_control = crate::packet::TsPacket(bytes).adaptation_field_control();
let payload_offset = match adaptation_control {
0b01 => 4,
0b10 if bytes[4] == 183 => TS_PACKET_BYTES,
0b10 if bytes[4] == 183 => TS_PACKET_SIZE,
0b11 if bytes[4] <= 182 => 5usize.saturating_add(usize::from(bytes[4])),
0b00 | 0b10 | 0b11 => {
return Some(HlsTsProbeOutcome::Malformed(HlsTsMalformedReason::InvalidPacketHeader));
}
_ => return Some(HlsTsProbeOutcome::Malformed(HlsTsMalformedReason::InvalidPacketHeader)),
};
let pid = (u16::from(bytes[1] & 0x1F) << 8) | u16::from(bytes[2]);
let pid = crate::packet::ts_packet_pid(bytes);
if self.context.is_psi_pid(pid) && bytes[1] & 0x40 != 0 {
let Some(payload) = bytes.get(payload_offset..) else {
return Some(HlsTsProbeOutcome::Malformed(HlsTsMalformedReason::InvalidPsiPointer));
+5 -5
View File
@@ -1,7 +1,7 @@
use super::{
demux::HlsPatSectionParser, HlsPatPacketFilter, HlsPmtEvidence, HlsPmtPacketFilter, HlsPmtSectionEvidence,
HlsPmtSectionParser, HlsPsiSectionConsumer, HlsTsDemuxContext, HlsTsMalformedReason, HlsTsProgramTopology,
PSI_SYNTAX_HEADER_BYTES, TS_PACKET_BYTES,
PSI_SYNTAX_HEADER_BYTES, TS_PACKET_SIZE,
};
use mpeg2ts_reader::{
demultiplex,
@@ -191,12 +191,12 @@ where
}
}
pub(super) fn synthetic_psi_packet(pid: Pid, starts_section: bool, payload: &[u8]) -> [u8; TS_PACKET_BYTES] {
pub(super) fn synthetic_psi_packet(pid: Pid, starts_section: bool, payload: &[u8]) -> [u8; TS_PACKET_SIZE] {
let pointer_bytes = usize::from(starts_section);
let payload_length = payload.len().saturating_add(pointer_bytes).min(184);
let pid = u16::from(pid);
let mut packet = [0xFF_u8; TS_PACKET_BYTES];
packet[0] = Packet::SYNC_BYTE;
let mut packet = [0xFF_u8; TS_PACKET_SIZE];
packet[0] = crate::packet::TS_SYNC_BYTE;
packet[1] = (u8::try_from(pid >> 8).unwrap_or(0) & 0x1F) | if starts_section { 0x40 } else { 0 };
packet[2] = pid.to_be_bytes()[1];
let payload_offset = if payload_length == 184 {
@@ -217,7 +217,7 @@ pub(super) fn synthetic_psi_packet(pid: Pid, starts_section: bool, payload: &[u8
} else {
payload_offset
};
let copy_length = payload.len().min(TS_PACKET_BYTES.saturating_sub(data_offset));
let copy_length = payload.len().min(TS_PACKET_SIZE.saturating_sub(data_offset));
packet[data_offset..data_offset.saturating_add(copy_length)].copy_from_slice(&payload[..copy_length]);
packet
}
+15 -15
View File
@@ -100,8 +100,8 @@ fn packetize_section(pid: u16, section: &[u8], first_counter: u8) -> Vec<u8> {
let mut counter = first_counter & 0x0F;
let mut first = true;
while offset < section.len() {
let mut packet = [0xFF_u8; TS_PACKET_BYTES];
packet[0] = Packet::SYNC_BYTE;
let mut packet = [0xFF_u8; TS_PACKET_SIZE];
packet[0] = crate::packet::TS_SYNC_BYTE;
packet[1] = u8::try_from(pid >> 8).unwrap_or(0) & 0x1F;
packet[2] = u8::try_from(pid & 0xFF).unwrap_or(0);
packet[3] = 0x10 | counter;
@@ -112,7 +112,7 @@ fn packetize_section(pid: u16, section: &[u8], first_counter: u8) -> Vec<u8> {
} else {
4
};
let copied = section.len().saturating_sub(offset).min(TS_PACKET_BYTES.saturating_sub(payload_start));
let copied = section.len().saturating_sub(offset).min(TS_PACKET_SIZE.saturating_sub(payload_start));
packet[payload_start..payload_start.saturating_add(copied)]
.copy_from_slice(&section[offset..offset.saturating_add(copied)]);
output.extend_from_slice(&packet);
@@ -133,10 +133,10 @@ fn packetize_section_with_split_syntax_header(pid: u16, section: &[u8], first_he
output
}
fn psi_start_packet_with_pointer(pid: u16, previous_section: &[u8], section_start: &[u8]) -> [u8; TS_PACKET_BYTES] {
fn psi_start_packet_with_pointer(pid: u16, previous_section: &[u8], section_start: &[u8]) -> [u8; TS_PACKET_SIZE] {
let payload_length = 1usize.saturating_add(previous_section.len()).saturating_add(section_start.len()).min(184);
let mut packet = [0xFF_u8; TS_PACKET_BYTES];
packet[0] = Packet::SYNC_BYTE;
let mut packet = [0xFF_u8; TS_PACKET_SIZE];
packet[0] = crate::packet::TS_SYNC_BYTE;
packet[1] = (u8::try_from(pid >> 8).unwrap_or(0) & 0x1F) | 0x40;
packet[2] = u8::try_from(pid & 0xFF).unwrap_or(0);
packet[3] = 0x30;
@@ -155,9 +155,9 @@ fn psi_start_packet_with_pointer(pid: u16, previous_section: &[u8], section_star
packet
}
fn null_packet() -> [u8; TS_PACKET_BYTES] {
let mut packet = [0xFF_u8; TS_PACKET_BYTES];
packet[0] = Packet::SYNC_BYTE;
fn null_packet() -> [u8; TS_PACKET_SIZE] {
let mut packet = [0xFF_u8; TS_PACKET_SIZE];
packet[0] = crate::packet::TS_SYNC_BYTE;
packet[1] = 0x1F;
packet[2] = 0xFF;
packet[3] = 0x10;
@@ -181,10 +181,10 @@ fn media_payload_packet(
payload_unit_start: bool,
discontinuity: bool,
payload: &[u8],
) -> [u8; TS_PACKET_BYTES] {
) -> [u8; TS_PACKET_SIZE] {
assert!(!payload.is_empty() && payload.len() <= 182);
let mut packet = [0xFF_u8; TS_PACKET_BYTES];
packet[0] = Packet::SYNC_BYTE;
let mut packet = [0xFF_u8; TS_PACKET_SIZE];
packet[0] = crate::packet::TS_SYNC_BYTE;
packet[1] = u8::try_from(pid >> 8).unwrap_or(0) & 0x1F;
if payload_unit_start {
packet[1] |= 0x40;
@@ -201,9 +201,9 @@ fn media_payload_packet(
packet
}
fn media_adaptation_only_packet(pid: u16, continuity_counter: u8, discontinuity: bool) -> [u8; TS_PACKET_BYTES] {
let mut packet = [0xFF_u8; TS_PACKET_BYTES];
packet[0] = Packet::SYNC_BYTE;
fn media_adaptation_only_packet(pid: u16, continuity_counter: u8, discontinuity: bool) -> [u8; TS_PACKET_SIZE] {
let mut packet = [0xFF_u8; TS_PACKET_SIZE];
packet[0] = crate::packet::TS_SYNC_BYTE;
packet[1] = u8::try_from(pid >> 8).unwrap_or(0) & 0x1F;
packet[2] = u8::try_from(pid & 0xFF).unwrap_or(0);
packet[3] = 0x20 | (continuity_counter & 0x0F);
@@ -424,10 +424,10 @@ fn ts_inspector_reports_probe_budget_without_reading_whole_source() {
}
let count = Rc::new(Cell::new(0));
let reader = CountingReader { inner: Cursor::new(bytes), bytes_read: Rc::clone(&count) };
let budget = HlsTsProbeBudget { max_bytes: (TS_PACKET_BYTES * 4) as u64, ..HlsTsProbeBudget::default() };
let budget = HlsTsProbeBudget { max_bytes: (TS_PACKET_SIZE * 4) as u64, ..HlsTsProbeBudget::default() };
let outcome = inspect_mpeg_ts(reader, HlsTsProbeProtection::Clear, budget).expect("probe succeeds");
assert!(matches!(outcome, HlsTsProbeOutcome::ProbeBudgetExhausted { .. }));
assert_eq!(count.get(), TS_PACKET_BYTES * 4);
assert_eq!(count.get(), TS_PACKET_SIZE * 4);
}
#[test]
@@ -496,7 +496,7 @@ fn ts_inspector_resynchronizes_only_within_named_budget() {
#[test]
fn ts_inspector_types_framing_crc_and_pmt_errors() {
let invalid_sync = vec![0_u8; TS_PACKET_BYTES * 3];
let invalid_sync = vec![0_u8; TS_PACKET_SIZE * 3];
assert_eq!(
inspect_mpeg_ts(Cursor::new(invalid_sync), HlsTsProbeProtection::Clear, HlsTsProbeBudget::default())
.expect("probe completes"),
+2 -2
View File
@@ -1,6 +1,6 @@
use super::{
HlsTsCompatibleSpliceEvidence, HlsTsPidBoundaryEvidence, HlsTsPidContinuityState, HlsTsSpliceEvidence,
HlsTsSpliceIncompatibility, HlsTsTrackSignature, HlsTsTransportStreamInspector, TS_PACKET_BYTES,
HlsTsSpliceIncompatibility, HlsTsTrackSignature, HlsTsTransportStreamInspector, TS_PACKET_SIZE,
};
use mpeg2ts_reader::packet::Packet;
@@ -38,7 +38,7 @@ impl HlsTsTransportStreamInspector {
if self.incompatible.is_some() {
return;
}
if bytes.len() != TS_PACKET_BYTES {
if bytes.len() != TS_PACKET_SIZE {
self.incompatible = Some(HlsTsSpliceIncompatibility::InvalidPacket { packet_index });
return;
}
@@ -0,0 +1,544 @@
use crate::processor::match_key::{extract_normalized_match_key, MatchKeyNormalizer};
use indexmap::IndexMap;
use shared::{
model::{
ChannelGroupOverridesDto, ChannelGroupPolicy, ChannelGroupsConfig, ChannelMemberRefDto, EpgConflictMode,
LiveStreamProperties, PlaylistGroup, PlaylistItem, StreamProperties, UUIDType, XtreamCluster,
MIN_GROUP_MEMBERS,
},
utils::{hash_string, token_quality, Internable},
};
use std::{
collections::{HashMap, HashSet},
sync::Arc,
};
use tuliprox_repository::{ChannelGroupOrigin, PendingChannelGroup};
pub(in crate::processor) fn strip_quality_words(value: &str) -> String {
let words: Vec<&str> = value.split_whitespace().collect();
let mut kept = Vec::with_capacity(words.len());
for word in words {
let core = word.trim_matches(|c: char| !c.is_alphanumeric());
if !core.is_empty() && token_quality(core).is_some() {
continue;
}
kept.push(word);
}
if kept.is_empty() {
value.to_string()
} else {
kept.join(" ")
}
}
fn is_eligible(item: &PlaylistItem) -> bool {
item.header.xtream_cluster == XtreamCluster::Live && !item.header.item_type.is_local()
}
fn item_member_ref(item: &PlaylistItem) -> Option<ChannelMemberRefDto> {
if !is_eligible(item) {
return None;
}
let stream_id = item.header.get_input_stream_id()?;
if stream_id.is_empty() {
return None;
}
Some(ChannelMemberRefDto { input: item.header.input_name.to_string(), stream_id: stream_id.to_string() })
}
struct GroupCandidate {
key: String,
origin: ChannelGroupOrigin,
policy: ChannelGroupPolicy,
positions: Vec<(usize, usize)>,
orphaned: Vec<ChannelMemberRefDto>,
epg_split: bool,
name_override: Option<String>,
logo_override: Option<String>,
epg_override: Option<String>,
playlist_group_override: Option<String>,
}
struct AutoBucket {
positions: Vec<(usize, usize)>,
has_dsl_forced: bool,
epg_split: bool,
}
struct GroupSlot {
id: u32,
title: Arc<str>,
xtream_cluster: XtreamCluster,
had_channels: bool,
channels: Vec<Option<PlaylistItem>>,
}
fn build_member_index(playlist: &[PlaylistGroup]) -> HashMap<ChannelMemberRefDto, (usize, usize)> {
let mut member_pos_map = HashMap::new();
for (group_idx, group) in playlist.iter().enumerate() {
if group.xtream_cluster != XtreamCluster::Live {
continue;
}
for (channel_idx, channel) in group.channels.iter().enumerate() {
if let Some(mref) = item_member_ref(channel) {
member_pos_map.entry(mref).or_insert((group_idx, channel_idx));
}
}
}
member_pos_map
}
fn resolve_manual_groups(
config: &ChannelGroupsConfig,
overrides: Option<&ChannelGroupOverridesDto>,
member_pos_map: &HashMap<ChannelMemberRefDto, (usize, usize)>,
taken_positions: &mut HashSet<(usize, usize)>,
candidates: &mut Vec<GroupCandidate>,
) {
let Some(overrides) = overrides else {
return;
};
for mg in &overrides.groups {
let mut matched_positions = Vec::new();
let mut orphaned = Vec::new();
for mref in &mg.members {
if let Some(&pos) = member_pos_map.get(mref) {
if !taken_positions.contains(&pos) {
taken_positions.insert(pos);
matched_positions.push(pos);
continue;
}
}
log::warn!(
"Manual channel group '{}' member '{}:{}' not found or already taken",
mg.id,
mref.input,
mref.stream_id
);
orphaned.push(mref.clone());
}
if matched_positions.len() < MIN_GROUP_MEMBERS {
continue;
}
candidates.push(GroupCandidate {
key: mg.id.clone(),
origin: ChannelGroupOrigin::Manual,
policy: mg.policy.unwrap_or(config.policy),
positions: matched_positions,
orphaned,
epg_split: false,
name_override: Some(mg.name.clone()),
logo_override: mg.logo.clone(),
epg_override: mg.epg_channel_id.clone(),
playlist_group_override: mg.playlist_group.clone(),
});
}
}
fn collect_auto_buckets(
config: &ChannelGroupsConfig,
playlist: &[PlaylistGroup],
excluded_set: &HashSet<&ChannelMemberRefDto>,
taken_positions: &HashSet<(usize, usize)>,
) -> IndexMap<String, AutoBucket> {
let normalizer = MatchKeyNormalizer::new(&config.match_normalize, config.match_as_ascii);
let mut auto_buckets: IndexMap<String, AutoBucket> = IndexMap::new();
for (group_idx, group) in playlist.iter().enumerate() {
if group.xtream_cluster != XtreamCluster::Live {
continue;
}
for (channel_idx, channel) in group.channels.iter().enumerate() {
let pos = (group_idx, channel_idx);
if taken_positions.contains(&pos) {
continue;
}
let Some(mref) = item_member_ref(channel) else {
continue;
};
if excluded_set.contains(&mref) {
continue;
}
let mut dsl_forced = false;
let key = match channel.header.channel_group.as_deref() {
Some("") => continue,
Some(forced) => {
dsl_forced = true;
normalizer.key(forced)
}
None => extract_normalized_match_key(
&channel.header,
config.match_by,
config.match_as_ascii,
Some(&normalizer),
),
};
if key.trim().is_empty() {
continue;
}
let bucket = auto_buckets.entry(key).or_insert_with(|| AutoBucket {
positions: Vec::new(),
has_dsl_forced: false,
epg_split: false,
});
bucket.positions.push(pos);
if dsl_forced {
bucket.has_dsl_forced = true;
}
}
}
auto_buckets
}
fn split_epg_buckets(
playlist: &[PlaylistGroup],
auto_buckets: IndexMap<String, AutoBucket>,
) -> IndexMap<String, AutoBucket> {
let mut split_buckets: IndexMap<String, AutoBucket> = IndexMap::new();
for (key, bucket) in auto_buckets {
if bucket.has_dsl_forced {
split_buckets.insert(key, bucket);
continue;
}
let mut epg_groups: IndexMap<String, Vec<(usize, usize)>> = IndexMap::new();
for &pos in &bucket.positions {
let ch = &playlist[pos.0].channels[pos.1];
if let Some(epg) = ch.header.epg_channel_id.as_deref() {
let trimmed = epg.trim().to_lowercase();
if !trimmed.is_empty() {
epg_groups.entry(trimmed).or_default().push(pos);
}
}
}
if epg_groups.len() >= 2 {
for (epg_id, sub_positions) in epg_groups {
let sub_key = format!("{key}|{epg_id}");
split_buckets
.insert(sub_key, AutoBucket { positions: sub_positions, has_dsl_forced: false, epg_split: true });
}
} else {
split_buckets.insert(key, bucket);
}
}
split_buckets
}
fn apply_auto_override(
bucket: &mut AutoBucket,
ao: &shared::model::AutoGroupOverrideDto,
playlist: &[PlaylistGroup],
member_pos_map: &HashMap<ChannelMemberRefDto, (usize, usize)>,
taken_positions: &HashSet<(usize, usize)>,
excluded_set: &HashSet<&ChannelMemberRefDto>,
) {
if !ao.remove.is_empty() {
bucket.positions.retain(|&pos| {
let ch = &playlist[pos.0].channels[pos.1];
!ao.remove.iter().any(|rem| {
ch.header.input_name.as_ref() == rem.input.as_str()
&& ch.header.get_input_stream_id().as_deref() == Some(rem.stream_id.as_str())
})
});
}
for add_ref in &ao.add {
if let Some(&pos) = member_pos_map.get(add_ref) {
if !taken_positions.contains(&pos) && !excluded_set.contains(add_ref) && !bucket.positions.contains(&pos) {
bucket.positions.push(pos);
}
}
}
if !ao.order.is_empty() {
let mut ordered = Vec::with_capacity(bucket.positions.len());
let mut used = HashSet::new();
for ord_ref in &ao.order {
if let Some(&pos) = bucket.positions.iter().find(|&&pos| {
let ch = &playlist[pos.0].channels[pos.1];
ch.header.input_name.as_ref() == ord_ref.input.as_str()
&& ch.header.get_input_stream_id().as_deref() == Some(ord_ref.stream_id.as_str())
}) {
if used.insert(pos) {
ordered.push(pos);
}
}
}
for &pos in &bucket.positions {
if !used.contains(&pos) {
ordered.push(pos);
}
}
bucket.positions = ordered;
}
}
fn build_group_metadata(
candidate: &GroupCandidate,
rep_header: &shared::model::PlaylistItemHeader,
slots: &[GroupSlot],
config: &ChannelGroupsConfig,
group_uuid: UUIDType,
) -> shared::model::PlaylistItemHeader {
let mut group_header = rep_header.clone();
group_header.uuid = group_uuid;
group_header.channel_group = None;
group_header.channel_group_uuid = Some(group_uuid);
if let Some(name) = &candidate.name_override {
group_header.name = name.as_str().intern();
group_header.title = name.as_str().intern();
} else if config.name_strip_quality {
group_header.name = strip_quality_words(&rep_header.name).intern();
group_header.title = strip_quality_words(&rep_header.title).intern();
}
if let Some(logo) = &candidate.logo_override {
group_header.logo = logo.as_str().intern();
} else if group_header.logo.is_empty() {
for &pos in &candidate.positions {
if let Some(item) = slots.get(pos.0).and_then(|g| g.channels.get(pos.1)).and_then(Option::as_ref) {
if !item.header.logo.is_empty() {
group_header.logo = item.header.logo.clone();
break;
}
}
}
}
if let Some(epg) = &candidate.epg_override {
group_header.epg_channel_id = Some(epg.as_str().intern());
} else if group_header.epg_channel_id.as_deref().is_none_or(|s| s.trim().is_empty()) {
for &pos in &candidate.positions {
if let Some(item) = slots.get(pos.0).and_then(|g| g.channels.get(pos.1)).and_then(Option::as_ref) {
if let Some(epg) = &item.header.epg_channel_id {
if !epg.trim().is_empty() {
group_header.epg_channel_id = Some(epg.clone());
break;
}
}
}
}
}
apply_archive_metadata(&mut group_header, candidate, slots);
group_header
}
fn apply_archive_metadata(
group_header: &mut shared::model::PlaylistItemHeader,
candidate: &GroupCandidate,
slots: &[GroupSlot],
) {
let mut has_archive = false;
let mut max_archive_duration: Option<i32> = None;
for &pos in &candidate.positions {
if let Some(item) = slots.get(pos.0).and_then(|g| g.channels.get(pos.1)).and_then(Option::as_ref) {
if let Some(StreamProperties::Live(live)) = &item.header.additional_properties {
if live.tv_archive.unwrap_or(0) > 0 {
has_archive = true;
if let Some(dur) = live.tv_archive_duration {
max_archive_duration = Some(max_archive_duration.map_or(dur, |m| m.max(dur)));
}
}
}
}
}
if has_archive {
match &mut group_header.additional_properties {
Some(StreamProperties::Live(live)) => {
live.tv_archive = Some(1);
live.tv_archive_duration = max_archive_duration;
}
_ => {
group_header.additional_properties = Some(StreamProperties::Live(Box::new(LiveStreamProperties {
name: group_header.name.clone(),
category_id: group_header.category_id,
stream_id: 0,
stream_icon: group_header.logo.clone(),
direct_source: "".intern(),
custom_sid: None,
added: None,
stream_type: None,
epg_channel_id: group_header.epg_channel_id.clone(),
tv_archive: Some(1),
tv_archive_duration: max_archive_duration,
is_adult: 0,
..Default::default()
})));
}
}
}
}
fn collect_group_candidates(
config: &ChannelGroupsConfig,
overrides: Option<&ChannelGroupOverridesDto>,
playlist: &[PlaylistGroup],
) -> Vec<GroupCandidate> {
let member_pos_map = build_member_index(playlist);
let mut taken_positions = HashSet::new();
let mut candidates = Vec::new();
resolve_manual_groups(config, overrides, &member_pos_map, &mut taken_positions, &mut candidates);
let excluded_set: HashSet<&ChannelMemberRefDto> =
overrides.map_or_else(HashSet::new, |o| o.excluded.iter().collect());
let mut auto_buckets = collect_auto_buckets(config, playlist, &excluded_set, &taken_positions);
if config.epg_conflict == EpgConflictMode::Split {
auto_buckets = split_epg_buckets(playlist, auto_buckets);
}
for (key, mut bucket) in auto_buckets {
let auto_override = overrides.and_then(|o| o.auto_overrides.iter().find(|ao| ao.auto_key == key));
let origin = if auto_override.is_some() { ChannelGroupOrigin::AutoEdited } else { ChannelGroupOrigin::Auto };
if let Some(ao) = auto_override {
apply_auto_override(&mut bucket, ao, playlist, &member_pos_map, &taken_positions, &excluded_set);
}
if bucket.positions.len() < MIN_GROUP_MEMBERS {
continue;
}
for &pos in &bucket.positions {
taken_positions.insert(pos);
}
candidates.push(GroupCandidate {
key: key.clone(),
origin,
policy: auto_override.and_then(|ao| ao.policy).unwrap_or(config.policy),
positions: bucket.positions,
orphaned: Vec::new(),
epg_split: bucket.epg_split,
name_override: auto_override.and_then(|ao| ao.name.clone()),
logo_override: auto_override.and_then(|ao| ao.logo.clone()),
epg_override: auto_override.and_then(|ao| ao.epg_channel_id.clone()),
playlist_group_override: auto_override.and_then(|ao| ao.playlist_group.clone()),
});
}
candidates
}
fn rebuild_playlist_from_slots(
slots: Vec<GroupSlot>,
deferred_placements: Vec<(String, PlaylistItem)>,
playlist: &mut Vec<PlaylistGroup>,
) {
for slot in slots {
let channels: Vec<PlaylistItem> = slot.channels.into_iter().flatten().collect();
if channels.is_empty() && slot.had_channels {
continue;
}
playlist.push(PlaylistGroup { id: slot.id, title: slot.title, channels, xtream_cluster: slot.xtream_cluster });
}
for (pg_override, mut item) in deferred_placements {
if let Some(g) = playlist.iter_mut().find(|g| g.title.as_ref() == pg_override.as_str()) {
item.header.category_id = g.id;
item.header.group = g.title.clone();
g.channels.push(item);
} else {
let next_id = playlist.iter().map(|g| g.id).max().unwrap_or(0) + 1;
item.header.category_id = next_id;
item.header.group = pg_override.as_str().intern();
playlist.push(PlaylistGroup {
id: next_id,
title: pg_override.as_str().intern(),
channels: vec![item],
xtream_cluster: XtreamCluster::Live,
});
}
}
}
fn materialize_groups(
candidates: &[GroupCandidate],
config: &ChannelGroupsConfig,
target_name: &str,
mut slots: Vec<GroupSlot>,
playlist: &mut Vec<PlaylistGroup>,
) -> Vec<PendingChannelGroup> {
let mut deferred_placements = Vec::new();
let mut pending_groups = Vec::with_capacity(candidates.len());
for candidate in candidates {
let rep_pos = candidate.positions[0];
let Some(rep_item) = slots.get(rep_pos.0).and_then(|g| g.channels.get(rep_pos.1)).and_then(Option::as_ref)
else {
continue;
};
let group_uuid = hash_string(&format!("channel-group|{target_name}|{}", candidate.key));
let group_header = build_group_metadata(candidate, &rep_item.header, &slots, config, group_uuid);
let mut members = Vec::with_capacity(candidate.positions.len());
for &pos in &candidate.positions {
if let Some(item) = slots.get_mut(pos.0).and_then(|g| g.channels.get_mut(pos.1)).and_then(Option::take) {
members.push(item);
}
}
let group_item = PlaylistItem { header: group_header };
if let Some(pg_override) = &candidate.playlist_group_override {
deferred_placements.push((pg_override.clone(), group_item));
} else if let Some(slot) = slots.get_mut(rep_pos.0).and_then(|g| g.channels.get_mut(rep_pos.1)) {
*slot = Some(group_item);
}
pending_groups.push(PendingChannelGroup {
group_uuid,
key: candidate.key.as_str().intern(),
origin: candidate.origin,
policy: candidate.policy,
members,
orphaned: candidate.orphaned.clone(),
epg_split: candidate.epg_split,
});
}
rebuild_playlist_from_slots(slots, deferred_placements, playlist);
pending_groups
}
pub(in crate::processor) fn group_playlist(
config: &ChannelGroupsConfig,
overrides: Option<&ChannelGroupOverridesDto>,
target_name: &str,
playlist: &mut Vec<PlaylistGroup>,
) -> Vec<PendingChannelGroup> {
if playlist.is_empty() {
return Vec::new();
}
let candidates = collect_group_candidates(config, overrides, playlist);
if candidates.is_empty() {
return Vec::new();
}
let slots: Vec<GroupSlot> = playlist
.drain(..)
.map(|g| GroupSlot {
id: g.id,
title: g.title,
xtream_cluster: g.xtream_cluster,
had_channels: !g.channels.is_empty(),
channels: g.channels.into_iter().map(Some).collect(),
})
.collect();
materialize_groups(&candidates, config, target_name, slots, playlist)
}

Some files were not shown because too many files have changed in this diff Show More