DVR Feature (#819)

feat: complete DVR and improve streaming, security, configuration, and UI

Complete the Digital Video Recorder subsystem and add a broad set of
reliability, security, streaming, configuration, processing, and Web UI
improvements across Tuliprox.

DVR:

* complete live recording and provider-aware VOD download support
* add recording queue, workers, scheduling, and recurring recording rules
* add conflict detection and capacity-aware scheduling
* add pause, resume, retry, edit, cancel, and delete workflows
* add recording quotas and configurable retention policies
* add crash recovery and startup reconciliation
* add durable lifecycle notifications with per-channel retries
* add DVR health monitoring and diagnostic tooling
* add secure access to recordings, thumbnails, and subtitles
* add WebSocket notifications for recording and rule changes
* add Web UI management for recordings, rules, progress, and task state
* add RBAC, configuration, documentation, and i18n support

Streaming and HLS:

* fix shared-stream idle handling and release dead provider streams correctly
* stop tee streams when both client and cache consumers are gone
* cancel provisioning probes when client streams terminate
* fix transient HLS origin work accounting and intermittent 503 responses
* make stream buffer byte limits configurable
* make shared subscriber idle timeout configurable
* make initial HLS manifest wait timeout configurable
* add configurable TS chunk packet count
* add configurable HLS refresh failure backoff
* centralize redirect limits and retry jitter handling
* improve provider DNS refresh behavior and failover tuning
* preserve UTF-8 characters in catchup templates
* improve stream history validation and persistence error handling

Security:

* use constant-time credential comparisons
* harden library and media path handling against traversal and symlink escapes
* only trust forwarded client IP headers from configured trusted proxies
* redact credentials and sensitive URL data from logs
* reject invalid authentication status-code configuration
* deny users with unresolved plans or invalid content filters
* improve authentication error handling across proxy and HLS endpoints

Configuration and reliability:

* prevent invalid api-proxy.yml reloads from terminating the running server
* fully validate API proxy configuration before persisting changes
* log configuration and EPG cleanup failures instead of silently discarding them
* keep the last valid configuration active after failed hot reloads
* align backend and shared media-server validation
* remove duplicated path and normalization logic
* improve DNS-store recovery and Windows rename fallback handling
* reject invalid duration, timestamp, and numeric conversions safely
* fix playlist bouquet save error handling
* fix provider record update detection
* fix cache boundary handling
* improve startup and persistence failure diagnostics

Filtering, search, sorting, and processing:

* add field-scoped playlist explorer search
* centralize shared stream-history search field definitions
* extend the filter DSL with string, set, and numeric operators
* add EPG ID, channel number, and detected quality as filterable fields
* add filter dry-run preview API with match statistics and samples
* report filter syntax errors with line and column information
* add natural numeric-aware sorting
* add quality-aware channel deduplication
* add accent-independent deduplication
* move natural sorting and quality detection helpers into shared code
* persist explorer search-field selection across reloads

User plans and content access:

* add reusable API user plans for capability tiers
* support inherited cluster and connection limits with per-user overrides
* add plan-level and user-level content filters
* enforce content filters across Xtream, M3U, direct playback, resource access,
  stream info, short EPG, categories, and XMLTV
* add trial plans with automatic expiry and Trial status
* add plan selection and content filtering to the user editor
* add full plan management to the API configuration Web UI
* migrate the API user database to schema V7 with plan and filter persistence

Web UI and accessibility:

* add live logging console to the stats page
* improve login error handling and prevent duplicate authentication requests
* add keyboard navigation to tabs, menus, tables, and search
* add ARIA roles, labels, validation state, and live-region feedback
* add confirmation dialogs for destructive actions
* add unsaved-change warnings and Ctrl/Cmd+S shortcuts
* add loading, progress, empty, and in-flight states across views
* improve dropdown and single-selection behavior
* add clipboard and credential-copy helpers
* persist table pagination and explorer search preferences
* improve error recovery when UI context providers are unavailable
* remove multiple panic-prone unwrap and browser API paths
* replace remaining hardcoded UI strings with translation keys

Maintenance:

* resolve backend and frontend compiler and Clippy warnings
* update packages and test fixtures
* consolidate duplicated helpers and validation logic
* improve documentation for configuration, filters, plans, DVR, and REST APIs
* add and update tests for migrations, filters, deduplication, sorting,
  configuration, streaming, and accessibility behavior
This commit is contained in:
euzu
2026-08-21 14:50:12 +02:00
committed by GitHub
parent df961e4bf7
commit 16d32d1298
270 changed files with 37731 additions and 2607 deletions
+1 -5
View File
@@ -74,11 +74,7 @@ pub fn is_cache_valid(status: &InputStatus, cluster: &str, cache_duration_second
return false;
}
let now = SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_secs();
if now > cluster_status.timestamp {
// check if age is within duration
// timestamp is creation time.
// wait, assuming timestamp is Last Update Time.
// now - timestamp < duration
if now >= cluster_status.timestamp {
return now - cluster_status.timestamp < cache_duration_seconds;
}
// Timestamp in future? Invalid.
+5 -1
View File
@@ -2,8 +2,9 @@ use crate::{
model::{
EPG_ATTRIB_CHANNEL, EPG_ATTRIB_ID, EPG_ATTRIB_LANG, EPG_TAG_CATEGORY, EPG_TAG_CHANNEL, EPG_TAG_DESC,
EPG_TAG_DISPLAY_NAME, EPG_TAG_ICON, EPG_TAG_LIVE, EPG_TAG_NEW, EPG_TAG_PROGRAMME, EPG_TAG_TITLE, EPG_TAG_TV,
EPG_TAG_PREVIOUSLY_SHOWN,
Epg, EpgSmartMatchConfig, IcsDummyPolicy, IcsEpgSourceConfig, PersistedEpgSource, PersistedEpgSourceKind,
TVGuide, XmlTag, XmlTagIcon,
TVGuide, XmlTag, XmlTagIcon
},
processing::{parser::ics, processor::EpgIdCache},
repository::{BPlusTree, BPlusTreeQuery, BPlusTreeUpdate, FlushPolicy},
@@ -210,6 +211,7 @@ impl TVGuide {
let mut categories = Vec::new();
let mut is_live = false;
let mut is_new = false;
let mut previously_shown = false;
if let Some(children) = tag.children.as_ref() {
for child in children {
match child.name.as_ref() {
@@ -229,6 +231,7 @@ impl TVGuide {
}
EPG_TAG_LIVE => is_live = true,
EPG_TAG_NEW => is_new = true,
EPG_TAG_PREVIOUSLY_SHOWN => previously_shown = true,
_ => {}
}
}
@@ -240,6 +243,7 @@ impl TVGuide {
programme.categories = categories;
programme.is_live = is_live;
programme.is_new = is_new;
programme.previously_shown = previously_shown;
Some(programme)
}
@@ -0,0 +1,191 @@
use shared::model::{DeduplicateConfig, DeduplicateKeep, DeduplicateMatchBy, PlaylistGroup, PlaylistItem, XtreamCluster};
use shared::utils::{deunicode_string, quality_rank, quality_tokens, token_quality};
use std::collections::HashMap;
/// Lowercased token join with quality tokens removed, so "News HD" and
/// "NEWS [FHD]" produce the same key.
fn normalized_dedup_key(value: &str, match_as_ascii: bool) -> String {
let value = if match_as_ascii { deunicode_string(value) } else { std::borrow::Cow::Borrowed(value) };
let mut key = String::with_capacity(value.len());
for token in quality_tokens(&value) {
if token_quality(token).is_some() {
continue;
}
if !key.is_empty() {
key.push(' ');
}
for ch in token.chars() {
key.extend(ch.to_lowercase());
}
}
key
}
fn match_value(config: DeduplicateConfig, item: &PlaylistItem) -> String {
normalized_dedup_key(raw_match_value(config, item), config.match_as_ascii)
}
fn raw_match_value(config: DeduplicateConfig, item: &PlaylistItem) -> &str {
let header = &item.header;
match config.match_by {
DeduplicateMatchBy::Caption => {
if header.title.is_empty() {
header.name.as_ref()
} else {
header.title.as_ref()
}
}
DeduplicateMatchBy::Name => header.name.as_ref(),
DeduplicateMatchBy::Title => header.title.as_ref(),
}
}
/// Collapse duplicate channels across the whole target playlist (per cluster).
/// Returns the number of removed channels. Empty match keys are never
/// deduplicated; ties keep the first occurrence in playlist order.
pub(in crate::processing::processor) fn deduplicate_playlist(
config: DeduplicateConfig,
playlist: &mut Vec<PlaylistGroup>,
) -> usize {
// winner per key: (quality rank, group index, channel index)
let mut winners: HashMap<(XtreamCluster, String), (u8, usize, usize)> = HashMap::new();
for (group_idx, group) in playlist.iter().enumerate() {
for (channel_idx, channel) in group.channels.iter().enumerate() {
let key_value = match_value(config, channel);
if key_value.is_empty() {
continue;
}
let rank = quality_rank(raw_match_value(config, channel));
match winners.entry((group.xtream_cluster, key_value)) {
std::collections::hash_map::Entry::Vacant(entry) => {
entry.insert((rank, group_idx, channel_idx));
}
std::collections::hash_map::Entry::Occupied(mut entry) => {
if config.keep == DeduplicateKeep::BestQuality && rank > entry.get().0 {
entry.insert((rank, group_idx, channel_idx));
}
}
}
}
}
let mut removed = 0;
// Only drop groups that deduplication emptied; groups that were already empty stay untouched.
let mut emptied_by_dedup = vec![false; playlist.len()];
for (group_idx, group) in playlist.iter_mut().enumerate() {
let cluster = group.xtream_cluster;
let before = group.channels.len();
let mut channel_idx = 0usize;
group.channels.retain(|channel| {
let idx = channel_idx;
channel_idx += 1;
let key_value = match_value(config, channel);
if key_value.is_empty() {
return true;
}
winners
.get(&(cluster, key_value))
.is_none_or(|(_, winner_group, winner_channel)| *winner_group == group_idx && *winner_channel == idx)
});
removed += before - group.channels.len();
emptied_by_dedup[group_idx] = before > 0 && group.channels.is_empty();
}
let mut group_idx = 0usize;
playlist.retain(|_| {
let keep = !emptied_by_dedup[group_idx];
group_idx += 1;
keep
});
removed
}
#[cfg(test)]
mod tests {
use super::*;
use shared::model::PlaylistItemHeader;
use shared::utils::Internable;
fn make_item(title: &str) -> PlaylistItem {
PlaylistItem { header: PlaylistItemHeader { title: title.intern(), ..Default::default() } }
}
fn make_group(title: &str, channels: Vec<PlaylistItem>) -> PlaylistGroup {
PlaylistGroup {
id: 1,
title: title.intern(),
channels,
xtream_cluster: XtreamCluster::Live,
}
}
#[test]
fn quality_rank_recognizes_tokens() {
assert_eq!(quality_rank("News [UHD]"), 5);
assert_eq!(quality_rank("News 1080p"), 3);
assert_eq!(quality_rank("News HD"), 2);
assert_eq!(quality_rank("News"), 0);
assert_eq!(quality_rank("HDTV News"), 0); // no partial token match
}
#[test]
fn dedup_keeps_best_quality() {
let mut playlist = vec![make_group(
"G",
vec![make_item("News HD"), make_item("News [FHD]"), make_item("News"), make_item("Sports HD")],
)];
let config =
DeduplicateConfig { match_by: DeduplicateMatchBy::Caption, keep: DeduplicateKeep::BestQuality, match_as_ascii: false };
let removed = deduplicate_playlist(config, &mut playlist);
assert_eq!(removed, 2);
let titles: Vec<_> = playlist[0].channels.iter().map(|c| c.header.title.to_string()).collect();
assert_eq!(titles, vec!["News [FHD]", "Sports HD"]);
}
#[test]
fn dedup_keep_first_preserves_playlist_order_winner() {
let mut playlist =
vec![make_group("G", vec![make_item("News HD"), make_item("News [FHD]")])];
let config = DeduplicateConfig { match_by: DeduplicateMatchBy::Caption, keep: DeduplicateKeep::First, match_as_ascii: false };
let removed = deduplicate_playlist(config, &mut playlist);
assert_eq!(removed, 1);
assert_eq!(playlist[0].channels[0].header.title.as_ref(), "News HD");
}
#[test]
fn dedup_ignores_empty_keys_and_drops_empty_groups() {
let mut playlist = vec![
make_group("Empty", vec![]),
make_group("A", vec![make_item("News HD")]),
make_group("B", vec![make_item("News FHD"), make_item("")]),
];
let config =
DeduplicateConfig { match_by: DeduplicateMatchBy::Caption, keep: DeduplicateKeep::BestQuality, match_as_ascii: false };
let removed = deduplicate_playlist(config, &mut playlist);
assert_eq!(removed, 1);
// "A" was emptied by dedup and dropped, the already-empty group survives
assert_eq!(playlist.len(), 2);
assert_eq!(playlist[0].title.as_ref(), "Empty");
assert_eq!(playlist[1].title.as_ref(), "B");
// the empty-caption item has no dedup key and is retained
let titles: Vec<_> = playlist[1].channels.iter().map(|c| c.header.title.to_string()).collect();
assert_eq!(titles, vec!["News FHD", ""]);
}
#[test]
fn dedup_match_as_ascii_collapses_accented_names() {
let mut playlist =
vec![make_group("G", vec![make_item("Café HD"), make_item("Cafe FHD")])];
let config =
DeduplicateConfig { match_by: DeduplicateMatchBy::Caption, keep: DeduplicateKeep::BestQuality, match_as_ascii: true };
let removed = deduplicate_playlist(config, &mut playlist);
assert_eq!(removed, 1);
assert_eq!(playlist[0].channels[0].header.title.as_ref(), "Cafe FHD");
// without the flag the accented name stays distinct
let mut playlist =
vec![make_group("G", vec![make_item("Café HD"), make_item("Cafe FHD")])];
let config =
DeduplicateConfig { match_by: DeduplicateMatchBy::Caption, keep: DeduplicateKeep::BestQuality, match_as_ascii: false };
assert_eq!(deduplicate_playlist(config, &mut playlist), 0);
}
}
+1
View File
@@ -5,6 +5,7 @@ mod xtream;
// mod affix;
mod xtream_vod;
mod xtream_series;
mod deduplicate;
mod epg;
mod sort;
mod trakt;
@@ -1595,6 +1595,18 @@ async fn finalize_prepared_target(
step.tick("playlist merge");
log_memory_snapshot(format!("target '{}' after_playlist_merge", target.name).as_str());
if let Some(dedup_config) = target.options.as_ref().and_then(|options| options.deduplicate.as_ref()) {
let removed = crate::processing::processor::deduplicate::deduplicate_playlist(
*dedup_config,
&mut flat_new_playlist,
);
if removed > 0 {
info!("Deduplicated {removed} channels for target {}", target.name);
}
step.tick("playlist dedup");
log_memory_snapshot(format!("target '{}' after_playlist_dedup", target.name).as_str());
}
if sort_playlist(target, &mut flat_new_playlist) {
step.tick("playlist sort");
log_memory_snapshot(format!("target '{}' after_playlist_sort", target.name).as_str());
+59 -6
View File
@@ -1,6 +1,7 @@
use crate::model::{ConfigSortRule, ConfigTarget};
use shared::foundation::ValueProvider;
use shared::model::{PlaylistGroup, SortOrder, SortTarget};
use shared::utils::natural_cmp;
use std::cmp::Ordering;
use std::sync::Arc;
use crate::utils::normalized_source_ordinal;
@@ -13,6 +14,14 @@ fn apply_sort_order(order: SortOrder, ordering: Ordering) -> Ordering {
}
}
fn compare_values(natural: bool, left: &str, right: &str) -> Ordering {
if natural {
natural_cmp(left, right)
} else {
left.cmp(right)
}
}
fn parse_capture_group_rank(name: &str) -> Option<u32> {
let suffix = name.strip_prefix('c')?;
if suffix.is_empty() || !suffix.bytes().all(|c| c.is_ascii_digit()) {
@@ -95,7 +104,7 @@ fn evaluate_sequence(plan: &SequencePlan, value: &str) -> SequenceMatch {
SequenceMatch::Unmatched
}
fn compare_sequence_match(a: &SequenceMatch, b: &SequenceMatch, order: SortOrder) -> Ordering {
fn compare_sequence_match(a: &SequenceMatch, b: &SequenceMatch, order: SortOrder, natural: bool) -> Ordering {
match (a, b) {
(
SequenceMatch::Matched { sequence_idx: idx_a, captures: captures_a },
@@ -110,7 +119,7 @@ fn compare_sequence_match(a: &SequenceMatch, b: &SequenceMatch, order: SortOrder
let capture_count = captures_a.len().max(captures_b.len());
for index in 0..capture_count {
let ord = match (captures_a.get(index), captures_b.get(index)) {
(Some(Some(v1)), Some(Some(v2))) => v1.cmp(v2),
(Some(Some(v1)), Some(Some(v2))) => compare_values(natural, v1, v2),
(Some(Some(_)), Some(None) | None) => Ordering::Greater,
(Some(None) | None, Some(Some(_))) => Ordering::Less,
_ => Ordering::Equal,
@@ -142,11 +151,13 @@ fn compare_rule_entries(rule: &PreparedRule, left: &RuleCacheEntry, right: &Rule
(Some(value_left), Some(value_right)) => {
if rule.sequence_plan.is_some() {
match (&left.sequence_match, &right.sequence_match) {
(Some(seq_left), Some(seq_right)) => compare_sequence_match(seq_left, seq_right, rule.rule.order),
(Some(seq_left), Some(seq_right)) => {
compare_sequence_match(seq_left, seq_right, rule.rule.order, rule.rule.natural)
}
_ => Ordering::Equal,
}
} else {
apply_sort_order(rule.rule.order, value_left.cmp(value_right))
apply_sort_order(rule.rule.order, compare_values(rule.rule.natural, value_left, value_right))
}
}
}
@@ -239,7 +250,7 @@ fn playlist_comparator(
let plan = SequencePlan::new(sequence);
let left = evaluate_sequence(&plan, value_a);
let right = evaluate_sequence(&plan, value_b);
compare_sequence_match(&left, &right, order)
compare_sequence_match(&left, &right, order, false)
} else {
apply_sort_order(order, value_a.cmp(value_b))
}
@@ -270,7 +281,7 @@ fn compare_cached_rule_entries(
let left = &cache[left_idx];
let right = &cache[right_idx];
if let (Some(va), Some(vb)) = (&left.value, &right.value) {
let fallback = apply_sort_order(rule.rule.order, va.cmp(vb));
let fallback = apply_sort_order(rule.rule.order, compare_values(rule.rule.natural, va, vb));
if fallback != Ordering::Equal {
return fallback;
}
@@ -419,6 +430,7 @@ mod tests {
target: SortTarget::Channel,
field: ItemField::Caption,
order: SortOrder::Asc,
natural: false,
sequence: Some(vec![
shared::model::REGEX_CACHE.get_or_compile(r"(?P<c1>.*?)\bUHD\b").unwrap(),
shared::model::REGEX_CACHE.get_or_compile(r"(?P<c1>.*?)\bFHD\b").unwrap(),
@@ -503,6 +515,7 @@ mod tests {
target: SortTarget::Channel,
field: ItemField::Caption,
order: SortOrder::Asc,
natural: false,
sequence: Some(vec![
shared::model::REGEX_CACHE.get_or_compile(r"^US\| EAST.*?\[\bUHD\b\](?P<c1>.*)").unwrap(),
shared::model::REGEX_CACHE.get_or_compile(r"^US\| EAST.*?\[\bFHD\b\](?P<c1>.*)").unwrap(),
@@ -603,6 +616,7 @@ mod tests {
target: SortTarget::Channel,
field: ItemField::Caption,
order: SortOrder::Desc,
natural: false,
sequence: Some(vec![shared::model::REGEX_CACHE.get_or_compile(r"^A-(?P<c1>\d+)$").unwrap()]),
filter: Filter::default(),
};
@@ -636,6 +650,7 @@ mod tests {
target: SortTarget::Channel,
field: ItemField::Caption,
order: SortOrder::Asc,
natural: false,
// Both values match the same sequence item and produce equal sequence priority.
sequence: Some(vec![shared::model::REGEX_CACHE.get_or_compile(r"^A-\d+-.$").unwrap()]),
filter: Filter::default(),
@@ -644,6 +659,7 @@ mod tests {
target: SortTarget::Channel,
field: ItemField::Caption,
order: SortOrder::Desc,
natural: false,
sequence: None,
filter: Filter::default(),
};
@@ -677,6 +693,7 @@ mod tests {
target: SortTarget::Group,
field: ItemField::Caption,
order: SortOrder::Asc,
natural: false,
// Both groups match the same sequence item and produce equal sequence priority.
sequence: Some(vec![shared::model::REGEX_CACHE.get_or_compile(r"^A-\d+-.$").unwrap()]),
filter: Filter::default(),
@@ -719,6 +736,7 @@ mod tests {
target: SortTarget::Group,
field: ItemField::Caption,
order: SortOrder::Asc,
natural: false,
// Both groups have the same sequence/rule priority and same caption value.
sequence: Some(vec![shared::model::REGEX_CACHE.get_or_compile(r"^Same Caption$").unwrap()]),
filter: Filter::default(),
@@ -751,6 +769,7 @@ mod tests {
target: SortTarget::Channel,
field: ItemField::Caption,
order: SortOrder::None,
natural: false,
sequence: Some(vec![
shared::model::REGEX_CACHE.get_or_compile(r"^UHD$").unwrap(),
shared::model::REGEX_CACHE.get_or_compile(r"^FHD$").unwrap(),
@@ -795,6 +814,7 @@ mod tests {
target: SortTarget::Group,
field: ItemField::Caption,
order: SortOrder::None,
natural: false,
sequence: Some(vec![
shared::model::REGEX_CACHE.get_or_compile(r"^UHD$").unwrap(),
shared::model::REGEX_CACHE.get_or_compile(r"^FHD$").unwrap(),
@@ -829,6 +849,7 @@ mod tests {
target: SortTarget::Channel,
field: ItemField::Caption,
order: SortOrder::None,
natural: false,
sequence: Some(vec![]),
filter: Filter::default(),
};
@@ -840,4 +861,36 @@ mod tests {
let expected = vec!["B", "A"].into_iter().map(Into::into).collect::<Vec<Arc<str>>>();
assert_eq!(expected, sorted);
}
#[test]
fn test_natural_sort_orders_embedded_numbers_numerically() {
let channels: Vec<PlaylistItem> = vec!["Chan 10", "Chan 2", "Chan 1", "Chan 002"]
.into_iter()
.enumerate()
.map(|(i, title)| PlaylistItem {
header: PlaylistItemHeader {
title: title.to_string().into(),
source_ordinal: u32::try_from(i + 1).unwrap(),
..Default::default()
},
})
.collect();
let channel_sort = ConfigSortRule {
target: SortTarget::Channel,
field: ItemField::Caption,
order: SortOrder::Asc,
natural: true,
sequence: None,
filter: Filter::default(),
};
let mut groups = vec![make_group(1, "G1", channels)];
sort_channels_in_groups(groups.as_mut_slice(), &[channel_sort], false);
let sorted = groups[0].channels.iter().map(|pli| pli.header.title.clone()).collect::<Vec<_>>();
let expected =
vec!["Chan 1", "Chan 2", "Chan 002", "Chan 10"].into_iter().map(Into::into).collect::<Vec<Arc<str>>>();
assert_eq!(expected, sorted);
}
}