mirror of
https://github.com/euzu/tuliprox.git
synced 2026-10-08 17:02:22 +02:00
84f66a07b8
* refactor(repository): return concrete playlist streams instead of Box<dyn Stream>
The three raw-playlist iterators each wrapped an already-concrete type in a
trait object: m3u produced a LockedReceiverStream and xtream/stalker produced a
ReceiverStream, then boxed it behind `dyn Stream + Send + Unpin`. Nothing in the
workspace stored these heterogeneously or needed object safety, so the box was
one allocation and one vtable per call for no benefit.
Name the concrete types in the signatures instead:
iter_raw_m3u_{target,input}_playlist -> LockedReceiverStream<Result<M3uPlaylistItem, _>>
iter_raw_xtream_{target,input}_playlist -> ReceiverStream<Result<XtreamPlaylistItem, _>>
iter_stalker_{items,series_roots} -> ReceiverStream<StalkerPlaylistItem>
Both concrete types are Stream + Unpin, so every caller keeps compiling
unchanged; the `futures::Stream` / `tokio_stream::Stream` imports are now unused
in two of the three files and are dropped.
Verified with cargo check on tuliprox-repository and tuliprox plus nightly fmt.
Not verified by tests.
* refactor(repository): extract PlaylistBackend so M3U and Xtream share one iterator
iter_raw_m3u_playlist and iter_raw_xtream_playlist were two transcriptions of
one design: the same two read locks, the same spawn_blocking producer feeding a
bounded channel, the same sorted-index reader with per-entry error logging,
differing only in the item type, the storage subdirectory and which
TuliproxError variant wrapped a failure. The storage-path pair
(m3u|xtream)_get_storage_path and ensure_(m3u|xtream)_storage_path were
identical modulo one const and one error constructor.
New `playlist_backend` module names those differences as associated types and
consts, so the shared body is written once:
trait PlaylistBackend { type Item; type SortKey; SUBDIR; LABEL;
HOLD_ITER_LOCK; repo_error(); storage_path() }
struct M3u; struct Xtream; // zero-sized markers
iter_raw_playlist<B, K, F>(..) -> Option<LockedReceiverStream<..>>
Dispatch is entirely static: the markers are ZSTs, the item filter is an
`impl Fn` monomorphized per call site, and the B+Tree key stays a function
parameter because M3U keys its target store by u32 and its input store by
Arc<str> — so the key is not a property of the backend.
Two deliberate non-unifications:
* Stalker is not a PlaylistBackend. Its store is keyed by input rather than
target, has no sorted index, and yields bare items rather than Results.
Folding it in would add methods that exist only to return None for one of
three implementors.
* HOLD_ITER_LOCK preserves an existing behavioural difference rather than
normalising it. M3U holds a read lock for the consumer's lifetime; Xtream
never did. Changing when that lock is released would change what a concurrent
writer can do mid-iteration, which is not this commit's business.
The only behaviour change is that M3U now also logs a reader-open failure
before forwarding it, which Xtream already did.
Net: 151 lines of duplicated logic in the two repositories replaced by one
shared implementation. Verified with cargo check --workspace and nightly fmt.
Not verified by tests.
* refactor(repository): replace boxed playlist iterators with concrete enums
PlaylistSourceOps is a private trait that is never used as a trait object --
PlaylistSource has always dispatched through the PlaylistSourceKind enum, and no
Box<dyn PlaylistSourceOps> exists anywhere in the workspace. Its three iterator
methods nevertheless returned Box<dyn Iterator<..> + Send + '_>, so the hottest
traversal in the repository paid one allocation per call plus one indirect,
uninlinable call per playlist item. The skip-set filter boxed a second time on
top of that.
The erasure could not simply be deleted: the bodies were composed adapter chains
(flat_map(..).map(..)) whose closure types cannot be named. New `playlist_items`
module replaces them with hand-rolled state machines whose types *are* nameable:
BTreeValues<'a, K, V> one B+Tree store, logging and skipping
unreadable entries (was filter_map(..))
BTreeStores<'a, K, V, const N> N stores back to back (was chain(..)):
N=1 single-file, 3 Xtream, 4 Stalker
MemoryDrain / MemoryItemsMut / MemoryItems the in-memory flat_map shapes
SourceItems / SourceCowItems / SourceItemsMut one variant per source kind
ClusterFiltered<I> skip-set filter, folded in rather than
wrapped -- a filtered traversal now
allocates nothing at all
PlaylistSource::into_items() -> ClusterFiltered<SourceItems<'_>>
Also in this commit, since all of it is the same erasure:
* update_playlist and obtain_resources drop BoxFuture for `async fn` (AFIT,
stable since 1.75). Static dispatch means auto-traits still propagate.
* The two Vec<(XtreamCluster, Box<dyn Iterator<..>>)> locals in take_groups
become Vec<(XtreamCluster, BTreeValues<..>)> -- every reader in those vecs
already had the same concrete type.
* A `dispatch!`/`dispatch_await!` macro pair replaces the hand-written
forwarding arms, and doubles as the conformance check the trait used to
provide: a variant missing a method fails to compile at the macro.
playlist_source.rs now contains zero `Box<dyn` and zero BoxFuture. Behaviour is
preserved, including the read-only warning on disk sources (now emitted where
the empty iterator is built) and the verbatim B+Tree skip-entry log message.
FetchedPlaylist forwards the concrete types rather than re-boxing them.
Verified with cargo check --workspace and nightly fmt, plus a structural check
that all five PlaylistSourceOps impls still carry their original method sets.
Not verified by tests.
* refactor(shared): key field access on a typed enum instead of a string
PlaylistItemHeader had two field-access paths and only one of them was cheap.
The filter engine in foundation/value_provider.rs matched on the typed
ItemField enum and returned borrowed &str. The mapper, sort and counter paths
went through FieldGetAccessor::get_field(&str), which walked a chain of ~20
eq_ignore_ascii_case comparisons and returned an owned Arc<str> -- so `chno`
and `type` did `.to_string().intern()`, a heap allocation *and* an interner
write lock, on every read, per item, per rule.
Adds the typed accessor the enum path deserved:
HeaderField every field addressable by name, with parse()/as_str()
FieldRef<'a> Shared(&Arc<str>) | Str(&str) | Num(u32) -- a read
never forces an allocation
FieldGet / FieldSet typed traits; the impl is a match on a discriminant
genre_ref! borrowing sibling of get_genre!
HeaderField is deliberately *wider* than ItemField rather than an extension of
it. ItemField is the config-facing vocabulary, and widening it would widen what
a user may write in a filter or sort rule; the accessor's domain is genuinely
larger (logo_small, audio_track and friends are addressable by name by the M3U
resource endpoint without being valid config fields). ItemField::header_field()
maps between them, returning None for Quality, which is derived from the caption
rather than stored.
FieldGetAccessor/FieldSetAccessor are kept as #[inline] shims over the typed
traits, so callers whose field name genuinely arrives as a string keep working
and keep their exact previous behaviour, interning of numbers included.
Migrated every call site that already held type information:
* sort.rs held an ItemField and called provider.get(field.as_ref()) -- turning
a typed enum into a string to do a linear string lookup. Now get_typed().
* MappingCounter.field becomes HeaderField, parsed once in prepare() instead of
re-parsed per channel inside the counter loop.
* trakt.rs's "caption" literals become HeaderField::Caption.
Left on the &str path deliberately: M3uPlaylistItem and XtreamPlaylistItem keep
their own string accessors. Their only caller is the M3U resource endpoint,
where the field name arrives in the URL and the lookup happens once per request,
not per item.
Behaviour is preserved throughout, including which fields are settable (input,
type and provider_id still return false) and which are absent per type (`id` is
a header field, `provider_id` an M3U one).
Verified with cargo check --workspace --all-targets and nightly fmt.
Not verified by tests.
* refactor(config): give HLS timing config units in the type system
Roughly 245 config and state fields across the workspace encode their unit as a
naming convention, and several sit adjacent inside the same struct.
HlsCacheConfigDto is the clearest case: origin_manifest_timeout_ms and
origin_segment_timeout_ms sit two lines above initial_manifest_wait_timeout_secs,
all four u64, nothing stopping a millisecond value reaching a seconds parameter.
Worse, cache_duration and session_idle_timeout do not carry the unit in their
names at all -- both are seconds, which is only discoverable from their defaults.
Adds Millis and Secs in shared/src/model/config/time_units.rs:
#[repr(transparent)] same layout as the u64 they replace -- no size cost,
no indirection, no allocation
#[serde(transparent)] same serialized form, so the YAML/JSON shape is
byte-identical and this lands with no config migration
and nothing user-visible
Conversions are methods, not From impls: `Secs -> Millis` is a multiplication
that can overflow, and forcing it to be spelled out at the call site is the
point of having the types. `as_duration_at_least_1ms` collects the `.max(1)`
that several origin deadlines applied individually, because they treat a zero
timeout as "do not wait" rather than "wait forever".
Applied to the six HLS timing fields on HlsCacheConfigDto and
HlsSegmentRepairConfigDto and, importantly, to their runtime counterparts in
tuliprox-core's HlsCacheConfig, so the type survives the Dto -> runtime hop
rather than being unwrapped at the boundary. The validation helper
ensure_min_u64 splits into ensure_min_millis / ensure_min_secs so a bound cannot
be compared against the wrong unit either.
Two manual secs->ms multiplications in backend/hls (gc.rs and manager.rs's
transient_resource_ttl_ms) become `.as_millis()`, which is where the conversion
was silently open-coded before.
Deliberate boundary: the structs inside backend/hls that do arithmetic on raw
millisecond counts keep u64 and take `.get()` at the edge. Pushing the newtypes
through those is the next slice; doing it here would have made this commit touch
most of the crate for no additional safety at the config surface.
Includes tests asserting the serialized form is a bare number, that layout is
transparent over u64 (Option<Millis> included), and that Secs::as_millis
saturates.
Verified with cargo check --workspace --all-targets and nightly fmt.
Not verified by tests.
* chore(lint): satisfy the nightly clippy gate after the static-dispatch work
Three findings from `cargo +nightly-2026-05-01 clippy --workspace -- -D warnings`:
* doc_markdown on a macro doc comment in shared/src/model/playlist.rs.
* clone_on_copy in the frontend's mapper counter view, now that
MappingCounter::field is a Copy HeaderField rather than a String.
* large_enum_variant on SourceItems and SourceCowItems, allowed with a
justification rather than fixed. The B+Tree disk iterators carry sizeable
cursor state, so holding three or four inline makes the enum ~1.9KB, and
clippy's suggested fix -- boxing the large variants -- would reintroduce
exactly the per-traversal heap allocation the type exists to remove. The cost
is stack space for one value per traversal, not per item: the enum is built
once and then driven through &mut, so it is never copied per element.
Gates now green: cargo build --workspace, clippy --workspace and
clippy --workspace --all-targets at -D warnings, fmt --all --check, and
bin/check-workspace-deps.sh (78 edges).
* refactor(shared): split TuliproxError into a Copy kind and a message
All 50 variants carried exactly one String and nothing else, so this was never
an enum of errors -- it was a category tag beside a message. Encoding it as an
enum cost a 50-arm `message()` whose only job was to return the payload, plus a
second 50-name list in `is_notify()` that had to be kept in sync by hand, and
made the category impossible to compare, store or return on its own.
enum ErrorKind { Config, ConfigApp, ... } // fieldless, Copy, Eq, Hash
struct TuliproxError { kind: ErrorKind, message: Box<str> }
An `error_kinds!` macro declares each category once -- variant, display label
and constructor -- so the label table and the notify set can no longer drift
apart. `message()` is now one field access, `is_notify()` delegates to the kind,
and `kind()` exposes the category directly.
The 755 construction sites are untouched. A tuple-variant constructor and an
associated function are invoked with identical syntax, so `TuliproxError::
Config(msg)` still compiles once `Config` is an `#[allow(non_snake_case)]`
associated fn. The non-snake-case names are the deliberate price of splitting
the type without rewriting every call site in one commit; renaming them to
`config()` style is a separate, mechanical follow-up.
Only the 10 sites that pattern-matched on a variant needed changing, and they
read better for it -- `err.kind() == ErrorKind::ApiXtream` instead of
`matches!(err, TuliproxError::ApiXtream(_))`, and for the two that also
inspected the payload, an explicit `kind()` plus `message().contains(..)`.
Two incidental improvements: the constructors take `impl Into<Box<str>>`, so
call sites holding a `&str` no longer need `.to_string()` (two test sites drop a
now-ambiguous `.into()`); and the error is one word smaller than the enum was.
Tests pin the behaviour that had to be preserved: the `"label: message"` Display
shape for four categories, that `message()` excludes the label, that both &str
and String are accepted, and the exact set of notifying categories.
Verified with cargo check --workspace --all-targets, clippy --all-targets at
-D warnings, nightly fmt, and the new unit tests.
* refactor(core): add a Clock seam and collapse 14 copies of current_time_millis
`fn current_time_millis() -> u64 { chrono::Utc::now().timestamp_millis()
.try_into().unwrap_or_default() }` was duplicated character for character in 14
files across backend/app and backend/hls. New tuliprox-core::utils::clock holds
the single copy, plus the trait that makes "what time is it" injectable:
trait Clock { fn now_ms(&self) -> Millis }
struct SystemClock; // ZST: no field, no vtable, no allocation
struct ManualClock(Arc<AtomicU64>); // deterministic; the Arc is confined here
Clock is meant to be held as a generic parameter defaulted to the ZST --
`struct Deadlines<C: Clock = SystemClock>` -- never as Arc<dyn Clock>. A test
asserts SystemClock is zero-sized and that owning one leaves a struct's layout
unchanged, which is the property that makes the seam free.
Scope is smaller than it first looked, and the reason is worth recording: most
of the deadline logic in tuliprox-hls already takes `now_ms: u64` as a function
parameter. HlsAccessLease, HlsAvailabilityReevaluationCycle and their neighbours
are already injectable and already tested that way, and passing the instant in
is a better pattern than reaching for a clock -- so they are left alone. The
trait is for the callers that have to *produce* the instant.
HlsTerminalCommitClock is deliberately left alone too. It fakes time with an
AtomicU64 sentinel, costing an atomic load per production read, and would be a
natural fit -- but it is owned by HlsProxy, so making it generic would push a
type parameter onto HlsProxy and from there onto AppState. A type parameter that
reaches the root state is exactly the case where the status quo wins; the module
docs say so, so the next person does not have to rediscover it.
The 14 call sites now import the shared function, preserving each one's original
visibility (two were pub(super) and re-export as such).
Verified with cargo check --workspace --all-targets, clippy --all-targets at
-D warnings, nightly fmt, and 4 new unit tests.
* refactor(shared): give config preparation one shape via a Prepare trait
Config types are deserialized first and resolved second -- templates expanded,
regexes compiled, filters parsed, derived fields computed. That second phase was
spread across ~98 inherent `prepare`/`validate` methods with no agreed
signature: some took nothing, some `Option<&[PatternTemplate]>`, some a storage
dir, a device number, a port or an `include_computed: bool`, returning variously
`()`, `Result<(), TuliproxError>`, `Result<(), &'static str>` or `bool`.
Because the shape was invisible, the recursive walk had to be hand-written at
every level, and a new config struct that forgot to call its children's prepare
failed silently at runtime rather than at compile time.
trait Prepare { type Ctx<'a>: Copy; fn prepare(&mut self, Self::Ctx<'_>) -> Result<(), TuliproxError> }
The context is an associated type, so a node needing pattern templates and a
node needing a port are both implementors without a lowest-common-denominator
argument list, and dispatch stays static -- the GAT is monomorphized per
implementor, with no trait object anywhere.
Migrated the 14 methods that already shared the exact signature
`prepare(&mut self, Option<&[PatternTemplate]>) -> Result<(), TuliproxError>`:
MapperOperation, MapperDto, MappingDto, MappingDefinitionDto, MappingsDto,
ConfigRenameDto, ConfigSortRuleDto, ConfigSortDto, ConfigFavouritesDto,
ConfigInputOptionsDto, and the four target-output types. `Ctx<'a>` is
`Option<&'a [PatternTemplate]>` for all of them, so they now visibly share a
contract instead of coincidentally sharing an argument list.
Blanket impls for Vec, slices, Option and Box are the payoff: two hand-written
child walks are gone, including a nested `Option<Vec<MapperDto>>` that took an
`if let` plus a `for` and is now one line.
Two things deliberately left alone:
* The `handle_tuliprox_error_result_list!` call sites in sort.rs and target.rs
aggregate every child's error rather than stopping at the first. Switching
them to the blanket impl would silently reduce a config report listing all
problems to one naming only the first. Aggregation is the better behaviour for
config validation, so those walks stay until Prepare can express it.
* The loop in ConfigTargetDto that prepares each output also counts output kinds
as it goes, so the prepare call is incidental to a loop doing more.
Includes 5 unit tests covering the blanket impls: same context to every child,
short-circuit on first failure matching the previous `?` behaviour, absent
Option as a no-op, nesting, and Box forwarding.
Verified with cargo check --workspace --all-targets, clippy --all-targets at
-D warnings, nightly fmt, and 970 tests across shared, core and config-loader.
* refactor(shared): add PrepareAll so config errors aggregate again
Prepare's collection impls short-circuit on the first failing child, which is
right for a nested walk but wrong for config validation: a user with three bad
sort rules should hear about all three in one pass, not fix them one round-trip
at a time. That is why the two remaining hand-written walks in sort.rs and
target.rs still used handle_tuliprox_error_result_list! after the Prepare
migration -- switching them to the blanket impl would have silently degraded a
config report to its first line. This closes that gap.
trait PrepareAll: Prepare { fn prepare_all(&mut self, Self::Ctx<'_>) -> Result<(), TuliproxError> }
Aggregation is byte-identical to the macro's: collect each failure's rendered
message, join with newlines, wrap in TuliproxError::Errors. Impls for Vec, [T],
Option and Box, so `Option<Vec<ConfigRenameDto>>` nests without a hand-written
`if let` plus loop -- target.rs's three-line walk becomes one line.
With both call sites converted the macro is dead, so it is removed along with
its re-export. `get_errors_notify_message!` next to it is still used by the
playlist processor and stays.
Three tests cover what the macro guaranteed: every failing child reported (not
just the first), every child still run despite earlier failures, and nesting
through Option.
Verified with cargo check --workspace --all-targets, clippy --all-targets at
-D warnings, nightly fmt, and 973 tests across shared, core and config-loader.
* refactor(shared): drop the field-accessor macros for explicit typed impls
Three macros generated the by-name field accessors. My earlier read of them as
"near-identical" was wrong in one important way, and the correction is the most
valuable part of this commit.
XtreamPlaylistItem's accessor is not a variant of the other two: it carries a
~80-line prefix-matched lookup into additional_properties for Xtream cover and
backdrop_path resources. It also has **no callers**. FieldGetAccessor is never a
generic bound and never a trait object, and the only call site in the workspace
is the M3U resource endpoint calling M3uPlaylistItem::get_field. So the whole
impl was unreachable; it is deleted rather than ported. The XC_PROP_* constants
it referenced are still used elsewhere, and the logic is in git history if the
Xtream resource endpoint ever needs cover-by-name.
The remaining two are collapsed onto the typed HeaderField/FieldGet/FieldRef
path from the earlier field-access work, written out explicitly instead of
macro-generated. A macro that exists to share ten repetitive match arms between
two types is not paying for itself: the explicit arms are the same length and
you can read them without expanding anything. M3uPlaylistItem also stops walking
a chain of eq_ignore_ascii_case comparisons and stops interning `chno` on every
read.
One behaviour is deliberately preserved rather than "fixed": M3uPlaylistItem
carries input_name, item_type and additional_properties, but none of them were
ever addressable by name, and the M3U resource endpoint resolves a URL path
segment through get_field -- so making them resolvable would turn a 404 into a
response. The explicit `Id | Input | Type | Genre => None` arm says so, and a
test asserts it.
Net 104 lines removed from playlist.rs, and zero accessor macros remain.
Two new tests: every HeaderField variant round-trips through parse/as_str
case-insensitively (with the epg_id alias), and the M3U item resolves
provider_id, name, chno and caption while refusing the header-only names.
Verified with cargo check --workspace --all-targets, clippy --all-targets at
-D warnings, nightly fmt, and the shared test suite.
* refactor(config): resolve ByteSize into a Bytes newtype once
`ByteSize(String)` was parsed with `parse_bytes() -> Result<u64, String>` at 15
sites, and the parsed value was typed as a bare u64 wherever it landed -- so a
resolved size was indistinguishable from any other u64, and a runtime struct
could hold either the string or the number with nothing marking which.
Adds `Bytes(u64)` beside `ByteSize`: #[repr(transparent)], #[serde(transparent)],
Copy. `parse_bytes()` now returns it, and the runtime structs hold it:
HlsCacheConfig::{cache_bytes, cache_bytes_per_session} u64 -> Bytes
FfprobeConfig::{probe_size_bytes, live_probe_size_bytes} u64 -> Bytes
`Bytes::at_least_1()` collects the `.max(1)` that probe sizes applied
individually, because they treat 0 as "unset" rather than "no bytes".
One correction to how I pitched this. I described FfprobeConfig's parallel
`probe_size: ByteSize` / `probe_size_bytes: u64` fields as redundant. They are
not: `From<&FfprobeConfig> for FfprobeConfigDto` reads the string form to send
the user's own spelling back to the web UI, so someone who wrote `10MB` sees
`10MB` rather than 10485760. Both fields stay, and a test records why so the
string one does not get "cleaned up" later.
Nor is this a performance change -- the parses happened at config load, not per
request. The win is that the resolved value now has a type: `Bytes` cannot be
passed where some unrelated u64 is wanted, and a runtime struct can no longer
hold an unparsed size by accident. Boundaries that genuinely need the number
(the ffmpeg CLI arg, the HLS cache-limit setter, the GC policy) take `.get()`
explicitly, as with Millis/Secs.
Two tests: transparent layout over u64 including Option, at_least_1 flooring,
and the ByteSize/Bytes division of labour.
Verified with cargo check --workspace --all-targets, clippy --all-targets at
-D warnings, nightly fmt.
* perf(parser): drop redundant Arc clones in the Xtream per-item parse loop
Inside `for stream in xtream_streams`, four fields wrapped a clone around a
method that already returns an owned value:
name: Arc::clone(&stream.get_name()) -> get_name() -> Arc<str>
logo: Arc::clone(&stream.get_stream_icon()) -> Arc<str>
title: Arc::clone(&stream.get_name()) -> Arc<str>
epg_channel_id: stream.get_epg_channel_id().clone() -> Option<Arc<str>>
Each one bumped the refcount to two and then dropped the temporary back to one:
four redundant atomic pairs per parsed Xtream stream, on the playlist parse path.
The same file already builds a header correctly without the outer clones 230
lines further down, so this was drift rather than intent.
The workspace now has zero `Arc::clone(&x.y())` sites. The three remaining
`.get_*().clone()` calls are correct -- those accessors return references
(`&Arc<str>`, `&Url`) and the clone is what makes the value owned.
Scope note: this is the substance of what I proposed as "borrowing accessors on
PlaylistEntry", but not the mechanism, because the mechanism did not survive
contact with the call sites. Of 23 `get_input_stream_id` calls, 19 are test
assertions; `get_provider_url` and `get_group` have one production caller each,
and the per-item `get_name` callers need an owned Arc to store. Adding four
borrowing methods to an already 13-method trait to save a refcount bump at ~4
non-hot call sites would grow the API surface for no measurable gain. Where a
caller genuinely only reads, borrowed access already exists via
`FieldGet`/`FieldRef::Shared` from the typed field-access work.
Verified with cargo check --workspace --all-targets, clippy --all-targets at
-D warnings, nightly fmt.
* refactor(shared): type the mapper and counter field allow-lists
COUNTER_FIELDS and MAPPER_FIELDS were `&[&str]` checked by a case-sensitive
string `contains` behind a `valid_property!` macro that added nothing over
`.contains()`. Both are now `&[HeaderField]`.
Three things fall out.
MAPPER_FIELDS loses an entry. The string list spelled the EPG channel id twice,
`epg_channel_id` and `epg_id`, to cover both accepted names. Aliases resolve in
HeaderField::parse, so the typed list names the field once and each spelling is
handled in exactly one place.
The counter path stops doing the same work twice. Since MappingCounter::field
became a HeaderField, prepare() ran a string `contains` and then parsed the same
name again; it is now one resolve-and-check.
`valid_property!` is deleted along with its export, replaced by
`is_allowed_field(name, allowed)`.
One deliberate behaviour change, called out because it loosens validation: the
allow-list was case-sensitive while set_field compared case-insensitively, so a
mapper naming `NAME` was rejected at config load even though writing it would
have worked. Resolving through parse makes the two agree. Every previously valid
config stays valid; some previously rejected ones are now accepted and behave
correctly.
Three tests: both EPG spellings resolve through the single entry, valid-but-
unlisted fields (`input`, `type`) are still rejected and COUNTER_FIELDS is a
strict subset of MAPPER_FIELDS, and casing now agrees with the accessor.
Verified with cargo check --workspace --all-targets, clippy --all-targets at
-D warnings, nightly fmt.
* refactor(shared): put genre on StreamProperties instead of in three macros
Video keeps its genre under `details`, Series keeps it inline, and Live and
Episode do not have one. That four-arm match was written out five times: in
`get_genre!`, in `genre_ref!`, in the Some-branch of `set_genre!`, inline in
`get_filter_value`, and inline in the header's typed field accessor.
Two methods on StreamProperties replace all of it:
fn genre(&self) -> Option<&Arc<str>>
fn set_genre(&mut self, value: &str) -> bool
`get_genre!` and `genre_ref!` are deleted outright -- one call site each, and
both are now `.additional_properties.as_ref().and_then(StreamProperties::genre)`
with an `Arc::clone` only where the caller actually needs to own it.
`set_genre!` stays a macro but loses its four-arm match, dropping from 79 lines
to 64. Its remaining bulk is the None-branch, which constructs a whole
StreamProperties::Video or ::Series *from the header* -- that needs the header,
not just the properties, so it does not belong on StreamProperties and stays
with the caller.
Left alone: ui_playlist_item.rs has the same Video-here/Series-there shape for
`rating`, but with one call site there is no repetition for an accessor to
remove, and adding API for a single caller is the trade this refactor exists to
avoid.
Two tests: genre round-trips through both storage shapes (including that setting
twice reuses existing Video details rather than replacing them), and Live and
Episode report no genre and refuse to take one.
Verified with cargo check --workspace --all-targets, clippy --all-targets at
-D warnings, nightly fmt, and the shared test suite.
* refactor(shared): single-source the item-type to cluster relation
The relation was written down twice and wrapped once:
* `PlaylistItemType::is_cluster(cluster)` matched every variant against a
cluster by hand.
* `impl TryFrom<PlaylistItemType> for XtreamCluster` matched every variant to a
cluster, independently, with nothing keeping the two in agreement.
* `cluster_from_item_type` in the repository wrapped the second with
`.unwrap_or(Live)`.
The TryFrom was total -- every arm returned `Ok` -- so its `Result` was a lie,
and the phantom error had spread defensive noise to 17 call sites across four
crates: `.unwrap_or(XtreamCluster::Live)` eleven times, plus
`.unwrap_or_default()`, `.ok()`, `.is_ok_and(..)`, `.unwrap_or(existing_cluster)`
and one `.map_err(..)?` building an error message that could never be produced.
Now there is one encoding:
PlaylistItemType::cluster(self) -> XtreamCluster // const, infallible
impl From<PlaylistItemType> for XtreamCluster // delegates
is_cluster(cluster) // delegates
All 17 sites drop their fallback. `cluster_from_item_type` is gone. The
`try_cluster!` macro in xtream_repository, whose `ok_or_else` could never fire,
becomes `cluster_or_item_type!` and its four call sites lose a `?`. The STRM
resolve path loses a five-line unreachable error branch.
Note this is source-compatible rather than a breaking change: std's blanket
`TryFrom for U where U: Into<T>` means `XtreamCluster::try_from(item_type)` still
compiles, now with `Error = Infallible`. The old call sites would have kept
working; they are cleaned up because the fallbacks are provably dead, not
because the compiler demanded it.
A test iterates every PlaylistItemType variant and asserts `From` agrees with
`cluster()`, that `is_cluster` accepts its own cluster, and that it rejects the
other two -- so the arms cannot drift apart again.
Verified with cargo check --workspace --all-targets, clippy --all-targets at
-D warnings, nightly fmt.
* refactor(shared): make VirtualId a real newtype and add ProviderId
`pub type VirtualId = u32` gave the reader a name and the compiler nothing. That
matters more than usual here: the same `BPlusTree<u32, XtreamPlaylistItem>` store
is keyed by a *virtual* id on the target path and a *provider* id on the input
path, chosen by a runtime `StorageKey` tag, and nothing stopped a lookup in one
key space using an id from the other.
#[repr(transparent)] #[serde(transparent)]
pub struct VirtualId(pub u32);
pub struct ProviderId(pub u32);
No `From<u32>`/`Into<u32>` on purpose: an implicit conversion would let a
provider id become a virtual id by inference, which is the confusion the types
exist to prevent. Crossing to a raw integer is spelled `new` and `get`.
On-disk compatibility was the gating question, since these are persisted B+Tree
keys and serialized struct fields. `backend/btree/src/codec.rs` gets a test
asserting a transparent newtype over u32 encodes byte-for-byte identically under
rmp_serde and cross-reads in both directions -- bytes written before the newtype
existed decode into it, and bytes it writes decode as a plain u32. Existing
databases are unaffected and there is no migration.
The type now flows through the playlist items (PlaylistItemHeader, M3u, Xtream,
Common), VirtualIdRecord and the whole TargetIdMapping id-index cluster
(disk_by_virtual_id, mem_by_uuid, mem_by_virtual_id, pending upserts,
find_virtual_ids, get_virtual_id_by_uuid, get_and_update_virtual_id), and the
metadata manager's provider->ids and uuid->id caches.
The compiler immediately found what the alias was hiding: the two `StorageKey`
arms in xtream_repository now have *incompatible types*, and
playlist_repository referred to one id-mapping store as both `BPlusTree<u32, _>`
and `BPlusTree<VirtualId, _>`.
Deliberate boundary: cross-boundary DTOs keep their u32 wire shape and unwrap
explicitly with `.get()` -- the Xtream-API-shaped documents, StreamInfo,
UiPlaylistItem, stream history, and the ffprobe/session interfaces. The B+Tree
stores themselves also stay u32-keyed for now: splitting them into VirtualId- and
ProviderId-keyed stores is item 23, and it needs per-site judgement about which
key space each of 24 call sites belongs to. Getting one wrong is a silent lookup
miss, so it is a focused follow-up rather than a rider on this commit.
Verified with cargo check --workspace --all-targets, clippy --all-targets at
-D warnings, nightly fmt, and 1,149 tests across btree, shared, repository and
metadata.
* refactor(repository): make the Xtream store key space a type, not a runtime tag
The Xtream playlist stores use one file layout and one value type for two
different id spaces: the target-side store is keyed by virtual id, the input-side
store by provider id. The only thing recording which was a `StorageKey` enum
matched *per item* inside the insert loop:
tree.insert(match storage_key {
StorageKey::VirtualId => item.virtual_id,
StorageKey::ProviderId => item.provider_id,
}, item);
`write_playlists_to_file` is now generic over the key with a `key_of` extractor,
so each call site names its own key space and the tree's type carries it:
write_playlists_to_file(.., |item| item.virtual_id, ..) // target
write_playlists_to_file(.., |item| ProviderId::new(item.provider_id), ..) // input
`StorageKey` is deleted. The per-item branch is gone -- the extractor is
monomorphized per call site -- and the two key spaces can no longer be swapped
by passing the wrong enum variant. Both keys are `#[serde(transparent)]` over
u32, so the on-disk encoding is identical either way, which the codec test in
backend/btree pins.
This is the write path only. The read-side store types are still
`BPlusTree<u32, XtreamPlaylistItem>`: typing those means classifying 24 call
sites as virtual- or provider-keyed, and misclassifying one is a silent lookup
miss rather than a compile error, so it wants its own focused pass with the
staging/publish paths read end to end.
Verified with cargo check --workspace --all-targets, clippy --all-targets at
-D warnings, nightly fmt, and 422 tests across repository and btree.
* feat(shared): open-world event ids for notifications
`MsgKind` is a closed enum, so adding one notification event kind meant
editing eight sites across three crates - and two of those sites failed
silently rather than at compile time: `discover_templates` hardcodes the
variant list, and Pushover has no template map at all.
Replace it as the extension point. An event is now identified by a dotted
`domain.event` string with a severity, subscriptions are glob patterns,
and adding an event is one `EventId` const plus one emit call.
The pattern grammar stays small enough to explain in a config comment:
`*` for everything, `recording.*` for a domain, `provider.*.expired` for a
single wildcard segment, and a leading `!` to exclude - so `["*",
"!system.info"]` reads the way it looks.
Every legacy `MsgKind` wire name stays a valid `notify_on` entry via
`LEGACY_ALIASES`, and the canonical recording ids produce exactly the
legacy template filenames (`recording.completed` ->
`telegram_recording_completed.templ`), so existing config and template
files keep working untouched.
`EventId` deserializes an unrecognised id to `registry::UNKNOWN` rather
than failing, so an outbox written by a newer build round-trips through an
older one instead of poisoning the whole file.
No behaviour change: nothing reads these types yet.
* feat(core): one notification envelope instead of a per-kind context
`TemplateContext` carries one `Option` field per message kind - `message`,
`stats`, `watch`, `processing`, `disk`, `recording`, `flat_stats` - which
is why adding a kind had to touch the renderer.
`NotificationEvent` is the shape every event fits: id, severity, timestamp,
instance, dedup key, title, body, and the typed payload serialized into
`fields`. The existing payload structs are unchanged and now travel inside
`fields`, so nothing about them had to move.
`title` and `body` are always populated. That is what lets Pushover, a
syslog channel or an email subject line render any event without a per-kind
match - the gap that had Pushover pushing raw `serde_json` dumps of watch
changes and playlist stats to phones. `body_for` gives watch changes and
processing stats a readable plain-text rendering for the first time, while
keeping the string and disk-alert output byte-identical to the old
`default_text_for`.
`from_content` lifts a legacy `MessageContent` into the envelope, so all six
existing emitters keep working untouched.
Timestamp is unix seconds rather than `DateTime<Utc>`: it matches the
outbox's existing `enqueued_at` representation and avoids pulling chrono's
serde feature into the workspace. `timestamp_rfc3339` renders it for
templates.
No behaviour change: nothing reads these types yet.
* feat(messaging): a NotificationChannel trait to open the channel set
Adding a channel meant editing ten sites across three crates: a
`MessagingChannel` variant, the `is_some()` chain in
`configured_channels`, the match in `send_message_to_channel`, a new
`send_*_message` fn, the hardcoded `tokio::join!`, a config field, a DTO
field, both `From` impls, `prepare`, and the template-discovery prefix.
This adds the abstraction that collapses those: one trait with a stable
`id()`, the operator's template lookup, a `wants()` routing hook, and
`send()`. The dispatcher never learns the channel's name.
`Delivery` replaces the old `Option<bool>`, which could not distinguish
"retry me" from "this URL is malformed and will fail identically forever".
That cost real attempts: a typo'd webhook burned all `max_attempts` with
exponential backoff before dead-lettering, and a `429` was retried straight
back into the rate limit it had just hit. `delivery_for_status` now
classifies once for every HTTP channel - `408`/`429`/`5xx` transient,
other `4xx` permanent - and `parse_retry_after` reads both legal header
forms so a provider-supplied delay is honoured.
A `Retry-After` HTTP-date already in the past clamps to zero ("retry now")
rather than being discarded, since the server is telling us the wait is
over.
Dispatch is dynamic on purpose: the channel set is an open world resolved
at config load and a send is bounded by a network round trip, so the
vtable hop is not measurable and static dispatch would reinstate the
closed-world match. Uses the boxed-future alias convention from
`tuliprox-processing`'s `SinkFuture` rather than adding `async-trait`.
No behaviour change: nothing implements the trait yet.
* perf(messaging): cache and precompile notification templates
`resolve_template` wrapped every template value in an `InputSource` and
called `download_text_content` - once per message, per channel. A `file://`
template was re-read from disk and an `http://` one re-fetched over the
network for every notification. Worse, nothing distinguished the three
cases, so an *inline* Handlebars string paid for a full download attempt
too, on every send.
Classify the source once (`Inline` / `File` / `Url`), then cache what is
actually resolved: local files revalidate on mtime so an edit applies
immediately, remote documents on a 5 minute TTL. Compiled templates are
kept in a registry keyed by the config value, so rendering is no longer a
re-parse.
When a remote template cannot be refreshed the cached copy is served
rather than silently degrading to the built-in text - the operator
configured a template for a reason, and a template-host blip should not
change what the notification looks like.
`validate` compile-checks a template body without sending, so config
validation can surface a malformed template at load instead of leaving a
per-send `error!` and a fallback that looks plausible.
`context_for` carries the new uniform `event.*` shape alongside every
legacy top-level key - `kind`, `message`, `stats`, `processing`, `disk`,
`watch` and the flattened first-input fields - so templates written against
the documented examples render identically. `kind` keeps its CamelCase
labels for the legacy ids for the same reason.
* refactor(messaging): dispatch through channels, promote the outbox
Replaces the closed-world dispatch with the trait and envelope added in the
previous commits, and moves the outbox out of the recording supervisor so
every notification gets durable retry.
Channels
--------
The four hand-written `send_*_message` functions, the `is_some()` chain in
`configured_channels`, the match in `send_message_to_channel` and the
hardcoded `tokio::join!` are gone. `channels::build` constructs whatever
the config declares and the dispatcher fans out over the trait, so adding
a channel is an impl plus a config field.
Pushover gains template support. It had none, so every notification took
the built-in text - which for watch changes and playlist stats was a raw
`serde_json` dump pushed to a phone. It also now sends `title` separately,
and maps event severity onto Pushover's own priority scale.
Sends are concurrent per channel and carry a 30s request timeout. The
shared client sets no request timeout at all, and the outbox awaited
channels in sequence, so one webhook host that accepted a connection and
never answered could stall every pending notification - including the
recording ones the outbox exists to protect.
Outbox
------
Moved to `tuliprox-messaging`, no longer gated on the recording config, and
started unconditionally at bind. Playlist stats, watch changes, disk alerts
and provider warnings previously called `send_message`, which fanned out
and discarded every outcome with `let _ =`; a transient 502 lost them
permanently. `send_event` is now a thin enqueue with a direct-send
fallback.
Entries key pending channels by stable string id rather than an enum
variant, so an outbox written by a build that knows a newer channel no
longer fails to deserialize and take the whole file - and with it every
pending notification - down with it. `notification_outbox.json` adopts any
entries left in `recording_notification_outbox.json` exactly once.
`Delivery::Permanent` dead-letters immediately instead of burning every
attempt on a request that will fail identically forever, and a provider's
`Retry-After` now wins over our own backoff rather than retrying straight
back into the rate limit.
Config
------
`notify_on` is a list of glob patterns; template maps are keyed by event id
wire name. Legacy `MsgKind` names still parse and are normalized to
canonical ids on load. Template discovery iterates the event registry
instead of a hardcoded eight-variant array - the site that silently made a
newly added kind undiscoverable - and still finds legacy filenames.
The frontend event picker is driven by the registry too, so an event added
in the backend appears in the UI without a frontend change.
The six pre-existing template tests are kept and now render through the new
pipeline, which is what proves the documented Discord and Telegram
templates still produce identical output.
* chore: lockfile for tokio-util and futures in tuliprox-messaging
* feat(messaging): per-channel routing, suppression and rate limiting
Routing was a single global `notify_on`: every enabled event went to every
configured channel, so "critical to Pushover, everything to Discord" could
not be expressed at all. Each channel now takes an optional `routing`
block:
telegram:
routing:
notify_on: ["provider.*", "system.disk.alert"]
min_severity: warn
quiet_hours: "23:00-07:00"
max_per_hour: 20
dedup_window_secs: 3600
An absent block inherits the global subscription, so existing configs are
unaffected.
Suppression by `dedup_key` generalizes the disk alert's
`repeat_interval_secs`, which lived in `sys_usage.rs` and was available to
nothing else - `provider.offline` needs exactly the same logic. The hourly
ceiling emits one "further notifications suppressed" audit line when it
trips and then goes quiet, so the silence is distinguishable from a
notifier that has died.
Quiet hours defer rather than drop. The outbox holds the entry until the
window closes, because an overnight outage nobody hears about afterwards is
worse than one that arrives late. An entry is only held while *every*
still-pending channel is asleep.
Two supporting fixes:
`channels::build` ran on every send, constructing a fresh `reqwest::Client`
each time - added in the previous commit and wrong. The channel set is now
cached and invalidated on reload, which is also what lets per-channel
suppression and rate-limit state survive between notifications.
Routing is boxed inside the channel DTOs. Four inline routing blocks made
`MessagingConfigDto` the largest `ConfigForm` variant at 857 bytes; the
`Option<Box<_>>` is niche-optimized and only allocates when routing is
actually configured.
Rate-limit windows are pure in `now`, so expiry is tested without sleeping
and without tests racing each other over shared global state.
* feat(app): bridge the event bus onto the notification pipeline
`EventMessage` already carries fourteen variants from thirteen emitters -
playlist updates, config changes, library scans, user connections, metadata
updates, recording changes - and every one of them reached the Web UI over
the websocket and nowhere else. The notification side had six emitters of
its own, and nothing connected the two.
One subscriber closes that gap, and every future `EventMessage` variant
comes along with it.
Three things keep it from being a firehose:
* The high-frequency variants map to `None`: progress ticks, download
deltas and periodic system info fire many times per operation and carry
nothing worth pushing to a phone. Their terminal counterparts are what
get through. The match is exhaustive, so a new variant is a compile
error rather than a silent firehose.
* Everything defaults to unsubscribed. An upgrade does not start messaging
anyone until `notify_on` asks for it.
* The broadcast channel has capacity 10, so `Lagged` is reported and
skipped rather than killing the bridge.
A partial playlist update maps to `completed` at `warn` rather than `info`,
so it does not read as a clean success. Config changes carry a per-file
dedup key, so a watcher that fires several times for one save produces one
notification.
The two genuinely chatty events - `user.connection.changed` and
`provider.connections.changed` - are registered and documented as high
frequency so subscribing to them is a deliberate act.
* feat(messaging): specific provider events, a test endpoint, and secret redaction
Three changes that together make the messaging config verifiable and safe to
expose.
Provider account events
-----------------------
The Xtream account-status, expiry-warning and expired messages all landed in
`MsgKind::Info`/`Error`, so subscribing to "my account is about to expire"
meant also receiving every processing error. They now emit
`provider.account.status_changed`, `.expiring` and `.expired` with a typed
payload.
All three carry a dedup key. They are re-evaluated on every playlist
refresh, so without one an account inside its final three days would notify
on every single update.
Test endpoint
-------------
`POST /api/v1/config/messaging/test` renders and optionally sends a chosen
event to a chosen channel, returning the per-channel outcome *and* the exact
rendered body. `preview: true` renders without sending, so a template can be
iterated without spamming a channel.
It deliberately bypasses `notify_on` and the suppression window: the
operator asked for this one explicitly, and a test that silently does
nothing because of a dedup window would be worse than useless.
Secret redaction
----------------
The config GET returned `config.yml` in full to any client holding
`ConfigRead` - including the Telegram bot token, the Pushover token and user
key, and any `Authorization` header on the REST channel. Those are now
masked on the way out.
`save_config_main` restores any secret the client echoes back still masked,
so a UI round-trip cannot overwrite a real token with the mask and silently
break the channel. A genuinely changed secret still writes through, an empty
secret is not replaced by a mask (an unconfigured channel must not look
configured), and non-credential REST headers keep their values.
* feat(messaging): five new channels, and HMAC signing for webhooks
The test of whether the channel trait actually opened the set. Each of
these is one impl plus one config field plus one line in the builder -
no dispatcher, outbox, renderer or template-discovery change.
* **ntfy** - self-hosted push, no account, no bot token. The natural
default for a homelab operator already running tuliprox in Docker.
ntfy headers must be ASCII and titles routinely carry emoji, so the
title is stripped rather than failing the send, with a fallback so it
never goes out empty.
* **Gotify** - same audience, same shape.
* **Slack** - not a Discord clone. Block Kit differs enough from Discord
embeds that reusing the Discord payload produces bad output, so it
builds header/section/context blocks. Slack has no severity field, so
non-info severities are marked in the header text where a human sees
them.
* **command** - runs a local program with the event JSON on stdin. The
escape hatch that means nobody waits for a channel to be added
upstream. Executed directly rather than through a shell, so there are
no quoting rules and no shell-injection surface from event content. A
missing binary is `Permanent` (retrying cannot help); a non-zero exit
or a timeout is `Retry`.
* **REST signing** - optional HMAC-SHA256 over `{timestamp}.{body}`,
sent as `X-Tuliprox-Signature`. The timestamp is inside the signed
payload, so a captured request cannot be replayed with a fresh header.
Verified against the RFC 4231 test vector.
Each severity maps onto the target's own priority scale rather than
being dropped, and the new secrets (ntfy token, Gotify token, REST
signing secret) join the existing ones in the redact/restore path.
`ConfigForm::Messaging` is boxed: eight channel configs made it dominate
the size of every other variant of that enum.
* docs(messaging): document the open-world event and channel model
Rewrites section 5 for what the messaging layer actually does now.
* The glob grammar for `notify_on`, with a note that every legacy event
name still works and is rewritten on the next save.
* A table of all 24 registered events with default severities.
* Per-channel `routing`: `notify_on`, `min_severity`, `quiet_hours`,
`max_per_hour`, `dedup_window_secs` - including that quiet hours defer
rather than drop, and that the hourly ceiling reports itself once so the
silence is not mistaken for a broken notifier.
* Delivery semantics: the outbox, per-channel retry, which statuses are
transient vs. permanent, and `Retry-After`.
* The four new channels (ntfy, Gotify, Slack, command), webhook HMAC
signing with the replay note, and secret masking in the Web UI.
* Templates: now supported on every channel including Pushover, keyed by
event id, resolved and compiled once rather than re-fetched per send,
with the uniform `event.*` context documented alongside the legacy keys.
* The test endpoint, so there is a feedback loop shorter than "save it and
wait for something to break".
The event table sits between generated-block markers and a test checks it
against the registry in both directions - a registered event missing from
the table, and a table row for an event that no longer exists. Verified
the test fails for both before restoring.
* fix(messaging): invalidate channel and template caches on config reload
The channel set and compiled templates are cached so a notification does
not rebuild every channel - and a fresh `reqwest::Client` with them - on
every send. Nothing invalidated those caches, so an edited bot token,
webhook URL or template would not take effect until a restart.
Also emits `config.reload_failed` alongside the existing `ServerError`, so
an operator can subscribe to "my config stopped loading" without taking
every server error with it.
* refactor(events): move EventMessage into shared
`EventMessage` lived in `tuliprox-session`, the crate that owns provider
allocation and the streaming-session runtime. That meant `metadata`, `dvr`
and `processing` all had to depend on the streaming runtime just to name an
event - a metadata refresh completing has nothing to do with which provider
a stream came from.
Every payload the enum carries was already a `shared::model` type, so the
move is mechanical: the taxonomy goes to `shared::model::event`, the bus
implementation stays in `session` next to the stream-meter registry it also
feeds.
`api::model` re-exports it from `shared` so the ~80 `crate::api::model::
EventMessage` call sites keep their path.
* refactor(events): emit through a static EventSink bound
There was no seam anywhere in the event path. `Arc<EventManager>` - the
concrete streaming-runtime bus - was baked into `MetadataUpdateCtx`,
`RecordingCtx`, and the playlist pipeline, which carried it as
`Option<Arc<EventManager>>` purely because tests have no bus to hand it.
That `Option` was then unwrapped at every one of its emit sites.
`shared::model::EventSink` is the seam: one method, `emit`, documented as
non-blocking because the bus is reached from the streaming data path.
It is a bound, never a trait object. The three context structs are generic
over their sink and monomorphise against the one they were built with, so
emitting stays a direct call. `NoopSink` is the absent case, and its `emit`
is an empty function - the pipeline's six `if let Some(events)` branches
collapse to unconditional calls that compile to nothing when that is the
instantiation, and `create_broadcast_callback` loses its noop arm.
`MetadataUpdateManager` stores its context and is itself held by `AppState`,
so it pins one instantiation (`BoundMetadataUpdateCtx`) rather than going
generic and dragging `AppState` with it. Functions that only read a context
stay generic.
`tuliprox-dvr` drops its dependency on `tuliprox-session` entirely - the
event bus was the only thing it wanted from the streaming runtime, and a
bound is not a dependency. 78 workspace edges, now 77.
`app_state_views!` grows a `ctx_field <- state_field` form for a context
that names a handle by the role it plays because it is written against a
bound rather than against `EventManager`.
* feat(events): give events an identity with EventKind
Subscribers each discriminated by exhaustively matching `EventMessage`: the
websocket mapped variants to permissions, the notification bridge computed
severity per variant, and the wire layer maps them to `ProtocolMessage`.
Adding a variant compiled cleanly while reaching none of them.
`EventKind` carries the payload-free identity, and everything that is a
property of the event rather than of one consumer hangs off it:
* `required_permission` - who may see this. Not a websocket concern: it
does not change with the transport, so `websocket_can_receive_runtime_
events` is now one line and cannot fall out of date.
* `severity` - on `EventMessage`, because it reads the payload: a playlist
update that failed is an error and one that succeeded is not. The bridge
now decides only which notification id to use.
* `is_high_frequency` - progress ticks, deltas, the periodic system-info
sample. A statement about rate, deliberately separate from the bridge's
notifiability decision, which also depends on whether a terminal
counterpart exists.
* `as_wire_name` / `from_wire_name` - stable strings for plugin
subscriptions and operator config, with the same
must-not-change-once-released contract a notification channel id carries.
`EventKind::ALL` fixes the bit order that the subscription mask uses.
No behaviour change: the permission mapping is the same table and the
notifiable set is untouched.
* perf(events): filtered subscriptions and cheap fat payloads
`get_event_channel` handed every subscriber all fourteen kinds and each one
filtered afterwards - after the broadcast channel had already cloned the
message for it. Two costs, addressed separately.
Filtering: `EventKindMask` is a one-word set over `EventKind`, and
`subscribe_filtered` returns a `FilteredEventReceiver` that drops what the
subscriber did not ask for before waking it. A concrete type, not a boxed
stream: the call site is a `select!` arm awaiting `recv()`, and a virtual
call per event buys nothing. `Lagged` still propagates - a gap in events the
subscriber did not want is still a gap in the ones it did.
The notification bridge is the first user. It handles ten kinds and dropped
four, and those four are the bulk of the traffic during a playlist refresh,
so it now never wakes for them. A test asserts the mask and the `None` arms
of `to_notification` agree, and that the sample list covers every
`EventKind` - so a variant added later cannot quietly fall out of either.
Payloads: `SystemInfoUpdate` and `DownloadsUpdate` carried `SystemInfo` and
`DownloadsResponse` by value, which is what `large_enum_variant` was
allowed for. Behind `Arc` they cost a refcount bump per receiver instead of
a deep copy, and the `allow` is gone. Only the websocket needs the value
itself; `unwrap_or_clone` there means the last subscriber standing pays
nothing. The frontend mirror of this enum has held both behind `Rc` all
along.
* refactor(session): split the stream-meter registry out of EventManager
`EventManager` was two components sharing a name: a pub/sub bus for fourteen
event kinds, and a metering subsystem with its own broadcast channel, its
own subscriber counter, its own background sampler task and three maps. No
event subscriber ever touched the second half, and no meter call site
touched the first.
`StreamMeterRegistry` is that second half. `EventManager` owns one and
forwards the meter methods, so the composition root still builds a single
handle and streams need not know metering is separate; `meters()` exposes
the registry for anything that does.
The three maps become one. `meters`, `meter_to_clients` and
`client_to_meter` had an invariant re-established by hand in four methods -
retain from the vec, remove the entry if it emptied, drop the index - and
`register_meter_client` spelled it differently from the other three. They
are now `HashMap<u32, MeterSlot>` plus the client index, with `detach_client`
and `remove_meter` as the only two mutators. The slot's handle stays
optional because the two halves genuinely arrive in either order: a client
can be assigned to a meter before the stream owning it registers.
`read_meter_qos` returns `Option<MeterQos>` instead of
`(Option<u64>, Option<u64>)`, where "both `None`" doubled as "this meter is
shared, ask no further" - a convention that lived only in a doc comment.
The sampler still declines to start outside a tokio runtime, but now says
so at debug rather than returning in silence.
All six meter tests pass unchanged, which is the point: this is a
rearrangement, not a behaviour change.
* feat(events): configurable bus capacity, and counters for what it drops
The bus was `broadcast::channel(10)` for everything. Ten is routinely
outrun: a playlist refresh emits progress ticks in a loop, and any
subscriber that awaits I/O per event falls behind within one target. The
evidence was already in the tree - the notification bridge carries a
dedicated `Lagged` arm with a comment about it, and the websocket has an
entire `ResyncStatus` recovery path that exists for no other reason.
Capacity is now `event_channel_capacity`, default 256, clamped to at least
1. `EventManager::new()` keeps the default for tests and early startup.
`EventBusStats` makes the drops visible. `send_event` returned a `bool`
that ~80 call sites discarded with `let _ =`, so "nobody received this" and
"nobody was listening" were equally invisible. It now counts emissions per
kind, emissions with no subscriber, and - reported by the subscribers,
since a broadcast channel drops for the receiver and not the sender - the
size of every gap a subscriber was told about. The notification bridge and
the websocket both report theirs.
This is also where the plugin system's promised drop counters come from,
rather than a second set of counters on a second queue.
* refactor(events): single-source the notification and wire mappings
Four parallel taxonomies described the same events: `EventMessage` on the
bus, `ProtocolMessage` on the wire, the frontend's own enum, and the
notification registry. Two of the three translation layers were hand-written
tables that had to be edited in lockstep, with nothing checking they agreed.
The notification id moves onto the event. `EventMessage::notification_id`
returns the registry id, or `None` for the kinds that are not notifiable -
one decision, made once. It lives on `EventMessage` rather than `EventKind`
because two of them read the payload: a playlist update that failed is a
different notification from one that succeeded, not merely a more severe
one. The bridge now decides only wording and attachments.
The wire mapping becomes a pure function. `handle_event_message` was a
hundred lines of `match` nested three deep inside the socket loop, where a
kind reaching no arm looked exactly like a kind deliberately ignored.
`to_protocol_message` is testable on its own, and the socket loop is now
guard, special case, send.
Both new tests iterate `EventKind::ALL` and assert the sample list covers
it, so a variant added later fails the tests instead of silently reaching
nobody - which is the failure mode all of this exists to prevent.
* feat(events): the seam the plugin host subscribes through
The plugin plan specified a second, independent event bus: its own bounded
mpsc, its own emit call sites in `process_sources`, the active-stream
tracker and user CRUD, and its own drop counters. Three of those four
emitters already publish to `EventManager`. Building the second one would
have meant two emit sites per event and two taxonomies to keep in step.
What the host actually needs from the existing bus, added here:
* `EventKindMask::from_wire_names` - a manifest's `events.*` list becomes a
subscription mask, with unknown names returned rather than silently
narrowing what the plugin asked for. An empty mask means no subscriber
task at all.
* `EventMessage::payload` - the JSON a plugin receives, defined on the event
so a new variant arrives with its payload shape decided instead of the
host growing a second match over the taxonomy. Never fails: an
unserialisable payload degrades to null rather than dropping the event.
* `EventKindMask::kinds`, for reporting what a subscription resolved to.
`plugin-system-plan.md` (untracked) is revised to match: the plugin queue
sits behind the bus rather than beside it, keeping its backpressure
isolation while making "emitters never block" structural - `broadcast::send`
cannot block - instead of a convention every emit site has to honour.
Tests assert the bit assignment is unique, wire names are unique and round
trip, and the taxonomy still fits the mask's `u32`.
Deliberately not done: `stream-start` / `stream-stop`, `user-created` and
`probe-result` still have no `EventMessage` variant. They belong in the
shared taxonomy rather than a plugin-private one, but stream start/stop
sits on the streaming hot path and should not gain a per-stream emission
speculatively, ahead of a consumer that needs it.
* feat(events): typed emit outcomes and nudge coalescing
Two shapes the messaging crate had already worked out for notification
channels, and the event bus lacked.
Typed outcomes. `send_event` returned a `bool` that conflated "nothing was
listening" with "this failed", told the caller nothing useful either way,
and was discarded by ~80 call sites. `EmitOutcome` distinguishes
`Delivered { receivers }`, `NoSubscribers` and `Coalesced` - the same
distinction `Delivery` draws for channels. Not `#[must_use]`: an emitter
genuinely may ignore all three, and pretending otherwise would just spread
`let _ =` further.
Coalescing. `rate_limit::admit` throttles notifications; nothing throttled
the bus. `RecordingChanged` is emitted from six routes, twice back-to-back
where deleting a recording also changes the rules, and once per item in a
bulk operation - each one making every Web UI session re-fetch the same
snapshot.
`EventKind::is_coalescable` marks the kinds where N occurrences and one are
indistinguishable to every consumer: payload-free nudges that everyone
answers by re-reading current state. Only the two recording nudges qualify.
An event carrying a payload is never coalescable however repetitive, because
a dropped progress tick loses the message it carried - the tests assert
exactly that, and that the two nudges do not suppress each other.
The window is 250ms and is measured from the last admitted send, so a
sustained stream is throttled to one per window rather than one per burst,
and a later user action always produces a visible refresh.
* feat(events): snapshots, an audit ring, graceful shutdown
The remaining smaller fixes from the event-manager review.
Latched snapshots. `EventKind::is_latched` marks the kinds that describe
current state rather than an occurrence - the last `SystemInfo` sample *is*
the system info. The bus retains the newest of each, and `snapshot()` hands
them to a session that connects between samples, which used to see empty
panels until the next one arrived up to three seconds later. Occurrences are
never latched: replaying "a playlist update finished" to a session that was
not there would be a lie, and the tests say so.
A recent-event ring. 256 entries, each with its kind, monotonic uptime and
outcome - including suppressed ones, since "it fired but was coalesced" is
exactly what the ring is consulted to find out. Exposed with the bus
counters at `GET /api/v1/events/stats`, behind `system.read` like `/status`.
"Why did my notification not fire?" was otherwise unanswerable without a
debug build.
Graceful shutdown. `Drop` cancelled the meter sampler but cannot await, so a
stream still running at shutdown lost its last window's bytes -
`flush_and_unregister_meter` already fixed that for one stream ending
between ticks. `EventManager::shutdown()` does it for the whole registry,
and runs after the connection manager so the final batch reports what the
streams actually transferred.
`send_provider_event` and `send_system_info` are gone. They wrapped
`send_event` only to log on failure, which `EventBusStats` now records
centrally - two calling conventions for one bus was one too many.
Also documents the new endpoint in the REST cookbook.
* fix(processing): pin the pipeline test context to NoopSink
`processing_context()` was made generic along with the pipeline it builds,
which left every call site unable to infer the sink type. These tests
exercise the pipeline, not the bus, so the helper names `NoopSink` and the
call sites say nothing.
Neither `cargo build --workspace` nor `cargo clippy --workspace` compiles
test code, so this only surfaced on the first full test run.
Also carries the lockfile change from `tuliprox-dvr` dropping its dependency
on `tuliprox-session`.
* refactor(messaging): static dispatch, no trait objects and no boxing
The notification layer used `Vec<Arc<dyn NotificationChannel>>` and a
`Pin<Box<dyn Future>>` per send. Both are gone.
`NotificationChannel::send` now returns `impl Future<Output = Delivery> +
Send` instead of a boxed future, which makes the trait deliberately *not*
object safe - there is no way to construct a trait object from it, so the
old shape cannot come back by accident.
`Channel` is an enum over the eight implementations. Every call goes
through a match the compiler turns into a direct call, so dispatch is
monomorphized and the send path allocates nothing. Adding a channel is now
a module, an enum variant, a config field and a line in `build` - one site
more than the trait-object version, and still far from the ten the original
closed-world dispatch cost.
The two `Box`es in the config types are gone as well:
* `ChannelRoutingDto` is stored inline again on all eight channel configs.
* `ConfigForm::Messaging` carries `MessagingConfigDto` directly.
Both were added only to silence `clippy::large_enum_variant`. That lint is
now allowed explicitly on `ConfigForm` with the reason recorded: the
variant is moved once per form submit and never in a hot path, so the
indirection bought nothing real.
The only remaining `dyn` in the crate is Handlebars' own `register_helper`
signature, which is pre-existing and part of that library's API.
No behaviour change - same 46 messaging tests, same delivery semantics.
* feat(events): put the notification-only lifecycle events on the bus
Six emit sites called `tuliprox_messaging` directly instead of publishing to
`EventManager`. Everything they emitted reached operators by mail and nothing
else: not the websocket, not the notification bridge, and - once a plugin
host subscribes to the bus - not plugins either. An event was
plugin-visible only by accident of which route its emitter happened to take.
Nine kinds move onto the bus, each keeping the registry id it already had:
* `DiskAlert` - and the *subscription* check in front of it is gone. The
thresholds still come from `messaging.disk_alert`, but gating emission on
someone being subscribed by mail meant a plugin watching for disk
pressure saw nothing unless an operator happened to want the same event.
The notification layer already drops unsubscribed events.
* `ConfigReloadFailed` - the typed counterpart to the `ServerError` string
the Web UI toasts. Both are emitted: the comment at that call site already
argued an operator should be able to subscribe to "my config stopped
loading" without taking every server error, and that argument applies to
plugins, which only ever saw the string.
* `PlaylistWatchChanged`
* `RecordingStarted` / `Completed` / `Failed`
* `ProviderAccountStatus` / `Expiring` / `Expired`
Recording lifecycle is published *alongside* its existing delivery, not
instead of it. That path is at-most-once: a durable marker is persisted
inside the queue-mutation boundary before delivery, and the outbox retries
per channel. A broadcast bus drops for a lagging subscriber, so routing it
through here would let a recording be marked delivered and then never sent.
The bridge ignores the kind for exactly that reason; `download_api` still
owns operator delivery.
`WatchChanges` and `RecordingLifecycleMessage` move to `shared` for the same
reason `EventMessage` did - an event payload has to be nameable by every
emitter - with `ProviderAccountEvent` and `ConfigReloadFailure` added there.
`tuliprox-core` re-exports the two that moved, so `MessageContent` and its
call sites are untouched. The bridge builds the three that already had a
`MessageContent` shape via `from_content`, so the templates that render them
see exactly the fields they saw before.
Severity is no longer a second table: it comes from the registered event's
descriptor, with one override for a partial playlist refresh, which shares
an id with a clean one but is not a clean success.
`tuliprox-iptv` drops its dependency on `tuliprox-messaging` - emitting an
event is not knowing how it gets delivered. 77 workspace edges, now 76.
* feat(events): carry the run summary on the playlist-update event
`PlaylistUpdate` carried the outcome enum alone, so "the refresh finished"
reached the bus but what it actually did did not. The run summary went
somewhere else entirely: straight to the notification layer as a second
message, built from `MessageContent::event_stats`.
Both messages resolve to `playlist.update.completed`, so a successful
refresh with statistics notified **twice** - once with the stats and once
with the bare outcome - and neither carried the other's content. A
subscriber on the bus saw an outcome with no detail, and the plugin plan's
`refresh-complete`, specified as "run summary as payload", had nothing to
read.
`PlaylistUpdateSummary` folds them: outcome, per-source statistics, and the
aggregated error text, emitted once at the end of `process_sources`. The
notification bridge renders it through `ProcessingStats`, so the "Stats" and
"Error" templates - which read `fields.stats` - see exactly the shape the
separate message used to hand them, while the id and severity come from the
event, because the pipeline's own `PlaylistUpdateState` is a better answer
than re-deriving the outcome from which fields happen to be populated.
The websocket frame is unchanged: it carries `summary.state`, so the Web UI
sees what it always did.
`tuliprox-processing` no longer sends notifications directly. The four
timeout and panic paths use `PlaylistUpdateSummary::state_only`, having no
statistics to report.
`SourceStats` and friends gain `PartialEq`, which `EventMessage` requires of
everything it carries.
* chore(processing): drop the now-unused messaging dependency
`tuliprox-processing` published its last notification directly when the
playlist run summary moved onto the bus. Emitting an event is not knowing
how it is delivered, so the edge goes. 76 workspace edges, now 75.
* chore: lockfile for the dropped iptv and processing messaging edges
* feat(events): user account lifecycle and failed stream probes
Two emitters that changed state and told nobody. Creating, editing or
deleting an API-proxy user wrote api_proxy.yml, swapped the live config
and returned 200; ffprobe failures went to the item store and a warn
line. Neither could be notified on, and neither had an audit trail.
Both follow the RecordingLifecycle / ProviderAccount shape: one payload
with a state, several EventKinds, so a subscriber can ask for deletions
or 404s alone.
UserLifecycleEvent carries username, target and state - not the
password or token. That record reaches Telegram, webhooks and shell
commands, several of which log; the secret is absent from the type
rather than redacted at each render site.
StreamProbeFailure has no success counterpart: a metadata run probes
every unknown stream, so success would fire thousands of times per
refresh. It is published from prepare_generic_stream_metadata rather
than from the manager because that is the only place the reason still
exists - both outcome enums collapse NotFound, Other and Cancelled into
one ProbeFailed. Cancelled is deliberately not published: it is this
server shutting down, not a statement about the stream. The URL is
sanitized at the emit site, since a resolved provider URL carries
account credentials.
Notifications for both are deduplicated - per account+state, and per
*input* for probes, so a provider outage notifies once instead of once
per channel behind it.
Also widens EventKindMask from u32 to u64. The taxonomy is at 27 of 32
bits; the cheap time to widen is before operators have subscription
lists to migrate.
And repairs every_event_kind_is_either_wire_mapped_or_deliberately_not,
which has been failing on DiskAlert since the notification-only
lifecycle events joined the bus: twelve kinds returned None from
to_protocol_message while HANDLED_ELSEWHERE listed three. The two
reasons a kind produces no frame are now two lists, because they are
not the same fact.
* refactor(auth): roles as a static bitset, one is_admin, one JWT decode
`Claims::roles` was a `Vec<String>`: an allocation per mint, a string
comparison per check, and - because five call sites compared with `==`
while a sixth used `eq_ignore_ascii_case` - two different answers to the
same question depending on where you landed.
Roles are now a `RoleSet`, built by the same `create_bitset!` macro that
backs `PermissionSet`. A role check is a `test` instruction on a `u8`.
The JWT wire format is unchanged: `role_names` serialises the set back to
the legacy `["ADMIN"]` string array, so tokens minted before this change
still verify and clients that read the payload see what they always saw.
Unknown role names deserialise to no bit rather than failing the parse,
so a token from a newer build fails closed.
`Claims::is_admin` / `is_api_user` replace the six open-coded checks.
Case-insensitivity is now uniform - the parse accepts either case, which
is the superset that cannot regress an existing token.
`validate_request` took a `fn(&str, &[u8]) -> bool` that re-decoded the
token from scratch, so every authenticated request paid for two JWT
decodes and the API-user path paid for three. It now takes a
`fn(&Claims) -> bool` - still a plain fn pointer, still static dispatch -
and reads the claims it already decoded. `verify_token_admin` and
`verify_token_api_user` are gone with it.
* refactor(auth): one typed rejection for every auth extractor, and 401 means 401
`AuthBasic`, `AuthBearer` and `Fingerprint` each declared
`type Rejection = (StatusCode, &'static str)` - the same alias, defined
twice, once in `tuliprox-auth` and once beside `Fingerprint` in
`tuliprox-core`. The tuple carried no structure, so each arm picked a
status by hand, and they all picked the same wrong one: a *missing*
`Authorization` header answered `403 Forbidden`. That tells a client "you
are authenticated and still may not do this" when the truth is "you did
not authenticate at all", and the two are not distinguishable from the
outside. No `WWW-Authenticate` challenge was sent either, so a 401 from
this server was never a well-formed 401.
`AuthRejection` replaces both aliases: an enum in `tuliprox-core` with a
`status()`, a `message()`, and an `IntoResponse` that attaches the
`WWW-Authenticate` challenge for the scheme the extractor wanted. Missing,
malformed and wrong-scheme headers are now 401; only an unresolvable peer
address stays a 400. `auth_middleware::rejection_for` gained the same
challenge header on its 401 path.
* fix(auth): validate the issuer, bound the token lifetime, honour pwd_version
Three holes in the token lifecycle, all of them the same shape - a field
that was written into every token and then never read back.
**`iss`**: `Validation::new(Algorithm::HS256)` checks `exp` and nothing
else, so the configured issuer was decoration. `verify_token` now takes
the expected issuer and sets it on the validation. The WebSocket paths
carried a bare `Vec<u8>` secret across task boundaries, which is exactly
why they could not check an issuer they never received - they now carry a
`TokenVerifier` that holds both.
**`token_ttl_mins`**: a configured `0` meant "expire in 100 years", which
is a permanent bearer credential written as if it were a configuration
convenience. `0` now means the 24-hour default, anything above the 30-day
ceiling is clamped, and both log why.
**`pwd_version`**: minted into every web token, checked in exactly one
place - the refresh endpoint - so changing a password invalidated nothing
on any guarded route. Together with the 100-year TTL above, a leaked token
was a permanent credential. `validate_password_version` is the check, and
it rejects `pwd_version == 0` rather than treating it as "skip", which is
how the refresh endpoint's version of this check could be bypassed.
`AuthError::PasswordChanged` is deliberately not refresh-required: a
refresh applies the same check, so the client must sign in again.
Enforcement on the request path lands in the next commit.
* fix(auth): scope-bind access tokens, wipe prompted passwords, drop dead file
**Access tokens** signed `(timestamp, ttl)` and nothing else, so any valid
token was valid at every place a token was accepted - one token minted for
any purpose opened all of them. The scope is now mixed into the keyed hash,
so a token minted for one capability does not verify against another. The
token string format is unchanged; only the signed payload grew.
The scope is a compile-time constant, never caller-supplied: both sides of
a handshake have to agree on the exact bytes or the signature fails, which
is what keeps mint and verify from drifting.
This binds to a capability, not to a resource. The internal web player
mints one token that travels the whole chain - webplayer or recording
entry point, the xtream handler they delegate to, and the
custom-video-stream fallback the stream layer redirects into - and those
sites do not share a target or a virtual id to bind against. Per-resource
binding needs that redirect chain traced against a running server; the
scope parameter is where it goes when it is.
**Prompted passwords** were left sitting in two `String`s after
`generate_password` returned. `UserCredential::zeroize` already applies
this discipline to a password that arrives over HTTP; one typed at a
terminal now gets the same treatment.
**`backend/auth/src/user.rs`** was never declared in `lib.rs`. It was a
dead duplicate of the `UserCredential` in `shared::model::auth::user`.
* fix(auth): enforce pwd_version and live permissions on every guarded route
Two checks existed and neither ran where it mattered.
**Password version.** `pwd_version` was consulted only by the refresh
endpoint, so changing a password invalidated nothing on any guarded route -
a token minted against the old password kept working until it expired.
`authenticate` now applies `validate_password_version` to every request
whose principal is a web user. Proxy API users authenticate against
`api_proxy.yml` and carry no password version, so there is nothing to
compare and the check is skipped for them rather than failing them.
The refresh endpoint's own copy read `pwd_version != 0 && pwd_version !=
current`, so a token carrying `0` skipped the check. It now uses the same
strict validator as everything else.
**Permission revocation.** `require_permission` read `claims.permissions`,
a snapshot from mint time, so revoking a group permission had no effect
until the token expired. The effective set is now the intersection of the
claim with what the live config grants: a revocation takes effect on the
next request, while a new *grant* still needs a refresh, because a token
must never end up with more authority than it was issued with. A principal
the web-auth config has never heard of has no live set and keeps its claim.
Three permission paths had drifted apart and now share one implementation:
`rbac_api` had a hand-rolled check that verified the signature and read the
claim directly - no schema gate, no subject gate, no password version, no
live intersection; `v1_api_config::decode_permissions` filtered config
output off the raw claim; and `get_username_from_auth_header` decoded with
a bare `Validation::new`, which checks `exp` and nothing else.
**`permission_layer!`** now expands to `require_permission::<{P as u32}>` -
a bare `fn` item with the permission fixed at monomorphisation, rather than
a closure capturing a runtime `Permission`. `Permission::from_repr` (new on
`create_bitset!`) recovers the variant on the other side. The 15 call sites
are unchanged.
**`AuthorizedClaims<const P>`** is the same requirement in a handler
signature, handing the handler the claims the layer already verified -
they used to be dropped, so handlers behind a layer decoded the token
again. Its rejection is a two-word `Copy` enum, not a rendered `Response`.
* refactor(processing): exec_processing takes a run, not twelve arguments
Twelve positional parameters, seven of them `Option<_>`. A call site was a
wall of `None`s and `Some(..)`s where the reader had to count commas to
work out which knob was being set, and the compiler could not catch two
same-typed arguments swapped. The CLI path was literally
`exec_processing(&client, cfg, targets, NoopSink, None, None, None, None,
None, None, None, None)`.
`ProcessingRun` takes the four that are always present as constructor
arguments and names the rest. The CLI path is now
`ProcessingRun::new(client, cfg, targets, NoopSink)`. Setters take
`impl Into<Option<T>>`, so a site that already holds an `Option` passes it
through unchanged and one that holds a value does not have to wrap it.
`client` moved from `&reqwest::Client` to an owned clone - the client is an
`Arc` internally, so this is a refcount bump, and it drops a lifetime
parameter from the struct.
* perf(events): drop the last four `as Arc<dyn EventSink>` casts
`exec_processing` has been generic over `E: EventSink` for a while, but
every caller handed it `Arc::clone(&event_manager) as Arc<dyn EventSink>`.
The blanket `impl<T: EventSink + ?Sized> EventSink for Arc<T>` made that
compile, so it looked done - but the monomorphisation was against
`Arc<dyn EventSink>`, and every `emit` on the update path still went
through a vtable.
Deleting the four casts is the whole change. `EventManager` is now the
concrete `E`, and the emit sites in the playlist pipeline are direct calls.
* perf(processing): the update bootstrap is a trait, not two layers of boxing
`PlaylistUpdateBootstrap` was
`Arc<dyn Fn() -> Pin<Box<dyn Future<Output = ()> + Send>> + Send + Sync>`:
an erased closure returning an erased, heap-allocated future, for work that
runs exactly once per update. Every call site had to spell out both
coercions - an `Arc::new`, a `Box::pin`, and two `as` casts - around a
three-line closure.
`UpdateBootstrap` is one trait with an RPITIT future and a blanket impl for
`Fn() -> impl Future`. `ProcessingRun` carries it as a type parameter, so
the call sites are now just the closure:
.with_bootstrap({
let state = Arc::clone(&app_state);
move || {
let state = Arc::clone(&state);
async move { sync_panel_api_exp_dates(&state).await }
}
})
`NoBootstrap` - the type parameter for a run without one - is a function
pointer rather than a unit struct, so it satisfies the same blanket `Fn`
impl and needs no second impl to conflict with it. No value of it is ever
constructed.
`with_bootstrap` changes the run's type parameter, so it rebuilds the
struct rather than mutating it; the other setters are unchanged.
* perf(processing): MetadataUpdateSink loses the vtable and the boxed futures
The pipeline held the metadata worker as `Arc<dyn MetadataUpdateSink>` and
the trait's two async methods returned
`SinkFuture<'_, T> = Pin<Box<dyn Future + Send>>`. That is a heap
allocation per `prepare_enqueue_state` - once per input - and a vtable hop
on `should_skip_enqueue`, which runs once per playlist item. There is
exactly one implementor, so nothing ever needed the erasure.
The futures are returned by value (RPITIT) and `PlaylistProcessingContext`
carries the sink as a type parameter. `NoopMetadataSink` names the
parameter for a run without a worker - the same role `NoopSink` plays for
events - and its methods are correct no-ops rather than `unreachable!`,
since the whole point of the type is to be absent.
Making the trait non-dyn-compatible is what forced the last four
`as Arc<dyn MetadataUpdateSink>` casts out of the composition root; the
compiler will not let them come back.
`PlaylistProcessingContext`'s `Clone` is written out rather than derived:
the derive would demand `M: Clone`, but the sink is behind an `Arc` and is
cloneable whatever `M` is.
* feat(auth): stable subject ids come from the identity registry
`create_jwt_web_user` and `create_jwt_api_user` synthesised the subject as
`format!("web:{username}")` / `format!("api:{username}")`, with a TODO
saying the registry would provide it. The registry was fully built - with
persistence, bootstrap, fail-closed recovery and an explicit `rename` that
preserves the id - and never wired into the server, so the TODO was the
live behaviour: the subject was a function of the display name, and
renaming a user reassigned every recording the old subject owned to a
principal that does not exist.
`IdentityRegistry` now lives on `AppState`, bootstrapped from the storage
dir with the current web and API principals. The login and refresh paths
resolve the subject through it - `register` is get-or-create, so a user
bootstrap already synced keeps their id and one added since gets a fresh
one. `register_api_user` and `lookup_api_by_username` are new: the API
namespace had a bootstrap sync path but no way in for a principal that
appears at runtime.
A corrupt registry refuses to start rather than inventing replacement ids,
which is the whole reason the registry's fail-closed path exists. The
pre-scan that would hand bootstrap the subject ids already referenced by
persisted recordings is still unwired, so a *missing* registry alongside
existing recordings initialises fresh; a corrupt one - the case a
half-written file actually produces - fails closed regardless.
* feat(auth): back off repeated failed sign-ins
`/auth/token` verified an argon2 hash, answered 401 and forgot. Nothing
counted how often that happened, so a password list could be worked against
it as fast as the hash function allows, for as long as the attacker liked.
The reverse-proxy rate limiter is opt-in, disabled by default, and applies
one blanket budget to every route - it is not a credential-stuffing control.
`LoginThrottle` tracks consecutive failures on two dimensions, because
either alone is defeatable: by client address, which stops one host
grinding a list but not an attacker with an address pool; and by username,
which stops a distributed attack converging on one account. The username
dimension is a denial-of-service lever if handled carelessly - anyone who
knows a username could lock its owner out - so its block is short (15
minutes at the ceiling) and a correct password clears it immediately.
Three free attempts, then 2s doubling to the ceiling, and a 429 carrying
`Retry-After` so a well-behaved client backs off rather than hammering. The
check runs *before* the argon2 verify: an attacker who can still force the
hash on every attempt has not been slowed down.
Usernames are canonicalised the way the rest of the auth path compares
them, so `Alice` and `alice` share one budget against one account.
The address dimension is only as trustworthy as the address, which this
server still takes from `X-Forwarded-For` with no trusted-proxy allowlist.
Until that is fixed the username dimension is the one doing the work.
* feat(auth): authentication decisions reach the event bus
Sign-ins, rejected sign-ins and permission denials went to `warn!` and
`debug!` and nowhere else. Nothing that subscribes to the bus - a
notification channel, a plugin, an audit sink - could see any of them, so
the events that matter most for spotting an intrusion were exactly the ones
the bus never carried.
`EventMessage::AuthAudit` carries one decision. Following
`UserLifecycleEvent`: one payload, four `EventKind`s, so a subscriber can
ask for the failures without being woken by every successful sign-in. Each
kind is registered with a severity and a description, so it shows up in the
notification config like every other event.
The record holds a username, an address and an outcome. The password and
the token are not in the type at all rather than being redacted at each
site that renders one - these records reach Telegram, webhooks and shell
commands, several of which log on their own. Notifications dedupe per
principal, address and outcome, so a password-guessing run is one piece of
news rather than one per attempt.
`required_permission` is `UserRead`, not the `SystemRead` the other
operational events take: who signed in is the same question as who may read
the user list, and that is the narrower answer. They are not pushed to the
Web UI socket - no panel renders them, and every connected admin does not
need every sign-in.
`token` split into `web_user_sign_in` and `api_user_sign_in` on the way
past. The two branches stay sequential and short-circuiting: the API branch
compares credentials and can allocate a persisted subject id, neither of
which a successful web sign-in should trigger. `SignInAttempt::Refused`
distinguishes "these are not credentials for this branch, try the next" from
a decided answer, which is what keeps a `ui_enabled: false` API user from
falling through to the generic 401.
* feat(auth): tokens can be revoked
The tokens this server mints are stateless JWTs: once issued, nothing could
take one back. A leaked token stayed valid until it expired, and there was
no way to end a session, sign a principal out of every device, or respond
to a compromise short of rotating the signing secret - which kills every
session for every principal at once, is not reversible, and leaves no
record of who did it or why.
`TokenRevocations` is a revocation *watermark* rather than a deny-list of
individual tokens: per subject, "everything issued at or before this
instant is dead", plus one global watermark for the same statement across
all subjects. Two things follow from that shape. It is bounded - a
deny-list grows with every revoked token and needs an expiry sweep; a
watermark is one timestamp per principal. And it revokes sessions a
deny-list cannot name: "sign out everywhere" and "revoke everything issued
before the breach" have no list of token ids behind them. The cost is
precision - it cannot revoke one session and spare another issued in the
same second - which is the right trade for what it is for.
`iat` is compared with `<=`, not `<`: it has one-second resolution, so a
token minted in the same second as the revocation would otherwise survive
it. Over-revoking by up to a second is the safe direction.
Revocations are persisted, because the tokens they revoke outlive the
process. An in-memory revocation would be a security control that quietly
stops applying at the next restart. A file that will not parse is an error
at startup rather than an empty store - reading it as "nothing is revoked"
would silently reinstate every revoked session.
`POST /auth/revoke/{username}` ends one principal's sessions across both
identity namespaces; `POST /auth/revoke` ends everyone's. Both state their
`UserWrite` requirement through `AuthorizedClaims` in the handler
signature, because the `/auth` routes are mounted before any state exists
to build a router layer from - which is the case that extractor was for.
The refresh endpoint checks revocation too: a revoked token that could be
exchanged for a fresh one would make revocation a formality.
* refactor(iptv): page arithmetic has one home
A Stalker catalog page is the last one when it came back empty, when it is
shorter than the advertised page size, when the accumulated row count reaches
`total_items`, or when `max_page` says so. That rule was written out four
times - in the accumulating paginator, in `parse_item_catalog_page`, in
`parse_series_catalog_page` and in the `apply_page_limit` guard - and the two
`parse_*_catalog_page` copies had already drifted from the loop's version in
how they measured progress against `total_items`.
`PageMeta::is_terminal` is now the single copy, with the progress measurement
made explicit: accumulating callers pass their real running count, single-page
callers pass `PageMeta::fetched_estimate`, which is deliberately zero when the
portal advertises no page size so the `total_items` arm stays inert rather
than truncating the catalog at page one.
The two per-row-type page parsers collapse into one generic walk as well. They
differed only in `T`, and both hand-rolled the same four-shape row collection
(bare array, `data` array, `data` object, id-keyed envelope).
Behaviour is unchanged. Not verified by tests - checked with cargo check and
nightly clippy on the crate.
* feat(iptv): catalogs can stream instead of accumulating
`get_live_streams` and friends buffered an entire provider catalog into a
`Vec` before the caller saw a single row, while the EPG path next door already
had the right shape - `stream_bulk_epg` hands over batches as they arrive.
Catalogs now have the same option.
The interesting part is not the callback but what it costs. The client tries
several endpoint candidates in turn, and a failure part-way through pagination
abandons that candidate and restarts on the next one. That retry is only sound
while nothing has left the client, so the sink decides: `CollectSink` holds
everything and can restart freely - the historical behaviour, and the reason a
truncated catalog is never returned as `Ok` - while `BatchSink` reports that it
can no longer restart once a page has been released, and the driver then
returns the failure as-is. The streaming methods document that an `Err` makes
the delivered batches an incomplete prefix.
`get_*_paginated` are now thin adapters over the same driver rather than a
second copy of the pagination loop.
Not verified by tests - checked with cargo check and nightly clippy.
* refactor(iptv): expiry rules take the instant, they no longer read the clock
Session staleness, cookie `Max-Age`, and the Xtream account-expiry warning were
all pure functions of "what time is it" that reached for the clock themselves.
That made every one of them assertable only by approximation: the session test
back-dated a struct field by hand, the cookie test slept for two milliseconds
and then checked the cookie was still there, and the three-day expiry window
had no test at all because reaching it meant waiting for the calendar.
Each now takes the instant as a parameter, following the pattern the workspace
clock module recommends for exactly this case - the deadline logic in
`tuliprox-hls` already does it this way. The old signatures survive as
wrappers over the system clock, so no caller changed.
`crate::clock` holds the one epoch-seconds conversion, and is the only place in
the crate that reads wall time without being handed it.
What this buys, concretely: the cookie boundary is now asserted at the exact
second it flips, a backwards-running clock is shown not to age a session, and
the expired / expiring / quiet split of the account-expiry warning has tests
for all three branches.
Not verified by test execution - checked with cargo check and nightly clippy.
* feat(iptv): the Stalker client's network and clock are seams
`StalkerApiClient` owned a `reqwest::Client` and read the system clock, which
put every interesting decision it makes behind a live portal: the recipe
fallback chain, endpoint-candidate failover, pagination termination, body caps,
and the portal's habit of reporting `{"code": 44, "text": "Account is blocked"}`
inside a `200 OK`. The module docs conceded it outright - "no HTTP requests are
issued from unit tests" - which is another way of saying none of that was
tested.
Both dependencies are now type parameters defaulted to the production
implementation, the shape the workspace clock module recommends. `new()` is
unchanged, `SystemClock` is zero-sized, and neither seam introduces a vtable or
an allocation, so nothing about the production path moved.
Only the *send* is abstracted - requests are still built with reqwest's builder,
because a fake has to build them too and re-modelling a query string buys
nothing. `execute` returns a domain error rather than `reqwest::Error`, which
callers converted anyway and which `reqwest` will not let anyone else construct.
Nine tests now cover paths that previously had no way to be reached at all: a
portal refusal hidden in a 200 body surfacing as a token rejection, an
over-cap body being refused rather than buffered, an HTML error page not
decoding as JSON, endpoint-candidate failover in priority order, exhaustion
reporting the last real failure rather than a synthetic one, pagination
stopping on the advertised last page, a truncated catalog never being returned
as success, the streaming variant making the opposite trade explicitly, and a
session ageing past its TTL on a clock that can be advanced.
`inspect_portal_code` became a free function - it never touched `self`, and as
an associated function on a now-generic type its callers could not infer `Tr`.
Checked with cargo check and nightly clippy across the crate and its two
consumers. The tests added here have NOT been executed.
* refactor(iptv): one redaction module and one error classification
The crate had three unrelated answers to "what must never reach a log line":
`safe_stalker_url` for error URLs, an inline key list inside the debug-dump
writer, and `sanitize_sensitive_info` on the Xtream side. The sibling
media-server crate already keeps that in one module; this is its counterpart,
so the sensitive-key list is defined once and every path that renders a
provider string goes through it. `safe_stalker_url` survives as the name its
callers already use.
Alongside it, `StalkerErrorKind` mirrors `MediaServerErrorKind`. Callers were
matching on variants to answer questions the variants were never organised
around - is the provider down or is my token stale, is this worth retrying -
and a 403 could arrive as either `TokenRejected` or `BadStatus` depending on
which layer noticed it. `kind()` collapses that, and `is_retryable()` is
deliberately false for auth failures so nothing loops on a rejected token.
Also adds a JSON redaction walk with a test that a nested `cmd` is caught,
which the debug-dump writer's own copy never had.
Not verified by test execution - cargo check and nightly clippy only.
* feat(iptv): providers remember what they already told us
Capability knowledge was discovered and then thrown away every refresh.
Whether a portal implements `get_all_channels` was inferred from the shape of
the error it returned; which bootstrap recipe worked was found by walking a
five-entry chain from the top; which of three endpoint candidates answered was
rediscovered per call. None of it survived, so every refresh re-probed
endpoints already known to 404 and replayed a chain whose answer was known.
That is not an ordinary cache miss. A handshake chain replayed from scratch
against a portal with stale credentials looks, from the provider's side, a lot
like credential stuffing - the failure mode the Xtream side already carries a
standing TODO about.
`ProviderCapabilities` is the snapshot, and it is a hint rather than a
contract: every claim carries the instant it was observed and expires after a
day, so a provider that starts implementing an action is picked up without
anyone clearing state by hand, and a remembered endpoint is moved to the front
of the candidate list rather than replacing it. A clock that has run backwards
leaves the snapshot alone instead of invalidating everything.
`CapabilityStore` has two implementations: in-memory for a single run, and one
JSON file per input written through the workspace's atomic-write helper. Input
names come from user config, so the filename is derived rather than taken
verbatim; a corrupt file is ignored rather than fatal, because re-probing is
always available.
Wired into three places that now behave differently: `get_all_channels` is not
re-probed on a portal that has refused it, the handshake chain starts at the
recipe that last worked, and endpoint candidates start at the one that last
answered. Tests cover each, including the cases where the remembered answer has
gone stale or gone away.
Not verified by test execution - cargo check and nightly clippy across the
crate and both consumers.
* feat(iptv): one shape for "fetch this input's playlist"
The three provider families were modelled three different ways - M3U as free
functions, Xtream as free functions with a different arity, Stalker as a struct
client orchestrated from another crate - and each returned a differently-shaped
tuple. The dispatcher paid for it: a ninety-line match whose eight arms each
hand-assembled a six-element tuple, padding the fields their provider does not
produce with literal zeros and `false`s, then destructured the lot by position.
Two of the six elements were dead on arrival - bound to `_m3u_error_count` and
`_xtream_error_count`, computed in two arms, read nowhere.
`PlaylistFetch` is that result with names. `PlaylistProvider` is the one method
every family implements. Provider-specific inputs stay on the provider value -
the event sink on Xtream, the refresh mode on Stalker - so `fetch` takes only
what all of them take.
Dispatch stays a match and stays statically dispatched; the providers share no
supertype and constructing one is free. What changed is that each arm now names
one type and awaits it, and the two unimplemented input types are an
`UnsupportedProvider` carrying its reason rather than a seven-line tuple
literal. Net 98 lines out of the dispatcher.
Stalker's orchestration did not move: it reaches into `tuliprox-repository` and
this crate's own processors, and `tuliprox-iptv` sits below both. The trait is
what lets it stay where it is and still answer in one shape - `StalkerProvider`,
`LibraryProvider` and `PlexProvider` live here, `M3uProvider` and
`XtreamProvider` ship with their clients. The Plex fetch moved out of the
dispatcher into its provider on the way.
Not verified by test execution - cargo check and nightly clippy across iptv,
processing and the binary.
* fix(iptv): catalog fetches honour the configured body cap again
Actions were `&'static str`, threaded from the call site into `send_json`, into
the body-cap lookup, and into six error variants as a `String`. The cap lookup
matched those strings with a silent fallback - and the strings it matched were
not the strings the call sites passed. Catalog fetches announced themselves as
`get_ordered_list` and `get_all_channels`; the lookup tested for `ordered_list`
and `all_channels`. Neither ever matched, so both fell through to the 8 MiB
fallback and a user who raised `ordered_list_mb` got 8 MiB anyway, with nothing
logged to say so. It went unnoticed because the fallback and the default happen
to be the same number.
`StalkerAction` is that set as an enum. Every action names its cap, the match is
exhaustive by construction, and the error variants carry something comparable
rather than a `String` each call site had to spell identically for a later
comparison to work - `is_unsupported_catalog_action` was doing exactly that.
The two handshake calls send the same `action=` query against different
endpoints, so the enum is the label rather than the wire value, and they stay
distinguishable in an error.
Behaviour changes only for users who configured a non-default `ordered_list_mb`
or `get_epg_mb`: they now get what they asked for. Session-shaped actions keep
the same fixed cap they landed on before.
Not verified by test execution - cargo check and nightly clippy across iptv,
processing and the binary.
* feat(iptv): provider failures answer one question
The three provider families report failure in two incompatible ways: Stalker
has a typed `StalkerError`, while M3U and Xtream hand back `Vec<TuliproxError>`
drawn from a workspace enum of forty-odd categories. So the dispatcher never
asked the questions it cares about - is this worth retrying, is the provider
down or is the config wrong - and every error was counted, logged and treated
identically.
Rather than a third error type for everything to convert through,
`ProviderErrorKind` is a classification both existing types map onto. Errors
keep their identity; the judgement is what gets unified. It is ordered by how
much attention a failure deserves, so a fetch that hit one timeout and one bad
portal URL is judged on the URL - `PlaylistFetch::error_kind` takes the worst,
not the first.
`Auth` is deliberately not retryable. It is recoverable, but by re-handshaking,
which is a different call; reporting it as retryable is how a client ends up
hammering a portal with a token that portal has already refused.
Also fixes the Stalker-to-workspace conversion, which flattened all fifteen
variants to `ProviderConnection` - "the network had a bad moment". A
misconfigured portal URL and a rejected password both came out as connection
trouble, and both counted as retryable. The conversion now preserves the class.
The workspace `ErrorKind` arms are enumerated rather than matched by name
prefix, so a renamed variant is a compile error rather than a silent
reclassification.
Not verified by test execution - cargo check and nightly clippy across iptv,
processing and the binary.
* feat(iptv): EPG acquisition has one shape too
EPG was wired per provider family and the two never met. Stalker streams
programme records straight into its repository from three calls in this crate.
M3U and Xtream have no EPG here at all: their XMLTV download lives in
processing, produces documents on disk, and is reached from a separate call
site. Neither knew the other existed.
What they share is the question - does this input have an EPG, and what did
fetching it produce - so that is what `EpgProvider` unifies. What they do not
share is the shape of the answer, and `EpgOutcome` says which happened rather
than forcing one into the other: pretending they were the same would mean
either materialising a several-hundred-megabyte XMLTV document into records in
memory, or teaching the Stalker client to write XMLTV it has no reason to
write. The guide handle is an associated type, so the file-based provider hands
back its `TVGuide` and the streaming one hands back nothing.
Two real implementors: `StalkerEpgProvider` here, `XmltvEpgProvider` in
processing next to the download it wraps, now driving `download_input_epg`.
Per-source EPG failures travel alongside the outcome rather than replacing it -
three sources where one 404s still yields a usable guide from the other two.
`EpgProgramRecord` is an alias rather than a new type: the workspace already
has exactly one programme record, in core rather than in any provider's module,
and its Stalker-flavoured name was the only thing provider-specific about it.
Not verified by test execution - cargo check, nightly clippy and the workspace
dependency gate across iptv, processing and the binary.
* style(iptv): format the new modules with the project rustfmt config
Applied to the eleven files added by this branch only. The rest of the crate
predates the current rustfmt.toml and running `cargo fmt --all` over it would
rewrite files this work never touched - that drift is left alone and called out
separately.
* chore: refresh Cargo.lock
Adds `http` as a dev-dependency of tuliprox-iptv (the fake transport builds
responses with it) and picks up the `dashmap` edge tuliprox-auth's manifest
already declared but the lockfile had not been regenerated for.
* docs(changelog): record the messaging, event bus and auth work
Sixty-six commits since a2de1164 had reached the changelog nowhere. The
file has not been touched since the multi-crate split, so every entry
below is new rather than a revision.
Four breaking changes, each one a field that was written and then never
read back: `token_ttl_mins: 0` no longer mints a ~100-year credential, a
request that never authenticated answers 401 rather than 403, `notify_on`
is glob patterns over dotted event ids, and config responses mask the
channel secrets they used to return in full.
The new features are three threads that turned out to be one. The
notification layer became open-world - an event is an id, a channel is an
impl - which is what let four channels, per-channel routing and a durable
outbox land without a dispatcher change. The event bus became the single
backbone underneath it, so an event emitted once reaches the Web UI, the
notification pipeline and a plugin rather than whichever one its emitter
happened to know about. And auth gained the three things a stateless-JWT
server was missing: a throttle in front of the hash, a revocation
watermark behind the token, and an audit trail for both.
Fixes, optimizations, new settings and maintenance are filled out from
the same range. The refactor commits are summarized by what they buy a
user - units in the type system, one shape for config preparation, no
boxing on the playlist traversal - rather than restated as diffs.
Two claims are deliberately narrower than their commit subjects. Provider
capability memory is described as lasting a client's lifetime, because
`JsonCapabilityStore` is built but not yet constructed in the composition
root. And the registered event count is 32, not the 24 the docs commit
wrote - the events added after it never reached the generated table, so
`every_registered_event_appears_in_the_docs_table` currently fails. That
is left for its own commit; this one only says what is true.
* fix(events): watch payloads carry counts, not prose
`handle_watch_notification` truncated its own lists by pushing a
synthesised sentence into them - "... 42 more added entries omitted"
beside real channel titles, or "5000 entries added. Detailed list
suppressed" replacing the list outright.
That was legible to the text template and to nothing else. `payload()`
serialises `WatchChanges` straight to JSON for plugins, and a plugin has
no way to tell a sentinel from a channel actually named that. The
notification subject line had the same problem from the other side: it
read `w.added.len()`, so a suppressed change of five thousand announced
itself as "1 channel(s) added".
`WatchChanges` grows `added_total`, `removed_total` and `truncated`. The
lists stay pure channel titles, the counts stay true whatever the lists
carry, and the plain-text renderer spells the omission out itself.
`WatchChanges::new` sets the totals from the lists so a caller that is
not truncating cannot get them out of step.
* fix(events): a failed library scan stops reporting itself as finished
`notification_id()` mapped `LibraryScanProgress` to
`LIBRARY_SCAN_COMPLETED` whatever the summary said, so the failure path
in `spawn_library_scan` - which emits the same variant with
`status: Error` - reached operators as "A local library scan finished"
at info severity.
The taxonomy now discriminates on the status the payload already
carries, the way `PlaylistUpdate` has always done on its state: success
keeps `library.scan.completed`, failure takes a new
`library.scan.failed` at error severity, and `severity()` picks that up
from the registry with no second table.
`EventKind` gains `LibraryScanFailed` alongside the progress kind rather
than reusing it. `LibraryScanProgress` is high-frequency and a scan
failure is not, so a subscriber that only wants failures should not have
to take the tick firehose to get them.
The emitter is unchanged - it was already reporting the status
correctly, and nothing downstream was reading it.
* feat(metadata): a broken input stops being silent
`InputMetadataUpdatesCompleted` only fires when a cycle drains *with
changes*, and a task that burns through its retries only reaches a
`debug!`. So an input whose resolves fail every time emitted a start and
then nothing, for as long as it stayed broken - on the bus it looked
exactly like one still working through a long queue.
The worker now counts the tasks that exhaust their retries during a
cycle and emits `InputMetadataUpdatesFailed` when that count is non-zero,
carrying the input, the count, whether anything resolved anyway, and the
last error.
Reported alongside the completion rather than instead of it. A cycle can
both produce changes and exhaust tasks, and the completion is what
triggers the downstream playlist update - suppressing it would trade one
silent failure for another.
Per cycle, not per task: a provider that has stopped answering fails
every item behind it, and `dedup_key` is per input for the same reason
`StreamProbeFailure` is.
`EventBusStats::Default` becomes a hand-written impl on the way past -
the taxonomy crossed 32 kinds and `Default` for arrays stops there.
* feat(events): the server says when it starts and stops
`system.started` and `system.shutdown` have been in the notification
registry - and in the documented event table - since it was written, with
nothing in the tree emitting either. An operator who subscribed got
silence.
`EventMessage::ServerLifecycle` carries both, one payload with two kinds
so a subscriber can ask for restarts alone. It reports the running
version, the bound address on start, and the signal name on stop.
Placement is the whole design here. The start event goes after
`spawn_notification_bridge`, not at the top of `main`: the bridge is what
turns a bus event into a notification, and anything published before it
subscribes reaches nobody. The stop event goes before
`cancel_all_service_tokens`, which stops the outbox that would carry it.
Neither reaches the websocket. There is no panel that renders them, and
the Web UI has necessarily disconnected by the time the second one fires.
* fix(admission): keep the strategy index aligned past a suppressed eviction
The strategy loop counted with a manual `idx` incremented at the end of the
body, but the eviction-reentry suppression arm exits via `continue` and so
skipped it. Every strategy evaluated after a suppressed eviction was handed an
index one too low.
That index is what `build_grace_ctx` stores as
`GraceResolutionContext.strategy_index`, and
`evaluate_remaining_strategies_after_grace` resumes at `strategy_index + 1`.
With `[EvictUserOldest, GraceHoldStream]` and the eviction suppressed, the
grace recorded index 0, so the fallback slice restarted at the grace strategy
itself and replayed it instead of moving past it.
Switched to `iter().enumerate()` so the index cannot drift from the item.
Not covered by a test run.
* feat(messaging): a lost notification reaches the bus
`notification.dead_lettered` was registered and documented; the outbox
detected the condition, bumped `health().dead_lettered` and logged to
`notification::audit`, and that was the end of it. Nothing subscribing to
the bus could learn that a notification had been permanently lost.
The outbox now takes an `EventSink` - generic, like the rest of the
emitters after the static-dispatch pass - and emits
`NotificationDeadLettered` at the point it gives up, carrying the event
id, the attempt count, the channels that never accepted it, and when it
was first enqueued.
It is deliberately *not* notifiable, and deliberately absent from
`NOTIFIABLE_KINDS` so the bridge is not even woken for it. This event
exists because delivery failed; enqueueing a notice about it into the
same outbox, against the same channels that just failed, is the loop its
registry entry warns about. Operators still get the audit line and the
counter - neither runs through the path that broke - and plugins see it
on the bus.
Emitted at the attempts-exhausted site only. A notification every channel
rejects as permanent is also dropped, but that path cannot currently be
told apart from a clean delivery without tracking that does not exist
yet.
* fix(admission): re-read admission after acquiring the per-user gate
`resolve_admission_with_strategies` read the admission state, then queued on
`acquire_user_admission`, then walked the eviction strategies using the snapshot
it had taken before it queued. A request that lost the race therefore acted on a
count the winner had already changed, and could evict a live connection to free
a slot that had been released while it waited.
The re-read is placed after the gate and after the empty-strategy check, so the
uncontended path costs nothing extra, and `build_grace_ctx` now captures the
fresh `kind`.
This does not close the wider check-then-register window: `connection_admission`
only inspects the counts, and the slot is registered by the caller outside this
gate, so two requests at `max_connections - 1` can still both be admitted. That
needs the registration brought under the same gate and is left alone here.
Not covered by a test run.
* fix(admission): stop evicting once a kick frees no connection
Eviction is destructive and is never rolled back. The strategy loop would kick a
target, find the retry still denied, and move on to the next eviction strategy -
so a request that ended up denied anyway could leave several other streams killed
behind it.
The loop now samples `user_connections` either side of the kick. A kick that
reduces the count is real progress and later strategies still run, which keeps
the over-limit case (a hot-swapped config that lowered `max_connections`)
converging. A kick that frees nothing sets `evictions_ineffective`, and further
`Evict` decisions are skipped for the rest of the walk; `Grace` strategies are
still evaluated.
Not covered by a test run.
* refactor(admission): drop the unused kind_for_exhausted parameter and refresh docs
`evaluate_admission_strategy_loop` took `_kind_for_exhausted` and never read it -
both callers construct the exhausted result themselves. Removing it takes the
argument count down by one on a function that needed
`#[allow(clippy::too_many_arguments)]`.
The doc block on `evaluate_remaining_strategies_after_grace` listed a `Deny`
rule, but `AdmissionDecision` only has `NoMatch`, `Grace` and `Evict`. Replaced
it with what the `Grace` arm actually does now that the strategy index it records
is correct.
Not covered by a test run.
* feat(processing): the watch feature stops failing silently
`watch` had one event for everything it knows, and three ways to stop
working without saying so:
* every pattern failed to compile, so `ConfigTarget::watch` became `None`
and `process_watch` returned immediately - the feature disabled itself
on a typo behind a single `warn!`;
* the target carries the reserved default name, which logs and returns;
* the watch state file could not be read or written, which either
re-baselines the group - losing the change it should have reported -
or drops it entirely.
All three now emit `playlist.watch.disabled` with the reason and, where
there is one, the underlying error.
The first needed the config layer to stop discarding the distinction. An
empty `Some` is now load-bearing: it means "configured and unusable",
which is not the same as "not configured", and only the runtime layer has
an event sink to report it from.
`playlist.watch.unmatched` covers the fourth silence. A pattern matching
no group looks exactly like a group that has not changed, so a typo in
`watch` is invisible. `EventKindMask::from_wire_names` already returns
unmatched subscription names for this reason, and its test says why: a
typo must surface, not silently narrow what was asked for.
Matching now walks the groups once and records which patterns hit,
instead of re-testing every pattern per group inside a filter.
* refactor(admission): make the empty-strategy-list rule explicit
`get_effective_admission_strategies` matched on `admission_strategies.is_some()`
and then re-unwrapped with `unwrap_or_default()`, so the guard proved something
the body checked again. More importantly, the rule that an explicitly empty list
suppresses the `grace_period_millis` fallback - while an absent list does not -
was implicit in the arm ordering.
Rewritten as a match on `admission_strategies.as_ref()` with the distinction
stated. Behaviour is unchanged: `Some(vec![])` still means "no strategies", not
"fall back to grace".
Not covered by a test run.
* perf(admission): carry the effective strategy list as Arc<[AdmissionStrategy]>
`GraceResolutionContext` is stored on `StreamInfo` and travels with every clone
of it, so a `Vec<AdmissionStrategy>` field meant reallocating the list on each
clone. `get_effective_admission_strategies` also handed back a fresh `Vec` that
`build_grace_ctx` then cloned again.
The list is immutable once resolved, so `Arc<[AdmissionStrategy]>` fits: the
context clone is now a refcount bump, and the one allocation left is the
`Arc::from` at resolution time. `evaluate_admission_strategy_loop` still takes a
plain `&[AdmissionStrategy]`, reached by deref, so slicing the remaining
strategies is unchanged.
Not covered by a test run.
* feat(processing): a target reports the groups it gains and loses
`watch` tracks channels inside named groups and is blind to the group set
itself, in both directions.
A group appearing was silent: `process_group_watch` found no baseline
file, wrote one and emitted nothing, so the group's entire channel list
read as "not new" from then on. A group vanishing was worse - it is
absent from the refreshed playlist, so nothing iterated it and no code
path observed the disappearance at all.
`process_target_groups_watch` diffs the target's group titles against a
persisted index and emits `playlist.groups.changed`. It runs before the
per-group fan-out and sees every group, not only the ones the watch
patterns name - the question is which groups exist, not what is inside
the watched ones.
The index sits beside the per-group directory rather than inside it
(`<target>.groups.bin`, not `<target>/__groups.bin`) so it cannot collide
with a group whose sanitized title matches. First sight writes the
baseline and says nothing; announcing every group as new on the first
refresh after an upgrade would be noise.
Sampled the same way `WatchChanges` now is: titles only in the lists,
counts that stay true whatever the lists carry.
Gated on `target.watch` being configured, so it costs nothing for targets
that never asked to be watched.
* refactor(admission): bundle the request-scoped admission arguments
`resolve_admission_with_strategies`, `evaluate_remaining_strategies_after_grace`,
`evaluate_admission_strategy_loop`, `get_admission_for_request` and
`should_suppress_eviction_for_recent_request` each threaded the same ten
positional parameters, three of them consecutive bare `bool`s
(`use_session_admission`, then `activate_unbound_session` a slot later). Callers
read `..., true, Some(session_token), true, guard)` - a shape where transposing
two arguments still compiles and silently changes which admission check runs.
They now take one `AdmissionRequest<'a>`, which names every field at the call
site. The `use_session_admission` comment that lived in the parameter list moved
onto the field it documents.
This removes three `#[allow(clippy::too_many_arguments)]` and one
`clippy::too_many_lines`.
Formatted with the project rustfmt config. Not covered by a test run.
* feat(session): the provider pool says when it runs out and when it falls back
Two moments the lineup manager knew about and told nobody.
`provider.pool.exhausted`: `log_exhausted_pool_snapshot` built a complete
picture - per-provider current/max plus expiry - and discarded it unless
debug logging happened to be on. `ActiveProvider` reports that connection
counts moved; nothing reported that a stream was refused because every
provider behind the input was full. The snapshot is now built
unconditionally on that path (already the slow one) and both the debug
line and the event render from the same structured data.
`provider.priority.fallback`: `acquire` walks priority groups highest to
lowest and silently falls through when the preferred ones are at
capacity, so "you are being served off your backup" was invisible.
Reported on transition, not per allocation - the fallthrough happens on
every request while the primary is full, and one event per stream start
would bury the thing worth hearing. A move back towards group zero is a
recovery and says so.
The group is resolved after the fact from the allocated provider rather
than by threading an index out of `acquire`, which keeps the allocation
path and its return type untouched.
Input and provider names go through `sanitize_sensitive_info` before
reaching the payload, for the reason `StreamProbeFailure` documents.
* feat(processing): a failed playlist fetch says what kind of failure it was
`ProviderErrorKind` already classifies every provider failure across all
three families, and already exposes `is_retryable()` and
`needs_operator()` - the two questions an operator actually asks. Nothing
consumed either. Every fetch failure was counted, logged and treated
identically.
The dispatcher now emits `provider.fetch.failed` when a fetch reports
errors, carrying the classification, the worst error's text, how many
there were, and whether any of the playlist came through anyway.
Severity follows the classification rather than the registry: a `Config`
failure will not fix itself and is an error, everything else may and is a
warning. That is the same payload-dependent override `PlaylistUpdate`
already uses for a partial refresh.
`ProviderFailureKind` mirrors `ProviderErrorKind` in `shared`, which
cannot host the original - it classifies a `StalkerError` that `shared`
does not know about. The conversion lives beside the original so a new
variant there is a compile error rather than a silent fallthrough.
Input name and error text go through `sanitize_sensitive_info`: both can
carry a provider URL with credentials in it.
* feat(app): a scheduled task that fails says so
The playlist update and the library scan both report their own outcomes.
The GeoIP refresh had no terminal event of its own - it logged one line
and moved on - so an operator running on a stale database never found
out.
`scheduled_task.failed` carries the task type, the cron expression that
triggered it, and the error. `GeoIpUpdateError::Disabled` stays silent:
the task ran and correctly found nothing to do, which is not a failure.
The task is typed as `ScheduleTaskType` rather than a free string, so a
task added to that enum cannot be reported under a name nothing
recognises.
* feat(session): a refused connection reaches the bus
`ActiveUser` reports connects and disconnects. A refusal is neither, so
the one outcome a user actually complains about was the one nothing
published - the admission ladder modelled it fully, as what is left after
every eviction strategy declines, and handed it to the caller and nobody
else.
`user.connection.denied` carries the user, the address the request was
attributed to, and the limit that was reached. It sits with the auth
events rather than the streaming ones and takes `UserRead`: "who was
turned away" is the same question as "who signed in", and strictly
narrower than the system-wide read.
Only the strategy path emits. An explicit `Terminate` also resolves to
`Exhausted`, but that is a requested teardown, not a denial.
`ActiveUserManager` gains an `events()` accessor rather than `AdmissionCtx`
growing a second handle to the same manager.
* refactor(app): the notification bridge stays a routing table
Fourteen new events left `to_notification` doing its own wording inline,
at which point it was 213 lines and clippy was right about it. Each new
event's wording moves into a `*_notification` builder beside the three
that already existed, and the match goes back to being one arm per
variant with no logic in it.
`push_sampled_list` is shared by the two events that carry sampled lists,
so the "... N more not listed" wording has one home rather than two.
Also completes the documented event table. Eight events registered by the
user-lifecycle and auth work - `user.created`, `user.updated`,
`user.deleted`, the four `auth.*` decisions and `stream.probe.failed` -
were never given a row, so `every_registered_event_appears_in_the_docs_table`
has been failing since before this branch. The rows are generated from
the descriptors so severity and description match exactly.
`stream.probe.failed` had a run of twenty-seven spaces inside its
description, left behind when a wrapped literal was collapsed onto one
line. Normalised in the registry and the table together.
* docs(changelog): record the bridge refactor and the completed event table
The eighteen commits since 4dee77a6 reached the changelog in d4b9c7d9,
which was about the notification bridge and carried CHANGELOG.md along
with it. This commit adds the two entries that commit earned and its own
message did not put in the file.
The docs table entry is the one an operator can act on: eight registered
events had no row, so `every_registered_event_appears_in_the_docs_table`
was failing and anyone reading section 5 to pick a `notify_on` pattern
could not see `user.*`, `auth.*` or `stream.probe.failed` at all.
The bridge refactor is recorded as maintenance because it buys one thing
a reader cares about - each event's wording has one home - rather than as
a line count.
* fix(xtream): a blank container extension falls back to the item url
A VOD document's `container_extension` comes from the provider's
`get_vod_streams` response, where a missing or null value collapses to an
empty string, and nothing fills it in afterwards unless a per-item
`get_vod_info` fetch or an ffprobe run happens to reach it. A provider
that omits the field therefore published `""` on every VOD, and a client
building `<stream_id>.<container_extension>` has nothing to append -
several stringify the blank and go on to request `813563.null`.
`get_vod_info` already fell back to the extension carried by the item URL.
The four builders that did not now share one helper on
`XtreamPlaylistItem`: the stream-list document, both no-properties paths,
and the resolved info document. That last one is post-processed on the
returned `XtreamInfoDocument::Video` rather than threading `url` through
`StreamProperties::to_info_document`, whose signature is shared with the
series and episode paths that have no URL to pass.
A non-empty provider value still wins, so the fallback cannot talk over
what the provider actually said, and an item whose URL carries no
extension either still reports the blank.
`create_vod_info_from_item` had the fallback already but kept the leading
dot that `extract_extension_from_url` returns, publishing `.mkv` into a
field that holds `mkv` everywhere else - a client would have built
`813563..mkv` from it. Stripped.
Series episodes are left alone. `SeriesStreamDetailEpisodeProperties`
carries no provider URL at that layer, only a `direct_source`, so the same
fallback has nothing to read there.
The six new tests are compiled but unrun.
* fix(frontend): reload plans when opening user create/edit screen
UserEdit previously fetched user plans only once on initial mount (use_effect_with((), ...)). Because panels are kept mounted in the SPA, newly created or modified plans in the Plans view did not appear in the plan dropdown until a full page reload or container restart.
Update effect dependencies to include active_page and selected_user so plans are re-fetched whenever navigating to UserlistPage::Edit.
* refactor(requests): replace reqwasm with gloo-net for HTTP requests
* refactor(locking): drop fs2 for std file locking
fs2's last release was 2018. Rust 1.89 stabilized File::lock,
lock_shared, try_lock, try_lock_shared and unlock together with
std::fs::TryLockError, and the workspace MSRV is 1.95 — so the
dependency comes out entirely rather than being swapped for fs4.
The syscalls underneath are unchanged. The one semantic shift is
try_lock: contention now arrives as TryLockError::WouldBlock instead
of an io::Error carrying ErrorKind::WouldBlock, which makes "lock is
held" a distinct variant from a real I/O failure at the type level.
* refactor(catchup): introduce effective_mode method to prioritize catchup-type
* refactor(catchup): enhance timestamp validation and error handling for catchup windows
* refactor(mapping): prepare extensible processing pipeline
* refactor(processing): compile target execution pipeline
* clippy fixes
* test fixes
* test fixes
---------
Co-authored-by: DarkBreakpoint <darkbreakpoint@github.com>
Co-authored-by: DarkBreakpoint <243206744+DarkBreakpoint@users.noreply.github.com>
Co-authored-by: euzu <euzu@proton.me>
132 KiB
132 KiB
🔌 Pillar 2: source.yml (Inputs, Panel API & Targets)
The source.yml serves as the central orchestration hub for all data flows within Tuliprox.
It defines the lifecycle of a stream—from the upstream provider to the end-user device—through three primary
architectural layers:
providers(Resilience Layer): Defines backend endpoints and failover logic. Use this to implement intelligent URL rotation and ensure high availability across multiple mirrors.inputs(Ingestion Layer): Manages upstream data sources. This layer handles credential management, connection pooling via Aliases, and automated account lifecycle management through Panel API integration.sources&targets(Egress Layer): The final mapping stage where ingested data is filtered, transformed, and routed to specific Targets (M3U, Xtream, Strm or HDHomeRun) for consumption by end devices.
Top-level entries
templates:
provider:
inputs:
sources:
| Block | Description | Link |
|---|---|---|
templates |
(Legacy) Inline templates for filter macros. Prefer template.yml. |
|
provider |
Provider Failover & DNS Rotation definitions. | See section |
inputs |
Data Sources (Providers, Files, Batches, Library). | See section |
sources |
Routing logic combining inputs to output targets. | See section |
1. Provider Failover & DNS Rotation (provider)
Tuliprox includes a robust failover engine for unstable IPTV providers. You can define backup URLs and intelligent IP rotation.
Define a provider block globally in source.yml to specify multiple backup URLs:
provider:
- name: my_failover_provider
urls:
- http://primary.example.com
- http://backup.example.com
provider_url_selection_policy: resume_last_working # or restart_from_first
dns:
enabled: true
refresh_secs: 300
prefer: ipv4 # system, ipv4, ipv6
schemes: [ http, https ]
keep_vhost: true
max_addrs: 2
on_resolve_error: keep_last_good # or fallback_to_hostname
on_connect_error: try_next_ip # or rotate_provider_url
overrides:
"primary.example.com":
- 203.0.113.10
Provider Parameters (provider[])
| Parameter | Type | Default | Technical Impact |
|---|---|---|---|
name |
String | required | Internal provider identifier referenced by provider://<name> URLs. Must be unique within the provider list. |
urls |
List | required | Ordered failover URL list for this provider. Tuliprox rotates through these URLs within a request when failover is triggered. |
provider_url_selection_policy |
Enum | resume_last_working |
Controls how a new request chooses its starting URL. resume_last_working starts at the last successful URL. restart_from_first always begins again at urls[0] and only fails over within that request. |
dns |
Object | unset | Optional DNS/IP rotation settings for the provider. See the table below. |
DNS Rotation Parameters (provider.dns)
| Parameter | Type | Default | Technical Impact |
|---|---|---|---|
refresh_secs |
Int | 300 |
The interval in seconds the background task resolves the hostnames. (Minimum effective value is 10). |
prefer |
Enum | system |
Which IP protocol to prefer during DNS resolution. Options: system, ipv4, ipv6. |
max_addrs |
Int | None |
Hard limit on the number of resolved IPs to retain per host. |
schemes |
List | [http, https] |
The HTTP schemes that IP connection rotation applies to. |
keep_vhost |
Bool | false |
If true, the Host header retains the original hostname[:port]. If false, it uses IP[:port]. Essential for reverse proxies upstream! |
on_resolve_error |
Enum | keep_last_good |
Policy on DNS resolution failure. Options: keep_last_good (uses cached IPs), fallback_to_hostname (clears cache, forcing host lookup). |
on_connect_error |
Enum | try_next_ip |
Policy on TCP connection failure. Options: try_next_ip (cycles to the next resolved IP for the same host), rotate_provider_url (instantly fails over to the next URL in the urls list). |
1.1 DNS Resolved IP Persistence
Resolved IPs are persisted to {storage_dir}/provider_dns_resolved.json (not to source.yml).
This file is written atomically after each DNS refresh cycle and read at startup to seed DNS caches before the
background resolver
completes its first cycle.
On config hot-reloads, DNS caches are carried over from previous provider instances so that resolved IPs are available
immediately.
Failover Triggers
Tuliprox automatically switches URLs or DNS IPs on failure. Failover DOES occur on:
- Network Timeouts
- HTTP 5xx errors (500, 502, 503, 504)
- HTTP 404 / 410 / 429
Failover DOES NOT trigger on:
- HTTP 401 / 403 (Authentication errors, to avoid rotating due to a banned account).
2. Inputs (Data Sources) (inputs)
An input represents an upstream provider or a local media library.
inputs:
- name: my_provider
type: xtream
url: provider://my_failover_provider
username: my_user
password: my_password
enabled: true
sequential_group: 1
cache_duration: 1d
persist: playlist_{}.m3u
method: GET
exp_date: "2028-11-30 12:34:12"
headers: { }
options: { }
epg: { }
aliases: [ ]
panel_api: { }
Input Base Parameters
| Parameter | Type | Required | Default | Technical Impact & Background |
|---|---|---|---|---|
name |
String | Yes | Internal reference ID for Tuliprox. Must be strictly unique. Critical for persistent UUID generation! | |
type |
Enum | No | m3u |
Allowed: m3u, xtream, stalker, library, staged, emby, jellyfin, plex, and m3u_batch / xtream_batch / stalker_batch (CSV offloading). Stalker inputs use the portal handshake/catalog flow instead of a plain playlist download. |
url |
String | Yes | The Provider URL. Tuliprox supports magic scheme prefixes: http(s)://, file://, batch://, and provider://my_failover_provider (for the Failover System above). |
|
username / password |
String | Often | Mandatory for xtream and for Stalker inputs using credentials_only or mac_plus_credentials. Stalker inputs using mac_only do not need them. |
|
enabled |
Bool | No | true |
If false, this input is completely ignored in all processing. |
sequential_group |
Int | No | Optional non-zero process-wide group ID. With process_parallel: true, complete refreshes of inputs sharing an ID run one after another, including all Stalker page slices. Different groups and ungrouped inputs may overlap. Not valid for staged inputs; a staged overlay inherits its provider job. |
|
cache_duration |
String | No | 0 |
Crucial: Determines how often Tuliprox actually downloads the raw list from the provider. At 1d (1 day), Tuliprox serves from its local .db for 24 hours, even if you trigger hourly updates. This heavily protects against provider bans! Supported units are s, m, h, and d. If cache_duration is set, the cached provider playlist stored on disk is reused for subsequent updates instead of downloading it again. |
persist |
String | No | Optional path template (e.g., ./playlist_{}.m3u) to permanently store the downloaded raw provider list locally on your disk. The {} in the filename is filled with the current timestamp. For m3u use a full filename. For xtream use a prefix like ./playlist_. |
|
method |
Enum | No | GET |
HTTP Request method for playlist downloads (GET or POST). |
exp_date |
Mixed | No | Expiration date as "YYYY-MM-DD HH:MM:SS" or Unix timestamp. In server mode, Tuliprox refreshes missing or soon-expiring Xtream account dates through the account's player_api.php credentials; see Automatic Xtream Expiration Refresh. |
|
headers |
Dict | No | Custom HTTP headers for the download (e.g., User-Agent: My-Player). |
|
epg |
Object | No | Allows mapping of external XMLTV files (see below). | |
aliases |
List | No | Connection pooling / Sub-accounts (see below). | |
staged |
Object | No | Staged overlay settings. Only valid when type: staged (see below). |
|
panel_api |
Object | No | Automated reseller account generation (see below). |
Minimal Stalker Input Example
inputs:
- name: stalker_main
type: stalker
url: http://portal.example.com/c/
stalker:
auth_mode: mac_only
mag_preset: generic_safe
endpoint_preference: auto
device:
mac_address: '00:1A:79:12:34:56'
enabled: true
options:
stalker_pre_resolve_playback: false
stalker_runtime_resolve_playback: true
Stalker refreshes write pages into an unpublished generation. Live, VOD, series, and EPG selected for one refresh become active together only after the complete selection is durable. Until then, Tuliprox continues serving the previous complete snapshot; a first import exposes no partial catalog. A saved checkpoint resumes after restart.
When process_parallel is enabled, progress messages include the input name. Targets wait for every enabled input in
their source and begin as soon as that source is ready, without waiting for unrelated sources.
Use stalker_pre_resolve_playback: true if you want Tuliprox to materialize playback URLs during refresh whenever the portal already grants them.
Keep stalker_runtime_resolve_playback: true when the portal uses expiring or session-bound temp links that may need a fresh create_link
call later during playback.
Stalker supports four authentication modes:
auto: requires either a MAC address or a complete username/password pair.mac_only: requiresstalker.device.mac_address; username/password are ignored.credentials_only: requires username and password; no MAC address is required.mac_plus_credentials: requires both a MAC address and username/password.
Input URL Schemes (inputs[].url)
Tuliprox utilizes a flexible URI-based system to define where input data originates. Depending on the prefix used, the engine switches between remote downloads, local file access, or internal failover logic.
| Scheme | Target Type | Technical Impact & Background |
|---|---|---|
http(s):// |
Remote Server | Standard method for downloading playlists from provider endpoints. |
file:// |
Local Storage | Reads a playlist directly from the host filesystem. Useful for manual backups or pre-processed files. |
provider:// |
Failover System | Resolves the URL via internal provider definitions. Pro-Tip: Use this to implement automatic rotation or failover between multiple mirrors/gateways of the same provider. New requests honor provider_url_selection_policy, so they can either resume from the last healthy URL or always restart from the first URL. |
batch:// |
CSV File | Dedicated scheme for bulk alias management. Points to a local ; separated CSV file (e.g., batch://./aliases.csv). |
Additional Notes
- Automatic Type Conversion: If the input
typeis set tom3uorxtreambut theurlstarts with thebatch://prefix, Tuliprox automatically upgrades the input tom3u_batchorxtream_batchrespectively. - Batch Constraints: For
m3u_batchandxtream_batch, only local CSV sources are permitted. You must use either thebatch://scheme or a plain absolute/relative filesystem path. - Protocol Restrictions: To ensure stability in batch processing, URI schemes such as
provider://,http(s)://, orfile://are strictly rejected when used within a batch context.
Input Subsections (Object Keys)
| Block | Description | Link |
|---|---|---|
headers |
Custom HTTP request headers for playlist and EPG downloads. | See Headers |
options |
Behavior controls for metadata resolution, stream probing, and skip logic. | See Options |
epg |
XMLTV source management and Smart Match fuzzy logic settings. | See EPG |
aliases |
Connection pooling for multiple subscriptions from the same provider. | See Aliases |
staged |
Overlay settings for first-class staged inputs. | See Staged |
panel_api |
Automated reseller panel integration (provisioning/renewal). | See Panel API |
2.1 Headers (headers)
Allows the injection of custom HTTP headers into outgoing requests for this specific provider.
This is often required for providers that enforce User-Agent whitelisting or specific authorization tokens.
| Parameter | Type | Technical Impact & Background |
|---|---|---|
User-Agent |
String | Mimics a specific player or browser to prevent 403 Forbidden errors. |
Authorization |
String | Manual token injection if required by the upstream API. |
Referer |
String | Can be used to bypass basic hotlink protections. |
headers:
User-Agent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Tuliprox/3.0"
X-Custom-Auth: "my-secret-token"
2.2 Input Options (options)
Controls the behavior during download and asynchronous metadata resolution (see the Metadata Update chapter) for this specific provider.
| Parameter | Type | Default | Technical Impact & Background |
|---|---|---|---|
skip_live / skip_vod / skip_series |
Bool | false |
Immediately ignores entire categories during Xtream or Stalker ingestion. Saves massive amounts of RAM and runtime if you only want specific clusters from a provider. |
xtream_live_stream_without_extension |
Bool | false |
Strips .ts from generated stream URLs. |
xtream_live_stream_use_prefix |
Bool | true |
Injects the /live/ prefix into URLs. |
disable_hls_streaming |
Bool | false |
Rewrites live .m3u8 requests to .ts and bypasses Tuliprox HLS handling. |
resolve_tmdb |
Bool | false |
Enables TMDB queries for this specific input based on parsed titles to fill missing posters and release years. |
probe_stream |
Bool | false |
Uses FFprobe to read A/V details (HDR, 4K). Respects max_connections. |
resolve_background |
Bool | true |
Metadata scans run asynchronously in the background so the general playlist update (which blocks clients) finishes instantly. |
resolve_series / resolve_vod |
Bool | false |
Fetches missing details like Plot or Cast via the Provider's API (get_vod_info / get_series_info). |
probe_series / probe_vod |
Bool | false |
Allows explicit FFprobe analysis of movies or entire TV show seasons. |
probe_live |
Bool | false |
Allows FFprobe to periodically tap into Live-TV streams in the background. |
probe_live_interval_hours |
Int | 120 |
Interval after which a Live stream is re-analyzed (Important as backup streams often change resolutions). |
resolve_delay / probe_delay |
Int | 2 |
Ban Protection: Hard wait time (in seconds) between API or Probe requests to the same provider! Prevents API spamming. |
resolve_filter |
String | - | Filter expression to selectively resolve only entries matching the condition. Uses the same Filter syntax. |
probe_filter |
String | - | Filter expression to selectively probe only entries matching the condition. Uses the same Filter syntax. |
stalker_pre_resolve_playback |
Bool | false |
Stalker-only: resolves create_link during playlist processing and persists the returned playback URL when the portal already grants one. Useful when you want the playlist/export step to materialize playable stream URLs up front. |
stalker_runtime_resolve_playback |
Bool | false |
Stalker-only: lets the reverse-proxy retry create_link during playback when a persisted Stalker URL has gone stale or is rejected by the portal. This is the recovery path for temp links and expired session-bound stream URLs. |
Note: For
resolve_vodandresolve_series, data is cached per input and only new or changed entries are updated.
Minimal Xtream MPEG-TS Example
inputs:
- name: ts-capable-provider
type: xtream
url: http://provider.example
username: user
password: pass
options:
disable_hls_streaming: true
Stalker playback notes
- Stalker playlist preview in the Web UI now works through the same protected playlist endpoints used for other input types.
stalker_pre_resolve_playbackandstalker_runtime_resolve_playbackare complementary:stalker_pre_resolve_playback: truetries to turn portalcmdvalues into concrete playback URLs during refresh.stalker_runtime_resolve_playback: trueretriescreate_linklater if the stored playback URL is stale, temp-link based, or rejected after processing.
- Temp-link variants (
nginx_secure_link,flussonic_tmp_link,wowza_tmp_link) are persisted as explicit playback modes and reused during runtime refresh, instead of being flattened into a generic direct-URL path. - When pre-resolve does not materialize a URL, Tuliprox keeps the Stalker item metadata and playback descriptor but does
not leak the raw
cmdinto the exported playlist URL field. - If pre-resolve is disabled or the portal refuses to resolve a specific item during refresh, the item can still remain playable later through runtime resolution, assuming the reverse-proxy path is used and runtime resolve is enabled.
- Runtime refresh reuses a cached Stalker client per input configuration and treats the session TTL as a soft re-handshake boundary. If refresh still cannot resolve a playable URL, Tuliprox invalidates the stale persisted URL instead of continuing to serve it indefinitely.
- Stalker EPG import now also consumes the portal bulk-EPG endpoint during processing when Stalker playback pre-resolve is enabled.
- The bulk-EPG path is streamed and batch-persisted to reduce peak memory pressure on large portals, but portal-specific tuning for pathological datasets is still a separate follow-up topic.
- Supported Stalker playback transports are currently
httpandhttpsonly.rtmp:///rtsp://commands are rejected explicitly because Tuliprox's reverse-proxy path does not relay those schemes. - Fresh temp-link resolution is implemented. The still-open edge case is whether a specific portal also requires extra forwarded cookies or headers on the final media request after temp-link resolution.
2.3 EPG Assignment & Smart Match (epg)
Tuliprox can load external EPG sources and map them intelligently using advanced fuzzy matching to streams that are missing a valid EPG ID.
The epg.sources list supports the following source types:
- XMLTV sources, which provide complete XMLTV channel and programme data.
- ICS calendar sources, which import iCalendar events and convert them into a virtual XMLTV-compatible EPG channel.
Tuliprox aggregates all configured EPG sources and assigns EPG data based on priority and matching rules.
Example Configuration
epg:
sources:
- url: "auto" # Automatically generated provider XMLTV URL
priority: -2 # High priority
logo_override: true # Replaces provider logos with EPG icons
- url: "http://localhost:3001/xmltv.php?epg_id=1"
priority: -1
- url: "http://localhost:3001/xmltv.php?epg_id=2"
priority: 3
- type: ics
url: "https://files-f1.motorsportcalendars.com/f1-calendar_p1_p2_p3_qualifying_sprint_gp.ics"
channel_id: "f1.calendar"
channel_title: "Formula 1"
priority: -10
ics:
timezone: "Europe/Budapest"
event:
title: "{summary}"
description: "{description}"
include_location: true
include_categories: true
dummy:
enabled: true
title: "No Formula 1 session"
description: "There is currently no scheduled Formula 1 session."
days_past: 1
days_future: 30
block_hours: 4
min_gap_minutes: 1
smart_match:
enabled: true
fuzzy_matching: true
match_threshold: 80
best_match_threshold: 99
name_prefix: { suffix: "." }
name_prefix_separator: [":", "|", "-"]
strip: ["3840p", "uhd", "fhd", "hd", "sd", "4k", "plus", "raw"]
normalize_regex: '[^a-zA-Z0-9\-]'
EPG Source Parameters (sources)
| Parameter | Type | Required | Default | Technical Impact & Background |
|---|---|---|---|---|
type |
Enum | No | xmltv |
Defines the source format. Supported values are xmltv and ics. Existing configurations without type are treated as xmltv. |
url |
String | Yes | The EPG source URL or local path. For XMLTV sources, use auto with Xtream inputs to automatically generate the native XMLTV URL using your credentials. For ICS sources, this points to an .ics calendar file. |
|
priority |
Int | No | 0 |
Determines the lookup order. Lower numbers have higher priority. For example, -2 is processed before 0. Use negative numbers for primary sources. |
logo_override |
Bool | No | false |
If set to true, channel logos from the provider are replaced by icons found in the EPG source. This mainly applies to XMLTV sources. |
channel_id |
String | ICS only | Required for type: ics. Defines the generated XMLTV channel ID for the imported calendar. Playlist entries can reference this value through their EPG ID or receive it through Smart Match. |
|
channel_title |
String | No | Optional display name for the generated ICS EPG channel. If omitted, channel_id is used as fallback. This value is also used as a Smart Match candidate. |
|
match_names |
List | No | [] |
Optional additional names for matching the generated ICS channel to playlist entries. Useful when the playlist channel name differs from the calendar title, for example F1, Formula One, or Formel 1. |
ics |
Object | ICS only | Additional configuration for type: ics. See ICS Calendar Source Parameters. |
XMLTV Sources
XMLTV is the default source type. Existing configurations remain valid and do not need to be changed.
epg:
sources:
- url: "auto"
priority: -2
logo_override: true
- url: "https://example.org/xmltv.xml"
priority: 0
The following configuration is equivalent:
epg:
sources:
- type: xmltv
url: "https://example.org/xmltv.xml"
priority: 0
For Xtream inputs, url: "auto" automatically generates the provider's native XMLTV endpoint from the configured
input URL, username, and password.
The ICS-only fields channel_id, channel_title, match_names, and ics are rejected on type: xmltv sources. This
keeps accidental calendar settings from changing XMLTV download or runtime behavior.
ICS Calendar Sources
ICS sources import iCalendar files and convert their VEVENT entries into XMLTV-compatible programme entries.
Recurring events using RRULE, RDATE, or EXDATE are detected but not expanded yet; Tuliprox imports the base
DTSTART/DTEND occurrence and logs one aggregated warning per parsed source when such entries are present.
An ICS source creates exactly one virtual EPG channel. It does not create a playlist channel or stream. The generated
channel_id must therefore be assigned to an existing playlist entry by one of the following mechanisms:
- the playlist entry already uses the same EPG ID,
- a mapper rule assigns the EPG ID,
- Smart Match matches the playlist channel name against
channel_id,channel_title, ormatch_names.
The generated programmes are written into the regular XMLTV output. M3U and Xtream use the same internal
epg_channel_id assignment, so the same channel_id works for both input types. If an Xtream provider supplies a valid
but unwanted EPG ID, use a mapping rule to overwrite the stream's EPG ID with the ICS channel_id.
Example using the public Formula 1 calendar:
epg:
sources:
- type: ics
url: "https://files-f1.motorsportcalendars.com/f1-calendar_p1_p2_p3_qualifying_sprint_gp.ics"
channel_id: "f1.calendar"
channel_title: "Formula 1"
priority: -10
match_names:
- "F1"
- "Formula One"
- "Formel 1"
ics:
timezone: "Europe/Budapest"
event:
title: "{summary}"
description: "{description}"
include_location: true
include_categories: true
dummy:
enabled: true
title: "No Formula 1 session"
description: "There is currently no scheduled Formula 1 session."
days_past: 1
days_future: 30
block_hours: 4
min_gap_minutes: 1
A playlist channel can then be linked explicitly by using the generated EPG channel ID:
#EXTINF:-1 tvg-id="f1.calendar" tvg-name="Formula 1",Formula 1
http://example.org/live/f1/index.m3u8
Alternatively, Smart Match can match playlist names such as Formula 1, F1, or Formel 1 when the corresponding
values are configured through channel_title or match_names.
ICS Calendar Source Parameters (ics)
| Parameter | Type | Required | Default | Technical Impact & Background |
|---|---|---|---|---|
timezone |
String | No | UTC |
Fallback timezone for floating ICS timestamps and the local dummy block calculation. Use an IANA zone such as Europe/Budapest, UTC, or America/New_York. |
event |
Object | No | Controls how calendar events are mapped to programme title and description. See ICS Event Mapping. | |
dummy |
Object | No | Controls optional gap filling with generated dummy programme entries. See ICS Dummy Gap Filling. | |
include_cancelled |
Bool | No | false |
If set to true, calendar events with STATUS:CANCELLED are imported. By default, cancelled events are skipped. |
max_events |
Int | No | 50000 |
Safety budget for encountered VEVENT blocks, including invalid or skipped events. Parsing stops when the budget is exhausted. Hard cap: 200000. |
max_download_bytes |
Int | No | 10485760 |
Maximum downloaded ICS size in bytes. The default is 10 MiB. Hard cap: 52428800 (50 MiB). |
max_decompressed_bytes |
Int | No | 20971520 |
Maximum decompressed ICS size in bytes. The default is 20 MiB. Hard cap: 104857600 (100 MiB). |
timezone must be a valid IANA timezone known to Tuliprox, for example UTC, Europe/Budapest, or
America/New_York.
ICS Event Mapping (event)
The event block controls how ICS VEVENT properties are converted into EPG programme fields.
ics:
event:
title: "{summary}"
description: "{description}"
include_location: true
include_categories: true
| Parameter | Type | Required | Default | Technical Impact & Background |
|---|---|---|---|---|
title |
String | No | {summary} |
Template used for the generated programme title. |
description |
String | No | {description} |
Template used for the generated programme description. |
include_location |
Bool | No | false |
Appends the ICS LOCATION value to the generated programme description when available. |
include_categories |
Bool | No | false |
Appends ICS CATEGORIES to the generated programme description when available. |
Supported template variables:
| Variable | Description |
|---|---|
{summary} |
The ICS SUMMARY value. Usually the event title. |
{description} |
The ICS DESCRIPTION value. |
{location} |
The ICS LOCATION value. |
{categories} |
The ICS CATEGORIES value. |
{uid} |
The ICS UID value. |
{start} |
The localized event start time. |
{end} |
The localized event end time. |
Example:
ics:
event:
title: "Formula 1: {summary}"
description: "{description}"
include_location: true
include_categories: true
ICS Time Handling
Tuliprox converts all imported calendar events into the internal EPG time format.
Supported ICS time forms:
| ICS Time Form | Behavior |
|---|---|
DTSTART:20260306T123000Z |
Treated as UTC. |
DTSTART;TZID=Europe/Berlin:20260306T1230 |
Interpreted in the specified TZID timezone and converted internally. |
DTSTART:20260306T123000 |
Treated as a floating timestamp and interpreted using ics.timezone. |
DTSTART;VALUE=DATE:20260306 |
All-day events. These are ignored. |
For regular programme entries, an ICS event must provide a valid start and end time. DTEND is preferred. If DTEND
is missing, Tuliprox may use DURATION when present. Events without a usable end time are skipped. For template
variables, {start} and {end} are formatted consistently; when DURATION supplies the end time, {end} uses the
same display timezone as DTSTART.
ICS Dummy Gap Filling (dummy)
ICS calendars often only contain real events, for example race sessions, meetings, or special broadcasts. This may leave large gaps in the EPG. The optional dummy gap filler can create placeholder programmes for these gaps.
Dummy entries are generated in local day blocks. By default, the block size is 4 hours:
00:00 - 04:00
04:00 - 08:00
08:00 - 12:00
12:00 - 16:00
16:00 - 20:00
20:00 - 24:00
Example:
ics:
timezone: "UTC"
dummy:
enabled: true
title: "No Formula 1 session"
description: "There is currently no scheduled Formula 1 session."
days_past: 1
days_future: 30
block_hours: 4
min_gap_minutes: 1
| Parameter | Type | Required | Default | Technical Impact & Background |
|---|---|---|---|---|
enabled |
Bool | No | false |
Enables generation of dummy programme entries for gaps without real events. |
title |
String | No | No programme entry |
Programme title for generated dummy entries. |
description |
String | No | empty | Programme description for generated dummy entries. |
days_past |
Int | No | 1 |
Number of days before the current day for which dummy entries are generated. |
days_future |
Int | No | 14 |
Number of days after the current day for which dummy entries are generated. |
block_hours |
Int | No | 4 |
Size of the local dummy blocks in hours. The value must divide 24 evenly. For example: 1, 2, 3, 4, 6, 8, or 12. |
min_gap_minutes |
Int | No | 1 |
Minimum gap length required to generate a dummy entry. Smaller gaps are ignored to avoid tiny placeholder programmes caused by second-level timestamp differences. |
Dummy entries never replace real programme entries. They only fill uncovered time ranges.
days_past is capped at 30; days_future is capped at 366. Dummy titles use the same length cap as event summaries
and dummy descriptions use the same length cap as event descriptions.
Example with one real event from 03:30 to 05:00:
00:00 - 03:30 No Formula 1 session
03:30 - 05:00 Real calendar event
05:00 - 08:00 No Formula 1 session
08:00 - 12:00 No Formula 1 session
...
Example with an event aligned to a block boundary:
00:00 - 04:00 No Formula 1 session
04:00 - 06:00 Real calendar event
06:00 - 08:00 No Formula 1 session
08:00 - 12:00 No Formula 1 session
...
If two events are adjacent, no dummy programme is inserted between them:
10:00 - 11:00 Real calendar event
11:00 - 12:00 Real calendar event
12:00 - 16:00 No Formula 1 session
Dummy block boundaries are calculated in ics.timezone. Around daylight saving time changes, the local block labels
remain stable, for example 00:00 - 04:00, 04:00 - 08:00, and so on, while the corresponding UTC duration may differ.
If a local boundary does not exist during a spring-forward transition, it maps to the first real instant after the gap.
If a boundary is ambiguous during a fall-back transition, the earlier occurrence is used. Any resulting zero-length or
negative UTC interval is omitted, so generated dummy programmes remain ordered, non-overlapping, and continuous over
the real local day.
ICS Download and Security Notes
ICS sources use the same download infrastructure as playlist and XMLTV EPG downloads. This means that input-level headers, disabled headers, default user-agent handling, cache handling, retry behavior, provider failover, and sanitized error logging are applied consistently.
Supported URL forms for ICS sources:
| URL Form | Description |
|---|---|
https://... |
Recommended for remote calendar files. |
http://... |
Supported when explicitly needed. |
webcal://... |
Normalized internally to https://.... |
provider://... |
Uses a configured Tuliprox provider failover definition. |
file://... |
Reads a local calendar file from the host filesystem. |
| local path | Reads a local relative or absolute file path, subject to path validation. |
Unsupported or unsafe schemes such as data:, ftp:, gopher:, javascript:, mailto:, or ssh: are rejected.
Tuliprox does not load external resources referenced inside the ICS file. Fields such as ATTACH, URL, ORGANIZER,
ATTENDEE, or IMAGE are treated as metadata only and must not trigger additional network requests.
To protect the server, ICS imports are bounded by safety limits such as maximum download size and maximum event count. Sensitive request data, such as credentials and authorization headers, is sanitized in logs and error messages.
Additional parser limits are enforced regardless of the configured values:
| Limit | Hard Cap |
|---|---|
| Physical or unfolded ICS line length | 128 KiB |
Properties per VEVENT |
256 |
Imported SUMMARY text |
4 KiB, truncated at a UTF-8 boundary |
Imported DESCRIPTION text |
64 KiB, truncated at a UTF-8 boundary |
If the file itself is unreadable, too large, not valid UTF-8, violates the line-length limit, or does not contain one
complete, correctly nested VCALENDAR envelope, the source fails. A correctly wrapped calendar without events is
valid. If a single VEVENT inside that envelope is malformed, has too many properties, has an unknown event TZID, or
lacks a usable
DTSTART/DTEND/DURATION, that event is skipped and the remaining events continue to be imported. The Web UI EPG
preview uses the same merged output behavior as the client XMLTV output, including ICS dummy gap filling. If a cached
.ics file cannot be parsed in preview, Tuliprox ignores the cached file and downloads the source again through the
normal EPG download path.
The graphical source editor currently preserves existing ICS fields when editing an input, but full creation and editing
of every ICS-specific field is YAML-first. Configure new ICS sources in source.yml.
Smart Match Parameters (smart_match)
The fuzzy matching logic attempts to "guess" the EPG ID by generating search keys based on the channel name.
For XMLTV sources, Tuliprox uses the channel IDs and display names from the XMLTV file.
For ICS sources, Tuliprox uses the generated virtual channel metadata:
channel_idchannel_titlematch_names
| Parameter | Type | Default | Technical Impact |
|---|---|---|---|
enabled |
Bool | false |
Activates the Smart Match engine for streams without a fixed tvg-id. |
fuzzy_matching |
Bool | false |
Fallback to phonetic and Jaro-Winkler similarity matching if exact ID match fails. |
match_threshold |
Int | 80 |
Minimum similarity score (10-100) required to accept a fuzzy match. |
best_match_threshold |
Int | 95 |
Minimum score for the strict fallback used when phonetic keys differ. |
name_prefix |
Object | ignore |
Options: ignore, suffix, prefix. For suffix/prefix, a concat string (e.g., { suffix: "." }) is required. |
name_prefix_separator |
List | [':', '|', '-'] |
Characters used by providers to delimit country codes (e.g., US:, FR|). |
strip |
List | (quality tags) | Resolution, codec and frame-rate markers stripped as complete terms before matching. |
normalize_regex |
String | [^a-zA-Z0-9._\-] |
Default pattern preserving the separators commonly found in XMLTV channel IDs. |
When upgrading, an explicitly configured legacy pattern such as [^a-zA-Z0-9\-] remains unchanged and continues to
remove periods and underscores. Remove that override or set [^a-zA-Z0-9._\-] to adopt the new default behavior.
How Smart-Matching works
If a stream is missing the tvg-id, Tuliprox performs the following steps:
- Normalization: The channel name (e.g.,
US: HBO HD 4K) is processed. - Prefix Extraction: Using
name_prefix_separator, Tuliprox identifies:and splits the name. It recognizesUSas the country prefix. - Cleaning: It strips terms defined in
strip("4K", "HD") and applies thenormalize_regex. The core name becomeshbo. - Reconstruction: Using
name_prefix.suffix(.), the country code is appended to the name. The target search key becomeshbo.us. - Exact Matching: Normalized XMLTV IDs and display names are checked first. Source priority resolves duplicate exact candidates. Quality variants sharing one normalized name can reuse a populated direct ID.
- Fuzzy Matching: When enabled, the engine scores every candidate in the matching Double Metaphone bucket and
keeps the globally best result rather than the first acceptable result. If the phonetic lookup yields nothing, a
same-initial fallback is allowed only at
best_match_threshold. - Safety Checks: Numeric signatures must agree (
TF1cannot matchTF1+1), tied candidates are rejected, and low-confidence candidates must beat the runner-up by a minimum margin. Explicit XMLTV ID country suffixes prevent cross-country matches, and decorative playlist separators are ignored. - Programme Validation: A channel declaration without programmes is not treated as a successful guide match. Tuliprox keeps ranked alternatives while parsing and selects the best candidate that actually contributes programme data. Existing IDs with no programmes can therefore be replaced by a populated, semantically compatible candidate. ICS channels with an enabled dummy policy also count as populated.
At debug level, each processed input emits one Smart EPG summary with the number of live channels whose original ID
was valid, whose ID was assigned by Smart Match, or which remained unresolved.
For an ICS source, the generated EPG channel participates in the same matching process. For example, this configuration:
epg:
sources:
- type: ics
url: "https://files-f1.motorsportcalendars.com/f1-calendar_p1_p2_p3_qualifying_sprint_gp.ics"
channel_id: "f1.calendar"
channel_title: "Formula 1"
match_names:
- "F1"
- "Formel 1"
can match playlist channel names such as:
Formula 1
F1
Formel 1
Note: Lower
match_thresholdvalues increase the chance of EPG assignment but may lead to incorrect matches for channels with very similar names.
2.4 Provider Aliases (aliases & batch://)
Tuliprox allows you to pool multiple subscriptions from the same provider into a single logical source. By merging these "aliases," Tuliprox tracks connection availability across all accounts, ensuring that if one subscription is at its limit, the next available connection from the pool is used.
Defining Aliases in YAML
Aliases are ideal for a small number of fixed accounts. Note that in YAML, max_connections: 0 signifies "unlimited,"
which is the default setting.
inputs:
- type: xtream
name: my_provider # Mandatory: Used for stable UUID generation
url: 'http://provider.net'
username: sub_1
password: pw1
max_connections: 1
aliases:
- name: my_provider_2
url: 'http://provider.net'
username: sub_2
password: pw2
priority: 1
max_connections: 2
exp_date: "2028-11-30 12:00:00"
enabled: true
Result: Tuliprox treats this as a single provider source with a total pool of 1 + 2 = 3 concurrent connections.
YAML Alias Fields
Alias entries use the same effective connection attributes as a normal input account. The input-level headers,
epg, options, persist, method, panel_api, cache_duration, and provider failover settings remain inherited
from the parent input; the alias fields below override the concrete account/connection identity.
| Parameter | Type | Required | Default | Technical Impact & Details |
|---|---|---|---|---|
id |
Int | No | Internal/generated alias ID. Normally omit this; Tuliprox assigns IDs from the input/alias order during config preparation. | |
name |
String | Yes | Unique alias name. Used for stable playlist UUID generation and consistent channel numbering across updates. | |
url |
String | Yes | Provider base URL, playlist URL, or provider://<name> reference. For M3U aliases, credentials can also be extracted from the URL query parameters. |
|
username |
String | Xtream | Account username. Mandatory for regular Xtream YAML aliases. Optional for M3U aliases when credentials are embedded in the playlist URL. | |
password |
String | Xtream | Account password. Mandatory for regular Xtream YAML aliases. Optional for M3U aliases when credentials are embedded in the playlist URL. | |
priority |
Int | No | 0 |
Connection selection priority for this alias. Lower numbers have higher priority; negative numbers are allowed. |
max_connections |
Int | No | 0 |
Allowed concurrent streams for this alias. In YAML, 0 means unlimited/no explicit limit. |
exp_date |
Mixed | No | Account expiration. Supports "YYYY-MM-DD HH:MM:SS" interpreted as UTC or Unix timestamps in seconds. Xtream dates participate in the automatic refresh described below. |
|
enabled |
Bool | No | true |
If false, this alias is ignored when Tuliprox builds the usable connection pool. |
Automatic Xtream Expiration Refresh
In server mode, Tuliprox runs an autonomous task that refreshes expiration dates for enabled Xtream inputs and aliases.
It calls the account's standard player_api.php endpoint with the configured URL, username, and password. This does not
require a reseller Panel API key and continues to work when panel_api provisioning is absent or disabled.
To limit provider traffic and avoid bans, the task uses these fixed rules:
- Accounts without an expiration date, or whose expiration is at most three days away, are eligible for refresh.
- An eligible account is queried at most once every 24 hours.
- Requests to accounts belonging to the same configured panel, including aliases, are spaced at least five minutes apart.
- Transport errors, HTTP 403/429 responses, and server errors put the complete panel into a six-hour cooldown.
Successful non-expired updates are collected for up to 15 minutes and then persisted as a batch. An expiration date at
or before the current time is persisted immediately and the account is set to enabled: false. Disabled accounts are
not queried and are not re-enabled automatically.
Tuliprox updates the main source YAML and local xtream_batch alias CSV files in place. Before changing either file it
creates a timestamped copy in backup_dir; CSV comments, column order, unknown columns, and environment placeholders
are preserved. The in-memory source configuration is refreshed once per persisted batch, without triggering a second
reload for Tuliprox's own file write.
Throttle and pending-batch state is stored in {storage_dir}/xtream_expiry_state.json, so restarts do not reset request
limits or discard already fetched expiration dates. The intervals above are currently fixed and have no configuration
keys.
Batch CSV Offloading (batch://)
For managing dozens or hundreds of accounts, Tuliprox supports offloading alias definitions to local CSV files using the
batch:// scheme.
| Scheme | Description |
|---|---|
batch://./file.csv |
Relative path to the CSV file. |
batch:///path/file.csv |
Absolute path to the CSV file. |
Note: Batch inputs only support local filesystem paths. Schemes like
http(s)://,file://, orprovider://are rejected for batch URL definitions. If an inputurlstarts withbatch://, Tuliprox automatically sets the corresponding batch type forxtream,m3u, orstalkerinputs.
Batch CSV Formats
Batch files use a semicolon (;) as a separator. Unlike standard YAML config, the default for max_connections in CSV
files is 1.
XtreamBatch
Used for Xtream Codes API accounts.
inputs:
- type: xtream_batch
name: my_provider
url: 'batch://./xtream_aliases.csv'
CSV Structure:
#name;username;password;url;max_connections;priority;exp_date;enabled
my_provider_1;user1;password1;http://p1.com:80;1;0;2028-11-23 12:34:23;true
my_provider_2;user2;password2;http://p2.com:8080;1;1;1732365263;true
M3uBatch
Used for plain M3U playlist URLs.
inputs:
- type: m3u_batch
name: m3u_pool
url: 'batch:///etc/tuliprox/m3u_aliases.csv'
CSV Structure:
#url;max_connections;priority;enabled
http://p1.com/get.php?username=u1&password=p1;1;0;true
http://p2.com/get.php?username=u2&password=p2;1;5;true
StalkerBatch
Used for Stalker/Ministra portals. The complete portal path is retained, so URLs such as /c/ must be included in
the CSV. The following credentials-based example uses config/stalker_aliases.csv:
inputs:
- type: stalker_batch
name: stalker_pool
url: 'batch://./config/stalker_aliases.csv'
stalker:
catalog_max_pages: 1000
CSV Structure:
#name;url;username;password;mac_address;auth_mode;mag_preset;endpoint_preference;max_connections;priority;exp_date;enabled
portal_primary;http://portal.example/c/;account1;secret1;00:1A:79:12:34:56;mac_plus_credentials;mag254_strict;portal;1;0;;true
portal_backup;https://backup.example/stalker_portal/c/;account2;secret2;;credentials_only;generic_safe;auto;1;10;;true
Columns are selected by the header and may appear in any order. mac_address, auth_mode, mag_preset, and
endpoint_preference are optional per-alias Stalker fields. An alias with no Stalker-specific values inherits the
complete parent configuration; otherwise empty enum fields use their normal defaults. Other device fields, size caps,
and the catalog page limit are configured on the parent input in the Web UI or YAML and inherited by every CSV alias.
Field Specifications
| Parameter | Technical Impact & Details |
|---|---|
url |
Provider base URL or full M3U playlist URL. Required in CSV rows. |
name |
Crucial: The first alias is automatically renamed with the name from the input definition (e.g., my_provider_1 gets my_provider). This is necessary for stable playlist UUID generation and consistent channel numbering across updates. |
username |
Xtream or Stalker account username. For M3U CSV rows, Tuliprox can also extract credentials from the URL query parameters. |
password |
Xtream or Stalker account password. For M3U CSV rows, Tuliprox can also extract credentials from the URL query parameters. |
mac_address |
Stalker alias MAC address. Required by mac_only and mac_plus_credentials; optional for credentials-only authentication. |
auth_mode |
Stalker alias authentication mode: auto, mac_only, credentials_only, or mac_plus_credentials. |
mag_preset |
Stalker alias MAG profile: generic_safe, mag250_legacy, mag254_strict, or ministra_modern. |
endpoint_preference |
Stalker portal endpoint selection: auto, server_load, or portal. |
max_connections |
Defines allowed concurrent streams. Default in CSV is 1. |
priority |
Lower numbers = higher priority. 0 is higher than 1. Negative numbers (e.g., -1) are allowed for top-tier priority. Items with the lowest values are processed first. |
exp_date |
Account expiration. Supports "YYYY-MM-DD HH:MM:SS" interpreted as UTC or Unix timestamps in seconds. Used for auto-cleanup or Panel API sync. |
enabled |
Enables/disables a CSV alias row. Empty values, 1, t, and true are treated as enabled; 0, f, or false disable the alias. |
2.5 Staged Sources (staged)
The staged input is a first-class input type for pre-formatted playlists. Tuliprox reads the selected playlist clusters from the staged source, then stores the merged result in the linked provider input. Stream delivery and API requests continue to use that provider input.
This is useful when an external playlist editor already has the desired channel order, groups, and original stream IDs. For example, an IPTV editor can provide the Live playlist layout while the actual streams are still opened against the Xtream or M3U provider.
Data flow:
staged input -> provider input: the staged input is an overlay for that provider. Clusters listed instaged.clustersare loaded from the staged input; the remaining clusters are loaded from the provider input itself. The merged playlist is persisted under the provider input, and streaming/API requests still target the provider input.staged input -> targetis not supported. Use a normalm3uorxtreaminput if the source should be connected directly to a target.
Configuration Example (Provider With Staged Live Overlay)
In this setup, Live-TV comes from the external staged playlist, while VOD and Series come from the original Xtream provider.
inputs:
- name: provider_main
type: xtream
url: http://provider-a.example:8080
username: main_user
password: main_pass
- name: provider_main_editor_live
type: staged
url: http://editor.example/provider-main-live.m3u
staged:
for_input: provider_main
clusters: [live]
Parameters
| Parameter | Type | Required | Default | Technical Impact & Background |
|---|---|---|---|---|
type |
Enum | Yes | Must be staged. |
|
staged_type |
Enum | No | m3u |
Format of the staged source. Allowed: m3u, xtream. |
url |
String | Yes | Download URL (HTTP/HTTPS) or local file path. For staged_type: xtream, use the base hostname:port. |
|
username / password |
String | Yes | Mandatory only if staged_type: xtream. Not inherited from the provider input. |
|
method |
Enum | No | GET |
HTTP request method (GET or POST). Not inherited from the provider input. |
headers |
Dict | No | Custom HTTP headers for the staged download. Not inherited from the provider input. | |
staged.for_input |
String | Yes | Provider input name. Must reference a non-staged m3u or xtream input. |
|
staged.clusters |
List | No | all | Clusters loaded from the staged input: live, vod, series. |
Staged Cluster Behavior & Validation
staged.clusters is the group of clusters loaded from the staged input.
- The referenced provider supplies all clusters not listed in
staged.clusters. staged.for_inputmust reference an existing non-stagedm3uorxtreaminput.- Each provider input can have at most one staged overlay.
staged.clustersmust not be empty.- Source definitions must reference the provider input, not the staged input.
- Staged inputs do not use
priority,max_connections, orcache_duration. The linked provider input controls stream limits and refresh cadence. If the provider input is still cached, Tuliprox does not query the staged input.
File Persistence (persist)
When using persist to save the staged data, follow these filename conventions:
- For
m3u: Use a full filename template like./staged_playlist_{}.m3u. - For
xtream: Use a prefix template like./staged_playlist_.
2.6 Provider Panel API (panel_api)
Tuliprox can optionally interface with a provider's reseller panel API to automate account lifecycle management. This allows the system to fetch credit balances, sync expiration dates, and automatically provision or renew alias accounts based on demand.
Important: Panel API accounts are managed as individual connections/aliases. Tuliprox does not assume unlimited provider access; each alias consumes a slot or credit according to your provider's rules.
panel_api:
url: '[https://panel.provider.com/api.php](https://panel.provider.com/api.php)'
api_key: 'YOUR_ADMIN_KEY'
credits: "0.0" # Persisted credit balance, updated via account_info
provisioning:
timeout_sec: 65
method: GET # Probe method (HEAD, GET, or POST)
probe_interval_sec: 10
cooldown_sec: 120 # Wait time after successful probe for DB finalization
offset: 12h # Pre-expiry window (e.g., 15m, 5h, 1d)
alias_pool:
size: { min: auto, max: auto }
remove_expired: true
query_parameter:
account_info: # Executed on boot/update to fetch credits
- { key: action, value: account_info }
- { key: api_key, value: auto }
client_info: # Mandatory for syncing exp_date
- { key: action, value: client_info }
- { key: username, value: auto }
- { key: password, value: auto }
- { key: api_key, value: auto }
client_new: # Create new account (type: m3u only)
- { key: action, value: new }
- { key: type, value: m3u }
- { key: sub, value: '1' }
- { key: api_key, value: auto }
client_renew: # Renew existing account (type: m3u only)
- { key: action, value: renew }
- { key: type, value: m3u }
- { key: username, value: auto }
- { key: password, value: auto }
- { key: sub, value: '1' }
- { key: api_key, value: auto }
client_adult_content: # Optional: Unlock adult content after new/renew
- { key: action, value: adult_content }
- { key: username, value: auto }
- { key: password, value: auto }
- { key: api_key, value: auto }
Configuration Parameters
| Block / Parameter | Type | Default | Technical Impact & Background |
|---|---|---|---|
url |
String | The base endpoint for the provider's reseller API. | |
api_key |
String | Your reseller administrative key. | |
alias_pool |
Object | Controls the lifecycle of active aliases. | |
↳ size.min |
Mixed | 1 |
Min accounts to keep. number or auto. If auto, it uses the count of enabled Tuliprox users (Active/Trial, not expired) mapped to this input's targets. |
↳ size.max |
Mixed | 1 |
Upper bound for aliases. If auto, checks are triggered upon user add/update. |
↳ remove_expired |
Bool | false |
If true, removes expired accounts from source.yml or batch CSVs during boot/update. (The root input is never removed). |
provisioning |
Object | Verification and renewal logic. | |
↳ offset |
String | None |
Pre-expiry window. If now + offset > exp_date, Tuliprox fires client_renew (falls back to client_new). |
↳ timeout_sec |
Int | 65 |
Max wait time for probing a new account before continuing boot/update. |
↳ method |
Enum | HEAD |
HTTP method for probes (HEAD, GET, POST). |
↳ cooldown_sec |
Int | 0 |
Extra wait time after a successful probe to mitigate 5XX errors during provider provisioning. |
Runtime Logic & Dynamic Values (auto)
The keyword auto acts as a placeholder for Tuliprox to inject runtime values dynamically into query parameters:
api_key: auto: Replaced bypanel_api.api_key.username / password: auto: Replaced by the specific credentials of the account being queried, renewed, or probed.
Response Evaluation & Fallback Logic
Tuliprox processes all Panel API responses as JSON and strictly requires status: true.
account_info: Extracts thecreditsfield and persists it. Uses root input credentials ifautois specified.client_info: Syncs theexpirefield, normalizing the timestamp/date to UTC.client_new: Attempts to extractusernameandpassworddirectly.- Fallback: If fields are missing, Tuliprox parses a
urlfield within the JSON response to extract credentials from the query string. - Failure to derive credentials results in a failed operation and no alias persistence.
- Fallback: If fields are missing, Tuliprox parses a
client_renew: Updates the expiration date without modifying existing credentials.client_adult_content: Optionally executed afterclient_neworclient_renewto toggle adult content settings on the provider side. Requiresstatus: truefor success.
3. Routing & Targets (sources)
This block links your inputs to one or more output targets and defines how Tuliprox transforms, filters, sorts, and
exports the resulting playlist.
Under sources:, you connect inputs from the inputs section of source.yml with one or more targets.
sources:
- inputs:
- my_provider
targets:
- name: my_target
filter: 'Group ~ ".*"'
output:
- type: m3u
3.1 inputs
inputs is a list of input names referencing entries defined in the inputs section of source.yml.
sources:
- inputs:
- my_input_a
- my_input_b
Note: The
inputslist only references previously defined input names. It does not define input behavior itself.
3.2 targets
A target defines the final transformed playlist that clients consume.
Tuliprox supports multiple targets per source, and each target can export to multiple output formats simultaneously.
sources:
- inputs:
- my_provider
targets:
- name: my_target
enabled: true
processing_order: frm
filter: 'Group ~ ".*"'
rename: [ ]
mapping: [ ]
sort: { }
options:
ignore_logo: false
epg_output:
lowercase_ids: false
lowercase_xmltv_display_names: false
share_live_streams:
hls: false
mpeg_ts: false
remove_duplicates: false
output:
- type: xtream
favourites: [ ]
watch: [ ]
use_memory_cache: false
targets:
- name: my_target
processing_order: rmf
filter: 'Group ~ "Sports.*"'
rename:
- field: group
pattern: '^UK '
new_name: ''
mapping:
- sports_map
output:
- type: m3u
Target Parameters
| Parameter | Type | Required | Default | Technical Impact & Background |
|---|---|---|---|---|
enabled |
Bool | No | true |
If set to false, Tuliprox skips building this target during normal processing. This reduces CPU, disk, and upstream workload, but the target can still be selected explicitly via CLI target execution if matched by -t. |
name |
String | No | default |
Logical target name. If not default, it must be unique. Unique names are important for selective execution (-t <target_name>) and for clearly separating output identities in Tuliprox's processing pipeline. |
processing_order |
Enum | No | frm |
Defines execution order for Filter, Rename, and Map. This directly changes which intermediate state downstream steps operate on and can therefore materially alter the final playlist result. |
filter |
String | Yes | Global filter DSL expression for the target. This determines which entries survive into the final target after the selected processing order has been applied. | |
rename |
List | No | Regex-based transformations applied to selected fields. This is commonly used to normalize channel/group labels before sorting, mapping, or export. | |
mapping |
List | No | References mapping IDs from mapping.yml for advanced transformation logic. This is where deep structural rewriting and metadata normalization can be applied. |
|
sort |
Object | No | Defines ordering for groups and channels after transformations. This affects the final playlist structure seen by clients and can significantly improve navigation quality in IPTV players. | |
options |
Object | No | Target-level behavior switches such as logo suppression, duplicate removal, and shared live-stream handling. These options influence memory usage, playlist cleanliness, and reverse-proxy behavior. | |
output |
List | Yes | Mandatory list of output formats. A single target can generate multiple output representations (e.g., xtream, m3u, strm, hdhomerun) from the same transformed result set. |
|
favourites |
List | No | Duplicates final transformed channels into dedicated favorite groups after processing is complete. This adds curated views without changing the original group structure. | |
watch |
List | No | Defines watched group patterns. If matching groups change during updates, Tuliprox emits Messaging events so operational changes become observable automatically. | |
use_memory_cache |
Bool | No | false |
If enabled, the final compiled playlist is cached in RAM. This reduces disk access and improves delivery speed, especially for M3U downloads, but increases memory consumption. |
3.2.1 processing_order
The processing order defines how Tuliprox applies:
- Filter
- Rename
- Map
Valid values are:
frm(default)fmrrfmrmfmfrmrf
Note: The selected processing order can change the final result significantly. For example, if renaming occurs before filtering, the filter must match the renamed state rather than the original source value.
processing_order only arranges the processing-stage mapping blocks around filter and rename. Mapping blocks that
opt into stage: after_epg always run once EPG enrichment has completed, regardless of processing_order.
3.2.2 filter
The target-level filter is a string-based expression using Tuliprox's filter DSL.
It defines which entries remain in the final target after the selected processing stages have been applied.
You can define complex strings or regex patterns exactly once in template.yml
and call them by wrapping the template name in exclamation marks: !MACRO_NAME!.
For less verbose expression definitions, inline filter definitions are also supported.
Tuliprox supports the following filter expression types:
- Use
NOTfor exclusion logic - Use
AND/ORfor boolean combinations - Type Comparison:
Type = vodorType = liveorType = series - Regular expression comparison:
([fieldname]) ~ "regexp"
The[fieldname]can beGroup,Title,Name,Caption,Url,Genre,Input,EpgIdorType. - String comparison (case-insensitive, no regex needed):
- Exact:
Group = "Sports"/ negated:Group != "Sports" - Substring:
Title CONTAINS "HD" - Prefix:
Caption STARTSWITH "DE:" - Case-insensitivity is ASCII-only: ASCII letters match regardless of case, non-ASCII characters must match
exactly.
Title CONTAINS "cinéma"matchesCinémabut notCINÉMA.
- Exact:
- Set membership (case-insensitive exact match against a list):
Group IN ["Sports", "News"] - Numeric comparison on the channel number:
Chno = 5,Chno != 5,Chno > 100,Chno >= 100,Chno < 200,Chno <= 200 - Numeric comparison on the detected quality tier:
Quality >= 3
The tier is derived from quality tokens in the caption:5= 4K/UHD/2160p,4= QHD/1440p,3= FHD/1080p,2= HD/720p,1= SD/480p/576p,0= no recognized quality token. - Filters don't have operator precedence, so please use parentheses
- You can apply Morgan’s Law
NOT (A) AND NOT (B)is the same asNOT( A OR B)
Note:
- If you use special characters like
+ | [ ] ( )inside the filter expression you must escape them correctly with backslashes.- When testing expressions externally, e.g. regex101.com, select the Rust flavor. > This helps avoid mismatches between development-time testing and Tuliprox runtime behavior.
⚠️ Warning: Filter expressions are evaluated using Rust-style regex behavior. Unsupported features such as lookarounds and backreferences are not available, so patterns copied from PCRE-based tools may need adjustment.
Example Filter
targets:
- name: regional_mix
filter: '((Group ~ "^DE.*") AND (NOT Title ~ ".*Shopping.*")) OR (Group ~ "^AU.*")'
output:
- type: m3u
This example keeps:
- entries from groups starting with
DE, except titles containingShopping - all entries from groups starting with
AU
3.2.3 rename
The rename block is a list of rename rules applied to selected fields.
Each rule performs regex-based search and replace using capture groups where needed.
Rename Parameters
| Parameter | Type | Required | Default | Technical Impact & Background |
|---|---|---|---|---|
field |
Enum | Yes | Field to transform, can be group, title, name, caption or url. This determines which part of the playlist entry Tuliprox rewrites before later stages such as sorting or final export. |
|
pattern |
String (Regex) | Yes | Regular expression used to match the current value of the selected field. This enables structural normalization of inconsistent source naming schemes. | |
new_name |
String | Yes | Replacement string. It can reference regex capture groups via $1, $2, and so on. This allows Tuliprox to preserve selected original content while reformatting labels. |
Rename Example
Example:
rename:
- field: group
pattern: '^DE(.*)'
new_name: '1. DE$1'
In above example, every group beginning with DE is renamed to start with 1., for example:
DE Sports→1. DE SportsDE Movies→1. DE Movies
This can be useful for players that ignore provider order and perform their own alphabetical sorting.
Note: The effective value that
renamesees depends onprocessing_order. If mapping runs before renaming, your rename pattern must match the already mapped value rather than the original source value.
3.2.4 mapping
The mapping block references a list of mapping identifiers (IDs) defined in your mapping files (
default: mapping.yml).
mapping:
- map_cleanup
- map_regional_groups
- map_vod_enrichment
Mapping Parameters
| Parameter | Type | Required | Default | Technical Impact & Background |
|---|---|---|---|---|
mapping |
List of Strings | No | Ordered list of mapping IDs to apply. Each referenced mapping can perform deep transformations on the playlist structure, metadata, grouping, or labels, making this one of the most powerful target-level processing stages in Tuliprox. |
To define a new mapping IDs see details in chapter Mapper DSL & Logic.
3.2.5 sort
The sort block defines ordering rules for groups and channels.
It has the following top-level attributes:
match_as_asciioptional, defaultfalserules
Sort Parameters
| Parameter | Type | Required | Default | Technical Impact & Background |
|---|---|---|---|---|
match_as_ascii |
Bool | No | false |
If enabled, Tuliprox normalizes accented characters during sorting comparisons. This improves deterministic ordering across multilingual playlists without modifying the original visible channel names. |
rules |
List | Yes | Ordered list of sort rules. Each rule is evaluated against the playlist after transformation, and directly shapes the browsing order clients see in the final target. |
rules
Each sort rule supports the following entries:
| Parameter | Type | Required | Default | Technical Impact & Background |
|---|---|---|---|---|
target |
Enum | Yes | Defines whether the rule sorts group or channel entries. This changes whether Tuliprox reorders category containers or items within those categories. |
|
field |
String | Yes | Sort field. For channel: title, name, caption, url, or quality (detected quality tier, best used with order: desc). For group: group. This determines which final-state value Tuliprox uses for ordering. |
|
filter |
String | Yes | Filter expression defining which entries the rule applies to. This makes it possible to sort only selected subsets of the playlist instead of the entire target uniformly. | |
order |
Enum | Yes | asc, desc, or none. none preserves source order for matched entries and is useful when provider order should remain untouched. |
|
natural |
Bool | No | false |
Natural sort: numbers embedded in values compare numerically instead of lexicographically, so Channel 2 sorts before Channel 10. Applies to the rule's value and sequence capture comparisons. |
sequence |
List | No | Ordered regex list used for index-based sorting. When present, Tuliprox prioritizes regex sequence position over order, enabling explicit semantic ordering such as quality tiers or curated group precedence. |
Note: Sort rules must be written with the configured
processing_orderin mind, because sorting operates on the transformed state that exists at that point in the pipeline.Multi-field sorting: rules are applied in the order they are declared. When a rule compares equal, the next rule decides — so a
channelrule ongroupfollowed by one oncaptionproduces group-then-caption ordering.
Sort Example
sort:
match_as_ascii: false
rules:
- target: group
order: asc
filter: 'Group ~ ".*"'
field: group
sequence:
- '^Freetv'
- '^Shopping'
- '^Entertainment'
- '^Sunrise'
- target: channel
order: asc
filter: 'Group ~ ".*"'
field: title
sequence:
- '(?P<c1>.*?)\bUHD\b'
- '(?P<c1>.*?)\bFHD\b'
- '(?P<c1>.*?)\bHD\b'
- '(?P<c1>.*?)\bSD\b'
Named Capture Groups in sequence
To sort by specific parts of a value, use named capture groups such as:
c1c2c3
Note:
- The numeric suffix defines priority. c1 > c2 > c3
This allows Tuliprox to perform structured multi-level sorting based on extracted fragments of a channel title or label.
In the example above:
- groups are ordered according to the explicit
sequence - Channels within the
Freetvgroup are first sorted byquality(as matched by the regexp sequence), and then by thecaptured prefix.
3.2.6 options
Target-level options control behavior of the final playlist independent of output type.
targets:
- name: xc_m3u
output:
- type: xtream
skip_live_direct_source: true
skip_video_direct_source: true
- type: m3u
- type: strm
directory: /tmp/kodi
- type: hdhomerun
username: hdhruser
device: hdhr1
use_output: xtream
options:
ignore_logo: false
epg_output:
lowercase_ids: true
lowercase_xmltv_display_names: false
share_live_streams:
hls: true
mpeg_ts: true
remove_duplicates: false
deduplicate:
match_by: caption
keep: best_quality
match_as_ascii: false
Target Option Parameters
| Parameter | Type | Required | Default | Technical Impact & Background |
|---|---|---|---|---|
ignore_logo |
Bool | No | false |
Ignores tvg-logo and tvg-logo-small attributes. This reduces downstream device-side logo caching and can keep generated M3U playlists leaner for clients with limited storage or poor cache invalidation behavior. |
share_live_streams.hls |
Bool | No | false |
Enables HLS live sharing for the new HLS cache proxy path. This is a configuration switch for the HLS cache feature and is independent from MPEG-TS stream sharing. |
share_live_streams.mpeg_ts |
Bool | No | false |
Allows Tuliprox to share MPEG-TS live stream connections in reverse proxy mode. This can reduce upstream provider connection usage when multiple clients watch the same channel, but it increases memory usage per shared channel. |
remove_duplicates |
Bool | No | false |
Legacy pre-transform identity deduplication. It runs independently for each input before the F/R/M pipe and removes repeated source identities before mapping can emit additional items. The field remains supported for backward compatibility. |
deduplicate |
Map | No | - | Post-merge, quality-aware content deduplication. It runs after all inputs have been transformed and merged. Channels whose match value is identical after stripping quality tokens (4K, UHD, 2160p, QHD, 1440p, FHD, 1080p, HD, 720p, SD, 480p, 576p) collapse to one entry. Sub-keys: match_by (caption (default), name, title), keep (best_quality (default), first) and match_as_ascii (default false). Matching is per cluster across all groups; ties keep the first occurrence. This lets mappings affect the final duplicate comparison results. |
epg_output.lowercase_ids |
Bool | No | false |
Canonicalizes visible technical EPG IDs with ASCII lowercase across M3U tvg-id, Xtream epg_channel_id, XMLTV channel/programme references, and EPG API responses. Changing this option requires a full target refresh. |
epg_output.lowercase_xmltv_display_names |
Bool | No | false |
Applies Unicode lowercase exclusively to XMLTV <display-name> values during serialization. Playlist names, Xtream names, programme titles, and programme descriptions remain unchanged; persisted target data does not require rebuilding. |
force_redirect |
Bool | No | false |
Optional redirect-related behavior switch. This influences how Tuliprox serves final stream delivery where redirect-style output handling is required by the deployment model. |
Shared HLS:
share_live_streams.hlsrequiresreverse_proxy.hls_cacheinconfig.yml. Start with Shared HLS Sessions for the feature overview and Shared HLS Configuration for the full setup checklist.Use the object form shown above. The old boolean style
share_live_streams: trueis not valid for this configuration, because HLS sharing and MPEG-TS sharing are independent switches.⚠️ Warning: When
share_live_streams.mpeg_tsis enabled, each shared channel consumes at least 12 MB of memory, regardless of the number of connected clients. If the reverse-proxy buffer size is increased above1024, memory usage increases accordingly. Example: with a buffer size of2048, each shared channel consumes at least 24 MB.
Quality-Aware Deduplication Example
Keep only the best-quality copy of every channel, collapsing entries like News HD, News FHD, and NEWS [4K]
into the single NEWS [4K] entry:
targets:
- name: clean_target
filter: 'Group ~ ".*"'
options:
deduplicate:
match_by: caption # caption (default) | name | title
keep: best_quality # best_quality (default) | first
match_as_ascii: false # true: "Café HD" matches "Cafe FHD"
output:
- type: m3u
- Matching compares the selected field with quality tokens stripped and remaining words lowercased, so unrelated channels never collapse.
keep: firstkeeps the first occurrence in playlist order instead of the highest quality tier (useful when provider ordering already encodes your preference).- Deduplication runs after group merging and before sorting; groups left empty are removed.
EPG Output Normalization
epg_output applies to the entire target so every output format uses the same EPG identity space. Both options
default to false; when they are omitted, Tuliprox preserves the source casing and existing output behavior.
targets:
- name: sample_target
options:
epg_output:
lowercase_ids: true
lowercase_xmltv_display_names: true
output:
- type: m3u
- type: xtream
With both options enabled, neutral input values such as Example.Channel and Sample Network produce a canonical
technical ID of example.channel and the following XMLTV output:
<channel id="example.channel">
<display-name>sample network</display-name>
</channel>
<programme channel="example.channel">
<title>Example Programme</title>
</programme>
lowercase_ids uses ASCII lowercase for technical identifiers. Non-ASCII characters in those IDs remain unchanged.
The canonical visible ID is used consistently for M3U tvg-id, Xtream epg_channel_id, XMLTV <channel id> and
<programme channel>, and Short EPG / Stream EPG responses. Target EPG database keys and API lookup keys use the same
target output casing: source case is preserved while the option is disabled, and ASCII lowercase is used after the
option is enabled and the target is refreshed. This keeps existing mixed-case target databases compatible while the
two XMLTV attributes continue to use exactly the same visible value.
lowercase_xmltv_display_names uses Unicode lowercase only while serializing XMLTV <display-name> values. It takes
effect for subsequent XMLTV responses and normally does not require rebuilding persisted target data. It does not
change playlist or Xtream names, programme titles or descriptions, input data, parser values, caches, or Smart Match
behavior.
Changing lowercase_ids requires a full target refresh so persisted playlist and EPG artifacts use the same visible
IDs; a configuration hot reload alone is insufficient. Clients may need to re-index their EPG data once after the
visible IDs change.
3.2.7 Output Formats (output)
A target can be exported to multiple formats simultaneously. The target-level filter, rename, mapping, and sort logic are applied first, and each output then formats the result differently.
Note: Output-specific filters are applied after all transformations have completed. Therefore, any filter inside an individual output block must refer to the final playlist state.
Output Block Parameters
Every output block contains at least:
| Parameter | Type | Required | Default | Technical Impact & Background |
|---|---|---|---|---|
type |
Enum | Yes | Output format type. Supported values include xtream, m3u, strm, and hdhomerun. This determines how Tuliprox serializes and serves the final playlist to downstream consumers. |
|
filter |
String | No | Optional output-level filter applied after all target transformations. This allows Tuliprox to derive specialized output subsets from the same target without duplicating upstream processing logic. |
Specific Output Properties are defined for each type:
1. Type xtream
output:
- type: xtream
skip_live_direct_source: true
skip_video_direct_source: true
skip_series_direct_source: true
update_strategy: instant
trakt:
api:
api_key: "YOUR_API_KEY"
version: "2"
url: "https://api.trakt.tv"
user_agent: "Mozilla/5.0"
lists:
- user: "gary"
list_slug: "latest-tv"
category_name: "Trending TV"
content_type: series
fuzzy_match_threshold: 80
xtream Parameters
| Parameter | Type | Required | Default | Technical Impact & Background |
|---|---|---|---|---|
type |
Enum | Yes | Must be xtream. Generates an Xtream-compatible API output backed by Tuliprox's processed data model. |
|
skip_live_direct_source |
Bool | No | true |
If true, Tuliprox ignores provider direct_source values for live content. This keeps playback under Tuliprox's delivery logic and avoids client behavior differences caused by bypass URLs. |
skip_video_direct_source |
Bool | No | true |
If true, Tuliprox ignores provider direct_source values for movies/VOD. This improves consistency across clients that otherwise may bypass Tuliprox for video playback. |
skip_series_direct_source |
Bool | No | true |
If true, Tuliprox ignores provider direct_source values for series entries. This ensures Tuliprox stays in control of series playback URL generation and proxy behavior. |
update_strategy |
Enum | No | instant |
instant writes changes immediately, while bundled batches write operations. This directly trades off freshness versus disk I/O load during background metadata enrichment and output maintenance. |
trakt |
Object | No | Trakt.tv integration block. Tuliprox can fetch Trakt lists, fuzzy-match them against playlist entries, and inject matched VOD or series entries into generated virtual categories. | |
filter |
String | No | Optional output-level filter for the Xtream export only. Useful when the same target should expose different subsets to different output formats. |
Note: IPTV players vary in how they resolve streams: some use the direct-source attribute, while others reconstruct URLs from server metadata. To ensure Tuliprox maintains control over the stream routing (Proxy/Redirect), the Direct Source Handling (skip_*_direct_source) attributes default to true.
⚠️ Warning: Setting
skip_*_direct_sourcetofalseforces the player to use the provider's originaldirect-sourceURL. This effectively bypasses Tuliprox, which will disable internal features like connection tracking, IP masking, and failover logic for those streams.
trakt Object in Xtream Output
Trakt.tv is an online platform for tracking, organizing, and discovering movies and TV shows. Tuliprox can query Trakt lists and match playlist entries using Jaro-Winkler-style fuzzy matching. Matching entries are then added to new virtual categories inside the Xtream output.
You can define a Trakt config like
inputs:
- name: my_xtream_input
type: xtream
options:
resolve_series: false
resolve_vod: false
sources:
- inputs:
- my_xtream_input
targets:
- name: iptv-trakt-example
filter: 'Group ~ ".*"'
output:
- type: xtream
skip_live_direct_source: true
skip_video_direct_source: true
skip_series_direct_source: true
trakt:
api:
api_key: "YOUR_API_KEY"
version: "2"
url: "https://api.trakt.tv"
user_agent: "Mozilla/5.0"
lists:
- user: "linaspurinis"
list_slug: "top-watched-movies-of-the-week"
category_name: "📈 Top Weekly Movies"
content_type: vod
fuzzy_match_threshold: 80
- user: "garycrawfordgc"
list_slug: "latest-tv-shows"
category_name: "📺 Latest TV Shows"
content_type: series
fuzzy_match_threshold: 80
charts:
- kind: movies
chart: trending
category_name: "🔥 Trending Movies"
tmdb_only: true
- kind: shows
chart: popular
category_name: "⭐ Popular Shows"
tmdb_only: true
This configuration creates additional virtual categories populated with matched entries from the configured Trakt user lists and public Trakt charts.
Trakt Parameters
| Parameter | Type | Required | Default | Technical Impact & Background |
|---|---|---|---|---|
api.api_key |
String | Yes | Trakt API key used for authenticated access. Without a valid key, Tuliprox cannot fetch remote list content. | |
api.version |
String | No | "2" |
API version header value. This ensures Tuliprox formats requests against the correct Trakt API version. |
api.url |
String | No | https://api.trakt.tv |
Base API URL for Trakt requests. This defines the remote endpoint Tuliprox queries for list data. |
api.user_agent |
String | No | Optional User-Agent used for Trakt API requests. This can help satisfy API gateway expectations or deployment-specific request policies. |
|
lists[].user |
String | Yes | Trakt username owning the list. This identifies which account namespace Tuliprox fetches list data from. | |
lists[].list_slug |
String | Yes | Trakt list slug. Combined with user, this uniquely identifies the remote list to load. |
|
lists[].category_name |
String | Yes | Name of the generated virtual category inside Tuliprox's Xtream output. This controls where matched entries appear to clients. | |
lists[].content_type |
Enum | Yes | vod or series. This determines which class of playlist entries Tuliprox will attempt to match and inject into the generated category. |
|
lists[].tmdb_only |
Bool | No | false |
If true, only exact TMDB-id matches are accepted for this list, disabling title/year fuzzy fallback and reducing false positives. |
lists[].fuzzy_match_threshold |
Integer | No | Fuzzy matching threshold for title matching. Higher values reduce false positives but may miss loosely matching items. | |
charts[] |
List | No | [] |
Public Trakt chart definitions. Unlike lists[], these are system charts and do not have a user/list owner. |
charts[].kind |
Enum | Yes | movies or shows. Aliases such as movie, vod, show, series, and tvshows are accepted. |
|
charts[].chart |
Enum | Yes | Public chart to fetch. MVP supports trending and popular. |
|
charts[].category_name |
String | Yes | Name of the generated virtual category inside Tuliprox's Xtream output. | |
charts[].tmdb_only |
Bool | No | false |
If true, only exact TMDB-id matches are accepted. This is recommended for dynamic charts to avoid fuzzy false positives. |
charts[].fuzzy_match_threshold |
Integer | No | Fuzzy matching threshold for chart title matching when tmdb_only is not enabled. |
The charts[] MVP intentionally supports only public, non-OAuth Trakt charts. User-specific recommendations and
account-scoped history feeds are not fetched by this block.
2. Type m3u
output:
- type: m3u
filename: custom_playlist.m3u
include_type_in_url: false
mask_redirect_url: false
filter: 'Type = live'
m3u Parameters
| Parameter | Type | Required | Default | Technical Impact & Background |
|---|---|---|---|---|
type |
Enum | Yes | Must be m3u. Generates a traditional playlist file suitable for IPTV players and related clients. |
|
filename |
String | No | Optional custom output filename. This affects how Tuliprox writes or exposes the generated playlist artifact. | |
include_type_in_url |
Bool | No | false |
If enabled, Tuliprox adds the stream type (live, movie, series) into generated stream URLs. This can improve downstream routing clarity and compatibility with clients that distinguish path structure by media type. |
mask_redirect_url |
Bool | No | false |
If enabled, Tuliprox uses URLs from api-proxy.yml for users operating in redirect proxy mode. This is important for multi-provider failover or cycling setups where exposing the provider URL directly would bypass Tuliprox's routing logic too early. |
filter |
String | No | Optional M3U-only post-transformation filter. This allows M3U consumers to receive a narrower subset than other output formats derived from the same target. |
Note:
mask_redirect_urlshould be enabled if you use multiple providers and want Tuliprox to preserve redirect-mode routing and cycling behavior without exposing the direct upstream endpoint in the initial playlist URL.
M3U Catchup & Archive Support
Tuliprox preserves the following catchup and archive attributes when it imports and exports M3U playlists:
catchupcatchup-dayscatchup-sourcecatchup-timecatchup-correctioncatchup-type- unknown attributes whose names start with
catchup-
When an output keeps direct provider URLs, the catchup metadata is written unchanged. For reverse-proxied outputs, Tuliprox replaces supported provider templates with authenticated local URLs. The provider URL and its credentials are not included in the generated catchup URL.
The template modes default, append, shift, xc, fs, and vod are supported. An unknown mode can still be
proxied when it supplies an explicit source template. Without a usable template, Tuliprox preserves the metadata but
cannot create a local catchup URL.
Native Flussonic playback is available for both HLS and MPEG-TS:
- HLS uses
.m3u8archive, relative-timeshift, and absolute-timeshift paths. - MPEG-TS uses absolute-timeshift
.tspaths. - Flat archive requests generated by TiviMate and nested Flussonic archive paths are accepted.
- A live item with a
timeshiftvalue but no catchup block is treated as Flussonic-style catchup.
For HLS archives, Tuliprox carries the archive start parameter into same-origin child playlists when the child does not already contain one. It does not add archive parameters to media segments, encryption keys, or initialization files.
When API server information is available, the generated #EXTM3U header contains url-tvg and x-tvg-url. Both point
to the authenticated Tuliprox XMLTV endpoint for the playlist user; a provider-supplied url-tvg value is not forwarded.
Tuliprox also reads a per-channel #EXTVLCOPT:http-user-agent directive. It writes the directive back when the output
contains the direct provider URL. For rewritten URLs, the value is kept internal and applied to the upstream HLS or
MPEG-TS request instead of being exposed in the playlist. Disabling the User-Agent header through the reverse-proxy
header settings takes precedence.
3. Type strm
output:
- type: strm
directory: /media/strm
username: local_user
style: jellyfin
flat: true
cleanup: false
underscore_whitespace: false
add_quality_to_filename: true
use_metadata: false
strm_props:
- "#KODIPROP:seekable=true"
- "#KODIPROP:inputstream=inputstream.ffmpeg"
filter: 'Type = vod'
Generates local .strm files for Emby, Jellyfin, or Kodi-based library ingestion.
Upgrade note:
style: plexis no longer supported. Existing STRM targets must switch tokodi,emby, orjellyfin, for examplestyle: plex->style: jellyfin. For Plex use cases, use thehdhomerunintegration instead. Leavingstyle: plexin the configuration will fail validation.
strm Parameters
| Parameter | Type | Required | Default | Technical Impact & Background |
|---|---|---|---|---|
type |
Enum | Yes | Must be strm. Generates filesystem-based .strm references instead of a network playlist format. |
|
directory |
String | Yes | Target directory where .strm files are written. This is the root Tuliprox manages for exported media stubs and must be chosen carefully to avoid overlap with real media directories. |
|
username |
String | No | Optional username context used when generating stream references. This affects which user-specific URL or access context Tuliprox embeds into the exported .strm files. |
|
underscore_whitespace |
Bool | No | false |
Replaces whitespace with _ in paths and filenames. This improves compatibility with environments or scrapers that prefer filesystem-safe, normalized naming. |
cleanup |
Bool | No | false |
If enabled, Tuliprox removes orphaned output files from the STRM directory. This keeps the export directory synchronized with the target, but can delete files if the directory points to an existing media folder. |
style |
Enum | Yes | Naming convention for the output structure. Supported values: kodi, emby, jellyfin. This affects scraper compatibility and how downstream media servers identify titles. |
|
flat |
Bool | No | false |
If enabled, Tuliprox creates a flatter directory structure. This changes how categories and group information are represented on disk and can simplify some media-server imports. |
strm_props |
List | No | Stream property lines inserted into .strm files, mainly for Kodi player behavior. This allows low-level playback hints to be embedded directly into generated files. |
|
add_quality_to_filename |
Bool | No | false |
Appends detected media quality tags such as [1080p 4K HEVC HDR] to the filename. This improves visibility in library UIs but depends on prior probing/enrichment data being available. |
use_metadata |
Bool | No | false |
Uses the media metadata name for STRM filenames and folders. By default, the target's processed title is used, so rename and mapping rules affect the generated paths. If metadata has no name, the processed title remains the fallback. |
filter |
String | No | Optional STRM-only output filter. Useful when only a subset of the target should be materialized as filesystem entries. |
Supported style Conventions
- Kodi:
Movie Name (Year) {tmdb=ID}/Movie Name (Year).strm - Emby:
Movie Name (Year) [tmdbid=ID]/Movie Name (Year).strm - Jellyfin:
Movie Name (Year) [tmdbid-ID]/Movie Name (Year).strm
Kodi-Specific Behavior
If style: kodi is selected:
#KODIPROP:seekable=true|falseis added automatically- if
strm_propsis not specified, Tuliprox additionally sets:#KODIPROP:inputstream=inputstream.ffmpeg#KODIPROP:http-reconnect=true
⚠️ Warning: If
cleanupis enabled, do not pointdirectoryat a real media library folder. Tuliprox may delete files that are no longer part of the generated target.
4. Type hdhomerun
output:
- type: hdhomerun
device: hdhr1
username: local_user
use_output: xtream
This binds the target to a configured HDHomeRun virtual tuner device from config.yml.
hdhomerun Parameters
| Parameter | Type | Required | Default | Technical Impact & Background |
|---|---|---|---|---|
type |
Enum | Yes | Must be hdhomerun. Exposes the target through Tuliprox's HDHomeRun emulation layer for tuner-style discovery by clients such as Plex or Jellyfin. |
|
device |
String | Yes | Must match a device name defined in config.yml. This links the playlist target to a specific emulated tuner endpoint. |
|
username |
String | Yes | Must match a user from api-proxy.yml. This determines which account context, access restrictions, and connection limits apply when clients consume the lineup through the tuner interface. |
|
use_output |
Enum | No | Selects whether the HDHomeRun stream URLs are based on m3u or xtream output behavior. This affects how playback URLs are generated and which delivery semantics back the tuner lineup. |
3.2.8 Favourites (favourites)
favourites lets you duplicate final transformed channels into dedicated favorite groups after
filtering, renaming, mapping, and other transformations are complete.
favourites:
- cluster: series
group: "My Favourites"
filter: 'Name ~ "Cinema"'
match_as_ascii: true
favourites Parameters
| Parameter | Type | Required | Default | Technical Impact & Background |
|---|---|---|---|---|
cluster |
String | No | Optional logical cluster, for example series. This influences how Tuliprox groups the duplicated entries internally for output generation. |
|
group |
String | Yes | Name of the favorite group created in the final playlist. This adds a curated access path without removing the original group membership. | |
filter |
String | Yes | Filter expression selecting which final entries should be duplicated into the favorites group. This operates on the transformed end state rather than the original raw input. | |
match_as_ascii |
Bool | No | false |
If enabled, Tuliprox normalizes accented characters during matching. This improves filter matching robustness across multilingual names while preserving the original visible title in output. |
3.2.9 Watch (watch)
For each target with a unique name, you can define watched groups. It is a list of group patterns Tuliprox monitors for content changes during updates.
If matching groups gain or lose channels, Tuliprox emits a Messaging event such as:
- channels added
- channels removed
watch:
- group: '^Sports'
- group: '^Movies'
watch Parameters
| Parameter | Type | Required | Default | Technical Impact & Background |
|---|---|---|---|---|
group |
String (Regex) | Yes | Regex pattern matched against final group names. This allows Tuliprox to detect meaningful content changes in selected areas of the playlist and notify operators automatically through the configured messaging backends. |
Note:
watchis especially useful for monitoring premium groups, VOD collections, or unstable provider segments where additions and removals should generate operational alerts.