mirror of
https://github.com/euzu/tuliprox.git
synced 2026-09-27 11:32:11 +02:00
84f66a07b8
* refactor(repository): return concrete playlist streams instead of Box<dyn Stream>
The three raw-playlist iterators each wrapped an already-concrete type in a
trait object: m3u produced a LockedReceiverStream and xtream/stalker produced a
ReceiverStream, then boxed it behind `dyn Stream + Send + Unpin`. Nothing in the
workspace stored these heterogeneously or needed object safety, so the box was
one allocation and one vtable per call for no benefit.
Name the concrete types in the signatures instead:
iter_raw_m3u_{target,input}_playlist -> LockedReceiverStream<Result<M3uPlaylistItem, _>>
iter_raw_xtream_{target,input}_playlist -> ReceiverStream<Result<XtreamPlaylistItem, _>>
iter_stalker_{items,series_roots} -> ReceiverStream<StalkerPlaylistItem>
Both concrete types are Stream + Unpin, so every caller keeps compiling
unchanged; the `futures::Stream` / `tokio_stream::Stream` imports are now unused
in two of the three files and are dropped.
Verified with cargo check on tuliprox-repository and tuliprox plus nightly fmt.
Not verified by tests.
* refactor(repository): extract PlaylistBackend so M3U and Xtream share one iterator
iter_raw_m3u_playlist and iter_raw_xtream_playlist were two transcriptions of
one design: the same two read locks, the same spawn_blocking producer feeding a
bounded channel, the same sorted-index reader with per-entry error logging,
differing only in the item type, the storage subdirectory and which
TuliproxError variant wrapped a failure. The storage-path pair
(m3u|xtream)_get_storage_path and ensure_(m3u|xtream)_storage_path were
identical modulo one const and one error constructor.
New `playlist_backend` module names those differences as associated types and
consts, so the shared body is written once:
trait PlaylistBackend { type Item; type SortKey; SUBDIR; LABEL;
HOLD_ITER_LOCK; repo_error(); storage_path() }
struct M3u; struct Xtream; // zero-sized markers
iter_raw_playlist<B, K, F>(..) -> Option<LockedReceiverStream<..>>
Dispatch is entirely static: the markers are ZSTs, the item filter is an
`impl Fn` monomorphized per call site, and the B+Tree key stays a function
parameter because M3U keys its target store by u32 and its input store by
Arc<str> — so the key is not a property of the backend.
Two deliberate non-unifications:
* Stalker is not a PlaylistBackend. Its store is keyed by input rather than
target, has no sorted index, and yields bare items rather than Results.
Folding it in would add methods that exist only to return None for one of
three implementors.
* HOLD_ITER_LOCK preserves an existing behavioural difference rather than
normalising it. M3U holds a read lock for the consumer's lifetime; Xtream
never did. Changing when that lock is released would change what a concurrent
writer can do mid-iteration, which is not this commit's business.
The only behaviour change is that M3U now also logs a reader-open failure
before forwarding it, which Xtream already did.
Net: 151 lines of duplicated logic in the two repositories replaced by one
shared implementation. Verified with cargo check --workspace and nightly fmt.
Not verified by tests.
* refactor(repository): replace boxed playlist iterators with concrete enums
PlaylistSourceOps is a private trait that is never used as a trait object --
PlaylistSource has always dispatched through the PlaylistSourceKind enum, and no
Box<dyn PlaylistSourceOps> exists anywhere in the workspace. Its three iterator
methods nevertheless returned Box<dyn Iterator<..> + Send + '_>, so the hottest
traversal in the repository paid one allocation per call plus one indirect,
uninlinable call per playlist item. The skip-set filter boxed a second time on
top of that.
The erasure could not simply be deleted: the bodies were composed adapter chains
(flat_map(..).map(..)) whose closure types cannot be named. New `playlist_items`
module replaces them with hand-rolled state machines whose types *are* nameable:
BTreeValues<'a, K, V> one B+Tree store, logging and skipping
unreadable entries (was filter_map(..))
BTreeStores<'a, K, V, const N> N stores back to back (was chain(..)):
N=1 single-file, 3 Xtream, 4 Stalker
MemoryDrain / MemoryItemsMut / MemoryItems the in-memory flat_map shapes
SourceItems / SourceCowItems / SourceItemsMut one variant per source kind
ClusterFiltered<I> skip-set filter, folded in rather than
wrapped -- a filtered traversal now
allocates nothing at all
PlaylistSource::into_items() -> ClusterFiltered<SourceItems<'_>>
Also in this commit, since all of it is the same erasure:
* update_playlist and obtain_resources drop BoxFuture for `async fn` (AFIT,
stable since 1.75). Static dispatch means auto-traits still propagate.
* The two Vec<(XtreamCluster, Box<dyn Iterator<..>>)> locals in take_groups
become Vec<(XtreamCluster, BTreeValues<..>)> -- every reader in those vecs
already had the same concrete type.
* A `dispatch!`/`dispatch_await!` macro pair replaces the hand-written
forwarding arms, and doubles as the conformance check the trait used to
provide: a variant missing a method fails to compile at the macro.
playlist_source.rs now contains zero `Box<dyn` and zero BoxFuture. Behaviour is
preserved, including the read-only warning on disk sources (now emitted where
the empty iterator is built) and the verbatim B+Tree skip-entry log message.
FetchedPlaylist forwards the concrete types rather than re-boxing them.
Verified with cargo check --workspace and nightly fmt, plus a structural check
that all five PlaylistSourceOps impls still carry their original method sets.
Not verified by tests.
* refactor(shared): key field access on a typed enum instead of a string
PlaylistItemHeader had two field-access paths and only one of them was cheap.
The filter engine in foundation/value_provider.rs matched on the typed
ItemField enum and returned borrowed &str. The mapper, sort and counter paths
went through FieldGetAccessor::get_field(&str), which walked a chain of ~20
eq_ignore_ascii_case comparisons and returned an owned Arc<str> -- so `chno`
and `type` did `.to_string().intern()`, a heap allocation *and* an interner
write lock, on every read, per item, per rule.
Adds the typed accessor the enum path deserved:
HeaderField every field addressable by name, with parse()/as_str()
FieldRef<'a> Shared(&Arc<str>) | Str(&str) | Num(u32) -- a read
never forces an allocation
FieldGet / FieldSet typed traits; the impl is a match on a discriminant
genre_ref! borrowing sibling of get_genre!
HeaderField is deliberately *wider* than ItemField rather than an extension of
it. ItemField is the config-facing vocabulary, and widening it would widen what
a user may write in a filter or sort rule; the accessor's domain is genuinely
larger (logo_small, audio_track and friends are addressable by name by the M3U
resource endpoint without being valid config fields). ItemField::header_field()
maps between them, returning None for Quality, which is derived from the caption
rather than stored.
FieldGetAccessor/FieldSetAccessor are kept as #[inline] shims over the typed
traits, so callers whose field name genuinely arrives as a string keep working
and keep their exact previous behaviour, interning of numbers included.
Migrated every call site that already held type information:
* sort.rs held an ItemField and called provider.get(field.as_ref()) -- turning
a typed enum into a string to do a linear string lookup. Now get_typed().
* MappingCounter.field becomes HeaderField, parsed once in prepare() instead of
re-parsed per channel inside the counter loop.
* trakt.rs's "caption" literals become HeaderField::Caption.
Left on the &str path deliberately: M3uPlaylistItem and XtreamPlaylistItem keep
their own string accessors. Their only caller is the M3U resource endpoint,
where the field name arrives in the URL and the lookup happens once per request,
not per item.
Behaviour is preserved throughout, including which fields are settable (input,
type and provider_id still return false) and which are absent per type (`id` is
a header field, `provider_id` an M3U one).
Verified with cargo check --workspace --all-targets and nightly fmt.
Not verified by tests.
* refactor(config): give HLS timing config units in the type system
Roughly 245 config and state fields across the workspace encode their unit as a
naming convention, and several sit adjacent inside the same struct.
HlsCacheConfigDto is the clearest case: origin_manifest_timeout_ms and
origin_segment_timeout_ms sit two lines above initial_manifest_wait_timeout_secs,
all four u64, nothing stopping a millisecond value reaching a seconds parameter.
Worse, cache_duration and session_idle_timeout do not carry the unit in their
names at all -- both are seconds, which is only discoverable from their defaults.
Adds Millis and Secs in shared/src/model/config/time_units.rs:
#[repr(transparent)] same layout as the u64 they replace -- no size cost,
no indirection, no allocation
#[serde(transparent)] same serialized form, so the YAML/JSON shape is
byte-identical and this lands with no config migration
and nothing user-visible
Conversions are methods, not From impls: `Secs -> Millis` is a multiplication
that can overflow, and forcing it to be spelled out at the call site is the
point of having the types. `as_duration_at_least_1ms` collects the `.max(1)`
that several origin deadlines applied individually, because they treat a zero
timeout as "do not wait" rather than "wait forever".
Applied to the six HLS timing fields on HlsCacheConfigDto and
HlsSegmentRepairConfigDto and, importantly, to their runtime counterparts in
tuliprox-core's HlsCacheConfig, so the type survives the Dto -> runtime hop
rather than being unwrapped at the boundary. The validation helper
ensure_min_u64 splits into ensure_min_millis / ensure_min_secs so a bound cannot
be compared against the wrong unit either.
Two manual secs->ms multiplications in backend/hls (gc.rs and manager.rs's
transient_resource_ttl_ms) become `.as_millis()`, which is where the conversion
was silently open-coded before.
Deliberate boundary: the structs inside backend/hls that do arithmetic on raw
millisecond counts keep u64 and take `.get()` at the edge. Pushing the newtypes
through those is the next slice; doing it here would have made this commit touch
most of the crate for no additional safety at the config surface.
Includes tests asserting the serialized form is a bare number, that layout is
transparent over u64 (Option<Millis> included), and that Secs::as_millis
saturates.
Verified with cargo check --workspace --all-targets and nightly fmt.
Not verified by tests.
* chore(lint): satisfy the nightly clippy gate after the static-dispatch work
Three findings from `cargo +nightly-2026-05-01 clippy --workspace -- -D warnings`:
* doc_markdown on a macro doc comment in shared/src/model/playlist.rs.
* clone_on_copy in the frontend's mapper counter view, now that
MappingCounter::field is a Copy HeaderField rather than a String.
* large_enum_variant on SourceItems and SourceCowItems, allowed with a
justification rather than fixed. The B+Tree disk iterators carry sizeable
cursor state, so holding three or four inline makes the enum ~1.9KB, and
clippy's suggested fix -- boxing the large variants -- would reintroduce
exactly the per-traversal heap allocation the type exists to remove. The cost
is stack space for one value per traversal, not per item: the enum is built
once and then driven through &mut, so it is never copied per element.
Gates now green: cargo build --workspace, clippy --workspace and
clippy --workspace --all-targets at -D warnings, fmt --all --check, and
bin/check-workspace-deps.sh (78 edges).
* refactor(shared): split TuliproxError into a Copy kind and a message
All 50 variants carried exactly one String and nothing else, so this was never
an enum of errors -- it was a category tag beside a message. Encoding it as an
enum cost a 50-arm `message()` whose only job was to return the payload, plus a
second 50-name list in `is_notify()` that had to be kept in sync by hand, and
made the category impossible to compare, store or return on its own.
enum ErrorKind { Config, ConfigApp, ... } // fieldless, Copy, Eq, Hash
struct TuliproxError { kind: ErrorKind, message: Box<str> }
An `error_kinds!` macro declares each category once -- variant, display label
and constructor -- so the label table and the notify set can no longer drift
apart. `message()` is now one field access, `is_notify()` delegates to the kind,
and `kind()` exposes the category directly.
The 755 construction sites are untouched. A tuple-variant constructor and an
associated function are invoked with identical syntax, so `TuliproxError::
Config(msg)` still compiles once `Config` is an `#[allow(non_snake_case)]`
associated fn. The non-snake-case names are the deliberate price of splitting
the type without rewriting every call site in one commit; renaming them to
`config()` style is a separate, mechanical follow-up.
Only the 10 sites that pattern-matched on a variant needed changing, and they
read better for it -- `err.kind() == ErrorKind::ApiXtream` instead of
`matches!(err, TuliproxError::ApiXtream(_))`, and for the two that also
inspected the payload, an explicit `kind()` plus `message().contains(..)`.
Two incidental improvements: the constructors take `impl Into<Box<str>>`, so
call sites holding a `&str` no longer need `.to_string()` (two test sites drop a
now-ambiguous `.into()`); and the error is one word smaller than the enum was.
Tests pin the behaviour that had to be preserved: the `"label: message"` Display
shape for four categories, that `message()` excludes the label, that both &str
and String are accepted, and the exact set of notifying categories.
Verified with cargo check --workspace --all-targets, clippy --all-targets at
-D warnings, nightly fmt, and the new unit tests.
* refactor(core): add a Clock seam and collapse 14 copies of current_time_millis
`fn current_time_millis() -> u64 { chrono::Utc::now().timestamp_millis()
.try_into().unwrap_or_default() }` was duplicated character for character in 14
files across backend/app and backend/hls. New tuliprox-core::utils::clock holds
the single copy, plus the trait that makes "what time is it" injectable:
trait Clock { fn now_ms(&self) -> Millis }
struct SystemClock; // ZST: no field, no vtable, no allocation
struct ManualClock(Arc<AtomicU64>); // deterministic; the Arc is confined here
Clock is meant to be held as a generic parameter defaulted to the ZST --
`struct Deadlines<C: Clock = SystemClock>` -- never as Arc<dyn Clock>. A test
asserts SystemClock is zero-sized and that owning one leaves a struct's layout
unchanged, which is the property that makes the seam free.
Scope is smaller than it first looked, and the reason is worth recording: most
of the deadline logic in tuliprox-hls already takes `now_ms: u64` as a function
parameter. HlsAccessLease, HlsAvailabilityReevaluationCycle and their neighbours
are already injectable and already tested that way, and passing the instant in
is a better pattern than reaching for a clock -- so they are left alone. The
trait is for the callers that have to *produce* the instant.
HlsTerminalCommitClock is deliberately left alone too. It fakes time with an
AtomicU64 sentinel, costing an atomic load per production read, and would be a
natural fit -- but it is owned by HlsProxy, so making it generic would push a
type parameter onto HlsProxy and from there onto AppState. A type parameter that
reaches the root state is exactly the case where the status quo wins; the module
docs say so, so the next person does not have to rediscover it.
The 14 call sites now import the shared function, preserving each one's original
visibility (two were pub(super) and re-export as such).
Verified with cargo check --workspace --all-targets, clippy --all-targets at
-D warnings, nightly fmt, and 4 new unit tests.
* refactor(shared): give config preparation one shape via a Prepare trait
Config types are deserialized first and resolved second -- templates expanded,
regexes compiled, filters parsed, derived fields computed. That second phase was
spread across ~98 inherent `prepare`/`validate` methods with no agreed
signature: some took nothing, some `Option<&[PatternTemplate]>`, some a storage
dir, a device number, a port or an `include_computed: bool`, returning variously
`()`, `Result<(), TuliproxError>`, `Result<(), &'static str>` or `bool`.
Because the shape was invisible, the recursive walk had to be hand-written at
every level, and a new config struct that forgot to call its children's prepare
failed silently at runtime rather than at compile time.
trait Prepare { type Ctx<'a>: Copy; fn prepare(&mut self, Self::Ctx<'_>) -> Result<(), TuliproxError> }
The context is an associated type, so a node needing pattern templates and a
node needing a port are both implementors without a lowest-common-denominator
argument list, and dispatch stays static -- the GAT is monomorphized per
implementor, with no trait object anywhere.
Migrated the 14 methods that already shared the exact signature
`prepare(&mut self, Option<&[PatternTemplate]>) -> Result<(), TuliproxError>`:
MapperOperation, MapperDto, MappingDto, MappingDefinitionDto, MappingsDto,
ConfigRenameDto, ConfigSortRuleDto, ConfigSortDto, ConfigFavouritesDto,
ConfigInputOptionsDto, and the four target-output types. `Ctx<'a>` is
`Option<&'a [PatternTemplate]>` for all of them, so they now visibly share a
contract instead of coincidentally sharing an argument list.
Blanket impls for Vec, slices, Option and Box are the payoff: two hand-written
child walks are gone, including a nested `Option<Vec<MapperDto>>` that took an
`if let` plus a `for` and is now one line.
Two things deliberately left alone:
* The `handle_tuliprox_error_result_list!` call sites in sort.rs and target.rs
aggregate every child's error rather than stopping at the first. Switching
them to the blanket impl would silently reduce a config report listing all
problems to one naming only the first. Aggregation is the better behaviour for
config validation, so those walks stay until Prepare can express it.
* The loop in ConfigTargetDto that prepares each output also counts output kinds
as it goes, so the prepare call is incidental to a loop doing more.
Includes 5 unit tests covering the blanket impls: same context to every child,
short-circuit on first failure matching the previous `?` behaviour, absent
Option as a no-op, nesting, and Box forwarding.
Verified with cargo check --workspace --all-targets, clippy --all-targets at
-D warnings, nightly fmt, and 970 tests across shared, core and config-loader.
* refactor(shared): add PrepareAll so config errors aggregate again
Prepare's collection impls short-circuit on the first failing child, which is
right for a nested walk but wrong for config validation: a user with three bad
sort rules should hear about all three in one pass, not fix them one round-trip
at a time. That is why the two remaining hand-written walks in sort.rs and
target.rs still used handle_tuliprox_error_result_list! after the Prepare
migration -- switching them to the blanket impl would have silently degraded a
config report to its first line. This closes that gap.
trait PrepareAll: Prepare { fn prepare_all(&mut self, Self::Ctx<'_>) -> Result<(), TuliproxError> }
Aggregation is byte-identical to the macro's: collect each failure's rendered
message, join with newlines, wrap in TuliproxError::Errors. Impls for Vec, [T],
Option and Box, so `Option<Vec<ConfigRenameDto>>` nests without a hand-written
`if let` plus loop -- target.rs's three-line walk becomes one line.
With both call sites converted the macro is dead, so it is removed along with
its re-export. `get_errors_notify_message!` next to it is still used by the
playlist processor and stays.
Three tests cover what the macro guaranteed: every failing child reported (not
just the first), every child still run despite earlier failures, and nesting
through Option.
Verified with cargo check --workspace --all-targets, clippy --all-targets at
-D warnings, nightly fmt, and 973 tests across shared, core and config-loader.
* refactor(shared): drop the field-accessor macros for explicit typed impls
Three macros generated the by-name field accessors. My earlier read of them as
"near-identical" was wrong in one important way, and the correction is the most
valuable part of this commit.
XtreamPlaylistItem's accessor is not a variant of the other two: it carries a
~80-line prefix-matched lookup into additional_properties for Xtream cover and
backdrop_path resources. It also has **no callers**. FieldGetAccessor is never a
generic bound and never a trait object, and the only call site in the workspace
is the M3U resource endpoint calling M3uPlaylistItem::get_field. So the whole
impl was unreachable; it is deleted rather than ported. The XC_PROP_* constants
it referenced are still used elsewhere, and the logic is in git history if the
Xtream resource endpoint ever needs cover-by-name.
The remaining two are collapsed onto the typed HeaderField/FieldGet/FieldRef
path from the earlier field-access work, written out explicitly instead of
macro-generated. A macro that exists to share ten repetitive match arms between
two types is not paying for itself: the explicit arms are the same length and
you can read them without expanding anything. M3uPlaylistItem also stops walking
a chain of eq_ignore_ascii_case comparisons and stops interning `chno` on every
read.
One behaviour is deliberately preserved rather than "fixed": M3uPlaylistItem
carries input_name, item_type and additional_properties, but none of them were
ever addressable by name, and the M3U resource endpoint resolves a URL path
segment through get_field -- so making them resolvable would turn a 404 into a
response. The explicit `Id | Input | Type | Genre => None` arm says so, and a
test asserts it.
Net 104 lines removed from playlist.rs, and zero accessor macros remain.
Two new tests: every HeaderField variant round-trips through parse/as_str
case-insensitively (with the epg_id alias), and the M3U item resolves
provider_id, name, chno and caption while refusing the header-only names.
Verified with cargo check --workspace --all-targets, clippy --all-targets at
-D warnings, nightly fmt, and the shared test suite.
* refactor(config): resolve ByteSize into a Bytes newtype once
`ByteSize(String)` was parsed with `parse_bytes() -> Result<u64, String>` at 15
sites, and the parsed value was typed as a bare u64 wherever it landed -- so a
resolved size was indistinguishable from any other u64, and a runtime struct
could hold either the string or the number with nothing marking which.
Adds `Bytes(u64)` beside `ByteSize`: #[repr(transparent)], #[serde(transparent)],
Copy. `parse_bytes()` now returns it, and the runtime structs hold it:
HlsCacheConfig::{cache_bytes, cache_bytes_per_session} u64 -> Bytes
FfprobeConfig::{probe_size_bytes, live_probe_size_bytes} u64 -> Bytes
`Bytes::at_least_1()` collects the `.max(1)` that probe sizes applied
individually, because they treat 0 as "unset" rather than "no bytes".
One correction to how I pitched this. I described FfprobeConfig's parallel
`probe_size: ByteSize` / `probe_size_bytes: u64` fields as redundant. They are
not: `From<&FfprobeConfig> for FfprobeConfigDto` reads the string form to send
the user's own spelling back to the web UI, so someone who wrote `10MB` sees
`10MB` rather than 10485760. Both fields stay, and a test records why so the
string one does not get "cleaned up" later.
Nor is this a performance change -- the parses happened at config load, not per
request. The win is that the resolved value now has a type: `Bytes` cannot be
passed where some unrelated u64 is wanted, and a runtime struct can no longer
hold an unparsed size by accident. Boundaries that genuinely need the number
(the ffmpeg CLI arg, the HLS cache-limit setter, the GC policy) take `.get()`
explicitly, as with Millis/Secs.
Two tests: transparent layout over u64 including Option, at_least_1 flooring,
and the ByteSize/Bytes division of labour.
Verified with cargo check --workspace --all-targets, clippy --all-targets at
-D warnings, nightly fmt.
* perf(parser): drop redundant Arc clones in the Xtream per-item parse loop
Inside `for stream in xtream_streams`, four fields wrapped a clone around a
method that already returns an owned value:
name: Arc::clone(&stream.get_name()) -> get_name() -> Arc<str>
logo: Arc::clone(&stream.get_stream_icon()) -> Arc<str>
title: Arc::clone(&stream.get_name()) -> Arc<str>
epg_channel_id: stream.get_epg_channel_id().clone() -> Option<Arc<str>>
Each one bumped the refcount to two and then dropped the temporary back to one:
four redundant atomic pairs per parsed Xtream stream, on the playlist parse path.
The same file already builds a header correctly without the outer clones 230
lines further down, so this was drift rather than intent.
The workspace now has zero `Arc::clone(&x.y())` sites. The three remaining
`.get_*().clone()` calls are correct -- those accessors return references
(`&Arc<str>`, `&Url`) and the clone is what makes the value owned.
Scope note: this is the substance of what I proposed as "borrowing accessors on
PlaylistEntry", but not the mechanism, because the mechanism did not survive
contact with the call sites. Of 23 `get_input_stream_id` calls, 19 are test
assertions; `get_provider_url` and `get_group` have one production caller each,
and the per-item `get_name` callers need an owned Arc to store. Adding four
borrowing methods to an already 13-method trait to save a refcount bump at ~4
non-hot call sites would grow the API surface for no measurable gain. Where a
caller genuinely only reads, borrowed access already exists via
`FieldGet`/`FieldRef::Shared` from the typed field-access work.
Verified with cargo check --workspace --all-targets, clippy --all-targets at
-D warnings, nightly fmt.
* refactor(shared): type the mapper and counter field allow-lists
COUNTER_FIELDS and MAPPER_FIELDS were `&[&str]` checked by a case-sensitive
string `contains` behind a `valid_property!` macro that added nothing over
`.contains()`. Both are now `&[HeaderField]`.
Three things fall out.
MAPPER_FIELDS loses an entry. The string list spelled the EPG channel id twice,
`epg_channel_id` and `epg_id`, to cover both accepted names. Aliases resolve in
HeaderField::parse, so the typed list names the field once and each spelling is
handled in exactly one place.
The counter path stops doing the same work twice. Since MappingCounter::field
became a HeaderField, prepare() ran a string `contains` and then parsed the same
name again; it is now one resolve-and-check.
`valid_property!` is deleted along with its export, replaced by
`is_allowed_field(name, allowed)`.
One deliberate behaviour change, called out because it loosens validation: the
allow-list was case-sensitive while set_field compared case-insensitively, so a
mapper naming `NAME` was rejected at config load even though writing it would
have worked. Resolving through parse makes the two agree. Every previously valid
config stays valid; some previously rejected ones are now accepted and behave
correctly.
Three tests: both EPG spellings resolve through the single entry, valid-but-
unlisted fields (`input`, `type`) are still rejected and COUNTER_FIELDS is a
strict subset of MAPPER_FIELDS, and casing now agrees with the accessor.
Verified with cargo check --workspace --all-targets, clippy --all-targets at
-D warnings, nightly fmt.
* refactor(shared): put genre on StreamProperties instead of in three macros
Video keeps its genre under `details`, Series keeps it inline, and Live and
Episode do not have one. That four-arm match was written out five times: in
`get_genre!`, in `genre_ref!`, in the Some-branch of `set_genre!`, inline in
`get_filter_value`, and inline in the header's typed field accessor.
Two methods on StreamProperties replace all of it:
fn genre(&self) -> Option<&Arc<str>>
fn set_genre(&mut self, value: &str) -> bool
`get_genre!` and `genre_ref!` are deleted outright -- one call site each, and
both are now `.additional_properties.as_ref().and_then(StreamProperties::genre)`
with an `Arc::clone` only where the caller actually needs to own it.
`set_genre!` stays a macro but loses its four-arm match, dropping from 79 lines
to 64. Its remaining bulk is the None-branch, which constructs a whole
StreamProperties::Video or ::Series *from the header* -- that needs the header,
not just the properties, so it does not belong on StreamProperties and stays
with the caller.
Left alone: ui_playlist_item.rs has the same Video-here/Series-there shape for
`rating`, but with one call site there is no repetition for an accessor to
remove, and adding API for a single caller is the trade this refactor exists to
avoid.
Two tests: genre round-trips through both storage shapes (including that setting
twice reuses existing Video details rather than replacing them), and Live and
Episode report no genre and refuse to take one.
Verified with cargo check --workspace --all-targets, clippy --all-targets at
-D warnings, nightly fmt, and the shared test suite.
* refactor(shared): single-source the item-type to cluster relation
The relation was written down twice and wrapped once:
* `PlaylistItemType::is_cluster(cluster)` matched every variant against a
cluster by hand.
* `impl TryFrom<PlaylistItemType> for XtreamCluster` matched every variant to a
cluster, independently, with nothing keeping the two in agreement.
* `cluster_from_item_type` in the repository wrapped the second with
`.unwrap_or(Live)`.
The TryFrom was total -- every arm returned `Ok` -- so its `Result` was a lie,
and the phantom error had spread defensive noise to 17 call sites across four
crates: `.unwrap_or(XtreamCluster::Live)` eleven times, plus
`.unwrap_or_default()`, `.ok()`, `.is_ok_and(..)`, `.unwrap_or(existing_cluster)`
and one `.map_err(..)?` building an error message that could never be produced.
Now there is one encoding:
PlaylistItemType::cluster(self) -> XtreamCluster // const, infallible
impl From<PlaylistItemType> for XtreamCluster // delegates
is_cluster(cluster) // delegates
All 17 sites drop their fallback. `cluster_from_item_type` is gone. The
`try_cluster!` macro in xtream_repository, whose `ok_or_else` could never fire,
becomes `cluster_or_item_type!` and its four call sites lose a `?`. The STRM
resolve path loses a five-line unreachable error branch.
Note this is source-compatible rather than a breaking change: std's blanket
`TryFrom for U where U: Into<T>` means `XtreamCluster::try_from(item_type)` still
compiles, now with `Error = Infallible`. The old call sites would have kept
working; they are cleaned up because the fallbacks are provably dead, not
because the compiler demanded it.
A test iterates every PlaylistItemType variant and asserts `From` agrees with
`cluster()`, that `is_cluster` accepts its own cluster, and that it rejects the
other two -- so the arms cannot drift apart again.
Verified with cargo check --workspace --all-targets, clippy --all-targets at
-D warnings, nightly fmt.
* refactor(shared): make VirtualId a real newtype and add ProviderId
`pub type VirtualId = u32` gave the reader a name and the compiler nothing. That
matters more than usual here: the same `BPlusTree<u32, XtreamPlaylistItem>` store
is keyed by a *virtual* id on the target path and a *provider* id on the input
path, chosen by a runtime `StorageKey` tag, and nothing stopped a lookup in one
key space using an id from the other.
#[repr(transparent)] #[serde(transparent)]
pub struct VirtualId(pub u32);
pub struct ProviderId(pub u32);
No `From<u32>`/`Into<u32>` on purpose: an implicit conversion would let a
provider id become a virtual id by inference, which is the confusion the types
exist to prevent. Crossing to a raw integer is spelled `new` and `get`.
On-disk compatibility was the gating question, since these are persisted B+Tree
keys and serialized struct fields. `backend/btree/src/codec.rs` gets a test
asserting a transparent newtype over u32 encodes byte-for-byte identically under
rmp_serde and cross-reads in both directions -- bytes written before the newtype
existed decode into it, and bytes it writes decode as a plain u32. Existing
databases are unaffected and there is no migration.
The type now flows through the playlist items (PlaylistItemHeader, M3u, Xtream,
Common), VirtualIdRecord and the whole TargetIdMapping id-index cluster
(disk_by_virtual_id, mem_by_uuid, mem_by_virtual_id, pending upserts,
find_virtual_ids, get_virtual_id_by_uuid, get_and_update_virtual_id), and the
metadata manager's provider->ids and uuid->id caches.
The compiler immediately found what the alias was hiding: the two `StorageKey`
arms in xtream_repository now have *incompatible types*, and
playlist_repository referred to one id-mapping store as both `BPlusTree<u32, _>`
and `BPlusTree<VirtualId, _>`.
Deliberate boundary: cross-boundary DTOs keep their u32 wire shape and unwrap
explicitly with `.get()` -- the Xtream-API-shaped documents, StreamInfo,
UiPlaylistItem, stream history, and the ffprobe/session interfaces. The B+Tree
stores themselves also stay u32-keyed for now: splitting them into VirtualId- and
ProviderId-keyed stores is item 23, and it needs per-site judgement about which
key space each of 24 call sites belongs to. Getting one wrong is a silent lookup
miss, so it is a focused follow-up rather than a rider on this commit.
Verified with cargo check --workspace --all-targets, clippy --all-targets at
-D warnings, nightly fmt, and 1,149 tests across btree, shared, repository and
metadata.
* refactor(repository): make the Xtream store key space a type, not a runtime tag
The Xtream playlist stores use one file layout and one value type for two
different id spaces: the target-side store is keyed by virtual id, the input-side
store by provider id. The only thing recording which was a `StorageKey` enum
matched *per item* inside the insert loop:
tree.insert(match storage_key {
StorageKey::VirtualId => item.virtual_id,
StorageKey::ProviderId => item.provider_id,
}, item);
`write_playlists_to_file` is now generic over the key with a `key_of` extractor,
so each call site names its own key space and the tree's type carries it:
write_playlists_to_file(.., |item| item.virtual_id, ..) // target
write_playlists_to_file(.., |item| ProviderId::new(item.provider_id), ..) // input
`StorageKey` is deleted. The per-item branch is gone -- the extractor is
monomorphized per call site -- and the two key spaces can no longer be swapped
by passing the wrong enum variant. Both keys are `#[serde(transparent)]` over
u32, so the on-disk encoding is identical either way, which the codec test in
backend/btree pins.
This is the write path only. The read-side store types are still
`BPlusTree<u32, XtreamPlaylistItem>`: typing those means classifying 24 call
sites as virtual- or provider-keyed, and misclassifying one is a silent lookup
miss rather than a compile error, so it wants its own focused pass with the
staging/publish paths read end to end.
Verified with cargo check --workspace --all-targets, clippy --all-targets at
-D warnings, nightly fmt, and 422 tests across repository and btree.
* feat(shared): open-world event ids for notifications
`MsgKind` is a closed enum, so adding one notification event kind meant
editing eight sites across three crates - and two of those sites failed
silently rather than at compile time: `discover_templates` hardcodes the
variant list, and Pushover has no template map at all.
Replace it as the extension point. An event is now identified by a dotted
`domain.event` string with a severity, subscriptions are glob patterns,
and adding an event is one `EventId` const plus one emit call.
The pattern grammar stays small enough to explain in a config comment:
`*` for everything, `recording.*` for a domain, `provider.*.expired` for a
single wildcard segment, and a leading `!` to exclude - so `["*",
"!system.info"]` reads the way it looks.
Every legacy `MsgKind` wire name stays a valid `notify_on` entry via
`LEGACY_ALIASES`, and the canonical recording ids produce exactly the
legacy template filenames (`recording.completed` ->
`telegram_recording_completed.templ`), so existing config and template
files keep working untouched.
`EventId` deserializes an unrecognised id to `registry::UNKNOWN` rather
than failing, so an outbox written by a newer build round-trips through an
older one instead of poisoning the whole file.
No behaviour change: nothing reads these types yet.
* feat(core): one notification envelope instead of a per-kind context
`TemplateContext` carries one `Option` field per message kind - `message`,
`stats`, `watch`, `processing`, `disk`, `recording`, `flat_stats` - which
is why adding a kind had to touch the renderer.
`NotificationEvent` is the shape every event fits: id, severity, timestamp,
instance, dedup key, title, body, and the typed payload serialized into
`fields`. The existing payload structs are unchanged and now travel inside
`fields`, so nothing about them had to move.
`title` and `body` are always populated. That is what lets Pushover, a
syslog channel or an email subject line render any event without a per-kind
match - the gap that had Pushover pushing raw `serde_json` dumps of watch
changes and playlist stats to phones. `body_for` gives watch changes and
processing stats a readable plain-text rendering for the first time, while
keeping the string and disk-alert output byte-identical to the old
`default_text_for`.
`from_content` lifts a legacy `MessageContent` into the envelope, so all six
existing emitters keep working untouched.
Timestamp is unix seconds rather than `DateTime<Utc>`: it matches the
outbox's existing `enqueued_at` representation and avoids pulling chrono's
serde feature into the workspace. `timestamp_rfc3339` renders it for
templates.
No behaviour change: nothing reads these types yet.
* feat(messaging): a NotificationChannel trait to open the channel set
Adding a channel meant editing ten sites across three crates: a
`MessagingChannel` variant, the `is_some()` chain in
`configured_channels`, the match in `send_message_to_channel`, a new
`send_*_message` fn, the hardcoded `tokio::join!`, a config field, a DTO
field, both `From` impls, `prepare`, and the template-discovery prefix.
This adds the abstraction that collapses those: one trait with a stable
`id()`, the operator's template lookup, a `wants()` routing hook, and
`send()`. The dispatcher never learns the channel's name.
`Delivery` replaces the old `Option<bool>`, which could not distinguish
"retry me" from "this URL is malformed and will fail identically forever".
That cost real attempts: a typo'd webhook burned all `max_attempts` with
exponential backoff before dead-lettering, and a `429` was retried straight
back into the rate limit it had just hit. `delivery_for_status` now
classifies once for every HTTP channel - `408`/`429`/`5xx` transient,
other `4xx` permanent - and `parse_retry_after` reads both legal header
forms so a provider-supplied delay is honoured.
A `Retry-After` HTTP-date already in the past clamps to zero ("retry now")
rather than being discarded, since the server is telling us the wait is
over.
Dispatch is dynamic on purpose: the channel set is an open world resolved
at config load and a send is bounded by a network round trip, so the
vtable hop is not measurable and static dispatch would reinstate the
closed-world match. Uses the boxed-future alias convention from
`tuliprox-processing`'s `SinkFuture` rather than adding `async-trait`.
No behaviour change: nothing implements the trait yet.
* perf(messaging): cache and precompile notification templates
`resolve_template` wrapped every template value in an `InputSource` and
called `download_text_content` - once per message, per channel. A `file://`
template was re-read from disk and an `http://` one re-fetched over the
network for every notification. Worse, nothing distinguished the three
cases, so an *inline* Handlebars string paid for a full download attempt
too, on every send.
Classify the source once (`Inline` / `File` / `Url`), then cache what is
actually resolved: local files revalidate on mtime so an edit applies
immediately, remote documents on a 5 minute TTL. Compiled templates are
kept in a registry keyed by the config value, so rendering is no longer a
re-parse.
When a remote template cannot be refreshed the cached copy is served
rather than silently degrading to the built-in text - the operator
configured a template for a reason, and a template-host blip should not
change what the notification looks like.
`validate` compile-checks a template body without sending, so config
validation can surface a malformed template at load instead of leaving a
per-send `error!` and a fallback that looks plausible.
`context_for` carries the new uniform `event.*` shape alongside every
legacy top-level key - `kind`, `message`, `stats`, `processing`, `disk`,
`watch` and the flattened first-input fields - so templates written against
the documented examples render identically. `kind` keeps its CamelCase
labels for the legacy ids for the same reason.
* refactor(messaging): dispatch through channels, promote the outbox
Replaces the closed-world dispatch with the trait and envelope added in the
previous commits, and moves the outbox out of the recording supervisor so
every notification gets durable retry.
Channels
--------
The four hand-written `send_*_message` functions, the `is_some()` chain in
`configured_channels`, the match in `send_message_to_channel` and the
hardcoded `tokio::join!` are gone. `channels::build` constructs whatever
the config declares and the dispatcher fans out over the trait, so adding
a channel is an impl plus a config field.
Pushover gains template support. It had none, so every notification took
the built-in text - which for watch changes and playlist stats was a raw
`serde_json` dump pushed to a phone. It also now sends `title` separately,
and maps event severity onto Pushover's own priority scale.
Sends are concurrent per channel and carry a 30s request timeout. The
shared client sets no request timeout at all, and the outbox awaited
channels in sequence, so one webhook host that accepted a connection and
never answered could stall every pending notification - including the
recording ones the outbox exists to protect.
Outbox
------
Moved to `tuliprox-messaging`, no longer gated on the recording config, and
started unconditionally at bind. Playlist stats, watch changes, disk alerts
and provider warnings previously called `send_message`, which fanned out
and discarded every outcome with `let _ =`; a transient 502 lost them
permanently. `send_event` is now a thin enqueue with a direct-send
fallback.
Entries key pending channels by stable string id rather than an enum
variant, so an outbox written by a build that knows a newer channel no
longer fails to deserialize and take the whole file - and with it every
pending notification - down with it. `notification_outbox.json` adopts any
entries left in `recording_notification_outbox.json` exactly once.
`Delivery::Permanent` dead-letters immediately instead of burning every
attempt on a request that will fail identically forever, and a provider's
`Retry-After` now wins over our own backoff rather than retrying straight
back into the rate limit.
Config
------
`notify_on` is a list of glob patterns; template maps are keyed by event id
wire name. Legacy `MsgKind` names still parse and are normalized to
canonical ids on load. Template discovery iterates the event registry
instead of a hardcoded eight-variant array - the site that silently made a
newly added kind undiscoverable - and still finds legacy filenames.
The frontend event picker is driven by the registry too, so an event added
in the backend appears in the UI without a frontend change.
The six pre-existing template tests are kept and now render through the new
pipeline, which is what proves the documented Discord and Telegram
templates still produce identical output.
* chore: lockfile for tokio-util and futures in tuliprox-messaging
* feat(messaging): per-channel routing, suppression and rate limiting
Routing was a single global `notify_on`: every enabled event went to every
configured channel, so "critical to Pushover, everything to Discord" could
not be expressed at all. Each channel now takes an optional `routing`
block:
telegram:
routing:
notify_on: ["provider.*", "system.disk.alert"]
min_severity: warn
quiet_hours: "23:00-07:00"
max_per_hour: 20
dedup_window_secs: 3600
An absent block inherits the global subscription, so existing configs are
unaffected.
Suppression by `dedup_key` generalizes the disk alert's
`repeat_interval_secs`, which lived in `sys_usage.rs` and was available to
nothing else - `provider.offline` needs exactly the same logic. The hourly
ceiling emits one "further notifications suppressed" audit line when it
trips and then goes quiet, so the silence is distinguishable from a
notifier that has died.
Quiet hours defer rather than drop. The outbox holds the entry until the
window closes, because an overnight outage nobody hears about afterwards is
worse than one that arrives late. An entry is only held while *every*
still-pending channel is asleep.
Two supporting fixes:
`channels::build` ran on every send, constructing a fresh `reqwest::Client`
each time - added in the previous commit and wrong. The channel set is now
cached and invalidated on reload, which is also what lets per-channel
suppression and rate-limit state survive between notifications.
Routing is boxed inside the channel DTOs. Four inline routing blocks made
`MessagingConfigDto` the largest `ConfigForm` variant at 857 bytes; the
`Option<Box<_>>` is niche-optimized and only allocates when routing is
actually configured.
Rate-limit windows are pure in `now`, so expiry is tested without sleeping
and without tests racing each other over shared global state.
* feat(app): bridge the event bus onto the notification pipeline
`EventMessage` already carries fourteen variants from thirteen emitters -
playlist updates, config changes, library scans, user connections, metadata
updates, recording changes - and every one of them reached the Web UI over
the websocket and nowhere else. The notification side had six emitters of
its own, and nothing connected the two.
One subscriber closes that gap, and every future `EventMessage` variant
comes along with it.
Three things keep it from being a firehose:
* The high-frequency variants map to `None`: progress ticks, download
deltas and periodic system info fire many times per operation and carry
nothing worth pushing to a phone. Their terminal counterparts are what
get through. The match is exhaustive, so a new variant is a compile
error rather than a silent firehose.
* Everything defaults to unsubscribed. An upgrade does not start messaging
anyone until `notify_on` asks for it.
* The broadcast channel has capacity 10, so `Lagged` is reported and
skipped rather than killing the bridge.
A partial playlist update maps to `completed` at `warn` rather than `info`,
so it does not read as a clean success. Config changes carry a per-file
dedup key, so a watcher that fires several times for one save produces one
notification.
The two genuinely chatty events - `user.connection.changed` and
`provider.connections.changed` - are registered and documented as high
frequency so subscribing to them is a deliberate act.
* feat(messaging): specific provider events, a test endpoint, and secret redaction
Three changes that together make the messaging config verifiable and safe to
expose.
Provider account events
-----------------------
The Xtream account-status, expiry-warning and expired messages all landed in
`MsgKind::Info`/`Error`, so subscribing to "my account is about to expire"
meant also receiving every processing error. They now emit
`provider.account.status_changed`, `.expiring` and `.expired` with a typed
payload.
All three carry a dedup key. They are re-evaluated on every playlist
refresh, so without one an account inside its final three days would notify
on every single update.
Test endpoint
-------------
`POST /api/v1/config/messaging/test` renders and optionally sends a chosen
event to a chosen channel, returning the per-channel outcome *and* the exact
rendered body. `preview: true` renders without sending, so a template can be
iterated without spamming a channel.
It deliberately bypasses `notify_on` and the suppression window: the
operator asked for this one explicitly, and a test that silently does
nothing because of a dedup window would be worse than useless.
Secret redaction
----------------
The config GET returned `config.yml` in full to any client holding
`ConfigRead` - including the Telegram bot token, the Pushover token and user
key, and any `Authorization` header on the REST channel. Those are now
masked on the way out.
`save_config_main` restores any secret the client echoes back still masked,
so a UI round-trip cannot overwrite a real token with the mask and silently
break the channel. A genuinely changed secret still writes through, an empty
secret is not replaced by a mask (an unconfigured channel must not look
configured), and non-credential REST headers keep their values.
* feat(messaging): five new channels, and HMAC signing for webhooks
The test of whether the channel trait actually opened the set. Each of
these is one impl plus one config field plus one line in the builder -
no dispatcher, outbox, renderer or template-discovery change.
* **ntfy** - self-hosted push, no account, no bot token. The natural
default for a homelab operator already running tuliprox in Docker.
ntfy headers must be ASCII and titles routinely carry emoji, so the
title is stripped rather than failing the send, with a fallback so it
never goes out empty.
* **Gotify** - same audience, same shape.
* **Slack** - not a Discord clone. Block Kit differs enough from Discord
embeds that reusing the Discord payload produces bad output, so it
builds header/section/context blocks. Slack has no severity field, so
non-info severities are marked in the header text where a human sees
them.
* **command** - runs a local program with the event JSON on stdin. The
escape hatch that means nobody waits for a channel to be added
upstream. Executed directly rather than through a shell, so there are
no quoting rules and no shell-injection surface from event content. A
missing binary is `Permanent` (retrying cannot help); a non-zero exit
or a timeout is `Retry`.
* **REST signing** - optional HMAC-SHA256 over `{timestamp}.{body}`,
sent as `X-Tuliprox-Signature`. The timestamp is inside the signed
payload, so a captured request cannot be replayed with a fresh header.
Verified against the RFC 4231 test vector.
Each severity maps onto the target's own priority scale rather than
being dropped, and the new secrets (ntfy token, Gotify token, REST
signing secret) join the existing ones in the redact/restore path.
`ConfigForm::Messaging` is boxed: eight channel configs made it dominate
the size of every other variant of that enum.
* docs(messaging): document the open-world event and channel model
Rewrites section 5 for what the messaging layer actually does now.
* The glob grammar for `notify_on`, with a note that every legacy event
name still works and is rewritten on the next save.
* A table of all 24 registered events with default severities.
* Per-channel `routing`: `notify_on`, `min_severity`, `quiet_hours`,
`max_per_hour`, `dedup_window_secs` - including that quiet hours defer
rather than drop, and that the hourly ceiling reports itself once so the
silence is not mistaken for a broken notifier.
* Delivery semantics: the outbox, per-channel retry, which statuses are
transient vs. permanent, and `Retry-After`.
* The four new channels (ntfy, Gotify, Slack, command), webhook HMAC
signing with the replay note, and secret masking in the Web UI.
* Templates: now supported on every channel including Pushover, keyed by
event id, resolved and compiled once rather than re-fetched per send,
with the uniform `event.*` context documented alongside the legacy keys.
* The test endpoint, so there is a feedback loop shorter than "save it and
wait for something to break".
The event table sits between generated-block markers and a test checks it
against the registry in both directions - a registered event missing from
the table, and a table row for an event that no longer exists. Verified
the test fails for both before restoring.
* fix(messaging): invalidate channel and template caches on config reload
The channel set and compiled templates are cached so a notification does
not rebuild every channel - and a fresh `reqwest::Client` with them - on
every send. Nothing invalidated those caches, so an edited bot token,
webhook URL or template would not take effect until a restart.
Also emits `config.reload_failed` alongside the existing `ServerError`, so
an operator can subscribe to "my config stopped loading" without taking
every server error with it.
* refactor(events): move EventMessage into shared
`EventMessage` lived in `tuliprox-session`, the crate that owns provider
allocation and the streaming-session runtime. That meant `metadata`, `dvr`
and `processing` all had to depend on the streaming runtime just to name an
event - a metadata refresh completing has nothing to do with which provider
a stream came from.
Every payload the enum carries was already a `shared::model` type, so the
move is mechanical: the taxonomy goes to `shared::model::event`, the bus
implementation stays in `session` next to the stream-meter registry it also
feeds.
`api::model` re-exports it from `shared` so the ~80 `crate::api::model::
EventMessage` call sites keep their path.
* refactor(events): emit through a static EventSink bound
There was no seam anywhere in the event path. `Arc<EventManager>` - the
concrete streaming-runtime bus - was baked into `MetadataUpdateCtx`,
`RecordingCtx`, and the playlist pipeline, which carried it as
`Option<Arc<EventManager>>` purely because tests have no bus to hand it.
That `Option` was then unwrapped at every one of its emit sites.
`shared::model::EventSink` is the seam: one method, `emit`, documented as
non-blocking because the bus is reached from the streaming data path.
It is a bound, never a trait object. The three context structs are generic
over their sink and monomorphise against the one they were built with, so
emitting stays a direct call. `NoopSink` is the absent case, and its `emit`
is an empty function - the pipeline's six `if let Some(events)` branches
collapse to unconditional calls that compile to nothing when that is the
instantiation, and `create_broadcast_callback` loses its noop arm.
`MetadataUpdateManager` stores its context and is itself held by `AppState`,
so it pins one instantiation (`BoundMetadataUpdateCtx`) rather than going
generic and dragging `AppState` with it. Functions that only read a context
stay generic.
`tuliprox-dvr` drops its dependency on `tuliprox-session` entirely - the
event bus was the only thing it wanted from the streaming runtime, and a
bound is not a dependency. 78 workspace edges, now 77.
`app_state_views!` grows a `ctx_field <- state_field` form for a context
that names a handle by the role it plays because it is written against a
bound rather than against `EventManager`.
* feat(events): give events an identity with EventKind
Subscribers each discriminated by exhaustively matching `EventMessage`: the
websocket mapped variants to permissions, the notification bridge computed
severity per variant, and the wire layer maps them to `ProtocolMessage`.
Adding a variant compiled cleanly while reaching none of them.
`EventKind` carries the payload-free identity, and everything that is a
property of the event rather than of one consumer hangs off it:
* `required_permission` - who may see this. Not a websocket concern: it
does not change with the transport, so `websocket_can_receive_runtime_
events` is now one line and cannot fall out of date.
* `severity` - on `EventMessage`, because it reads the payload: a playlist
update that failed is an error and one that succeeded is not. The bridge
now decides only which notification id to use.
* `is_high_frequency` - progress ticks, deltas, the periodic system-info
sample. A statement about rate, deliberately separate from the bridge's
notifiability decision, which also depends on whether a terminal
counterpart exists.
* `as_wire_name` / `from_wire_name` - stable strings for plugin
subscriptions and operator config, with the same
must-not-change-once-released contract a notification channel id carries.
`EventKind::ALL` fixes the bit order that the subscription mask uses.
No behaviour change: the permission mapping is the same table and the
notifiable set is untouched.
* perf(events): filtered subscriptions and cheap fat payloads
`get_event_channel` handed every subscriber all fourteen kinds and each one
filtered afterwards - after the broadcast channel had already cloned the
message for it. Two costs, addressed separately.
Filtering: `EventKindMask` is a one-word set over `EventKind`, and
`subscribe_filtered` returns a `FilteredEventReceiver` that drops what the
subscriber did not ask for before waking it. A concrete type, not a boxed
stream: the call site is a `select!` arm awaiting `recv()`, and a virtual
call per event buys nothing. `Lagged` still propagates - a gap in events the
subscriber did not want is still a gap in the ones it did.
The notification bridge is the first user. It handles ten kinds and dropped
four, and those four are the bulk of the traffic during a playlist refresh,
so it now never wakes for them. A test asserts the mask and the `None` arms
of `to_notification` agree, and that the sample list covers every
`EventKind` - so a variant added later cannot quietly fall out of either.
Payloads: `SystemInfoUpdate` and `DownloadsUpdate` carried `SystemInfo` and
`DownloadsResponse` by value, which is what `large_enum_variant` was
allowed for. Behind `Arc` they cost a refcount bump per receiver instead of
a deep copy, and the `allow` is gone. Only the websocket needs the value
itself; `unwrap_or_clone` there means the last subscriber standing pays
nothing. The frontend mirror of this enum has held both behind `Rc` all
along.
* refactor(session): split the stream-meter registry out of EventManager
`EventManager` was two components sharing a name: a pub/sub bus for fourteen
event kinds, and a metering subsystem with its own broadcast channel, its
own subscriber counter, its own background sampler task and three maps. No
event subscriber ever touched the second half, and no meter call site
touched the first.
`StreamMeterRegistry` is that second half. `EventManager` owns one and
forwards the meter methods, so the composition root still builds a single
handle and streams need not know metering is separate; `meters()` exposes
the registry for anything that does.
The three maps become one. `meters`, `meter_to_clients` and
`client_to_meter` had an invariant re-established by hand in four methods -
retain from the vec, remove the entry if it emptied, drop the index - and
`register_meter_client` spelled it differently from the other three. They
are now `HashMap<u32, MeterSlot>` plus the client index, with `detach_client`
and `remove_meter` as the only two mutators. The slot's handle stays
optional because the two halves genuinely arrive in either order: a client
can be assigned to a meter before the stream owning it registers.
`read_meter_qos` returns `Option<MeterQos>` instead of
`(Option<u64>, Option<u64>)`, where "both `None`" doubled as "this meter is
shared, ask no further" - a convention that lived only in a doc comment.
The sampler still declines to start outside a tokio runtime, but now says
so at debug rather than returning in silence.
All six meter tests pass unchanged, which is the point: this is a
rearrangement, not a behaviour change.
* feat(events): configurable bus capacity, and counters for what it drops
The bus was `broadcast::channel(10)` for everything. Ten is routinely
outrun: a playlist refresh emits progress ticks in a loop, and any
subscriber that awaits I/O per event falls behind within one target. The
evidence was already in the tree - the notification bridge carries a
dedicated `Lagged` arm with a comment about it, and the websocket has an
entire `ResyncStatus` recovery path that exists for no other reason.
Capacity is now `event_channel_capacity`, default 256, clamped to at least
1. `EventManager::new()` keeps the default for tests and early startup.
`EventBusStats` makes the drops visible. `send_event` returned a `bool`
that ~80 call sites discarded with `let _ =`, so "nobody received this" and
"nobody was listening" were equally invisible. It now counts emissions per
kind, emissions with no subscriber, and - reported by the subscribers,
since a broadcast channel drops for the receiver and not the sender - the
size of every gap a subscriber was told about. The notification bridge and
the websocket both report theirs.
This is also where the plugin system's promised drop counters come from,
rather than a second set of counters on a second queue.
* refactor(events): single-source the notification and wire mappings
Four parallel taxonomies described the same events: `EventMessage` on the
bus, `ProtocolMessage` on the wire, the frontend's own enum, and the
notification registry. Two of the three translation layers were hand-written
tables that had to be edited in lockstep, with nothing checking they agreed.
The notification id moves onto the event. `EventMessage::notification_id`
returns the registry id, or `None` for the kinds that are not notifiable -
one decision, made once. It lives on `EventMessage` rather than `EventKind`
because two of them read the payload: a playlist update that failed is a
different notification from one that succeeded, not merely a more severe
one. The bridge now decides only wording and attachments.
The wire mapping becomes a pure function. `handle_event_message` was a
hundred lines of `match` nested three deep inside the socket loop, where a
kind reaching no arm looked exactly like a kind deliberately ignored.
`to_protocol_message` is testable on its own, and the socket loop is now
guard, special case, send.
Both new tests iterate `EventKind::ALL` and assert the sample list covers
it, so a variant added later fails the tests instead of silently reaching
nobody - which is the failure mode all of this exists to prevent.
* feat(events): the seam the plugin host subscribes through
The plugin plan specified a second, independent event bus: its own bounded
mpsc, its own emit call sites in `process_sources`, the active-stream
tracker and user CRUD, and its own drop counters. Three of those four
emitters already publish to `EventManager`. Building the second one would
have meant two emit sites per event and two taxonomies to keep in step.
What the host actually needs from the existing bus, added here:
* `EventKindMask::from_wire_names` - a manifest's `events.*` list becomes a
subscription mask, with unknown names returned rather than silently
narrowing what the plugin asked for. An empty mask means no subscriber
task at all.
* `EventMessage::payload` - the JSON a plugin receives, defined on the event
so a new variant arrives with its payload shape decided instead of the
host growing a second match over the taxonomy. Never fails: an
unserialisable payload degrades to null rather than dropping the event.
* `EventKindMask::kinds`, for reporting what a subscription resolved to.
`plugin-system-plan.md` (untracked) is revised to match: the plugin queue
sits behind the bus rather than beside it, keeping its backpressure
isolation while making "emitters never block" structural - `broadcast::send`
cannot block - instead of a convention every emit site has to honour.
Tests assert the bit assignment is unique, wire names are unique and round
trip, and the taxonomy still fits the mask's `u32`.
Deliberately not done: `stream-start` / `stream-stop`, `user-created` and
`probe-result` still have no `EventMessage` variant. They belong in the
shared taxonomy rather than a plugin-private one, but stream start/stop
sits on the streaming hot path and should not gain a per-stream emission
speculatively, ahead of a consumer that needs it.
* feat(events): typed emit outcomes and nudge coalescing
Two shapes the messaging crate had already worked out for notification
channels, and the event bus lacked.
Typed outcomes. `send_event` returned a `bool` that conflated "nothing was
listening" with "this failed", told the caller nothing useful either way,
and was discarded by ~80 call sites. `EmitOutcome` distinguishes
`Delivered { receivers }`, `NoSubscribers` and `Coalesced` - the same
distinction `Delivery` draws for channels. Not `#[must_use]`: an emitter
genuinely may ignore all three, and pretending otherwise would just spread
`let _ =` further.
Coalescing. `rate_limit::admit` throttles notifications; nothing throttled
the bus. `RecordingChanged` is emitted from six routes, twice back-to-back
where deleting a recording also changes the rules, and once per item in a
bulk operation - each one making every Web UI session re-fetch the same
snapshot.
`EventKind::is_coalescable` marks the kinds where N occurrences and one are
indistinguishable to every consumer: payload-free nudges that everyone
answers by re-reading current state. Only the two recording nudges qualify.
An event carrying a payload is never coalescable however repetitive, because
a dropped progress tick loses the message it carried - the tests assert
exactly that, and that the two nudges do not suppress each other.
The window is 250ms and is measured from the last admitted send, so a
sustained stream is throttled to one per window rather than one per burst,
and a later user action always produces a visible refresh.
* feat(events): snapshots, an audit ring, graceful shutdown
The remaining smaller fixes from the event-manager review.
Latched snapshots. `EventKind::is_latched` marks the kinds that describe
current state rather than an occurrence - the last `SystemInfo` sample *is*
the system info. The bus retains the newest of each, and `snapshot()` hands
them to a session that connects between samples, which used to see empty
panels until the next one arrived up to three seconds later. Occurrences are
never latched: replaying "a playlist update finished" to a session that was
not there would be a lie, and the tests say so.
A recent-event ring. 256 entries, each with its kind, monotonic uptime and
outcome - including suppressed ones, since "it fired but was coalesced" is
exactly what the ring is consulted to find out. Exposed with the bus
counters at `GET /api/v1/events/stats`, behind `system.read` like `/status`.
"Why did my notification not fire?" was otherwise unanswerable without a
debug build.
Graceful shutdown. `Drop` cancelled the meter sampler but cannot await, so a
stream still running at shutdown lost its last window's bytes -
`flush_and_unregister_meter` already fixed that for one stream ending
between ticks. `EventManager::shutdown()` does it for the whole registry,
and runs after the connection manager so the final batch reports what the
streams actually transferred.
`send_provider_event` and `send_system_info` are gone. They wrapped
`send_event` only to log on failure, which `EventBusStats` now records
centrally - two calling conventions for one bus was one too many.
Also documents the new endpoint in the REST cookbook.
* fix(processing): pin the pipeline test context to NoopSink
`processing_context()` was made generic along with the pipeline it builds,
which left every call site unable to infer the sink type. These tests
exercise the pipeline, not the bus, so the helper names `NoopSink` and the
call sites say nothing.
Neither `cargo build --workspace` nor `cargo clippy --workspace` compiles
test code, so this only surfaced on the first full test run.
Also carries the lockfile change from `tuliprox-dvr` dropping its dependency
on `tuliprox-session`.
* refactor(messaging): static dispatch, no trait objects and no boxing
The notification layer used `Vec<Arc<dyn NotificationChannel>>` and a
`Pin<Box<dyn Future>>` per send. Both are gone.
`NotificationChannel::send` now returns `impl Future<Output = Delivery> +
Send` instead of a boxed future, which makes the trait deliberately *not*
object safe - there is no way to construct a trait object from it, so the
old shape cannot come back by accident.
`Channel` is an enum over the eight implementations. Every call goes
through a match the compiler turns into a direct call, so dispatch is
monomorphized and the send path allocates nothing. Adding a channel is now
a module, an enum variant, a config field and a line in `build` - one site
more than the trait-object version, and still far from the ten the original
closed-world dispatch cost.
The two `Box`es in the config types are gone as well:
* `ChannelRoutingDto` is stored inline again on all eight channel configs.
* `ConfigForm::Messaging` carries `MessagingConfigDto` directly.
Both were added only to silence `clippy::large_enum_variant`. That lint is
now allowed explicitly on `ConfigForm` with the reason recorded: the
variant is moved once per form submit and never in a hot path, so the
indirection bought nothing real.
The only remaining `dyn` in the crate is Handlebars' own `register_helper`
signature, which is pre-existing and part of that library's API.
No behaviour change - same 46 messaging tests, same delivery semantics.
* feat(events): put the notification-only lifecycle events on the bus
Six emit sites called `tuliprox_messaging` directly instead of publishing to
`EventManager`. Everything they emitted reached operators by mail and nothing
else: not the websocket, not the notification bridge, and - once a plugin
host subscribes to the bus - not plugins either. An event was
plugin-visible only by accident of which route its emitter happened to take.
Nine kinds move onto the bus, each keeping the registry id it already had:
* `DiskAlert` - and the *subscription* check in front of it is gone. The
thresholds still come from `messaging.disk_alert`, but gating emission on
someone being subscribed by mail meant a plugin watching for disk
pressure saw nothing unless an operator happened to want the same event.
The notification layer already drops unsubscribed events.
* `ConfigReloadFailed` - the typed counterpart to the `ServerError` string
the Web UI toasts. Both are emitted: the comment at that call site already
argued an operator should be able to subscribe to "my config stopped
loading" without taking every server error, and that argument applies to
plugins, which only ever saw the string.
* `PlaylistWatchChanged`
* `RecordingStarted` / `Completed` / `Failed`
* `ProviderAccountStatus` / `Expiring` / `Expired`
Recording lifecycle is published *alongside* its existing delivery, not
instead of it. That path is at-most-once: a durable marker is persisted
inside the queue-mutation boundary before delivery, and the outbox retries
per channel. A broadcast bus drops for a lagging subscriber, so routing it
through here would let a recording be marked delivered and then never sent.
The bridge ignores the kind for exactly that reason; `download_api` still
owns operator delivery.
`WatchChanges` and `RecordingLifecycleMessage` move to `shared` for the same
reason `EventMessage` did - an event payload has to be nameable by every
emitter - with `ProviderAccountEvent` and `ConfigReloadFailure` added there.
`tuliprox-core` re-exports the two that moved, so `MessageContent` and its
call sites are untouched. The bridge builds the three that already had a
`MessageContent` shape via `from_content`, so the templates that render them
see exactly the fields they saw before.
Severity is no longer a second table: it comes from the registered event's
descriptor, with one override for a partial playlist refresh, which shares
an id with a clean one but is not a clean success.
`tuliprox-iptv` drops its dependency on `tuliprox-messaging` - emitting an
event is not knowing how it gets delivered. 77 workspace edges, now 76.
* feat(events): carry the run summary on the playlist-update event
`PlaylistUpdate` carried the outcome enum alone, so "the refresh finished"
reached the bus but what it actually did did not. The run summary went
somewhere else entirely: straight to the notification layer as a second
message, built from `MessageContent::event_stats`.
Both messages resolve to `playlist.update.completed`, so a successful
refresh with statistics notified **twice** - once with the stats and once
with the bare outcome - and neither carried the other's content. A
subscriber on the bus saw an outcome with no detail, and the plugin plan's
`refresh-complete`, specified as "run summary as payload", had nothing to
read.
`PlaylistUpdateSummary` folds them: outcome, per-source statistics, and the
aggregated error text, emitted once at the end of `process_sources`. The
notification bridge renders it through `ProcessingStats`, so the "Stats" and
"Error" templates - which read `fields.stats` - see exactly the shape the
separate message used to hand them, while the id and severity come from the
event, because the pipeline's own `PlaylistUpdateState` is a better answer
than re-deriving the outcome from which fields happen to be populated.
The websocket frame is unchanged: it carries `summary.state`, so the Web UI
sees what it always did.
`tuliprox-processing` no longer sends notifications directly. The four
timeout and panic paths use `PlaylistUpdateSummary::state_only`, having no
statistics to report.
`SourceStats` and friends gain `PartialEq`, which `EventMessage` requires of
everything it carries.
* chore(processing): drop the now-unused messaging dependency
`tuliprox-processing` published its last notification directly when the
playlist run summary moved onto the bus. Emitting an event is not knowing
how it is delivered, so the edge goes. 76 workspace edges, now 75.
* chore: lockfile for the dropped iptv and processing messaging edges
* feat(events): user account lifecycle and failed stream probes
Two emitters that changed state and told nobody. Creating, editing or
deleting an API-proxy user wrote api_proxy.yml, swapped the live config
and returned 200; ffprobe failures went to the item store and a warn
line. Neither could be notified on, and neither had an audit trail.
Both follow the RecordingLifecycle / ProviderAccount shape: one payload
with a state, several EventKinds, so a subscriber can ask for deletions
or 404s alone.
UserLifecycleEvent carries username, target and state - not the
password or token. That record reaches Telegram, webhooks and shell
commands, several of which log; the secret is absent from the type
rather than redacted at each render site.
StreamProbeFailure has no success counterpart: a metadata run probes
every unknown stream, so success would fire thousands of times per
refresh. It is published from prepare_generic_stream_metadata rather
than from the manager because that is the only place the reason still
exists - both outcome enums collapse NotFound, Other and Cancelled into
one ProbeFailed. Cancelled is deliberately not published: it is this
server shutting down, not a statement about the stream. The URL is
sanitized at the emit site, since a resolved provider URL carries
account credentials.
Notifications for both are deduplicated - per account+state, and per
*input* for probes, so a provider outage notifies once instead of once
per channel behind it.
Also widens EventKindMask from u32 to u64. The taxonomy is at 27 of 32
bits; the cheap time to widen is before operators have subscription
lists to migrate.
And repairs every_event_kind_is_either_wire_mapped_or_deliberately_not,
which has been failing on DiskAlert since the notification-only
lifecycle events joined the bus: twelve kinds returned None from
to_protocol_message while HANDLED_ELSEWHERE listed three. The two
reasons a kind produces no frame are now two lists, because they are
not the same fact.
* refactor(auth): roles as a static bitset, one is_admin, one JWT decode
`Claims::roles` was a `Vec<String>`: an allocation per mint, a string
comparison per check, and - because five call sites compared with `==`
while a sixth used `eq_ignore_ascii_case` - two different answers to the
same question depending on where you landed.
Roles are now a `RoleSet`, built by the same `create_bitset!` macro that
backs `PermissionSet`. A role check is a `test` instruction on a `u8`.
The JWT wire format is unchanged: `role_names` serialises the set back to
the legacy `["ADMIN"]` string array, so tokens minted before this change
still verify and clients that read the payload see what they always saw.
Unknown role names deserialise to no bit rather than failing the parse,
so a token from a newer build fails closed.
`Claims::is_admin` / `is_api_user` replace the six open-coded checks.
Case-insensitivity is now uniform - the parse accepts either case, which
is the superset that cannot regress an existing token.
`validate_request` took a `fn(&str, &[u8]) -> bool` that re-decoded the
token from scratch, so every authenticated request paid for two JWT
decodes and the API-user path paid for three. It now takes a
`fn(&Claims) -> bool` - still a plain fn pointer, still static dispatch -
and reads the claims it already decoded. `verify_token_admin` and
`verify_token_api_user` are gone with it.
* refactor(auth): one typed rejection for every auth extractor, and 401 means 401
`AuthBasic`, `AuthBearer` and `Fingerprint` each declared
`type Rejection = (StatusCode, &'static str)` - the same alias, defined
twice, once in `tuliprox-auth` and once beside `Fingerprint` in
`tuliprox-core`. The tuple carried no structure, so each arm picked a
status by hand, and they all picked the same wrong one: a *missing*
`Authorization` header answered `403 Forbidden`. That tells a client "you
are authenticated and still may not do this" when the truth is "you did
not authenticate at all", and the two are not distinguishable from the
outside. No `WWW-Authenticate` challenge was sent either, so a 401 from
this server was never a well-formed 401.
`AuthRejection` replaces both aliases: an enum in `tuliprox-core` with a
`status()`, a `message()`, and an `IntoResponse` that attaches the
`WWW-Authenticate` challenge for the scheme the extractor wanted. Missing,
malformed and wrong-scheme headers are now 401; only an unresolvable peer
address stays a 400. `auth_middleware::rejection_for` gained the same
challenge header on its 401 path.
* fix(auth): validate the issuer, bound the token lifetime, honour pwd_version
Three holes in the token lifecycle, all of them the same shape - a field
that was written into every token and then never read back.
**`iss`**: `Validation::new(Algorithm::HS256)` checks `exp` and nothing
else, so the configured issuer was decoration. `verify_token` now takes
the expected issuer and sets it on the validation. The WebSocket paths
carried a bare `Vec<u8>` secret across task boundaries, which is exactly
why they could not check an issuer they never received - they now carry a
`TokenVerifier` that holds both.
**`token_ttl_mins`**: a configured `0` meant "expire in 100 years", which
is a permanent bearer credential written as if it were a configuration
convenience. `0` now means the 24-hour default, anything above the 30-day
ceiling is clamped, and both log why.
**`pwd_version`**: minted into every web token, checked in exactly one
place - the refresh endpoint - so changing a password invalidated nothing
on any guarded route. Together with the 100-year TTL above, a leaked token
was a permanent credential. `validate_password_version` is the check, and
it rejects `pwd_version == 0` rather than treating it as "skip", which is
how the refresh endpoint's version of this check could be bypassed.
`AuthError::PasswordChanged` is deliberately not refresh-required: a
refresh applies the same check, so the client must sign in again.
Enforcement on the request path lands in the next commit.
* fix(auth): scope-bind access tokens, wipe prompted passwords, drop dead file
**Access tokens** signed `(timestamp, ttl)` and nothing else, so any valid
token was valid at every place a token was accepted - one token minted for
any purpose opened all of them. The scope is now mixed into the keyed hash,
so a token minted for one capability does not verify against another. The
token string format is unchanged; only the signed payload grew.
The scope is a compile-time constant, never caller-supplied: both sides of
a handshake have to agree on the exact bytes or the signature fails, which
is what keeps mint and verify from drifting.
This binds to a capability, not to a resource. The internal web player
mints one token that travels the whole chain - webplayer or recording
entry point, the xtream handler they delegate to, and the
custom-video-stream fallback the stream layer redirects into - and those
sites do not share a target or a virtual id to bind against. Per-resource
binding needs that redirect chain traced against a running server; the
scope parameter is where it goes when it is.
**Prompted passwords** were left sitting in two `String`s after
`generate_password` returned. `UserCredential::zeroize` already applies
this discipline to a password that arrives over HTTP; one typed at a
terminal now gets the same treatment.
**`backend/auth/src/user.rs`** was never declared in `lib.rs`. It was a
dead duplicate of the `UserCredential` in `shared::model::auth::user`.
* fix(auth): enforce pwd_version and live permissions on every guarded route
Two checks existed and neither ran where it mattered.
**Password version.** `pwd_version` was consulted only by the refresh
endpoint, so changing a password invalidated nothing on any guarded route -
a token minted against the old password kept working until it expired.
`authenticate` now applies `validate_password_version` to every request
whose principal is a web user. Proxy API users authenticate against
`api_proxy.yml` and carry no password version, so there is nothing to
compare and the check is skipped for them rather than failing them.
The refresh endpoint's own copy read `pwd_version != 0 && pwd_version !=
current`, so a token carrying `0` skipped the check. It now uses the same
strict validator as everything else.
**Permission revocation.** `require_permission` read `claims.permissions`,
a snapshot from mint time, so revoking a group permission had no effect
until the token expired. The effective set is now the intersection of the
claim with what the live config grants: a revocation takes effect on the
next request, while a new *grant* still needs a refresh, because a token
must never end up with more authority than it was issued with. A principal
the web-auth config has never heard of has no live set and keeps its claim.
Three permission paths had drifted apart and now share one implementation:
`rbac_api` had a hand-rolled check that verified the signature and read the
claim directly - no schema gate, no subject gate, no password version, no
live intersection; `v1_api_config::decode_permissions` filtered config
output off the raw claim; and `get_username_from_auth_header` decoded with
a bare `Validation::new`, which checks `exp` and nothing else.
**`permission_layer!`** now expands to `require_permission::<{P as u32}>` -
a bare `fn` item with the permission fixed at monomorphisation, rather than
a closure capturing a runtime `Permission`. `Permission::from_repr` (new on
`create_bitset!`) recovers the variant on the other side. The 15 call sites
are unchanged.
**`AuthorizedClaims<const P>`** is the same requirement in a handler
signature, handing the handler the claims the layer already verified -
they used to be dropped, so handlers behind a layer decoded the token
again. Its rejection is a two-word `Copy` enum, not a rendered `Response`.
* refactor(processing): exec_processing takes a run, not twelve arguments
Twelve positional parameters, seven of them `Option<_>`. A call site was a
wall of `None`s and `Some(..)`s where the reader had to count commas to
work out which knob was being set, and the compiler could not catch two
same-typed arguments swapped. The CLI path was literally
`exec_processing(&client, cfg, targets, NoopSink, None, None, None, None,
None, None, None, None)`.
`ProcessingRun` takes the four that are always present as constructor
arguments and names the rest. The CLI path is now
`ProcessingRun::new(client, cfg, targets, NoopSink)`. Setters take
`impl Into<Option<T>>`, so a site that already holds an `Option` passes it
through unchanged and one that holds a value does not have to wrap it.
`client` moved from `&reqwest::Client` to an owned clone - the client is an
`Arc` internally, so this is a refcount bump, and it drops a lifetime
parameter from the struct.
* perf(events): drop the last four `as Arc<dyn EventSink>` casts
`exec_processing` has been generic over `E: EventSink` for a while, but
every caller handed it `Arc::clone(&event_manager) as Arc<dyn EventSink>`.
The blanket `impl<T: EventSink + ?Sized> EventSink for Arc<T>` made that
compile, so it looked done - but the monomorphisation was against
`Arc<dyn EventSink>`, and every `emit` on the update path still went
through a vtable.
Deleting the four casts is the whole change. `EventManager` is now the
concrete `E`, and the emit sites in the playlist pipeline are direct calls.
* perf(processing): the update bootstrap is a trait, not two layers of boxing
`PlaylistUpdateBootstrap` was
`Arc<dyn Fn() -> Pin<Box<dyn Future<Output = ()> + Send>> + Send + Sync>`:
an erased closure returning an erased, heap-allocated future, for work that
runs exactly once per update. Every call site had to spell out both
coercions - an `Arc::new`, a `Box::pin`, and two `as` casts - around a
three-line closure.
`UpdateBootstrap` is one trait with an RPITIT future and a blanket impl for
`Fn() -> impl Future`. `ProcessingRun` carries it as a type parameter, so
the call sites are now just the closure:
.with_bootstrap({
let state = Arc::clone(&app_state);
move || {
let state = Arc::clone(&state);
async move { sync_panel_api_exp_dates(&state).await }
}
})
`NoBootstrap` - the type parameter for a run without one - is a function
pointer rather than a unit struct, so it satisfies the same blanket `Fn`
impl and needs no second impl to conflict with it. No value of it is ever
constructed.
`with_bootstrap` changes the run's type parameter, so it rebuilds the
struct rather than mutating it; the other setters are unchanged.
* perf(processing): MetadataUpdateSink loses the vtable and the boxed futures
The pipeline held the metadata worker as `Arc<dyn MetadataUpdateSink>` and
the trait's two async methods returned
`SinkFuture<'_, T> = Pin<Box<dyn Future + Send>>`. That is a heap
allocation per `prepare_enqueue_state` - once per input - and a vtable hop
on `should_skip_enqueue`, which runs once per playlist item. There is
exactly one implementor, so nothing ever needed the erasure.
The futures are returned by value (RPITIT) and `PlaylistProcessingContext`
carries the sink as a type parameter. `NoopMetadataSink` names the
parameter for a run without a worker - the same role `NoopSink` plays for
events - and its methods are correct no-ops rather than `unreachable!`,
since the whole point of the type is to be absent.
Making the trait non-dyn-compatible is what forced the last four
`as Arc<dyn MetadataUpdateSink>` casts out of the composition root; the
compiler will not let them come back.
`PlaylistProcessingContext`'s `Clone` is written out rather than derived:
the derive would demand `M: Clone`, but the sink is behind an `Arc` and is
cloneable whatever `M` is.
* feat(auth): stable subject ids come from the identity registry
`create_jwt_web_user` and `create_jwt_api_user` synthesised the subject as
`format!("web:{username}")` / `format!("api:{username}")`, with a TODO
saying the registry would provide it. The registry was fully built - with
persistence, bootstrap, fail-closed recovery and an explicit `rename` that
preserves the id - and never wired into the server, so the TODO was the
live behaviour: the subject was a function of the display name, and
renaming a user reassigned every recording the old subject owned to a
principal that does not exist.
`IdentityRegistry` now lives on `AppState`, bootstrapped from the storage
dir with the current web and API principals. The login and refresh paths
resolve the subject through it - `register` is get-or-create, so a user
bootstrap already synced keeps their id and one added since gets a fresh
one. `register_api_user` and `lookup_api_by_username` are new: the API
namespace had a bootstrap sync path but no way in for a principal that
appears at runtime.
A corrupt registry refuses to start rather than inventing replacement ids,
which is the whole reason the registry's fail-closed path exists. The
pre-scan that would hand bootstrap the subject ids already referenced by
persisted recordings is still unwired, so a *missing* registry alongside
existing recordings initialises fresh; a corrupt one - the case a
half-written file actually produces - fails closed regardless.
* feat(auth): back off repeated failed sign-ins
`/auth/token` verified an argon2 hash, answered 401 and forgot. Nothing
counted how often that happened, so a password list could be worked against
it as fast as the hash function allows, for as long as the attacker liked.
The reverse-proxy rate limiter is opt-in, disabled by default, and applies
one blanket budget to every route - it is not a credential-stuffing control.
`LoginThrottle` tracks consecutive failures on two dimensions, because
either alone is defeatable: by client address, which stops one host
grinding a list but not an attacker with an address pool; and by username,
which stops a distributed attack converging on one account. The username
dimension is a denial-of-service lever if handled carelessly - anyone who
knows a username could lock its owner out - so its block is short (15
minutes at the ceiling) and a correct password clears it immediately.
Three free attempts, then 2s doubling to the ceiling, and a 429 carrying
`Retry-After` so a well-behaved client backs off rather than hammering. The
check runs *before* the argon2 verify: an attacker who can still force the
hash on every attempt has not been slowed down.
Usernames are canonicalised the way the rest of the auth path compares
them, so `Alice` and `alice` share one budget against one account.
The address dimension is only as trustworthy as the address, which this
server still takes from `X-Forwarded-For` with no trusted-proxy allowlist.
Until that is fixed the username dimension is the one doing the work.
* feat(auth): authentication decisions reach the event bus
Sign-ins, rejected sign-ins and permission denials went to `warn!` and
`debug!` and nowhere else. Nothing that subscribes to the bus - a
notification channel, a plugin, an audit sink - could see any of them, so
the events that matter most for spotting an intrusion were exactly the ones
the bus never carried.
`EventMessage::AuthAudit` carries one decision. Following
`UserLifecycleEvent`: one payload, four `EventKind`s, so a subscriber can
ask for the failures without being woken by every successful sign-in. Each
kind is registered with a severity and a description, so it shows up in the
notification config like every other event.
The record holds a username, an address and an outcome. The password and
the token are not in the type at all rather than being redacted at each
site that renders one - these records reach Telegram, webhooks and shell
commands, several of which log on their own. Notifications dedupe per
principal, address and outcome, so a password-guessing run is one piece of
news rather than one per attempt.
`required_permission` is `UserRead`, not the `SystemRead` the other
operational events take: who signed in is the same question as who may read
the user list, and that is the narrower answer. They are not pushed to the
Web UI socket - no panel renders them, and every connected admin does not
need every sign-in.
`token` split into `web_user_sign_in` and `api_user_sign_in` on the way
past. The two branches stay sequential and short-circuiting: the API branch
compares credentials and can allocate a persisted subject id, neither of
which a successful web sign-in should trigger. `SignInAttempt::Refused`
distinguishes "these are not credentials for this branch, try the next" from
a decided answer, which is what keeps a `ui_enabled: false` API user from
falling through to the generic 401.
* feat(auth): tokens can be revoked
The tokens this server mints are stateless JWTs: once issued, nothing could
take one back. A leaked token stayed valid until it expired, and there was
no way to end a session, sign a principal out of every device, or respond
to a compromise short of rotating the signing secret - which kills every
session for every principal at once, is not reversible, and leaves no
record of who did it or why.
`TokenRevocations` is a revocation *watermark* rather than a deny-list of
individual tokens: per subject, "everything issued at or before this
instant is dead", plus one global watermark for the same statement across
all subjects. Two things follow from that shape. It is bounded - a
deny-list grows with every revoked token and needs an expiry sweep; a
watermark is one timestamp per principal. And it revokes sessions a
deny-list cannot name: "sign out everywhere" and "revoke everything issued
before the breach" have no list of token ids behind them. The cost is
precision - it cannot revoke one session and spare another issued in the
same second - which is the right trade for what it is for.
`iat` is compared with `<=`, not `<`: it has one-second resolution, so a
token minted in the same second as the revocation would otherwise survive
it. Over-revoking by up to a second is the safe direction.
Revocations are persisted, because the tokens they revoke outlive the
process. An in-memory revocation would be a security control that quietly
stops applying at the next restart. A file that will not parse is an error
at startup rather than an empty store - reading it as "nothing is revoked"
would silently reinstate every revoked session.
`POST /auth/revoke/{username}` ends one principal's sessions across both
identity namespaces; `POST /auth/revoke` ends everyone's. Both state their
`UserWrite` requirement through `AuthorizedClaims` in the handler
signature, because the `/auth` routes are mounted before any state exists
to build a router layer from - which is the case that extractor was for.
The refresh endpoint checks revocation too: a revoked token that could be
exchanged for a fresh one would make revocation a formality.
* refactor(iptv): page arithmetic has one home
A Stalker catalog page is the last one when it came back empty, when it is
shorter than the advertised page size, when the accumulated row count reaches
`total_items`, or when `max_page` says so. That rule was written out four
times - in the accumulating paginator, in `parse_item_catalog_page`, in
`parse_series_catalog_page` and in the `apply_page_limit` guard - and the two
`parse_*_catalog_page` copies had already drifted from the loop's version in
how they measured progress against `total_items`.
`PageMeta::is_terminal` is now the single copy, with the progress measurement
made explicit: accumulating callers pass their real running count, single-page
callers pass `PageMeta::fetched_estimate`, which is deliberately zero when the
portal advertises no page size so the `total_items` arm stays inert rather
than truncating the catalog at page one.
The two per-row-type page parsers collapse into one generic walk as well. They
differed only in `T`, and both hand-rolled the same four-shape row collection
(bare array, `data` array, `data` object, id-keyed envelope).
Behaviour is unchanged. Not verified by tests - checked with cargo check and
nightly clippy on the crate.
* feat(iptv): catalogs can stream instead of accumulating
`get_live_streams` and friends buffered an entire provider catalog into a
`Vec` before the caller saw a single row, while the EPG path next door already
had the right shape - `stream_bulk_epg` hands over batches as they arrive.
Catalogs now have the same option.
The interesting part is not the callback but what it costs. The client tries
several endpoint candidates in turn, and a failure part-way through pagination
abandons that candidate and restarts on the next one. That retry is only sound
while nothing has left the client, so the sink decides: `CollectSink` holds
everything and can restart freely - the historical behaviour, and the reason a
truncated catalog is never returned as `Ok` - while `BatchSink` reports that it
can no longer restart once a page has been released, and the driver then
returns the failure as-is. The streaming methods document that an `Err` makes
the delivered batches an incomplete prefix.
`get_*_paginated` are now thin adapters over the same driver rather than a
second copy of the pagination loop.
Not verified by tests - checked with cargo check and nightly clippy.
* refactor(iptv): expiry rules take the instant, they no longer read the clock
Session staleness, cookie `Max-Age`, and the Xtream account-expiry warning were
all pure functions of "what time is it" that reached for the clock themselves.
That made every one of them assertable only by approximation: the session test
back-dated a struct field by hand, the cookie test slept for two milliseconds
and then checked the cookie was still there, and the three-day expiry window
had no test at all because reaching it meant waiting for the calendar.
Each now takes the instant as a parameter, following the pattern the workspace
clock module recommends for exactly this case - the deadline logic in
`tuliprox-hls` already does it this way. The old signatures survive as
wrappers over the system clock, so no caller changed.
`crate::clock` holds the one epoch-seconds conversion, and is the only place in
the crate that reads wall time without being handed it.
What this buys, concretely: the cookie boundary is now asserted at the exact
second it flips, a backwards-running clock is shown not to age a session, and
the expired / expiring / quiet split of the account-expiry warning has tests
for all three branches.
Not verified by test execution - checked with cargo check and nightly clippy.
* feat(iptv): the Stalker client's network and clock are seams
`StalkerApiClient` owned a `reqwest::Client` and read the system clock, which
put every interesting decision it makes behind a live portal: the recipe
fallback chain, endpoint-candidate failover, pagination termination, body caps,
and the portal's habit of reporting `{"code": 44, "text": "Account is blocked"}`
inside a `200 OK`. The module docs conceded it outright - "no HTTP requests are
issued from unit tests" - which is another way of saying none of that was
tested.
Both dependencies are now type parameters defaulted to the production
implementation, the shape the workspace clock module recommends. `new()` is
unchanged, `SystemClock` is zero-sized, and neither seam introduces a vtable or
an allocation, so nothing about the production path moved.
Only the *send* is abstracted - requests are still built with reqwest's builder,
because a fake has to build them too and re-modelling a query string buys
nothing. `execute` returns a domain error rather than `reqwest::Error`, which
callers converted anyway and which `reqwest` will not let anyone else construct.
Nine tests now cover paths that previously had no way to be reached at all: a
portal refusal hidden in a 200 body surfacing as a token rejection, an
over-cap body being refused rather than buffered, an HTML error page not
decoding as JSON, endpoint-candidate failover in priority order, exhaustion
reporting the last real failure rather than a synthetic one, pagination
stopping on the advertised last page, a truncated catalog never being returned
as success, the streaming variant making the opposite trade explicitly, and a
session ageing past its TTL on a clock that can be advanced.
`inspect_portal_code` became a free function - it never touched `self`, and as
an associated function on a now-generic type its callers could not infer `Tr`.
Checked with cargo check and nightly clippy across the crate and its two
consumers. The tests added here have NOT been executed.
* refactor(iptv): one redaction module and one error classification
The crate had three unrelated answers to "what must never reach a log line":
`safe_stalker_url` for error URLs, an inline key list inside the debug-dump
writer, and `sanitize_sensitive_info` on the Xtream side. The sibling
media-server crate already keeps that in one module; this is its counterpart,
so the sensitive-key list is defined once and every path that renders a
provider string goes through it. `safe_stalker_url` survives as the name its
callers already use.
Alongside it, `StalkerErrorKind` mirrors `MediaServerErrorKind`. Callers were
matching on variants to answer questions the variants were never organised
around - is the provider down or is my token stale, is this worth retrying -
and a 403 could arrive as either `TokenRejected` or `BadStatus` depending on
which layer noticed it. `kind()` collapses that, and `is_retryable()` is
deliberately false for auth failures so nothing loops on a rejected token.
Also adds a JSON redaction walk with a test that a nested `cmd` is caught,
which the debug-dump writer's own copy never had.
Not verified by test execution - cargo check and nightly clippy only.
* feat(iptv): providers remember what they already told us
Capability knowledge was discovered and then thrown away every refresh.
Whether a portal implements `get_all_channels` was inferred from the shape of
the error it returned; which bootstrap recipe worked was found by walking a
five-entry chain from the top; which of three endpoint candidates answered was
rediscovered per call. None of it survived, so every refresh re-probed
endpoints already known to 404 and replayed a chain whose answer was known.
That is not an ordinary cache miss. A handshake chain replayed from scratch
against a portal with stale credentials looks, from the provider's side, a lot
like credential stuffing - the failure mode the Xtream side already carries a
standing TODO about.
`ProviderCapabilities` is the snapshot, and it is a hint rather than a
contract: every claim carries the instant it was observed and expires after a
day, so a provider that starts implementing an action is picked up without
anyone clearing state by hand, and a remembered endpoint is moved to the front
of the candidate list rather than replacing it. A clock that has run backwards
leaves the snapshot alone instead of invalidating everything.
`CapabilityStore` has two implementations: in-memory for a single run, and one
JSON file per input written through the workspace's atomic-write helper. Input
names come from user config, so the filename is derived rather than taken
verbatim; a corrupt file is ignored rather than fatal, because re-probing is
always available.
Wired into three places that now behave differently: `get_all_channels` is not
re-probed on a portal that has refused it, the handshake chain starts at the
recipe that last worked, and endpoint candidates start at the one that last
answered. Tests cover each, including the cases where the remembered answer has
gone stale or gone away.
Not verified by test execution - cargo check and nightly clippy across the
crate and both consumers.
* feat(iptv): one shape for "fetch this input's playlist"
The three provider families were modelled three different ways - M3U as free
functions, Xtream as free functions with a different arity, Stalker as a struct
client orchestrated from another crate - and each returned a differently-shaped
tuple. The dispatcher paid for it: a ninety-line match whose eight arms each
hand-assembled a six-element tuple, padding the fields their provider does not
produce with literal zeros and `false`s, then destructured the lot by position.
Two of the six elements were dead on arrival - bound to `_m3u_error_count` and
`_xtream_error_count`, computed in two arms, read nowhere.
`PlaylistFetch` is that result with names. `PlaylistProvider` is the one method
every family implements. Provider-specific inputs stay on the provider value -
the event sink on Xtream, the refresh mode on Stalker - so `fetch` takes only
what all of them take.
Dispatch stays a match and stays statically dispatched; the providers share no
supertype and constructing one is free. What changed is that each arm now names
one type and awaits it, and the two unimplemented input types are an
`UnsupportedProvider` carrying its reason rather than a seven-line tuple
literal. Net 98 lines out of the dispatcher.
Stalker's orchestration did not move: it reaches into `tuliprox-repository` and
this crate's own processors, and `tuliprox-iptv` sits below both. The trait is
what lets it stay where it is and still answer in one shape - `StalkerProvider`,
`LibraryProvider` and `PlexProvider` live here, `M3uProvider` and
`XtreamProvider` ship with their clients. The Plex fetch moved out of the
dispatcher into its provider on the way.
Not verified by test execution - cargo check and nightly clippy across iptv,
processing and the binary.
* fix(iptv): catalog fetches honour the configured body cap again
Actions were `&'static str`, threaded from the call site into `send_json`, into
the body-cap lookup, and into six error variants as a `String`. The cap lookup
matched those strings with a silent fallback - and the strings it matched were
not the strings the call sites passed. Catalog fetches announced themselves as
`get_ordered_list` and `get_all_channels`; the lookup tested for `ordered_list`
and `all_channels`. Neither ever matched, so both fell through to the 8 MiB
fallback and a user who raised `ordered_list_mb` got 8 MiB anyway, with nothing
logged to say so. It went unnoticed because the fallback and the default happen
to be the same number.
`StalkerAction` is that set as an enum. Every action names its cap, the match is
exhaustive by construction, and the error variants carry something comparable
rather than a `String` each call site had to spell identically for a later
comparison to work - `is_unsupported_catalog_action` was doing exactly that.
The two handshake calls send the same `action=` query against different
endpoints, so the enum is the label rather than the wire value, and they stay
distinguishable in an error.
Behaviour changes only for users who configured a non-default `ordered_list_mb`
or `get_epg_mb`: they now get what they asked for. Session-shaped actions keep
the same fixed cap they landed on before.
Not verified by test execution - cargo check and nightly clippy across iptv,
processing and the binary.
* feat(iptv): provider failures answer one question
The three provider families report failure in two incompatible ways: Stalker
has a typed `StalkerError`, while M3U and Xtream hand back `Vec<TuliproxError>`
drawn from a workspace enum of forty-odd categories. So the dispatcher never
asked the questions it cares about - is this worth retrying, is the provider
down or is the config wrong - and every error was counted, logged and treated
identically.
Rather than a third error type for everything to convert through,
`ProviderErrorKind` is a classification both existing types map onto. Errors
keep their identity; the judgement is what gets unified. It is ordered by how
much attention a failure deserves, so a fetch that hit one timeout and one bad
portal URL is judged on the URL - `PlaylistFetch::error_kind` takes the worst,
not the first.
`Auth` is deliberately not retryable. It is recoverable, but by re-handshaking,
which is a different call; reporting it as retryable is how a client ends up
hammering a portal with a token that portal has already refused.
Also fixes the Stalker-to-workspace conversion, which flattened all fifteen
variants to `ProviderConnection` - "the network had a bad moment". A
misconfigured portal URL and a rejected password both came out as connection
trouble, and both counted as retryable. The conversion now preserves the class.
The workspace `ErrorKind` arms are enumerated rather than matched by name
prefix, so a renamed variant is a compile error rather than a silent
reclassification.
Not verified by test execution - cargo check and nightly clippy across iptv,
processing and the binary.
* feat(iptv): EPG acquisition has one shape too
EPG was wired per provider family and the two never met. Stalker streams
programme records straight into its repository from three calls in this crate.
M3U and Xtream have no EPG here at all: their XMLTV download lives in
processing, produces documents on disk, and is reached from a separate call
site. Neither knew the other existed.
What they share is the question - does this input have an EPG, and what did
fetching it produce - so that is what `EpgProvider` unifies. What they do not
share is the shape of the answer, and `EpgOutcome` says which happened rather
than forcing one into the other: pretending they were the same would mean
either materialising a several-hundred-megabyte XMLTV document into records in
memory, or teaching the Stalker client to write XMLTV it has no reason to
write. The guide handle is an associated type, so the file-based provider hands
back its `TVGuide` and the streaming one hands back nothing.
Two real implementors: `StalkerEpgProvider` here, `XmltvEpgProvider` in
processing next to the download it wraps, now driving `download_input_epg`.
Per-source EPG failures travel alongside the outcome rather than replacing it -
three sources where one 404s still yields a usable guide from the other two.
`EpgProgramRecord` is an alias rather than a new type: the workspace already
has exactly one programme record, in core rather than in any provider's module,
and its Stalker-flavoured name was the only thing provider-specific about it.
Not verified by test execution - cargo check, nightly clippy and the workspace
dependency gate across iptv, processing and the binary.
* style(iptv): format the new modules with the project rustfmt config
Applied to the eleven files added by this branch only. The rest of the crate
predates the current rustfmt.toml and running `cargo fmt --all` over it would
rewrite files this work never touched - that drift is left alone and called out
separately.
* chore: refresh Cargo.lock
Adds `http` as a dev-dependency of tuliprox-iptv (the fake transport builds
responses with it) and picks up the `dashmap` edge tuliprox-auth's manifest
already declared but the lockfile had not been regenerated for.
* docs(changelog): record the messaging, event bus and auth work
Sixty-six commits since a2de1164 had reached the changelog nowhere. The
file has not been touched since the multi-crate split, so every entry
below is new rather than a revision.
Four breaking changes, each one a field that was written and then never
read back: `token_ttl_mins: 0` no longer mints a ~100-year credential, a
request that never authenticated answers 401 rather than 403, `notify_on`
is glob patterns over dotted event ids, and config responses mask the
channel secrets they used to return in full.
The new features are three threads that turned out to be one. The
notification layer became open-world - an event is an id, a channel is an
impl - which is what let four channels, per-channel routing and a durable
outbox land without a dispatcher change. The event bus became the single
backbone underneath it, so an event emitted once reaches the Web UI, the
notification pipeline and a plugin rather than whichever one its emitter
happened to know about. And auth gained the three things a stateless-JWT
server was missing: a throttle in front of the hash, a revocation
watermark behind the token, and an audit trail for both.
Fixes, optimizations, new settings and maintenance are filled out from
the same range. The refactor commits are summarized by what they buy a
user - units in the type system, one shape for config preparation, no
boxing on the playlist traversal - rather than restated as diffs.
Two claims are deliberately narrower than their commit subjects. Provider
capability memory is described as lasting a client's lifetime, because
`JsonCapabilityStore` is built but not yet constructed in the composition
root. And the registered event count is 32, not the 24 the docs commit
wrote - the events added after it never reached the generated table, so
`every_registered_event_appears_in_the_docs_table` currently fails. That
is left for its own commit; this one only says what is true.
* fix(events): watch payloads carry counts, not prose
`handle_watch_notification` truncated its own lists by pushing a
synthesised sentence into them - "... 42 more added entries omitted"
beside real channel titles, or "5000 entries added. Detailed list
suppressed" replacing the list outright.
That was legible to the text template and to nothing else. `payload()`
serialises `WatchChanges` straight to JSON for plugins, and a plugin has
no way to tell a sentinel from a channel actually named that. The
notification subject line had the same problem from the other side: it
read `w.added.len()`, so a suppressed change of five thousand announced
itself as "1 channel(s) added".
`WatchChanges` grows `added_total`, `removed_total` and `truncated`. The
lists stay pure channel titles, the counts stay true whatever the lists
carry, and the plain-text renderer spells the omission out itself.
`WatchChanges::new` sets the totals from the lists so a caller that is
not truncating cannot get them out of step.
* fix(events): a failed library scan stops reporting itself as finished
`notification_id()` mapped `LibraryScanProgress` to
`LIBRARY_SCAN_COMPLETED` whatever the summary said, so the failure path
in `spawn_library_scan` - which emits the same variant with
`status: Error` - reached operators as "A local library scan finished"
at info severity.
The taxonomy now discriminates on the status the payload already
carries, the way `PlaylistUpdate` has always done on its state: success
keeps `library.scan.completed`, failure takes a new
`library.scan.failed` at error severity, and `severity()` picks that up
from the registry with no second table.
`EventKind` gains `LibraryScanFailed` alongside the progress kind rather
than reusing it. `LibraryScanProgress` is high-frequency and a scan
failure is not, so a subscriber that only wants failures should not have
to take the tick firehose to get them.
The emitter is unchanged - it was already reporting the status
correctly, and nothing downstream was reading it.
* feat(metadata): a broken input stops being silent
`InputMetadataUpdatesCompleted` only fires when a cycle drains *with
changes*, and a task that burns through its retries only reaches a
`debug!`. So an input whose resolves fail every time emitted a start and
then nothing, for as long as it stayed broken - on the bus it looked
exactly like one still working through a long queue.
The worker now counts the tasks that exhaust their retries during a
cycle and emits `InputMetadataUpdatesFailed` when that count is non-zero,
carrying the input, the count, whether anything resolved anyway, and the
last error.
Reported alongside the completion rather than instead of it. A cycle can
both produce changes and exhaust tasks, and the completion is what
triggers the downstream playlist update - suppressing it would trade one
silent failure for another.
Per cycle, not per task: a provider that has stopped answering fails
every item behind it, and `dedup_key` is per input for the same reason
`StreamProbeFailure` is.
`EventBusStats::Default` becomes a hand-written impl on the way past -
the taxonomy crossed 32 kinds and `Default` for arrays stops there.
* feat(events): the server says when it starts and stops
`system.started` and `system.shutdown` have been in the notification
registry - and in the documented event table - since it was written, with
nothing in the tree emitting either. An operator who subscribed got
silence.
`EventMessage::ServerLifecycle` carries both, one payload with two kinds
so a subscriber can ask for restarts alone. It reports the running
version, the bound address on start, and the signal name on stop.
Placement is the whole design here. The start event goes after
`spawn_notification_bridge`, not at the top of `main`: the bridge is what
turns a bus event into a notification, and anything published before it
subscribes reaches nobody. The stop event goes before
`cancel_all_service_tokens`, which stops the outbox that would carry it.
Neither reaches the websocket. There is no panel that renders them, and
the Web UI has necessarily disconnected by the time the second one fires.
* fix(admission): keep the strategy index aligned past a suppressed eviction
The strategy loop counted with a manual `idx` incremented at the end of the
body, but the eviction-reentry suppression arm exits via `continue` and so
skipped it. Every strategy evaluated after a suppressed eviction was handed an
index one too low.
That index is what `build_grace_ctx` stores as
`GraceResolutionContext.strategy_index`, and
`evaluate_remaining_strategies_after_grace` resumes at `strategy_index + 1`.
With `[EvictUserOldest, GraceHoldStream]` and the eviction suppressed, the
grace recorded index 0, so the fallback slice restarted at the grace strategy
itself and replayed it instead of moving past it.
Switched to `iter().enumerate()` so the index cannot drift from the item.
Not covered by a test run.
* feat(messaging): a lost notification reaches the bus
`notification.dead_lettered` was registered and documented; the outbox
detected the condition, bumped `health().dead_lettered` and logged to
`notification::audit`, and that was the end of it. Nothing subscribing to
the bus could learn that a notification had been permanently lost.
The outbox now takes an `EventSink` - generic, like the rest of the
emitters after the static-dispatch pass - and emits
`NotificationDeadLettered` at the point it gives up, carrying the event
id, the attempt count, the channels that never accepted it, and when it
was first enqueued.
It is deliberately *not* notifiable, and deliberately absent from
`NOTIFIABLE_KINDS` so the bridge is not even woken for it. This event
exists because delivery failed; enqueueing a notice about it into the
same outbox, against the same channels that just failed, is the loop its
registry entry warns about. Operators still get the audit line and the
counter - neither runs through the path that broke - and plugins see it
on the bus.
Emitted at the attempts-exhausted site only. A notification every channel
rejects as permanent is also dropped, but that path cannot currently be
told apart from a clean delivery without tracking that does not exist
yet.
* fix(admission): re-read admission after acquiring the per-user gate
`resolve_admission_with_strategies` read the admission state, then queued on
`acquire_user_admission`, then walked the eviction strategies using the snapshot
it had taken before it queued. A request that lost the race therefore acted on a
count the winner had already changed, and could evict a live connection to free
a slot that had been released while it waited.
The re-read is placed after the gate and after the empty-strategy check, so the
uncontended path costs nothing extra, and `build_grace_ctx` now captures the
fresh `kind`.
This does not close the wider check-then-register window: `connection_admission`
only inspects the counts, and the slot is registered by the caller outside this
gate, so two requests at `max_connections - 1` can still both be admitted. That
needs the registration brought under the same gate and is left alone here.
Not covered by a test run.
* fix(admission): stop evicting once a kick frees no connection
Eviction is destructive and is never rolled back. The strategy loop would kick a
target, find the retry still denied, and move on to the next eviction strategy -
so a request that ended up denied anyway could leave several other streams killed
behind it.
The loop now samples `user_connections` either side of the kick. A kick that
reduces the count is real progress and later strategies still run, which keeps
the over-limit case (a hot-swapped config that lowered `max_connections`)
converging. A kick that frees nothing sets `evictions_ineffective`, and further
`Evict` decisions are skipped for the rest of the walk; `Grace` strategies are
still evaluated.
Not covered by a test run.
* refactor(admission): drop the unused kind_for_exhausted parameter and refresh docs
`evaluate_admission_strategy_loop` took `_kind_for_exhausted` and never read it -
both callers construct the exhausted result themselves. Removing it takes the
argument count down by one on a function that needed
`#[allow(clippy::too_many_arguments)]`.
The doc block on `evaluate_remaining_strategies_after_grace` listed a `Deny`
rule, but `AdmissionDecision` only has `NoMatch`, `Grace` and `Evict`. Replaced
it with what the `Grace` arm actually does now that the strategy index it records
is correct.
Not covered by a test run.
* feat(processing): the watch feature stops failing silently
`watch` had one event for everything it knows, and three ways to stop
working without saying so:
* every pattern failed to compile, so `ConfigTarget::watch` became `None`
and `process_watch` returned immediately - the feature disabled itself
on a typo behind a single `warn!`;
* the target carries the reserved default name, which logs and returns;
* the watch state file could not be read or written, which either
re-baselines the group - losing the change it should have reported -
or drops it entirely.
All three now emit `playlist.watch.disabled` with the reason and, where
there is one, the underlying error.
The first needed the config layer to stop discarding the distinction. An
empty `Some` is now load-bearing: it means "configured and unusable",
which is not the same as "not configured", and only the runtime layer has
an event sink to report it from.
`playlist.watch.unmatched` covers the fourth silence. A pattern matching
no group looks exactly like a group that has not changed, so a typo in
`watch` is invisible. `EventKindMask::from_wire_names` already returns
unmatched subscription names for this reason, and its test says why: a
typo must surface, not silently narrow what was asked for.
Matching now walks the groups once and records which patterns hit,
instead of re-testing every pattern per group inside a filter.
* refactor(admission): make the empty-strategy-list rule explicit
`get_effective_admission_strategies` matched on `admission_strategies.is_some()`
and then re-unwrapped with `unwrap_or_default()`, so the guard proved something
the body checked again. More importantly, the rule that an explicitly empty list
suppresses the `grace_period_millis` fallback - while an absent list does not -
was implicit in the arm ordering.
Rewritten as a match on `admission_strategies.as_ref()` with the distinction
stated. Behaviour is unchanged: `Some(vec![])` still means "no strategies", not
"fall back to grace".
Not covered by a test run.
* perf(admission): carry the effective strategy list as Arc<[AdmissionStrategy]>
`GraceResolutionContext` is stored on `StreamInfo` and travels with every clone
of it, so a `Vec<AdmissionStrategy>` field meant reallocating the list on each
clone. `get_effective_admission_strategies` also handed back a fresh `Vec` that
`build_grace_ctx` then cloned again.
The list is immutable once resolved, so `Arc<[AdmissionStrategy]>` fits: the
context clone is now a refcount bump, and the one allocation left is the
`Arc::from` at resolution time. `evaluate_admission_strategy_loop` still takes a
plain `&[AdmissionStrategy]`, reached by deref, so slicing the remaining
strategies is unchanged.
Not covered by a test run.
* feat(processing): a target reports the groups it gains and loses
`watch` tracks channels inside named groups and is blind to the group set
itself, in both directions.
A group appearing was silent: `process_group_watch` found no baseline
file, wrote one and emitted nothing, so the group's entire channel list
read as "not new" from then on. A group vanishing was worse - it is
absent from the refreshed playlist, so nothing iterated it and no code
path observed the disappearance at all.
`process_target_groups_watch` diffs the target's group titles against a
persisted index and emits `playlist.groups.changed`. It runs before the
per-group fan-out and sees every group, not only the ones the watch
patterns name - the question is which groups exist, not what is inside
the watched ones.
The index sits beside the per-group directory rather than inside it
(`<target>.groups.bin`, not `<target>/__groups.bin`) so it cannot collide
with a group whose sanitized title matches. First sight writes the
baseline and says nothing; announcing every group as new on the first
refresh after an upgrade would be noise.
Sampled the same way `WatchChanges` now is: titles only in the lists,
counts that stay true whatever the lists carry.
Gated on `target.watch` being configured, so it costs nothing for targets
that never asked to be watched.
* refactor(admission): bundle the request-scoped admission arguments
`resolve_admission_with_strategies`, `evaluate_remaining_strategies_after_grace`,
`evaluate_admission_strategy_loop`, `get_admission_for_request` and
`should_suppress_eviction_for_recent_request` each threaded the same ten
positional parameters, three of them consecutive bare `bool`s
(`use_session_admission`, then `activate_unbound_session` a slot later). Callers
read `..., true, Some(session_token), true, guard)` - a shape where transposing
two arguments still compiles and silently changes which admission check runs.
They now take one `AdmissionRequest<'a>`, which names every field at the call
site. The `use_session_admission` comment that lived in the parameter list moved
onto the field it documents.
This removes three `#[allow(clippy::too_many_arguments)]` and one
`clippy::too_many_lines`.
Formatted with the project rustfmt config. Not covered by a test run.
* feat(session): the provider pool says when it runs out and when it falls back
Two moments the lineup manager knew about and told nobody.
`provider.pool.exhausted`: `log_exhausted_pool_snapshot` built a complete
picture - per-provider current/max plus expiry - and discarded it unless
debug logging happened to be on. `ActiveProvider` reports that connection
counts moved; nothing reported that a stream was refused because every
provider behind the input was full. The snapshot is now built
unconditionally on that path (already the slow one) and both the debug
line and the event render from the same structured data.
`provider.priority.fallback`: `acquire` walks priority groups highest to
lowest and silently falls through when the preferred ones are at
capacity, so "you are being served off your backup" was invisible.
Reported on transition, not per allocation - the fallthrough happens on
every request while the primary is full, and one event per stream start
would bury the thing worth hearing. A move back towards group zero is a
recovery and says so.
The group is resolved after the fact from the allocated provider rather
than by threading an index out of `acquire`, which keeps the allocation
path and its return type untouched.
Input and provider names go through `sanitize_sensitive_info` before
reaching the payload, for the reason `StreamProbeFailure` documents.
* feat(processing): a failed playlist fetch says what kind of failure it was
`ProviderErrorKind` already classifies every provider failure across all
three families, and already exposes `is_retryable()` and
`needs_operator()` - the two questions an operator actually asks. Nothing
consumed either. Every fetch failure was counted, logged and treated
identically.
The dispatcher now emits `provider.fetch.failed` when a fetch reports
errors, carrying the classification, the worst error's text, how many
there were, and whether any of the playlist came through anyway.
Severity follows the classification rather than the registry: a `Config`
failure will not fix itself and is an error, everything else may and is a
warning. That is the same payload-dependent override `PlaylistUpdate`
already uses for a partial refresh.
`ProviderFailureKind` mirrors `ProviderErrorKind` in `shared`, which
cannot host the original - it classifies a `StalkerError` that `shared`
does not know about. The conversion lives beside the original so a new
variant there is a compile error rather than a silent fallthrough.
Input name and error text go through `sanitize_sensitive_info`: both can
carry a provider URL with credentials in it.
* feat(app): a scheduled task that fails says so
The playlist update and the library scan both report their own outcomes.
The GeoIP refresh had no terminal event of its own - it logged one line
and moved on - so an operator running on a stale database never found
out.
`scheduled_task.failed` carries the task type, the cron expression that
triggered it, and the error. `GeoIpUpdateError::Disabled` stays silent:
the task ran and correctly found nothing to do, which is not a failure.
The task is typed as `ScheduleTaskType` rather than a free string, so a
task added to that enum cannot be reported under a name nothing
recognises.
* feat(session): a refused connection reaches the bus
`ActiveUser` reports connects and disconnects. A refusal is neither, so
the one outcome a user actually complains about was the one nothing
published - the admission ladder modelled it fully, as what is left after
every eviction strategy declines, and handed it to the caller and nobody
else.
`user.connection.denied` carries the user, the address the request was
attributed to, and the limit that was reached. It sits with the auth
events rather than the streaming ones and takes `UserRead`: "who was
turned away" is the same question as "who signed in", and strictly
narrower than the system-wide read.
Only the strategy path emits. An explicit `Terminate` also resolves to
`Exhausted`, but that is a requested teardown, not a denial.
`ActiveUserManager` gains an `events()` accessor rather than `AdmissionCtx`
growing a second handle to the same manager.
* refactor(app): the notification bridge stays a routing table
Fourteen new events left `to_notification` doing its own wording inline,
at which point it was 213 lines and clippy was right about it. Each new
event's wording moves into a `*_notification` builder beside the three
that already existed, and the match goes back to being one arm per
variant with no logic in it.
`push_sampled_list` is shared by the two events that carry sampled lists,
so the "... N more not listed" wording has one home rather than two.
Also completes the documented event table. Eight events registered by the
user-lifecycle and auth work - `user.created`, `user.updated`,
`user.deleted`, the four `auth.*` decisions and `stream.probe.failed` -
were never given a row, so `every_registered_event_appears_in_the_docs_table`
has been failing since before this branch. The rows are generated from
the descriptors so severity and description match exactly.
`stream.probe.failed` had a run of twenty-seven spaces inside its
description, left behind when a wrapped literal was collapsed onto one
line. Normalised in the registry and the table together.
* docs(changelog): record the bridge refactor and the completed event table
The eighteen commits since 4dee77a6 reached the changelog in d4b9c7d9,
which was about the notification bridge and carried CHANGELOG.md along
with it. This commit adds the two entries that commit earned and its own
message did not put in the file.
The docs table entry is the one an operator can act on: eight registered
events had no row, so `every_registered_event_appears_in_the_docs_table`
was failing and anyone reading section 5 to pick a `notify_on` pattern
could not see `user.*`, `auth.*` or `stream.probe.failed` at all.
The bridge refactor is recorded as maintenance because it buys one thing
a reader cares about - each event's wording has one home - rather than as
a line count.
* fix(xtream): a blank container extension falls back to the item url
A VOD document's `container_extension` comes from the provider's
`get_vod_streams` response, where a missing or null value collapses to an
empty string, and nothing fills it in afterwards unless a per-item
`get_vod_info` fetch or an ffprobe run happens to reach it. A provider
that omits the field therefore published `""` on every VOD, and a client
building `<stream_id>.<container_extension>` has nothing to append -
several stringify the blank and go on to request `813563.null`.
`get_vod_info` already fell back to the extension carried by the item URL.
The four builders that did not now share one helper on
`XtreamPlaylistItem`: the stream-list document, both no-properties paths,
and the resolved info document. That last one is post-processed on the
returned `XtreamInfoDocument::Video` rather than threading `url` through
`StreamProperties::to_info_document`, whose signature is shared with the
series and episode paths that have no URL to pass.
A non-empty provider value still wins, so the fallback cannot talk over
what the provider actually said, and an item whose URL carries no
extension either still reports the blank.
`create_vod_info_from_item` had the fallback already but kept the leading
dot that `extract_extension_from_url` returns, publishing `.mkv` into a
field that holds `mkv` everywhere else - a client would have built
`813563..mkv` from it. Stripped.
Series episodes are left alone. `SeriesStreamDetailEpisodeProperties`
carries no provider URL at that layer, only a `direct_source`, so the same
fallback has nothing to read there.
The six new tests are compiled but unrun.
* fix(frontend): reload plans when opening user create/edit screen
UserEdit previously fetched user plans only once on initial mount (use_effect_with((), ...)). Because panels are kept mounted in the SPA, newly created or modified plans in the Plans view did not appear in the plan dropdown until a full page reload or container restart.
Update effect dependencies to include active_page and selected_user so plans are re-fetched whenever navigating to UserlistPage::Edit.
* refactor(requests): replace reqwasm with gloo-net for HTTP requests
* refactor(locking): drop fs2 for std file locking
fs2's last release was 2018. Rust 1.89 stabilized File::lock,
lock_shared, try_lock, try_lock_shared and unlock together with
std::fs::TryLockError, and the workspace MSRV is 1.95 — so the
dependency comes out entirely rather than being swapped for fs4.
The syscalls underneath are unchanged. The one semantic shift is
try_lock: contention now arrives as TryLockError::WouldBlock instead
of an io::Error carrying ErrorKind::WouldBlock, which makes "lock is
held" a distinct variant from a real I/O failure at the type level.
* refactor(catchup): introduce effective_mode method to prioritize catchup-type
* refactor(catchup): enhance timestamp validation and error handling for catchup windows
* refactor(mapping): prepare extensible processing pipeline
* refactor(processing): compile target execution pipeline
* clippy fixes
* test fixes
* test fixes
---------
Co-authored-by: DarkBreakpoint <darkbreakpoint@github.com>
Co-authored-by: DarkBreakpoint <243206744+DarkBreakpoint@users.noreply.github.com>
Co-authored-by: euzu <euzu@proton.me>
78 KiB
78 KiB
🏛️ config.yml (Core System)
The config.yml is the primary configuration file of Tuliprox. It dictates the engine's core runtime behavior, memory
management,
caching mechanisms, background schedulers, and external integrations (like HDHomeRun, GeoIP, and the Web UI).
Top-level entries
process_parallel: false
disk_based_processing: false
storage_dir: ./data
default_user_agent: Tuliprox/...
backup_dir: ./data/backup
user_config_dir: ./data/user
mapping_path: mapping.yml
template_path: template.yml
update_on_boot: false
config_hot_reload: false
accept_insecure_ssl_certificates: false
sleep_timer_mins: null
connect_timeout_secs: 10
interner_gc_interval_secs: 180
interner_gc_min_pool_size: 100
user_access_control: false
custom_stream_response_path: null
custom_stream_response_timeout_secs: 0
custom_stream_response_enabled: true # default: true (serve the fallback videos)
custom_stream_response_error_status: 502 # 4xx/5xx only; returned when custom_stream_response_enabled is false
api:
web_ui:
log:
schedules:
messaging:
video:
proxy:
ipcheck:
hdhomerun:
library:
reverse_proxy:
metadata_update:
Global System & Storage Settings (Flat Keys)
| Parameter | Type | Required | Default | Technical Impact & Background |
|---|---|---|---|---|
process_parallel |
Bool | No | false |
Enables dependency-aware parallel updates. Inputs from the same source and independent sources may download concurrently. A source's targets start as soon as all of its enabled inputs are ready; disjoint target outputs may then finalize concurrently. With false, inputs and targets keep configured sequential order. Use inputs[].sequential_group to serialize credentials that must never overlap. |
disk_based_processing |
Bool | No | false |
Tradeoff Guidance: Normally, Tuliprox loads playlists into RAM. With true, every chunk is manipulated directly on disk (using a B+Tree database). Use this on low-end hardware (e.g., Raspberry Pi) or with massive playlists (>500k streams) to prevent Out-Of-Memory crashes. It significantly increases Disk I/O load, so it is slower but much safer for tight memory footprints. |
storage_dir |
String | No | ./data |
Root directory for all runtime data (B+Tree databases, downloads, caches). Relative paths are resolved against the Tuliprox Home Directory. Be aware that different configurations (e.g. user bouquets) alongside the playlists are stored in this directory. |
default_user_agent |
String | No | Tuliprox/... |
Fallback HTTP User-Agent used for upstream provider requests if the input definition or client request does not explicitly provide one. |
backup_dir |
String | No | {storage_dir}/backup |
Storage location for config backups (e.g., triggered via "Save Configuration" in the Web UI). |
user_config_dir |
String | No | {storage_dir}/user |
Storage location for user-specific configurations (like favorites or custom bouquets created via the Web UI). |
mapping_path |
String | No | mapping.yml |
Path to the mapping file. Pro-Tip: If you specify a folder path here (e.g., ./config/mappings/), Tuliprox loads all .yml files in that folder in alphanumeric order and merges them. Note: This is a lexicographic sort, meaning m_10.yml comes before m_2.yml. Name files carefully (e.g., m_01.yml, m_02.yml). |
template_path |
String | No | template.yml |
Path to the template macro file. Specifying a folder here is also possible and highly recommended (see above). |
update_on_boot |
Bool | No | false |
Forces Tuliprox to immediately query all providers and rebuild all playlists upon startup. If false, the proxy serves the local DB cache from the last run until the scheduler triggers the next update. |
config_hot_reload |
Bool | No | false |
Spawns a filesystem watcher for mapping.yml and api-proxy.yml. Upon saving, mappings and user credentials become active immediately without requiring a server restart. (See Bind-Mount Note below) |
accept_insecure_ssl_certificates |
Bool | No | false |
Set to true if your upstream provider uses expired, self-signed, or improperly configured HTTPS certificates. Otherwise, the HTTP client drops the connection securely. |
sleep_timer_mins |
Int | No | null |
Automatic kill-switch for proxied streams. Forcibly terminates active stream connections after X minutes (Protects against users falling asleep with the TV on). |
connect_timeout_secs |
Int | No | 10 |
Maximum time (in seconds) Tuliprox waits to establish the initial TCP connection to a provider. 0 disables the timeout and the connection attempt continues until the provider closes it or a network timeout occurs (Warning: risk of hanging threads!). |
interner_gc_interval_secs |
Int | No | 180 |
Interval in seconds between background string interner garbage-collection checks. Lower values reclaim unused interned strings sooner, but run the GC gate more often. Higher values reduce GC overhead, but can retain more memory for longer. Must be greater than 0. |
interner_gc_min_pool_size |
Int | No | 100 |
Minimum number of interned strings required before the background interner GC runs. This avoids paying the interner write-lock cost when the pool is very small. Lower values favor aggressive cleanup; higher values favor throughput and lower lock contention at the cost of extra retained memory. |
event_channel_capacity |
Int | No | 256 |
Buffer depth of the in-process event broadcast channel, per subscriber. Every runtime event — playlist-update progress, config reloads, download deltas, recording changes — passes through it on the way to the Web UI websocket and the notification pipeline. A subscriber that falls this many events behind is told it lagged and resumes from the newest event, having missed the gap; the Web UI recovers by re-requesting a status snapshot. Raise it if the bus reports lag under load. 0 is clamped to 1. |
user_access_control |
Bool | No | false |
Security: If true, Tuliprox actively enforces status (Active/Banned), exp_date, and max_connections constraints for users defined in api-proxy.yml. If false, those fields are ignored. |
custom_stream_response_path |
String | No | null |
Directory path where Tuliprox looks for custom fallback .ts files. See section Custom Stream Response for exact filenames. |
custom_stream_response_timeout_secs |
Int | No | 0 |
Hard timeout (in seconds) that forces the fallback video stream to terminate to prevent infinite bandwidth usage. 0 means endless loop. |
custom_stream_response_enabled |
Bool | No | true |
Enables configured custom MPEG-TS fallback videos for stream errors. When false, factories skip the video body and call sites return custom_stream_response_error_status; useful behind reverse proxies that intercept 4xx/5xx responses instead of keeping fallback sockets open. |
custom_stream_response_error_status |
Int | No | 502 |
HTTP status code returned when custom_stream_response_enabled is false. Must be a 4xx or 5xx code (the prepare() step rejects anything outside that range; 0 is silently clamped to 502). Common choices: 404 (channel not found), 502 (bad gateway — upstream failed), 503 (service unavailable — overloaded). |
⚠️ Important: config_hot_reload & Bind-Mounts
If you use Bind-Mounts (e.g., in fstab or Docker), the filesystem watcher may report the original source path
instead of your mount point.
- Example Setup:
/home/tuliprox/config(Source) →/config(Mount Point). - Behavior: If you configure Tuliprox to watch
/config, the watcher might still trigger events using the path/home/tuliprox/config. - Solution: Ensure your internal Tuliprox paths match the paths reported by the OS kernel to ensure the hot-reload trigger fires correctly.
Subsections (Object Keys)
| Block | Description | Link |
|---|---|---|
api |
Internal web server binding settings. | See section |
web_ui |
Web Dashboard, RBAC, and Authentication. | See section |
log |
Console output verbosity and sanitization. | See section |
schedules |
Automated background tasks (Cronjobs). | See section |
messaging |
Webhooks & Push-Notifications (Telegram, Discord, etc.). | See section |
video |
Extension mapping and Web UI download behavior. | See section |
proxy |
SOCKS5/HTTP proxy settings for outgoing requests. | See section |
ipcheck |
IP detection to verify in the Web UI which public IP Tuliprox is currently using. | See section |
hdhomerun |
Virtual DVB-C/T network tuner emulation. | See section |
library |
Local Media Library integration. | See Local Library |
reverse_proxy |
Streaming buffers, rate limits, caching. | See Reverse Proxy |
metadata_update |
TMDB matching, FFprobe processing, Job Queues. | See Metadata Update |
(Note: The advanced topics Local Library, Reverse Proxy and Metadata Update are extremely extensive and have their own dedicated subchapters. Here we cover the global base settings.)
1. API Server (api)
Controls the internal web server of Tuliprox. This does not dictate the public URLs given to clients (those belong in
api-proxy.yml), but rather the physical socket binding on your host machine.
api:
host: 0.0.0.0
port: 8901
web_root: ./web
| Parameter | Type | Required | Default | Technical Impact & Background |
|---|---|---|---|---|
host |
String | No | 0.0.0.0 |
Bind interface. 0.0.0.0 listens on all network cards. 127.0.0.1 restricts access to localhost (useful if you force traffic through a local Nginx/Traefik reverse proxy). |
port |
Int | No | 8901 |
The listening port for proxy streams, the Web UI, and all REST APIs. |
web_root |
String | No | ./web |
Physical path to the compiled Wasm/JS/CSS frontend assets of the Web UI. |
2. Web UI & Administration (web_ui)
Tuliprox ships with a comprehensive Web Dashboard containing a Web Player, Playlist Editor, User Management, and Live Logs.
web_ui:
enabled: true
user_ui_enabled: true
path: admin
player_server: default
kick_secs: 90
combine_views_stats_streams: false
landing_page: dashboard
stream_info:
hide_group: false
hide_ip: false
hide_country: false
hide_shared: false
hide_duration: false
hide_bandwidth: false
hide_transferred: false
hide_player: false
hide_user_comment: false
hide_epg: false
content_security_policy:
enabled: true
custom-attributes:
- "style-src 'self' 'nonce-{nonce_b64}'"
- "img-src 'self' data:"
auth:
enabled: true
issuer: tuliprox
secret: "YOUR_SECRET_JWT_KEY_HERE"
token_ttl_mins: 30
userfile: user.txt
groupfile: groups.txt
2.1 Web UI Parameters
| Parameter | Type | Default | Technical Impact & Background |
|---|---|---|---|
enabled |
Bool | true |
Completely toggles the Web Dashboard and its REST API endpoints on or off. |
user_ui_enabled |
Bool | true |
Allows standard proxy users (not just admins) to log into the Web UI to manage their own favorites/bouquets. |
path |
String | "" |
Base path for the UI (e.g., admin). Critical for reverse proxy subfolder setups so assets load from example.com/admin/assets/. |
player_server |
String | default |
Determines which virtual server block from api-proxy.yml is used to construct the streaming URLs when playing a channel directly within the Web UI player. |
kick_secs |
Int | 90 |
Background: When you kick a user via the Dashboard, they are not only disconnected but hard-blocked at the IP/User level for X seconds. This prevents their IPTV player's auto-reconnect logic from instantly stealing the provider slot back. |
combine_views_stats_streams |
Bool | false |
Combines the "Server Stats" and "Active Streams" views into a single unified window in the UI. |
landing_page |
String | dashboard |
Set the initial landing page for the webui. Possible values are dashboard, stats, streams, stream_history, downloads, users, config, source_editor, playlist_update, playlist_settings, playlist_explorer, playlist_epg, rbac |
stream_info |
Object | null |
Optional visibility controls for fields shown in the active stream display. If omitted, or if all nested hide_* flags are false, the default dashboard view is unchanged. |
2.2 Stream Display Visibility (stream_info)
Use this optional block to hide selected fields in the active stream display shown in the dashboard.
web_ui:
stream_info:
hide_group: true
hide_ip: true
hide_epg: true
| Parameter | Type | Default | Technical Impact & Background |
|---|---|---|---|
hide_group |
Bool | false |
Hides the stream group/category label. |
hide_ip |
Bool | false |
Hides the client IP shown for an active stream. |
hide_country |
Bool | false |
Hides the detected client country flag/code. |
hide_shared |
Bool | false |
Hides the shared-stream indicator. |
hide_duration |
Bool | false |
Hides the running playback duration. |
hide_bandwidth |
Bool | false |
Hides the live bandwidth badge when metrics are available. |
hide_transferred |
Bool | false |
Hides the transferred-bytes badge when metrics are available. |
hide_player |
Bool | false |
Hides the player / user-agent field. |
hide_user_comment |
Bool | false |
Hides the user comment next to the active stream title. |
hide_epg |
Bool | false |
Hides per-stream EPG information and skips the associated dashboard EPG fetch/update work for that display. |
- If every
hide_*flag isfalse, Tuliprox treats the wholestream_infoblock as empty/default. - This setting is global for the Web UI. It is not currently per-user.
2.3 Content Security Policy (content_security_policy)
This block enhances security by restricting which resources the browser is allowed to load.
- Default Directives: When
enabled: true, Tuliprox automatically applies:default-src 'self'script-src 'self' 'wasm-unsafe-eval' 'nonce-{nonce_b64}'frame-ancestors 'none'
- Customization: Use
custom-attributesto add specific rules (e.g., allowing external channel logos viaimg-src).
2.4 Authentication & RBAC (auth)
Tuliprox features a robust Role-Based Access Control (RBAC) system.
| Parameter | Type | Default | Technical Impact & Background |
|---|---|---|---|
enabled |
Bool | true |
Master switch for UI authentication. Disabling this exposes the dashboard to anyone with network access. |
issuer |
String | tuliprox |
The identifier for the JWT "iss" field. |
secret |
String | (Random) |
Critical for JWT encryption. Use a static 64-character hex string using Node.js (see Secret Generation) to keep sessions valid across restarts. If omitted, Tuliprox generates one in-memory, but all active logins will invalidate on every server restart! |
token_ttl_mins |
Int | 30 |
How long a login session remains valid. Setting this to 0 makes the token effectively valid for 100 years (Extreme Security Risk!). |
userfile |
String | user.txt |
The file storing Admins and Web Users. |
groupfile |
String | groups.txt |
The RBAC (Role-Based Access Control) definition file. |
Technical Background
- File Resolution: If
userfileis not defined with an absolute path, Tuliprox automatically looks for it within your globalconfig_dir. Ensure the process has sufficient read permissions for this directory. - RBAC (Role-Based Access Control): This system manages Web UI access levels (e.g., Admins vs. Bouquet-Editors)
by assigning users to specific permission groups defined in
groups.txt.
Structure of user.txt
This file stores users, Argon2 password hashes, and RBAC groups. Generate secure passwords via CLI:
./tuliprox --genpwd.
The userfile has the following format per line: username:argon2_hash[:group1,group2]
Example:
# A normal Admin (No group specified = Fallback to built-in Admin role)
admin:$argon2id$v=19$m=19456,t=2,p=1$QUp...
# An Editor assigned to specific permission groups
editor:$argon2id$v=19$m=19456,t=2,p=1$Y2F...:playlist_manager,user_manager
Structure of groups.txt
Define group permissions here. An editor might be allowed to update playlists (playlist.write) but forbidden from
viewing or changing config.yml (config.read).
Format: group_name:permission1,permission2,...
viewer:config.read,source.read,playlist.read,system.read,library.read
playlist_manager:playlist.read,playlist.write,source.read
Available Permissions: config.read/write, source.read/write, user.read/write, playlist.read/write,
library.read/write,
system.read/write, epg.read/write, download.read/write. Note: Write does not imply Read. A group must explicitly
grant both if users need
to view
and edit content.
Generating Passwords
To ensure security, Tuliprox does not store plain-text passwords. You must generate an encrypted hash using the built-in generator:
Local Installation:
./tuliprox --genpwd
Docker Installation:
docker container exec -it tuliprox ./tuliprox --genpwd
After running the command, copy the generated Argon2id string and manually paste it into your userfile next to the
desired username.
JWT Secret Generation
As mentioned in the table above, a static secret is required to keep sessions valid across restarts.
You can generate a secure 32-byte (64-character hex) key using Node.js:
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
Troubleshooting Access
- File Location: If Tuliprox cannot find the file, check your
storage_diror provide an absolute path in theuserfileparameter. - Permissions: Ensure the Tuliprox process has read access to the
userfileandgroupfile. - Session Invalidated: If you change the
secretinconfig.yml, all currently logged-in users will be forced to log in again.
3. Logging (log)
Controls console output verbosity and sanitization.
log:
sanitize_sensitive_info: true
log_active_user: false
log_level: info
runtime_config_report_enabled: false
runtime_config_report_format: yaml
| Parameter | Type | Default | Technical Impact & Background |
|---|---|---|---|
log_level |
String | info |
Verbosity. Possible values: trace, debug, info, warn, error. Can be overridden per-module (e.g., tuliprox=debug,hyper_util=warn). |
sanitize_sensitive_info |
Bool | true |
Critical: Masks passwords, provider URLs, and external client IPs in the logs with ***. Highly recommended to keep true so you can safely share logs on GitHub/Discord for support without leaking credentials. |
log_active_user |
Bool | false |
Periodically writes the current active client connection count as an INFO message to the log file. |
runtime_config_report_enabled |
Bool | false |
Opt-in startup dump of the complete effective runtime configuration after defaults, path resolution, and runtime preparation have been applied. Sensitive values such as passwords, secrets, tokens, and API keys are masked as ***. |
runtime_config_report_format |
String | yaml |
Output format for the runtime config report. Supported values: json and yaml. Both formats are emitted as pretty, multi-line output. Ignored unless runtime_config_report_enabled is true. |
If enabled, Tuliprox keeps the existing startup summary and prints the additional full runtime report once during startup. The report includes:
- prepared
config.yml - prepared
source.yml - loaded mappings/templates/api-proxy sections when present
- resolved runtime
paths
The report is intended for diagnostics and verification of effective defaults. Sensitive values are redacted before logging.
4. Schedules (schedules)
Automate your background updates to keep your playlists, library, and Geo-IP data synchronized.
⚠️ Provider Safety Warning: Do not schedule updates to run every second or minute. Excessive requests can lead to your IP being banned by your provider. Updating twice a day is generally sufficient for most use cases.
Cron Syntax
Tuliprox uses a standard cron syntax but strictly requires 7 fields, starting with seconds:
# ┌──────────────────────────── second (0 - 59)
# │ ┌──────────────────────── minute (0 - 59)
# │ │ ┌──────────────────── hour (0 - 23)
# │ │ │ ┌─────────────── day of month (1 - 31)
# │ │ │ │ ┌─────────── month (1 - 12)
# │ │ │ │ │ ┌─────── day of week (0 - 6) (Sunday to Saturday; 7 is also Sunday)
# │ │ │ │ │ │ ┌─── year (optional, e.g., 2026)
# │ │ │ │ │ │ │
# sec min hour dom mon dow year
0 0 8 * * * *
Configuration
In current versions, schedules are defined as a list of tasks.
schedules:
# Every morning at 08:00:00 (Playlist Update for specific targets)
- schedule: "0 0 8 * * * *"
type: PlaylistUpdate
targets: [ "m3u_target", "xtream_target" ]
# Every evening at 20:00:00 (Full Library Scan)
- schedule: "0 0 20 * * * *"
type: LibraryScan
# Every Monday at 04:00:00 (Geo-IP Database Refresh)
- schedule: "0 0 4 * * 1 *"
type: GeoIpUpdate
# Every 1st of the month at 04:00:00
- schedule: "0 0 4 1 * * *"
type: GeoIpUpdate
| Parameter | Type | Default | Description |
|---|---|---|---|
schedule |
String | - | Cron expression with 7 fields (Seconds included at the start). |
type |
Enum | PlaylistUpdate |
The task to execute. See Task Types below. |
targets |
List | - | (Optional, only for PlaylistUpdate) List of target names to restrict the update to. If omitted, all enabled targets are updated. |
Task Types
PlaylistUpdate: Triggers the processing pipeline for your target playlists. It downloads provider data, applies filters/maps, and updates the local databases.LibraryScan: Initiates a scan of the local media library. Note: This requires thelibraryconfiguration to be enabled.GeoIpUpdate: Downloads the latest MaxMind/Geo-IP database and rebuilds the internal binary file. Note: Requiresreverse_proxy.geoip.enabled: true.
5. Messaging (messaging)
Tuliprox can proactively notify you via Push-Notifications when updates fail, finish, or when specific channels are added/removed from a watched group. Why is this useful? Because it allows you to instantly detect upstream provider issues or simply let you know when new movies are added to your playlist.
5.1 Configuration & Opt-In
Messaging is strictly opt-in. Nothing is sent until notify_on asks for it.
notify_on is a list of glob patterns over event ids. An event id is a
dotted domain.event string:
| Pattern | Matches |
|---|---|
* |
every event |
recording.* |
every event in the recording domain |
recording.completed |
that event only |
provider.*.expired |
one wildcard segment |
!system.info |
excludes, whatever else matched |
A subscription matches when at least one positive pattern matches and no
negative pattern does, so ["*", "!system.info"] reads the way it looks.
Upgrading: the old event names (
info,stats,error,watch,disk_alert,recording_started,recording_completed,recording_failed) are still accepted and are rewritten to their canonical ids the next time the config is saved. Existing configs and existing template files keep working untouched.
Available Events
| Event id | Default severity | Description |
|---|---|---|
system.info |
info | A general informational message. |
system.error |
error | A general error message. |
system.disk.alert |
warn | Disk usage crossed the warn or critical threshold. |
system.started |
info | The server finished starting up. |
system.shutdown |
info | The server is shutting down cleanly. |
playlist.update.completed |
info | A playlist update finished; carries per-source statistics. |
playlist.update.failed |
error | A playlist update failed. |
playlist.watch.changed |
info | Channels were added to or removed from a watched group. |
playlist.groups.changed |
info | Groups were added to or removed from a target. |
playlist.watch.disabled |
warn | A target's watch configuration is set but not working. |
playlist.watch.unmatched |
warn | Watch patterns matched no group in the refreshed playlist. |
recording.started |
info | A recording started. |
recording.completed |
info | A recording completed. |
recording.failed |
error | A recording failed. |
provider.account.status_changed |
warn | A provider reported a changed account status. |
provider.account.expiring |
warn | A provider account is approaching its expiry date. |
provider.account.expired |
error | A provider account has expired. |
provider.fetch.failed |
error | An input's playlist could not be fetched. |
provider.pool.exhausted |
warn | Every provider behind an input was at capacity. |
provider.priority.fallback |
warn | An input started being served from a different provider priority group. |
config.changed |
info | A configuration file was changed and reloaded. |
config.reload_failed |
error | A configuration file changed but could not be reloaded. |
library.scan.completed |
info | A local library scan finished. |
library.scan.failed |
error | A local library scan could not complete. |
metadata.update.started |
info | A metadata update started for an input. |
metadata.update.completed |
info | A metadata update finished for an input. |
metadata.update.failed |
error | A metadata update cycle ended with tasks it could not finish. |
user.connection.changed |
info | A user connected or disconnected. High frequency - subscribe deliberately. |
provider.connections.changed |
info | A provider's active connection count changed. High frequency - subscribe deliberately. |
user.connection.denied |
warn | A user was refused a connection because their limits were full. |
recording.queue.changed |
info | The recording queue changed. |
recording.rules.changed |
info | The recording rule set changed. |
scheduled_task.failed |
error | A scheduled task could not complete. |
notification.dead_lettered |
error | A notification was permanently undeliverable and has been dropped. |
user.created |
info | An API-proxy user account was created. |
user.updated |
info | An API-proxy user account was changed. |
user.deleted |
warn | An API-proxy user account was deleted. |
auth.sign_in.succeeded |
info | A principal signed in and was issued a token. |
auth.sign_in.failed |
warn | A sign-in was rejected. Deduplicated per principal and address, so a password-guessing run notifies once rather than per attempt. |
auth.sign_in.throttled |
warn | A sign-in was refused without checking credentials because the caller is backing off after repeated failures. |
auth.permission.denied |
warn | An authenticated principal asked for something its permissions do not cover. |
stream.probe.failed |
warn | A stream probe returned no metadata; the stream may be dead. Deduplicated per input, so a provider outage notifies once rather than per channel. |
Severity is one of info, warn, error, critical, in that order.
Example
messaging:
notify_on: [ "playlist.update.failed", "recording.*", "provider.account.*", "system.disk.alert" ]
# Telegram: Supports Markdown and Group Topics
telegram:
markdown: true
bot_token: "<TOKEN>"
chat_ids:
- "<CHAT_ID>"
- "<CHAT_ID>:<MESSAGE_THREAD_ID>" # Use colon to target a specific topic/thread
templates:
playlist.update.completed: 'file:///config/messaging_templates/telegram_playlist_update_completed.templ'
recording.completed: 'file:///config/messaging_templates/telegram_recording_completed.templ'
# Discord: Webhook integration
discord:
url: "<WEBHOOK_URL>"
templates:
system.info: '{"content": "🚀 Tuliprox Info: {{message}}"}'
# Pushover: Simple mobile push alerts
pushover:
token: "<API_TOKEN>"
user: "<USER_KEY>"
# ntfy: Self-hosted push, no account needed
ntfy:
url: "https://ntfy.sh"
topic: "my-tuliprox"
# token: "<BEARER_TOKEN>" # only for a protected topic
# Gotify
gotify:
url: "https://gotify.local"
token: "<APP_TOKEN>"
# Slack incoming webhook
slack:
url: "<WEBHOOK_URL>"
# REST: Generic webhook/API support
rest:
url: "https://my-api.local/alert"
method: "POST"
headers:
- "Content-Type: application/json"
signing_secret: "<SHARED_SECRET>" # optional, see 5.4
templates:
system.error: '{"text": "Alert: {{message}}", "type": "{{kind}}"}'
# command: run a local program with the event JSON on stdin
command:
program: "/config/scripts/notify.sh"
args: [ "--from-tuliprox" ]
timeout_secs: 30
5.1.1 Per-Channel Routing
Every channel takes an optional routing block. Without one it inherits the
global notify_on, so existing configs are unaffected.
messaging:
notify_on: [ "*" ]
pushover:
token: "<API_TOKEN>"
user: "<USER_KEY>"
routing:
min_severity: critical # only wake me for real problems
discord:
url: "<WEBHOOK_URL>"
routing:
notify_on: [ "playlist.*", "recording.*" ]
quiet_hours: "23:00-07:00" # deferred, never dropped
max_per_hour: 20 # circuit breaker
dedup_window_secs: 3600 # suppress a repeated event
| Field | Meaning |
|---|---|
notify_on |
Overrides the global subscription for this channel. Same glob grammar. |
min_severity |
Drop anything below info | warn | error | critical. |
quiet_hours |
HH:MM-HH:MM local time. Wrapping windows (23:00-07:00) work. Notifications are deferred until the window closes, never dropped - an overnight outage must not be silently invisible. |
max_per_hour |
Circuit breaker. On tripping, the channel emits one "further notifications suppressed" audit line and then goes quiet for the rest of the hour, so the silence is distinguishable from a broken notifier. |
dedup_window_secs |
Suppress a repeated event with the same dedup key for this long. Generalizes what used to be disk_alert.repeat_interval_secs. |
5.1.2 Delivery, Retry and the Outbox
Every notification goes through a durable outbox before it is sent. Entries
are persisted to {storage_dir}/notification_outbox.json before the first
attempt, retried per channel with capped exponential backoff, and
dead-lettered after max_attempts.
Retry is at-most-once per channel: a message that reached Telegram but not Discord is retried only against Discord, so a retry never duplicates a delivered message.
- A
429or503is retried, and a provider'sRetry-Afteris honoured rather than being retried straight back into the same rate limit. - A
404or401is permanent - retrying a malformed webhook URL cannot help, so it is dead-lettered immediately instead of consuming every attempt. - Each channel has a 30 second request timeout, and channels are attempted concurrently, so one unresponsive host cannot stall the others.
Tuning lives under video.download.recording.notifications
(outbox_buffer, max_attempts, backoff_initial_secs,
backoff_max_secs).
5.1.3 Testing Your Configuration
POST /api/v1/config/messaging/test renders and optionally sends a chosen
event, and returns the exact rendered body per channel.
{ "event": "recording.completed", "channel": "telegram", "preview": true }
event- any event id. Defaults tosystem.info.channel- restrict to one channel id. Omit for all configured channels.preview- render only, send nothing. Use this to iterate on a template without spamming a channel.
The test deliberately bypasses notify_on and the suppression window: you
asked for this one explicitly.
5.2 Templating (Handlebars)
Every channel supports templates - Telegram, Discord, REST, Pushover, ntfy, Gotify, Slack and command.
Tuliprox uses Handlebars to format message bodies, which allows rich, structured
notifications (e.g. Discord embeds or Markdown tables).
Templates are keyed by event id:
telegram:
templates:
recording.completed: 'file:///config/messaging_templates/telegram_recording_completed.templ'
A {prefix}_{event}.templ file dropped into /config/messaging_templates/ is picked up automatically -
telegram_recording_completed.templ, discord_playlist_update_failed.templ, and so on. Dots in the event id become
underscores in the filename.
Templates are resolved and compiled once and then cached: local files are re-read when their mtime changes, remote templates revalidate on a 5 minute TTL, and a remote template that cannot be refreshed serves the cached copy rather than silently falling back to the built-in text.
If no template applies, the channel sends the event's built-in title and body, which are always populated and
always human-readable.
Loading Methods
- Raw String: Define the template directly in your
config.yaml(best for simple one-liners). - URI: Reference a local file (
file://...) or a remote resource (http://...).
Note: If you save your configuration via the Web UI, raw template strings are automatically moved to individual files in
/config/messaging_templates/to keep your main configuration clean.
Available Context Variables
The Handlebars engine provides a rich context object. Depending on the kind of notification, different variables are
populated:
Every template sees a uniform event object, plus the legacy top-level keys documented below:
{{event.id}}: The canonical event id, e.g.recording.completed.{{event.severity}}:info,warn,errororcritical.{{event.title}}: One-line summary. Always present.{{event.body}}: Plain-text body. Always present.{{event.fields}}: The typed payload for this event.{{event.timestamp}}: Unix seconds. Use{{timestamp}}for the RFC 3339 form.
The legacy keys below continue to work exactly as before, so existing templates need no change:
-
{{message}}: The primary text payload. Used for human-readableinfomessages or the summary of anerror. -
{{kind}}: The event category (Info,Stats,Error,Watch, ...). Use this in Handlebars helpers (e.g.,{{#if (eq kind "error")}}) to create conditional layouts. -
{{severity}}: The event severity. -
{{timestamp}}: The event occurrence time in UTC (ISO 8601 / RFC3339 format). -
{{stats}}: Execution Metrics & Performance. A comprehensive list of statistics for the last update cycle, covering both ingestion and generation phases.- Structure: A nested list containing
inputs(Source-level metrics) andtargets(Output-level metrics). - Access: Iterate over the main list to access individual source or target reports. Use nested loops for
detailed input/target breakdowns:
{{#each stats}} {{#each inputs}} ... {{/each}} {{/each}}. - Key Properties:
- Metadata: Access
name,type, andtook(execution duration) for each entry. - Error Tracking:
errorsprovides a count of failed items or connection issues during that specific phase. - Filtering Delta: Compare
rawcounts (total items received from the provider) vs.processedcounts ( items that survived your Mapping DSL and filters) to monitor your "Red Thread" efficiency. - Target Processing: A target may include a
processingobject with exact filter inspected/retained/removed counts, renamed item/field counts, matched mapping rules, emitted virtual items, and mapping diagnostics. The object is omitted when no processing work was recorded, so existing templates remain compatible.
- Metadata: Access
- Structure: A nested list containing
-
{{processing}}: Engine State & Telemetry. Provides insight into the internal execution environment during the task. It includes data on memory allocation peaks, active worker threads, and non-blocking diagnostic warnings.- Access: Access properties directly via dot-notation (e.g.,
{{processing.memory_peak_mb}}). Use this to monitor system health and resource consumption during heavy mapping cycles.
- Access: Access properties directly via dot-notation (e.g.,
-
{{watch}}: Change Tracking Data. Specifically available for thewatchevent kind. It contains a diff-style object showing exactly which groups or channels were added, removed, or modified compared to the previous state.- Access: Iterate over the change sets using loops. Common keys include
added,removed, andmodified. - Example: Use
{{#each watch.added}} • {{name}} {{/each}}to list all new channels detected in the monitored groups.
- Access: Iterate over the change sets using loops. Common keys include
-
{{recording}}: DVR Lifecycle Data. Available forrecording_started,recording_completed, andrecording_failed. Common fields areprogramme_title,channel,effective_start,effective_end,visibility,output_filename, andfailure_reasonfor failed recordings. -
{{disk}}: Disk Alert Data. Available forsystem.disk.alert:level,percent,total_bytes,used_bytes,free_bytes.
Recording lifecycle notifications are global-channel notifications. Tuliprox sends them for shared recordings, legacy administrator recordings, and built-in administrator private recordings. Private recordings owned by regular users are suppressed.
Template Examples
Telegram (Markdown Report):
*🔄 Playlist Update Report*
{{#each stats}}
*📥 Source Stats*
{{#each inputs}}
• *{{name}}* (`{{type}}`)
⏱️ Took: `{{took}}` | ❌ Errors: `{{errors}}`
📊 `{{raw.groups}}`/`{{raw.channels}}` ➔ *`{{processed.groups}}`*/*`{{processed.channels}}`*
{{/each}}
{{/each}}
Discord (Complex Embed):
{
"content": "Tuliprox Notification",
"embeds": [{
"title": "Event: {{kind}}",
"description": "{{message}}",
"color": 3447003,
"fields": [
{{#each stats}}
{ "name": "Source {{@index}}", "value": "Processed {{#each inputs}}{{name}} {{/each}}", "inline": false }
{{/each}}
],
"footer": { "text": "Reported at {{timestamp}}" }
}]
}
5.3 Disk-Space Alerts (messaging.disk_alert)
Tuliprox monitors the free space on the current working directory's mount and notifies you via the
standard messaging channels (Telegram, Discord, Pushover, REST) before the disk fills up. The feature is
opt-in: no disk_alert block means no monitoring and no notifications.
How it works: A background loop in backend/app/src/api/sys_usage.rs ticks every 2 seconds (fixed
sampling interval — see SYSTEM_USAGE_INTERVAL in that file, not configurable) and compares the
current working directory's percent-used against two thresholds. The DiskAlertMonitor state machine
(backend/app/src/api/sys_usage.rs::DiskAlertMonitor) decides when to emit a DiskAlert:
| Level | Default Trigger | Operator Action |
|---|---|---|
Normal |
< warn_percent |
All good. |
Warn |
≥ warn_percent (default 80.0) |
Plan a cleanup. |
Critical |
≥ critical_percent (default 95.0) |
Cleanup is urgent; stream generation may fail. |
A DiskAlert is emitted whenever the level is non-Normal and either:
- the level just changed (Normal → Warn, Warn → Critical, or any step back down), or
- the disk has stayed in the same alert state for at least
repeat_interval_secsseconds since the previous notification.
This periodic re-notification is intentional: a long-running full-disk situation that nobody fixes
would otherwise go silent after the first transition, defeating the purpose of the feature. With the
default repeat_interval_secs: 3600 (1 hour), a stuck-at-87% disk produces one Warn notification per
hour instead of a single one-shot alert.
messaging:
notify_on: [ "info", "stats", "error", "watch" ]
# Disk-space alerts are sent through the channels above. Configure the
# thresholds and the re-arm interval here. The 2-second sampling tick is
# internal and not configurable.
disk_alert:
# Percent-used at which the Warn level is reached. Default: 80.0.
# Must be in [0, 100].
warn_percent: 80.0
# Percent-used at which the Critical level is reached. Default: 95.0.
# Must be in (warn_percent, 100].
critical_percent: 95.0
# Re-arm interval in seconds. While the disk stays in the same alert
# state, the alert is re-sent after this many seconds. Default: 3600
# (1 hour). This is NOT the sampling interval — sampling is fixed at
# 2 seconds and is not currently configurable.
repeat_interval_secs: 3600
telegram:
bot_token: "<TOKEN>"
chat_ids: [ "<CHAT_ID>" ]
templates:
# Custom templates per level. Plain-text defaults are used if omitted.
disk_alert_warn: "⚠️ Disk is {{percent}}% full (Warn threshold)."
disk_alert_critical: "🚨 Disk is {{percent}}% full — cleanup now!"
disk_alert_normal: "✅ Disk back to normal at {{percent}}%."
Note on
mounts: Earlier drafts of this section described amountsconfig field. The current implementation has no such field — the disk probe is hard-coded to sample the current working directory's mount (seeDiskProbe::for_cwd()inbackend/app/src/api/sys_usage.rs). If you need to monitor a different path, change the working directory of the Tuliprox process.
Available Handlebars Variables (all disk_alert_* templates):
{{level}}— The new level after the transition (Warn,Critical, orNormal).{{percent}}— Current percent-used, rounded to one decimal place.{{free_bytes}}/{{total_bytes}}/{{used_bytes}}— Raw byte counts for the sampled mount.{{timestamp}}— Event time in UTC (ISO 8601 / RFC3339).
Available template keys per channel (Handlebars): disk_alert_warn, disk_alert_critical,
disk_alert_normal. If a template is not configured for a level, Tuliprox falls back to a clear plain-text
default line, e.g.:
[WARN] Disk is 82.3% full (free=3.4GB total=19.0GB).Note: Disk alerts are sent through every configured messaging channel. If you want a channel to ignore disk alerts, simply do not declare the matchingdisk_alert_*template key for it and it will still receive the plain-text default. Use the standardnotify_onlist to control which event kinds are sent at all.
5.4 Webhook Signing
Set rest.signing_secret and every request carries:
X-Tuliprox-Timestamp: 1700000000
X-Tuliprox-Signature: sha256=<hex>
The signature is HMAC-SHA256 over {timestamp}.{body} using the shared secret. The timestamp is inside the signed
payload, so a captured request cannot be replayed later with a fresh timestamp header. Verify it by recomputing the
HMAC and comparing in constant time, and reject requests whose timestamp is too far from your clock.
5.5 Running a Local Program
The command channel runs a local program with the event JSON on stdin, plus TULIPROX_EVENT_ID,
TULIPROX_EVENT_SEVERITY and TULIPROX_EVENT_TITLE in the environment.
messaging:
command:
program: "/config/scripts/notify.sh"
args: [ "--from-tuliprox" ]
timeout_secs: 30
The program is executed directly, never through a shell, so there are no quoting rules to get wrong and event content cannot be interpreted as shell syntax.
Security: this runs arbitrary code as the tuliprox process user. It is opt-in and never configured by default. Treat the script as part of your trusted computing base.
5.6 Secrets in the Web UI
Channel secrets - the Telegram bot token, Pushover token and user key, ntfy and Gotify tokens, the REST signing
secret, and any Authorization-style REST header - are masked as ******** when the configuration is read back
through the API. Leaving a masked value untouched when you save keeps the stored secret; replacing it writes the new
one through.
6. Video & Web Search (video)
Optional video-related behaviors, mostly utilized by the Web UI.
video:
web_search: "https://www.imdb.com/search/title/?title={}"
extensions: [ "mkv", "mp4", "avi", "ts", "webm" ]
download:
directory: /tmp/tuliprox_downloads
headers:
User-Agent: "AppleTV/tvOS/9.1.1"
Accept: "video/*"
organize_into_directories: true
episode_pattern: '.*(?P<episode>[Ss]\d{1,2}(.*?)[Ee]\d{1,2}).*'
web_search: A template URL used in the Web UI to quickly search for a movie title (replaces{}with the title).extensions: Defines which file endings Tuliprox categorizes as VOD/Video content when transforming M3U to Xtream.download: Configuration for the Web UI download and recording manager.directory: Where downloaded files and recordings are saved.headers(optional): Custom HTTP headers used for the download request. This is useful for bypassing basic user-agent filters or setting specific media types.organize_into_directories: If true, Tuliprox automatically creates neat subfolders for series.episode_pattern: Crucial for the directory organization. It uses the mandatory Named Capture Group(?P<episode>...)in the Regex to identify and strip the episode identifier (e.g.,S01E01) from the filename, ensuring all episodes of a show land in the same base-show folder.download_priority: Default provider priority for VOD/series/episode downloads. Lower values mean higher priority.recording_priority: Default provider priority for live recordings. Lower values mean higher priority.reserve_slots_for_users: Keeps provider headroom for normal foreground users before background-priority transfers may consume the last slots.max_background_per_provider: Limits how many background-priority transfers may run in parallel against one provider.retry_backoff_initial_secs: Initial retry delay for transient download/recording failures.retry_backoff_multiplier: Growth factor applied to each later retry delay.retry_backoff_max_secs: Maximum retry delay once the backoff curve reaches its cap.retry_backoff_jitter_percent: Randomizes retry delays to avoid retry spikes after shared upstream problems.retry_max_attempts: Maximum number of transient retries before a transfer is marked as failed.
Tuliprox handles these transfers like provider-bound background streams:
- They respect provider limits, user priorities, and connection preemption instead of bypassing normal stream capacity.
- Waiting for provider capacity is notify-based, not polling-based.
- The Web UI loads an initial transfer snapshot and then stays synchronized through websocket updates.
- Changes to
video.downloadparticipate in hot config reloads. The background scheduler restarts and active transfers are
re-queued so they continue under the updated download configuration. - RBAC integration is explicit:
download.readallows opening the downloads view and receiving transfer snapshots.download.writeallows queueing, pausing, cancelling, retrying, and removing transfers.recording.readallows opening DVR task, quota, library, and recurring-rule views.recording.writeallows creating, editing, cancelling, deleting, and managing DVR tasks and rules.
- Persisted queue recovery is tolerant of corruption. If
downloads_state.jsoncannot be deserialized,
Tuliprox renames it to a timestamped*_corrupt.*.jsonbackup and starts with an empty transfer queue instead of aborting server boot.
6.1 DVR Runtime Files
The DVR runtime keeps durable state under storage_dir:
downloads_state.json: queued, scheduled, active, and finished downloads and recordings.recording_rules.json: recurring recording rules and tombstones.
Live recordings use a partial-file lifecycle. The worker writes to <filename>.partial and renames it to the final
path only after ffmpeg exits successfully and the final path is still free.
See also: the full DVR Operator Reference — configuration reference, directory layout, filename placeholders, lifecycle / restart, quota charge-by-state, disk admission, safe deletion, authorization matrix, identity-registry bootstrap, token refresh, deprecated
/file/record, REST + WebSocket surface, conflict preview, recurring-rule matching + DST + reconciliation, at-most-once notification protocol, migration checklist, and the 32-scenario acceptance sweep.
6.1.1 Filename placeholders
The filename template supports these placeholders (filename only — never the directory):
| Placeholder | Resolves to |
|---|---|
{channel} |
Channel name |
{program_title} |
Programme title (sanitized) |
{start_time} |
UTC YYYY-MM-DDTHH-MM |
{end_time} |
UTC YYYY-MM-DDTHH-MM |
{episode} |
Episode identifier extracted by episode_pattern |
{owner} |
The owner principal id (user:<id> or legacy:admin) |
Security:
{owner}is allowed only in the filename template, never in directory templates. Directory templates are resolved against the caller's identity, so the directory part is intrinsically owner-scoped.
6.1.2 Authorization matrix
The DVR layer runs an additional authorization pass on top of recording.read / recording.write.
| Visibility | Owner | Admin (builtin:admin) |
Foreign user | Notes |
|---|---|---|---|---|
private |
read + write + delete | read + write + delete | — | Foreign reads return 404 |
shared |
— | read + write + delete | read | Only admins create shared recordings |
legacy |
— (orphan) | read + write + delete | — | Created by the deprecated /file/record |
6.1.3 Identity bootstrap
If the users table is empty on first boot:
- Tuliprox reads
TULIPROX_BOOTSTRAP_ADMINfrom the environment. - The built-in
builtin:adminrole is assigned to that user. - The bootstrap admin must use
POST /api/v1/auth/loginto obtain a JWT. - From that point on, the admin creates additional users via
POST /api/v1/users. - If
TULIPROX_BOOTSTRAP_ADMINis unset and the table is empty, the server fails closed at boot. - The bootstrap admin cannot be deleted while it is the sole
builtin:adminmember.
6.1.4 Token refresh on schema bump
When the JWT schema version is bumped (a new field is added), existing tokens are rejected with
401 Unauthorized and an X-Token-Refresh: required response header. The frontend automatically calls
POST /api/v1/auth/refresh to mint a new token. The wire code is recording_token_refresh_required so the
toastr surfaces a stable, translatable message. Operators upgrading across a schema-bump release do not need
to do anything manually.
6.1.5 Deprecated /file/record
POST /api/v1/file/record is the legacy recording endpoint. It is still functional and admin-gated, but
returns a recording_forbidden error for non-admin principals and is scheduled for removal in the next
major version. New code should use POST /api/v1/recording/tasks with a CreateRecordingTaskBody payload
(see REST API cookbook).
Note: The named capture group
(?P<episode>...)is mandatory for this to function correctly.Example:
.*(?P<episode>[Ss]\d{1,2}(.*?)[Ee]\d{1,2}).*
7. Outgoing Proxy (proxy)
If Tuliprox itself must operate behind a corporate proxy or VPN (e.g., a Gluetun WireGuard container):
proxy:
url: socks5://192.168.1.6:8123
username: "opt_user"
password: "opt_password"
Setting this forces every outgoing request (Playlist downloads, TMDB API calls, FFprobe stream analysis, and Reverse-Proxy Video Streaming) through this proxy.
8. IP-Check (ipcheck)
To verify in the Web UI which public IP Tuliprox is currently using (crucial when verifying VPN routing or diagnosing geo-blocks), Tuliprox queries external detection APIs. This ensures that your traffic is actually routed through the intended gateway or VPN tunnel.
ipcheck:
url: "[https://api64.ipify.org?format=json](https://api64.ipify.org?format=json)" # Generic URL for both IP versions
url_ipv4: "[https://ipinfo.io/ip](https://ipinfo.io/ip)" # Dedicated IPv4 fetch
url_ipv6: "[https://v6.ident.me](https://v6.ident.me)" # Dedicated IPv6 fetch
pattern_ipv4: '(?:\d{1,3}\.){3}\d{1,3}' # Optional Regex for extraction
pattern_ipv6: '([0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}'
| Parameter | Type | Technical Impact |
|---|---|---|
url |
String | A generic endpoint that may return both IPv4 and IPv6 in a single response (often JSON). |
url_ipv4 |
String | A dedicated URL to fetch only the public IPv4 address. |
url_ipv6 |
String | A dedicated URL to fetch only the public IPv6 address. |
pattern_ipv4 |
Regex | Optional regex pattern to extract the IPv4 string from complex API responses (e.g., if the API returns a full JSON object). |
pattern_ipv6 |
Regex | Optional regex pattern to extract the IPv6 string from complex API responses. |
Technical Background
- VPN Validation: This feature is primarily used to confirm that Tuliprox is successfully using a VPN or Proxy. If the displayed IP in the Web UI matches your home ISP instead of your VPN provider, your routing is likely misconfigured.
- Regex Extraction: If your preferred IP-API returns data in a format like
{"ip": "1.2.3.4", "city": "Berlin"}, you can use thepattern_ipv4to isolate just the IP address for the Tuliprox UI. - Execution: The check is performed periodically or on-demand when accessing the Dashboard to provide real-time connectivity status.
9. HDHomeRun Emulation (hdhomerun)
Deep-Dive Feature: Tuliprox can masquerade on the local network as a physical SiliconDust HDHomeRun DVB-C/S/T network tuner. Media servers like Plex, Jellyfin, Emby, or TVHeadend will automatically discover Tuliprox via UPnP as a real hardware antenna and ingest Live-TV natively into their Live-DVR systems.
Tuliprox utilizes standardized UPnP/SSDP (UDP Port 1900) for broad compatibility and the proprietary SiliconDust protocol (UDP Port 65001) for compatibility with official HDHomeRun tools.
hdhomerun:
enabled: true
auth: false # If true, lineup.json requires Basic Auth (using the assigned user's credentials)
devices:
- name: hdhr1 # MUST match the 'device' name in the source.yml hdhomerun output!
tuner_count: 4 # Number of concurrent streams the client thinks are available
port: 5004 # Unique TCP Port for this specific virtual tuner API
device_id: "107ABCDF" # 8-char hex code (Port 65001). If left blank, Tuliprox generates a valid one with correct checksums.
device_udn: "uuid:..." # Unique Device Name (Port 1900). Recommended to leave blank for auto-generation.
friendly_name: "Tuliprox Living Room"
Configuration Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
enabled |
Bool | false |
Master switch for the entire emulation engine. |
auth |
Bool | false |
Requires HTTP Basic Auth for the /lineup.json endpoint using the assigned user's credentials. |
devices |
List | [] |
A list of virtual HDHomeRun devices to emulate. |
9.1. Device-Specific Fields
| Parameter | Type | Default | Technical Impact & Background |
|---|---|---|---|
name |
String | Required | Unique internal identifier. Must match the device field in your source.yml target mapping. |
tuner_count |
Int | 1 |
Number of virtual tuners reported to the client. Defines how many concurrent streams Plex/Emby thinks the "hardware" can handle. |
port |
Int | API+1 |
TCP port for the HTTP API (/device.xml, /lineup.json). Each virtual device must have a unique port. |
friendly_name |
String | (Auto) |
The display name in client applications (e.g., "Tuliprox Living Room"). |
device_id |
Hex | (Auto) |
8-char hex ID for SiliconDust protocol (Port 65001). Tuliprox automatically corrects invalid IDs by calculating the required checksum. |
device_udn |
UUID | (Auto) |
Unique Device Name for UPnP/SSDP (Port 1900). Recommended to leave blank for auto-generation. |
manufacturer |
String | SiliconDust |
Customizes the manufacturer string reported to clients. |
model_name |
String | HDTC-2US |
Mimics a specific hardware model for maximum compatibility with official apps. |
firmware_name |
String | hdhomerun3_atsc |
The firmware type reported during the discovery handshake. |
Note: Advanced metadata fields like
model_numberandfirmware_versioncan also be overridden but are safe to leave at their defaults to ensure the best "plug-and-play" experience with media servers.
9.2. Linking Devices to Playlists
The name of each device in config.yml must correspond to a device reference in a hdhomerun output target within
your source.yml.
Example source.yml snippet:
targets:
- name: my-tv-lineup
output:
- type: hdhomerun
device: hdhr1 # Link to the device defined above
username: local # The user whose credentials/playlist will be served
Discovery Protocols: A Technical Distinction
To satisfy both official SiliconDust hardware scanners and generic third-party UPnP discovery, Tuliprox handles two distinct layers:
device_id(Port 65001): Used by proprietary SiliconDust tools. It requires a specific hexadecimal format and checksum.device_udn(Port 1900): Used by the standard SSDP/UPnP protocol (e.g., by Plex or VLC). This identifies the device as a unique UUID on the network.
Pro-Tip: If Plex fails to find your device, ensure that the UDP ports 1900 and 65001 are not blocked by your firewall and that Tuliprox is running on the same network subnet as your media server.
Additional Information
Custom Stream Responses (Fallback Videos)
When a stream fails to load at the provider (HTTP 404/502) or a user reaches their connection limit, Tuliprox can
seamlessly
substitute a fallback info-video (as a .ts stream) instead of brutally closing the TCP connection. Dropping
connections often
causes hardware players or Smart TVs to freeze.
custom_stream_response_path: /home/tuliprox/resources
custom_stream_response_timeout_secs: 20
| Name | Type | Default | Technical Impact & Background |
|---|---|---|---|
custom_stream_response_path |
String | (Empty) |
Directory path where Tuliprox looks for exactly named .ts files. |
custom_stream_response_timeout_secs |
Int | 0 |
Hard timeout (in seconds) that forces the fallback video stream to terminate to prevent infinite bandwidth usage. 0 means the fallback loops endlessly until the user switches channels. |
Filenames searched for in the directory:
channel_unavailable.ts(Provider returns 404/502/Timeout)user_connections_exhausted.ts(User hit theirmax_connectionslimit)provider_connections_exhausted.ts(Provider has no free slots left)low_priority_preempted.ts(User was kicked by an Admin with higher priority)user_account_expired.ts(User'sexp_datereached)panel_api_provisioning.ts(Loops while a new Provider Account is generated via Panel API)hls_session_or_lease_expired.ts(Shared-HLS session or access lease expired; restart stream)
Note
: These Video files are all available in the docker image.
How to create your own fallback video stream:
You can simply convert an image with ffmpeg.
ffmpeg -y -nostdin -loop 1 -framerate 30 -i blank_screen.jpg -f lavfi \
-i anullsrc=channel_layout=stereo:sample_rate=48000 -t 10 -shortest -c:v libx264 \
-pix_fmt yuv420p -preset veryfast -crf 23 -x264-params "keyint=30:min-keyint=30:scenecut=0:bframes=0:open_gop=0" \
-c:a aac -b:a 128k -ac 2 -ar 48000 -mpegts_flags +resend_headers -muxdelay 0 -muxpreload 0 -f mpegts blank_screen.ts
The filename identifies the file inside the path custom_stream_response_path.
How it works:
Tuliprox searches the specified folder for exactly named .ts files. If found, they are looped back to the client. The
custom_stream_response_timeout_secs parameter hard-kills the fallback stream after X seconds to prevent infinite
bandwidth usage.