diff --git a/.cargo/config.toml b/.cargo/config.toml index 12f60697a..21de2fbae 100644 --- a/.cargo/config.toml +++ b/.cargo/config.toml @@ -7,8 +7,8 @@ linker = "aarch64-linux-gnu-gcc" [target.x86_64-pc-windows-gnu] linker = "x86_64-w64-mingw32-gcc" -[target.x86_64-apple-darwin] -linker = "x86_64-apple-darwin14-clang" # Requires osxcross +#[target.x86_64-apple-darwin] +#linker = "x86_64-apple-darwin14-clang" # Requires osxcross -[target.aarch64-apple-darwin] -linker = "aarch64-apple-darwin20.4-clang" # Requires osxcross +#[target.aarch64-apple-darwin] +#linker = "aarch64-apple-darwin20.4-clang" # Requires osxcross diff --git a/CHANGELOG.md b/CHANGELOG.md index ace9cc9a7..21544f9ce 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,10 @@ ## ⚠️ Breaking Changes +- **The Web UI WebSocket protocol is now version 4.** Playlist update completion messages carry the correlated + run ID and execution order instead of a bare status. Reload existing browser tabs after upgrading the server; + version-3 clients are rejected during the handshake rather than receiving incompatible update messages. + - **Smart EPG normalization now preserves XMLTV ID separators by default.** The default `normalize_regex` changed from `[^a-zA-Z0-9\-]` to `[^a-zA-Z0-9._\-]`. Existing configurations that explicitly set the former pattern keep the legacy separator-removal behavior; remove the override or use the new pattern to adopt the new default. @@ -112,6 +116,44 @@ effect on the next playlist update. Leaving every cluster unselected means no bouquet restriction; an individual cluster may intentionally have no selected groups while another cluster remains configured. +- **Per-cluster update quality guards for Xtream, Stalker, and M3U inputs.** Optional `update_quality` thresholds + compare Live, VOD, and Series item counts with the last accepted cluster and retain its data when a candidate falls + outside the configured range. Xtream retains its active cluster database and categories; Stalker retains its active + manifest entry and generation. M3U evaluates the clusters within one downloaded document and combines accepted + candidates with retained clusters before persistence. Quality rejections are reported as partial updates, separately + from technical failures. Omitted thresholds default to `0`, disabling the guard; `100` requires an unchanged item + count. The Source Editor exposes these settings through the shared range-slider component. + +- **Input-focused playlist updates in the Web UI.** The Update tab now provides compact input cards with consistent + status badges, last-update times, and selection of affected targets by stable ID. Available actions follow each + input's capabilities: Xtream, Stalker, and M3U offer Update, Refresh, and Force Update; Plex offers Update and Refresh; + Library offers Rescan. Update may reuse a valid cache, Refresh bypasses cache reads while enforcing quality, and + Force Update bypasses both for the selected input without changing saved configuration. Selected targets rebuild + with all their required inputs; the bulk action remains separate. Existing target-name API requests remain supported. + +- **Collapsible pipeline transparency instead of a permanent update log.** Update Details separates input/cluster + outcomes from target processing and output status. It shows available quality thresholds and evaluations, actual + cache/provider origin, effective update mode, and correlated run details. Target summaries use the configured + processing order and label Filter/Rename/Mapping counts as configured rules, not runtime hits. Technical failures, + quality rejections, and unavailable facts remain distinct. A running target rebuild keeps its run's status and + details until completion, even when a follow-up update has already been queued. + +- **Reload-safe input and cluster status.** Completed input results and one latest snapshot per cluster use the + existing `status.json`, without a separate history or browser persistence. Available snapshots retain the run's + policy, actual data source, historical quality-guard threshold, quality evaluation, active count, and technical + outcome. Current correlated runs take priority over persisted snapshots; older status files remain readable and + missing facts stay unknown. Browser reloads can also restore in-progress state from the running server without + persisting queued or updating states as completed results. + +- **Library rescans rebuild selected targets and expose catalog and scan facts separately.** Rescan completes the + filesystem scan before rebuilding targets; scan errors prevent the rebuild. A complete, error-free empty scan is + authoritative, including deletion of the last item, while other required inputs remain available to the target's + configured filters. Library details show Movies, Series, Episodes, and Total items from the stored catalog; Episodes + counts stored episode records while Total items remains the movie/series-entry count. Last rescan shows the actual + scanned-file, media-group, added, updated, removed, and error counts for the correlated run, including reported scan + failures. Catalog counts survive reloads; unavailable rescan metrics remain unknown rather than being inferred from + the catalog. Library cards no longer present their catalog as a generic provider/cache acquisition. + - **Target filters can run during processing or immediately before persistence.** The existing scalar `filter` syntax remains the `processing` stage. The staged map accepts optional `processing` and `persist` filters; `persist` sees the fully finalized state after EPG processing, mappings, merge, deduplication, sorting, numbering, and counters. @@ -934,6 +976,15 @@ ## 🐛 Fixes +- **Playlist update status reads no longer block an async request worker on filesystem I/O.** The existing input + status reader runs on the blocking pool; its status format and reload projection are unchanged. +- **Xtream disk processing preserves successful cluster downloads when another cluster request fails.** Existing + per-cluster quality checks still apply, while the failed cluster retains its previous data. +- **Stalker acquisition failures now identify the affected requested clusters.** Portal, client, storage and handshake + errors mark only requested, non-skipped clusters as failed instead of leaving their status unknown. +- **The Library catalog status initializes storage before its first read.** A fresh installation reports zero catalog + counts, while initialization failures and corrupt catalog data remain errors rather than appearing empty. + - **Trakt curation now requires an explicitly configured Client ID.** Tuliprox no longer bundles or falls back to a shared Client ID. Blank or header-invalid `trakt.api.api_key` values now produce one target-scoped warning and skip only optional Trakt curation without making an HTTP request; other target processing continues. Trakt `401`, `403`, diff --git a/Cargo.toml b/Cargo.toml index 323de1c61..dd9900430 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -89,6 +89,16 @@ lto = false incremental = true codegen-units = 256 +[profile.dev-slim] +inherits = "dev" +opt-level = 0 +strip = "none" +lto = "off" +debug = "line-tables-only" + +[profile.dev-slim.package."*"] +debug = false + [profile.release.package.frontend] opt-level = 'z' # Size optimization for frontend diff --git a/backend/app/src/api/auth_middleware.rs b/backend/app/src/api/auth_middleware.rs index 722db1fa0..f5b4916de 100644 --- a/backend/app/src/api/auth_middleware.rs +++ b/backend/app/src/api/auth_middleware.rs @@ -188,7 +188,7 @@ pub async fn require_permission( next.run(request).await } -fn check_permission( +pub(crate) fn check_permission( app_state: &Arc, claims: &Claims, client_ip: Option<&str>, diff --git a/backend/app/src/api/endpoints/api_playlist_utils.rs b/backend/app/src/api/endpoints/api_playlist_utils.rs index dff5f8dc8..55f4c77f0 100644 --- a/backend/app/src/api/endpoints/api_playlist_utils.rs +++ b/backend/app/src/api/endpoints/api_playlist_utils.rs @@ -145,23 +145,24 @@ pub(in crate::api::endpoints) async fn get_playlist_for_input( // TODO refactor let stalker_cluster = stalker_cluster(cluster); let client = app_state.http_client.load(); - let (groups, errors, _, partial) = download_stalker_playlist( + let fetch = download_stalker_playlist( &app_state.app_config, client.as_ref(), input, Some(&[stalker_cluster]), crate::processing::processor::StalkerRefreshMode::ServerSlice, true, + shared::model::UpdateQualityPolicy::Enforce, ) .await; - if groups.is_empty() { - if partial { + if fetch.groups.is_empty() { + if fetch.partial { return stalker_refresh_pending_response(accept); } - if errors.is_empty() { + if fetch.errors.is_empty() { return json_or_bin_response(accept, &Vec::::new()).into_response(); } - let error_strings: Vec = errors.iter().map(ToString::to_string).collect(); + let error_strings: Vec = fetch.errors.iter().map(ToString::to_string).collect(); return (axum::http::StatusCode::BAD_REQUEST, axum::Json(json!({"error": error_strings.join(", ")}))) .into_response(); } @@ -169,7 +170,8 @@ pub(in crate::api::endpoints) async fn get_playlist_for_input( let web_ui_path = config.web_ui.as_ref().and_then(|web_ui| web_ui.path.as_ref()).map_or("", String::as_str); let resource_url = concat_path_leading_slash(web_ui_path, "api/v1/playlist/resource"); let encrypt_secret = app_state.get_encrypt_secret(); - let channels: Vec = groups + let channels: Vec = fetch + .groups .iter() .flat_map(|group| group.channels.iter()) .map(UiPlaylistItem::from) @@ -199,7 +201,7 @@ pub(in crate::api::endpoints) async fn get_playlist_for_custom_provider( (playlist, errors, false) } InputPersistence::Xtream => { - let (pl, err, _) = xtream::download_xtream_playlist( + let fetch = xtream::download_xtream_playlist_direct( &app_state.app_config, client, &app_state.event_manager, @@ -207,7 +209,7 @@ pub(in crate::api::endpoints) async fn get_playlist_for_custom_provider( Some(&[cluster]), ) .await; - (pl, err, false) + (fetch.groups, fetch.errors, fetch.partial) } InputPersistence::Library => { return ( @@ -225,20 +227,19 @@ pub(in crate::api::endpoints) async fn get_playlist_for_custom_provider( } InputPersistence::Stalker => { let stalker_cluster = stalker_cluster(cluster); - // The Stalker processor mirrors the M3U/Xtream path: it returns - // `PlaylistGroup`s for the requested cluster only. The third - // tuple element (`use_disk_based_processing`) is irrelevant for - // a live preview — we always surface the in-memory items here. - let (groups, errs, _, partial) = download_stalker_playlist( + // A live preview materializes the requested active cluster; provider-side + // persistence remains an implementation detail of the named fetch result. + let fetch = download_stalker_playlist( &app_state.app_config, client, input, Some(&[stalker_cluster]), crate::processing::processor::StalkerRefreshMode::ServerSlice, true, + shared::model::UpdateQualityPolicy::Enforce, ) .await; - (groups, errs, partial) + (fetch.groups, fetch.errors, fetch.partial) } }; if result.is_empty() { diff --git a/backend/app/src/api/endpoints/library_api.rs b/backend/app/src/api/endpoints/library_api.rs index 5d2b87fbf..af82f9c00 100644 --- a/backend/app/src/api/endpoints/library_api.rs +++ b/backend/app/src/api/endpoints/library_api.rs @@ -53,7 +53,7 @@ async fn scan_library( permit, ); - (axum::http::StatusCode::ACCEPTED, axum::Json(OperationRunAccepted {})).into_response() + (axum::http::StatusCode::ACCEPTED, axum::Json(OperationRunAccepted::default())).into_response() } /// Gets Library status @@ -61,42 +61,32 @@ async fn get_library_status( axum::extract::State(app_state): axum::extract::State>, ) -> axum::response::Response { let config_snapshot = app_state.app_config.config.load(); - if let Some(config) = config_snapshot.library.as_ref() { - if config.enabled { - let client = app_state.http_client.load_full().as_ref().clone(); - // Get statistics from processor - let processor = LibraryProcessor::new( - config.clone(), - config_snapshot.metadata_update.as_ref(), - client, - &config_snapshot.storage_dir, - ); - let entries = processor.get_all_entries().await; - - let movies = entries.iter().filter(|e| e.metadata.is_movie()).count(); - let series = entries.iter().filter(|e| e.metadata.is_series()).count(); - - let response = LibraryStatus { - enabled: true, - total_items: entries.len(), - movies, - series, - path: Some( - resolve_metadata_storage_path( - config_snapshot.metadata_update.as_ref(), - &config_snapshot.storage_dir, - ) - .to_string_lossy() - .to_string(), - ), - }; - - return axum::Json(response).into_response(); + match read_library_status(&config_snapshot, app_state.http_client.load_full().as_ref().clone()).await { + Ok(status) => axum::Json(status).into_response(), + Err(err) => { + log::error!("Failed to read Library catalog status: {err}"); + axum::http::StatusCode::INTERNAL_SERVER_ERROR.into_response() } } +} - let response = LibraryStatus::default(); - axum::Json(response).into_response() +// Both counts and the exposed path refer to the scanner's canonical catalog. +async fn read_library_status( + config: &tuliprox_core::model::Config, + client: reqwest::Client, +) -> std::io::Result { + let Some(library) = config.library.as_ref().filter(|library| library.enabled) else { + return Ok(LibraryStatus::default()); + }; + let processor = + LibraryProcessor::new(library.clone(), config.metadata_update.as_ref(), client, &config.storage_dir); + let mut status = processor.catalog_status().await?; + status.path = Some( + resolve_metadata_storage_path(config.metadata_update.as_ref(), &config.storage_dir) + .to_string_lossy() + .into_owned(), + ); + Ok(status) } async fn get_thumbnail( @@ -133,6 +123,57 @@ async fn get_thumbnail( serve_thumbnail_hash(&storage, &id, etag, &headers).await } +#[cfg(test)] +mod tests { + use super::*; + + #[tokio::test] + async fn library_status_read_uses_canonical_catalog_with_episodes_without_a_new_scan() { + use crate::library::{EpisodeMetadata, MediaMetadata, MetadataCacheEntry, SeriesMetadata}; + use shared::model::LibraryConfigDto; + use tuliprox_core::model::{Config, LibraryConfig, MetadataUpdateConfig}; + let temp = tempfile::tempdir().unwrap(); + let config = Config { + storage_dir: temp.path().to_string_lossy().into_owned(), + library: Some(LibraryConfig::from(&LibraryConfigDto { enabled: true, ..LibraryConfigDto::default() })), + metadata_update: Some(MetadataUpdateConfig { + cache_path: "canonical".into(), + ..MetadataUpdateConfig::default() + }), + ..Config::default() + }; + let storage = + MetadataStorage::new(resolve_metadata_storage_path(config.metadata_update.as_ref(), &config.storage_dir)); + storage.initialize().await.unwrap(); + storage + .store(&MetadataCacheEntry::new( + "/not-scanned".into(), + 0, + 0, + MediaMetadata::Series(SeriesMetadata { + episodes: Some(vec![EpisodeMetadata::default(); 3]), + number_of_episodes: 999, + ..SeriesMetadata::default() + }), + )) + .await + .unwrap(); + let status = read_library_status(&config, reqwest::Client::new()).await.unwrap(); + assert_eq!((status.movies, status.series, status.episodes, status.total_items), (0, 1, 3, 1)); + assert_eq!(status.path.as_deref(), temp.path().join("canonical").to_str()); + let response = axum::Json(status.clone()).into_response(); + let json: serde_json::Value = + serde_json::from_slice(&axum::body::to_bytes(response.into_body(), usize::MAX).await.unwrap()).unwrap(); + assert_eq!(json["episodes"], 3); + assert_eq!(read_library_status(&config, reqwest::Client::new()).await.unwrap(), status); + assert!(!json.to_string().contains("files_scanned")); + assert_eq!( + read_library_status(&Config::default(), reqwest::Client::new()).await.unwrap(), + LibraryStatus::default() + ); + } +} + async fn serve_thumbnail_hash( storage: &MetadataStorage, hash: &str, diff --git a/backend/app/src/api/endpoints/user_api.rs b/backend/app/src/api/endpoints/user_api.rs index 40ac43db9..e54d39b07 100644 --- a/backend/app/src/api/endpoints/user_api.rs +++ b/backend/app/src/api/endpoints/user_api.rs @@ -68,24 +68,23 @@ async fn playlist_categories( } let target_name = &target.name; let xtream_stream = if target.has_output(TargetType::Xtream) { - let config = &app_state.app_config.config.load(); let live_categories = if user.allows_cluster(XtreamCluster::Live) { get_categories_from_xtream( - xtream_get_playlist_categories(config, target_name, XtreamCluster::Live).await, + xtream_get_playlist_categories(&app_state.app_config, target_name, XtreamCluster::Live).await, ) } else { Vec::new() }; let vod_categories = if user.allows_cluster(XtreamCluster::Video) { get_categories_from_xtream( - xtream_get_playlist_categories(config, target_name, XtreamCluster::Video).await, + xtream_get_playlist_categories(&app_state.app_config, target_name, XtreamCluster::Video).await, ) } else { Vec::new() }; let series_categories = if user.allows_cluster(XtreamCluster::Series) { get_categories_from_xtream( - xtream_get_playlist_categories(config, target_name, XtreamCluster::Series).await, + xtream_get_playlist_categories(&app_state.app_config, target_name, XtreamCluster::Series).await, ) } else { Vec::new() @@ -160,8 +159,7 @@ async fn save_playlist_bouquet( if user.permission_denied(&app_state.app_config) { return axum::http::StatusCode::FORBIDDEN.into_response(); } - let config = &app_state.app_config.config.load(); - match save_user_bouquet(config, &target.name, &username, &bouquet).await { + match save_user_bouquet(&app_state.app_config, &target.name, &username, &bouquet).await { Ok(()) => { return axum::http::StatusCode::OK.into_response(); } diff --git a/backend/app/src/api/endpoints/v1_api_playlist.rs b/backend/app/src/api/endpoints/v1_api_playlist.rs index 8a06edf03..d16865dbb 100644 --- a/backend/app/src/api/endpoints/v1_api_playlist.rs +++ b/backend/app/src/api/endpoints/v1_api_playlist.rs @@ -6,7 +6,7 @@ use crate::{ create_api_proxy_user, json_or_bin_response, resource_response, try_option_bad_request, try_result_bad_request, try_unwrap_body, ResourceFetchPolicy, }, - auth_middleware::permission_layer, + auth_middleware::{check_permission, permission_layer, VerifiedClaims}, endpoints::{ api_playlist_utils::{ get_playlist_for_custom_provider, get_playlist_for_input, get_playlist_for_target, @@ -29,7 +29,8 @@ use crate::{ iptv::{stalker::client::validate_public_playable_url, xtream}, model::{ parse_xmltv_for_web_ui_from_file, parse_xmltv_for_web_ui_from_url, AppConfig, ConfigInput, ConfigInputFlags, - ConfigInputOptions, EpgSource, EpgSourceType, IcsDummyPolicy, InputSource, + ConfigInputOptions, ConfigInputUpdateQuality, EpgSource, EpgSourceType, IcsDummyPolicy, InputSource, + ProcessTargets, SourcesConfig, }, processing::{ epg::get_input_raw_epg_file_path, @@ -53,9 +54,13 @@ use shared::{ error::TuliproxError, foundation::{get_filter_detailed, Filter, ValueProvider}, model::{ - permission::Permission, stalker::StalkerStreamKind, EpgChannel, InputType, OperationRunAccepted, - PlaylistEpgRequest, PlaylistItem, PlaylistRequest, PlaylistUrlResolveRequest, ProxyType, TargetType, - UiPlaylistItem, VirtualId, XtreamCluster, + permission::Permission, stalker::StalkerStreamKind, EpgChannel, InputPlaylistUpdateStatusDto, InputType, + InputUpdateAction, InputUpdateCapabilities, InputUpdateRequest, OperationRunAccepted, + PersistedPlaylistUpdateClusterState, PersistedPlaylistUpdateClusterStatusDto, + PersistedPlaylistUpdateInputResult, PlaylistEpgRequest, PlaylistItem, PlaylistRequest, + PlaylistUpdateRequestDto, PlaylistUpdateRequestPayload, PlaylistUpdateRunId, PlaylistUpdateState, + PlaylistUpdateStatusDto, PlaylistUrlResolveRequest, ProxyType, TargetType, UiPlaylistItem, VirtualId, + XtreamCluster, }, utils::{concat_path_leading_slash, deobfuscate_text, sanitize_sensitive_info, Internable}, }; @@ -63,6 +68,26 @@ use std::{path::Path, str::FromStr, sync::Arc}; use tokio_stream::StreamExt; use url::Url; +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum ManualUpdateEnqueueError { + Busy, + Unavailable, +} + +fn enqueue_manual_playlist_update( + sender: &tokio::sync::mpsc::Sender, + targets: Arc, + input_action: Option, +) -> Result { + let permit = sender.try_reserve().map_err(|error| match error { + tokio::sync::mpsc::error::TrySendError::Full(()) => ManualUpdateEnqueueError::Busy, + tokio::sync::mpsc::error::TrySendError::Closed(()) => ManualUpdateEnqueueError::Unavailable, + })?; + let run_id = PlaylistUpdateRunId::generate(); + permit.send(crate::api::model::ManualPlaylistUpdateRequest { run_id: run_id.clone(), targets, input_action }); + Ok(run_id) +} + fn create_config_input_for_m3u(url: &str) -> ConfigInput { ConfigInput { id: 0, @@ -72,6 +97,7 @@ fn create_config_input_for_m3u(url: &str) -> ConfigInput { enabled: true, options: Some(ConfigInputOptions { flags: ConfigInputFlags::XtreamLiveStreamUsePrefix | ConfigInputFlags::ResolveBackground, + update_quality: ConfigInputUpdateQuality::default(), resolve_delay: shared::defaults::default_resolve_delay_secs(), probe_delay: shared::defaults::default_probe_delay_secs(), probe_live_interval_hours: 120, @@ -93,6 +119,7 @@ fn create_config_input_for_xtream(username: &str, password: &str, host: &str) -> enabled: true, options: Some(ConfigInputOptions { flags: ConfigInputFlags::XtreamLiveStreamUsePrefix | ConfigInputFlags::ResolveBackground, + update_quality: ConfigInputUpdateQuality::default(), resolve_delay: shared::defaults::default_resolve_delay_secs(), probe_delay: shared::defaults::default_probe_delay_secs(), probe_live_interval_hours: 120, @@ -475,26 +502,49 @@ async fn load_epg_channels_for_input( async fn playlist_update( axum::extract::State(app_state): axum::extract::State>, - axum::extract::Json(targets): axum::extract::Json>, + claims: Option>, + axum::extract::Json(payload): axum::extract::Json, ) -> impl axum::response::IntoResponse + Send { - let user_targets = if targets.is_empty() { None } else { Some(targets) }; - let process_targets = app_state.app_config.sources.load().validate_targets(user_targets.as_ref()); + let request = payload.into_request(); + let input_action = match request.manual_input_update() { + Ok(action) => action, + Err(error) => { + return (axum::http::StatusCode::BAD_REQUEST, axum::Json(json!({"error": error}))).into_response() + } + }; + if input_action.is_some_and(|request| request.action == InputUpdateAction::Rescan) { + let config = app_state.app_config.config.load(); + if !config.library.as_ref().is_some_and(|library| library.enabled) { + return (axum::http::StatusCode::BAD_REQUEST, axum::Json(json!({"error": "Library is not enabled"}))) + .into_response(); + } + if config.web_ui.as_ref().and_then(|web| web.auth.as_ref()).is_some() { + let Some(axum::Extension(VerifiedClaims(claims))) = claims else { + return axum::http::StatusCode::UNAUTHORIZED.into_response(); + }; + if check_permission::<{ Permission::LibraryWrite as u32 }>(&app_state, &claims, None).is_err() { + return axum::http::StatusCode::FORBIDDEN.into_response(); + } + } + } + let process_targets = resolve_manual_playlist_update_targets(&app_state.app_config.sources.load(), &request); match process_targets { Ok(valid_targets) => { let valid_targets = Arc::new(valid_targets); - // Deduplicate rapid clicks: the channel has capacity 1, so at most one - // update is queued at any time. Additional requests while the channel - // is full are silently dropped — the pending run already covers them. - match app_state - .manual_update_sender - .try_send(crate::api::model::ManualPlaylistUpdateRequest { targets: valid_targets }) - { - Ok(()) => (axum::http::StatusCode::ACCEPTED, axum::Json(OperationRunAccepted {})).into_response(), - Err(tokio::sync::mpsc::error::TrySendError::Full(_)) => { - debug!("Manual playlist update deduplicated: an update is already pending or running"); - (axum::http::StatusCode::ACCEPTED, axum::Json(OperationRunAccepted {})).into_response() + match enqueue_manual_playlist_update(&app_state.manual_update_sender, valid_targets, input_action) { + Ok(run_id) => { + (axum::http::StatusCode::ACCEPTED, axum::Json(OperationRunAccepted::playlist_update(run_id))) + .into_response() } - Err(tokio::sync::mpsc::error::TrySendError::Closed(_)) => { + Err(ManualUpdateEnqueueError::Busy) => { + debug!("Manual playlist update rejected: another update is already queued"); + ( + axum::http::StatusCode::CONFLICT, + axum::Json(json!({"error": "Another playlist update is already queued; retry this request"})), + ) + .into_response() + } + Err(ManualUpdateEnqueueError::Unavailable) => { debug!("Manual playlist update rejected: worker channel closed (server shutting down)"); axum::http::StatusCode::SERVICE_UNAVAILABLE.into_response() } @@ -507,6 +557,156 @@ async fn playlist_update( } } +fn resolve_manual_playlist_update_targets( + sources: &SourcesConfig, + request: &PlaylistUpdateRequestDto, +) -> Result { + let manual_action = + request.manual_input_update().map_err(|error| TuliproxError::ConfigSource(error.to_string()))?; + let manual_input = if let Some(manual) = manual_action { + let input = sources + .inputs + .iter() + .find(|input| input.id == manual.input_id) + .ok_or_else(|| TuliproxError::ConfigSource(format!("No input found for id {}", manual.input_id)))?; + if !InputUpdateCapabilities::for_input_type(input.input_type, input.enabled).supports_action(manual.action) { + return Err(TuliproxError::ConfigSource("Manual action is not supported for this input".to_string())); + } + // Only the additive action wire form requires IDs. Legacy input_refresh + // requests retain the existing name resolution, including empty = all. + if request.input_action.is_some() && request.target_ids.is_none() { + return Err(TuliproxError::ConfigSource("Manual input actions require explicit target IDs".to_string())); + } + Some(input) + } else { + None + }; + let Some(target_ids) = request.target_ids.as_deref() else { + let targets = (!request.targets.is_empty()).then_some(&request.targets); + return sources.validate_targets(targets); + }; + if !request.targets.is_empty() { + return Err(TuliproxError::ConfigSource( + "Manual playlist update cannot contain both target names and target IDs".to_string(), + )); + } + if target_ids.is_empty() { + return Err(TuliproxError::ConfigSource( + "Manual playlist update target ID selection must not be empty".to_string(), + )); + } + + let mut targets = Vec::with_capacity(target_ids.len()); + let mut target_names = Vec::with_capacity(target_ids.len()); + for target_id in target_ids { + let Some(target) = sources.get_target_by_id(*target_id) else { + return Err(TuliproxError::ConfigSource(format!("No target found for id {target_id}"))); + }; + if manual_input.is_some_and(|input| { + !sources.sources.iter().any(|source| { + source.inputs.contains(&input.name) && source.targets.iter().any(|target| target.id == *target_id) + }) + }) { + return Err(TuliproxError::ConfigSource("Target does not belong to the selected input".to_string())); + } + targets.push(target.id); + target_names.push(target.name.clone()); + } + + Ok(ProcessTargets { + enabled: true, + inputs: sources.inputs.iter().map(|input| input.id).collect(), + targets, + target_names, + }) +} + +async fn read_playlist_update_input_status( + read: impl FnOnce() -> crate::processing::input_cache::InputStatus + Send + 'static, +) -> Result { + tokio::task::spawn_blocking(read).await.map_err(|error| { + error!("Playlist update status reader failed: {error}"); + axum::http::StatusCode::INTERNAL_SERVER_ERROR + }) +} + +async fn playlist_update_status( + axum::extract::State(app_state): axum::extract::State>, +) -> Result, axum::http::StatusCode> { + let storage_dir = app_state.app_config.config.load().storage_dir.clone(); + let inputs = app_state.app_config.sources.load().inputs.clone(); + let status_reads = futures::stream::iter(inputs.clone().into_iter().map(|input| { + let storage_dir = storage_dir.clone(); + let input_name = input.name.clone(); + async move { + let storage_path = + crate::processing::input_cache::resolve_input_storage_path(&storage_dir, &input_name).await; + read_playlist_update_input_status(move || crate::processing::input_cache::load_input_status(&storage_path)) + .await + } + })); + let status_reads = { + use futures::StreamExt as _; + status_reads.buffered(8) + }; + let input_statuses: Vec<_> = futures::StreamExt::collect(status_reads).await; + let input_statuses = input_statuses.into_iter().collect::, _>>()?; + let mut statuses = Vec::with_capacity(inputs.len()); + for (input, input_status) in inputs.into_iter().zip(input_statuses) { + let cluster_based = input.input_type.is_xtream() || input.input_type.is_stalker(); + let last_update = + input_status.clusters.values().map(|cluster| cluster.timestamp).filter(|timestamp| *timestamp > 0).max(); + let clusters = if cluster_based || input.input_type == InputType::M3u { + [XtreamCluster::Live, XtreamCluster::Video, XtreamCluster::Series] + .into_iter() + .filter_map(|cluster| { + input_status.clusters.get(cluster.as_ref()).map(|persisted| { + PersistedPlaylistUpdateClusterStatusDto { + cluster, + status: match &persisted.status { + crate::processing::input_cache::ClusterState::Ok => { + PersistedPlaylistUpdateClusterState::Ok + } + crate::processing::input_cache::ClusterState::Failed => { + PersistedPlaylistUpdateClusterState::Failed + } + }, + timestamp: persisted.timestamp, + last_update: persisted.last_update, + } + }) + }) + .collect() + } else { + Vec::new() + }; + let last_input_update = input_status.last_input_update.or_else(|| { + // Legacy cluster-free inputs already persisted a general default status. + // Do not infer an aggregate input/target or Quality result from clusters. + if cluster_based { + return None; + } + input_status.clusters.get("default").filter(|status| status.timestamp > 0).map(|status| { + PersistedPlaylistUpdateInputResult { + state: match status.status { + crate::processing::input_cache::ClusterState::Ok => PlaylistUpdateState::Success, + crate::processing::input_cache::ClusterState::Failed => PlaylistUpdateState::Failure, + }, + timestamp: status.timestamp, + } + }) + }); + statuses.push(InputPlaylistUpdateStatusDto { input_id: input.id, last_update, last_input_update, clusters }); + } + let active_updates = app_state + .event_manager + .playlist_update_snapshot() + .into_iter() + .filter(|event| event.input_id.is_some_and(|id| statuses.iter().any(|status| status.input_id == id))) + .collect(); + Ok(axum::Json(PlaylistUpdateStatusDto { inputs: statuses, active_updates })) +} + async fn playlist_content( accept: Option, app_state: &Arc, @@ -1120,6 +1320,7 @@ pub fn v1_api_playlist_register_protected(router: Router>) -> axum router .route("/playlist/resolve_url", axum::routing::post(playlist_resolve_url)) .route("/playlist/update", axum::routing::post(playlist_update)) + .route("/playlist/update/status", axum::routing::get(playlist_update_status)) .route("/playlist/epg", axum::routing::post(playlist_epg)) .route("/playlist/epg/stream", axum::routing::post(stream_epg_api)) .route("/playlist/live", axum::routing::post(playlist_content_live)) @@ -1145,6 +1346,7 @@ pub fn v1_api_playlist_register_with_permissions( app_state: &Arc, ) -> axum::Router> { let read_routes = Router::new() + .route("/update/status", axum::routing::get(playlist_update_status)) .route("/live", axum::routing::post(playlist_content_live)) .route("/vod", axum::routing::post(playlist_content_vod)) .route("/series", axum::routing::post(playlist_content_series)) @@ -1203,8 +1405,8 @@ mod tests { SharedStreamManager, }, model::{ - AppConfig, Config, ConfigInput, ConfigInputOptions, ConfigProvider, ConfigSource, ConfigTarget, - SourcesConfig, StreamHistoryConfig, VideoDownloadConfig, + AppConfig, Config, ConfigInput, ConfigInputOptions, ConfigInputUpdateQuality, ConfigProvider, ConfigSource, + ConfigTarget, SourcesConfig, StreamHistoryConfig, VideoDownloadConfig, }, processing::epg::{get_input_raw_epg_file_path, get_input_raw_xmltv_file_path}, repository::GeoIp, @@ -1223,8 +1425,12 @@ mod tests { foundation::Filter, model::{ provider_saturation::build_group_lookup, ConfigPaths, ConfigProviderDto, EpgChannel, EpgConfigDto, - EpgProgramme, EpgSourceDto, EpgSourceTypeDto, IcsDummyConfigDto, IcsEpgSourceConfigDto, InputType, - PlaylistRequest, ProcessingOrder, XtreamCluster, + EpgProgramme, EpgSourceDto, EpgSourceTypeDto, IcsDummyConfigDto, IcsEpgSourceConfigDto, + InputRefreshOverride, InputRefreshPolicy, InputType, OperationRunAccepted, + PersistedPlaylistUpdateClusterSnapshot, PersistedPlaylistUpdateClusterState, + PersistedPlaylistUpdateQualityDecision, PersistedPlaylistUpdateQualitySnapshot, + PersistedPlaylistUpdateTechnicalState, PlaylistRequest, PlaylistUpdateDataSource, PlaylistUpdateRequestDto, + PlaylistUpdateRequestPayload, PlaylistUpdateRunId, PlaylistUpdateStatusDto, ProcessingOrder, XtreamCluster, }, utils::Internable, }; @@ -1235,6 +1441,967 @@ mod tests { use tower::ServiceExt; use url::Url; + fn manual_update_request( + input_id: u16, + policy: InputRefreshPolicy, + ) -> crate::api::model::ManualPlaylistUpdateRequest { + crate::api::model::ManualPlaylistUpdateRequest { + run_id: PlaylistUpdateRunId::generate(), + targets: Arc::new(crate::model::ProcessTargets { + enabled: true, + inputs: vec![input_id], + targets: Vec::new(), + target_names: Vec::new(), + }), + input_action: Some(shared::model::InputUpdateRequest { + input_id, + action: shared::model::InputUpdateAction::Provider(policy), + }), + } + } + + fn playlist_update_target(id: u16, name: &str) -> Arc { + Arc::new(ConfigTarget { + id, + enabled: true, + name: name.to_string(), + options: None, + sort: None, + filter: Filter::default().into(), + output: vec![], + rename: None, + mapping_ids: None, + mapping: Arc::default(), + favourites: None, + processing_order: ProcessingOrder::default(), + execution_plan: tuliprox_core::model::TargetExecutionPlan::default(), + watch: None, + use_memory_cache: false, + }) + } + + #[test] + fn manual_update_capabilities_reject_unsupported_actions_and_keep_all_required_inputs() { + use shared::model::{InputUpdateAction, InputUpdateRequest}; + for (input_type, allowed) in [ + (InputType::Xtream, 3), + (InputType::Stalker, 3), + (InputType::M3u, 3), + (InputType::Plex, 2), + (InputType::Library, 1), + (InputType::Jellyfin, 0), + (InputType::Emby, 0), + (InputType::M3uBatch, 0), + (InputType::XtreamBatch, 0), + (InputType::StalkerBatch, 0), + (InputType::Staged, 0), + ] { + for enabled in [false, true] { + let input = Arc::new(ConfigInput { + id: 2, + name: "selected".intern(), + input_type, + enabled, + ..ConfigInput::default() + }); + let companion = Arc::new(ConfigInput { + id: 3, + name: "companion".intern(), + input_type: InputType::M3u, + enabled: true, + ..ConfigInput::default() + }); + let sources = SourcesConfig { + inputs: vec![input.clone(), companion.clone()], + sources: vec![ConfigSource { + inputs: vec![input.name.clone(), companion.name.clone()], + targets: vec![playlist_update_target(20, "target")], + }], + ..SourcesConfig::default() + }; + for (index, action) in [ + InputUpdateAction::Provider(InputRefreshPolicy::NORMAL), + InputUpdateAction::Provider(InputRefreshPolicy::REFRESH), + InputUpdateAction::Provider(InputRefreshPolicy::FORCE), + InputUpdateAction::Rescan, + ] + .into_iter() + .enumerate() + { + let request = PlaylistUpdateRequestDto { + target_ids: Some(vec![20]), + input_action: Some(InputUpdateRequest { input_id: 2, action }), + ..PlaylistUpdateRequestDto::default() + }; + let result = super::resolve_manual_playlist_update_targets(&sources, &request); + let expected = + enabled && if input_type == InputType::Library { index == 3 } else { index < allowed }; + assert_eq!(result.is_ok(), expected, "{input_type} {action:?}"); + if let Ok(targets) = result { + assert_eq!(targets.targets, vec![20]); + assert_eq!(targets.inputs, vec![2, 3]); + } + } + } + } + } + + #[tokio::test] + async fn manual_update_library_rescan_uses_same_queue_and_requires_library_permission() { + use shared::model::{ + InputUpdateAction, InputUpdateRequest, LibraryConfigDto, WebAuthConfigDto, WebUiConfigDto, + }; + let input = Arc::new(ConfigInput { + id: 2, + name: "library".intern(), + input_type: InputType::Library, + enabled: true, + ..ConfigInput::default() + }); + let app_config = Arc::new(test_app_config( + input.clone(), + ConfigSource { inputs: vec![input.name.clone()], targets: vec![playlist_update_target(20, "target")] }, + )); + let mut config = (**app_config.config.load()).clone(); + config.library = + Some(crate::model::LibraryConfig::from(&LibraryConfigDto { enabled: true, ..LibraryConfigDto::default() })); + config.web_ui = Some(crate::model::WebUiConfig::from(&WebUiConfigDto { + auth: Some(WebAuthConfigDto::default()), + ..WebUiConfigDto::default() + })); + app_config.config.store(Arc::new(config)); + let mut sources = (**app_config.sources.load()).clone(); + sources.inputs.push(Arc::new(ConfigInput { + id: 3, + name: "other-input".intern(), + input_type: InputType::M3u, + enabled: true, + ..ConfigInput::default() + })); + sources.sources.push(ConfigSource { + inputs: vec!["other-input".intern()], + targets: vec![playlist_update_target(21, "unrelated-target")], + }); + app_config.sources.store(Arc::new(sources)); + let (sender, mut receiver) = mpsc::channel(1); + let app_state = test_app_state_with_manual_update_sender(app_config, sender); + let action = InputUpdateRequest { input_id: 2, action: InputUpdateAction::Rescan }; + let request = PlaylistUpdateRequestDto { + target_ids: Some(vec![20]), + input_action: Some(action), + ..PlaylistUpdateRequestDto::default() + }; + let mut claims: shared::model::Claims = + serde_json::from_value(serde_json::json!({"username":"test", "iss":"test", "iat":0, "exp":0})).unwrap(); + let rejected = super::playlist_update( + State(app_state.clone()), + Some(axum::Extension(super::VerifiedClaims(claims.clone()))), + Json(PlaylistUpdateRequestPayload::Current(request.clone())), + ) + .await + .into_response(); + assert_eq!(rejected.status(), StatusCode::FORBIDDEN); + assert!(receiver.try_recv().is_err()); + let unauthenticated = super::playlist_update( + State(app_state.clone()), + None, + Json(PlaylistUpdateRequestPayload::Current(request.clone())), + ) + .await + .into_response(); + assert_eq!(unauthenticated.status(), StatusCode::UNAUTHORIZED); + assert!(receiver.try_recv().is_err()); + claims.permissions.set(shared::model::permission::Permission::LibraryWrite); + // The legacy-name compatibility must never relax the new Rescan wire form. + let invalid_requests = [ + PlaylistUpdateRequestDto { target_ids: None, ..request.clone() }, + PlaylistUpdateRequestDto { targets: vec!["target".to_string()], target_ids: None, ..request.clone() }, + PlaylistUpdateRequestDto { target_ids: Some(Vec::new()), ..request.clone() }, + PlaylistUpdateRequestDto { target_ids: Some(vec![99]), ..request.clone() }, + PlaylistUpdateRequestDto { target_ids: Some(vec![21]), ..request.clone() }, + PlaylistUpdateRequestDto { targets: vec!["target".to_string()], ..request.clone() }, + PlaylistUpdateRequestDto { + input_refresh: Some(InputRefreshOverride { input_id: 2, policy: InputRefreshPolicy::NORMAL }), + ..request.clone() + }, + ]; + for invalid in invalid_requests { + let rejected = super::playlist_update( + State(app_state.clone()), + Some(axum::Extension(super::VerifiedClaims(claims.clone()))), + Json(PlaylistUpdateRequestPayload::Current(invalid)), + ) + .await + .into_response(); + assert_eq!(rejected.status(), StatusCode::BAD_REQUEST); + assert!(receiver.try_recv().is_err()); + } + let accepted = super::playlist_update( + State(app_state), + Some(axum::Extension(super::VerifiedClaims(claims))), + Json(PlaylistUpdateRequestPayload::Current(request)), + ) + .await + .into_response(); + assert_eq!(accepted.status(), StatusCode::ACCEPTED); + let queued = receiver.recv().await.unwrap(); + assert_eq!(queued.input_action, Some(action)); + assert_eq!(queued.targets.targets, vec![20]); + } + + #[tokio::test] + async fn manual_update_target_ids_keep_conflict_instead_of_accepting_a_dropped_force_request() { + for pending_policy in [InputRefreshPolicy::NORMAL, InputRefreshPolicy::REFRESH] { + let input = Arc::new(ConfigInput { + id: 17, + name: "force-input".intern(), + input_type: InputType::Xtream, + enabled: true, + ..ConfigInput::default() + }); + let app_config = Arc::new(test_app_config( + Arc::clone(&input), + ConfigSource { + inputs: vec![Arc::clone(&input.name)], + targets: vec![playlist_update_target(1, "queued-target")], + }, + )); + let (sender, mut receiver) = mpsc::channel(1); + sender.send(manual_update_request(1, InputRefreshPolicy::NORMAL)).await.unwrap(); + let running = receiver.recv().await.expect("running request"); + assert_eq!( + running.input_action.map(|request| request.action), + Some(shared::model::InputUpdateAction::Provider(InputRefreshPolicy::NORMAL)) + ); + sender.send(manual_update_request(2, pending_policy)).await.unwrap(); + let app_state = test_app_state_with_manual_update_sender(app_config, sender); + + let response = super::playlist_update( + State(app_state), + None, + Json(PlaylistUpdateRequestPayload::Current(PlaylistUpdateRequestDto { + targets: Vec::new(), + target_ids: Some(vec![1]), + input_refresh: Some(InputRefreshOverride { input_id: input.id, policy: InputRefreshPolicy::FORCE }), + input_action: None, + })), + ) + .await + .into_response(); + + assert_eq!(response.status(), StatusCode::CONFLICT); + let pending = receiver.recv().await.expect("pending request remains queued"); + assert_eq!( + pending.input_action, + Some(shared::model::InputUpdateRequest { + input_id: 2, + action: shared::model::InputUpdateAction::Provider(pending_policy) + }) + ); + assert!(receiver.try_recv().is_err()); + } + } + + #[tokio::test] + async fn manual_update_run_target_ids_and_refresh_override_keep_the_accepted_queue_identity() { + let input = Arc::new(ConfigInput { + id: 17, + name: "stable-input".intern(), + input_type: InputType::Xtream, + enabled: true, + ..ConfigInput::default() + }); + let app_config = Arc::new(test_app_config( + Arc::clone(&input), + ConfigSource { + inputs: vec![Arc::clone(&input.name)], + targets: vec![playlist_update_target(1, "shared-name"), playlist_update_target(4, "shared-name")], + }, + )); + let (sender, mut receiver) = mpsc::channel(1); + let app_state = test_app_state_with_manual_update_sender(app_config, sender); + let input_refresh = InputRefreshOverride { input_id: input.id, policy: InputRefreshPolicy::FORCE }; + + let response = super::playlist_update( + State(app_state), + None, + Json(PlaylistUpdateRequestPayload::Current(PlaylistUpdateRequestDto { + targets: Vec::new(), + target_ids: Some(vec![4, 1]), + input_refresh: Some(input_refresh), + input_action: None, + })), + ) + .await + .into_response(); + + assert_eq!(response.status(), StatusCode::ACCEPTED); + let body = axum::body::to_bytes(response.into_body(), usize::MAX).await.expect("accepted response body"); + let accepted = serde_json::from_slice::(&body).expect("accepted response payload"); + let queued = receiver.recv().await.expect("accepted update request"); + assert_eq!(accepted.run_id.as_ref(), Some(&queued.run_id)); + assert!(queued.targets.enabled); + assert_eq!(queued.targets.inputs, vec![input.id]); + assert_eq!(queued.targets.targets, vec![4, 1]); + assert_eq!(queued.targets.target_names, vec!["shared-name", "shared-name"]); + assert_eq!( + queued.input_action, + Some(shared::model::InputUpdateRequest { + input_id: input_refresh.input_id, + action: shared::model::InputUpdateAction::Provider(input_refresh.policy) + }) + ); + } + + #[tokio::test] + async fn manual_update_target_ids_reject_unknown_empty_and_ambiguous_selections() { + let input = Arc::new(ConfigInput { + id: 17, + name: "stable-input".intern(), + input_type: InputType::Xtream, + enabled: true, + ..ConfigInput::default() + }); + let app_config = Arc::new(test_app_config( + Arc::clone(&input), + ConfigSource { + inputs: vec![Arc::clone(&input.name)], + targets: vec![playlist_update_target(1, "known-target")], + }, + )); + let (sender, mut receiver) = mpsc::channel(1); + let app_state = test_app_state_with_manual_update_sender(app_config, sender); + let input_refresh = Some(InputRefreshOverride { input_id: input.id, policy: InputRefreshPolicy::REFRESH }); + let invalid_requests = [ + PlaylistUpdateRequestDto { + targets: vec!["known-target".to_string()], + input_action: Some(shared::model::InputUpdateRequest { + input_id: input.id, + action: shared::model::InputUpdateAction::Provider(InputRefreshPolicy::REFRESH), + }), + ..PlaylistUpdateRequestDto::default() + }, + PlaylistUpdateRequestDto { + targets: Vec::new(), + target_ids: Some(vec![99]), + input_refresh, + input_action: None, + }, + PlaylistUpdateRequestDto { + targets: Vec::new(), + target_ids: Some(Vec::new()), + input_refresh, + input_action: None, + }, + PlaylistUpdateRequestDto { + targets: vec!["known-target".to_string()], + target_ids: Some(vec![1]), + input_refresh, + input_action: None, + }, + ]; + + for request in invalid_requests { + let response = super::playlist_update( + State(Arc::clone(&app_state)), + None, + Json(PlaylistUpdateRequestPayload::Current(request)), + ) + .await + .into_response(); + + assert_eq!(response.status(), StatusCode::BAD_REQUEST); + assert!(receiver.try_recv().is_err()); + } + } + + #[tokio::test] + async fn manual_update_target_ids_preserve_named_legacy_and_empty_bulk_requests() { + let input = Arc::new(ConfigInput { + id: 17, + name: "stable-input".intern(), + input_type: InputType::Xtream, + enabled: true, + ..ConfigInput::default() + }); + let app_config = Arc::new(test_app_config( + Arc::clone(&input), + ConfigSource { + inputs: vec![Arc::clone(&input.name)], + targets: vec![playlist_update_target(1, "shared-name"), playlist_update_target(4, "shared-name")], + }, + )); + let (sender, mut receiver) = mpsc::channel(1); + let app_state = test_app_state_with_manual_update_sender(app_config, sender); + let requests = [ + PlaylistUpdateRequestPayload::Current(PlaylistUpdateRequestDto { + targets: vec!["shared-name".to_string()], + target_ids: None, + input_refresh: None, + input_action: None, + }), + PlaylistUpdateRequestPayload::LegacyTargets(vec!["shared-name".to_string()]), + PlaylistUpdateRequestPayload::Current(PlaylistUpdateRequestDto { + targets: Vec::new(), + target_ids: None, + input_refresh: None, + input_action: None, + }), + ]; + + for (index, request) in requests.into_iter().enumerate() { + let response = + super::playlist_update(State(Arc::clone(&app_state)), None, Json(request)).await.into_response(); + + assert_eq!(response.status(), StatusCode::ACCEPTED); + let queued = receiver.recv().await.expect("accepted legacy request"); + assert_eq!(queued.input_action, None); + if index < 2 { + assert!(queued.targets.enabled); + assert_eq!(queued.targets.targets, vec![1, 4]); + assert_eq!(queued.targets.target_names, vec!["shared-name", "shared-name"]); + } else { + assert!(!queued.targets.enabled); + assert!(queued.targets.targets.is_empty()); + assert!(queued.targets.target_names.is_empty()); + } + } + } + + #[tokio::test] + async fn manual_update_m3u_force_is_queued_by_stable_ids_and_preserves_conflict() { + use shared::model::{InputUpdateAction, InputUpdateRequest}; + let input = Arc::new(ConfigInput { + id: 17, + name: "m3u-force".intern(), + input_type: InputType::M3u, + enabled: true, + ..ConfigInput::default() + }); + let config = Arc::new(test_app_config( + input.clone(), + ConfigSource { + inputs: vec![input.name.clone()], + targets: vec![playlist_update_target(1, "selected"), playlist_update_target(2, "other")], + }, + )); + let (sender, mut receiver) = mpsc::channel(1); + let app_state = test_app_state_with_manual_update_sender(config, sender); + let action = + InputUpdateRequest { input_id: input.id, action: InputUpdateAction::Provider(InputRefreshPolicy::FORCE) }; + let request = PlaylistUpdateRequestPayload::Current(PlaylistUpdateRequestDto { + target_ids: Some(vec![1]), + input_action: Some(action), + ..PlaylistUpdateRequestDto::default() + }); + let accepted = + super::playlist_update(State(app_state.clone()), None, Json(request.clone())).await.into_response(); + assert_eq!(accepted.status(), StatusCode::ACCEPTED); + let conflict = super::playlist_update(State(app_state), None, Json(request)).await.into_response(); + assert_eq!(conflict.status(), StatusCode::CONFLICT); + let queued = receiver.try_recv().unwrap(); + assert_eq!(queued.input_action, Some(action)); + assert_eq!(queued.targets.targets, [1]); + assert_eq!(queued.targets.inputs, [17]); + assert!(receiver.try_recv().is_err()); + } + + #[tokio::test] + async fn manual_update_legacy_input_refresh_preserves_named_and_all_scope_policy_and_queue_conflicts() { + let input = Arc::new(ConfigInput { + id: 17, + name: "provider".intern(), + input_type: InputType::Xtream, + enabled: true, + ..ConfigInput::default() + }); + let app_config = Arc::new(test_app_config( + input.clone(), + ConfigSource { + inputs: vec![input.name.clone()], + targets: vec![playlist_update_target(1, "selected"), playlist_update_target(4, "other")], + }, + )); + let (sender, mut receiver) = mpsc::channel(1); + let app_state = test_app_state_with_manual_update_sender(app_config, sender); + for policy in [InputRefreshPolicy::NORMAL, InputRefreshPolicy::REFRESH, InputRefreshPolicy::FORCE] { + for names in [vec!["selected".to_string()], Vec::new()] { + let expected = app_state + .app_config + .sources + .load() + .validate_targets((!names.is_empty()).then_some(&names)) + .unwrap(); + let request: PlaylistUpdateRequestPayload = serde_json::from_value(json!({ + "targets": names, + "input_refresh": {"input_id": input.id, "policy": policy} + })) + .unwrap(); + let response = + super::playlist_update(State(app_state.clone()), None, Json(request.clone())).await.into_response(); + assert_eq!(response.status(), StatusCode::ACCEPTED); + let body = axum::body::to_bytes(response.into_body(), usize::MAX).await.unwrap(); + let accepted: OperationRunAccepted = serde_json::from_slice(&body).unwrap(); + let conflict = + super::playlist_update(State(app_state.clone()), None, Json(request)).await.into_response(); + assert_eq!(conflict.status(), StatusCode::CONFLICT); + let queued = receiver.try_recv().unwrap(); + assert_eq!(accepted.run_id.as_ref(), Some(&queued.run_id)); + assert_eq!(queued.targets.enabled, expected.enabled); + assert_eq!(queued.targets.inputs, expected.inputs); + assert_eq!(queued.targets.targets, expected.targets); + assert_eq!(queued.targets.target_names, expected.target_names); + assert_eq!( + queued.input_action, + Some(shared::model::InputUpdateRequest { + input_id: input.id, + action: shared::model::InputUpdateAction::Provider(policy), + }) + ); + assert!(receiver.try_recv().is_err()); + } + } + } + + #[tokio::test] + async fn m3u_update_quality_playlist_update_status_reads_newest_timestamp_from_canonically_resolved_input_path() { + let temp_dir = tempdir().expect("temp dir"); + let input = Arc::new(ConfigInput { + id: 7, + name: "timestamped/provider legacy".intern(), + input_type: InputType::Xtream, + enabled: true, + ..ConfigInput::default() + }); + let source = ConfigSource { inputs: vec![Arc::clone(&input.name)], targets: Vec::new() }; + let app_config = test_app_config(Arc::clone(&input), source); + app_config.config.store(Arc::new(Config { + storage_dir: temp_dir.path().to_string_lossy().to_string(), + ..Config::default() + })); + let mut sources = app_config.sources.load().as_ref().clone(); + sources.inputs.push(Arc::new(ConfigInput { + id: 8, + name: "never-updated".intern(), + input_type: InputType::Stalker, + enabled: true, + ..ConfigInput::default() + })); + sources.inputs.push(Arc::new(ConfigInput { + id: 9, + name: "m3u-with-legacy-cluster".intern(), + input_type: InputType::M3u, + enabled: true, + ..ConfigInput::default() + })); + app_config.sources.store(Arc::new(sources)); + + let never_updated_path = temp_dir.path().join("input_never_updated"); + assert!(!never_updated_path.exists()); + let storage_path = crate::processing::input_cache::resolve_input_storage_path( + temp_dir.path().to_string_lossy().as_ref(), + &input.name, + ) + .await; + assert_eq!(storage_path.file_name().and_then(|name| name.to_str()), Some("input_timestamped_provider_legacy")); + let mut persisted = crate::processing::input_cache::InputStatus::default(); + persisted.clusters.insert( + XtreamCluster::Live.as_ref().to_string(), + crate::processing::input_cache::ClusterStatus { + status: crate::processing::input_cache::ClusterState::Ok, + timestamp: 101, + last_update: None, + }, + ); + persisted.clusters.insert( + XtreamCluster::Video.as_ref().to_string(), + crate::processing::input_cache::ClusterStatus { + status: crate::processing::input_cache::ClusterState::Failed, + timestamp: 303, + last_update: Some(PersistedPlaylistUpdateClusterSnapshot { + policy: Some(InputRefreshPolicy::REFRESH), + source: Some(PlaylistUpdateDataSource::Provider), + quality_guard_threshold: None, + quality: Some(PersistedPlaylistUpdateQualitySnapshot { + threshold: 95, + baseline_count: Some(4_812), + candidate_count: Some(3_104), + achieved_quality: Some(64), + decision: PersistedPlaylistUpdateQualityDecision::Rejected, + }), + active_count: Some(4_812), + technical_state: Some(PersistedPlaylistUpdateTechnicalState::Succeeded), + }), + }, + ); + persisted.clusters.insert( + XtreamCluster::Series.as_ref().to_string(), + crate::processing::input_cache::ClusterStatus { + status: crate::processing::input_cache::ClusterState::Ok, + timestamp: 202, + last_update: None, + }, + ); + crate::processing::input_cache::save_input_status(&storage_path, &persisted); + + let m3u_storage_path = crate::processing::input_cache::resolve_input_storage_path( + temp_dir.path().to_string_lossy().as_ref(), + "m3u-with-legacy-cluster", + ) + .await; + let mut m3u_persisted = crate::processing::input_cache::InputStatus::default(); + m3u_persisted.clusters.insert( + XtreamCluster::Live.as_ref().to_string(), + crate::processing::input_cache::ClusterStatus { + status: crate::processing::input_cache::ClusterState::Failed, + timestamp: 404, + last_update: None, + }, + ); + crate::processing::input_cache::save_input_status(&m3u_storage_path, &m3u_persisted); + + let app_state = test_app_state(Arc::new(app_config)); + let router = super::v1_api_playlist_register_protected(Router::new()).with_state(app_state); + let response = router + .into_service::() + .oneshot(Request::builder().uri("/playlist/update/status").body(Body::empty()).expect("request")) + .await + .expect("response"); + + assert_eq!(response.status(), StatusCode::OK); + let body = axum::body::to_bytes(response.into_body(), usize::MAX).await.expect("body"); + let status = serde_json::from_slice::(&body).expect("status response"); + assert_eq!(status.inputs.len(), 3); + assert_eq!(status.inputs[0].input_id, 7); + assert_eq!(status.inputs[0].last_update, Some(303)); + assert_eq!(status.inputs[0].clusters.len(), 3); + assert_eq!(status.inputs[0].clusters[0].cluster, XtreamCluster::Live); + assert_eq!(status.inputs[0].clusters[0].status, PersistedPlaylistUpdateClusterState::Ok); + assert_eq!(status.inputs[0].clusters[0].timestamp, 101); + assert_eq!(status.inputs[0].clusters[1].cluster, XtreamCluster::Video); + assert_eq!(status.inputs[0].clusters[1].status, PersistedPlaylistUpdateClusterState::Failed); + assert_eq!(status.inputs[0].clusters[1].timestamp, 303); + assert_eq!( + status.inputs[0].clusters[1].last_update, + Some(PersistedPlaylistUpdateClusterSnapshot { + policy: Some(InputRefreshPolicy::REFRESH), + source: Some(PlaylistUpdateDataSource::Provider), + quality_guard_threshold: None, + quality: Some(PersistedPlaylistUpdateQualitySnapshot { + threshold: 95, + baseline_count: Some(4_812), + candidate_count: Some(3_104), + achieved_quality: Some(64), + decision: PersistedPlaylistUpdateQualityDecision::Rejected, + }), + active_count: Some(4_812), + technical_state: Some(PersistedPlaylistUpdateTechnicalState::Succeeded), + }) + ); + assert_eq!(status.inputs[0].clusters[2].cluster, XtreamCluster::Series); + assert_eq!(status.inputs[0].clusters[2].status, PersistedPlaylistUpdateClusterState::Ok); + assert_eq!(status.inputs[0].clusters[2].timestamp, 202); + assert_eq!(status.inputs[1].input_id, 8); + assert_eq!(status.inputs[1].last_update, None); + assert!(status.inputs[1].clusters.is_empty()); + assert_eq!(status.inputs[2].input_id, 9); + assert_eq!(status.inputs[2].last_update, Some(404)); + assert_eq!(status.inputs[2].clusters.len(), 1, "Only the actually persisted M3U cluster is exposed"); + assert_eq!(status.inputs[2].clusters[0].cluster, XtreamCluster::Live); + assert_eq!(status.inputs[2].clusters[0].status, PersistedPlaylistUpdateClusterState::Failed); + assert!(never_updated_path.is_dir()); + } + + #[tokio::test] + async fn playlist_update_status_reader_runs_off_the_async_worker() { + use crate::processing::input_cache::{ + load_input_status, save_input_status, ClusterState, ClusterStatus, InputStatus, + }; + + let temp = tempdir().unwrap(); + let expected = InputStatus { + clusters: HashMap::from([( + "live".to_owned(), + ClusterStatus { status: ClusterState::Ok, timestamp: 17, last_update: None }, + )]), + ..InputStatus::default() + }; + save_input_status(temp.path(), &expected); + let before = std::fs::read(temp.path().join("status.json")).unwrap(); + let worker = std::thread::current().id(); + let path = temp.path().to_path_buf(); + let actual = super::read_playlist_update_input_status(move || { + assert_ne!(std::thread::current().id(), worker); + load_input_status(&path) + }) + .await + .unwrap(); + + assert_eq!(actual.clusters, expected.clusters); + assert_eq!(std::fs::read(temp.path().join("status.json")).unwrap(), before); + } + + #[tokio::test] + async fn playlist_update_status_reader_join_failure_returns_server_error_not_empty_status() { + let result = super::read_playlist_update_input_status(|| panic!("simulated status reader failure")).await; + assert_eq!(result.unwrap_err(), StatusCode::INTERNAL_SERVER_ERROR); + } + + #[tokio::test] + async fn playlist_update_status_reload_returns_active_bus_facts_without_persisting_updating() { + use shared::model::{EventMessage, PlaylistUpdateProgressEvent, PlaylistUpdateState, PlaylistUpdateSummary}; + let temp = tempdir().unwrap(); + let input = Arc::new(ConfigInput { + id: 7, + name: "reload-input".intern(), + input_type: InputType::M3u, + enabled: true, + ..ConfigInput::default() + }); + let config = + test_app_config(input.clone(), ConfigSource { inputs: vec![input.name.clone()], targets: Vec::new() }); + config + .config + .store(Arc::new(Config { storage_dir: temp.path().to_string_lossy().into_owned(), ..Config::default() })); + let app = test_app_state(Arc::new(config)); + let path = + crate::processing::input_cache::resolve_input_storage_path(&temp.path().to_string_lossy(), &input.name) + .await; + let persisted = crate::processing::input_cache::InputStatus { + last_input_update: Some(shared::model::PersistedPlaylistUpdateInputResult { + state: PlaylistUpdateState::Success, + timestamp: 1, + }), + ..crate::processing::input_cache::InputStatus::default() + }; + crate::processing::input_cache::save_input_status(&path, &persisted); + let before = std::fs::read(path.join("status.json")).unwrap(); + let mut progress = + PlaylistUpdateProgressEvent::for_run_input("running".into(), 17.into(), 7, "reload-input", "updating"); + app.event_manager.send_event(EventMessage::PlaylistUpdateProgress(progress.clone())); + let unknown = PlaylistUpdateProgressEvent::for_run_input("other".into(), 18.into(), 99, "removed", "updating"); + app.event_manager.send_event(EventMessage::PlaylistUpdateProgress(unknown)); + let Json(status) = super::playlist_update_status(State(app.clone())).await.unwrap(); + assert_eq!(status.active_updates, vec![progress.clone()]); + assert_eq!(status.inputs[0].last_input_update, persisted.last_input_update); + progress.state = Some(PlaylistUpdateState::Partial); + app.event_manager.send_event(EventMessage::PlaylistUpdateProgress(progress.clone())); + let Json(status) = super::playlist_update_status(State(app.clone())).await.unwrap(); + assert_eq!(status.active_updates, vec![progress], "completed input still belongs to a running target rebuild"); + app.event_manager.send_event(EventMessage::PlaylistUpdate(PlaylistUpdateSummary::for_run( + "running".into(), + 17.into(), + PlaylistUpdateState::Failure, + ))); + let Json(status) = super::playlist_update_status(State(app)).await.unwrap(); + assert!(status.active_updates.is_empty()); + assert_eq!( + std::fs::read(path.join("status.json")).unwrap(), + before, + "read-only runtime projection never writes transient status" + ); + } + + #[tokio::test] + async fn playlist_update_status_reloads_staged_own_completion_without_default_or_parent_inference() { + use crate::processing::input_cache::{ + load_input_status, resolve_input_storage_path, save_input_status, ClusterState, ClusterStatus, InputStatus, + }; + use shared::model::{ + PersistedPlaylistUpdateClusterStatusDto, PersistedPlaylistUpdateInputResult, PlaylistUpdateState, + }; + for own_state in [Some(PlaylistUpdateState::Success), Some(PlaylistUpdateState::Failure), None] { + let temp = tempdir().unwrap(); + let parent = Arc::new(ConfigInput { + id: 7, + name: "parent".intern(), + input_type: InputType::Xtream, + ..ConfigInput::default() + }); + let app_config = test_app_config( + parent.clone(), + ConfigSource { inputs: vec![parent.name.clone()], targets: Vec::new() }, + ); + app_config.config.store(Arc::new(Config { + storage_dir: temp.path().to_string_lossy().into_owned(), + ..Config::default() + })); + let mut staged = ConfigInput { + id: 8, + name: "staged/provider own".intern(), + input_type: InputType::Staged, + staged_type: shared::model::StagedInputType::Xtream, + ..ConfigInput::default() + }; + staged.staged = Some(tuliprox_core::model::ConfigInputStaged { + for_input: Some(parent.name.clone()), + clusters: shared::model::ClusterFlags::Live, + }); + staged.resolve_staged_download_type(); + let mut sources = app_config.sources.load().as_ref().clone(); + sources.inputs.insert(0, Arc::new(staged.clone())); // Deliberately not ID or source order. + app_config.sources.store(Arc::new(sources)); + let path = resolve_input_storage_path(&temp.path().to_string_lossy(), &staged.name).await; + let snapshot = PersistedPlaylistUpdateClusterSnapshot { + policy: Some(InputRefreshPolicy::REFRESH), + source: Some(PlaylistUpdateDataSource::Cache), + quality_guard_threshold: Some(95), + ..PersistedPlaylistUpdateClusterSnapshot::default() + }; + let mut persisted = InputStatus::default(); + persisted.clusters.insert( + "live".into(), + ClusterStatus { status: ClusterState::Ok, timestamp: 123, last_update: Some(snapshot) }, + ); + persisted.last_input_update = + own_state.map(|state| PersistedPlaylistUpdateInputResult { state, timestamp: 456 }); + save_input_status(&path, &persisted); + let before = std::fs::read(path.join("status.json")).unwrap(); + let parent_path = resolve_input_storage_path(&temp.path().to_string_lossy(), &parent.name).await; + save_input_status( + &parent_path, + &InputStatus { + last_input_update: Some(PersistedPlaylistUpdateInputResult { + state: PlaylistUpdateState::Partial, + timestamp: 789, + }), + ..InputStatus::default() + }, + ); + let router = super::v1_api_playlist_register_protected(Router::new()) + .with_state(test_app_state(Arc::new(app_config))); + let response = router + .into_service::() + .oneshot(Request::builder().uri("/playlist/update/status").body(Body::empty()).unwrap()) + .await + .unwrap(); + assert_eq!(response.status(), StatusCode::OK); + let body = axum::body::to_bytes(response.into_body(), usize::MAX).await.unwrap(); + let dto: PlaylistUpdateStatusDto = serde_json::from_slice(&body).unwrap(); + let own = dto.inputs.iter().find(|input| input.input_id == 8).unwrap(); + assert_eq!(own.last_input_update, persisted.last_input_update); + assert_eq!(own.last_update, Some(123)); + assert_eq!( + own.clusters, + vec![PersistedPlaylistUpdateClusterStatusDto { + cluster: XtreamCluster::Live, + status: PersistedPlaylistUpdateClusterState::Ok, + timestamp: 123, + last_update: Some(snapshot) + }] + ); + assert_eq!( + dto.inputs.iter().find(|input| input.input_id == 7).unwrap().last_input_update.unwrap().state, + PlaylistUpdateState::Partial + ); + assert!(!load_input_status(&path).clusters.contains_key("default")); + assert_eq!( + std::fs::read(path.join("status.json")).unwrap(), + before, + "read endpoint must not rewrite canonical status" + ); + } + } + + #[tokio::test] + async fn playlist_update_status_exposes_last_input_results_and_legacy_default_without_inventing_clusters() { + use crate::processing::input_cache::{ + load_input_status, resolve_input_storage_path, save_input_status, ClusterState, ClusterStatus, InputStatus, + }; + use shared::model::{PersistedPlaylistUpdateInputResult, PlaylistUpdateState}; + for last_state in [ + None, + Some(PlaylistUpdateState::Success), + Some(PlaylistUpdateState::Partial), + Some(PlaylistUpdateState::Failure), + ] { + let temp = tempdir().unwrap(); + let app_config = test_app_config( + Arc::new(ConfigInput::default()), + ConfigSource { inputs: Vec::new(), targets: Vec::new() }, + ); + app_config.config.store(Arc::new(Config { + storage_dir: temp.path().to_string_lossy().into_owned(), + ..Config::default() + })); + let mut sources = (**app_config.sources.load()).clone(); + sources.inputs.clear(); + for (index, input_type) in [ + InputType::Xtream, + InputType::XtreamBatch, + InputType::Stalker, + InputType::StalkerBatch, + InputType::M3u, + InputType::M3uBatch, + InputType::Library, + InputType::Plex, + InputType::Emby, + InputType::Jellyfin, + InputType::Staged, + ] + .into_iter() + .enumerate() + { + let input = Arc::new(ConfigInput { + id: u16::try_from(index + 1).unwrap(), + name: format!("input-{index}").intern(), + input_type, + enabled: index % 2 == 0, + ..ConfigInput::default() + }); + let path = resolve_input_storage_path(&temp.path().to_string_lossy(), &input.name).await; + let mut persisted = InputStatus::default(); + persisted.clusters.insert( + "default".to_string(), + ClusterStatus { + status: if input.enabled { ClusterState::Ok } else { ClusterState::Failed }, + timestamp: 123, + last_update: None, + }, + ); + persisted.last_input_update = + last_state.map(|state| PersistedPlaylistUpdateInputResult { state, timestamp: 456 }); + save_input_status(&path, &persisted); + sources.inputs.push(input); + } + let inputs = sources.inputs.clone(); + app_config.sources.store(Arc::new(sources)); + let router = super::v1_api_playlist_register_protected(Router::new()) + .with_state(test_app_state(Arc::new(app_config))); + let response = router + .into_service::() + .oneshot(Request::builder().uri("/playlist/update/status").body(Body::empty()).unwrap()) + .await + .unwrap(); + assert_eq!(response.status(), StatusCode::OK); + let body = axum::body::to_bytes(response.into_body(), usize::MAX).await.unwrap(); + let dto: PlaylistUpdateStatusDto = serde_json::from_slice(&body).unwrap(); + assert_eq!(dto.inputs.len(), inputs.len()); + for (input, status) in inputs.iter().zip(dto.inputs) { + assert_eq!(status.input_id, input.id); + assert_eq!(status.last_update, Some(123), "Last update retains the cache timestamp contract"); + let expected = if let Some(state) = last_state { + Some(PersistedPlaylistUpdateInputResult { state, timestamp: 456 }) + } else if input.input_type.is_xtream() || input.input_type.is_stalker() { + None // Do not reconstruct an aggregate input result from incomplete legacy clusters. + } else { + Some(PersistedPlaylistUpdateInputResult { + state: if input.enabled { PlaylistUpdateState::Success } else { PlaylistUpdateState::Failure }, + timestamp: 123, + }) + }; + assert_eq!(status.last_input_update, expected); + assert!(status.clusters.is_empty(), "no synthetic Live/VOD/Series from default"); + let path = resolve_input_storage_path(&temp.path().to_string_lossy(), &input.name).await; + assert_eq!( + load_input_status(&path).last_input_update.map(|result| result.state), + last_state, + "read endpoint must not persist a legacy projection" + ); + } + } + } + #[tokio::test] async fn stable_recording_route_rejects_target_and_input_from_different_sources() { let app_config = Arc::new(test_app_config( @@ -1407,6 +2574,14 @@ mod tests { } fn test_app_state(app_cfg: Arc) -> Arc { + let (manual_update_sender, _) = mpsc::channel::(1); + test_app_state_with_manual_update_sender(app_cfg, manual_update_sender) + } + + fn test_app_state_with_manual_update_sender( + app_cfg: Arc, + manual_update_sender: mpsc::Sender, + ) -> Arc { let event_manager = Arc::new(EventManager::new()); let active_provider = Arc::new(ActiveProviderManager::new(&app_cfg, &event_manager)); let shared_stream_manager = Arc::new(SharedStreamManager::new(Arc::clone(&active_provider))); @@ -1435,7 +2610,6 @@ mod tests { hls_cache: CancellationToken::new(), }; let metadata_manager = Arc::new(MetadataUpdateManager::new(tokens.metadata.clone())); - let (manual_update_sender, _) = mpsc::channel::(1); Arc::new(AppState { forced_targets: Arc::new(ArcSwap::from_pointee(crate::model::ProcessTargets { @@ -2221,6 +3395,7 @@ mod tests { enabled: true, options: Some(ConfigInputOptions { flags: crate::model::ConfigInputFlagsSet::new(), + update_quality: ConfigInputUpdateQuality::default(), resolve_delay: shared::defaults::default_resolve_delay_secs(), probe_delay: shared::defaults::default_probe_delay_secs(), probe_live_interval_hours: 120, diff --git a/backend/app/src/api/endpoints/websocket_api.rs b/backend/app/src/api/endpoints/websocket_api.rs index fb2fe9a76..fd9873cfd 100644 --- a/backend/app/src/api/endpoints/websocket_api.rs +++ b/backend/app/src/api/endpoints/websocket_api.rs @@ -13,8 +13,9 @@ use log::{error, trace}; use shared::{ defaults::default_kick_secs, model::{ - Claims, Permission, ProtocolHandler, ProtocolHandlerMemory, ProtocolMessage, RoleSet, UserCommand, UserId, - UserRole, WsCloseCode, CURRENT_PERMISSION_SCHEMA_VERSION, PERM_ALL, PROTOCOL_VERSION, TOKEN_NO_AUTH, + Claims, Permission, PlaylistUpdateRunStateEvent, ProtocolHandler, ProtocolHandlerMemory, ProtocolMessage, + RoleSet, UserCommand, UserId, UserRole, WsCloseCode, CURRENT_PERMISSION_SCHEMA_VERSION, PERM_ALL, + PROTOCOL_VERSION, TOKEN_NO_AUTH, }, utils::concat_path_leading_slash, }; @@ -444,11 +445,18 @@ fn to_protocol_message(event: EventMessage) -> Option<(ProtocolMessage, &'static EventMessage::ConfigChange(config) => { (ProtocolMessage::ConfigChangeResponse(config), "Configuration files change event") } - // The wire carries the outcome only. The run summary rides along on + // The wire carries the correlated outcome only. Statistics stay on // the bus for notifications and plugins; the Web UI re-fetches its // own view rather than reading statistics off this frame. EventMessage::PlaylistUpdate(summary) => { - (ProtocolMessage::PlaylistUpdateResponse(summary.state), "Playlist update event") + ( + ProtocolMessage::PlaylistUpdateResponse(PlaylistUpdateRunStateEvent { + run_id: summary.run_id, + execution_order: summary.execution_order, + state: summary.state, + }), + "Playlist update event", + ) } EventMessage::PlaylistUpdateProgress(progress) => { (ProtocolMessage::PlaylistUpdateProgressResponse(progress), "Playlist update progress event") @@ -661,7 +669,8 @@ mod tests { AuthAuditEvent, AuthAuditOutcome, Claims, ConfigReloadFailure, DiskAlert, DiskAlertLevel, DownloadsDelta, DownloadsResponse, FileDownloadDto, LibraryScanProgressEvent, LibraryScanSummary, LibraryScanSummaryStatus, MetadataUpdateFailure, MsgKind, Permission, PlaylistGroupsChanged, PlaylistUpdateProgressEvent, - ProtocolHandler, ProtocolHandlerMemory, ProviderAccountEvent, ProviderAccountState, ProviderFailureKind, + PlaylistUpdateRunId, PlaylistUpdateRunOrder, PlaylistUpdateState, PlaylistUpdateSummary, ProtocolHandler, + ProtocolHandlerMemory, ProtocolMessage, ProviderAccountEvent, ProviderAccountState, ProviderFailureKind, ProviderFetchFailure, ProviderPoolExhausted, ProviderPriorityFallback, RecordingLifecycleMessage, RoleSet, TaskKindDto, TaskPriorityDto, TransferStatusDto, UserId, UserRole, WatchChanges, WatchDisabled, WatchDisabledReason, WatchUnmatched, CURRENT_PERMISSION_SCHEMA_VERSION, PERM_ALL, PROTOCOL_VERSION, @@ -739,6 +748,25 @@ mod tests { } } + #[test] + fn playlist_update_run_websocket_terminal_keeps_the_processing_identity() { + let run_id = PlaylistUpdateRunId::from("websocket-run"); + let execution_order = PlaylistUpdateRunOrder::from(29); + let mapped = to_protocol_message(EventMessage::PlaylistUpdate(PlaylistUpdateSummary::for_run( + run_id.clone(), + execution_order, + PlaylistUpdateState::Partial, + ))) + .expect("playlist update event is wire mapped"); + + let ProtocolMessage::PlaylistUpdateResponse(terminal) = mapped.0 else { + panic!("expected playlist update terminal response"); + }; + assert_eq!(terminal.run_id, Some(run_id)); + assert_eq!(terminal.execution_order, Some(execution_order)); + assert_eq!(terminal.state, PlaylistUpdateState::Partial); + } + fn provider_fetch_failure() -> ProviderFetchFailure { ProviderFetchFailure { input: "i".into(), @@ -787,10 +815,7 @@ mod tests { EventMessage::ActiveProvider("p".into(), 1), EventMessage::ConfigChange(ConfigType::Config), EventMessage::PlaylistUpdate(PlaylistUpdateSummary::state_only(PlaylistUpdateState::Success)), - EventMessage::PlaylistUpdateProgress(PlaylistUpdateProgressEvent { - target: String::new(), - message: String::new(), - }), + EventMessage::PlaylistUpdateProgress(PlaylistUpdateProgressEvent::global("", "")), EventMessage::SystemInfoUpdate(Arc::new(SystemInfo { cpu_usage: 0.0, memory_usage: 0, @@ -1047,10 +1072,7 @@ mod tests { assert!(websocket_can_receive_runtime_events( &mem, - &EventMessage::PlaylistUpdateProgress(PlaylistUpdateProgressEvent { - target: "target".to_string(), - message: "step".to_string(), - }) + &EventMessage::PlaylistUpdateProgress(PlaylistUpdateProgressEvent::global("target", "step")) )); } diff --git a/backend/app/src/api/main_api.rs b/backend/app/src/api/main_api.rs index b74d61cba..74004b209 100644 --- a/backend/app/src/api/main_api.rs +++ b/backend/app/src/api/main_api.rs @@ -34,7 +34,7 @@ use crate::{ }, }, model::{AppConfig, Config, HdHomeRunFlags, Healthcheck, ProcessTargets, RateLimitConfig}, - processing::processor::{exec_processing, ProcessingRun}, + processing::processor::{exec_processing, next_playlist_update_run_order, ProcessingRun}, repository::{get_geoip_path, GeoIp}, utils::{exec_file_lock_prune, get_default_web_root_path}, VERSION, @@ -49,7 +49,7 @@ use dashmap::DashSet; use log::{debug, error, info, warn}; use shared::{ error::TuliproxError, - model::ServerLifecycleEvent, + model::{PlaylistUpdateState, PlaylistUpdateSummary, ServerLifecycleEvent}, utils::{concat_path_leading_slash, sanitize_sensitive_info}, }; use std::{ @@ -471,15 +471,22 @@ async fn run_manual_update_worker( ) { while let Some(request) = rx.recv().await { let Some(permit) = app_state.update_guard.acquire_playlist_lock().await else { + app_state.event_manager.send_event(EventMessage::PlaylistUpdate(PlaylistUpdateSummary::for_run( + request.run_id, + next_playlist_update_run_order(), + PlaylistUpdateState::Failure, + ))); break; }; exec_processing( - ProcessingRun::new( + ProcessingRun::for_run( + request.run_id, client.clone(), Arc::clone(&app_state.app_config), request.targets, Arc::clone(&app_state.event_manager), ) + .with_manual_input_update(request.input_action) .with_bootstrap({ let state = Arc::clone(&app_state); move || { diff --git a/backend/app/src/api/model/app_state.rs b/backend/app/src/api/model/app_state.rs index e49325b4d..b8f653598 100644 --- a/backend/app/src/api/model/app_state.rs +++ b/backend/app/src/api/model/app_state.rs @@ -26,7 +26,7 @@ use reqwest::Client; use shared::{ create_bitset, error::TuliproxError, - model::{UserConnectionPermission, VideoDownloadConfigDto, WebAuthConfigDto}, + model::{PlaylistUpdateRunId, UserConnectionPermission, VideoDownloadConfigDto, WebAuthConfigDto}, utils::small_vecs_equal_unordered, }; use std::{ @@ -418,7 +418,9 @@ fn video_download_changed(a: &crate::model::VideoDownloadConfig, b: &crate::mode #[derive(Clone)] pub struct ManualPlaylistUpdateRequest { + pub run_id: PlaylistUpdateRunId, pub targets: Arc, + pub input_action: Option, } #[derive(Clone)] diff --git a/backend/app/src/api/source_yml_patch.rs b/backend/app/src/api/source_yml_patch.rs index 56744d82d..5bce723b6 100644 --- a/backend/app/src/api/source_yml_patch.rs +++ b/backend/app/src/api/source_yml_patch.rs @@ -456,6 +456,25 @@ pub(crate) fn apply_sources_yml_patches( let idx = *inputs_by_name.get(input_name).ok_or_else(|| { TuliproxError::ConfigPanelApi(format!("source.yml patch target input '{input_name}' was not found")) })?; + let mut existing = + doc.inputs[idx].aliases.iter().flatten().filter(|alias| alias.name.trim() == alias_name.trim()); + if let Some(alias) = existing.next() { + if existing.next().is_some() { + return Err(TuliproxError::ConfigPanelApi(format!( + "source.yml patch target alias '{alias_name}' under input '{input_name}' is ambiguous" + ))); + } + if alias.url == *base_url + && alias.username.as_deref() == Some(username.as_str()) + && alias.password.as_deref() == Some(password.as_str()) + && alias.exp_date == *exp_date + { + continue; + } + return Err(TuliproxError::ConfigPanelApi(format!( + "source.yml patch: alias '{alias_name}' already exists under input '{input_name}' with different data" + ))); + } let alias_idx = append_sources_yml_alias( input_name, &mut doc.inputs[idx], @@ -926,7 +945,7 @@ pub(crate) async fn execute_source_yml_patches( #[cfg(test)] mod tests { use super::*; - use crate::config_loader::source_patch::{ensure_block_style_for_insertion, TextEdit}; + use crate::config_loader::source_patch::TextEdit; use shared::model::PanelApiConfigDto; fn edits_to_text(text: &str, edits: Vec) -> String { @@ -1046,21 +1065,20 @@ mod tests { } #[test] - fn flow_style_insertion_is_rejected_with_actionable_error() { - // `find_input` is the entry point; it does not reject flow-style itself. The rejector is - // `ensure_block_style_for_insertion`. We drive it directly to keep the unit test focused. + fn flow_style_insertion_adds_a_mapping_entry() { let fixture = "inputs:\n - { name: provider, url: http://main.example }\n"; let doc: crate::config_loader::source_patch::SourcePatchDocument = serde_saphyr::from_str(fixture).expect("parse"); let input = &doc.inputs[0].value; let anchor = span_byte_range(&input.name).expect("name span"); - let err = ensure_block_style_for_insertion(fixture, &anchor, "exp_date").expect_err("must reject"); - let msg = format!("{err}"); - assert!( - msg.contains("cannot insert optional field 'exp_date' into a flow-style YAML mapping"), - "unexpected error: {msg}" - ); + let edit = build_field_insertion_edit(fixture, &anchor, "enabled", "false").expect("build edit"); + let patched = edits_to_text(fixture, vec![edit]); + let reparsed: crate::config_loader::source_patch::SourcePatchDocument = + serde_saphyr::from_str(&patched).expect("reparse"); + + assert!(!reparsed.inputs[0].value.enabled.as_ref().expect("enabled").value); + assert!(patched.contains("name: provider, enabled: false")); } // ----------------------------------------------------------------------- @@ -1074,7 +1092,10 @@ mod tests { model::{ConfigPaths, InputType, SourcesConfigDto}, utils::Internable, }; - use std::time::{SystemTime, UNIX_EPOCH}; + use std::{ + fmt::Write as _, + time::{SystemTime, UNIX_EPOCH}, + }; use tuliprox_core::{ model::{Config, ConfigInput, MediaToolCapabilities, SourcesConfig}, utils::FileLockManager, @@ -1145,6 +1166,196 @@ sources: [] std::env::temp_dir().join(format!("tuliprox-source-yml-patch-{nanos}-{name}")) } + fn panel_alias_fixture(style: &str, indent: usize, newline: &str) -> String { + let mut text = String::from(concat!( + "inputs:\n- name: provider\n type: xtream\n url: provider://example\n", + " username: root\n password: pass\n", + " options:\n update_quality:\n live: 85\n vod: 85\n series: 85\n", + " resolve_background: false\n cache_duration: 20h\n", + " # aliases:\n # - {name: commented-old, url: provider://example}\n", + " max_connections: 1\n exp_date: 1788779143\n aliases:\n", + )); + for (name, expiry) in [("expired", 1), ("current", 4_102_444_800_u64)] { + let item = if style == "flow" || (style == "mixed" && name == "current") { + format!("- {{name: {name}, url: provider://example, username: {name}, password: pass, max_connections: 1, exp_date: {expiry}}}\n") + } else { + format!("- name: {name}\n url: provider://example\n username: {name}\n password: pass\n max_connections: 1\n exp_date: {expiry}\n") + }; + for line in item.lines() { + writeln!(text, "{}{line}", " ".repeat(indent)).expect("fixture line"); + } + } + text.push_str(concat!( + " # keep panel settings\n panel_api:\n enabled: true\n", + " url: http://panel.example\n api_key: synthetic-key\n", + " alias_pool:\n size:\n min: 1\n max: auto\n remove_expired: true\n", + "- name: next-input\n url: http://next.example\nsources: []\n", + )); + text.replace('\n', newline) + } + + #[tokio::test] + async fn panel_alias_layout_root_update_round_trips_block_flow_and_mixed_lists() { + for style in ["block", "flow", "mixed"] { + for indent in [2, 4] { + for newline in ["\n", "\r\n"] { + let directory = tempfile::tempdir().expect("temporary fixture"); + let source_path = directory.path().join("source.yml"); + let backup_dir = directory.path().join("backup"); + std::fs::create_dir(&backup_dir).expect("backup directory"); + let fixture = panel_alias_fixture(style, indent, newline); + std::fs::write(&source_path, &fixture).expect("write fixture"); + let before: SourcesConfigDto = serde_saphyr::from_str(&fixture).expect("original DTO"); + let app_cfg = build_app_config(&backup_dir); + let patches = [ + SourcesYmlPatch::SortAliases { + input_name: "provider".into(), + order: AliasExpDateSortOrder::NewestFirst, + }, + SourcesYmlPatch::UpdateRootCredentials { + input_name: "provider".into(), + username: "renewed-root".into(), + password: "renewed-pass".into(), + exp_date: Some(4_102_444_900), + }, + SourcesYmlPatch::UpdateAliasCredentials { + input_name: "provider".into(), + alias_name: "current".into(), + username: "renewed-alias".into(), + password: "pass # with: {}, commas".into(), + exp_date: Some(4_102_444_900), + }, + SourcesYmlPatch::AddAlias { + input_name: "provider".into(), + alias_name: "added".into(), + base_url: "provider://example".into(), + username: "added-user".into(), + password: "added-pass".into(), + exp_date: Some(4_102_445_000), + }, + SourcesYmlPatch::RemoveExpiredAliases { input_name: "provider".into() }, + SourcesYmlPatch::SortAliases { + input_name: "provider".into(), + order: AliasExpDateSortOrder::NewestFirst, + }, + ]; + assert!(execute_source_yml_patches(&app_cfg, &source_path, &patches).await.expect("patch")); + let patched = std::fs::read_to_string(&source_path).expect("read patched"); + let mut parsed: SourcesConfigDto = serde_saphyr::from_str(&patched).expect("reparse"); + let input = &parsed.inputs[0]; + assert_eq!(input.username.as_deref(), Some("renewed-root")); + assert_eq!(input.exp_date, Some(4_102_444_900)); + assert_eq!(input.options, before.inputs[0].options); + assert_eq!(input.panel_api, before.inputs[0].panel_api); + assert_eq!(parsed.inputs[1], before.inputs[1]); + let aliases = input.aliases.as_ref().expect("aliases"); + assert_eq!( + aliases.iter().map(|alias| alias.name.as_ref()).collect::>(), + ["added", "current"] + ); + assert_eq!(aliases[1].password.as_deref(), Some("pass # with: {}, commas")); + assert!(patched.contains("# - {name: commented-old, url: provider://example}")); + let panel_start = fixture.find(" # keep panel settings").expect("panel"); + assert!(patched.ends_with(&fixture[panel_start..])); + let current_prefix = if style == "block" { "- name: current" } else { "- {name: current" }; + assert!(patched.contains(current_prefix)); + // The initial sort makes the older entry the last item. In + // mixed lists its block style is the insertion convention. + let added_prefix = if style == "flow" { "- {name: added" } else { "- name: added" }; + assert!(patched.contains(added_prefix)); + assert!(std::fs::read_dir(&backup_dir).expect("backups").any(|entry| { + let path = entry.expect("backup entry").path(); + std::fs::read(path).is_ok_and(|bytes| bytes == fixture.as_bytes()) + })); + parsed.inputs[0] + .prepare(0, false, &HashSet::from(["example".to_string()]), None) + .expect("prepare updated config"); + assert_eq!(parsed.inputs[0].cache_duration_seconds, 72_000); + } + } + } + } + + #[tokio::test] + async fn panel_alias_layout_repeated_add_is_noop_and_conflict_preserves_root() { + for style in ["block", "flow"] { + let directory = tempfile::tempdir().expect("temporary fixture"); + let source_path = directory.path().join("source.yml"); + let backup_dir = directory.path().join("backup"); + std::fs::create_dir(&backup_dir).expect("backup directory"); + let original = panel_alias_fixture(style, 2, "\n"); + std::fs::write(&source_path, &original).expect("fixture"); + let app_cfg = build_app_config(&backup_dir); + let add = SourcesYmlPatch::AddAlias { + input_name: "provider".into(), + alias_name: "current".into(), + base_url: "provider://example".into(), + username: "current".into(), + password: "pass".into(), + exp_date: Some(4_102_444_800), + }; + assert!(!execute_source_yml_patches(&app_cfg, &source_path, &[add]).await.expect("identical addition")); + let patches = [ + SourcesYmlPatch::UpdateRootCredentials { + input_name: "provider".into(), + username: "new-root".into(), + password: "new-pass".into(), + exp_date: Some(4_102_444_900), + }, + SourcesYmlPatch::AddAlias { + input_name: "provider".into(), + alias_name: "current".into(), + base_url: "provider://example".into(), + username: "current".into(), + password: "conflicting-password".into(), + exp_date: Some(4_102_444_800), + }, + ]; + let error = execute_source_yml_patches(&app_cfg, &source_path, &patches).await.expect_err("conflict"); + assert!(error.to_string().contains("already exists")); + assert!(!error.to_string().contains("conflicting-password")); + assert_eq!(std::fs::read(&source_path).expect("unchanged file"), original.as_bytes()); + assert_eq!(std::fs::read_dir(&backup_dir).expect("backups").count(), 0); + } + } + + #[tokio::test] + async fn panel_alias_layout_replenishes_empty_list_with_block_default() { + for style in ["block", "flow"] { + let directory = tempfile::tempdir().expect("temporary fixture"); + let source_path = directory.path().join("source.yml"); + let backup_dir = directory.path().join("backup"); + std::fs::create_dir(&backup_dir).expect("backup directory"); + let original = panel_alias_fixture(style, 2, "\n").replace("4102444800", "2"); + std::fs::write(&source_path, &original).expect("fixture"); + let app_cfg = build_app_config(&backup_dir); + let patches = [ + SourcesYmlPatch::RemoveExpiredAliases { input_name: "provider".into() }, + SourcesYmlPatch::UpdateRootCredentials { + input_name: "provider".into(), + username: "new-root".into(), + password: "new-pass".into(), + exp_date: Some(4_102_444_900), + }, + SourcesYmlPatch::AddAlias { + input_name: "provider".into(), + alias_name: "added".into(), + base_url: "provider://example".into(), + username: "added-user".into(), + password: "added-pass".into(), + exp_date: Some(4_102_445_000), + }, + ]; + assert!(execute_source_yml_patches(&app_cfg, &source_path, &patches).await.expect("replenish")); + let result = std::fs::read_to_string(&source_path).expect("read"); + let parsed: SourcesConfigDto = serde_saphyr::from_str(&result).expect("reparse"); + assert_eq!(parsed.inputs[0].username.as_deref(), Some("new-root")); + assert_eq!(parsed.inputs[0].aliases.as_ref().expect("aliases").len(), 1); + assert!(result.contains(" aliases:\n - name: added\n")); + assert!(result.ends_with(&original[original.find(" # keep panel settings").expect("panel")..])); + } + } + #[tokio::test] async fn end_to_end_patch_inserts_exp_date_and_writes_backup() { let dir = unique_path("insert"); @@ -1220,6 +1431,64 @@ sources: [] let _ = std::fs::remove_dir_all(&dir); } + #[tokio::test] + async fn flow_style_expiry_disables_root_and_alias_accounts() { + let dir = unique_path("flow-expiry"); + std::fs::create_dir_all(&dir).expect("mkdir"); + let source_path = dir.join("source.yml"); + let backup_dir = dir.join("backup"); + std::fs::create_dir_all(&backup_dir).expect("mkdir backup"); + let fixture = concat!( + "inputs:\n", + " - {name: flow-root, type: xtream, url: http://root.example, username: root-user, password: root-pass, exp_date: 100}\n", + " - name: provider\n", + " type: xtream\n", + " url: http://main.example\n", + " username: user\n", + " password: pass\n", + " aliases:\n", + " - {name: flow-alias, url: http://alias.example, username: alias-user, password: alias-pass}\n", + "sources: []\n", + ); + tokio::fs::write(&source_path, fixture).await.expect("write fixture"); + let app_cfg = build_app_config(&backup_dir); + let commands = [ + SourcesYmlPatch::SetFetchedExpiry { + input_name: Arc::from("flow-root"), + account_name: Arc::from("flow-root"), + exp_date: 100, + disable: true, + }, + SourcesYmlPatch::SetFetchedExpiry { + input_name: Arc::from("provider"), + account_name: Arc::from("flow-alias"), + exp_date: 200, + disable: true, + }, + ]; + + assert!(execute_source_yml_patches(&app_cfg, &source_path, &commands).await.expect("patch")); + let patched_text = tokio::fs::read_to_string(&source_path).await.expect("read"); + let parsed: SourcesConfigDto = serde_saphyr::from_str(&patched_text).expect("reparse"); + let root = parsed.inputs.iter().find(|input| input.name.as_ref() == "flow-root").expect("root"); + let provider = parsed.inputs.iter().find(|input| input.name.as_ref() == "provider").expect("provider"); + let alias = provider + .aliases + .as_ref() + .and_then(|aliases| aliases.iter().find(|alias| alias.name.as_ref() == "flow-alias")) + .expect("alias"); + + assert!(!root.enabled); + assert_eq!(root.exp_date, Some(100)); + assert!(!alias.enabled); + assert_eq!(alias.exp_date, Some(200)); + assert!(patched_text.contains("name: flow-root, enabled: false")); + assert!(patched_text.contains("name: flow-alias, enabled: false")); + assert!(patched_text.contains("password: alias-pass, exp_date: 200")); + + let _ = std::fs::remove_dir_all(&dir); + } + #[tokio::test] async fn duplicate_alias_names_make_patch_fail_without_corrupting_file() { let dir = unique_path("dup"); @@ -1328,6 +1597,60 @@ sources: [] let _ = std::fs::remove_dir_all(&dir); } + #[tokio::test] + async fn adding_first_alias_preserves_panel_api_and_following_input() { + let dir = unique_path("first-alias"); + std::fs::create_dir_all(&dir).expect("mkdir"); + let source_path = dir.join("source.yml"); + let backup_dir = dir.join("backup"); + std::fs::create_dir_all(&backup_dir).expect("mkdir backup"); + let fixture = concat!( + "inputs:\n", + " - name: provider\n", + " type: xtream\n", + " url: http://main.example\n", + " username: user\n", + " password: pass\n", + " panel_api:\n", + " enabled: true\n", + " url: http://panel.example\n", + " api_key: test-key\n", + " alias_pool:\n", + " size:\n", + " min: 1\n", + " max: auto\n", + " remove_expired: true\n", + " - name: next\n", + " url: http://next.example\n", + "sources: []\n", + ); + tokio::fs::write(&source_path, fixture).await.expect("write fixture"); + let app_cfg = build_app_config(&backup_dir); + let patches = [SourcesYmlPatch::AddAlias { + input_name: Arc::from("provider"), + alias_name: Arc::from("provider-new"), + base_url: "http://new.example".to_string(), + username: "new-user".to_string(), + password: "new-pass".to_string(), + exp_date: Some(300), + }]; + + assert!(execute_source_yml_patches(&app_cfg, &source_path, &patches).await.expect("patch")); + let patched = tokio::fs::read_to_string(&source_path).await.expect("read"); + let parsed: SourcesConfigDto = serde_saphyr::from_str(&patched).expect("reparse"); + + assert_eq!(parsed.inputs.len(), 2); + assert!(parsed.inputs[0].panel_api.is_some()); + let aliases = parsed.inputs[0].aliases.as_ref().expect("aliases"); + assert_eq!(aliases.len(), 1); + assert_eq!(aliases[0].name.as_ref(), "provider-new"); + assert_eq!(parsed.inputs[1].name.as_ref(), "next"); + assert!(patched.contains(" aliases:\n - name: provider-new")); + assert!(patched.contains(" remove_expired: true")); + + let _ = std::fs::remove_dir_all(&dir); + } + #[tokio::test] async fn scalar_update_and_sort_aliases_are_both_persisted() { let dir = unique_path("update-sort"); diff --git a/backend/app/src/api/tasks/notification_bridge.rs b/backend/app/src/api/tasks/notification_bridge.rs index 4b66be14d..566b3fb36 100644 --- a/backend/app/src/api/tasks/notification_bridge.rs +++ b/backend/app/src/api/tasks/notification_bridge.rs @@ -565,10 +565,7 @@ mod tests { EventMessage::ActiveProvider("p".into(), 1), EventMessage::ConfigChange(ConfigType::Config), EventMessage::PlaylistUpdate(PlaylistUpdateSummary::state_only(PlaylistUpdateState::Success)), - EventMessage::PlaylistUpdateProgress(PlaylistUpdateProgressEvent { - target: String::new(), - message: String::new(), - }), + EventMessage::PlaylistUpdateProgress(PlaylistUpdateProgressEvent::global("", "")), EventMessage::SystemInfoUpdate(Arc::new(SystemInfo { cpu_usage: 0.0, memory_usage: 0, diff --git a/backend/config-loader/src/source_patch.rs b/backend/config-loader/src/source_patch.rs index 3ff713a33..a11677f62 100644 --- a/backend/config-loader/src/source_patch.rs +++ b/backend/config-loader/src/source_patch.rs @@ -4,7 +4,7 @@ use shared::{ model::{ConfigInputAliasDto, SourcesConfigDto}, utils::is_nullish, }; -use std::{collections::HashMap, fmt::Write, ops::Range}; +use std::{collections::HashMap, ops::Range}; #[derive(Debug, Clone)] pub struct TextEdit { @@ -119,40 +119,84 @@ pub fn line_end_offset(text: &str, byte_pos: usize) -> usize { } } -/// Returns true when the mapping owning `byte_range` is written in flow style (`{...}`). +/// Opening byte offset when the scalar owning `byte_range` is inside a flow mapping (`{...}`). /// -/// Scanning starts at the beginning of the line holding the value and walks backwards over -/// preceding lines while they belong to the same flow scope, so a value on a continuation line -/// of a multi-line flow mapping is still detected. -fn is_flow_style_mapping_at(text: &str, byte_range: &Range) -> bool { - let mut depth = 0i32; - for ch in text[..byte_range.start.min(text.len())].chars().rev() { - match ch { - '}' | ']' => depth += 1, - '{' | '[' if depth == 0 => return true, - '{' | '[' => depth -= 1, - '\n' if depth == 0 => break, +/// The lightweight scanner ignores quoted values and comments while tracking nested flow +/// collections. Opening delimiters are only recognized at YAML token boundaries so braces in +/// plain values such as environment placeholders do not create false flow scopes. +fn flow_mapping_start_at(text: &str, byte_range: &Range) -> Option { + let bytes = text.as_bytes(); + let limit = byte_range.start.min(bytes.len()); + let mut flow_stack = Vec::<(u8, usize)>::new(); + let mut in_single_quote = false; + let mut in_double_quote = false; + let mut in_comment = false; + let mut index = 0; + + while index < limit { + let byte = bytes[index]; + if in_comment { + if byte == b'\n' { + in_comment = false; + } + index += 1; + continue; + } + if in_single_quote { + if byte == b'\'' { + if index + 1 < limit && bytes[index + 1] == b'\'' { + index += 2; + continue; + } + in_single_quote = false; + } + index += 1; + continue; + } + if in_double_quote { + if byte == b'\\' { + index = (index + 2).min(limit); + continue; + } + if byte == b'"' { + in_double_quote = false; + } + index += 1; + continue; + } + + match byte { + b'\'' => in_single_quote = true, + b'"' => in_double_quote = true, + b'#' if index == 0 + || bytes[index - 1].is_ascii_whitespace() + || matches!(bytes[index - 1], b',' | b'{' | b'[') => + { + in_comment = true; + } + b'{' | b'[' + if index == 0 + || bytes[index - 1].is_ascii_whitespace() + || matches!(bytes[index - 1], b'-' | b':' | b',' | b'{' | b'[') => + { + flow_stack.push((byte, index)); + } + b'}' if flow_stack.last().is_some_and(|(delimiter, _)| *delimiter == b'{') => { + flow_stack.pop(); + } + b']' if flow_stack.last().is_some_and(|(delimiter, _)| *delimiter == b'[') => { + flow_stack.pop(); + } _ => {} } + index += 1; } - false + + flow_stack.last().filter(|(delimiter, _)| *delimiter == b'{').map(|(_, start)| *start) } -/// Rejects inserting a new key into a flow-style mapping. -/// -/// Replacing an existing scalar inside a flow mapping is safe because its span is explicit, -/// but inserting a whole `key: value` line is not representable there. -pub fn ensure_block_style_for_insertion( - text: &str, - anchor: &Range, - field_name: &str, -) -> Result<(), TuliproxError> { - if is_flow_style_mapping_at(text, anchor) { - return Err(TuliproxError::Config(format!( - "cannot insert optional field '{field_name}' into a flow-style YAML mapping; edit the account in block style" - ))); - } - Ok(()) +fn is_flow_style_mapping_at(text: &str, byte_range: &Range) -> bool { + flow_mapping_start_at(text, byte_range).is_some() } /// Byte offset of the end of the line holding `byte_pos`, excluding its line break. @@ -198,17 +242,19 @@ pub fn sibling_key_indent(text: &str, byte_pos: usize) -> usize { indent } -/// Builds an edit that inserts `key: value` on its own line directly below the anchor line. +/// Builds an edit that inserts `key: value` next to an existing scalar in the same mapping. /// -/// The anchor range must point at an existing sibling scalar in the same mapping; its line -/// indentation and the document's newline style are reused so no existing byte changes. +/// Block mappings receive a sibling line below the anchor. Flow mappings receive the new entry +/// immediately after the anchor value, before its existing separator or closing brace. pub fn build_field_insertion_edit( text: &str, anchor: &Range, key: &str, value: &str, ) -> Result { - ensure_block_style_for_insertion(text, anchor, key)?; + if is_flow_style_mapping_at(text, anchor) { + return Ok(TextEdit { range: anchor.end..anchor.end, replacement: format!(", {key}: {value}") }); + } let insert_at = line_content_end(text, anchor.end); let indent = sibling_key_indent(text, anchor.start); let newline = detect_newline(text); @@ -320,17 +366,30 @@ fn ensure_unique_alias_names(input: &serde_saphyr::Spanned, input_na Ok(()) } +fn alias_item_line_start(text: &str, alias: &serde_saphyr::Spanned) -> Result { + let alias_range = span_byte_range(alias)?; + let name_range = span_byte_range(&alias.value.name)?; + // A projected multiline flow map starts at its first child, not necessarily + // at the '{'. Use the owning map's opener to retain the sequence item's '-'. + let mapping_start = flow_mapping_start_at(text, &name_range).unwrap_or(alias_range.start); + let line_start = text[..mapping_start].rfind('\n').map_or(0, |pos| pos + 1); + if !text[line_start..mapping_start].trim_start().starts_with('-') && line_start > 0 { + let previous_start = text[..line_start - 1].rfind('\n').map_or(0, |pos| pos + 1); + if text[previous_start..line_start].trim() == "-" { + return Ok(previous_start); + } + } + Ok(line_start) +} + pub fn alias_item_block_range( text: &str, alias_span: &serde_saphyr::Spanned, ) -> Result, TuliproxError> { - let alias_range = span_byte_range(alias_span)?; let name_range = span_byte_range(&alias_span.value.name)?; - let line_start = text[..alias_range.start].rfind('\n').map_or(0, |p| p + 1); - let dash_pos = text[line_start..alias_range.start].find('-').map_or(line_start, |p| line_start + p); - let block_start = text[..dash_pos].rfind('\n').map_or(0, |p| p + 1); + let block_start = alias_item_line_start(text, alias_span)?; - let comment_start = find_owned_comment_start(text, block_start, line_start); + let comment_start = find_owned_comment_start(text, block_start, block_start); let block_end; let item_indent = text[block_start..].chars().take_while(|c| *c == ' ').count(); @@ -353,13 +412,21 @@ pub fn alias_item_block_range( continue; } let next_indent = next_line.chars().take_while(|c| *c == ' ').count(); - if next_indent < item_indent || (next_indent == item_indent && stripped.starts_with('-')) { + // Indentless block sequences put their '-' at the same column as the + // parent's next key. That key ends the item just like a following '-'. + // A closing line of a multiline flow mapping still belongs to the item. + if (next_indent < item_indent || (next_indent == item_indent && !stripped.starts_with('#'))) + && !is_flow_style_mapping_at(text, &(next_line_start..next_line_start)) + { block_end = next_line_start; break; } - if next_indent == item_indent && stripped.starts_with('#') { + if next_indent == item_indent + && stripped.starts_with('#') + && !is_flow_style_mapping_at(text, &(next_line_start..next_line_start)) + { let mut lookahead = next_line_end; - let mut comments_belong_to_next_item = false; + let mut comments_belong_to_next_node = false; while lookahead < text.len() { let lookahead_end = match text[lookahead..].find('\n') { Some(off) => lookahead + off + 1, @@ -372,13 +439,10 @@ pub fn alias_item_block_range( lookahead = lookahead_end; continue; } - if candidate_indent == item_indent && candidate_stripped.starts_with('-') { - comments_belong_to_next_item = true; - break; - } + comments_belong_to_next_node = candidate_indent <= item_indent && !candidate_stripped.is_empty(); break; } - if comments_belong_to_next_item { + if comments_belong_to_next_node { block_end = next_line_start; break; } @@ -418,32 +482,43 @@ pub fn serialize_alias_block( indent: usize, newline: &str, ) -> Result { + serialize_alias_item(alias, indent, newline, AliasMappingStyle::Block) +} + +#[derive(Clone, Copy)] +enum AliasMappingStyle { + Block, + Flow, +} + +fn alias_mapping_style(text: &str, alias: &PatchAlias) -> Result { + let name = span_byte_range(&alias.name)?; + Ok(if is_flow_style_mapping_at(text, &name) { AliasMappingStyle::Flow } else { AliasMappingStyle::Block }) +} + +fn serialize_alias_item( + alias: &ConfigInputAliasDto, + indent: usize, + newline: &str, + style: AliasMappingStyle, +) -> Result { + // Use the same DTO serializer as a regular config save, excluding the + // computed ID. Only the existing item's explicit flow style overrides it. + let mut alias = alias.clone(); + alias.id = 0; + let serialized = match style { + AliasMappingStyle::Block => serialize_yaml_scalar(&[alias])?, + AliasMappingStyle::Flow => serialize_yaml_scalar(&[serde_saphyr::FlowMap(alias)])?, + }; let pad = " ".repeat(indent); - let mut out = String::new(); - let _ = writeln!(out, "{pad}- name: {}", serialize_yaml_scalar(&alias.name.as_ref())?); - let _ = writeln!(out, "{pad} url: {}", serialize_yaml_scalar(&alias.url)?); - if let Some(username) = &alias.username { - let _ = writeln!(out, "{pad} username: {}", serialize_yaml_scalar(username)?); + Ok(serialized.lines().map(|line| format!("{pad}{line}")).collect::>().join(newline)) +} + +fn append_alias_text(output: &mut String, item: &str, newline: &str) { + if !output.is_empty() && !output.ends_with('\n') { + output.push_str(newline); } - if let Some(password) = &alias.password { - let _ = writeln!(out, "{pad} password: {}", serialize_yaml_scalar(password)?); - } - if alias.max_connections != 0 { - let _ = writeln!(out, "{pad} max_connections: {}", serialize_yaml_scalar(&alias.max_connections)?); - } - if let Some(exp_date) = alias.exp_date { - let _ = writeln!(out, "{pad} exp_date: {}", serialize_yaml_scalar(&exp_date)?); - } - if !alias.enabled { - let _ = writeln!(out, "{pad} enabled: false"); - } - if out.ends_with('\n') { - out.truncate(out.len() - 1); - } - if newline == "\r\n" { - out = out.replace('\n', "\r\n"); - } - Ok(out) + output.push_str(item); } pub fn build_alias_addition_edit( @@ -457,13 +532,40 @@ pub fn build_alias_addition_edit( if let Some(aliases_spanned) = &input.value.aliases { let aliases = &aliases_spanned.value; + if aliases.iter().any(|existing| existing.value.name.value == alias.name.as_ref()) { + return Err(TuliproxError::Config(format!( + "source.yml patch: alias '{}' already exists under input '{input_name}'", + alias.name + ))); + } if aliases.is_empty() { let aliases_range = span_byte_range(aliases_spanned)?; - let line_end = line_end_offset(text, aliases_range.end); - let indent = line_indent_at(text, aliases_range.start) + 2; - let block = serialize_alias_block(alias, indent, newline)?; - let insertion = format!("{newline}{block}{newline}"); - return Ok(TextEdit { range: line_end..line_end, replacement: insertion }); + let line_start = text[..aliases_range.start].rfind('\n').map_or(0, |pos| pos + 1); + let line_end = line_content_end(text, aliases_range.end); + let line = &text[line_start..line_end]; + let field_indent = line.chars().take_while(|ch| *ch == ' ' || *ch == '\t').count(); + let value = line[field_indent..] + .strip_prefix("aliases:") + .map(str::trim_start) + .ok_or_else(|| TuliproxError::Config("source.yml patch: empty aliases field not found".to_string()))?; + let closing_bracket = value.find(']').ok_or_else(|| { + TuliproxError::Config("source.yml patch: unsupported empty aliases sequence layout".to_string()) + })?; + if !value.starts_with('[') || !value[1..closing_bracket].trim().is_empty() { + return Err(TuliproxError::Config( + "source.yml patch: unsupported empty aliases sequence layout".to_string(), + )); + } + let trailing = value[closing_bracket + 1..].trim_start(); + if !trailing.is_empty() && !trailing.starts_with('#') { + return Err(TuliproxError::Config( + "source.yml patch: unsupported content after empty aliases sequence".to_string(), + )); + } + let block = serialize_alias_block(alias, field_indent + 2, newline)?; + let comment = if trailing.is_empty() { String::new() } else { format!(" {trailing}") }; + let replacement = format!("{:field_indent$}aliases:{comment}{newline}{block}", ""); + return Ok(TextEdit { range: line_start..line_end, replacement }); } let Some(last_alias) = aliases.last() else { return Err(TuliproxError::Config(format!( @@ -471,20 +573,20 @@ pub fn build_alias_addition_edit( ))); }; let last_block = alias_item_block_range(text, last_alias)?; - let item_indent = text[last_block.start..].chars().take_while(|c| *c == ' ').count(); - let block = serialize_alias_block(alias, item_indent, newline)?; - let insertion = format!("{block}{newline}"); + let item_indent = line_indent_at(text, alias_item_line_start(text, last_alias)?); + let style = alias_mapping_style(text, &last_alias.value)?; + let block = serialize_alias_item(alias, item_indent, newline, style)?; + let separator = if text[..last_block.end].ends_with('\n') { "" } else { newline }; + let insertion = format!("{separator}{block}{newline}"); return Ok(TextEdit { range: last_block.end..last_block.end, replacement: insertion }); } let input_span_range = span_byte_range(&input.value.name)?; - let input_indent = line_indent_at(text, input_span_range.start); - let aliases_indent = input_indent; - let item_indent = aliases_indent + 2; + let field_indent = sibling_key_indent(text, input_span_range.start); + let item_indent = field_indent + 2; // `aliases: null` (or an empty `aliases:`) deserializes to `None` but still occupies a line. // Replacing that line in place is what turns the null marker into a real block sequence. - let field_indent = sibling_key_indent(text, input_span_range.start); if let Some(null_line) = find_null_aliases_line(text, input_span_range.end, field_indent) { let block = serialize_alias_block(alias, field_indent + 2, newline)?; let pad = " ".repeat(field_indent); @@ -492,42 +594,30 @@ pub fn build_alias_addition_edit( return Ok(TextEdit { range: null_line, replacement }); } - let mut last_field_end: Option = None; - if let Some(f) = &input.value.panel_api { - last_field_end = Some(span_byte_range(f)?.end); + // Anchor the new sibling only to scalar input fields. A span from the reduced + // `PatchPanelApi` projection does not describe the complete nested mapping and can point + // into its first child instead of behind the `panel_api` block. + let mut anchor_end = input_span_range.end; + if let Some(field) = &input.value.max_connections { + anchor_end = anchor_end.max(span_byte_range(field)?.end); } - if last_field_end.is_none() { - if let Some(f) = &input.value.max_connections { - last_field_end = Some(span_byte_range(f)?.end); - } + if let Some(field) = &input.value.exp_date { + anchor_end = anchor_end.max(span_byte_range(field)?.end); } - if last_field_end.is_none() { - if let Some(f) = &input.value.exp_date { - last_field_end = Some(span_byte_range(f)?.end); - } + if let Some(field) = &input.value.password { + anchor_end = anchor_end.max(span_byte_range(field)?.end); } - if last_field_end.is_none() { - if let Some(f) = &input.value.password { - last_field_end = Some(span_byte_range(f)?.end); - } + if let Some(field) = &input.value.username { + anchor_end = anchor_end.max(span_byte_range(field)?.end); } - if last_field_end.is_none() { - if let Some(f) = &input.value.username { - last_field_end = Some(span_byte_range(f)?.end); - } - } - if last_field_end.is_none() { - if let Some(f) = &input.value.url { - last_field_end = Some(span_byte_range(f)?.end); - } + if let Some(field) = &input.value.url { + anchor_end = anchor_end.max(span_byte_range(field)?.end); } - let anchor_end = last_field_end.unwrap_or(input_span_range.end); - let line_end = line_end_offset(text, anchor_end); + let insert_at = line_content_end(text, anchor_end); let block = serialize_alias_block(alias, item_indent, newline)?; - let insertion = - format!("{newline}{:aliases_indent$}aliases:{newline}{block}{newline}", "", aliases_indent = aliases_indent); - Ok(TextEdit { range: line_end..line_end, replacement: insertion }) + let insertion = format!("{newline}{:field_indent$}aliases:{newline}{block}", "", field_indent = field_indent); + Ok(TextEdit { range: insert_at..insert_at, replacement: insertion }) } /// Scans forward from `start` for an `aliases:` line at `key_indent` whose value is empty or @@ -622,7 +712,7 @@ pub fn build_alias_sort_edit( "source.yml patch: alias '{name}' not found during sort for input '{input_name}'" )) })?; - reordered_text.push_str(&text[range.clone()]); + append_alias_text(&mut reordered_text, &text[range.clone()], detect_newline(text)); } Ok(Some(TextEdit { range: first_block_start..last_block_end, replacement: reordered_text })) @@ -662,17 +752,19 @@ pub fn build_alias_sequence_edit( .map(|range| range.end) .max() .ok_or_else(|| TuliproxError::Config("source.yml patch: alias sequence has no end".to_string()))?; - let first_name_start = - aliases.first().map(|alias| span_byte_range(&alias.value.name)).transpose()?.map_or(start, |range| range.start); - let item_indent = line_indent_at(text, first_name_start); + let last_alias = aliases + .last() + .ok_or_else(|| TuliproxError::Config("source.yml patch: alias sequence has no last item".to_string()))?; + let item_indent = line_indent_at(text, alias_item_line_start(text, last_alias)?); + let style = alias_mapping_style(text, &last_alias.value)?; let newline = detect_newline(text); let mut replacement = String::new(); for alias in expected_aliases { if let Some(range) = blocks.get(alias.name.as_ref()) { - replacement.push_str(&text[range.clone()]); + append_alias_text(&mut replacement, &text[range.clone()], newline); } else { - replacement.push_str(&serialize_alias_block(alias, item_indent, newline)?); + append_alias_text(&mut replacement, &serialize_alias_item(alias, item_indent, newline, style)?, newline); replacement.push_str(newline); } } @@ -683,6 +775,157 @@ pub fn build_alias_sequence_edit( #[cfg(test)] mod tests { use super::*; + use std::fmt::Write as _; + + fn alias_layout_fixture(indent: usize, style: &str, newline: &str) -> String { + let mut text = + String::from("inputs:\n- name: provider\n type: xtream\n url: provider://example\n aliases:\n"); + for (index, name) in ["older", "newer"].iter().enumerate() { + let item = match style { + "flow" => format!("- {{name: {name}, url: provider://example, exp_date: {index}}}\n"), + "multiline-flow" => { + format!("- {{\n name: {name},\n url: provider://example,\n exp_date: {index}\n# comment inside flow item\n}}\n") + } + "mixed" if index == 1 => format!("- {{name: {name}, url: provider://example, exp_date: {index}}}\n"), + "block" | "mixed" => format!("- name: {name}\n url: provider://example\n exp_date: {index}\n"), + _ => panic!("unsupported test style"), + }; + writeln!(text, "{}# comment for {name}", " ".repeat(indent)).expect("fixture comment"); + for line in item.lines() { + writeln!(text, "{}{line}", " ".repeat(indent)).expect("fixture line"); + } + } + text.push_str(concat!( + " # panel settings belong to the input\n", + " panel_api:\n", + " url: http://panel.example\n", + " api_key: synthetic-key\n", + " alias_pool:\n", + " size:\n", + " min: 1\n", + " max: auto\n", + " remove_expired: true\n", + "- name: next-input\n", + " url: http://next.example\n", + "sources: []\n", + )); + text.replace('\n', newline) + } + + #[test] + fn alias_layout_preserves_flow_block_and_mixed_items_and_following_panel() { + for indent in [2, 4] { + for style in ["block", "flow", "multiline-flow", "mixed"] { + for newline in ["\n", "\r\n"] { + let original = alias_layout_fixture(indent, style, newline); + let doc = parse_patch_document(&original).expect("projection"); + let before: SourcesConfigDto = serde_saphyr::from_str(&original).expect("original config"); + let aliases = &doc.inputs[0].value.aliases.as_ref().expect("aliases").value; + let suffix_start = original.find(" # panel settings").expect("panel comment"); + let prefix = &original[..original.find("aliases:").expect("aliases")]; + for alias in aliases { + let range = alias_item_block_range(&original, alias).expect("item range"); + assert!( + range.end <= suffix_start, + "indent={indent}, style={style}: {}", + &original[range.clone()] + ); + assert!(!original[range].contains("panel_api:")); + } + + let added = ConfigInputAliasDto { + id: 42, + name: "added".into(), + url: "provider://example".to_string(), + username: Some("user: name".to_string()), + password: Some("pass # with: {}, commas and 'quotes'".to_string()), + priority: 3, + exp_date: Some(500), + enabled: true, + ..Default::default() + }; + let mut expected = before.clone(); + expected.inputs[0].aliases.as_mut().expect("aliases").push(added.clone()); + let edit = build_alias_addition_edit(&original, &doc, "provider", &added).expect("add"); + let appended = apply_scalar_edits(&original, vec![edit]).expect("apply"); + parse_and_validate_patched_text(&appended, &expected).expect("validate addition"); + assert!(appended.starts_with(prefix)); + assert!(appended.ends_with(&original[suffix_start..])); + let new_item_prefix = if style == "block" { "- name: added" } else { "- {name: added" }; + assert!(appended.contains(new_item_prefix), "style={style}: {appended}"); + assert!(!appended.contains("id: 42")); + if newline == "\r\n" { + assert!(!appended.replace("\r\n", "").contains('\n')); + } + + let mut sorted_expected = before.clone(); + sorted_expected.inputs[0].aliases.as_mut().expect("aliases").reverse(); + let sort = build_alias_sort_edit(&original, &doc, "provider", &["newer", "older"]) + .expect("sort") + .expect("sort edit"); + let sorted = apply_scalar_edits(&original, vec![sort]).expect("apply sort"); + parse_and_validate_patched_text(&sorted, &sorted_expected).expect("validate sort"); + assert!(sorted.ends_with(&original[suffix_start..])); + + let sorted_aliases = sorted_expected.inputs[0].aliases.as_mut().expect("aliases"); + sorted_aliases.insert(0, added.clone()); + let sequence_edit = build_alias_sequence_edit(&original, &doc, "provider", sorted_aliases) + .expect("rebuild") + .expect("sequence edit"); + let rebuilt = apply_scalar_edits(&original, vec![sequence_edit]).expect("apply rebuild"); + parse_and_validate_patched_text(&rebuilt, &sorted_expected).expect("validate rebuild"); + assert!(rebuilt.contains(new_item_prefix)); + assert!(rebuilt.ends_with(&original[suffix_start..])); + + for removed_names in [&["newer"][..], &["older", "newer"][..]] { + let mut removal_expected = before.clone(); + removal_expected.inputs[0] + .aliases + .as_mut() + .expect("aliases") + .retain(|alias| !removed_names.contains(&alias.name.as_ref())); + let edits = + build_alias_removal_edits(&original, &doc, "provider", removed_names).expect("remove"); + let removed = apply_scalar_edits(&original, edits).expect("apply removal"); + parse_and_validate_patched_text(&removed, &removal_expected).expect("validate removal"); + assert!(removed.ends_with(&original[suffix_start..])); + } + } + } + } + } + + #[test] + fn alias_layout_stops_at_root_scalars_and_preserves_commented_old_aliases() { + let original = alias_layout_fixture(2, "flow", "\n") + .replace(" # panel settings", " max_connections: 1\n exp_date: 1788779143\n # panel settings") + .replace(" aliases:\n", " # aliases:\n # - {name: disabled-old, url: provider://example}\n aliases:\n"); + let doc = parse_patch_document(&original).expect("projection"); + let mut expected: SourcesConfigDto = serde_saphyr::from_str(&original).expect("config"); + expected.inputs[0].aliases = None; + let edits = build_alias_removal_edits(&original, &doc, "provider", &["older", "newer"]).expect("remove"); + let result = apply_scalar_edits(&original, edits).expect("apply"); + parse_and_validate_patched_text(&result, &expected).expect("root fields retained"); + assert!(result.contains(" # - {name: disabled-old, url: provider://example}")); + assert!(result.contains(" max_connections: 1\n exp_date: 1788779143\n")); + } + + #[test] + fn alias_layout_add_and_sort_handle_missing_final_newline() { + let original = "sources: []\ninputs:\n- name: provider\n url: provider://example\n aliases:\n - {name: older, url: provider://example}\n - {name: newer, url: provider://example}"; + let doc = parse_patch_document(original).expect("projection"); + let mut expected: SourcesConfigDto = serde_saphyr::from_str(original).expect("config"); + expected.inputs[0].aliases.as_mut().expect("aliases").reverse(); + let sort = build_alias_sort_edit(original, &doc, "provider", &["newer", "older"]).expect("sort").expect("edit"); + let sorted = apply_scalar_edits(original, vec![sort]).expect("apply sort"); + parse_and_validate_patched_text(&sorted, &expected).expect("separate item lines after sort"); + let added = + ConfigInputAliasDto { name: "added".into(), url: "provider://example".into(), ..Default::default() }; + let edit = build_alias_addition_edit(original, &doc, "provider", &added).expect("add"); + let appended = apply_scalar_edits(original, vec![edit]).expect("apply add"); + assert!(appended.contains("url: provider://example}\n - {name: added")); + parse_patch_document(&appended).expect("separate item lines after addition"); + } const FIXTURE: &str = concat!( "templates:\n", @@ -877,6 +1120,36 @@ mod tests { assert_eq!(patched, "inputs:\n - {name: test, exp_date: 99}\n"); } + #[test] + fn flow_style_field_insertion_preserves_single_and_multiline_mappings() { + let fixtures = [ + "inputs:\n - {name: test, url: http://main.example}\n", + concat!( + "inputs:\n", + " - {\n", + " name: test, # keep account note\n", + " url: http://main.example\n", + " }\n", + ), + ]; + + for fixture in fixtures { + let doc: SourcePatchDocument = serde_saphyr::from_str(fixture).expect("projection parses"); + let name_range = span_byte_range(&doc.inputs[0].value.name).expect("name span"); + let edit = build_field_insertion_edit(fixture, &name_range, "enabled", "false").expect("build edit"); + let patched = apply_scalar_edits(fixture, vec![edit]).expect("apply"); + let reparsed: SourcePatchDocument = serde_saphyr::from_str(&patched).expect("reparse"); + let input = &reparsed.inputs[0].value; + + assert!(!input.enabled.as_ref().expect("enabled").value); + assert_eq!(input.url.as_ref().expect("url").value, "http://main.example"); + assert!(patched.contains("name: test, enabled: false")); + if fixture.contains("# keep account note") { + assert!(patched.contains("# keep account note")); + } + } + } + #[test] fn quoted_values_survive_unchanged() { let doc: SourcePatchDocument = serde_saphyr::from_str(FIXTURE).expect("projection parses"); @@ -956,8 +1229,21 @@ mod tests { } #[test] - fn add_first_alias_when_aliases_absent() { - let no_aliases_fixture = "inputs:\n - name: solo\n url: http://solo.example\n"; + fn add_first_alias_when_aliases_absent_preserves_nested_fields_and_next_input() { + let no_aliases_fixture = concat!( + "inputs:\n", + " - name: solo\n", + " url: http://solo.example\n", + " panel_api:\n", + " enabled: true\n", + " alias_pool:\n", + " size:\n", + " min: 1\n", + " max: auto\n", + " remove_expired: true\n", + " - name: next\n", + " url: http://next.example\n", + ); let doc: SourcePatchDocument = serde_saphyr::from_str(no_aliases_fixture).expect("projection parses"); let new_alias = ConfigInputAliasDto { name: "solo-alias".into(), @@ -973,6 +1259,46 @@ mod tests { assert!(patched.contains("aliases:")); assert!(patched.contains("- name: solo-alias")); assert!(patched.contains("url: http://alias.example")); + assert!(patched.contains(" aliases:\n - name: solo-alias")); + assert!(patched.contains(" remove_expired: true")); + + let reparsed: SourcePatchDocument = serde_saphyr::from_str(&patched).expect("reparse"); + assert_eq!(reparsed.inputs.len(), 2); + let aliases = reparsed.inputs[0].value.aliases.as_ref().expect("aliases"); + assert_eq!(aliases.value.len(), 1); + assert_eq!(aliases.value[0].value.name.value, "solo-alias"); + assert_eq!(reparsed.inputs[1].value.name.value, "next"); + } + + #[test] + fn add_first_alias_replaces_empty_flow_sequence() { + let fixture = concat!( + "inputs:\n", + " - name: provider\n", + " url: http://main.example\n", + " aliases: [] # keep alias note\n", + " - name: next\n", + " url: http://next.example\n", + ); + let doc: SourcePatchDocument = serde_saphyr::from_str(fixture).expect("projection parses"); + let new_alias = ConfigInputAliasDto { + name: "provider-backup".into(), + url: "http://backup.example".to_string(), + enabled: true, + ..Default::default() + }; + + let edit = build_alias_addition_edit(fixture, &doc, "provider", &new_alias).expect("build edit"); + let patched = apply_scalar_edits(fixture, vec![edit]).expect("apply"); + + assert!(!patched.contains("aliases: []")); + assert!( + patched.contains("aliases: # keep alias note\n - name: provider-backup"), + "unexpected patch:\n{patched}" + ); + let reparsed: SourcePatchDocument = serde_saphyr::from_str(&patched).expect("reparse"); + assert_eq!(reparsed.inputs.len(), 2); + assert_eq!(reparsed.inputs[0].value.aliases.as_ref().expect("aliases").value.len(), 1); } #[test] @@ -1057,7 +1383,7 @@ mod tests { assert_eq!(aliases.value[0].value.name.value, "newest"); assert_eq!(aliases.value[0].value.max_connections.as_ref().expect("max connections").value, 2); assert_eq!(aliases.value[1].value.name.value, "oldest"); - assert_eq!(aliases.value[1].value.enabled.as_ref().expect("enabled").value, false); + assert!(!aliases.value[1].value.enabled.as_ref().expect("enabled").value); } #[test] diff --git a/backend/core/src/model/config/input.rs b/backend/core/src/model/config/input.rs index ce5933658..19d2efbd7 100644 --- a/backend/core/src/model/config/input.rs +++ b/backend/core/src/model/config/input.rs @@ -1,5 +1,5 @@ use crate::{ - model::{macros, ConfigProvider, EpgConfig, PanelApiConfig}, + model::{macros, ConfigInputUpdateQuality, ConfigProvider, EpgConfig, PanelApiConfig}, utils::get_csv_file_path, }; use chrono::Utc; @@ -58,6 +58,7 @@ create_bitset!( #[derive(Debug, Clone)] pub struct ConfigInputOptions { pub flags: ConfigInputFlagsSet, + pub update_quality: ConfigInputUpdateQuality, pub resolve_delay: u16, pub probe_delay: u16, pub probe_live_interval_hours: u32, @@ -103,6 +104,7 @@ impl From<&ConfigInputOptionsDto> for ConfigInputOptions { Self { flags, + update_quality: ConfigInputUpdateQuality::from(&dto.update_quality), resolve_delay: dto.resolve_delay, probe_delay: dto.probe_delay, probe_live_interval_hours: dto.probe_live_interval_hours, @@ -1079,7 +1081,8 @@ mod tests { use super::*; use crate::model::ConfigProvider; use shared::model::{ - ConfigProviderDto, MediaServerInputConfigDto, MediaServerLibrarySelector, ProviderUrlSelectionPolicy, + ConfigInputUpdateQualityDto, ConfigProviderDto, MediaServerInputConfigDto, MediaServerLibrarySelector, + ProviderUrlSelectionPolicy, XtreamCluster, }; use std::{borrow::Cow, sync::Arc}; @@ -1092,11 +1095,92 @@ mod tests { #[test] fn input_options_conversion_sets_disable_hls_streaming_flag() { - let dto = ConfigInputOptionsDto { disable_hls_streaming: true, ..ConfigInputOptionsDto::default() }; + let dto = ConfigInputOptionsDto { + disable_hls_streaming: true, + update_quality: ConfigInputUpdateQualityDto { live: 95, vod: 90, series: 85 }, + ..ConfigInputOptionsDto::default() + }; let options = ConfigInputOptions::from(&dto); assert!(options.has_flag(ConfigInputFlags::DisableHlsStreaming)); + assert_eq!(options.update_quality.threshold(XtreamCluster::Live), 95); + assert_eq!(options.update_quality.threshold(XtreamCluster::Video), 90); + assert_eq!(options.update_quality.threshold(XtreamCluster::Series), 85); + } + + fn quality_options_dto() -> ConfigInputOptionsDto { + ConfigInputOptionsDto { + update_quality: ConfigInputUpdateQualityDto { live: 95, vod: 90, series: 85 }, + ..ConfigInputOptionsDto::default() + } + } + + fn assert_update_quality_is_preserved(input: &ConfigInput) { + let options = input.options.as_ref().expect("runtime input options"); + assert_eq!(options.update_quality.threshold(XtreamCluster::Live), 95); + assert_eq!(options.update_quality.threshold(XtreamCluster::Video), 90); + assert_eq!(options.update_quality.threshold(XtreamCluster::Series), 85); + } + + #[test] + fn alias_conversion_inherits_update_quality() { + let dto = ConfigInputDto { + input_type: InputType::Xtream, + options: Some(quality_options_dto()), + aliases: Some(vec![ConfigInputAliasDto { + name: "alias".into(), + url: "https://alias.example.invalid".to_string(), + ..ConfigInputAliasDto::default() + }]), + ..ConfigInputDto::default() + }; + let input = ConfigInput::from(&dto); + let alias = input.aliases.as_ref().and_then(|aliases| aliases.first()).expect("runtime alias"); + + let alias_input = input.as_input(alias); + + assert_update_quality_is_preserved(&alias_input); + } + + #[test] + fn batch_conversion_preserves_update_quality_when_promoting_alias() { + let mut input = ConfigInput::from(&ConfigInputDto { + input_type: InputType::XtreamBatch, + url: "batch:///tmp/aliases.csv".to_string(), + enabled: true, + options: Some(quality_options_dto()), + aliases: Some(vec![ConfigInputAliasDto { + name: "alias".into(), + url: "https://alias.example.invalid".to_string(), + username: Some("user".to_string()), + password: Some("password".to_string()), + enabled: true, + ..ConfigInputAliasDto::default() + }]), + ..ConfigInputDto::default() + }); + + let _ = input.prepare_batch(); + + assert_eq!(input.input_type, InputType::Xtream); + assert_update_quality_is_preserved(&input); + } + + #[test] + fn staged_xtream_conversion_preserves_update_quality() { + let mut input = ConfigInput::from(&ConfigInputDto { + input_type: InputType::Staged, + staged_type: StagedInputType::Xtream, + options: Some(quality_options_dto()), + staged: Some(ConfigInputStagedDto { for_input: Some("provider".into()), clusters: ClusterFlags::all() }), + ..ConfigInputDto::default() + }); + + input.resolve_staged_download_type(); + + assert_eq!(input.input_type, InputType::Xtream); + assert_update_quality_is_preserved(&input); } #[test] diff --git a/backend/core/src/model/mod.rs b/backend/core/src/model/mod.rs index e20820eaf..31f60a11b 100644 --- a/backend/core/src/model/mod.rs +++ b/backend/core/src/model/mod.rs @@ -25,6 +25,7 @@ pub mod stalker_record; pub mod stream_error; pub mod target_bouquet; pub mod update_guard; +mod update_quality; pub mod update_task; pub mod user_api_request; pub mod xtream_response; @@ -33,6 +34,6 @@ pub use self::{ auth_rejection::*, batch_result_collector::*, config::*, custom_stream_flags::*, fingerprint::*, healthcheck::*, input_source::*, mapping::*, messaging::*, notification::*, playlist_filter::*, playlist_key::*, provider::*, proxy_redirect::*, stalker_record::*, stream_error::*, stream_history::*, target_bouquet::*, update_guard::*, - update_task::*, xmltv::*, xtream::*, + update_quality::*, update_task::*, xmltv::*, xtream::*, }; pub use shared::model::xtream_const::*; diff --git a/backend/core/src/model/update_quality.rs b/backend/core/src/model/update_quality.rs new file mode 100644 index 000000000..8405a67e9 --- /dev/null +++ b/backend/core/src/model/update_quality.rs @@ -0,0 +1,439 @@ +use shared::model::{ConfigInputUpdateQualityDto, XtreamCluster}; + +const MAX_UPDATE_QUALITY_THRESHOLD: u8 = 100; + +/// Resolved per-cluster thresholds for the input update quality guard. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub struct ConfigInputUpdateQuality { + live: u8, + vod: u8, + series: u8, +} + +impl ConfigInputUpdateQuality { + /// Returns the configured threshold for an Xtream-compatible cluster. + #[must_use] + pub const fn threshold(&self, cluster: XtreamCluster) -> u8 { + match cluster { + XtreamCluster::Live => self.live, + XtreamCluster::Video => self.vod, + XtreamCluster::Series => self.series, + } + } +} + +impl From<&ConfigInputUpdateQualityDto> for ConfigInputUpdateQuality { + fn from(dto: &ConfigInputUpdateQualityDto) -> Self { Self { live: dto.live, vod: dto.vod, series: dto.series } } +} + +/// Typed outcome of evaluating one candidate cluster against its active baseline. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum UpdateQualityDecision { + Disabled, + BootstrapAccepted { candidate: usize, threshold: u8 }, + Accepted { current: usize, candidate: usize, threshold: u8, quality: u8 }, + Rejected { current: usize, candidate: usize, threshold: u8, quality: u8 }, + RejectedWithoutBaseline { candidate: usize, threshold: u8 }, +} + +impl UpdateQualityDecision { + /// Preserves an already evaluated accepted Quality result for downstream reporting. + #[must_use] + pub const fn acceptance(self, cluster: XtreamCluster) -> Option { + match self { + Self::BootstrapAccepted { candidate, threshold } => Some(ClusterUpdateAcceptance { + cluster, + current_count: None, + candidate_count: candidate, + threshold, + quality: None, + }), + Self::Accepted { current, candidate, threshold, quality } => Some(ClusterUpdateAcceptance { + cluster, + current_count: Some(current), + candidate_count: candidate, + threshold, + quality: Some(quality), + }), + Self::Disabled | Self::Rejected { .. } | Self::RejectedWithoutBaseline { .. } => None, + } + } + + /// Preserves an already evaluated rejected Quality result for downstream reporting. + #[must_use] + pub const fn rejection(self, cluster: XtreamCluster) -> Option { + match self { + Self::Rejected { current, candidate, threshold, quality } => Some(ClusterUpdateRejection { + cluster, + current_count: current, + candidate_count: candidate, + threshold, + quality, + }), + Self::RejectedWithoutBaseline { candidate, threshold } => Some(ClusterUpdateRejection { + cluster, + current_count: 0, + candidate_count: candidate, + threshold, + quality: 0, + }), + Self::Disabled | Self::BootstrapAccepted { .. } | Self::Accepted { .. } => None, + } + } +} + +/// Completed normal Quality evaluation whose candidate was accepted. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct ClusterUpdateAcceptance { + pub cluster: XtreamCluster, + pub current_count: Option, + pub candidate_count: usize, + pub threshold: u8, + pub quality: Option, +} + +/// Nonfatal report for one cluster whose candidate update was rejected. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct ClusterUpdateRejection { + pub cluster: XtreamCluster, + pub current_count: usize, + pub candidate_count: usize, + pub threshold: u8, + pub quality: u8, +} + +/// Audit report for one cluster accepted through a request-local quality bypass. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct ClusterForceUpdate { + pub cluster: XtreamCluster, + pub candidate_count: usize, + pub configured_threshold: u8, +} + +/// Evaluates a candidate count without I/O or floating-point arithmetic. +/// +/// A missing count is treated as having no baseline. Config +/// validation guarantees thresholds in `0..=100`; directly supplied values +/// above 100 are defensively normalized to 100. +#[must_use] +pub fn evaluate_update_quality(current: Option, candidate: usize, threshold: u8) -> UpdateQualityDecision { + let threshold = threshold.min(MAX_UPDATE_QUALITY_THRESHOLD); + if threshold == 0 { + return UpdateQualityDecision::Disabled; + } + + let Some(current) = current.filter(|value| *value > 0) else { + return if candidate > 0 { + UpdateQualityDecision::BootstrapAccepted { candidate, threshold } + } else { + UpdateQualityDecision::RejectedWithoutBaseline { candidate, threshold } + }; + }; + + if current == 0 { + return if candidate == 0 { + UpdateQualityDecision::Accepted { current: 0, candidate: 0, threshold, quality: 100 } + } else { + UpdateQualityDecision::Rejected { current: 0, candidate, threshold, quality: 0 } + }; + } + + let difference = current.abs_diff(candidate) as u128; + let current_wide = current as u128; + let scaled_difference = difference * 100; + let allowed_difference = current_wide * u128::from(100_u8.saturating_sub(threshold)); + let accepted = scaled_difference <= allowed_difference; + + let rounded_deviation = + scaled_difference / current_wide + u128::from(!scaled_difference.is_multiple_of(current_wide)); + let Ok(quality) = u8::try_from(100_u128.saturating_sub(rounded_deviation)) else { + return UpdateQualityDecision::Rejected { current, candidate, threshold, quality: 0 }; + }; + + if accepted { + UpdateQualityDecision::Accepted { current, candidate, threshold, quality } + } else { + UpdateQualityDecision::Rejected { current, candidate, threshold, quality } + } +} + +#[cfg(test)] +mod tests { + use super::{ + evaluate_update_quality, ClusterUpdateAcceptance, ClusterUpdateRejection, ConfigInputUpdateQuality, + UpdateQualityDecision, + }; + use shared::model::{ConfigInputUpdateQualityDto, XtreamCluster}; + + struct Case { + name: &'static str, + current: Option, + candidate: usize, + threshold: u8, + expected: UpdateQualityDecision, + } + + fn assert_cases(cases: &[Case]) { + for case in cases { + let decision = evaluate_update_quality(case.current, case.candidate, case.threshold); + if let UpdateQualityDecision::Accepted { threshold, quality, .. } = decision { + assert!(threshold <= 100, "accepted threshold exceeds 100 in case: {}", case.name); + assert!(quality >= threshold, "accepted quality is below threshold in case: {}", case.name); + } + assert_eq!(decision, case.expected, "case: {}", case.name); + } + } + + #[test] + fn runtime_policy_maps_thresholds_to_clusters() { + let policy = ConfigInputUpdateQuality::from(&ConfigInputUpdateQualityDto { live: 95, vod: 90, series: 85 }); + + assert_eq!(policy.threshold(XtreamCluster::Live), 95); + assert_eq!(policy.threshold(XtreamCluster::Video), 90); + assert_eq!(policy.threshold(XtreamCluster::Series), 85); + } + + #[test] + fn evaluates_disabled_and_bootstrap_cases() { + assert_cases(&[ + Case { + name: "disabled check preserves existing behavior", + current: Some(1_000), + candidate: 0, + threshold: 0, + expected: UpdateQualityDecision::Disabled, + }, + Case { + name: "missing baseline accepts a non-empty bootstrap", + current: None, + candidate: 42, + threshold: 90, + expected: UpdateQualityDecision::BootstrapAccepted { candidate: 42, threshold: 90 }, + }, + Case { + name: "zero baseline accepts a non-empty bootstrap", + current: Some(0), + candidate: 42, + threshold: 90, + expected: UpdateQualityDecision::BootstrapAccepted { candidate: 42, threshold: 90 }, + }, + Case { + name: "missing baseline rejects an empty candidate", + current: None, + candidate: 0, + threshold: 90, + expected: UpdateQualityDecision::RejectedWithoutBaseline { candidate: 0, threshold: 90 }, + }, + ]); + } + + #[test] + fn pipeline_transparency_quality_reports_preserve_evaluated_acceptance_and_rejection() { + let accepted = evaluate_update_quality(Some(1_000), 950, 90); + assert_eq!( + accepted.acceptance(XtreamCluster::Live), + Some(ClusterUpdateAcceptance { + cluster: XtreamCluster::Live, + current_count: Some(1_000), + candidate_count: 950, + threshold: 90, + quality: Some(95), + }) + ); + assert_eq!(accepted.rejection(XtreamCluster::Live), None); + + let bootstrap = evaluate_update_quality(None, 42, 90); + assert_eq!( + bootstrap.acceptance(XtreamCluster::Video), + Some(ClusterUpdateAcceptance { + cluster: XtreamCluster::Video, + current_count: None, + candidate_count: 42, + threshold: 90, + quality: None, + }) + ); + + let rejected = evaluate_update_quality(None, 0, 90); + assert_eq!( + rejected.rejection(XtreamCluster::Series), + Some(ClusterUpdateRejection { + cluster: XtreamCluster::Series, + current_count: 0, + candidate_count: 0, + threshold: 90, + quality: 0, + }) + ); + assert_eq!(evaluate_update_quality(Some(100), 50, 0).acceptance(XtreamCluster::Live), None); + } + + #[test] + fn evaluates_90_percent_boundaries() { + assert_cases(&[ + Case { + name: "lower 90 percent boundary is accepted", + current: Some(1_000), + candidate: 900, + threshold: 90, + expected: UpdateQualityDecision::Accepted { + current: 1_000, + candidate: 900, + threshold: 90, + quality: 90, + }, + }, + Case { + name: "upper 90 percent boundary is accepted", + current: Some(1_000), + candidate: 1_100, + threshold: 90, + expected: UpdateQualityDecision::Accepted { + current: 1_000, + candidate: 1_100, + threshold: 90, + quality: 90, + }, + }, + Case { + name: "one below lower 90 percent boundary is rejected", + current: Some(1_000), + candidate: 899, + threshold: 90, + expected: UpdateQualityDecision::Rejected { + current: 1_000, + candidate: 899, + threshold: 90, + quality: 89, + }, + }, + Case { + name: "one above upper 90 percent boundary is rejected", + current: Some(1_000), + candidate: 1_101, + threshold: 90, + expected: UpdateQualityDecision::Rejected { + current: 1_000, + candidate: 1_101, + threshold: 90, + quality: 89, + }, + }, + ]); + } + + #[test] + fn evaluates_exact_empty_and_large_counter_cases() { + assert_cases(&[ + Case { + name: "threshold 100 accepts an exact count", + current: Some(1_000), + candidate: 1_000, + threshold: 100, + expected: UpdateQualityDecision::Accepted { + current: 1_000, + candidate: 1_000, + threshold: 100, + quality: 100, + }, + }, + Case { + name: "threshold 100 rejects any difference", + current: Some(1_000), + candidate: 999, + threshold: 100, + expected: UpdateQualityDecision::Rejected { + current: 1_000, + candidate: 999, + threshold: 100, + quality: 99, + }, + }, + Case { + name: "existing baseline rejects an empty candidate", + current: Some(1_000), + candidate: 0, + threshold: 90, + expected: UpdateQualityDecision::Rejected { current: 1_000, candidate: 0, threshold: 90, quality: 0 }, + }, + Case { + name: "large counters do not overflow", + current: Some(usize::MAX), + candidate: usize::MAX - 1, + threshold: 99, + expected: UpdateQualityDecision::Accepted { + current: usize::MAX, + candidate: usize::MAX - 1, + threshold: 99, + quality: 99, + }, + }, + Case { + name: "large difference products do not overflow", + current: Some(usize::MAX), + candidate: 0, + threshold: 1, + expected: UpdateQualityDecision::Rejected { + current: usize::MAX, + candidate: 0, + threshold: 1, + quality: 0, + }, + }, + ]); + } + + #[test] + fn normalizes_out_of_range_thresholds_before_deciding() { + assert_cases(&[ + Case { + name: "threshold 101 accepts an identical count as normalized 100", + current: Some(1_000), + candidate: 1_000, + threshold: 101, + expected: UpdateQualityDecision::Accepted { + current: 1_000, + candidate: 1_000, + threshold: 100, + quality: 100, + }, + }, + Case { + name: "threshold 101 rejects a differing count as normalized 100", + current: Some(1_000), + candidate: 999, + threshold: 101, + expected: UpdateQualityDecision::Rejected { + current: 1_000, + candidate: 999, + threshold: 100, + quality: 99, + }, + }, + Case { + name: "maximum u8 threshold accepts an identical count as normalized 100", + current: Some(1_000), + candidate: 1_000, + threshold: u8::MAX, + expected: UpdateQualityDecision::Accepted { + current: 1_000, + candidate: 1_000, + threshold: 100, + quality: 100, + }, + }, + Case { + name: "maximum u8 threshold rejects a differing count as normalized 100", + current: Some(1_000), + candidate: 1_001, + threshold: u8::MAX, + expected: UpdateQualityDecision::Rejected { + current: 1_000, + candidate: 1_001, + threshold: 100, + quality: 99, + }, + }, + ]); + } +} diff --git a/backend/iptv/src/m3u/client.rs b/backend/iptv/src/m3u/client.rs index d05216055..54224b455 100644 --- a/backend/iptv/src/m3u/client.rs +++ b/backend/iptv/src/m3u/client.rs @@ -22,13 +22,36 @@ pub async fn download_m3u_playlist_from_source( input: &ConfigInput, explicit_source: Option, ) -> (Vec, Vec) { - let storage_dir = &cfg.storage_dir; - let input_source: InputSource = explicit_source.unwrap_or_else(|| input.into()); - let persist_file_path = prepare_file_path(input.persist.as_deref(), storage_dir, ""); - match request::get_input_text_content_as_stream(app_config, client, &input_source, storage_dir, persist_file_path) - .await - { + match download_reader(app_config, client, cfg, input, explicit_source).await { Ok(reader) => (m3u::parse_m3u(cfg, input, reader).await, vec![]), Err(err) => (vec![], vec![err]), } } + +pub(crate) async fn download_m3u_playlist_for_update( + app_config: &Arc, + client: &reqwest::Client, + cfg: &Arc, + input: &ConfigInput, +) -> (Vec, Vec) { + match download_reader(app_config, client, cfg, input, None).await { + Ok(reader) => match m3u::parse_m3u_for_update(cfg, input, reader).await { + Ok(groups) => (groups, vec![]), + Err(error) => (vec![], vec![TuliproxError::Download(format!("M3U update parsing failed: {error}"))]), + }, + Err(error) => (vec![], vec![error]), + } +} + +async fn download_reader( + app_config: &Arc, + client: &reqwest::Client, + cfg: &Config, + input: &ConfigInput, + explicit_source: Option, +) -> Result { + let storage_dir = &cfg.storage_dir; + let input_source: InputSource = explicit_source.unwrap_or_else(|| input.into()); + let persist_file_path = prepare_file_path(input.persist.as_deref(), storage_dir, ""); + request::get_input_text_content_as_stream(app_config, client, &input_source, storage_dir, persist_file_path).await +} diff --git a/backend/iptv/src/provider.rs b/backend/iptv/src/provider.rs index 16c23622b..98f068233 100644 --- a/backend/iptv/src/provider.rs +++ b/backend/iptv/src/provider.rs @@ -19,10 +19,12 @@ use crate::error::ProviderErrorKind; use shared::{ error::TuliproxError, - model::{PlaylistGroup, XtreamCluster}, + model::{PlaylistGroup, UpdateQualityPolicy, XtreamCluster}, }; use std::{future::Future, sync::Arc}; -use tuliprox_core::model::{AppConfig, Config, ConfigInput}; +use tuliprox_core::model::{ + AppConfig, ClusterForceUpdate, ClusterUpdateAcceptance, ClusterUpdateRejection, Config, ConfigInput, +}; /// What a provider produced for one input. /// @@ -32,6 +34,16 @@ use tuliprox_core::model::{AppConfig, Config, ConfigInput}; pub struct PlaylistFetch { pub groups: Vec, pub errors: Vec, + /// Clusters explicitly identified at a failing acquisition/publication boundary. + /// Unscoped errors must not be assigned to clusters by consumers. + pub failed_clusters: Vec, + /// Completed normal Quality evaluations whose candidates were accepted. + pub quality_acceptances: Vec, + /// Completed cluster updates that were rejected by the domain policy. + /// These are nonfatal and are intentionally separate from `errors`. + pub quality_rejections: Vec, + /// Clusters accepted through an explicit request-local quality bypass. + pub force_updates: Vec, /// The provider wrote the playlist to disk itself, so the caller must not. pub persisted: bool, /// The fetch stopped part-way and should be resumed rather than treated as a result. @@ -40,6 +52,14 @@ pub struct PlaylistFetch { } impl PlaylistFetch { + /// Keep the existing error and its known cluster together, without duplicate identities. + pub fn record_cluster_error(&mut self, cluster: XtreamCluster, error: TuliproxError) { + self.errors.push(error); + if !self.failed_clusters.contains(&cluster) { + self.failed_clusters.push(cluster); + } + } + /// A successful fetch of `groups`. #[must_use] pub fn groups(groups: Vec) -> Self { Self { groups, ..Self::default() } } @@ -59,6 +79,24 @@ impl PlaylistFetch { self } + #[must_use] + pub fn with_quality_acceptances(mut self, quality_acceptances: Vec) -> Self { + self.quality_acceptances = quality_acceptances; + self + } + + #[must_use] + pub fn with_quality_rejections(mut self, quality_rejections: Vec) -> Self { + self.quality_rejections = quality_rejections; + self + } + + #[must_use] + pub fn with_force_updates(mut self, force_updates: Vec) -> Self { + self.force_updates = force_updates; + self + } + #[must_use] pub fn persisted(mut self, persisted: bool) -> Self { self.persisted = persisted; @@ -71,8 +109,9 @@ impl PlaylistFetch { self } - /// Whether the fetch completed without errors. A fetch that produced no groups and no - /// errors counts as successful — an empty catalog is a legitimate answer. + /// Whether the fetch completed without technical errors. A fetch that produced no groups + /// and no errors counts as successful — an empty catalog is a legitimate answer. Quality + /// rejections are completed, nonfatal decisions and therefore do not make this false. #[must_use] pub fn is_ok(&self) -> bool { self.errors.is_empty() && !self.partial } @@ -106,6 +145,9 @@ pub struct PlaylistFetchRequest<'a> { /// Which Xtream clusters still need fetching. `None` means "whatever the input /// allows"; providers that have no clusters ignore it. pub xtream_clusters: Option<&'a [XtreamCluster]>, + /// Request-local quality behavior for playlist updates. Direct provider + /// callers do not construct this request and retain their own scope. + pub update_quality: UpdateQualityPolicy, } /// Fetch one input's playlist. @@ -124,8 +166,13 @@ impl PlaylistProvider for M3uProvider { fn name(&self) -> &'static str { "m3u" } async fn fetch(&self, request: &PlaylistFetchRequest<'_>) -> PlaylistFetch { - let (groups, errors) = - crate::m3u::download_m3u_playlist(request.app_config, request.client, request.config, request.input).await; + let (groups, errors) = crate::m3u::download_m3u_playlist_for_update( + request.app_config, + request.client, + request.config, + request.input, + ) + .await; PlaylistFetch::groups(groups).with_errors(errors) } } @@ -144,15 +191,15 @@ impl PlaylistProvider for XtreamProvider<'_, E> { fn name(&self) -> &'static str { "xtream" } async fn fetch(&self, request: &PlaylistFetchRequest<'_>) -> PlaylistFetch { - let (groups, errors, persisted) = crate::xtream::download_xtream_playlist( + crate::xtream::download_xtream_playlist( request.app_config, request.client, self.events, request.input, request.xtream_clusters, + request.update_quality, ) - .await; - PlaylistFetch::groups(groups).with_errors(errors).persisted(persisted) + .await } } @@ -195,7 +242,8 @@ impl PlaylistProvider for BatchContainerProvider { #[cfg(test)] mod tests { use super::PlaylistFetch; - use shared::error::TuliproxError; + use shared::{error::TuliproxError, model::XtreamCluster}; + use tuliprox_core::model::ClusterUpdateRejection; #[test] fn an_empty_catalog_is_a_success_not_a_failure() { @@ -246,4 +294,21 @@ mod tests { assert!(fetch.needs_operator(), "one blip must not hide a config problem"); assert!(!fetch.is_retryable()); } + + #[test] + fn a_quality_rejection_is_not_a_technical_fetch_failure() { + let rejection = ClusterUpdateRejection { + cluster: XtreamCluster::Video, + current_count: 12_543, + candidate_count: 217, + threshold: 90, + quality: 1, + }; + let fetch = PlaylistFetch::groups(Vec::new()).with_quality_rejections(vec![rejection]); + + assert!(fetch.is_ok()); + assert!(fetch.errors.is_empty()); + assert!(!fetch.partial); + assert_eq!(fetch.quality_rejections, vec![rejection]); + } } diff --git a/backend/iptv/src/xtream/mod.rs b/backend/iptv/src/xtream/mod.rs index 7e630f70a..2347f3c6e 100644 --- a/backend/iptv/src/xtream/mod.rs +++ b/backend/iptv/src/xtream/mod.rs @@ -1,32 +1,36 @@ +use crate::provider::PlaylistFetch; use chrono::{DateTime, Utc}; use log::{error, info, warn}; use shared::{ concat_string, error::TuliproxError, model::{ - EventMessage, EventSink, InputType, PlaylistEntry, PlaylistGroup, ProviderAccountEvent, ProviderAccountState, - ProxyUserStatus, SeriesStreamProperties, StreamProperties, VideoStreamProperties, XtreamCluster, - XtreamLoginInfo, XtreamPlaylistItem, XtreamSeriesInfo, XtreamVideoInfo, XtreamVideoInfoDoc, + EventMessage, EventSink, InputType, PlaylistEntry, PlaylistGroup, PlaylistItem, PlaylistItemType, + ProviderAccountEvent, ProviderAccountState, ProviderId, ProxyUserStatus, SeriesStreamProperties, + StreamProperties, UpdateQualityPolicy, VideoStreamProperties, XtreamCluster, XtreamLoginInfo, + XtreamPlaylistItem, XtreamSeriesInfo, XtreamVideoInfo, XtreamVideoInfoDoc, }, utils::{ extract_extension_from_url, get_i64_from_serde_value, get_string_from_serde_value, sanitize_sensitive_info, Internable, PROVIDER_SCHEME_PREFIX, }, }; -use std::{collections::HashMap, io::Error, str::FromStr, sync::Arc}; +use std::{collections::HashMap, future::Future, io::Error, str::FromStr, sync::Arc}; use tuliprox_core::{ model::{ - is_input_expired, xtream_mapping_option_from_target_options, AppConfig, ConfigInput, ConfigInputFlags, - ConfigTarget, InputSource, ProxyUserCredentials, XtreamTargetOutput, + evaluate_update_quality, is_input_expired, xtream_mapping_option_from_target_options, AppConfig, + ClusterForceUpdate, ConfigInput, ConfigInputFlags, ConfigTarget, InputSource, ProxyUserCredentials, + XtreamTargetOutput, }, utils::request, }; use tuliprox_parser::{xtream, xtream::parse_xtream_series_info}; use tuliprox_repository::{ - get_input_storage_path, get_target_id_mapping, get_target_storage_path, persist_input_vod_info, - persist_input_xtream_playlist_clusters_to_disk, persists_input_series_info, + count_input_xtream_cluster, get_input_storage_path, get_target_id_mapping, get_target_storage_path, + persist_input_vod_info, persist_input_xtream_playlist_clusters_to_disk, persists_input_series_info, rewrite_provider_series_info_episode_virtual_id, write_playlist_batch_item_upsert, write_playlist_item_update, - PlaylistStorageState, ProviderEpisodeKey, VirtualIdRecord, + PlaylistStorageState, ProviderEpisodeKey, VirtualIdRecord, XtreamClusterPublishBatchResult, + XtreamClusterQualityPolicy, XtreamClusterRefreshRequest, }; const THREE_DAYS_IN_SECS: i64 = 3 * 24 * 60 * 60; @@ -503,6 +507,142 @@ pub fn requested_clusters(requested: Option<&[XtreamCluster]>, skip_cluster: &[X .collect() } +fn apply_cluster_update_quality( + fetch: &mut PlaylistFetch, + cluster: XtreamCluster, + current_count: Option, + threshold: u8, + mut candidate_groups: Vec, +) { + let candidate_count = candidate_groups.iter().map(|group| group.channels.len()).sum(); + let decision = evaluate_update_quality(current_count, candidate_count, threshold); + if let Some(rejection) = decision.rejection(cluster) { + fetch.quality_rejections.push(rejection); + } else { + if let Some(acceptance) = decision.acceptance(cluster) { + fetch.quality_acceptances.push(acceptance); + } + fetch.groups.append(&mut candidate_groups); + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum XtreamDownloadScope { + PlaylistUpdate { quality: UpdateQualityPolicy }, + Direct, +} + +struct XtreamDownloadContext { + source_input_type: InputType, + scope: XtreamDownloadScope, +} + +fn disk_cluster_quality_policy( + scope: XtreamDownloadScope, + input: &ConfigInput, + cluster: XtreamCluster, +) -> XtreamClusterQualityPolicy { + let configured_threshold = input.options.as_ref().map_or(0, |options| options.update_quality.threshold(cluster)); + match scope { + XtreamDownloadScope::PlaylistUpdate { quality: UpdateQualityPolicy::Enforce } => { + XtreamClusterQualityPolicy::Enforce { threshold: configured_threshold } + } + XtreamDownloadScope::PlaylistUpdate { quality: UpdateQualityPolicy::Bypass } => { + XtreamClusterQualityPolicy::Bypass { configured_threshold } + } + XtreamDownloadScope::Direct => XtreamClusterQualityPolicy::Enforce { threshold: 0 }, + } +} + +fn apply_disk_cluster_publish_result(fetch: &mut PlaylistFetch, mut result: XtreamClusterPublishBatchResult) { + fetch.quality_acceptances.append(&mut result.quality_acceptances); + fetch.quality_rejections.append(&mut result.quality_rejections); + fetch.force_updates.append(&mut result.force_updates); + fetch.errors.append(&mut result.errors); + if let Some(cluster) = result.failed_cluster { + fetch.failed_clusters.push(cluster); + } +} + +struct InMemoryXtreamCandidateWinner { + group_index: usize, + item: PlaylistItem, +} + +fn input_btree_provider_id(item: &PlaylistItem) -> ProviderId { + let header = &item.header; + let missing_live_input_identity = header.input_stream_id.is_empty() + && (header.item_type.is_live() || header.item_type == PlaylistItemType::Catchup); + let provider_id = if missing_live_input_identity { None } else { item.get_provider_id() }; + + // The input BTree uses the converted item's provider id as its key. Keeping + // the same unset sentinel here is required for candidate/baseline parity. + ProviderId::new(provider_id.unwrap_or_default()) +} + +fn normalize_in_memory_xtream_candidate(mut candidate_groups: Vec) -> Vec { + let candidate_count = candidate_groups.iter().map(|group| group.channels.len()).sum(); + let mut winning_position_by_provider_id = HashMap::::with_capacity(candidate_count); + let mut candidates = Vec::>::with_capacity(candidate_count); + + for (group_index, group) in candidate_groups.iter_mut().enumerate() { + for item in std::mem::take(&mut group.channels) { + let provider_id = input_btree_provider_id(&item); + let candidate_position = candidates.len(); + if let Some(previous_position) = winning_position_by_provider_id.insert(provider_id, candidate_position) { + candidates[previous_position] = None; + } + candidates.push(Some(InMemoryXtreamCandidateWinner { group_index, item })); + } + } + + for winner in candidates.into_iter().flatten() { + candidate_groups[winner.group_index].channels.push(winner.item); + } + candidate_groups.retain(|group| !group.channels.is_empty()); + candidate_groups +} + +async fn apply_in_memory_cluster_download( + scope: XtreamDownloadScope, + input: &ConfigInput, + cluster: XtreamCluster, + mut candidate_groups: Vec, + fetch: &mut PlaylistFetch, + load_baseline_count: B, +) -> Result<(), TuliproxError> +where + B: FnOnce() -> Fut, + Fut: Future, TuliproxError>>, +{ + match scope { + XtreamDownloadScope::Direct => { + fetch.groups.append(&mut candidate_groups); + return Ok(()); + } + XtreamDownloadScope::PlaylistUpdate { quality: UpdateQualityPolicy::Bypass } => { + candidate_groups = normalize_in_memory_xtream_candidate(candidate_groups); + let candidate_count = candidate_groups.iter().map(|group| group.channels.len()).sum(); + let configured_threshold = + input.options.as_ref().map_or(0, |options| options.update_quality.threshold(cluster)); + fetch.groups.append(&mut candidate_groups); + fetch.force_updates.push(ClusterForceUpdate { cluster, candidate_count, configured_threshold }); + return Ok(()); + } + XtreamDownloadScope::PlaylistUpdate { quality: UpdateQualityPolicy::Enforce } => {} + } + + let threshold = input.options.as_ref().map_or(0, |options| options.update_quality.threshold(cluster)); + let current_count = if threshold == 0 { + None + } else { + candidate_groups = normalize_in_memory_xtream_candidate(candidate_groups); + load_baseline_count().await? + }; + apply_cluster_update_quality(fetch, cluster, current_count, threshold, candidate_groups); + Ok(()) +} + /// Downloads xtream clusters from a single source (either main input or staged input). async fn download_xtream_from_source( app_config: &Arc, @@ -510,9 +650,9 @@ async fn download_xtream_from_source( events: &E, input: &ConfigInput, input_source: &InputSource, - source_input_type: InputType, clusters: &[XtreamCluster], -) -> (Vec, Vec, bool) { + context: XtreamDownloadContext, +) -> PlaylistFetch { let (username, password) = (input_source.username.as_deref().unwrap_or(""), input_source.password.as_deref().unwrap_or("")); @@ -522,7 +662,7 @@ async fn download_xtream_from_source( } else { match input.resolve_url(&input_source.url) { Ok(url) => url.into_owned(), - Err(err) => return (Vec::new(), vec![err], false), + Err(err) => return PlaylistFetch { failed_clusters: clusters.to_vec(), ..PlaylistFetch::failed(err) }, } }; @@ -531,16 +671,14 @@ async fn download_xtream_from_source( if let Err(err) = xtream_login(app_config, client, events, &input_source_login, username).await { error!("Could not log in with xtream user {username} for provider {}. {err}", input.name); - return (Vec::new(), vec![err], false); + return PlaylistFetch { failed_clusters: clusters.to_vec(), ..PlaylistFetch::failed(err) }; } - let mut playlist_groups: Vec = Vec::with_capacity(128); - let cfg = app_config.config.load(); let storage_dir = &cfg.storage_dir; - let use_disk_based_processing = cfg.disk_based_processing && source_input_type.is_xtream(); + let use_disk_based_processing = cfg.disk_based_processing && context.source_input_type.is_xtream(); + let mut fetch = PlaylistFetch::groups(Vec::with_capacity(128)).persisted(use_disk_based_processing); - let mut errors = vec![]; let mut disk_cluster_readers = Vec::new(); for (xtream_cluster, category, stream) in &ACTIONS { if !clusters.contains(xtream_cluster) { @@ -565,12 +703,28 @@ async fn download_xtream_from_source( ) { (Ok(category_content), Ok(stream_content)) => { if use_disk_based_processing { - disk_cluster_readers.push((*xtream_cluster, category_content, stream_content)); + disk_cluster_readers.push(XtreamClusterRefreshRequest { + cluster: *xtream_cluster, + quality: disk_cluster_quality_policy(context.scope, input, *xtream_cluster), + categories: category_content, + streams: stream_content, + }); } else { match xtream::parse_xtream(input, *xtream_cluster, category_content, stream_content).await { Ok(sub_playlist_parsed) => { - if let Some(mut xtream_sub_playlist) = sub_playlist_parsed { - playlist_groups.append(&mut xtream_sub_playlist); + if let Some(xtream_sub_playlist) = sub_playlist_parsed { + if let Err(err) = apply_in_memory_cluster_download( + context.scope, + input, + *xtream_cluster, + xtream_sub_playlist, + &mut fetch, + || count_input_xtream_cluster(app_config, input, *xtream_cluster), + ) + .await + { + fetch.record_cluster_error(*xtream_cluster, err); + } } else { error!( "Could not parse playlist {xtream_cluster} for input {}: {}", @@ -579,58 +733,100 @@ async fn download_xtream_from_source( ); } } - Err(err) => errors.push(err), + Err(err) => fetch.record_cluster_error(*xtream_cluster, err), } } } (Err(err1), Err(err2)) => { - errors.extend([err1, err2]); + fetch.record_cluster_error(*xtream_cluster, err1); + fetch.record_cluster_error(*xtream_cluster, err2); } - (_, Err(err)) | (Err(err), _) => errors.push(err), + (_, Err(err)) | (Err(err), _) => fetch.record_cluster_error(*xtream_cluster, err), } } - if use_disk_based_processing && errors.is_empty() && !disk_cluster_readers.is_empty() { - if let Err(err) = persist_input_xtream_playlist_clusters_to_disk(app_config, input, disk_cluster_readers).await - { + if use_disk_based_processing && !disk_cluster_readers.is_empty() { + let result = persist_input_xtream_playlist_clusters_to_disk(app_config, input, disk_cluster_readers).await; + for err in &result.errors { error!("persist_input_xtream_playlist_clusters_to_disk failed: {err}"); - errors.push(err); } + apply_disk_cluster_publish_result(&mut fetch, result); } - (playlist_groups, errors, use_disk_based_processing) + fetch } +async fn download_xtream_playlist_with_scope( + app_config: &Arc, + client: &reqwest::Client, + events: &E, + input: &ConfigInput, + clusters: Option<&[XtreamCluster]>, + scope: XtreamDownloadScope, +) -> PlaylistFetch { + let skip_cluster = get_skip_cluster(input); + let main_clusters = requested_clusters(clusters, &skip_cluster); + + let mut fetch = PlaylistFetch::groups(Vec::with_capacity(128)); + + if !main_clusters.is_empty() { + check_alias_user_state(events, input); + let source: InputSource = input.into(); + let mut source_fetch = download_xtream_from_source( + app_config, + client, + events, + input, + &source, + &main_clusters, + XtreamDownloadContext { source_input_type: input.input_type, scope }, + ) + .await; + fetch.groups.append(&mut source_fetch.groups); + fetch.errors.append(&mut source_fetch.errors); + fetch.failed_clusters.append(&mut source_fetch.failed_clusters); + fetch.quality_acceptances.append(&mut source_fetch.quality_acceptances); + fetch.quality_rejections.append(&mut source_fetch.quality_rejections); + fetch.force_updates.append(&mut source_fetch.force_updates); + fetch.persisted |= source_fetch.persisted; + } + + for (grp_id, plg) in (1_u32..).zip(fetch.groups.iter_mut()) { + plg.id = grp_id; + } + + fetch +} + +/// Downloads an Xtream candidate for the playlist-update flow and applies the configured quality policy. pub async fn download_xtream_playlist( app_config: &Arc, client: &reqwest::Client, events: &E, input: &ConfigInput, clusters: Option<&[XtreamCluster]>, -) -> (Vec, Vec, bool) { - let skip_cluster = get_skip_cluster(input); - let main_clusters = requested_clusters(clusters, &skip_cluster); + quality: UpdateQualityPolicy, +) -> PlaylistFetch { + download_xtream_playlist_with_scope( + app_config, + client, + events, + input, + clusters, + XtreamDownloadScope::PlaylistUpdate { quality }, + ) + .await +} - let mut all_groups = Vec::with_capacity(128); - let mut all_errors = Vec::new(); - let mut any_disk = false; - - if !main_clusters.is_empty() { - check_alias_user_state(events, input); - let source: InputSource = input.into(); - let (g, e, d) = - download_xtream_from_source(app_config, client, events, input, &source, input.input_type, &main_clusters) - .await; - all_groups.extend(g); - all_errors.extend(e); - any_disk |= d; - } - - for (grp_id, plg) in (1_u32..).zip(all_groups.iter_mut()) { - plg.id = grp_id; - } - - (all_groups, all_errors, any_disk) +/// Downloads Xtream data for a direct caller without consulting the persisted update baseline. +pub async fn download_xtream_playlist_direct( + app_config: &Arc, + client: &reqwest::Client, + events: &E, + input: &ConfigInput, + clusters: Option<&[XtreamCluster]>, +) -> PlaylistFetch { + download_xtream_playlist_with_scope(app_config, client, events, input, clusters, XtreamDownloadScope::Direct).await } fn check_alias_user_state(events: &E, input: &ConfigInput) { @@ -714,14 +910,31 @@ pub fn create_vod_info_from_item(pli: &XtreamPlaylistItem) -> String { #[cfg(test)] mod tests { - use super::{create_vod_info_from_item, requested_clusters}; + use super::{ + apply_cluster_update_quality, apply_disk_cluster_publish_result, apply_in_memory_cluster_download, + create_vod_info_from_item, disk_cluster_quality_policy, input_btree_provider_id, requested_clusters, + XtreamDownloadScope, + }; + use crate::provider::PlaylistFetch; use serde_json::Value; use shared::{ - model::{InputType, PlaylistItemType, ProxyType, XtreamCluster, XtreamPlaylistItem}, + error::TuliproxError, + model::{ + ConfigInputOptionsDto, ConfigInputUpdateQualityDto, InputType, PlaylistGroup, PlaylistItem, + PlaylistItemHeader, PlaylistItemType, ProxyType, UpdateQualityPolicy, XtreamCluster, XtreamPlaylistItem, + }, utils::Internable, }; + use std::sync::{ + atomic::{AtomicUsize, Ordering}, + Arc, + }; use tuliprox_core::model::{ - ConfigInput, ConfigInputFlags, ConfigInputFlagsSet, ConfigInputOptions, ProxyUserCredentials, + ClusterForceUpdate, ClusterUpdateAcceptance, ClusterUpdateRejection, ConfigInput, ConfigInputFlags, + ConfigInputFlagsSet, ConfigInputOptions, ProxyUserCredentials, + }; + use tuliprox_repository::{ + XtreamClusterPublishBatchResult, XtreamClusterPublishOutcome, XtreamClusterQualityPolicy, }; /// Records what reached the bus, so the expiry branches can be asserted rather than @@ -811,6 +1024,456 @@ mod tests { assert_eq!(clusters, vec![XtreamCluster::Live, XtreamCluster::Video]); } + fn candidate_group(cluster: XtreamCluster, category_id: u32, count: usize) -> PlaylistGroup { + let title = format!("{cluster}-{category_id}").intern(); + let channels = (0..count) + .map(|index| { + let stream_id = format!("{category_id}{index:03}").intern(); + PlaylistItem { + header: PlaylistItemHeader { + id: Arc::clone(&stream_id), + input_stream_id: stream_id, + name: format!("stream-{index}").intern(), + title: format!("stream-{index}").intern(), + group: Arc::clone(&title), + url: format!("http://provider.example/{cluster}/{index}").intern(), + item_type: PlaylistItemType::from(cluster), + xtream_cluster: cluster, + category_id, + input_name: "provider".intern(), + ..PlaylistItemHeader::default() + }, + } + }) + .collect(); + PlaylistGroup { id: category_id, title, channels, xtream_cluster: cluster } + } + + fn set_candidate_provider_id(item: &mut PlaylistItem, provider_id: u32) { + let provider_id = provider_id.to_string().intern(); + item.header.id = Arc::clone(&provider_id); + item.header.input_stream_id = provider_id; + } + + #[test] + fn in_memory_candidate_key_matches_persisted_input_btree_key() { + let resolved = candidate_group(XtreamCluster::Video, 1, 1).channels.remove(0); + + let mut unresolved = candidate_group(XtreamCluster::Video, 2, 1).channels.remove(0); + unresolved.header.id = "not-an-id".intern(); + unresolved.header.url = "http://provider.example/movie/not-an-id".intern(); + + let mut missing_live_input_identity = candidate_group(XtreamCluster::Live, 3, 1).channels.remove(0); + missing_live_input_identity.header.input_stream_id = "".intern(); + + for (name, item) in [ + ("resolved", resolved), + ("unresolved", unresolved), + ("missing live input identity", missing_live_input_identity), + ] { + assert_eq!( + input_btree_provider_id(&item).get(), + XtreamPlaylistItem::from(&item).provider_id, + "case: {name}" + ); + } + } + + #[tokio::test] + async fn in_memory_quality_direct_download_skips_baseline_while_update_rejects_same_candidate() { + let options = ConfigInputOptions::from(&ConfigInputOptionsDto { + update_quality: ConfigInputUpdateQualityDto { live: 90, ..ConfigInputUpdateQualityDto::default() }, + ..ConfigInputOptionsDto::default() + }); + let input = ConfigInput { + name: "provider".intern(), + input_type: InputType::Xtream, + options: Some(options), + ..ConfigInput::default() + }; + let candidate = vec![candidate_group(XtreamCluster::Live, 1, 89)]; + let baseline_reads = AtomicUsize::new(0); + + let mut direct_fetch = PlaylistFetch::default(); + apply_in_memory_cluster_download( + XtreamDownloadScope::Direct, + &input, + XtreamCluster::Live, + candidate.clone(), + &mut direct_fetch, + || async { + baseline_reads.fetch_add(1, Ordering::SeqCst); + Ok(Some(100)) + }, + ) + .await + .expect("direct candidate should be accepted"); + + assert_eq!(baseline_reads.load(Ordering::SeqCst), 0); + assert_eq!(direct_fetch.groups.len(), 1); + assert_eq!(direct_fetch.groups[0].channels.len(), 89); + assert!(direct_fetch.quality_rejections.is_empty()); + + let mut update_fetch = PlaylistFetch::default(); + apply_in_memory_cluster_download( + XtreamDownloadScope::PlaylistUpdate { quality: UpdateQualityPolicy::Enforce }, + &input, + XtreamCluster::Live, + candidate, + &mut update_fetch, + || async { + baseline_reads.fetch_add(1, Ordering::SeqCst); + Ok(Some(100)) + }, + ) + .await + .expect("update candidate should produce a domain decision"); + + assert_eq!(baseline_reads.load(Ordering::SeqCst), 1); + assert!(update_fetch.groups.is_empty()); + assert_eq!( + update_fetch.quality_rejections, + vec![tuliprox_core::model::ClusterUpdateRejection { + cluster: XtreamCluster::Live, + current_count: 100, + candidate_count: 89, + threshold: 90, + quality: 89, + }] + ); + } + + #[tokio::test] + async fn pipeline_transparency_in_memory_quality_preserves_accepted_evaluation() { + let options = ConfigInputOptions::from(&ConfigInputOptionsDto { + update_quality: ConfigInputUpdateQualityDto { live: 100, ..ConfigInputUpdateQualityDto::default() }, + ..ConfigInputOptionsDto::default() + }); + let input = ConfigInput { + name: "provider".intern(), + input_type: InputType::Xtream, + options: Some(options), + ..ConfigInput::default() + }; + let mut first_group = candidate_group(XtreamCluster::Live, 1, 1); + set_candidate_provider_id(&mut first_group.channels[0], 7); + let mut winning_group = candidate_group(XtreamCluster::Live, 2, 2); + set_candidate_provider_id(&mut winning_group.channels[0], 7); + set_candidate_provider_id(&mut winning_group.channels[1], 8); + winning_group.channels[0].header.name = "winning-duplicate".intern(); + winning_group.channels[0].header.title = "winning-duplicate".intern(); + let baseline_reads = AtomicUsize::new(0); + let mut fetch = PlaylistFetch::default(); + + apply_in_memory_cluster_download( + XtreamDownloadScope::PlaylistUpdate { quality: UpdateQualityPolicy::Enforce }, + &input, + XtreamCluster::Live, + vec![first_group, winning_group], + &mut fetch, + || async { + baseline_reads.fetch_add(1, Ordering::SeqCst); + Ok(Some(2)) + }, + ) + .await + .expect("deduplicated candidate should be accepted"); + + assert_eq!(baseline_reads.load(Ordering::SeqCst), 1); + assert!(fetch.quality_rejections.is_empty()); + assert_eq!( + fetch.quality_acceptances, + vec![ClusterUpdateAcceptance { + cluster: XtreamCluster::Live, + current_count: Some(2), + candidate_count: 2, + threshold: 100, + quality: Some(100), + }] + ); + assert_eq!(fetch.groups.len(), 1, "the emptied first category must be removed"); + assert_eq!(fetch.groups[0].id, 2); + assert_eq!(fetch.groups[0].channels.len(), 2); + let winner = + fetch.groups[0].channels.iter().find(|item| item.header.id.as_ref() == "7").expect("winning duplicate"); + assert_eq!(winner.header.name.as_ref(), "winning-duplicate"); + assert_eq!(winner.header.category_id, 2); + } + + #[tokio::test] + async fn in_memory_quality_direct_and_disabled_paths_preserve_duplicate_rows_without_baseline_io() { + let guarded_options = ConfigInputOptions::from(&ConfigInputOptionsDto { + update_quality: ConfigInputUpdateQualityDto { live: 100, ..ConfigInputUpdateQualityDto::default() }, + ..ConfigInputOptionsDto::default() + }); + let guarded_input = ConfigInput { + name: "provider".intern(), + input_type: InputType::Xtream, + options: Some(guarded_options), + ..ConfigInput::default() + }; + let disabled_input = ConfigInput { + name: "provider".intern(), + input_type: InputType::Xtream, + options: Some(ConfigInputOptions::from(&ConfigInputOptionsDto::default())), + ..ConfigInput::default() + }; + let mut first_group = candidate_group(XtreamCluster::Live, 1, 1); + set_candidate_provider_id(&mut first_group.channels[0], 7); + let mut second_group = candidate_group(XtreamCluster::Live, 2, 1); + set_candidate_provider_id(&mut second_group.channels[0], 7); + let candidate = vec![first_group, second_group]; + let baseline_reads = AtomicUsize::new(0); + + let mut direct_fetch = PlaylistFetch::default(); + apply_in_memory_cluster_download( + XtreamDownloadScope::Direct, + &guarded_input, + XtreamCluster::Live, + candidate.clone(), + &mut direct_fetch, + || async { + baseline_reads.fetch_add(1, Ordering::SeqCst); + Ok(Some(2)) + }, + ) + .await + .expect("direct candidate should remain unguarded"); + + let mut disabled_fetch = PlaylistFetch::default(); + apply_in_memory_cluster_download( + XtreamDownloadScope::PlaylistUpdate { quality: UpdateQualityPolicy::Enforce }, + &disabled_input, + XtreamCluster::Live, + candidate, + &mut disabled_fetch, + || async { + baseline_reads.fetch_add(1, Ordering::SeqCst); + Ok(Some(2)) + }, + ) + .await + .expect("disabled quality guard should preserve the candidate"); + + assert_eq!(baseline_reads.load(Ordering::SeqCst), 0); + assert_eq!(direct_fetch.groups.iter().map(|group| group.channels.len()).sum::(), 2); + assert_eq!(disabled_fetch.groups.iter().map(|group| group.channels.len()).sum::(), 2); + assert!(direct_fetch.quality_rejections.is_empty()); + assert!(disabled_fetch.quality_rejections.is_empty()); + } + + #[tokio::test] + async fn in_memory_force_normalizes_and_accepts_without_baseline_io() { + let options = ConfigInputOptions::from(&ConfigInputOptionsDto { + update_quality: ConfigInputUpdateQualityDto { live: 100, ..ConfigInputUpdateQualityDto::default() }, + ..ConfigInputOptionsDto::default() + }); + let input = ConfigInput { + name: "provider".intern(), + input_type: InputType::Xtream, + options: Some(options), + ..ConfigInput::default() + }; + let mut first_group = candidate_group(XtreamCluster::Live, 1, 1); + set_candidate_provider_id(&mut first_group.channels[0], 7); + let mut winning_group = candidate_group(XtreamCluster::Live, 2, 2); + set_candidate_provider_id(&mut winning_group.channels[0], 7); + set_candidate_provider_id(&mut winning_group.channels[1], 8); + winning_group.channels[0].header.name = "winning-duplicate".intern(); + let baseline_reads = AtomicUsize::new(0); + let mut fetch = PlaylistFetch::default(); + + apply_in_memory_cluster_download( + XtreamDownloadScope::PlaylistUpdate { quality: UpdateQualityPolicy::Bypass }, + &input, + XtreamCluster::Live, + vec![first_group, winning_group], + &mut fetch, + || async { + baseline_reads.fetch_add(1, Ordering::SeqCst); + Ok(Some(100)) + }, + ) + .await + .expect("forced candidate should be accepted"); + + assert_eq!(baseline_reads.load(Ordering::SeqCst), 0); + assert!(fetch.quality_rejections.is_empty()); + assert_eq!(fetch.groups.len(), 1); + assert_eq!(fetch.groups[0].channels.len(), 2); + assert_eq!(fetch.groups[0].channels[0].header.name.as_ref(), "winning-duplicate"); + assert_eq!( + fetch.force_updates, + vec![ClusterForceUpdate { cluster: XtreamCluster::Live, candidate_count: 2, configured_threshold: 100 }] + ); + } + + #[test] + fn in_memory_quality_applies_exact_90_percent_boundaries() { + for (name, candidate_count, rejected) in [ + ("lower boundary", 90, false), + ("below lower boundary", 89, true), + ("upper boundary", 110, false), + ("above upper boundary", 111, true), + ] { + let mut fetch = PlaylistFetch::default(); + apply_cluster_update_quality( + &mut fetch, + XtreamCluster::Live, + Some(100), + 90, + vec![candidate_group(XtreamCluster::Live, 1, candidate_count)], + ); + + assert_eq!(!fetch.quality_rejections.is_empty(), rejected, "case: {name}"); + assert_eq!(fetch.groups.is_empty(), rejected, "case: {name}"); + if rejected { + let rejection = fetch.quality_rejections.first().expect("rejection report"); + assert_eq!(rejection.current_count, 100, "case: {name}"); + assert_eq!(rejection.candidate_count, candidate_count, "case: {name}"); + assert_eq!(rejection.threshold, 90, "case: {name}"); + assert_eq!(rejection.quality, 89, "case: {name}"); + } + } + } + + #[test] + fn in_memory_quality_preserves_disabled_and_bootstrap_behavior() { + let mut disabled = PlaylistFetch::default(); + apply_cluster_update_quality( + &mut disabled, + XtreamCluster::Live, + Some(100), + 0, + vec![candidate_group(XtreamCluster::Live, 1, 1)], + ); + assert_eq!(disabled.groups[0].channels.len(), 1); + assert!(disabled.quality_rejections.is_empty()); + + let mut bootstrap = PlaylistFetch::default(); + apply_cluster_update_quality( + &mut bootstrap, + XtreamCluster::Video, + None, + 90, + vec![candidate_group(XtreamCluster::Video, 2, 3)], + ); + assert_eq!(bootstrap.groups[0].channels.len(), 3); + assert!(bootstrap.quality_rejections.is_empty()); + + let mut empty_bootstrap = PlaylistFetch::default(); + apply_cluster_update_quality(&mut empty_bootstrap, XtreamCluster::Series, None, 90, Vec::new()); + assert!(empty_bootstrap.groups.is_empty()); + assert_eq!( + empty_bootstrap.quality_rejections, + vec![tuliprox_core::model::ClusterUpdateRejection { + cluster: XtreamCluster::Series, + current_count: 0, + candidate_count: 0, + threshold: 90, + quality: 0, + }] + ); + } + + #[test] + fn rejected_vod_does_not_block_accepted_live_and_series_clusters() { + let mut fetch = PlaylistFetch::default(); + apply_cluster_update_quality( + &mut fetch, + XtreamCluster::Live, + Some(100), + 90, + vec![candidate_group(XtreamCluster::Live, 1, 90)], + ); + apply_cluster_update_quality( + &mut fetch, + XtreamCluster::Video, + Some(100), + 90, + vec![candidate_group(XtreamCluster::Video, 2, 40), candidate_group(XtreamCluster::Video, 3, 49)], + ); + apply_cluster_update_quality( + &mut fetch, + XtreamCluster::Series, + Some(100), + 90, + vec![candidate_group(XtreamCluster::Series, 4, 110)], + ); + + assert_eq!(fetch.groups.len(), 2); + assert!(fetch.groups.iter().any(|group| group.xtream_cluster == XtreamCluster::Live)); + assert!(fetch.groups.iter().any(|group| group.xtream_cluster == XtreamCluster::Series)); + assert!(!fetch.groups.iter().any(|group| group.xtream_cluster == XtreamCluster::Video)); + assert_eq!(fetch.quality_rejections.len(), 1); + assert_eq!(fetch.quality_rejections[0].cluster, XtreamCluster::Video); + assert_eq!(fetch.quality_rejections[0].candidate_count, 89); + } + + #[test] + fn pipeline_transparency_disk_batch_transports_quality_and_error_facts_together() { + let acceptance = ClusterUpdateAcceptance { + cluster: XtreamCluster::Live, + current_count: Some(100), + candidate_count: 90, + threshold: 90, + quality: Some(90), + }; + let rejection = ClusterUpdateRejection { + cluster: XtreamCluster::Video, + current_count: 100, + candidate_count: 89, + threshold: 90, + quality: 89, + }; + let force_update = + ClusterForceUpdate { cluster: XtreamCluster::Series, candidate_count: 80, configured_threshold: 95 }; + let technical_error = TuliproxError::RepositoryXtream("injected post-quality failure".to_string()); + let mut fetch = PlaylistFetch::default().persisted(true); + + apply_disk_cluster_publish_result( + &mut fetch, + XtreamClusterPublishBatchResult { + outcomes: vec![XtreamClusterPublishOutcome::QualityAccepted(acceptance)], + quality_acceptances: vec![acceptance], + quality_rejections: vec![rejection], + force_updates: vec![force_update], + errors: vec![technical_error], + failed_cluster: Some(XtreamCluster::Series), + }, + ); + + assert!(fetch.persisted); + assert!(fetch.groups.is_empty()); + assert_eq!(fetch.quality_acceptances, vec![acceptance]); + assert_eq!(fetch.quality_rejections, vec![rejection]); + assert_eq!(fetch.force_updates, vec![force_update]); + assert_eq!(fetch.errors.len(), 1); + assert_eq!(fetch.failed_clusters, [XtreamCluster::Series]); + } + + #[test] + fn disk_quality_scope_disables_guard_for_direct_downloads() { + let options = ConfigInputOptions::from(&ConfigInputOptionsDto { + update_quality: ConfigInputUpdateQualityDto { live: 90, ..ConfigInputUpdateQualityDto::default() }, + ..ConfigInputOptionsDto::default() + }); + let input = ConfigInput { options: Some(options), ..ConfigInput::default() }; + + assert_eq!( + disk_cluster_quality_policy( + XtreamDownloadScope::PlaylistUpdate { quality: UpdateQualityPolicy::Enforce }, + &input, + XtreamCluster::Live, + ), + XtreamClusterQualityPolicy::Enforce { threshold: 90 } + ); + assert_eq!( + disk_cluster_quality_policy(XtreamDownloadScope::Direct, &input, XtreamCluster::Live), + XtreamClusterQualityPolicy::Enforce { threshold: 0 } + ); + } + fn test_vod_item() -> XtreamPlaylistItem { XtreamPlaylistItem { virtual_id: shared::model::VirtualId::new(176_141), diff --git a/backend/library/src/library/metadata_iterator.rs b/backend/library/src/library/metadata_iterator.rs index d65eba78f..78507614e 100644 --- a/backend/library/src/library/metadata_iterator.rs +++ b/backend/library/src/library/metadata_iterator.rs @@ -1,6 +1,9 @@ use crate::library::MetadataCacheEntry; use log::error; -use std::path::{Path, PathBuf}; +use std::{ + io, + path::{Path, PathBuf}, +}; pub struct MetadataAsyncIter { paths: Vec, @@ -21,6 +24,19 @@ async fn collect_paths(storage_dir: &Path) -> Vec { } impl MetadataAsyncIter { + /// Opens the canonical catalog without treating an unreadable directory as empty. + pub async fn try_new(storage_dir: &Path) -> io::Result { + let mut paths = Vec::new(); + let mut read_dir = tokio::fs::read_dir(storage_dir).await?; + while let Some(entry) = read_dir.next_entry().await? { + let path = entry.path(); + if path.extension().and_then(|s| s.to_str()) == Some("json") { + paths.push(path); + } + } + Ok(Self { paths, index: 0 }) + } + pub async fn new(storage_dir: &Path) -> Self { let paths = collect_paths(storage_dir).await; Self { paths, index: 0 } @@ -28,21 +44,23 @@ impl MetadataAsyncIter { pub async fn next(&mut self) -> Option { while self.index < self.paths.len() { - let path = &self.paths[self.index]; - self.index += 1; - - match tokio::fs::read_to_string(path).await { - Ok(content) => match serde_json::from_str::(&content) { - Ok(entry) => return Some(entry), - Err(e) => { - error!("Failed to parse library metadata {}: {}", path.display(), e); - } - }, - Err(e) => { - error!("Failed to read library metadata {}: {}", path.display(), e); - } + match self.try_next().await { + Ok(entry) => return entry, + Err(e) => error!("Failed to read library metadata: {e}"), } } None } + + /// Reads a stored entry without hiding corruption or read failures from publishers. + pub async fn try_next(&mut self) -> io::Result> { + let Some(path) = self.paths.get(self.index) else { return Ok(None) }; + self.index += 1; + let content = tokio::fs::read_to_string(path) + .await + .map_err(|err| io::Error::new(err.kind(), format!("{}: {err}", path.display())))?; + serde_json::from_str(&content) + .map(Some) + .map_err(|err| io::Error::new(io::ErrorKind::InvalidData, format!("{}: {err}", path.display()))) + } } diff --git a/backend/library/src/library/metadata_storage.rs b/backend/library/src/library/metadata_storage.rs index 77b8e0dd7..ee8fcaf6a 100644 --- a/backend/library/src/library/metadata_storage.rs +++ b/backend/library/src/library/metadata_storage.rs @@ -23,6 +23,16 @@ pub struct MetadataStorage { } impl MetadataStorage { + /// Loads a complete canonical catalog, preserving read errors for authoritative consumers. + pub async fn load_all_complete(&self) -> std::io::Result> { + let mut iter = super::MetadataAsyncIter::try_new(&self.storage_dir.join(LIBRARY_PATH)).await?; + let mut entries = Vec::new(); + while let Some(entry) = iter.try_next().await? { + entries.push(entry); + } + Ok(entries) + } + // Creates a new metadata storage instance pub fn new(storage_dir: PathBuf) -> Self { Self { storage_dir, mutation_guard: Arc::new(Mutex::new(())) } } diff --git a/backend/library/src/library/processor.rs b/backend/library/src/library/processor.rs index 9dde40775..6d56f36df 100644 --- a/backend/library/src/library/processor.rs +++ b/backend/library/src/library/processor.rs @@ -8,7 +8,7 @@ use crate::library::{ }; use log::{debug, error, info, warn}; use path_clean::PathClean; -use shared::model::{LibraryMetadataFormat, LibraryScanResult}; +use shared::model::{LibraryMetadataFormat, LibraryScanResult, LibraryStatus}; use std::{collections::HashMap, fmt, future::Future, io, path::PathBuf, pin::Pin, sync::Arc}; use tuliprox_core::{ model::{LibraryConfig, MetadataUpdateConfig}, @@ -150,19 +150,35 @@ impl LibraryProcessor { } } - // Performs a full Library scan + // Performs a full Library scan with the existing standalone best-effort behavior. pub async fn scan(&self, force_rescan: bool) -> Result { + self.scan_with_completeness(force_rescan, super::scanner::ScanCompleteness::BestEffort).await + } + + /// Uses the same scanner and writer, but refuses incomplete discovery before a target rebuild. + pub async fn scan_for_target_rebuild(&self) -> Result { + self.scan_with_completeness(false, super::scanner::ScanCompleteness::Complete).await + } + + async fn scan_with_completeness( + &self, + force_rescan: bool, + completeness: super::scanner::ScanCompleteness, + ) -> Result { info!("Starting Library scan (force_rescan: {force_rescan})"); // Initialize storage self.storage.initialize().await?; // Load existing metadata cache - let existing_entries = self.storage.load_all().await; + let existing_entries = match completeness { + super::scanner::ScanCompleteness::Complete => self.storage.load_all_complete().await?, + super::scanner::ScanCompleteness::BestEffort => self.storage.load_all().await, + }; let existing_map: HashMap<_, _> = existing_entries.iter().map(|e| (e.file_path.clone(), e.clone())).collect(); // Scan for video files - let scanned_files = self.scanner.scan_all().await?; + let scanned_files = self.scanner.scan_all_with_completeness(completeness).await?; let scanned_files_count = scanned_files.len(); info!("Scanned {scanned_files_count} video files"); let media_groups = MediaGrouper::group(scanned_files); @@ -223,6 +239,9 @@ impl LibraryProcessor { debug!("Removing orphaned entry for: {}", entry.file_path); if let Err(e) = self.storage.delete_by_uuid(&entry.uuid).await { error!("Failed to delete orphaned entry: {e}"); + if completeness == super::scanner::ScanCompleteness::Complete { + result.errors += 1; + } } else { result.files_removed += 1; } @@ -672,6 +691,24 @@ impl LibraryProcessor { // Gets all cached metadata entries pub async fn get_all_entries(&self) -> Vec { self.storage.load_all().await } + /// Current catalog counts, independent of any scan's file/group counters. + pub async fn catalog_status(&self) -> io::Result { + self.storage.initialize().await?; + let entries = self.storage.load_all_complete().await?; + let mut status = + LibraryStatus { enabled: self.config.enabled, total_items: entries.len(), ..LibraryStatus::default() }; + for entry in entries { + match entry.metadata { + MediaMetadata::Movie(_) => status.movies += 1, + MediaMetadata::Series(series) => { + status.series += 1; + status.episodes += series.episodes.as_ref().map_or(0, Vec::len); + } + } + } + Ok(status) + } + async fn is_local_ffprobe_enabled(&self) -> bool { if let Some(probes) = &self.tool_probes { return (probes.ffprobe_enabled)().await; @@ -769,6 +806,76 @@ mod tests { use shared::model::{FfprobeConfigDto, LibraryConfigDto}; use tuliprox_core::model::MetadataUpdateConfig; + #[tokio::test] + async fn library_catalog_status_counts_canonical_episodes_and_preserves_total_items_on_reload() { + let temp = tempfile::tempdir().unwrap(); + let config = LibraryConfig::from(&LibraryConfigDto { enabled: true, ..LibraryConfigDto::default() }); + let processor = + LibraryProcessor::new(config.clone(), None, reqwest::Client::new(), temp.path().to_str().unwrap()); + let catalog_path = resolve_metadata_storage_path(None, temp.path().to_str().unwrap()).join("library"); + assert!(!catalog_path.exists(), "status must also work before the first scan"); + let empty = processor.catalog_status().await.unwrap(); + assert_eq!((empty.movies, empty.series, empty.episodes, empty.total_items), (0, 0, 0, 0)); + assert!(catalog_path.is_dir()); + for metadata in [ + MediaMetadata::Movie(super::super::MovieMetadata::default()), + MediaMetadata::Series(SeriesMetadata { + // Deliberately not the catalog's actual episode count. + number_of_episodes: 999, + episodes: Some(vec![EpisodeMetadata::default(); 3]), + ..SeriesMetadata::default() + }), + MediaMetadata::Series(SeriesMetadata::default()), + ] { + // No media files exist; the status is derived exclusively from stored metadata. + processor + .storage + .store(&MetadataCacheEntry::new("/not-a-real-media-file".into(), 0, 0, metadata)) + .await + .unwrap(); + } + let reloaded = LibraryProcessor::new(config, None, reqwest::Client::new(), temp.path().to_str().unwrap()); + let status = reloaded.catalog_status().await.unwrap(); + assert_eq!((status.movies, status.series, status.episodes, status.total_items), (1, 2, 3, 3)); + assert_eq!(status, processor.catalog_status().await.unwrap()); + } + + #[tokio::test] + async fn library_catalog_status_and_complete_scan_reject_corrupt_metadata() { + let temp = tempfile::tempdir().unwrap(); + let processor = LibraryProcessor::new( + LibraryConfig::from(&LibraryConfigDto::default()), + None, + reqwest::Client::new(), + temp.path().to_str().unwrap(), + ); + processor.storage.initialize().await.unwrap(); + let path = resolve_metadata_storage_path(None, temp.path().to_str().unwrap()).join("library/broken.json"); + tokio::fs::write(path, "not json").await.unwrap(); + assert_eq!(processor.catalog_status().await.unwrap_err().kind(), io::ErrorKind::InvalidData); + assert_eq!(processor.scan_for_target_rebuild().await.unwrap_err().kind(), io::ErrorKind::InvalidData); + } + + #[tokio::test] + async fn library_catalog_status_reports_storage_initialization_failure() { + let temp = tempfile::tempdir().unwrap(); + let metadata_path = temp.path().join("not-a-directory"); + tokio::fs::write(&metadata_path, "occupied").await.unwrap(); + let metadata = MetadataUpdateConfig { + cache_path: metadata_path.to_string_lossy().into_owned(), + ..MetadataUpdateConfig::default() + }; + let processor = LibraryProcessor::new( + LibraryConfig::from(&LibraryConfigDto::default()), + Some(&metadata), + reqwest::Client::new(), + temp.path().to_str().unwrap(), + ); + + assert!(processor.catalog_status().await.is_err(), "unreadable storage is not a zero-count catalog"); + assert_eq!(tokio::fs::read(metadata_path).await.unwrap(), b"occupied"); + } + #[test] fn test_scan_result_creation() { let result = LibraryScanResult { diff --git a/backend/library/src/library/scanner.rs b/backend/library/src/library/scanner.rs index c0aa0f315..c4715a608 100644 --- a/backend/library/src/library/scanner.rs +++ b/backend/library/src/library/scanner.rs @@ -181,10 +181,24 @@ pub struct LibraryScanner { config: LibraryConfig, } +/// Completeness required by a caller that will publish targets from this discovery. +#[derive(Clone, Copy, PartialEq, Eq)] +pub(super) enum ScanCompleteness { + BestEffort, + Complete, +} + impl LibraryScanner { pub fn new(config: LibraryConfig) -> Self { Self { config } } pub async fn scan_all(&self) -> Result, io::Error> { + self.scan_all_with_completeness(ScanCompleteness::BestEffort).await + } + + pub(super) async fn scan_all_with_completeness( + &self, + completeness: ScanCompleteness, + ) -> Result, io::Error> { if !self.config.enabled { info!("Library media scanning is disabled"); return Ok(Vec::new()); @@ -192,6 +206,10 @@ impl LibraryScanner { let mut all_files = Vec::new(); + if completeness == ScanCompleteness::Complete && !self.config.scan_directories.iter().any(|dir| dir.enabled) { + return Err(io::Error::other("Library rescan requires an enabled scan directory")); + } + for scan_dir in &self.config.scan_directories { if !scan_dir.enabled { debug!("Skipping disabled scan directory: {}", scan_dir.path); @@ -199,13 +217,16 @@ impl LibraryScanner { } info!("Scanning directory: {}", scan_dir.path); - match self.scan_directory(scan_dir).await { + match self.scan_directory(scan_dir, completeness).await { Ok(mut files) => { info!("Found {} video files in {}", files.len(), scan_dir.path); all_files.append(&mut files); } Err(err) => { error!("Failed to scan directory {}: {err}", scan_dir.path); + if completeness == ScanCompleteness::Complete { + return Err(err); + } } } } @@ -215,22 +236,36 @@ impl LibraryScanner { } // Recursively scans a single directory for video files - async fn scan_directory(&self, scan_directory: &LibraryScanDirectory) -> io::Result> { + async fn scan_directory( + &self, + scan_directory: &LibraryScanDirectory, + completeness: ScanCompleteness, + ) -> io::Result> { let path = Path::new(&scan_directory.path); - if !fs::try_exists(path).await.unwrap_or(false) { + if completeness == ScanCompleteness::BestEffort && !fs::try_exists(path).await.unwrap_or(false) { warn!("Directory does not exist or is not readable: {}", scan_directory.path); return Ok(Vec::new()); } let dir_metadata = fs::metadata(path).await?; if !dir_metadata.is_dir() { + if completeness == ScanCompleteness::Complete { + return Err(io::Error::other("Library scan root is not a directory")); + } warn!("Path is not a directory: {}", scan_directory.path); return Ok(Vec::new()); } let mut files = Vec::new(); - self.scan_directory_recursive(path, scan_directory.recursive, scan_directory.content_type, &mut files).await?; + self.scan_directory_recursive( + path, + scan_directory.recursive, + scan_directory.content_type, + &mut files, + completeness, + ) + .await?; Ok(files) } @@ -240,6 +275,7 @@ impl LibraryScanner { recursive: bool, content_type: LibraryContentType, files: &'a mut Vec, + completeness: ScanCompleteness, ) -> std::pin::Pin> + Send + 'a>> { Box::pin(async move { let mut entries = fs::read_dir(path).await?; @@ -250,6 +286,9 @@ impl LibraryScanner { Ok(m) => m, Err(err) => { error!("Failed to read metadata for {}: {err}", entry_path.display()); + if completeness == ScanCompleteness::Complete { + return Err(err); + } continue; } }; @@ -257,10 +296,14 @@ impl LibraryScanner { if metadata.is_dir() { if recursive { // Recursively scan subdirectories - if let Err(err) = - self.scan_directory_recursive(&entry_path, recursive, content_type, files).await + if let Err(err) = self + .scan_directory_recursive(&entry_path, recursive, content_type, files, completeness) + .await { error!("Failed to scan subdirectory {}: {err}", entry_path.display()); + if completeness == ScanCompleteness::Complete { + return Err(err); + } } } } else if metadata.is_file() { @@ -275,6 +318,9 @@ impl LibraryScanner { } Err(err) => { error!("Failed to read metadata for {}: {err}", entry_path.display()); + if completeness == ScanCompleteness::Complete { + return Err(err); + } } } } diff --git a/backend/parser/src/m3u.rs b/backend/parser/src/m3u.rs index e0095e47e..18d58a257 100644 --- a/backend/parser/src/m3u.rs +++ b/backend/parser/src/m3u.rs @@ -515,7 +515,23 @@ fn parse_extvlcopt_user_agent(line: &str) -> Option<&str> { Some(value.trim()) } -pub async fn consume_m3u(cfg: &Config, input: &ConfigInput, lines: DynReader, mut visit: F) { +#[derive(Clone, Copy, PartialEq, Eq)] +enum ParseScope { + Direct, + Update, +} + +pub async fn consume_m3u(cfg: &Config, input: &ConfigInput, lines: DynReader, visit: F) { + let _ = consume_m3u_scoped(cfg, input, lines, visit, ParseScope::Direct).await; +} + +async fn consume_m3u_scoped( + cfg: &Config, + input: &ConfigInput, + lines: DynReader, + mut visit: F, + scope: ParseScope, +) -> std::io::Result<()> { let mut header: Option = None; let mut group: Option> = None; let mut upstream_user_agent: Option> = None; @@ -528,10 +544,27 @@ pub async fn consume_m3u(cfg: &Config, input: &ConfigInp }; let mut lines = tokio::io::BufReader::new(lines).lines(); let mut ord_counter: u32 = 1; - while let Ok(Some(line)) = lines.next_line().await { + let mut document_seen = false; + loop { + let line = match lines.next_line().await { + Ok(Some(line)) => line, + Err(error) if scope == ParseScope::Update => return Err(error), + Ok(None) | Err(_) => break, + }; + let line = + if scope == ParseScope::Update { line.trim_start_matches('\u{feff}').trim().to_owned() } else { line }; + if scope == ParseScope::Update && line.is_empty() { + continue; + } let bytes = line.as_bytes(); if let Some(b'#') = bytes.first().copied() { if bytes.starts_with(b"#EXTINF") { + if scope == ParseScope::Update + && (header.is_some() || !line.starts_with("#EXTINF:") || !line.contains(',')) + { + return Err(std::io::Error::new(std::io::ErrorKind::InvalidData, "Incomplete M3U entry")); + } + document_seen = true; header = Some(line); upstream_user_agent = None; continue; @@ -543,6 +576,7 @@ pub async fn consume_m3u(cfg: &Config, input: &ConfigInp continue; } if let Some(rest) = line.strip_prefix("#EXTM3U") { + document_seen = true; default_catchup_correction = parse_extm3u_catchup_correction(rest); continue; } @@ -552,6 +586,9 @@ pub async fn consume_m3u(cfg: &Config, input: &ConfigInp } continue; } + if scope == ParseScope::Update && header.is_none() { + return Err(std::io::Error::new(std::io::ErrorKind::InvalidData, "M3U content without EXTINF")); + } let group_value = group.take(); if let Some(header_value) = header.take() { let mut item = PlaylistItem { @@ -585,6 +622,10 @@ pub async fn consume_m3u(cfg: &Config, input: &ConfigInp visit(item); } } + if scope == ParseScope::Update && (!document_seen || header.is_some()) { + return Err(std::io::Error::new(std::io::ErrorKind::InvalidData, "Incomplete M3U document")); + } + Ok(()) } fn build_series_info(cfg: &Config, items: Vec) -> Option { @@ -676,27 +717,51 @@ fn build_series_info(cfg: &Config, items: Vec) -> Option Vec { + parse_m3u_scoped(cfg, input, lines, ParseScope::Direct).await.unwrap_or_default() +} + +/// Parses an update candidate completely; read/truncation errors must not become Quality fallback. +pub async fn parse_m3u_for_update( + cfg: &Config, + input: &ConfigInput, + lines: DynReader, +) -> std::io::Result> { + parse_m3u_scoped(cfg, input, lines, ParseScope::Update).await +} + +async fn parse_m3u_scoped( + cfg: &Config, + input: &ConfigInput, + lines: DynReader, + scope: ParseScope, +) -> std::io::Result> { let mut group_map: IndexMap> = IndexMap::new(); let mut series_map: IndexMap<(Arc, Arc), Vec> = IndexMap::new(); - consume_m3u(cfg, input, lines, |item| { - if item.header.xtream_cluster.is_series() { - let key = ( - shared::utils::deunicode_string(&item.header.group).to_lowercase().intern(), - shared::utils::deunicode_string(&item.header.parent_code).to_lowercase().intern(), - ); - series_map.entry(key).or_default().push(item); - return; - } + consume_m3u_scoped( + cfg, + input, + lines, + |item| { + if item.header.xtream_cluster.is_series() { + let key = ( + shared::utils::deunicode_string(&item.header.group).to_lowercase().intern(), + shared::utils::deunicode_string(&item.header.parent_code).to_lowercase().intern(), + ); + series_map.entry(key).or_default().push(item); + return; + } - let key = { - let header = &item.header; - let normalized_group = shared::utils::deunicode_string(&header.group).to_lowercase().intern(); - (header.xtream_cluster, normalized_group) - }; - group_map.entry(key).or_default().push(item); - }) - .await; + let key = { + let header = &item.header; + let normalized_group = shared::utils::deunicode_string(&header.group).to_lowercase().intern(); + (header.xtream_cluster, normalized_group) + }; + group_map.entry(key).or_default().push(item); + }, + scope, + ) + .await?; for ((_category, _series_name), items) in series_map { if let Some(series_info) = build_series_info(cfg, items) { @@ -706,7 +771,7 @@ pub async fn parse_m3u(cfg: &Config, input: &ConfigInput, lines: DynReader) -> V } let mut grp_id = 0; - group_map + Ok(group_map .into_values() .filter_map(|channels| { // create a group based on the first playlist item @@ -718,7 +783,7 @@ pub async fn parse_m3u(cfg: &Config, input: &ConfigInput, lines: DynReader) -> V None } }) - .collect() + .collect()) } #[cfg(test)] @@ -755,6 +820,51 @@ mod test { } } + #[tokio::test] + async fn m3u_update_quality_parse_failure_does_not_yield_a_partial_candidate() { + for text in [ + "", + "unavailable", + "#EXTM3U\n#EXTINF:-1,Missing URL\n", + "#EXTM3U\n#EXTINF:-1,One\n#EXTINF:-1,Two\nhttp://stream.example/two\n", + ] { + assert!( + super::parse_m3u_for_update(&Config::default(), &test_input(), make_reader(text)).await.is_err(), + "{text}" + ); + } + let complete = "#EXTM3U\n#EXTINF:-1,Channel\nhttp://stream.example/channel\n"; + let checked = + super::parse_m3u_for_update(&Config::default(), &test_input(), make_reader(complete)).await.unwrap(); + let direct = parse_m3u(&Config::default(), &test_input(), make_reader(complete)).await; + let projection = |groups: &[shared::model::PlaylistGroup]| { + groups + .iter() + .flat_map(|group| { + group.channels.iter().map(|item| { + (group.xtream_cluster, group.title.clone(), item.header.id.clone(), item.header.url.clone()) + }) + }) + .collect::>() + }; + assert_eq!(projection(&checked), projection(&direct)); + assert!(super::parse_m3u_for_update(&Config::default(), &test_input(), make_reader("#EXTM3U\n")) + .await + .unwrap() + .is_empty()); + } + + #[tokio::test] + async fn m3u_update_quality_read_error_is_not_a_successful_short_document() { + let mut bytes = b"#EXTM3U\n#EXTINF:-1,Channel\nhttp://stream.example/channel\n".to_vec(); + bytes.extend_from_slice(b"\xff\n"); + let reader: DynReader = Box::pin(std::io::Cursor::new(bytes.clone())); + assert!(super::parse_m3u_for_update(&Config::default(), &test_input(), reader).await.is_err()); + let reader: DynReader = Box::pin(std::io::Cursor::new(bytes)); + let direct = parse_m3u(&Config::default(), &test_input(), reader).await; + assert_eq!(direct.iter().map(|g| g.channels.len()).sum::(), 1, "Direct compatibility is unchanged"); + } + #[test] fn test_classify_token_length_dispatch_keeps_expected_mapping() { assert!(matches!(classify_token("xui-id"), M3uToken::ProviderId)); diff --git a/backend/processing/src/input_cache.rs b/backend/processing/src/input_cache.rs index 7781d47ba..3fa12c2f6 100644 --- a/backend/processing/src/input_cache.rs +++ b/backend/processing/src/input_cache.rs @@ -1,5 +1,6 @@ use log::{error, warn}; use serde::{Deserialize, Serialize}; +use shared::model::{PersistedPlaylistUpdateClusterSnapshot, PersistedPlaylistUpdateInputResult}; use std::{ collections::HashMap, fs, @@ -10,23 +11,28 @@ use tuliprox_repository::{build_input_storage_path, get_input_storage_path}; pub const STATUS_FILE: &str = "status.json"; -#[derive(Serialize, Deserialize, Default, Debug, Clone, PartialEq)] +#[derive(Serialize, Deserialize, Default, Debug, Clone, Copy, PartialEq, Eq)] pub enum ClusterState { #[default] Ok, Failed, } -#[derive(Serialize, Deserialize, Debug, Clone)] +#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq)] pub struct ClusterStatus { pub status: ClusterState, pub timestamp: u64, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub last_update: Option, } -#[derive(Serialize, Deserialize, Debug, Default, Clone)] +#[derive(Serialize, Deserialize, Debug, Default, Clone, PartialEq, Eq)] pub struct InputStatus { #[serde(default)] pub clusters: HashMap, + /// Last input completion for every input type; never used to establish cache validity. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub last_input_update: Option, } pub async fn resolve_input_storage_path(storage_dir: &str, input_name: &str) -> PathBuf { @@ -89,5 +95,139 @@ pub fn is_cache_valid(status: &InputStatus, cluster: &str, cache_duration_second pub fn update_cluster_status(status: &mut InputStatus, cluster: &str, state: ClusterState) { let now = SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_secs(); - status.clusters.insert(cluster.to_string(), ClusterStatus { status: state, timestamp: now }); + status.clusters.insert(cluster.to_string(), ClusterStatus { status: state, timestamp: now, last_update: None }); +} + +/// Replaces exactly one cluster's last snapshot while retaining its cache timestamp. +/// +/// Stalker stores cache validity under `default`; its timestamp and state seed a +/// per-cluster read projection without changing the cache key used by processing. +pub fn replace_cluster_snapshot( + status: &mut InputStatus, + cluster: &str, + snapshot: PersistedPlaylistUpdateClusterSnapshot, +) -> bool { + if let Some(cluster_status) = status.clusters.get_mut(cluster) { + if cluster_status.last_update == Some(snapshot) { + return false; + } + cluster_status.last_update = Some(snapshot); + return true; + } + + let Some(default_status) = status.clusters.get("default") else { + return false; + }; + status.clusters.insert( + cluster.to_string(), + ClusterStatus { + status: default_status.status, + timestamp: default_status.timestamp, + last_update: Some(snapshot), + }, + ); + true +} + +/// Replaces a cluster read projection from the canonical `default` cache status. +pub fn replace_cluster_snapshot_from_default_status( + status: &mut InputStatus, + cluster: &str, + snapshot: PersistedPlaylistUpdateClusterSnapshot, +) -> bool { + let Some(default_status) = status.clusters.get("default") else { + return false; + }; + let replacement = ClusterStatus { + status: default_status.status, + timestamp: default_status.timestamp, + last_update: Some(snapshot), + }; + if status.clusters.get(cluster) == Some(&replacement) { + return false; + } + status.clusters.insert(cluster.to_string(), replacement); + true +} + +#[cfg(test)] +mod tests { + use super::{ + load_input_status, replace_cluster_snapshot, replace_cluster_snapshot_from_default_status, save_input_status, + ClusterState, ClusterStatus, InputStatus, + }; + use shared::model::{ + PersistedPlaylistUpdateClusterSnapshot, PersistedPlaylistUpdateTechnicalState, PlaylistUpdateDataSource, + XtreamCluster, + }; + + #[test] + fn persisted_cluster_snapshot_reads_legacy_status_json_without_last_update() { + let temp = tempfile::tempdir().expect("temporary status directory"); + std::fs::write(temp.path().join(super::STATUS_FILE), r#"{"clusters":{"live":{"status":"Ok","timestamp":17}}}"#) + .expect("legacy status fixture"); + + let status = load_input_status(temp.path()); + + let live = status.clusters.get("live").expect("legacy live status"); + assert_eq!(live.status, ClusterState::Ok); + assert_eq!(live.timestamp, 17); + assert_eq!(live.last_update, None); + assert_eq!(status.last_input_update, None); + } + + #[test] + fn persisted_cluster_snapshot_replaces_one_value_without_history() { + let temp = tempfile::tempdir().expect("temporary status directory"); + let mut status = InputStatus::default(); + super::update_cluster_status(&mut status, XtreamCluster::Live.as_ref(), ClusterState::Ok); + let first = PersistedPlaylistUpdateClusterSnapshot { + source: Some(PlaylistUpdateDataSource::Provider), + technical_state: Some(PersistedPlaylistUpdateTechnicalState::Succeeded), + ..PersistedPlaylistUpdateClusterSnapshot::default() + }; + let replacement = PersistedPlaylistUpdateClusterSnapshot { + source: Some(PlaylistUpdateDataSource::Cache), + technical_state: Some(PersistedPlaylistUpdateTechnicalState::Succeeded), + ..PersistedPlaylistUpdateClusterSnapshot::default() + }; + + assert!(replace_cluster_snapshot(&mut status, XtreamCluster::Live.as_ref(), first)); + assert!(replace_cluster_snapshot(&mut status, XtreamCluster::Live.as_ref(), replacement)); + save_input_status(temp.path(), &status); + + let reloaded = load_input_status(temp.path()); + assert_eq!(reloaded.clusters.len(), 1); + assert_eq!( + reloaded.clusters["live"].last_update, + Some(replacement), + "the new completion must replace, not append to, the prior snapshot" + ); + let encoded = std::fs::read_to_string(temp.path().join(super::STATUS_FILE)).expect("persisted status"); + assert_eq!(encoded.matches("last_update").count(), 1); + } + + #[test] + fn persisted_cluster_snapshot_uses_current_canonical_default_status_for_projection() { + let mut status = InputStatus::default(); + status.clusters.insert( + "default".to_string(), + ClusterStatus { status: ClusterState::Failed, timestamp: 23, last_update: None }, + ); + status.clusters.insert( + XtreamCluster::Live.as_ref().to_string(), + ClusterStatus { status: ClusterState::Ok, timestamp: 17, last_update: None }, + ); + let snapshot = PersistedPlaylistUpdateClusterSnapshot { + source: Some(PlaylistUpdateDataSource::Provider), + ..PersistedPlaylistUpdateClusterSnapshot::default() + }; + + assert!(replace_cluster_snapshot_from_default_status(&mut status, XtreamCluster::Live.as_ref(), snapshot)); + + let projected = &status.clusters[XtreamCluster::Live.as_ref()]; + assert_eq!(projected.status, ClusterState::Failed); + assert_eq!(projected.timestamp, 23); + assert_eq!(projected.last_update, Some(snapshot)); + } } diff --git a/backend/processing/src/processor/library.rs b/backend/processing/src/processor/library.rs index fb9ca9208..bae2c44ef 100644 --- a/backend/processing/src/processor/library.rs +++ b/backend/processing/src/processor/library.rs @@ -66,16 +66,19 @@ pub async fn download_library_playlist( input: &ConfigInput, ) -> (Vec, Vec) { let config = &*app_config.config.load(); - let Some(library_config) = config.library.as_ref() else { return (vec![], vec![]) }; - if !library_config.enabled { - return (vec![], vec![]); - } + let failure = |error: String| (vec![], vec![TuliproxError::RepositoryLibrary(error)]); + let Some(library_config) = config.library.as_ref().filter(|library| library.enabled) else { + return failure("Library is disabled or unavailable".into()); + }; let api_base_path = concat_path_leading_slash(config.web_ui.as_ref().and_then(|w| w.path.as_deref()).unwrap_or(""), "api/v1"); let storage_path = resolve_metadata_storage_path(config.metadata_update.as_ref(), &config.storage_dir).join("library"); - let mut metadata_iter = MetadataAsyncIter::new(&storage_path).await; + let mut metadata_iter = match MetadataAsyncIter::try_new(&storage_path).await { + Ok(iter) => iter, + Err(err) => return failure(format!("Failed to open Library catalog: {err}")), + }; let mut group_movies = PlaylistGroup { id: 0, title: library_config.playlist.movie_category.clone(), @@ -88,7 +91,12 @@ pub async fn download_library_playlist( channels: vec![], xtream_cluster: XtreamCluster::Series, }; - while let Some(entry) = metadata_iter.next().await { + loop { + let entry = match metadata_iter.try_next().await { + Ok(Some(entry)) => entry, + Ok(None) => break, + Err(err) => return failure(format!("Failed to read Library catalog: {err}")), + }; match entry.metadata { MediaMetadata::Movie(_) => { to_playlist_item( diff --git a/backend/processing/src/processor/playlist/fetch_outcome.rs b/backend/processing/src/processor/playlist/fetch_outcome.rs new file mode 100644 index 000000000..e1b533cfb --- /dev/null +++ b/backend/processing/src/processor/playlist/fetch_outcome.rs @@ -0,0 +1,286 @@ +use crate::input_cache::{self, ClusterState, InputStatus}; +use log::warn; +use shared::model::{ + EventMessage, EventSink, PlaylistUpdateProgressEvent, PlaylistUpdateRunId, PlaylistUpdateRunOrder, XtreamCluster, +}; +use tuliprox_core::model::{ClusterForceUpdate, ClusterUpdateRejection, ConfigInput}; +use tuliprox_iptv::provider::PlaylistFetch; + +/// Cache entries whose state is determined by one completed provider fetch. +#[derive(Clone, Copy)] +pub(super) enum CacheStatusScope<'a> { + Default, + RequestedClusters(&'a [XtreamCluster]), +} + +/// Applies the observable and cache-state effects of a completed provider fetch. +/// +/// Quality rejections remain separate from technical errors. For per-cluster +/// fetches, only requested clusters are updated, so valid cached clusters keep +/// both their state and timestamp. +pub(super) fn apply_playlist_fetch_outcome( + events: &E, + run_id: &PlaylistUpdateRunId, + execution_order: PlaylistUpdateRunOrder, + input: &ConfigInput, + status: &mut InputStatus, + cache_scope: CacheStatusScope<'_>, + fetch: &PlaylistFetch, +) -> bool { + report_quality_rejections(events, run_id, execution_order, input, &fetch.quality_rejections); + if fetch.persisted { + report_force_updates(events, run_id, execution_order, input, &fetch.force_updates); + } + + let technical_failure = fetch.partial || !fetch.errors.is_empty(); + match cache_scope { + CacheStatusScope::RequestedClusters(requested_clusters) => { + for cluster in requested_clusters { + let rejected = fetch.quality_rejections.iter().any(|rejection| rejection.cluster == *cluster); + let state = if technical_failure || rejected { ClusterState::Failed } else { ClusterState::Ok }; + input_cache::update_cluster_status(status, cluster.as_ref(), state); + } + !requested_clusters.is_empty() + } + CacheStatusScope::Default => { + let default_state = if technical_failure || !fetch.quality_rejections.is_empty() { + ClusterState::Failed + } else { + ClusterState::Ok + }; + input_cache::update_cluster_status(status, "default", default_state); + + // M3U still acquires/caches one document, but its completed decisions + // and reload snapshots are per cluster. A rejected cluster invalidates + // the document cache without labelling accepted siblings Failed. + if input.get_download_input_type() == shared::model::InputType::M3u { + for cluster in super::ingest::PIPELINE_TRANSPARENCY_CLUSTERS { + if super::ingest::cluster_is_configured(input, cluster) { + let rejected = fetch.quality_rejections.iter().any(|item| item.cluster == cluster); + let state = if technical_failure || rejected { ClusterState::Failed } else { ClusterState::Ok }; + input_cache::update_cluster_status(status, cluster.as_ref(), state); + } + } + } + + for rejection in &fetch.quality_rejections { + input_cache::update_cluster_status(status, rejection.cluster.as_ref(), ClusterState::Failed); + } + true + } + } +} + +pub(super) fn report_force_updates( + events: &E, + run_id: &PlaylistUpdateRunId, + execution_order: PlaylistUpdateRunOrder, + input: &ConfigInput, + force_updates: &[ClusterForceUpdate], +) { + for force_update in force_updates { + let message = format!( + "Input '{}' cluster '{}' force-published: candidate={} configured_threshold={}", + input.name, + quality_cluster_name(force_update.cluster), + force_update.candidate_count, + force_update.configured_threshold + ); + log::info!("{message}"); + events.emit(EventMessage::PlaylistUpdateProgress(PlaylistUpdateProgressEvent::for_run_input( + run_id.clone(), + execution_order, + input.id, + input.name.to_string(), + message, + ))); + } +} + +fn report_quality_rejections( + events: &E, + run_id: &PlaylistUpdateRunId, + execution_order: PlaylistUpdateRunOrder, + input: &ConfigInput, + rejections: &[ClusterUpdateRejection], +) { + for rejection in rejections { + let message = format!( + "Input '{}' cluster '{}' rejected: current={} candidate={} threshold={} quality={}; retaining previous cluster", + input.name, + quality_cluster_name(rejection.cluster), + rejection.current_count, + rejection.candidate_count, + rejection.threshold, + rejection.quality + ); + warn!("{message}"); + events.emit(EventMessage::PlaylistUpdateProgress(PlaylistUpdateProgressEvent::for_run_input( + run_id.clone(), + execution_order, + input.id, + input.name.to_string(), + message, + ))); + } +} + +const fn quality_cluster_name(cluster: XtreamCluster) -> &'static str { + match cluster { + XtreamCluster::Live => "live", + XtreamCluster::Video => "vod", + XtreamCluster::Series => "series", + } +} + +#[cfg(test)] +mod tests { + use super::{apply_playlist_fetch_outcome, CacheStatusScope}; + use crate::input_cache::{is_cache_valid, ClusterState, ClusterStatus, InputStatus}; + use shared::model::{ + EventMessage, EventSink, InputType, PlaylistUpdateRunId, PlaylistUpdateRunOrder, XtreamCluster, + }; + use std::sync::{Arc, Mutex}; + use tuliprox_core::model::{ClusterForceUpdate, ClusterUpdateRejection, ConfigInput}; + use tuliprox_iptv::provider::PlaylistFetch; + + #[derive(Clone, Default)] + struct CollectSink(Arc>>); + + impl EventSink for CollectSink { + fn emit(&self, event: EventMessage) { self.0.lock().expect("event sink lock").push(event); } + } + + fn rejection(cluster: XtreamCluster) -> ClusterUpdateRejection { + ClusterUpdateRejection { cluster, current_count: 12_543, candidate_count: 217, threshold: 90, quality: 1 } + } + + fn run_id() -> PlaylistUpdateRunId { "fetch-outcome-run".into() } + + fn execution_order() -> PlaylistUpdateRunOrder { PlaylistUpdateRunOrder::from(41) } + + #[test] + fn playlist_update_run_mixed_cluster_fetch_reports_correlated_rejection() { + let input = ConfigInput { name: Arc::from("provider-a"), input_type: InputType::Xtream, ..Default::default() }; + let events = CollectSink::default(); + let mut status = InputStatus::default(); + status + .clusters + .insert("series".to_string(), ClusterStatus { status: ClusterState::Ok, timestamp: 17, last_update: None }); + let fetch = PlaylistFetch::groups(Vec::new()).with_quality_rejections(vec![rejection(XtreamCluster::Video)]); + + let changed = apply_playlist_fetch_outcome( + &events, + &run_id(), + execution_order(), + &input, + &mut status, + CacheStatusScope::RequestedClusters(&[XtreamCluster::Live, XtreamCluster::Video]), + &fetch, + ); + + assert!(changed); + assert_eq!( + status.clusters.get(XtreamCluster::Live.as_ref()).map(|entry| &entry.status), + Some(&ClusterState::Ok) + ); + assert_eq!( + status.clusters.get(XtreamCluster::Video.as_ref()).map(|entry| &entry.status), + Some(&ClusterState::Failed) + ); + let series = status.clusters.get("series").expect("cached series status"); + assert_eq!(series.status, ClusterState::Ok); + assert_eq!(series.timestamp, 17); + assert!(!status.clusters.contains_key("default")); + + let emitted = events.0.lock().expect("event sink lock"); + assert_eq!(emitted.len(), 1); + let EventMessage::PlaylistUpdateProgress(progress) = &emitted[0] else { + panic!("expected playlist update progress event"); + }; + assert_eq!(progress.run_id.as_ref().map(AsRef::as_ref), Some("fetch-outcome-run")); + assert_eq!(progress.execution_order, Some(execution_order())); + assert_eq!(progress.target, "provider-a"); + assert_eq!( + progress.message, + "Input 'provider-a' cluster 'vod' rejected: current=12543 candidate=217 threshold=90 quality=1; retaining previous cluster" + ); + } + + #[test] + fn synthetic_default_fetch_invalidates_default_and_cluster_cache_after_rejection() { + let input = ConfigInput { name: Arc::from("provider-a"), input_type: InputType::Stalker, ..Default::default() }; + let events = CollectSink::default(); + let mut status = InputStatus::default(); + status.clusters.insert( + "default".to_string(), + ClusterStatus { status: ClusterState::Ok, timestamp: 23, last_update: None }, + ); + let fetch = PlaylistFetch::groups(Vec::new()).with_quality_rejections(vec![rejection(XtreamCluster::Series)]); + + let changed = apply_playlist_fetch_outcome( + &events, + &run_id(), + execution_order(), + &input, + &mut status, + CacheStatusScope::Default, + &fetch, + ); + + assert!(changed); + let default = status.clusters.get("default").expect("default cache status"); + assert_eq!(default.status, ClusterState::Failed); + assert!(!is_cache_valid(&status, "default", u64::MAX)); + assert_eq!( + status.clusters.get(XtreamCluster::Series.as_ref()).map(|entry| &entry.status), + Some(&ClusterState::Failed) + ); + } + + #[test] + fn playlist_update_run_force_publication_emits_correlated_audit_progress() { + let input = ConfigInput { name: Arc::from("provider-a"), input_type: InputType::Xtream, ..Default::default() }; + let events = CollectSink::default(); + let mut status = InputStatus::default(); + status.clusters.insert( + XtreamCluster::Video.as_ref().to_string(), + ClusterStatus { status: ClusterState::Failed, timestamp: 17, last_update: None }, + ); + let fetch = PlaylistFetch::default() + .with_force_updates(vec![ClusterForceUpdate { + cluster: XtreamCluster::Video, + candidate_count: 12_450, + configured_threshold: 95, + }]) + .persisted(true); + + let changed = apply_playlist_fetch_outcome( + &events, + &run_id(), + execution_order(), + &input, + &mut status, + CacheStatusScope::RequestedClusters(&[XtreamCluster::Video]), + &fetch, + ); + + assert!(changed); + assert_eq!( + status.clusters.get(XtreamCluster::Video.as_ref()).map(|entry| &entry.status), + Some(&ClusterState::Ok) + ); + let emitted = events.0.lock().expect("event sink lock"); + assert_eq!(emitted.len(), 1); + let EventMessage::PlaylistUpdateProgress(progress) = &emitted[0] else { + panic!("expected playlist update progress event"); + }; + assert_eq!(progress.run_id.as_ref().map(AsRef::as_ref), Some("fetch-outcome-run")); + assert_eq!(progress.execution_order, Some(execution_order())); + assert_eq!(progress.target, "provider-a"); + assert_eq!( + progress.message, + "Input 'provider-a' cluster 'vod' force-published: candidate=12450 configured_threshold=95" + ); + } +} diff --git a/backend/processing/src/processor/playlist/ingest.rs b/backend/processing/src/processor/playlist/ingest.rs index 93818b89b..68fbb891f 100644 --- a/backend/processing/src/processor/playlist/ingest.rs +++ b/backend/processing/src/processor/playlist/ingest.rs @@ -1,5 +1,8 @@ #![allow(clippy::wildcard_imports)] -use super::*; +use super::{ + fetch_outcome::{apply_playlist_fetch_outcome, report_force_updates, CacheStatusScope}, + *, +}; // Inputs disabled in the config are always disabled. // Command-line targets can only restrict enabled inputs, never enable them. @@ -28,9 +31,34 @@ pub(crate) async fn with_sequential_group( pub(crate) struct PlaylistDownloadResult { pub downloaded_playlist: Vec, pub download_err: Vec, + pub quality_rejections: Vec, + pub force_updates: Vec, pub was_cached: bool, pub persisted: bool, pub partial: bool, + pub input_telemetry: Option, + pending_status: Option, + provider_failure_evidence: ProviderFailureEvidence, +} + +struct PendingInputStatusUpdate { + storage_path: PathBuf, + status: input_cache::InputStatus, + status_changed: bool, + cluster_status_source: ClusterStatusSource, +} + +#[derive(Clone, Copy)] +enum ClusterStatusSource { + PerCluster, + Default, +} + +#[derive(Clone, Copy, Default)] +enum ProviderFailureEvidence { + #[default] + None, + TechnicalFailure, } impl PlaylistDownloadResult { @@ -40,22 +68,405 @@ impl PlaylistDownloadResult { was_cached: bool, persisted: bool, ) -> Self { - Self { downloaded_playlist, download_err, was_cached, persisted, partial: false } + Self { + downloaded_playlist, + download_err, + quality_rejections: Vec::new(), + force_updates: Vec::new(), + was_cached, + persisted, + partial: false, + input_telemetry: None, + pending_status: None, + provider_failure_evidence: ProviderFailureEvidence::None, + } } - pub(crate) fn with_partial(mut self, partial: bool) -> Self { - self.partial = partial; + fn with_input_telemetry(mut self, input_telemetry: PlaylistUpdateInputTelemetry) -> Self { + self.input_telemetry = Some(input_telemetry); + self + } + + fn with_pending_status(mut self, pending_status: PendingInputStatusUpdate) -> Self { + self.pending_status = Some(pending_status); self } } +impl From for PlaylistDownloadResult { + fn from(fetch: PlaylistFetch) -> Self { + let provider_failure_evidence = if fetch.partial || !fetch.errors.is_empty() { + ProviderFailureEvidence::TechnicalFailure + } else { + ProviderFailureEvidence::None + }; + Self { + downloaded_playlist: fetch.groups, + download_err: fetch.errors, + quality_rejections: fetch.quality_rejections, + force_updates: fetch.force_updates, + was_cached: false, + persisted: fetch.persisted, + partial: fetch.partial, + input_telemetry: None, + pending_status: None, + provider_failure_evidence, + } + } +} + +pub(super) const PIPELINE_TRANSPARENCY_CLUSTERS: [XtreamCluster; 3] = + [XtreamCluster::Live, XtreamCluster::Video, XtreamCluster::Series]; + +fn is_cluster_input(input_type: InputType) -> bool { + matches!(input_type, InputType::Xtream | InputType::Stalker | InputType::M3u) +} + +pub(super) fn cluster_is_configured(input: &ConfigInput, cluster: XtreamCluster) -> bool { + let skip_flag = match cluster { + XtreamCluster::Live => ConfigInputFlags::SkipLive, + XtreamCluster::Video => ConfigInputFlags::SkipVod, + XtreamCluster::Series => ConfigInputFlags::SkipSeries, + }; + !input.has_flag(skip_flag) +} + +fn input_telemetry_for_fetch( + input: &ConfigInput, + refresh_policy: InputRefreshPolicy, + source: PlaylistUpdateDataSource, + provider_clusters: &[XtreamCluster], + fetch: Option<&PlaylistFetch>, +) -> PlaylistUpdateInputTelemetry { + let input_type = input.get_download_input_type(); + if !is_cluster_input(input_type) { + return PlaylistUpdateInputTelemetry { refresh_policy, source: Some(source), clusters: Vec::new() }; + } + + let mut candidate_counts = HashMap::::new(); + if let Some(fetch) = fetch { + for group in &fetch.groups { + *candidate_counts.entry(group.xtream_cluster).or_default() += group.channels.len(); + } + if fetch.errors.is_empty() && !fetch.partial && !fetch.persisted { + for cluster in provider_clusters { + candidate_counts.entry(*cluster).or_default(); + } + } + } + + let clusters = PIPELINE_TRANSPARENCY_CLUSTERS + .into_iter() + .map(|cluster| { + let requested = cluster_is_configured(input, cluster); + let source = requested.then(|| { + if provider_clusters.contains(&cluster) { + PlaylistUpdateDataSource::Provider + } else { + PlaylistUpdateDataSource::Cache + } + }); + let configured_threshold = + input.options.as_ref().map_or(0, |options| options.update_quality.threshold(cluster)); + let mut telemetry = PlaylistUpdateClusterTelemetry { + cluster, + requested, + source, + baseline_count: None, + candidate_count: candidate_counts.get(&cluster).copied(), + active_count: None, + threshold: (configured_threshold > 0).then_some(configured_threshold), + quality: None, + decision: None, + technical_state: None, + }; + + if let Some(fetch) = fetch { + if requested && fetch.failed_clusters.contains(&cluster) { + telemetry.technical_state = Some(PersistedPlaylistUpdateTechnicalState::Failed); + } + if let Some(rejection) = fetch.quality_rejections.iter().find(|item| item.cluster == cluster) { + telemetry.baseline_count = Some(rejection.current_count); + telemetry.candidate_count = Some(rejection.candidate_count); + telemetry.active_count = Some(rejection.current_count); + telemetry.threshold = Some(rejection.threshold); + telemetry.quality = Some(rejection.quality); + telemetry.decision = Some(PlaylistUpdateClusterDecision::Rejected); + } else if let Some(acceptance) = fetch.quality_acceptances.iter().find(|item| item.cluster == cluster) { + telemetry.baseline_count = acceptance.current_count; + telemetry.candidate_count = Some(acceptance.candidate_count); + telemetry.threshold = Some(acceptance.threshold); + telemetry.quality = acceptance.quality; + telemetry.decision = Some(PlaylistUpdateClusterDecision::Accepted); + } else if let Some(force_update) = fetch.force_updates.iter().find(|item| item.cluster == cluster) { + telemetry.candidate_count = Some(force_update.candidate_count); + telemetry.threshold = + (force_update.configured_threshold > 0).then_some(force_update.configured_threshold); + telemetry.decision = Some(PlaylistUpdateClusterDecision::Accepted); + } else if source == Some(PlaylistUpdateDataSource::Provider) { + if configured_threshold == 0 + && (candidate_counts.contains_key(&cluster) || (fetch.errors.is_empty() && !fetch.partial)) + { + telemetry.decision = Some(PlaylistUpdateClusterDecision::Accepted); + } else if (provider_clusters == [cluster] || input_type == InputType::M3u) + && fetch.groups.is_empty() + && !fetch.persisted + && !fetch.errors.is_empty() + { + telemetry.decision = Some(PlaylistUpdateClusterDecision::TechnicalError); + } + } + } + telemetry + }) + .collect(); + + PlaylistUpdateInputTelemetry { refresh_policy, source: None, clusters } +} + +fn confirm_published_cluster_counts(input_telemetry: &mut PlaylistUpdateInputTelemetry) { + for cluster in &mut input_telemetry.clusters { + if cluster.decision == Some(PlaylistUpdateClusterDecision::Accepted) { + cluster.active_count = cluster.candidate_count; + } + } +} + +fn finalize_input_telemetry( + playlist_download_result: &mut PlaylistDownloadResult, + storage_error: Option<&TuliproxError>, +) { + let activation_is_uncertain = storage_error.is_some() + || playlist_download_result.partial + || !playlist_download_result.download_err.is_empty(); + if let Some(input_telemetry) = playlist_download_result.input_telemetry.as_mut() { + if activation_is_uncertain { + for cluster in &mut input_telemetry.clusters { + cluster.active_count = None; + } + } else { + confirm_published_cluster_counts(input_telemetry); + } + } +} + +pub(crate) fn neutralize_overlaid_cluster_facts( + input_telemetry: &mut PlaylistUpdateInputTelemetry, + overlaid_clusters: ClusterFlags, +) { + for cluster in &mut input_telemetry.clusters { + if cluster_selected(cluster.cluster, overlaid_clusters) { + cluster.source = None; + cluster.baseline_count = None; + cluster.candidate_count = None; + cluster.active_count = None; + cluster.quality = None; + cluster.decision = None; + cluster.technical_state = None; + } + } +} + +fn persisted_quality_snapshot( + cluster: &PlaylistUpdateClusterTelemetry, + effective_policy: InputRefreshPolicy, +) -> Option { + if effective_policy.bypasses_quality() { + return None; + } + let threshold = cluster.threshold?; + let decision = match cluster.decision? { + PlaylistUpdateClusterDecision::Accepted => PersistedPlaylistUpdateQualityDecision::Accepted, + PlaylistUpdateClusterDecision::Rejected => PersistedPlaylistUpdateQualityDecision::Rejected, + PlaylistUpdateClusterDecision::TechnicalError => return None, + }; + Some(PersistedPlaylistUpdateQualitySnapshot { + threshold, + baseline_count: cluster.baseline_count, + candidate_count: cluster.candidate_count, + achieved_quality: cluster.quality, + decision, + }) +} + +#[derive(Clone, Copy)] +struct PersistedSnapshotContext { + effective_policy: InputRefreshPolicy, + request_policy: Option, + technical_failure: bool, + storage_failure: bool, + provider_failure_evidence: ProviderFailureEvidence, + provider_cluster_count: usize, +} + +fn persisted_technical_state( + cluster: &PlaylistUpdateClusterTelemetry, + context: PersistedSnapshotContext, +) -> Option { + let source = cluster.source?; + if cluster.technical_state == Some(PersistedPlaylistUpdateTechnicalState::Failed) { + return cluster.technical_state; + } + if cluster.decision == Some(PlaylistUpdateClusterDecision::TechnicalError) { + return Some(PersistedPlaylistUpdateTechnicalState::Failed); + } + if context.storage_failure && source == PlaylistUpdateDataSource::Provider { + return Some(PersistedPlaylistUpdateTechnicalState::Failed); + } + if !context.technical_failure { + return Some(PersistedPlaylistUpdateTechnicalState::Succeeded); + } + (source == PlaylistUpdateDataSource::Provider + && context.provider_cluster_count == 1 + && matches!(context.provider_failure_evidence, ProviderFailureEvidence::TechnicalFailure)) + .then_some(PersistedPlaylistUpdateTechnicalState::Failed) +} + +fn persisted_cluster_snapshot( + cluster: &PlaylistUpdateClusterTelemetry, + context: PersistedSnapshotContext, +) -> PersistedPlaylistUpdateClusterSnapshot { + let quality = persisted_quality_snapshot(cluster, context.effective_policy); + PersistedPlaylistUpdateClusterSnapshot { + policy: cluster.source.and(context.request_policy), + source: cluster.source, + quality_guard_threshold: (quality.is_none() && cluster.source.is_some()) + .then_some(cluster.threshold.unwrap_or(0)), + quality, + active_count: cluster.active_count, + technical_state: persisted_technical_state(cluster, context), + } +} + +fn persist_finalized_cluster_snapshots( + playlist_download_result: &mut PlaylistDownloadResult, + request_policy: Option, + storage_error: Option<&TuliproxError>, +) { + let technical_failure = storage_error.is_some() + || playlist_download_result.partial + || !playlist_download_result.download_err.is_empty(); + let provider_failure_evidence = playlist_download_result.provider_failure_evidence; + let snapshots = playlist_download_result.input_telemetry.as_ref().map_or_else(Vec::new, |telemetry| { + let provider_cluster_count = telemetry + .clusters + .iter() + .filter(|cluster| cluster.requested && cluster.source == Some(PlaylistUpdateDataSource::Provider)) + .count(); + telemetry + .clusters + .iter() + .filter(|cluster| cluster.requested) + .map(|cluster| { + let snapshot = persisted_cluster_snapshot( + cluster, + PersistedSnapshotContext { + effective_policy: telemetry.refresh_policy, + request_policy, + technical_failure, + storage_failure: storage_error.is_some(), + provider_failure_evidence, + provider_cluster_count, + }, + ); + (cluster.cluster, snapshot) + }) + .collect() + }); + + let Some(mut pending_status) = playlist_download_result.pending_status.take() else { + return; + }; + let mut changed = pending_status.status_changed; + for (cluster, snapshot) in snapshots { + changed |= match pending_status.cluster_status_source { + ClusterStatusSource::PerCluster => { + input_cache::replace_cluster_snapshot(&mut pending_status.status, cluster.as_ref(), snapshot) + } + ClusterStatusSource::Default => input_cache::replace_cluster_snapshot_from_default_status( + &mut pending_status.status, + cluster.as_ref(), + snapshot, + ), + }; + } + if changed { + input_cache::save_input_status(&pending_status.storage_path, &pending_status.status); + } +} + +fn cached_playlist_download_result(input: &ConfigInput, refresh_policy: InputRefreshPolicy) -> PlaylistDownloadResult { + PlaylistDownloadResult::new(vec![], vec![], true, false).with_input_telemetry(input_telemetry_for_fetch( + input, + refresh_policy, + PlaylistUpdateDataSource::Cache, + &[], + None, + )) +} + +/// Loads an input already acquired in this run without reporting a second acquisition. +fn in_run_reuse_playlist_download_result() -> PlaylistDownloadResult { + PlaylistDownloadResult::new(vec![], vec![], true, false) +} + +fn forced_empty_cluster_flags(force_updates: &[ClusterForceUpdate]) -> ClusterFlags { + force_updates.iter().filter(|update| update.candidate_count == 0).fold( + ClusterFlags::empty(), + |mut clusters, update| { + clusters.insert(match update.cluster { + XtreamCluster::Live => ClusterFlags::Live, + XtreamCluster::Video => ClusterFlags::Vod, + XtreamCluster::Series => ClusterFlags::Series, + }); + clusters + }, + ) +} + pub(crate) fn collect_effective_skip_clusters(input: &ConfigInput) -> Vec { - if !input.input_type.is_xtream() { + if !input.input_type.is_xtream() && input.input_type != InputType::M3u { return vec![]; } xtream::get_skip_cluster(input) } +fn report_forced_update_request( + ctx: &PlaylistProcessingContext, + input: &ConfigInput, + refresh_policy: InputRefreshPolicy, +) { + if !refresh_policy.bypasses_quality() { + return; + } + let input_type = input.get_download_input_type(); + if !is_cluster_input(input_type) { + return; + } + let clusters = [ + (ConfigInputFlags::SkipLive, "live"), + (ConfigInputFlags::SkipVod, "vod"), + (ConfigInputFlags::SkipSeries, "series"), + ] + .into_iter() + .filter_map(|(skip_flag, name)| (!input.has_flag(skip_flag)).then_some(name)) + .collect::>() + .join(","); + let message = + format!("Input '{}': forced update requested; cache and update-quality bypassed for {clusters}", input.name); + info!("{message}"); + ctx.events.emit(EventMessage::PlaylistUpdateProgress( + PlaylistUpdateProgressEvent::for_run_input( + ctx.run_id.clone(), + ctx.execution_order, + input.id, + input.name.to_string(), + message, + ) + .with_detail(shared::model::PlaylistUpdateProgressDetail::ForcedUpdateRequested), + )); +} + pub(crate) fn filter_skipped_clusters_from_source(source: PlaylistSource, input: &ConfigInput) -> PlaylistSource { let skip_clusters = collect_effective_skip_clusters(input); if skip_clusters.is_empty() { @@ -99,14 +510,46 @@ pub(crate) fn should_apply_staged_overlay(download_result: &PlaylistDownloadResu !download_result.was_cached } +#[derive(Clone, Copy)] +struct PlaylistUpdateExecutionRef<'a> { + run_id: &'a PlaylistUpdateRunId, + execution_order: PlaylistUpdateRunOrder, +} + +/// Local discovery is not a provider Refresh policy. Both can require a fresh read. +#[derive(Clone, Copy)] +enum InputAcquisition { + Provider(InputRefreshPolicy), + RescannedLibrary, +} + +impl InputAcquisition { + fn refresh_policy(self) -> InputRefreshPolicy { + match self { + Self::Provider(policy) => policy, + Self::RescannedLibrary => InputRefreshPolicy::NORMAL, + } + } + + fn bypasses_cache(self) -> bool { + match self { + Self::Provider(policy) => policy.bypasses_cache(), + Self::RescannedLibrary => true, + } + } +} + #[allow(clippy::too_many_lines)] -pub(crate) async fn playlist_download_from_input( +async fn playlist_download_from_input( client: &reqwest::Client, app_config: &Arc, events: &E, + execution: PlaylistUpdateExecutionRef<'_>, input: &ConfigInput, stalker_refresh_mode: StalkerRefreshMode, + acquisition: InputAcquisition, ) -> PlaylistDownloadResult { + let refresh_policy = acquisition.refresh_policy(); let config = &*app_config.config.load(); let storage_dir = &config.storage_dir; @@ -138,26 +581,50 @@ pub(crate) async fn playlist_download_from_input( let xtream_cache_candidates = xtream::requested_clusters(None, &skip_cluster); for cluster in xtream_cache_candidates { - if !input_cache::is_cache_valid(&status, cluster.as_ref(), cache_duration) { + if acquisition.bypasses_cache() || !input_cache::is_cache_valid(&status, cluster.as_ref(), cache_duration) { xtream_clusters_to_download.push(cluster); } } xtream_clusters_to_download.is_empty() } else { - input_cache::is_cache_valid(&status, "default", cache_duration) + !acquisition.bypasses_cache() && input_cache::is_cache_valid(&status, "default", cache_duration) + }; + let cluster_status_source = if use_per_cluster_cache || download_input_type == InputType::M3u { + ClusterStatusSource::PerCluster + } else { + ClusterStatusSource::Default }; if fully_cached { - return PlaylistDownloadResult::new(vec![], vec![], true, false); + return cached_playlist_download_result(input, refresh_policy).with_pending_status(PendingInputStatusUpdate { + storage_path, + status, + status_changed: false, + cluster_status_source, + }); } + let provider_clusters = if is_cluster_input(download_input_type) { + if use_per_cluster_cache { + xtream_clusters_to_download.clone() + } else { + PIPELINE_TRANSPARENCY_CLUSTERS + .into_iter() + .filter(|cluster| cluster_is_configured(input, *cluster)) + .collect() + } + } else { + Vec::new() + }; + let request = PlaylistFetchRequest { app_config, config: &app_config.config.load(), client, input, xtream_clusters: Some(xtream_clusters_to_download.as_slice()), + update_quality: refresh_policy.quality, }; // Each arm builds the provider its input type needs and awaits it in place: the @@ -165,7 +632,15 @@ pub(crate) async fn playlist_download_from_input( // and stays statically dispatched. What changed is the result - one named // `PlaylistFetch` instead of a six-element tuple assembled by position. let fetch = match download_input_type { - InputType::M3u => M3uProvider.fetch(&request).await, + InputType::M3u => { + super::m3u_quality::effective_m3u_fetch( + app_config, + input, + refresh_policy.quality, + M3uProvider.fetch(&request).await, + ) + .await + } InputType::Xtream => XtreamProvider::new(events).fetch(&request).await, InputType::M3uBatch | InputType::XtreamBatch | InputType::StalkerBatch => { BatchContainerProvider.fetch(&request).await @@ -214,55 +689,118 @@ pub(crate) async fn playlist_download_from_input( })); } - let PlaylistFetch { groups: playlist, errors, persisted, partial } = fetch; - - // Update Status - let save_status; - if partial { - input_cache::update_cluster_status(&mut status, "default", ClusterState::Failed); - save_status = true; - } else if errors.is_empty() { - if use_per_cluster_cache { - for cluster in &xtream_clusters_to_download { - input_cache::update_cluster_status(&mut status, cluster.as_ref(), ClusterState::Ok); - } - save_status = !xtream_clusters_to_download.is_empty(); - } else { - input_cache::update_cluster_status(&mut status, "default", ClusterState::Ok); - save_status = true; - } - } else if use_per_cluster_cache { - for cluster in &xtream_clusters_to_download { - input_cache::update_cluster_status(&mut status, cluster.as_ref(), ClusterState::Failed); - } - save_status = !xtream_clusters_to_download.is_empty(); + let cache_scope = if use_per_cluster_cache { + CacheStatusScope::RequestedClusters(&xtream_clusters_to_download) } else { - input_cache::update_cluster_status(&mut status, "default", ClusterState::Failed); - save_status = true; - } + CacheStatusScope::Default + }; + let save_status = apply_playlist_fetch_outcome( + events, + execution.run_id, + execution.execution_order, + input, + &mut status, + cache_scope, + &fetch, + ); - if save_status { - input_cache::save_input_status(&storage_path, &status); - } - - PlaylistDownloadResult::new(playlist, errors, false, persisted).with_partial(partial) + let input_telemetry = input_telemetry_for_fetch( + input, + refresh_policy, + PlaylistUpdateDataSource::Provider, + &provider_clusters, + Some(&fetch), + ); + PlaylistDownloadResult::from(fetch).with_input_telemetry(input_telemetry).with_pending_status( + PendingInputStatusUpdate { storage_path, status, status_changed: save_status, cluster_status_source }, + ) } -#[derive(Clone, Copy, Eq, PartialEq)] +#[derive(Clone, Copy, Debug, Eq, PartialEq)] pub(crate) enum InputJobState { Ready, Pending, Failed, } +pub(crate) struct InputDownloadResult { + /// Only a completely read Library catalog (or its persisted input) may be empty. + pub(crate) authoritative_library: bool, + pub(crate) errors: Vec, + pub(crate) source: PlaylistSource, + pub(crate) storage_error: Option, + pub(crate) partial: bool, + pub(crate) quality_rejections: Vec, + pub(crate) accepted_empty_clusters: ClusterFlags, + pub(crate) input_telemetry: Option, +} + +impl InputDownloadResult { + pub(super) fn update_state(&mut self) -> PlaylistUpdateState { + InputCompletionFacts { + job_state: self.job_state(), + had_errors: !self.errors.is_empty() || self.storage_error.is_some(), + had_quality_rejections: !self.quality_rejections.is_empty(), + } + .update_state() + } + + pub(crate) fn job_state(&mut self) -> InputJobState { + if self.partial { + InputJobState::Pending + } else if self.storage_error.is_some() + || (self.source.is_empty() && self.accepted_empty_clusters.is_empty() && !self.authoritative_library) + { + InputJobState::Failed + } else { + InputJobState::Ready + } + } +} + pub(crate) struct InputJobResult { pub(crate) index: usize, + pub(crate) input_id: u16, pub(crate) input_name: Arc, pub(crate) state: InputJobState, pub(crate) source: Option, pub(crate) epg: Option, pub(crate) stat: InputStats, pub(crate) errors: Vec, + pub(crate) accepted_empty_clusters: ClusterFlags, + pub(crate) had_quality_rejections: bool, + pub(crate) input_telemetry: Option, +} + +impl InputJobResult { + pub(crate) fn update_state(&self) -> PlaylistUpdateState { + InputCompletionFacts { + job_state: self.state, + had_errors: !self.errors.is_empty(), + had_quality_rejections: self.had_quality_rejections, + } + .update_state() + } +} + +// The same completion contract applies to a source job (including EPG errors) +// and an indirect staged download, which has no separate source job. +struct InputCompletionFacts { + job_state: InputJobState, + had_errors: bool, + had_quality_rejections: bool, +} + +impl InputCompletionFacts { + fn update_state(self) -> PlaylistUpdateState { + if self.job_state == InputJobState::Failed || self.had_errors { + PlaylistUpdateState::Failure + } else if self.job_state == InputJobState::Pending || self.had_quality_rejections { + PlaylistUpdateState::Partial + } else { + PlaylistUpdateState::Success + } + } } pub(crate) async fn process_input_job( @@ -287,38 +825,36 @@ pub(crate) async fn process_input_job_inner InputJobResult { let start_time = Instant::now(); let input_type = input.get_download_input_type(); - let broadcast_step = create_broadcast_callback(&ctx.events); + let broadcast_step = create_input_broadcast_callback(&ctx.events, &ctx.run_id, ctx.execution_order, input.id); broadcast_step("Playlist download", &format!("Downloading input '{}'", input.name)); - let (mut errors, mut source, storage_error, partial) = download_input(ctx, input, false).await; - let storage_failed = storage_error.is_some(); - if let Some(err) = storage_error { + let mut download = download_input(ctx, input, false).await; + let state = download.job_state(); + let storage_failed = download.storage_error.is_some(); + let had_quality_rejections = !download.quality_rejections.is_empty(); + if had_quality_rejections { + ctx.had_quality_rejections.store(true, std::sync::atomic::Ordering::Release); + } + if let Some(err) = download.storage_error.take() { broadcast_step("Playlist download", &format!("Failed to persist/load input '{}' playlist", input.name)); error!("Failed to persist input playlist {}", input.name); - errors.push(err); + download.errors.push(err); } - let epg = if input_type == InputType::Library || partial || storage_failed { + let epg = if input_type == InputType::Library || download.partial || storage_failed { None } else { - download_input_epg(ctx, input, &mut errors).await - }; - let group_count = source.get_group_count(); - let channel_count = source.get_channel_count(); - let state = if partial { - InputJobState::Pending - } else if storage_failed || source.is_empty() { - if source.is_empty() { - broadcast_step("Playlist download", &format!("Input '{}' playlist is empty", input.name)); - errors.push(TuliproxError::RepositoryPlaylist(format!("Source is empty {}", input.name))); - } - InputJobState::Failed - } else { - InputJobState::Ready + download_input_epg(ctx, input, &mut download.errors).await }; + let group_count = download.source.get_group_count(); + let channel_count = download.source.get_channel_count(); + if state == InputJobState::Failed && download.source.is_empty() && download.accepted_empty_clusters.is_empty() { + broadcast_step("Playlist download", &format!("Input '{}' playlist is empty", input.name)); + download.errors.push(TuliproxError::RepositoryPlaylist(format!("Source is empty {}", input.name))); + } let stat = create_input_stat( group_count, channel_count, - errors.len(), + download.errors.len(), input_type, &input.name, start_time.elapsed().as_secs(), @@ -326,12 +862,16 @@ pub(crate) async fn process_input_job_inner InputJobR let error = TuliproxError::RepositoryPlaylist(format!("Input '{}' processing panicked", input.name)); InputJobResult { index, + input_id: input.id, input_name: input.name.clone(), state: InputJobState::Failed, source: None, epg: None, stat: create_input_stat(0, 0, 1, input.get_download_input_type(), &input.name, 0), errors: vec![error], + accepted_empty_clusters: ClusterFlags::empty(), + had_quality_rejections: false, + input_telemetry: None, } } +pub(crate) fn report_input_job_completion( + events: &E, + run_id: &PlaylistUpdateRunId, + execution_order: PlaylistUpdateRunOrder, + result: &InputJobResult, +) { + report_input_completion( + events, + run_id, + execution_order, + result.input_id, + &result.input_name, + result.update_state(), + result.input_telemetry.as_ref(), + ); +} + +pub(super) fn report_input_completion( + events: &E, + run_id: &PlaylistUpdateRunId, + execution_order: PlaylistUpdateRunOrder, + input_id: u16, + input_name: &str, + state: PlaylistUpdateState, + input_telemetry: Option<&PlaylistUpdateInputTelemetry>, +) { + let input_name = sanitize_sensitive_info(input_name).into_owned(); + let message = match state { + PlaylistUpdateState::Success => format!("Input '{input_name}' completed successfully"), + PlaylistUpdateState::Partial => format!("Input '{input_name}' completed partially"), + PlaylistUpdateState::Failure => format!("Input '{input_name}' failed during update"), + }; + let progress = PlaylistUpdateProgressEvent::input_completed( + run_id.clone(), + execution_order, + input_id, + state, + input_name, + message, + ); + let progress = match input_telemetry { + Some(input_telemetry) => progress.with_input_telemetry(input_telemetry.clone()), + None => progress, + }; + events.emit(EventMessage::PlaylistUpdateProgress(progress)); +} + #[allow(clippy::too_many_lines)] pub(crate) async fn process_source( source_idx: usize, @@ -360,7 +951,7 @@ pub(crate) async fn process_source::new(); if let Some(source) = sources.get_source_at(source_idx) { let mut source_playlists = Vec::with_capacity(source.inputs.len()); - let broadcast_step = create_broadcast_callback(&ctx.events); + let broadcast_step = create_broadcast_callback(&ctx.events, &ctx.run_id, ctx.execution_order); let process_parallel = ctx.config.config.load().process_parallel; let mut disabled_inputs: Vec> = vec![]; let mut enabled_inputs = Vec::with_capacity(source.inputs.len()); @@ -400,10 +991,14 @@ pub(crate) async fn process_source, input: &Arc, allow_staged_input: bool, -) -> (Vec, PlaylistSource, Option, bool) { +) -> InputDownloadResult { if input.staged.is_some() && !allow_staged_input { - return (vec![], MemoryPlaylistSource::default().into_source(), None, false); + return InputDownloadResult { + authoritative_library: false, + errors: Vec::new(), + source: MemoryPlaylistSource::default().into_source(), + storage_error: None, + partial: false, + quality_rejections: Vec::new(), + accepted_empty_clusters: ClusterFlags::empty(), + input_telemetry: None, + }; } let staged_overlay = if input.staged.is_none() { @@ -534,6 +1139,7 @@ pub(crate) async fn download_input)> = None; @@ -586,8 +1202,10 @@ pub(crate) async fn download_input (pl_source, None), Err(e) => (MemoryPlaylistSource::default().into_source(), Some(e)), } } else { debug!("Persisting input '{}' playlist", input.name); - let (pl, err) = persist_input_playlist(&ctx.config, input, playlist_download_result.downloaded_playlist).await; + let (pl, err) = persist_input_playlist_with_options( + &ctx.config, + input, + std::mem::take(&mut playlist_download_result.downloaded_playlist), + InputPlaylistPersistOptions { accepted_empty_clusters }, + ) + .await; (MemoryPlaylistSource::new(pl).into_source(), err) }; @@ -622,17 +1262,30 @@ pub(crate) async fn download_input( @@ -677,19 +1360,18 @@ async fn download_m3u_alias_playlists(events: &E) -> StepMeasureCallback { +pub(crate) fn create_broadcast_callback( + events: &E, + run_id: &PlaylistUpdateRunId, + execution_order: PlaylistUpdateRunOrder, +) -> StepMeasureCallback { let events = events.clone(); + let run_id = run_id.clone(); Box::new(move |context: &str, msg: &str| { - events.emit(EventMessage::PlaylistUpdateProgress(PlaylistUpdateProgressEvent { - target: context.to_owned(), - message: msg.to_owned(), - })); + events.emit(EventMessage::PlaylistUpdateProgress(PlaylistUpdateProgressEvent::for_run_global( + run_id.clone(), + execution_order, + context, + msg, + ))); + }) +} + +fn create_input_broadcast_callback( + events: &E, + run_id: &PlaylistUpdateRunId, + execution_order: PlaylistUpdateRunOrder, + input_id: u16, +) -> StepMeasureCallback { + let events = events.clone(); + let run_id = run_id.clone(); + Box::new(move |context: &str, msg: &str| { + events.emit(EventMessage::PlaylistUpdateProgress(PlaylistUpdateProgressEvent::for_run_input( + run_id.clone(), + execution_order, + input_id, + context, + msg, + ))); }) } @@ -727,6 +1435,8 @@ pub(crate) fn create_input_stat( pub struct PlaylistProcessingContext { pub client: reqwest::Client, + pub run_id: PlaylistUpdateRunId, + pub execution_order: PlaylistUpdateRunOrder, pub config: Arc, pub user_targets: Arc, pub events: E, @@ -743,6 +1453,9 @@ pub struct PlaylistProcessingContext>>>, + /// Completion precedence for this context's `run_id`, keyed by stable input ID. + /// Fresh for every `exec_processing`; clones for parallel sources share it. + pub(super) input_completions: Arc>>, #[allow(clippy::type_complexity)] pub input_locks: Arc, Weak>>>>, @@ -751,7 +1464,13 @@ pub struct PlaylistProcessingContext>, pub pre_processed_inputs: Option>>>, pub stalker_refresh_mode: StalkerRefreshMode, + /// Resumable Stalker work that must remain `Pending` at input level. pub partial_refresh: Arc, + /// Completed, nonfatal quality decisions that make only the overall run partial. + pub had_quality_rejections: Arc, + /// Optional request-local behavior for one manually selected input. + pub input_refresh: Option, + pub(crate) library_update_mode: LibraryUpdateMode, } // Written out rather than derived: `#[derive(Clone)]` would demand `M: Clone`, @@ -760,23 +1479,44 @@ impl Clone for PlaylistProcessingCo fn clone(&self) -> Self { Self { client: self.client.clone(), + run_id: self.run_id.clone(), + execution_order: self.execution_order, config: Arc::clone(&self.config), user_targets: Arc::clone(&self.user_targets), events: self.events.clone(), playlist_state: self.playlist_state.clone(), disabled_headers: self.disabled_headers.clone(), processed_inputs: Arc::clone(&self.processed_inputs), + input_completions: Arc::clone(&self.input_completions), input_locks: Arc::clone(&self.input_locks), provider_manager: self.provider_manager.clone(), metadata_manager: self.metadata_manager.clone(), pre_processed_inputs: self.pre_processed_inputs.clone(), stalker_refresh_mode: self.stalker_refresh_mode, partial_refresh: Arc::clone(&self.partial_refresh), + had_quality_rejections: Arc::clone(&self.had_quality_rejections), + input_refresh: self.input_refresh, + library_update_mode: self.library_update_mode, } } } impl PlaylistProcessingContext { + #[must_use] + pub fn refresh_policy(&self, input_id: u16) -> InputRefreshPolicy { + self.input_refresh + .filter(|input_refresh| input_refresh.input_id == input_id) + .map_or(InputRefreshPolicy::NORMAL, |input_refresh| input_refresh.policy) + } + + fn acquisition(&self, input_id: u16) -> InputAcquisition { + if self.library_update_mode.reloads_input(input_id) { + InputAcquisition::RescannedLibrary + } else { + InputAcquisition::Provider(self.refresh_policy(input_id)) + } + } + pub async fn is_input_downloaded(&self, input_name: &str) -> bool { let processed = self.processed_inputs.lock().await; processed.contains(input_name) @@ -873,3 +1613,639 @@ pub(crate) async fn process_sources ConfigInput { + ConfigInput { id: 17, name: Arc::from("provider-a"), input_type, enabled: true, ..ConfigInput::default() } + } + + fn cluster(telemetry: &PlaylistUpdateInputTelemetry, cluster: XtreamCluster) -> &PlaylistUpdateClusterTelemetry { + telemetry.clusters.iter().find(|item| item.cluster == cluster).expect("configured cluster telemetry") + } + + const fn completed_snapshot_context( + effective_policy: InputRefreshPolicy, + request_policy: Option, + provider_cluster_count: usize, + ) -> PersistedSnapshotContext { + PersistedSnapshotContext { + effective_policy, + request_policy, + technical_failure: false, + storage_failure: false, + provider_failure_evidence: ProviderFailureEvidence::None, + provider_cluster_count, + } + } + + const fn failed_provider_snapshot_context( + effective_policy: InputRefreshPolicy, + request_policy: Option, + provider_cluster_count: usize, + ) -> PersistedSnapshotContext { + PersistedSnapshotContext { + effective_policy, + request_policy, + technical_failure: true, + storage_failure: false, + provider_failure_evidence: ProviderFailureEvidence::TechnicalFailure, + provider_cluster_count, + } + } + + #[test] + fn pipeline_transparency_telemetry_preserves_cache_rejection_and_force_decisions() { + let input = input(InputType::Xtream); + let fetch = PlaylistFetch::groups(Vec::new()) + .with_quality_rejections(vec![ClusterUpdateRejection { + cluster: XtreamCluster::Video, + current_count: 12_543, + candidate_count: 217, + threshold: 90, + quality: 1, + }]) + .with_force_updates(vec![ClusterForceUpdate { + cluster: XtreamCluster::Series, + candidate_count: 8_412, + configured_threshold: 100, + }]); + + let telemetry = input_telemetry_for_fetch( + &input, + InputRefreshPolicy::FORCE, + PlaylistUpdateDataSource::Provider, + &[XtreamCluster::Video, XtreamCluster::Series], + Some(&fetch), + ); + + let live = cluster(&telemetry, XtreamCluster::Live); + assert!(live.requested); + assert_eq!(live.source, Some(PlaylistUpdateDataSource::Cache)); + assert_eq!(live.decision, None); + + let video = cluster(&telemetry, XtreamCluster::Video); + assert_eq!(video.source, Some(PlaylistUpdateDataSource::Provider)); + assert_eq!(video.decision, Some(PlaylistUpdateClusterDecision::Rejected)); + assert_eq!( + (video.baseline_count, video.candidate_count, video.active_count), + (Some(12_543), Some(217), Some(12_543)) + ); + assert_eq!((video.threshold, video.quality), (Some(90), Some(1))); + + let series = cluster(&telemetry, XtreamCluster::Series); + assert_eq!(series.source, Some(PlaylistUpdateDataSource::Provider)); + assert_eq!(series.decision, Some(PlaylistUpdateClusterDecision::Accepted)); + assert_eq!((series.candidate_count, series.active_count), (Some(8_412), None)); + assert_eq!(series.threshold, Some(100)); + + let mut published = telemetry; + confirm_published_cluster_counts(&mut published); + assert_eq!(cluster(&published, XtreamCluster::Series).active_count, Some(8_412)); + } + + #[test] + fn pipeline_transparency_telemetry_preserves_accepted_quality_and_bootstrap_facts() { + let mut input = input(InputType::Xtream); + input.options = Some(ConfigInputOptions::from(&ConfigInputOptionsDto { + update_quality: ConfigInputUpdateQualityDto { live: 90, vod: 90, series: 0 }, + ..ConfigInputOptionsDto::default() + })); + let fetch = PlaylistFetch::groups(Vec::new()).with_quality_acceptances(vec![ + ClusterUpdateAcceptance { + cluster: XtreamCluster::Live, + current_count: Some(12_543), + candidate_count: 12_000, + threshold: 90, + quality: Some(95), + }, + ClusterUpdateAcceptance { + cluster: XtreamCluster::Video, + current_count: None, + candidate_count: 217, + threshold: 90, + quality: None, + }, + ]); + + let mut telemetry = input_telemetry_for_fetch( + &input, + InputRefreshPolicy::NORMAL, + PlaylistUpdateDataSource::Provider, + &PIPELINE_TRANSPARENCY_CLUSTERS, + Some(&fetch), + ); + + let live = cluster(&telemetry, XtreamCluster::Live); + assert_eq!(live.decision, Some(PlaylistUpdateClusterDecision::Accepted)); + assert_eq!((live.baseline_count, live.candidate_count), (Some(12_543), Some(12_000))); + assert_eq!((live.threshold, live.quality, live.active_count), (Some(90), Some(95), None)); + + let video = cluster(&telemetry, XtreamCluster::Video); + assert_eq!(video.decision, Some(PlaylistUpdateClusterDecision::Accepted)); + assert_eq!((video.baseline_count, video.candidate_count), (None, Some(217))); + assert_eq!((video.threshold, video.quality, video.active_count), (Some(90), None, None)); + + confirm_published_cluster_counts(&mut telemetry); + assert_eq!(cluster(&telemetry, XtreamCluster::Live).active_count, Some(12_000)); + assert_eq!(cluster(&telemetry, XtreamCluster::Video).active_count, Some(217)); + } + + #[test] + fn pipeline_transparency_accepted_quality_survives_fetch_error_without_active_claim() { + let mut input = input(InputType::Xtream); + input.options = Some(ConfigInputOptions::from(&ConfigInputOptionsDto { + update_quality: ConfigInputUpdateQualityDto { live: 90, ..ConfigInputUpdateQualityDto::default() }, + ..ConfigInputOptionsDto::default() + })); + let acceptance = ClusterUpdateAcceptance { + cluster: XtreamCluster::Live, + current_count: Some(100), + candidate_count: 95, + threshold: 90, + quality: Some(95), + }; + let fetch = PlaylistFetch::groups(Vec::new()) + .with_quality_acceptances(vec![acceptance]) + .with_errors(vec![TuliproxError::RepositoryXtream("post-quality publish failure".to_string())]); + let telemetry = input_telemetry_for_fetch( + &input, + InputRefreshPolicy::NORMAL, + PlaylistUpdateDataSource::Provider, + &[XtreamCluster::Live], + Some(&fetch), + ); + let mut download_result = PlaylistDownloadResult::from(fetch).with_input_telemetry(telemetry); + + finalize_input_telemetry(&mut download_result, None); + + let telemetry = download_result.input_telemetry.as_ref().expect("input telemetry"); + let live = cluster(telemetry, XtreamCluster::Live); + assert_eq!(live.decision, Some(PlaylistUpdateClusterDecision::Accepted)); + assert_eq!((live.baseline_count, live.candidate_count), (Some(100), Some(95))); + assert_eq!((live.threshold, live.quality), (Some(90), Some(95))); + assert_eq!(live.active_count, None); + assert_eq!(download_result.download_err.len(), 1); + } + + #[test] + fn pipeline_transparency_telemetry_keeps_zero_threshold_quality_disabled() { + let input = input(InputType::Xtream); + let fetch = PlaylistFetch::groups(Vec::new()); + + let telemetry = input_telemetry_for_fetch( + &input, + InputRefreshPolicy::NORMAL, + PlaylistUpdateDataSource::Provider, + &[XtreamCluster::Live], + Some(&fetch), + ); + + let live = cluster(&telemetry, XtreamCluster::Live); + assert_eq!(live.decision, Some(PlaylistUpdateClusterDecision::Accepted)); + assert_eq!(live.threshold, None); + assert_eq!(live.quality, None); + assert_eq!(live.active_count, None); + } + + #[test] + fn pipeline_transparency_telemetry_reports_non_cluster_cache_without_synthetic_clusters() { + let telemetry = input_telemetry_for_fetch( + &input(InputType::Plex), + InputRefreshPolicy::NORMAL, + PlaylistUpdateDataSource::Cache, + &[], + None, + ); + + assert_eq!(telemetry.refresh_policy, InputRefreshPolicy::NORMAL); + assert_eq!(telemetry.source, Some(PlaylistUpdateDataSource::Cache)); + assert!(telemetry.clusters.is_empty()); + } + + #[test] + fn pipeline_transparency_known_cluster_failure_keeps_quality_decision_and_neutralizes_overlay() { + let input = input(InputType::Xtream); + let acceptance = ClusterUpdateAcceptance { + cluster: XtreamCluster::Series, + current_count: Some(100), + candidate_count: 95, + threshold: 90, + quality: Some(95), + }; + let mut fetch = PlaylistFetch::groups(Vec::new()).with_quality_acceptances(vec![acceptance]); + fetch.record_cluster_error( + XtreamCluster::Series, + TuliproxError::RepositoryXtream("publish failure".to_string()), + ); + let telemetry = input_telemetry_for_fetch( + &input, + InputRefreshPolicy::NORMAL, + PlaylistUpdateDataSource::Provider, + &PIPELINE_TRANSPARENCY_CLUSTERS, + Some(&fetch), + ); + let mut result = PlaylistDownloadResult::from(fetch).with_input_telemetry(telemetry); + finalize_input_telemetry(&mut result, None); + let telemetry = result.input_telemetry.as_mut().unwrap(); + let shows = cluster(telemetry, XtreamCluster::Series); + assert_eq!(shows.decision, Some(PlaylistUpdateClusterDecision::Accepted)); + assert_eq!(shows.technical_state, Some(PersistedPlaylistUpdateTechnicalState::Failed)); + assert_eq!(shows.active_count, None); + let context = failed_provider_snapshot_context(InputRefreshPolicy::NORMAL, None, 3); + let snapshot = persisted_cluster_snapshot(shows, context); + assert_eq!(snapshot.technical_state, Some(PersistedPlaylistUpdateTechnicalState::Failed)); + assert_eq!(snapshot.quality.unwrap().decision, PersistedPlaylistUpdateQualityDecision::Accepted); + neutralize_overlaid_cluster_facts(telemetry, ClusterFlags::Series); + let overlaid = cluster(telemetry, XtreamCluster::Series); + assert_eq!(overlaid.technical_state, None); + assert_eq!(persisted_cluster_snapshot(overlaid, context).technical_state, None); + } + + #[test] + fn pipeline_transparency_telemetry_types_complete_provider_failure_without_guessing_metrics() { + let mut input = input(InputType::Xtream); + input.options = Some(ConfigInputOptions::from(&ConfigInputOptionsDto { + skip_vod: true, + skip_series: true, + ..ConfigInputOptionsDto::default() + })); + let fetch = PlaylistFetch::failed(TuliproxError::Download("provider unavailable".to_string())); + + let telemetry = input_telemetry_for_fetch( + &input, + InputRefreshPolicy::REFRESH, + PlaylistUpdateDataSource::Provider, + &[XtreamCluster::Live], + Some(&fetch), + ); + + assert_eq!(telemetry.refresh_policy, InputRefreshPolicy::REFRESH); + let live = cluster(&telemetry, XtreamCluster::Live); + assert!(live.requested); + assert_eq!(live.source, Some(PlaylistUpdateDataSource::Provider)); + assert_eq!(live.decision, Some(PlaylistUpdateClusterDecision::TechnicalError)); + assert!(live.baseline_count.is_none()); + assert!(live.candidate_count.is_none()); + assert!(live.active_count.is_none()); + assert!(live.quality.is_none()); + } + + #[test] + fn pipeline_transparency_telemetry_does_not_assign_an_unscoped_error_to_multiple_clusters() { + let input = input(InputType::Stalker); + let fetch = PlaylistFetch::failed(TuliproxError::Download("provider unavailable".to_string())); + + let telemetry = input_telemetry_for_fetch( + &input, + InputRefreshPolicy::NORMAL, + PlaylistUpdateDataSource::Provider, + &PIPELINE_TRANSPARENCY_CLUSTERS, + Some(&fetch), + ); + + assert!(telemetry.clusters.iter().all(|item| item.requested && item.decision.is_none())); + } + + #[test] + fn pipeline_transparency_telemetry_does_not_infer_an_unreported_quality_acceptance() { + let mut input = input(InputType::Xtream); + input.options = Some(ConfigInputOptions::from(&ConfigInputOptionsDto { + skip_vod: true, + update_quality: ConfigInputUpdateQualityDto { live: 90, ..ConfigInputUpdateQualityDto::default() }, + ..ConfigInputOptionsDto::default() + })); + let fetch = PlaylistFetch::groups(Vec::new()); + + let telemetry = input_telemetry_for_fetch( + &input, + InputRefreshPolicy::NORMAL, + PlaylistUpdateDataSource::Provider, + &[XtreamCluster::Live], + Some(&fetch), + ); + + let live = cluster(&telemetry, XtreamCluster::Live); + assert_eq!(live.threshold, Some(90)); + assert_eq!(live.decision, None, "an accepted quality decision was not retained by the provider result"); + let video = cluster(&telemetry, XtreamCluster::Video); + assert!(!video.requested); + assert_eq!(video.source, None); + assert_eq!(video.decision, None); + } + + #[test] + fn persisted_cluster_snapshot_preserves_quality_facts_without_merging_technical_state() { + let accepted = PlaylistUpdateClusterTelemetry { + cluster: XtreamCluster::Live, + requested: true, + source: Some(PlaylistUpdateDataSource::Provider), + baseline_count: Some(1_000), + candidate_count: Some(950), + active_count: Some(950), + threshold: Some(90), + quality: Some(95), + decision: Some(PlaylistUpdateClusterDecision::Accepted), + technical_state: None, + }; + + let completed = persisted_cluster_snapshot( + &accepted, + completed_snapshot_context(InputRefreshPolicy::NORMAL, Some(InputRefreshPolicy::REFRESH), 1), + ); + assert_eq!(completed.policy, Some(InputRefreshPolicy::REFRESH)); + assert_eq!(completed.source, Some(PlaylistUpdateDataSource::Provider)); + assert_eq!(completed.quality_guard_threshold, None, "an evaluation already owns its threshold"); + assert_eq!( + completed.quality, + Some(PersistedPlaylistUpdateQualitySnapshot { + threshold: 90, + baseline_count: Some(1_000), + candidate_count: Some(950), + achieved_quality: Some(95), + decision: PersistedPlaylistUpdateQualityDecision::Accepted, + }) + ); + assert_eq!(completed.active_count, Some(950)); + assert_eq!(completed.technical_state, Some(PersistedPlaylistUpdateTechnicalState::Succeeded)); + + let mut bootstrap = accepted.clone(); + bootstrap.baseline_count = None; + bootstrap.candidate_count = Some(42); + bootstrap.active_count = Some(42); + bootstrap.quality = None; + let bootstrap = + persisted_cluster_snapshot(&bootstrap, completed_snapshot_context(InputRefreshPolicy::NORMAL, None, 1)); + assert_eq!(bootstrap.policy, None, "an automatic run must not acquire a default policy"); + assert_eq!(bootstrap.quality_guard_threshold, None, "bootstrap evaluation already owns its threshold"); + assert_eq!( + bootstrap.quality, + Some(PersistedPlaylistUpdateQualitySnapshot { + threshold: 90, + baseline_count: None, + candidate_count: Some(42), + achieved_quality: None, + decision: PersistedPlaylistUpdateQualityDecision::Accepted, + }), + "bootstrap acceptance must not invent achieved Quality" + ); + + let mut rejected = accepted.clone(); + rejected.candidate_count = Some(217); + rejected.active_count = Some(1_000); + rejected.quality = Some(21); + rejected.decision = Some(PlaylistUpdateClusterDecision::Rejected); + let rejected = + persisted_cluster_snapshot(&rejected, completed_snapshot_context(InputRefreshPolicy::NORMAL, None, 1)); + assert_eq!( + rejected.quality.map(|quality| quality.decision), + Some(PersistedPlaylistUpdateQualityDecision::Rejected) + ); + assert_eq!(rejected.active_count, Some(1_000), "only the typed active baseline is retained"); + assert_eq!(rejected.quality_guard_threshold, None, "a rejection evaluation already owns its threshold"); + assert_eq!(rejected.technical_state, Some(PersistedPlaylistUpdateTechnicalState::Succeeded)); + + let mut accepted_then_failed = accepted; + accepted_then_failed.active_count = None; + let accepted_then_failed = persisted_cluster_snapshot( + &accepted_then_failed, + PersistedSnapshotContext { + effective_policy: InputRefreshPolicy::NORMAL, + request_policy: None, + technical_failure: true, + storage_failure: true, + provider_failure_evidence: ProviderFailureEvidence::None, + provider_cluster_count: 1, + }, + ); + assert_eq!( + accepted_then_failed.quality.map(|quality| quality.decision), + Some(PersistedPlaylistUpdateQualityDecision::Accepted) + ); + assert_eq!(accepted_then_failed.technical_state, Some(PersistedPlaylistUpdateTechnicalState::Failed)); + assert_eq!(accepted_then_failed.active_count, None); + } + + #[test] + fn persisted_cluster_snapshot_keeps_policy_source_and_disabled_guard_independent() { + let forced = PlaylistUpdateClusterTelemetry { + cluster: XtreamCluster::Video, + requested: true, + source: Some(PlaylistUpdateDataSource::Provider), + baseline_count: None, + candidate_count: Some(73), + active_count: Some(73), + threshold: Some(95), + quality: None, + decision: Some(PlaylistUpdateClusterDecision::Accepted), + technical_state: None, + }; + + let forced = persisted_cluster_snapshot( + &forced, + completed_snapshot_context(InputRefreshPolicy::FORCE, Some(InputRefreshPolicy::FORCE), 1), + ); + assert_eq!(forced.policy, Some(InputRefreshPolicy::FORCE)); + assert_eq!(forced.source, Some(PlaylistUpdateDataSource::Provider)); + assert_eq!(forced.quality_guard_threshold, Some(95)); + assert_eq!(forced.quality, None, "a Force bypass is not a Quality evaluation"); + + let disabled = PlaylistUpdateClusterTelemetry { + cluster: XtreamCluster::Series, + requested: true, + source: Some(PlaylistUpdateDataSource::Provider), + baseline_count: None, + candidate_count: Some(12), + active_count: Some(12), + threshold: None, + quality: None, + decision: Some(PlaylistUpdateClusterDecision::Accepted), + technical_state: None, + }; + let disabled = persisted_cluster_snapshot( + &disabled, + completed_snapshot_context(InputRefreshPolicy::FORCE, Some(InputRefreshPolicy::FORCE), 1), + ); + assert_eq!(disabled.quality_guard_threshold, Some(0)); + assert_eq!(disabled.quality, None, "threshold zero stays disabled even during Force"); + + let cached = PlaylistUpdateClusterTelemetry { + cluster: XtreamCluster::Live, + requested: true, + source: Some(PlaylistUpdateDataSource::Cache), + baseline_count: None, + candidate_count: None, + active_count: None, + threshold: Some(90), + quality: None, + decision: None, + technical_state: None, + }; + let cached = persisted_cluster_snapshot( + &cached, + completed_snapshot_context(InputRefreshPolicy::REFRESH, Some(InputRefreshPolicy::REFRESH), 0), + ); + assert_eq!(cached.policy, Some(InputRefreshPolicy::REFRESH)); + assert_eq!(cached.quality_guard_threshold, Some(90)); + assert_eq!( + cached.source, + Some(PlaylistUpdateDataSource::Cache), + "the actual source remains stronger than request policy" + ); + } + + #[test] + fn persisted_cluster_snapshot_final_status_boundary_replaces_provider_with_cache_without_history() { + let temp = tempfile::tempdir().expect("temporary status directory"); + let mut status = input_cache::InputStatus::default(); + status.clusters.insert( + XtreamCluster::Live.as_ref().to_string(), + input_cache::ClusterStatus { status: input_cache::ClusterState::Ok, timestamp: 17, last_update: None }, + ); + let provider_telemetry = PlaylistUpdateInputTelemetry { + refresh_policy: InputRefreshPolicy::REFRESH, + source: None, + clusters: vec![PlaylistUpdateClusterTelemetry { + cluster: XtreamCluster::Live, + requested: true, + source: Some(PlaylistUpdateDataSource::Provider), + baseline_count: Some(100), + candidate_count: Some(95), + active_count: Some(95), + threshold: Some(90), + quality: Some(95), + decision: Some(PlaylistUpdateClusterDecision::Accepted), + technical_state: None, + }], + }; + let mut provider_result = PlaylistDownloadResult::new(Vec::new(), Vec::new(), false, false) + .with_input_telemetry(provider_telemetry) + .with_pending_status(PendingInputStatusUpdate { + storage_path: temp.path().to_path_buf(), + status, + status_changed: true, + cluster_status_source: ClusterStatusSource::PerCluster, + }); + + persist_finalized_cluster_snapshots(&mut provider_result, Some(InputRefreshPolicy::REFRESH), None); + + let provider_status = input_cache::load_input_status(temp.path()); + let provider_snapshot = provider_status.clusters["live"].last_update.expect("provider snapshot"); + assert_eq!(provider_snapshot.source, Some(PlaylistUpdateDataSource::Provider)); + assert!(provider_snapshot.quality.is_some()); + + let cache_telemetry = PlaylistUpdateInputTelemetry { + refresh_policy: InputRefreshPolicy::NORMAL, + source: None, + clusters: vec![PlaylistUpdateClusterTelemetry { + cluster: XtreamCluster::Live, + requested: true, + source: Some(PlaylistUpdateDataSource::Cache), + baseline_count: None, + candidate_count: None, + active_count: None, + threshold: Some(90), + quality: None, + decision: None, + technical_state: None, + }], + }; + let mut cache_result = PlaylistDownloadResult::new(Vec::new(), Vec::new(), true, false) + .with_input_telemetry(cache_telemetry) + .with_pending_status(PendingInputStatusUpdate { + storage_path: temp.path().to_path_buf(), + status: provider_status, + status_changed: false, + cluster_status_source: ClusterStatusSource::PerCluster, + }); + + persist_finalized_cluster_snapshots(&mut cache_result, Some(InputRefreshPolicy::NORMAL), None); + + let cached_status = input_cache::load_input_status(temp.path()); + assert_eq!(cached_status.clusters.len(), 1); + let cached_snapshot = cached_status.clusters["live"].last_update.expect("replacement cache snapshot"); + assert_eq!(cached_snapshot.policy, Some(InputRefreshPolicy::NORMAL)); + assert_eq!(cached_snapshot.source, Some(PlaylistUpdateDataSource::Cache)); + assert_eq!(cached_snapshot.quality_guard_threshold, Some(90)); + assert_eq!(cached_snapshot.quality, None); + assert_eq!(cached_snapshot.active_count, None); + assert_eq!(cached_snapshot.technical_state, Some(PersistedPlaylistUpdateTechnicalState::Succeeded)); + let encoded = + std::fs::read_to_string(temp.path().join(input_cache::STATUS_FILE)).expect("persisted status JSON"); + assert_eq!(encoded.matches(r#""last_update""#).count(), 1); + assert!(!encoded.contains("history")); + } + + #[test] + fn persisted_cluster_snapshot_discards_unproven_staged_overlay_facts() { + let mut telemetry = PlaylistUpdateInputTelemetry { + refresh_policy: InputRefreshPolicy::FORCE, + source: None, + clusters: vec![PlaylistUpdateClusterTelemetry { + cluster: XtreamCluster::Video, + requested: true, + source: Some(PlaylistUpdateDataSource::Provider), + baseline_count: Some(100), + candidate_count: Some(95), + active_count: Some(95), + threshold: Some(90), + quality: Some(95), + decision: Some(PlaylistUpdateClusterDecision::Accepted), + technical_state: None, + }], + }; + neutralize_overlaid_cluster_facts(&mut telemetry, ClusterFlags::Vod); + + let snapshot = persisted_cluster_snapshot( + cluster(&telemetry, XtreamCluster::Video), + completed_snapshot_context(telemetry.refresh_policy, Some(InputRefreshPolicy::FORCE), 0), + ); + + assert_eq!(snapshot, PersistedPlaylistUpdateClusterSnapshot::default()); + } + + #[test] + fn persisted_cluster_snapshot_leaves_unscoped_multi_cluster_failure_unknown() { + let cluster = PlaylistUpdateClusterTelemetry { + cluster: XtreamCluster::Live, + requested: true, + source: Some(PlaylistUpdateDataSource::Provider), + baseline_count: Some(100), + candidate_count: Some(95), + active_count: None, + threshold: Some(90), + quality: Some(95), + decision: Some(PlaylistUpdateClusterDecision::Accepted), + technical_state: None, + }; + + let snapshot = + persisted_cluster_snapshot(&cluster, failed_provider_snapshot_context(InputRefreshPolicy::NORMAL, None, 2)); + + assert_eq!(snapshot.technical_state, None); + assert_eq!( + snapshot.quality.map(|quality| quality.decision), + Some(PersistedPlaylistUpdateQualityDecision::Accepted) + ); + + let unscoped_ancillary_failure = persisted_cluster_snapshot( + &cluster, + PersistedSnapshotContext { + effective_policy: InputRefreshPolicy::NORMAL, + request_policy: None, + technical_failure: true, + storage_failure: false, + provider_failure_evidence: ProviderFailureEvidence::None, + provider_cluster_count: 1, + }, + ); + assert_eq!( + unscoped_ancillary_failure.technical_state, None, + "an input-wide ancillary failure is not invented as a cluster-scoped provider failure" + ); + } +} diff --git a/backend/processing/src/processor/playlist/input_status.rs b/backend/processing/src/processor/playlist/input_status.rs new file mode 100644 index 000000000..95f58efbd --- /dev/null +++ b/backend/processing/src/processor/playlist/input_status.rs @@ -0,0 +1,75 @@ +use super::{InputDownloadResult, InputJobResult, MetadataUpdateSink, PlaylistProcessingContext}; +use crate::input_cache::{load_input_status, resolve_input_storage_path, save_input_status}; +use shared::model::{EventSink, PersistedPlaylistUpdateInputResult, PlaylistUpdateState}; +use std::{ + sync::Arc, + time::{SystemTime, UNIX_EPOCH}, +}; + +pub(super) async fn persist_input_job_result( + ctx: &PlaylistProcessingContext, + result: &InputJobResult, +) { + // Successful in-run reuse has no new acquisition facts and must not replace the + // original result (in particular Partial) or advance its completion timestamp. + let state = result.update_state(); + if result.input_telemetry.is_some() || state == PlaylistUpdateState::Failure { + persist_input_completion(ctx, result.input_id, &result.input_name, state).await; + } +} + +pub(super) async fn complete_staged_input( + ctx: &PlaylistProcessingContext, + input: &tuliprox_core::model::ConfigInput, + result: &mut InputDownloadResult, +) { + let state = result.update_state(); + if result.input_telemetry.is_some() || state == PlaylistUpdateState::Failure { + persist_input_completion(ctx, input.id, &input.name, state).await; + // Report the staged input's own facts before its parent consumes them. + // Successful in-run reuse has no new acquisition and stays silent. + super::ingest::report_input_completion( + &ctx.events, + &ctx.run_id, + ctx.execution_order, + input.id, + &input.name, + state, + result.input_telemetry.as_ref(), + ); + } +} + +pub(super) async fn persist_input_completion( + ctx: &PlaylistProcessingContext, + input_id: u16, + input_name: &Arc, + state: PlaylistUpdateState, +) { + // Same per-input lock and canonical writer as acquisition. Called after its + // own critical section (the staged caller still owns the parent lock). + let _input_lock = ctx.get_input_lock(input_name).await; + let storage_dir = ctx.config.config.load().storage_dir.clone(); + let path = resolve_input_storage_path(&storage_dir, input_name).await; + { + // Shared by source contexts of this run only; never compare with the + // previous run's persisted result. Weaker/equal completions do not move + // the timestamp belonging to an already recorded stronger result. + let mut completions = ctx.input_completions.lock().await; + if let Some(current) = completions.get(&input_id) { + use PlaylistUpdateState::{Failure, Partial, Success}; + match (current, state) { + (Failure, _) | (Partial, Partial | Success) | (Success, Success) => return, + (Partial, Failure) | (Success, Partial | Failure) => {} + } + } + completions.insert(input_id, state); + } + let mut status = load_input_status(&path); + status.last_input_update = Some(PersistedPlaylistUpdateInputResult { + state, + timestamp: SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_secs(), + }); + // Do not touch the cache's cluster status/timestamp or its proven runtime facts. + save_input_status(&path, &status); +} diff --git a/backend/processing/src/processor/playlist/input_status_tests.rs b/backend/processing/src/processor/playlist/input_status_tests.rs new file mode 100644 index 000000000..01cfeb650 --- /dev/null +++ b/backend/processing/src/processor/playlist/input_status_tests.rs @@ -0,0 +1,404 @@ +mod persisted_input_status { + use super::*; + use crate::processor::playlist::input_status::{persist_input_completion, persist_input_job_result}; + use shared::model::PersistedPlaylistUpdateInputResult; + + fn acquired_result(state: PlaylistUpdateState) -> InputJobResult { + let mut result = super::super::playlist_update_run_result( + 7, + InputJobState::Ready, + vec![], + state == PlaylistUpdateState::Partial, + ); + result.input_telemetry = Some(PlaylistUpdateInputTelemetry { + refresh_policy: InputRefreshPolicy::REFRESH, + source: Some(PlaylistUpdateDataSource::Provider), + clusters: Vec::new(), + }); + result + } + + #[tokio::test] + async fn playlist_update_status_run_precedence_is_isolated_by_stable_input_id() { + let temp = tempfile::tempdir().unwrap(); + let ctx = processing_context(temp.path()); + let failed = super::super::playlist_update_run_result(7, InputJobState::Failed, vec![], false); + persist_input_job_result(&ctx, &failed).await; + let mut independent = acquired_result(PlaylistUpdateState::Success); + independent.input_id = 8; + independent.input_name = "independent".intern(); + persist_input_job_result(&ctx, &independent).await; + for (result, state) in [(failed, PlaylistUpdateState::Failure), (independent, PlaylistUpdateState::Success)] { + let path = + input_cache::resolve_input_storage_path(&temp.path().to_string_lossy(), &result.input_name).await; + assert_eq!(input_cache::load_input_status(&path).last_input_update.unwrap().state, state); + } + } + + #[tokio::test] + async fn m3u_update_quality_playlist_update_status_staged_completion_is_independent_of_parent_failure() { + for (parent_fails, staged_fails) in [(false, false), (true, false), (false, true)] { + let temp = tempfile::tempdir().unwrap(); + let events = CollectSink::default(); + let ctx = processing_context_with_events(temp.path(), events.clone()); + let parent_path = temp.path().join("parent.m3u"); + let staged_path = temp.path().join("overlay.m3u"); + for (path, fails) in [(&parent_path, parent_fails), (&staged_path, staged_fails)] { + if !fails { + std::fs::write(path, "#EXTM3U\n#EXTINF:-1,Fixture\nhttp://stream.example/fixture\n").unwrap(); + } + } + let parent = Arc::new(ConfigInput { + id: 7, + name: "parent".intern(), + input_type: InputType::M3u, + url: parent_path.to_string_lossy().into_owned(), + enabled: true, + ..ConfigInput::default() + }); + let mut staged = ConfigInput { + id: 8, + name: "staged".intern(), + input_type: InputType::Staged, + url: staged_path.to_string_lossy().into_owned(), + enabled: true, + staged: Some(tuliprox_core::model::ConfigInputStaged { + for_input: Some(parent.name.clone()), + clusters: ClusterFlags::Live, + }), + ..ConfigInput::default() + }; + staged.resolve_staged_download_type(); + let staged = Arc::new(staged); + ctx.config.sources.store(Arc::new(SourcesConfig { + inputs: vec![parent.clone(), staged.clone()], + ..SourcesConfig::default() + })); + let result = download_input(&ctx, &parent, false).await; + assert_eq!(!result.errors.is_empty(), parent_fails || staged_fails); + let path = input_storage_path(&ctx, &staged).await; + let persisted = input_cache::load_input_status(&path); + assert_eq!( + persisted.last_input_update.unwrap().state, + if staged_fails { PlaylistUpdateState::Failure } else { PlaylistUpdateState::Success } + ); + assert_eq!(persisted.clusters.len(), 4); + assert!(persisted.clusters["default"].last_update.is_none(), "document cache and cluster facts stay separate"); + for cluster in XTREAM_CLUSTER { + assert!(persisted.clusters[cluster.as_ref()].last_update.is_some()); + } + let completions = staged_completion::input_completions(&events, staged.id); + assert_eq!(completions.len(), 1, "real staged download must report its own completion"); + assert_eq!(completions[0].state, persisted.last_input_update.map(|result| result.state)); + assert_eq!(completions[0].run_id, Some(ctx.run_id.clone())); + assert_eq!(completions[0].execution_order, Some(ctx.execution_order)); + let telemetry = completions[0].input_telemetry.as_ref().unwrap(); + assert_eq!(telemetry.source, None); + assert!(telemetry.clusters.iter().all(|cluster| cluster.source == Some(PlaylistUpdateDataSource::Provider))); + assert!(staged_completion::input_completions(&events, parent.id).is_empty()); + } + } + + #[tokio::test] + async fn playlist_update_status_delayed_source_results_keep_strongest_completion_until_next_run() { + for weaker in [PlaylistUpdateState::Success, PlaylistUpdateState::Partial] { + for failure_first in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let events = CollectSink::default(); + let ctx = processing_context_with_events(temp.path(), events.clone()); + let acquired = acquired_result(weaker); + let failed_reuse = super::super::playlist_update_run_result(7, InputJobState::Failed, vec![], false); + let (first, delayed) = if failure_first { (failed_reuse, acquired) } else { (acquired, failed_reuse) }; + let path = + input_cache::resolve_input_storage_path(&temp.path().to_string_lossy(), &first.input_name).await; + let mut original = input_cache::InputStatus::default(); + original.clusters.insert( + "live".into(), + input_cache::ClusterStatus { + status: input_cache::ClusterState::Failed, + timestamp: 123, + last_update: Some(PersistedPlaylistUpdateClusterSnapshot { + source: Some(PlaylistUpdateDataSource::Cache), + policy: Some(InputRefreshPolicy::FORCE), + quality_guard_threshold: Some(95), + ..PersistedPlaylistUpdateClusterSnapshot::default() + }), + }, + ); + input_cache::save_input_status(&path, &original); + let (release, other_input_finished) = tokio::sync::oneshot::channel(); + let delayed_ctx = ctx.clone(); + // Source A already holds its shared-input result, but cannot drain + // its sorted result list until another input job has completed. + let delayed_source = async move { + let source_results = vec![delayed]; + other_input_finished.await.unwrap(); + for result in source_results { + persist_input_job_result(&delayed_ctx, &result).await; + report_input_job_completion( + &delayed_ctx.events, + &delayed_ctx.run_id, + delayed_ctx.execution_order, + &result, + ); + } + }; + let first_source = async { + persist_input_job_result(&ctx, &first).await; + report_input_job_completion(&ctx.events, &ctx.run_id, ctx.execution_order, &first); + let mut persisted = input_cache::load_input_status(&path); + assert_eq!(persisted.last_input_update.unwrap().state, first.update_state()); + // Distinct sentinel proves a delayed weaker result does not + // advance the stronger completion's timestamp, without sleeps. + persisted.last_input_update.as_mut().unwrap().timestamp = 456; + input_cache::save_input_status(&path, &persisted); + release.send(()).unwrap(); + }; + tokio::join!(delayed_source, first_source); + let final_status = input_cache::load_input_status(&path); + let completion = final_status.last_input_update.unwrap(); + assert_eq!( + completion.state, + PlaylistUpdateState::Failure, + "weaker={weaker:?}, failure_first={failure_first}" + ); + if failure_first { + assert_eq!(completion.timestamp, 456); + } + assert_eq!(final_status.clusters, original.clusters); + let states: Vec<_> = events + .0 + .lock() + .unwrap() + .iter() + .filter_map(|event| match event { + EventMessage::PlaylistUpdateProgress(progress) => progress.state, + _ => None, + }) + .collect(); + assert_eq!( + states, + if failure_first { + vec![PlaylistUpdateState::Failure, weaker] + } else { + vec![weaker, PlaylistUpdateState::Failure] + } + ); + + let mut next = processing_context(temp.path()); + next.run_id = "next-successful-run".into(); + next.execution_order = 2.into(); + let success = acquired_result(PlaylistUpdateState::Success); + persist_input_job_result(&next, &success).await; + let replacement = input_cache::load_input_status(&path); + assert_eq!(replacement.last_input_update.unwrap().state, PlaylistUpdateState::Success); + assert_eq!(replacement.clusters, original.clusters); + } + } + } + + #[tokio::test] + async fn playlist_update_status_parent_download_persists_staged_own_per_cluster_success_and_failure() { + for fails in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let mut responses = fixture_responses([1, 0, 0]); + if fails { + responses.insert("get_live_streams".into(), "not JSON".into()); + } + let server = TestXtreamServer::start_with_responses(responses); + let events = CollectSink::default(); + let ctx = processing_context_with_events(temp.path(), events.clone()); + let parent_file = temp.path().join("parent.m3u"); + std::fs::write(&parent_file, "#EXTM3U\n#EXTINF:-1,Parent\nhttp://stream.example/parent\n").unwrap(); + let parent = Arc::new(ConfigInput { + id: 7, + name: "parent".intern(), + input_type: InputType::M3u, + url: parent_file.to_string_lossy().into_owned(), + enabled: true, + ..ConfigInput::default() + }); + let mut staged = + (*test_input(&server.base_url, ConfigInputUpdateQualityDto { live: 95, vod: 0, series: 0 }, 0, &[])) + .clone(); + staged.id = 8; + staged.name = "own-staged".intern(); + staged.input_type = InputType::Staged; + staged.staged_type = shared::model::StagedInputType::Xtream; + staged.staged = Some(tuliprox_core::model::ConfigInputStaged { + for_input: Some(parent.name.clone()), + clusters: ClusterFlags::Live, + }); + staged.options = Some(ConfigInputOptions::from(&ConfigInputOptionsDto { + skip_vod: true, + skip_series: true, + update_quality: ConfigInputUpdateQualityDto { live: 95, vod: 0, series: 0 }, + ..ConfigInputOptionsDto::default() + })); + staged.resolve_staged_download_type(); // Same resolver used when SourcesConfig is built. + let staged = Arc::new(staged); + ctx.config.sources.store(Arc::new(SourcesConfig { + inputs: vec![parent.clone(), staged.clone()], + ..SourcesConfig::default() + })); + let staged_path = input_storage_path(&ctx, &staged).await; + // Referencing or directly skipping a staged input is not an acquisition. + let skipped = download_input(&ctx, &staged, false).await; + assert!(skipped.input_telemetry.is_none()); + assert!(input_cache::load_input_status(&staged_path).last_input_update.is_none()); + assert!(staged_completion::input_completions(&events, staged.id).is_empty()); + + let result = download_input(&ctx, &parent, false).await; + let status = input_cache::load_input_status(&staged_path); + assert!(!status.clusters.contains_key("default")); + assert_eq!(status.clusters.len(), 1); + let completion = + status.last_input_update.expect("actual indirect staged acquisition has its own completion"); + assert_eq!( + completion.state, + if fails { PlaylistUpdateState::Failure } else { PlaylistUpdateState::Success } + ); + assert!(completion.timestamp > 0); + let completions = staged_completion::input_completions(&events, staged.id); + assert_eq!(completions.len(), 1); + assert_eq!(completions[0].state, Some(completion.state)); + assert_eq!(completions[0].run_id, Some(ctx.run_id.clone())); + assert_eq!(completions[0].execution_order, Some(ctx.execution_order)); + let snapshot = status.clusters["live"].last_update.unwrap(); + assert_eq!(snapshot.source, Some(PlaylistUpdateDataSource::Provider)); + assert_eq!(snapshot.policy, None); + if fails { + assert_eq!(snapshot.quality_guard_threshold, Some(95)); + assert_eq!(snapshot.quality, None); + } else { + assert_eq!(snapshot.quality_guard_threshold, None, "the evaluation already owns its threshold"); + let quality = snapshot.quality.unwrap(); + assert_eq!(quality.threshold, 95); + assert_eq!(quality.decision, PersistedPlaylistUpdateQualityDecision::Accepted); + assert_eq!(quality.achieved_quality, None, "bootstrap has no invented Quality"); + } + assert_eq!(result.errors.is_empty(), !fails); + let parent_path = input_storage_path(&ctx, &parent).await; + assert_ne!(parent_path, staged_path); + assert!( + input_cache::load_input_status(&parent_path).last_input_update.is_none(), + "staged result does not invent a parent job completion" + ); + if !fails { + let bytes = std::fs::read(staged_path.join(input_cache::STATUS_FILE)).unwrap(); + let reuse = download_input(&ctx, &parent, false).await; + assert!(reuse.input_telemetry.is_none()); + let mut staged_reuse = download_input(&ctx, &staged, true).await; + assert!(staged_reuse.input_telemetry.is_none()); + super::super::input_status::complete_staged_input(&ctx, &staged, &mut staged_reuse).await; + assert_eq!(std::fs::read(staged_path.join(input_cache::STATUS_FILE)).unwrap(), bytes); + assert_eq!(staged_completion::input_completions(&events, staged.id), completions); + } + let requests = server.finish(); + assert_eq!( + requests.iter().filter(|action| action.as_str() == "get_live_streams").count(), + 1, + "no extra staged acquisition on parent reuse" + ); + } + } + + #[tokio::test] + async fn playlist_update_status_completion_replaces_one_result_without_touching_cache_or_cluster_snapshots() { + let temp = tempfile::tempdir().unwrap(); + let ctx = processing_context(temp.path()); + let input_name: Arc = "result-input".intern(); + let path = input_cache::resolve_input_storage_path(&temp.path().to_string_lossy(), &input_name).await; + let mut original = input_cache::InputStatus::default(); + input_cache::update_cluster_status(&mut original, "default", input_cache::ClusterState::Ok); + original.clusters.get_mut("default").unwrap().timestamp = 1; // Expired cache must stay expired. + input_cache::replace_cluster_snapshot( + &mut original, + "default", + PersistedPlaylistUpdateClusterSnapshot { + source: Some(PlaylistUpdateDataSource::Cache), + ..PersistedPlaylistUpdateClusterSnapshot::default() + }, + ); + input_cache::save_input_status(&path, &original); + for state in [PlaylistUpdateState::Success, PlaylistUpdateState::Partial, PlaylistUpdateState::Failure] { + persist_input_completion(&ctx, 7, &input_name, state).await; + let reloaded = input_cache::load_input_status(&path); + let result = reloaded.last_input_update.unwrap(); + assert_eq!(result.state, state); + assert!(result.timestamp > 1); + assert_eq!(reloaded.clusters, original.clusters); + assert!(!input_cache::is_cache_valid(&reloaded, "default", 3600)); + let json: serde_json::Value = + serde_json::from_slice(&std::fs::read(path.join(input_cache::STATUS_FILE)).unwrap()).unwrap(); + assert!(json["last_input_update"].is_object()); + assert_eq!(json.as_object().unwrap().len(), 2, "only clusters + one result, no history"); + } + } + + #[tokio::test] + async fn playlist_update_status_in_run_reuse_preserves_original_partial_result_and_timestamp() { + let temp = tempfile::tempdir().unwrap(); + let ctx = processing_context(temp.path()); + let mut result = super::super::playlist_update_run_result(7, InputJobState::Ready, vec![], true); + result.input_telemetry = Some(PlaylistUpdateInputTelemetry { + refresh_policy: InputRefreshPolicy::NORMAL, + source: Some(PlaylistUpdateDataSource::Provider), + clusters: Vec::new(), + }); + persist_input_job_result(&ctx, &result).await; + let path = input_cache::resolve_input_storage_path(&temp.path().to_string_lossy(), &result.input_name).await; + let mut original = input_cache::load_input_status(&path); + assert_eq!(original.last_input_update.unwrap().state, PlaylistUpdateState::Partial); + original.last_input_update.as_mut().unwrap().timestamp = 123; + input_cache::save_input_status(&path, &original); + let before = std::fs::read(path.join(input_cache::STATUS_FILE)).unwrap(); + result.had_quality_rejections = false; + result.input_telemetry = None; // Same existing absence of acquisition facts as a real in-run reuse. + persist_input_job_result(&ctx, &result).await; + assert_eq!(std::fs::read(path.join(input_cache::STATUS_FILE)).unwrap(), before); + result.state = InputJobState::Failed; // A genuine technical completion must still replace the prior result. + persist_input_job_result(&ctx, &result).await; + assert_eq!( + input_cache::load_input_status(&path).last_input_update.unwrap().state, + PlaylistUpdateState::Failure + ); + } + + #[tokio::test] + async fn playlist_update_status_completion_covers_every_input_type_without_artificial_clusters() { + let temp = tempfile::tempdir().unwrap(); + let ctx = processing_context(temp.path()); + for (id, input_type) in [ + InputType::Xtream, + InputType::XtreamBatch, + InputType::Stalker, + InputType::StalkerBatch, + InputType::M3u, + InputType::M3uBatch, + InputType::Library, + InputType::Plex, + InputType::Emby, + InputType::Jellyfin, + InputType::Staged, + ] + .into_iter() + .enumerate() + { + let input = ConfigInput { + id: u16::try_from(id).unwrap(), + input_type, + name: format!("input-{id}").intern(), + ..ConfigInput::default() + }; + let result = panicked_input_job(id, &input); + persist_input_job_result(&ctx, &result).await; + let path = input_cache::resolve_input_storage_path(&temp.path().to_string_lossy(), &input.name).await; + let persisted = input_cache::load_input_status(&path); + assert!(persisted.clusters.is_empty()); + assert!( + matches!(persisted.last_input_update, Some(PersistedPlaylistUpdateInputResult {state: PlaylistUpdateState::Failure, timestamp}) if timestamp > 0) + ); + } + } +} diff --git a/backend/processing/src/processor/playlist/m3u_quality.rs b/backend/processing/src/processor/playlist/m3u_quality.rs new file mode 100644 index 000000000..6e663fc18 --- /dev/null +++ b/backend/processing/src/processor/playlist/m3u_quality.rs @@ -0,0 +1,111 @@ +//! M3U's document adapter at the existing locked input acquisition boundary. +use super::ingest::{cluster_is_configured, PIPELINE_TRANSPARENCY_CLUSTERS}; +use shared::model::{M3uPlaylistItem, UpdateQualityPolicy}; +use std::{ + collections::{HashMap, HashSet}, + sync::Arc, +}; +use tuliprox_core::model::{evaluate_update_quality, AppConfig, ClusterForceUpdate, ConfigInput}; +use tuliprox_iptv::provider::PlaylistFetch; +use tuliprox_repository::{get_input_m3u_playlist_file_path, get_input_storage_path, load_input_m3u_playlist}; + +/// Called while the existing per-input lock protects baseline, decision and subsequent persistence. +pub(super) async fn effective_m3u_fetch( + config: &Arc, + input: &ConfigInput, + policy: UpdateQualityPolicy, + mut fetch: PlaylistFetch, +) -> PlaylistFetch { + if !fetch.is_ok() { + return fetch; + } + // Match the existing M3U writer's last-wins primary key population. This also + // keeps the target source and raw group catalogs consistent with its BTree. + let mut winners = HashMap::new(); + for (group_index, group) in fetch.groups.iter_mut().enumerate() { + for item in std::mem::take(&mut group.channels) { + winners.insert(M3uPlaylistItem::from(&item).provider_id, (group_index, item)); + } + } + let mut winners: Vec<_> = winners.into_values().collect(); + winners.sort_by_key(|(group, item)| (*group, item.header.source_ordinal)); + for (group_index, item) in winners { + fetch.groups[group_index].channels.push(item); + } + fetch.groups.retain(|group| !group.channels.is_empty()); + + if policy == UpdateQualityPolicy::Enforce + && input.options.as_ref().is_none_or(|options| { + PIPELINE_TRANSPARENCY_CLUSTERS.into_iter().all(|cluster| options.update_quality.threshold(cluster) == 0) + }) + { + return fetch; + } + + let needs_previous = PIPELINE_TRANSPARENCY_CLUSTERS.into_iter().any(|cluster| { + !cluster_is_configured(input, cluster) + || (policy == UpdateQualityPolicy::Enforce + && input.options.as_ref().is_some_and(|options| options.update_quality.threshold(cluster) > 0)) + }); + let previous = if needs_previous { + let storage_dir = config.config.load().storage_dir.clone(); + let loaded = async { + let storage = get_input_storage_path(&input.name, &storage_dir).await.map_err(|error| { + shared::error::TuliproxError::RepositoryM3u(format!("Cannot read M3U baseline: {error}")) + })?; + load_input_m3u_playlist(config, &get_input_m3u_playlist_file_path(&storage, &input.name)).await + } + .await; + match loaded { + Ok(groups) => groups, + Err(error) => return PlaylistFetch::failed(error), + } + } else { + Vec::new() + }; + let mut retained = Vec::new(); + for cluster in PIPELINE_TRANSPARENCY_CLUSTERS { + if !cluster_is_configured(input, cluster) { + retained.push(cluster); + continue; + } + let threshold = input.options.as_ref().map_or(0, |options| options.update_quality.threshold(cluster)); + let candidate_count = + fetch.groups.iter().filter(|group| group.xtream_cluster == cluster).map(|group| group.channels.len()).sum(); + let current_count = previous + .iter() + .filter(|group| group.xtream_cluster == cluster) + .map(|group| group.channels.len()) + .reduce(|total, count| total + count); + if policy == UpdateQualityPolicy::Bypass { + fetch.force_updates.push(ClusterForceUpdate { cluster, candidate_count, configured_threshold: threshold }); + continue; + } + let decision = evaluate_update_quality(current_count, candidate_count, threshold); + if let Some(rejection) = decision.rejection(cluster) { + fetch.quality_rejections.push(rejection); + retained.push(cluster); + } else if let Some(acceptance) = decision.acceptance(cluster) { + fetch.quality_acceptances.push(acceptance); + } + } + // Quality decisions must not reorder accepted candidate groups. Append retained + // groups in their existing persisted order, without regrouping either source. + fetch.groups.retain(|group| !retained.contains(&group.xtream_cluster)); + fetch.groups.extend(previous.into_iter().filter(|group| retained.contains(&group.xtream_cluster))); + // Retention can combine an old cluster with a newly reclassified primary key. + // The existing document BTree cannot represent both; fail before publication + // instead of losing a retained row or exposing different target/catalog facts. + let mut keys = HashSet::new(); + if fetch + .groups + .iter() + .flat_map(|group| &group.channels) + .any(|item| !keys.insert(M3uPlaylistItem::from(item).provider_id)) + { + return PlaylistFetch::failed(shared::error::TuliproxError::RepositoryM3u( + "Conflicting M3U keys across retained and candidate clusters".to_owned(), + )); + } + fetch +} diff --git a/backend/processing/src/processor/playlist/m3u_quality_tests.rs b/backend/processing/src/processor/playlist/m3u_quality_tests.rs new file mode 100644 index 000000000..05c7ceabd --- /dev/null +++ b/backend/processing/src/processor/playlist/m3u_quality_tests.rs @@ -0,0 +1,520 @@ +mod m3u_update_quality { + use super::*; + use std::fmt::Write; + use tuliprox_repository::{load_input_m3u_playlist, raw_group_catalog_path, RawInputGroupCatalog}; + + fn document(prefix: &str, counts: [usize; 3]) -> String { + let mut text = String::from("#EXTM3U\n"); + for (cluster, count) in XTREAM_CLUSTER.into_iter().zip(counts) { + for index in 0..count { + let (kind, group, title) = match cluster { + XtreamCluster::Live => ("live", format!("{prefix}-live"), format!("Channel {index}")), + XtreamCluster::Video => ("movie", format!("{prefix}-vod"), format!("Movie {index}")), + XtreamCluster::Series => { + ("series", format!("{prefix}-show-{index}"), format!("Show {index} S01E01")) + } + }; + let id = match cluster { + XtreamCluster::Live => 1000, + XtreamCluster::Video => 2000, + XtreamCluster::Series => 3000, + } + index; + writeln!(text, "#EXTGRP:{prefix}-{cluster}\n#EXTINF:-1 xui-id=\"{id}\" tvg-type=\"{kind}\" group-title=\"{group}\",{title}\nhttp://stream.example/{prefix}/{cluster}/{id}").unwrap(); + } + } + text + } + + async fn fixture( + root: &Path, + previous: Option<[usize; 3]>, + thresholds: [u8; 3], + ) -> (PlaylistProcessingContext, Arc) { + let ctx = processing_context(root); + let input = Arc::new(ConfigInput { + id: 27, + name: "m3u-quality".intern(), + input_type: InputType::M3u, + enabled: true, + url: root.join("provider.m3u").to_string_lossy().into_owned(), + options: Some(ConfigInputOptions::from(&ConfigInputOptionsDto { + update_quality: ConfigInputUpdateQualityDto { + live: thresholds[0], + vod: thresholds[1], + series: thresholds[2], + }, + ..ConfigInputOptionsDto::default() + })), + ..ConfigInput::default() + }); + if let Some(counts) = previous { + tokio::fs::write(&input.url, document("previous", counts)).await.unwrap(); + let (groups, errors) = + tuliprox_iptv::m3u::download_m3u_playlist(&ctx.config, &ctx.client, &ctx.config.config.load(), &input) + .await; + assert!(errors.is_empty()); + let (_, error) = tuliprox_repository::persist_input_playlist(&ctx.config, &input, groups).await; + assert!(error.is_none(), "{error:?}"); + } + (ctx, input) + } + + async fn run( + ctx: &PlaylistProcessingContext, + input: &Arc, + counts: [usize; 3], + ) -> InputDownloadResult { + tokio::fs::write(&input.url, document("candidate", counts)).await.unwrap(); + download_input(ctx, input, false).await + } + + fn counts(groups: &[PlaylistGroup]) -> [usize; 3] { + XTREAM_CLUSTER + .map(|cluster| groups.iter().filter(|g| g.xtream_cluster == cluster).map(|g| g.channels.len()).sum()) + } + + async fn verify_effective( + ctx: &PlaylistProcessingContext, + input: &ConfigInput, + result: &mut InputDownloadResult, + expected: [usize; 3], + ) { + assert!(result.errors.is_empty(), "{:?}", result.errors); + assert!(result.storage_error.is_none(), "{:?}", result.storage_error); + assert_eq!(result.job_state(), InputJobState::Ready); + let effective = result.source.take_groups(); + assert_eq!(counts(&effective), expected); + let storage = tuliprox_repository::get_input_storage_path(&input.name, &ctx.config.config.load().storage_dir) + .await + .unwrap(); + let saved = load_input_m3u_playlist( + &ctx.config, + &tuliprox_repository::get_input_m3u_playlist_file_path(&storage, &input.name), + ) + .await + .unwrap(); + assert_eq!(counts(&saved), expected); + for cluster in XTREAM_CLUSTER { + let catalog: RawInputGroupCatalog = + serde_json::from_slice(&tokio::fs::read(raw_group_catalog_path(&storage, cluster)).await.unwrap()) + .unwrap(); + let mut expected_groups: Vec<_> = + effective.iter().filter(|g| g.xtream_cluster == cluster).map(|g| g.title.to_string()).collect(); + expected_groups.sort(); + expected_groups.dedup(); + assert_eq!(catalog.groups, expected_groups); + } + } + + #[tokio::test] + async fn m3u_update_quality_all_clusters_accepted() { + let temp = tempfile::tempdir().unwrap(); + let (ctx, input) = fixture(temp.path(), Some([10, 10, 10]), [90, 90, 90]).await; + let mut result = run(&ctx, &input, [9, 11, 10]).await; + assert!(result.quality_rejections.is_empty()); + assert_eq!(result.update_state(), PlaylistUpdateState::Success); + verify_effective(&ctx, &input, &mut result, [9, 11, 10]).await; + } + + async fn order_candidate( + ctx: &PlaylistProcessingContext, + input: &ConfigInput, + prefix: &str, + ) -> tuliprox_iptv::provider::PlaylistFetch { + let mut body = String::from("#EXTM3U\n"); + for (id, kind, group) in [ + (2000, "movie", "obsolete"), + (2000, "movie", "movies-a"), + (1000, "live", "live-a"), + (2001, "movie", "movies-b"), + (1001, "live", "live-b"), + (3000, "series", "shows-a"), + (3001, "series", "shows-b"), + ] { + writeln!(body, "#EXTGRP:{prefix}-{group}\n#EXTINF:-1 tvg-type=\"{kind}\" group-title=\"{prefix}-{group}\",Item {id} S01E01\nhttp://stream.example/{kind}/{id}").unwrap(); + } + tokio::fs::write(&input.url, body).await.unwrap(); + let (groups, errors) = + tuliprox_iptv::m3u::download_m3u_playlist(&ctx.config, &ctx.client, &ctx.config.config.load(), input).await; + assert!(errors.is_empty()); + assert_eq!( + group_names(&groups), + ["obsolete", "movies-a", "live-a", "movies-b", "live-b", "shows-a", "shows-b"] + .map(|name| format!("{prefix}-{name}")) + ); + tuliprox_iptv::provider::PlaylistFetch::groups(groups) + } + + fn group_names(groups: &[PlaylistGroup]) -> Vec { + groups.iter().map(|group| group.title.to_string()).collect() + } + + #[tokio::test] + async fn m3u_update_quality_preserves_candidate_group_order_when_disabled() { + let temp = tempfile::tempdir().unwrap(); + let (ctx, input) = fixture(temp.path(), None, [0, 0, 0]).await; + let candidate = order_candidate(&ctx, &input, "candidate").await; + let effective = super::super::super::m3u_quality::effective_m3u_fetch( + &ctx.config, + &input, + shared::model::UpdateQualityPolicy::Enforce, + candidate, + ) + .await; + assert!(effective.errors.is_empty()); + assert!(effective.quality_acceptances.is_empty() && effective.quality_rejections.is_empty()); + assert_eq!( + group_names(&effective.groups), + ["movies-a", "live-a", "movies-b", "live-b", "shows-a", "shows-b"] + .map(|name| format!("candidate-{name}")) + ); + assert_eq!(counts(&effective.groups), [2, 2, 2], "disabled quality processing must collapse duplicate keys"); + } + + #[tokio::test] + async fn m3u_update_quality_preserves_candidate_group_order_when_all_clusters_accepted() { + let temp = tempfile::tempdir().unwrap(); + let (ctx, input) = fixture(temp.path(), Some([2, 2, 2]), [100, 100, 100]).await; + let candidate = order_candidate(&ctx, &input, "candidate").await; + let effective = super::super::super::m3u_quality::effective_m3u_fetch( + &ctx.config, + &input, + shared::model::UpdateQualityPolicy::Enforce, + candidate, + ) + .await; + assert!(effective.errors.is_empty() && effective.quality_rejections.is_empty()); + assert_eq!(effective.quality_acceptances.len(), 3); + assert_eq!( + group_names(&effective.groups), + ["movies-a", "live-a", "movies-b", "live-b", "shows-a", "shows-b"].map(|name| format!("candidate-{name}")) + ); + } + + #[tokio::test] + async fn m3u_update_quality_mixed_fallback_preserves_relative_group_order() { + for _ in 0..3 { + let temp = tempfile::tempdir().unwrap(); + let (ctx, input) = fixture(temp.path(), None, [100, 100, 100]).await; + let mut previous = order_candidate(&ctx, &input, "previous").await.groups; + previous.retain(|group| group.title.as_ref() != "previous-obsolete"); + let (_, error) = tuliprox_repository::persist_input_playlist(&ctx.config, &input, previous).await; + assert!(error.is_none()); + let mut previous = load_input_playlist(&ctx.config, &input, None).await.unwrap(); + let previous = previous.take_groups(); + assert_eq!( + previous + .iter() + .filter(|group| group.xtream_cluster == XtreamCluster::Series) + .map(|group| group.title.to_string()) + .collect::>(), + ["previous-shows-a", "previous-shows-b"] + ); + let mut candidate = order_candidate(&ctx, &input, "candidate").await; + candidate.groups.retain(|group| group.title.as_ref() != "candidate-shows-b"); + let effective = super::super::super::m3u_quality::effective_m3u_fetch( + &ctx.config, + &input, + shared::model::UpdateQualityPolicy::Enforce, + candidate, + ) + .await; + assert!(effective.errors.is_empty()); + assert_eq!(effective.quality_rejections.len(), 1); + assert_eq!(effective.quality_rejections[0].cluster, XtreamCluster::Series); + assert_eq!( + group_names(&effective.groups), + [ + "candidate-movies-a", + "candidate-live-a", + "candidate-movies-b", + "candidate-live-b", + "previous-shows-a", + "previous-shows-b" + ] + ); + } + } + + #[tokio::test] + async fn m3u_update_quality_retains_only_rejected_cluster() { + for disk in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let (ctx, input) = fixture(temp.path(), Some([10, 10, 10]), [90, 90, 90]).await; + let mut config = (**ctx.config.config.load()).clone(); + config.disk_based_processing = disk; + ctx.config.config.store(Arc::new(config)); + let mut result = run(&ctx, &input, [9, 11, 8]).await; + assert_eq!(result.update_state(), PlaylistUpdateState::Partial); + assert_eq!(result.quality_rejections.len(), 1); + assert_eq!(result.quality_rejections[0].cluster, XtreamCluster::Series); + verify_effective(&ctx, &input, &mut result, [9, 11, 10]).await; + } + } + + #[tokio::test] + async fn m3u_update_quality_zero_candidate_retains_previous_cluster() { + let temp = tempfile::tempdir().unwrap(); + let (ctx, input) = fixture(temp.path(), Some([10, 10, 10]), [90, 90, 90]).await; + let mut result = run(&ctx, &input, [10, 10, 0]).await; + assert_eq!(result.quality_rejections[0].candidate_count, 0); + verify_effective(&ctx, &input, &mut result, [10, 10, 10]).await; + } + + #[tokio::test] + async fn m3u_update_quality_disabled() { + let temp = tempfile::tempdir().unwrap(); + let (ctx, input) = fixture(temp.path(), Some([10, 10, 10]), [0, 0, 0]).await; + let mut result = run(&ctx, &input, [1, 1, 0]).await; + assert!(result.quality_rejections.is_empty()); + assert!(result + .input_telemetry + .as_ref() + .unwrap() + .clusters + .iter() + .all(|c| c.quality.is_none() && c.threshold.is_none())); + verify_effective(&ctx, &input, &mut result, [1, 1, 0]).await; + } + + #[tokio::test] + async fn m3u_update_quality_first_import() { + let temp = tempfile::tempdir().unwrap(); + let (ctx, input) = fixture(temp.path(), None, [90, 90, 90]).await; + let mut result = run(&ctx, &input, [1, 1, 1]).await; + assert!(result.quality_rejections.is_empty()); + assert!(result.input_telemetry.as_ref().unwrap().clusters.iter().all(|c| c.quality.is_none() + && c.baseline_count.is_none() + && c.decision == Some(PlaylistUpdateClusterDecision::Accepted))); + verify_effective(&ctx, &input, &mut result, [1, 1, 1]).await; + } + + #[tokio::test] + async fn m3u_update_quality_download_or_parse_failure() { + for body in [None, Some("provider unavailable"), Some("#EXTM3U\n#EXTINF:-1,Truncated\n")] { + let temp = tempfile::tempdir().unwrap(); + let (ctx, input) = fixture(temp.path(), Some([10, 10, 10]), [90, 90, 90]).await; + if let Some(body) = body { + tokio::fs::write(&input.url, body).await.unwrap(); + } else { + tokio::fs::remove_file(&input.url).await.unwrap(); + } + let mut result = download_input(&ctx, &input, false).await; + assert!(!result.errors.is_empty()); + assert!(result.quality_rejections.is_empty()); + assert_eq!(result.update_state(), PlaylistUpdateState::Failure); + let mut saved = load_input_playlist(&ctx.config, &input, None).await.unwrap(); + assert_eq!(counts(&saved.take_groups()), [10, 10, 10]); + } + } + + #[tokio::test] + async fn m3u_update_quality_empty_bootstrap_and_structural_empty_use_existing_domain() { + for previous in [None, Some([10, 10, 0])] { + let temp = tempfile::tempdir().unwrap(); + let (ctx, input) = fixture(temp.path(), previous, [90, 90, 90]).await; + let mut result = run(&ctx, &input, if previous.is_some() { [10, 10, 0] } else { [0, 0, 0] }).await; + if previous.is_none() { + assert_eq!(result.update_state(), PlaylistUpdateState::Failure); + assert_eq!(result.quality_rejections.len(), 3); + } else { + assert_eq!(result.update_state(), PlaylistUpdateState::Partial); + assert_eq!(result.quality_rejections.len(), 1); + assert_eq!(result.quality_rejections[0].current_count, 0); + assert_eq!(result.quality_rejections[0].cluster, XtreamCluster::Series); + } + } + } + + #[tokio::test] + async fn m3u_update_quality_skip_preserves_previous_and_reuse_emits_no_new_facts() { + let temp = tempfile::tempdir().unwrap(); + let (ctx, mut input) = fixture(temp.path(), Some([10, 10, 10]), [90, 90, 90]).await; + Arc::make_mut(&mut input).options.as_mut().unwrap().flags.set(ConfigInputFlags::SkipSeries); + let mut result = run(&ctx, &input, [10, 10, 0]).await; + assert_eq!(result.update_state(), PlaylistUpdateState::Success); + assert!(result.quality_rejections.is_empty()); + let mut saved = load_input_playlist(&ctx.config, &input, None).await.unwrap(); + assert_eq!(counts(&result.source.take_groups()), [10, 10, 0]); + assert_eq!(counts(&saved.take_groups()), [10, 10, 10]); + let telemetry = result.input_telemetry.unwrap(); + assert!(!telemetry.clusters.iter().find(|c| c.cluster == XtreamCluster::Series).unwrap().requested); + tokio::fs::remove_file(&input.url).await.unwrap(); + let reused = download_input(&ctx, &input, false).await; + assert!(reused.errors.is_empty()); + assert!(reused.input_telemetry.is_none()); + } + + #[tokio::test] + async fn m3u_update_quality_force_bypasses_guard_and_cache_without_inventing_quality() { + for threshold in [0, 95] { + for policy in [InputRefreshPolicy::REFRESH, InputRefreshPolicy::FORCE] { + let temp = tempfile::tempdir().unwrap(); + let (mut ctx, mut input) = fixture(temp.path(), Some([10, 10, 10]), [threshold; 3]).await; + Arc::make_mut(&mut input).cache_duration_seconds = 3_600; + let storage = input_storage_path(&ctx, &input).await; + let mut status = input_cache::InputStatus::default(); + input_cache::update_cluster_status(&mut status, "default", input_cache::ClusterState::Ok); + input_cache::save_input_status(&storage, &status); + set_refresh_policy(&mut ctx, &input, policy); + let mut result = run(&ctx, &input, [1, 1, 1]).await; + let rejected = threshold > 0 && policy == InputRefreshPolicy::REFRESH; + assert_eq!( + result.update_state(), + if rejected { PlaylistUpdateState::Partial } else { PlaylistUpdateState::Success } + ); + assert_eq!(counts(&result.source.take_groups()), if rejected { [10; 3] } else { [1; 3] }); + let telemetry = result.input_telemetry.as_ref().unwrap(); + assert_eq!(telemetry.refresh_policy, policy); + assert!(telemetry + .clusters + .iter() + .all(|cluster| cluster.source == Some(PlaylistUpdateDataSource::Provider))); + let saved = input_cache::load_input_status(&storage); + for cluster in &telemetry.clusters { + if policy == InputRefreshPolicy::FORCE { + assert_eq!(cluster.decision, Some(PlaylistUpdateClusterDecision::Accepted)); + assert_eq!((cluster.baseline_count, cluster.quality), (None, None)); + assert_eq!((cluster.candidate_count, cluster.active_count), (Some(1), Some(1))); + let snapshot = saved.clusters[cluster.cluster.as_ref()].last_update.unwrap(); + assert_eq!(snapshot.quality, None); + assert_eq!(snapshot.quality_guard_threshold, Some(threshold)); + assert_eq!(snapshot.policy, Some(policy)); + } + } + let reused = download_input(&ctx, &input, false).await; + assert!(reused.input_telemetry.is_none(), "in-run reuse must not create another Force acquisition"); + } + } + } + + #[tokio::test] + async fn m3u_update_quality_force_keeps_parse_and_empty_publication_errors() { + for body in ["#EXTM3U\n", "#EXTM3U\n#EXTINF:-1,Truncated\n"] { + let temp = tempfile::tempdir().unwrap(); + let (mut ctx, input) = fixture(temp.path(), Some([10; 3]), [95; 3]).await; + set_refresh_policy(&mut ctx, &input, InputRefreshPolicy::FORCE); + tokio::fs::write(&input.url, body).await.unwrap(); + let mut result = download_input(&ctx, &input, false).await; + assert_eq!(result.update_state(), PlaylistUpdateState::Failure); + assert!(result.quality_rejections.is_empty()); + let mut saved = load_input_playlist(&ctx.config, &input, None).await.unwrap(); + assert_eq!(counts(&saved.take_groups()), [10; 3]); + } + } + + #[tokio::test] + async fn m3u_update_quality_target_processing_uses_retained_and_accepted_groups() { + for policy in [InputRefreshPolicy::NORMAL, InputRefreshPolicy::FORCE] { + let temp = tempfile::tempdir().unwrap(); + let (mut ctx, input) = fixture(temp.path(), Some([10, 10, 10]), [90, 90, 90]).await; + set_refresh_policy(&mut ctx, &input, policy); + tokio::fs::write(&input.url, document("candidate", [10, 0, 10])).await.unwrap(); + let output = temp.path().join("client.m3u"); + let mut target = ConfigTarget::from(&ConfigTargetDto { + id: 1, + name: "m3u-target".into(), + enabled: true, + ..ConfigTargetDto::default() + }); + target.filter = get_filter(r#"Name ~ ".*""#, None).unwrap().into(); + target.output = vec![TargetOutput::M3u(tuliprox_core::model::M3uTargetOutput { + filename: Some(output.to_string_lossy().into_owned()), + include_type_in_url: false, + mask_redirect_url: false, + filter: None, + })]; + ctx.config.sources.store(Arc::new(SourcesConfig { + inputs: vec![input.clone()], + sources: vec![ConfigSource { inputs: vec![input.name.clone()], targets: vec![Arc::new(target)] }], + ..SourcesConfig::default() + })); + let (_, targets, errors) = process_source(0, Arc::new(ctx)).await; + assert!(errors.is_empty(), "{errors:?}"); + assert_eq!(targets.len(), 1); + let published = tokio::fs::read_to_string(output).await.unwrap(); + assert!(published.contains("/candidate/live/") && published.contains("/candidate/series/"), "{published}"); + assert_eq!(published.contains("/previous/video/"), policy == InputRefreshPolicy::NORMAL, "{published}"); + assert!(!published.contains("/previous/live/") && !published.contains("/candidate/video/")); + } + } + + #[tokio::test] + async fn m3u_update_quality_raw_group_catalog_uses_effective_state() { + let temp = tempfile::tempdir().unwrap(); + let (ctx, input) = fixture(temp.path(), Some([10, 10, 10]), [90, 90, 90]).await; + let mut result = run(&ctx, &input, [10, 10, 0]).await; + let storage = input_cache::resolve_input_storage_path(&ctx.config.config.load().storage_dir, &input.name).await; + let status = input_cache::load_input_status(&storage); + for cluster in XTREAM_CLUSTER { + let saved = &status.clusters[cluster.as_ref()]; + let rejected = cluster == XtreamCluster::Series; + assert_eq!( + saved.status, + if rejected { input_cache::ClusterState::Failed } else { input_cache::ClusterState::Ok } + ); + let snapshot = saved.last_update.unwrap(); + assert_eq!( + snapshot.quality.unwrap().decision, + if rejected { + PersistedPlaylistUpdateQualityDecision::Rejected + } else { + PersistedPlaylistUpdateQualityDecision::Accepted + } + ); + assert_eq!(snapshot.active_count, Some(10)); + } + assert_eq!(status.clusters["default"].status, input_cache::ClusterState::Failed); + let series: RawInputGroupCatalog = serde_json::from_slice( + &tokio::fs::read(raw_group_catalog_path(&storage, XtreamCluster::Series)).await.unwrap(), + ) + .unwrap(); + assert_eq!(series.groups, ["previous-series"]); + verify_effective(&ctx, &input, &mut result, [10, 10, 10]).await; + } + + #[tokio::test] + async fn m3u_update_quality_duplicate_keys_match_the_existing_persisted_population() { + let temp = tempfile::tempdir().unwrap(); + let (ctx, input) = fixture(temp.path(), Some([1, 1, 0]), [100, 100, 0]).await; + let candidate = document("superseded", [1, 1, 0]) + &document("last", [1, 1, 0]); + tokio::fs::write(&input.url, candidate).await.unwrap(); + let mut result = download_input(&ctx, &input, false).await; + assert_eq!(result.update_state(), PlaylistUpdateState::Success); + assert!(result.source.take_groups().iter().all(|group| group.title.starts_with("last"))); + let mut saved = load_input_playlist(&ctx.config, &input, None).await.unwrap(); + assert_eq!(counts(&saved.take_groups()), [1, 1, 0]); + } + + #[tokio::test] + async fn m3u_update_quality_duplicate_keys_match_the_existing_persisted_population_when_disabled() { + let temp = tempfile::tempdir().unwrap(); + let (ctx, input) = fixture(temp.path(), None, [0, 0, 0]).await; + let candidate = document("superseded", [1, 1, 0]) + &document("last", [1, 1, 0]); + tokio::fs::write(&input.url, candidate).await.unwrap(); + let mut result = download_input(&ctx, &input, false).await; + assert_eq!(result.update_state(), PlaylistUpdateState::Success); + assert!(result.source.take_groups().iter().all(|group| group.title.starts_with("last"))); + let mut saved = load_input_playlist(&ctx.config, &input, None).await.unwrap(); + assert_eq!(counts(&saved.take_groups()), [1, 1, 0]); + } + + #[tokio::test] + async fn m3u_update_quality_conflicting_retained_key_cannot_diverge_from_storage() { + let temp = tempfile::tempdir().unwrap(); + let (ctx, input) = fixture(temp.path(), Some([1, 1, 0]), [90, 0, 0]).await; + let candidate = document("candidate", [0, 1, 0]).replace("2000", "1000"); + tokio::fs::write(&input.url, candidate).await.unwrap(); + let mut result = download_input(&ctx, &input, false).await; + assert_eq!(result.update_state(), PlaylistUpdateState::Failure); + assert!(!result.errors.is_empty()); + let mut saved = load_input_playlist(&ctx.config, &input, None).await.unwrap(); + assert_eq!(counts(&saved.take_groups()), [1, 1, 0]); + let storage = input_storage_path(&ctx, &input).await; + let catalog: RawInputGroupCatalog = serde_json::from_slice( + &tokio::fs::read(raw_group_catalog_path(&storage, XtreamCluster::Live)).await.unwrap(), + ) + .unwrap(); + assert_eq!(catalog.groups, ["previous-live"]); + } +} diff --git a/backend/processing/src/processor/playlist/manual_update.rs b/backend/processing/src/processor/playlist/manual_update.rs new file mode 100644 index 000000000..ae45a54b4 --- /dev/null +++ b/backend/processing/src/processor/playlist/manual_update.rs @@ -0,0 +1,198 @@ +use super::{process_sources, MetadataUpdateSink, PlaylistProcessingContext}; +use shared::{ + error::TuliproxError, + model::{ + EventMessage, EventSink, LibraryScanResult, PlaylistUpdateProgressDetail, PlaylistUpdateProgressEvent, + PlaylistUpdateState, SourceStats, + }, +}; +use std::{future::Future, io}; +use tuliprox_core::model::UpdateGuard; +use tuliprox_library::library::LibraryProcessor; + +/// Bulk and scheduled runs read the existing catalog. Only an explicit card request scans. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub(crate) enum LibraryUpdateMode { + #[default] + ExistingCatalog, + Rescan { + input_id: u16, + }, +} + +impl LibraryUpdateMode { + pub(crate) fn reloads_input(self, input_id: u16) -> bool { self == Self::Rescan { input_id } } +} + +/// The rebuild is not even started until the existing scanner reports complete success. +async fn rebuild_after_scan( + scan: impl Future>, + report: impl FnOnce(LibraryScanResult), + rebuild: impl FnOnce() -> T, +) -> io::Result { + let result = scan.await?; + let complete = result.errors == 0; + report(result); + if !complete { + return Err(io::Error::other("Library rescan reported incomplete input data")); + } + Ok(rebuild()) +} + +pub(super) async fn process_manual_update( + ctx: &PlaylistProcessingContext, + update_guard: Option<&UpdateGuard>, +) -> (Vec, Vec) { + let LibraryUpdateMode::Rescan { input_id } = ctx.library_update_mode else { + return process_sources(ctx).await; + }; + let progress = |message: &str, detail| { + ctx.events.emit(EventMessage::PlaylistUpdateProgress( + PlaylistUpdateProgressEvent::for_run_input( + ctx.run_id.clone(), + ctx.execution_order, + input_id, + "Library rescan", + message, + ) + .with_detail(detail), + )); + }; + progress( + "Library rescan started; selected target rebuild is waiting", + PlaylistUpdateProgressDetail::LibraryRescanStarted, + ); + let result = async { + // The worker already owns the playlist permit. Never wait for a second lock: + // a concurrent standalone scan makes this request fail, without a lock cycle. + let _library_permit = update_guard + .and_then(UpdateGuard::try_library) + .ok_or_else(|| io::Error::other("Library scan is already running or its guard is unavailable"))?; + let (library, metadata, storage_dir) = { + let config = ctx.config.config.load(); + let library = config + .library + .as_ref() + .filter(|library| library.enabled) + .ok_or_else(|| io::Error::other("Library is not enabled"))? + .clone(); + (library, config.metadata_update.clone(), config.storage_dir.clone()) + }; + let processor = LibraryProcessor::new(library, metadata.as_ref(), ctx.client.clone(), &storage_dir); + let rebuild = rebuild_after_scan( + processor.scan_for_target_rebuild(), + |result| { + if result.errors > 0 { + log::error!("Library rescan failed for input {input_id}: incomplete scan; {result:?}"); + } + ctx.events.emit(EventMessage::PlaylistUpdateProgress( + PlaylistUpdateProgressEvent::for_run_input( + ctx.run_id.clone(), + ctx.execution_order, + input_id, + "Library rescan", + "Library scan result", + ) + .with_library_scan_result(result) + .with_detail(PlaylistUpdateProgressDetail::LibraryScanResult), + )); + }, + || { + progress( + "Library rescan completed; loading input data and rebuilding selected targets", + PlaylistUpdateProgressDetail::LibraryRescanCompleted, + ); + process_sources(ctx) + }, + ) + .await?; + // Retain the existing library permit until the catalog-dependent rebuild finishes. + Ok::<_, io::Error>(rebuild.await) + } + .await; + match result { + Ok(result) => result, + Err(error) => { + log::error!("Library rescan failed for input {input_id}: {error}"); + if let Some(input) = ctx.config.get_input_by_id(input_id) { + super::input_status::persist_input_completion(ctx, input.id, &input.name, PlaylistUpdateState::Failure) + .await; + } + ctx.events.emit(EventMessage::PlaylistUpdateProgress( + PlaylistUpdateProgressEvent::input_completed( + ctx.run_id.clone(), + ctx.execution_order, + input_id, + PlaylistUpdateState::Failure, + "Library rescan", + "Library rescan failed; selected targets were not rebuilt", + ) + .with_detail(PlaylistUpdateProgressDetail::LibraryRescanFailed), + )); + (Vec::new(), vec![TuliproxError::RepositoryLibrary(format!("Library rescan failed: {error}"))]) + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::cell::Cell; + + fn scan_result(errors: usize) -> LibraryScanResult { + LibraryScanResult { + files_scanned: 0, + groups_scanned: 0, + files_added: 0, + files_updated: 0, + files_removed: 0, + errors, + } + } + + #[tokio::test] + async fn manual_update_library_rebuild_waits_for_successful_scan() { + let completed = Cell::new(false); + let scan = async { + tokio::task::yield_now().await; + completed.set(true); + Ok(scan_result(0)) + }; + let rebuilt = rebuild_after_scan( + scan, + |_| {}, + || { + assert!(completed.get()); + "common target pipeline" + }, + ) + .await; + assert_eq!(rebuilt.unwrap(), "common target pipeline"); + } + + #[tokio::test] + async fn manual_update_library_errors_and_partial_scan_never_start_target_rebuild() { + for result in [Err(io::Error::other("scan failed")), Ok(LibraryScanResult { errors: 1, ..scan_result(0) })] { + let rebuilt = Cell::new(false); + let reported = std::cell::RefCell::new(None); + let expected = result.as_ref().ok().cloned(); + assert!(rebuild_after_scan( + async { result }, + |scan| *reported.borrow_mut() = Some(scan), + || rebuilt.set(true) + ) + .await + .is_err()); + assert_eq!(*reported.borrow(), expected); + assert!(!rebuilt.get()); + } + } + + #[test] + fn manual_update_library_bulk_reuses_catalog_without_scan_or_policy_override() { + assert_eq!(LibraryUpdateMode::default(), LibraryUpdateMode::ExistingCatalog); + assert!(!LibraryUpdateMode::ExistingCatalog.reloads_input(2)); + assert!(LibraryUpdateMode::Rescan { input_id: 2 }.reloads_input(2)); + assert!(!LibraryUpdateMode::Rescan { input_id: 2 }.reloads_input(3)); + } +} diff --git a/backend/processing/src/processor/playlist/manual_update_integration_tests.rs b/backend/processing/src/processor/playlist/manual_update_integration_tests.rs new file mode 100644 index 000000000..87cc1672e --- /dev/null +++ b/backend/processing/src/processor/playlist/manual_update_integration_tests.rs @@ -0,0 +1,725 @@ +mod manual_update_integration { + use super::*; + use shared::model::{ConfigRenameDto, ItemField, LibraryConfigDto, LibraryContentType, LibraryScanResult}; + use tuliprox_core::model::{LibraryConfig, LibraryScanDirectory, M3uTargetOutput, MetadataUpdateConfig}; + + async fn library_context(root: &Path, order: ProcessingOrder) -> PlaylistProcessingContext { + let ctx = processing_context_with_events(root, CollectSink::default()); + let media = root.join("media"); + tokio::fs::create_dir_all(&media).await.unwrap(); + tokio::fs::write(media.join("Library Movie (2025).mp4"), b"fixture").await.unwrap(); + let companion_path = root.join("companion.m3u"); + tokio::fs::write(&companion_path, "#EXTM3U\n#EXTINF:-1,Companion\nhttp://stream.example/companion\n") + .await + .unwrap(); + let mut config = (**ctx.config.config.load()).clone(); + let mut library = LibraryConfig::from(&LibraryConfigDto { enabled: true, ..LibraryConfigDto::default() }); + library.metadata.fallback_to_filename = true; + library.thumbnails.enabled = false; + library.scan_directories = vec![LibraryScanDirectory { + enabled: true, + path: media.to_string_lossy().into_owned(), + content_type: LibraryContentType::Movie, + recursive: true, + }]; + config.library = Some(library); + let mut metadata = MetadataUpdateConfig::default(); + metadata.ffprobe.enabled = false; + metadata.tmdb.enabled = false; + config.metadata_update = Some(metadata); + ctx.config.config.store(Arc::new(config)); + let library_input = Arc::new(ConfigInput { + id: 2, + name: "local".intern(), + input_type: InputType::Library, + enabled: true, + cache_duration_seconds: 3600, + ..ConfigInput::default() + }); + let companion = Arc::new(ConfigInput { + id: 3, + name: "companion".intern(), + input_type: InputType::M3u, + enabled: true, + url: companion_path.to_string_lossy().into_owned(), + ..ConfigInput::default() + }); + let mut target = ConfigTarget::from(&ConfigTargetDto { + id: 20, + enabled: true, + name: "selected".into(), + processing_order: order, + rename: Some(vec![ConfigRenameDto { + field: ItemField::Name, + pattern: "^MAPPED$".into(), + new_name: "RENAMED".into(), + t_pattern: None, + }]), + ..ConfigTargetDto::default() + }); + target.output = vec![TargetOutput::M3u(M3uTargetOutput { + filename: Some(root.join("selected.m3u").to_string_lossy().into_owned()), + include_type_in_url: false, + mask_redirect_url: false, + filter: None, + })]; + target.filter = get_filter(r#"Name ~ ".*""#, None).unwrap().into(); + let mapped = super::super::mapping_stage::build_target( + vec![super::super::mapping_stage::build_mapping("manual", MappingStage::Processing, r#"@Name = "MAPPED""#)], + false, + ); + target.mapping = mapped.mapping; + let mut unselected = target.clone(); + unselected.id = 21; + unselected.name = "unselected".into(); + unselected.output = vec![TargetOutput::M3u(M3uTargetOutput { + filename: Some(root.join("unselected.m3u").to_string_lossy().into_owned()), + include_type_in_url: false, + mask_redirect_url: false, + filter: None, + })]; + ctx.config.sources.store(Arc::new(SourcesConfig { + inputs: vec![library_input.clone(), companion.clone()], + sources: vec![ConfigSource { + inputs: vec![library_input.name.clone(), companion.name.clone()], + targets: vec![Arc::new(target), Arc::new(unselected)], + }], + ..SourcesConfig::default() + })); + ctx + } + + fn rescan_run(ctx: &PlaylistProcessingContext, guard: UpdateGuard) -> ProcessingRun { + ProcessingRun::for_run( + "manual-library-run".into(), + ctx.client.clone(), + ctx.config.clone(), + Arc::new(ProcessTargets { + enabled: true, + inputs: vec![2, 3], + targets: vec![20], + target_names: vec!["selected".into()], + }), + ctx.events.clone(), + ) + .with_update_guard(guard) + .with_manual_input_update(Some(InputUpdateRequest { input_id: 2, action: InputUpdateAction::Rescan })) + } + + fn scan_results(ctx: &PlaylistProcessingContext) -> Vec { + ctx.events + .0 + .lock() + .unwrap() + .iter() + .filter_map(|event| match event { + EventMessage::PlaylistUpdateProgress(progress) if progress.input_id == Some(2) => { + if progress.library_scan_result.is_some() { + assert_eq!( + progress.run_id.as_ref().map(PlaylistUpdateRunId::as_ref), + Some("manual-library-run") + ); + assert!(progress.execution_order.is_some()); + } + progress.library_scan_result.clone() + } + _ => None, + }) + .collect() + } + + fn assert_run_state(ctx: &PlaylistProcessingContext, state: PlaylistUpdateState) { + let events = ctx.events.0.lock().unwrap(); + assert!( + events.iter().any(|event| matches!(event, EventMessage::PlaylistUpdate(summary) if summary.state == state)), + "{events:?}" + ); + assert!(events.iter().any(|event| matches!(event, EventMessage::PlaylistUpdateProgress(progress) if progress.input_id == Some(2) && progress.state == Some(state))), "{events:?}"); + } + + #[tokio::test] + async fn manual_update_library_empty_rescan_persists_zero_to_zero_and_nonzero_to_zero_with_companion() { + for disk in [false, true] { + for previous_population in [false, true] { + for companion in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let ctx = library_context(temp.path(), ProcessingOrder::Frm).await; + let mut config = (**ctx.config.config.load()).clone(); + config.disk_based_processing = disk; + ctx.config.config.store(Arc::new(config)); + if !companion { + let mut sources = (**ctx.config.sources.load()).clone(); + sources.sources[0].inputs.retain(|name| name.as_ref() == "local"); + ctx.config.sources.store(Arc::new(sources)); + } + if previous_population { + exec_processing(rescan_run(&ctx, UpdateGuard::new())).await; + assert_run_state(&ctx, PlaylistUpdateState::Success); + ctx.events.0.lock().unwrap().clear(); + } + tokio::fs::remove_file(temp.path().join("media/Library Movie (2025).mp4")).await.unwrap(); + exec_processing(rescan_run(&ctx, UpdateGuard::new())).await; + assert_run_state(&ctx, PlaylistUpdateState::Success); + for name in std::iter::once("local").chain(companion.then_some("companion")) { + let path = input_cache::resolve_input_storage_path(&temp.path().to_string_lossy(), name).await; + let status = input_cache::load_input_status(&path); + assert_eq!(status.last_input_update.unwrap().state, PlaylistUpdateState::Success, "{name}"); + } + assert_eq!( + scan_results(&ctx), + vec![LibraryScanResult { + files_scanned: 0, + groups_scanned: 0, + files_added: 0, + files_updated: 0, + files_removed: usize::from(previous_population), + errors: 0, + }] + ); + let output = tokio::fs::read_to_string(temp.path().join("selected.m3u")).await.unwrap(); + assert_eq!(output.matches("#EXTINF").count(), usize::from(companion), "{output}"); + assert_eq!(output.contains("http://stream.example/companion"), companion); + assert!(!output.contains("Library%20Movie") && !output.contains("file://")); + assert!(!temp.path().join("unselected.m3u").exists()); + let input = ctx.config.sources.load().inputs[0].clone(); + let mut persisted = load_input_playlist(&ctx.config, &input, None).await.unwrap(); + assert!(persisted.is_empty(), "Empty Library must replace its old persisted input"); + assert!(!ctx.events.0.lock().unwrap().iter().any(|event| matches!(event, EventMessage::PlaylistUpdateProgress(progress) if progress.message.contains("playlist is empty")))); + // Same canonical empty input remains readable on subsequent cache/reuse paths. + let first = download_input(&ctx, &input, false).await; + assert!(first.errors.is_empty() && first.storage_error.is_none()); + let mut reuse = download_input(&ctx, &input, false).await; + assert_eq!(reuse.job_state(), InputJobState::Ready); + assert_eq!(reuse.input_telemetry, None, "In-run reuse must not emit another acquisition"); + assert_eq!(scan_results(&ctx).len(), 1, "Reuse must not manufacture a second rescan"); + } + } + } + } + + #[tokio::test] + async fn coderabbit_library_unavailable_is_not_an_authoritative_empty_catalog() { + for disabled in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let ctx = library_context(temp.path(), ProcessingOrder::Frm).await; + exec_processing(rescan_run(&ctx, UpdateGuard::new())).await; + assert_run_state(&ctx, PlaylistUpdateState::Success); + let mut input = (*ctx.config.sources.load().inputs[0]).clone(); + input.cache_duration_seconds = 0; + let input = Arc::new(input); + let storage = input_storage_path(&ctx, &input).await; + let file = tuliprox_repository::get_input_local_library_playlist_file_path(&storage, &input.name); + let before = tokio::fs::read(&file).await.unwrap(); + let output = tokio::fs::read(temp.path().join("selected.m3u")).await.unwrap(); + let mut config = (**ctx.config.config.load()).clone(); + if disabled { + config.library.as_mut().unwrap().enabled = false; + } else { + config.library = None; + } + ctx.config.config.store(Arc::new(config)); + + let mut result = download_input(&ctx, &input, false).await; + assert_eq!(result.job_state(), InputJobState::Failed); + assert_eq!(result.update_state(), PlaylistUpdateState::Failure); + assert!(!result.authoritative_library); + assert!(!result.errors.is_empty()); + assert_eq!(tokio::fs::read(&file).await.unwrap(), before); + assert_eq!(tokio::fs::read(temp.path().join("selected.m3u")).await.unwrap(), output); + } + } + + #[tokio::test] + async fn coderabbit_library_missing_input_triggers_catalog_recovery_not_empty_success() { + for disk in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let ctx = library_context(temp.path(), ProcessingOrder::Frm).await; + let mut config = (**ctx.config.config.load()).clone(); + config.disk_based_processing = disk; + ctx.config.config.store(Arc::new(config)); + exec_processing(rescan_run(&ctx, UpdateGuard::new())).await; + assert_run_state(&ctx, PlaylistUpdateState::Success); + let input = ctx.config.sources.load().inputs[0].clone(); + let storage = input_storage_path(&ctx, &input).await; + let file = tuliprox_repository::get_input_local_library_playlist_file_path(&storage, &input.name); + tokio::fs::remove_file(&file).await.unwrap(); + + assert!(load_input_playlist(&ctx.config, &input, None).await.is_err()); + let mut recovered = download_input(&ctx, &input, false).await; + assert_eq!(recovered.job_state(), InputJobState::Ready); + assert!(recovered.errors.is_empty() && recovered.storage_error.is_none()); + assert!(!recovered.source.is_empty(), "missing cache must be repaired from the populated catalog"); + assert!(file.exists()); + let mut persisted = load_input_playlist(&ctx.config, &input, None).await.unwrap(); + assert!(!persisted.is_empty()); + + let media_server = ConfigInput { input_type: InputType::Plex, ..(*input).clone() }; + assert!( + load_input_playlist(&ctx.config, &media_server, None).await.is_err(), + "the shared loader must not turn a missing Plex cache into a successful empty input" + ); + } + } + + #[tokio::test] + async fn manual_update_library_scan_errors_keep_real_result_and_block_rebuild() { + let temp = tempfile::tempdir().unwrap(); + let ctx = library_context(temp.path(), ProcessingOrder::Frm).await; + let mut config = (**ctx.config.config.load()).clone(); + config.library.as_mut().unwrap().metadata.fallback_to_filename = false; + ctx.config.config.store(Arc::new(config)); + exec_processing(rescan_run(&ctx, UpdateGuard::new())).await; + assert_run_state(&ctx, PlaylistUpdateState::Failure); + let path = input_cache::resolve_input_storage_path(&temp.path().to_string_lossy(), "local").await; + assert_eq!( + input_cache::load_input_status(&path).last_input_update.unwrap().state, + PlaylistUpdateState::Failure + ); + assert_eq!( + scan_results(&ctx), + vec![LibraryScanResult { + files_scanned: 1, + groups_scanned: 1, + files_added: 0, + files_updated: 0, + files_removed: 0, + errors: 1, + }] + ); + assert!(!temp.path().join("selected.m3u").exists()); + } + + #[tokio::test] + async fn manual_update_library_persisted_input_status_keeps_failed_companion_separate() { + let temp = tempfile::tempdir().unwrap(); + let ctx = library_context(temp.path(), ProcessingOrder::Frm).await; + exec_processing(rescan_run(&ctx, UpdateGuard::new())).await; + let path = input_cache::resolve_input_storage_path(&temp.path().to_string_lossy(), "companion").await; + assert_eq!( + input_cache::load_input_status(&path).last_input_update.unwrap().state, + PlaylistUpdateState::Success + ); + let before = std::fs::read(temp.path().join("selected.m3u")).unwrap(); + tokio::fs::write(temp.path().join("companion.m3u"), "#EXTM3U\n").await.unwrap(); + exec_processing(rescan_run(&ctx, UpdateGuard::new())).await; + assert_eq!( + input_cache::load_input_status(&path).last_input_update.unwrap().state, + PlaylistUpdateState::Failure + ); + let library_path = input_cache::resolve_input_storage_path(&temp.path().to_string_lossy(), "local").await; + assert_eq!( + input_cache::load_input_status(&library_path).last_input_update.unwrap().state, + PlaylistUpdateState::Success + ); + assert_eq!(std::fs::read(temp.path().join("selected.m3u")).unwrap(), before); + } + + #[tokio::test] + async fn manual_update_library_empty_xtream_and_strm_use_existing_publish_and_keep_companions() { + use tuliprox_repository::{xtream_get_file_path, xtream_get_storage_path, BPlusTreeQuery}; + for companion in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let ctx = library_context(temp.path(), ProcessingOrder::Frm).await; + tokio::fs::write( + temp.path().join("companion.m3u"), + "#EXTM3U\n#EXTINF:-1 type=\"vod\",Companion\nhttp://stream.example/companion.mp4\n", + ) + .await + .unwrap(); + let mut sources = (**ctx.config.sources.load()).clone(); + if !companion { + sources.sources[0].inputs.retain(|name| name.as_ref() == "local"); + } + let target = Arc::make_mut(&mut sources.sources[0].targets[0]); + target.output.extend([ + TargetOutput::Xtream(XtreamTargetOutput { + flags: XtreamTargetFlagsSet::new(), + trakt: None, + filter: None, + }), + TargetOutput::Strm(tuliprox_core::model::StrmTargetOutput::from(&shared::model::StrmTargetOutputDto { + directory: temp.path().join("strm").to_string_lossy().into_owned(), + flat: true, + ..shared::model::StrmTargetOutputDto::default() + })), + ]); + ctx.config.sources.store(Arc::new(sources)); + exec_processing(rescan_run(&ctx, UpdateGuard::new())).await; + assert_run_state(&ctx, PlaylistUpdateState::Success); + ctx.events.0.lock().unwrap().clear(); + tokio::fs::remove_file(temp.path().join("media/Library Movie (2025).mp4")).await.unwrap(); + exec_processing(rescan_run(&ctx, UpdateGuard::new())).await; + assert_run_state(&ctx, PlaylistUpdateState::Success); + let config = ctx.config.config.load(); + let xtream = xtream_get_storage_path(&config, "selected").unwrap(); + for (cluster, expected) in + [(XtreamCluster::Live, 0), (XtreamCluster::Video, usize::from(companion)), (XtreamCluster::Series, 0)] + { + let path = xtream_get_file_path(&xtream, cluster); + if !path.exists() { + assert_eq!(expected, 0); + continue; + } + let mut query = BPlusTreeQuery::::try_new(&path).unwrap(); + assert_eq!(query.iter().count(), expected, "{cluster:?}"); + } + let entries = strm_artifacts(&temp.path().join("strm")).await; + assert_eq!(entries.len(), usize::from(companion), "{entries:?}"); + if companion { + assert!(entries[0].1.contains("http://stream.example/companion")); + } + } + } + + #[tokio::test] + async fn manual_update_library_empty_permission_does_not_loosen_other_provider_inputs() { + let temp = tempfile::tempdir().unwrap(); + let ctx = library_context(temp.path(), ProcessingOrder::Frm).await; + for input_type in [InputType::M3u, InputType::Xtream, InputType::Stalker, InputType::Plex] { + let input = + ConfigInput { input_type, name: format!("empty-{input_type:?}").intern(), ..ConfigInput::default() }; + let (_, error) = tuliprox_repository::persist_input_playlist(&ctx.config, &input, Vec::new()).await; + assert!(error.is_some(), "{input_type:?} must retain its generic empty-input guard"); + } + exec_processing(rescan_run(&ctx, UpdateGuard::new())).await; + let before = tokio::fs::read(temp.path().join("selected.m3u")).await.unwrap(); + ctx.events.0.lock().unwrap().clear(); + tokio::fs::write(temp.path().join("companion.m3u"), "#EXTM3U\n").await.unwrap(); + exec_processing(rescan_run(&ctx, UpdateGuard::new())).await; + { + let events = ctx.events.0.lock().unwrap(); + assert!(events.iter().any(|event| matches!(event, EventMessage::PlaylistUpdate(summary) if summary.state == PlaylistUpdateState::Failure))); + assert!(events.iter().any(|event| matches!(event, EventMessage::PlaylistUpdateProgress(progress) if progress.input_id == Some(3) && progress.state == Some(PlaylistUpdateState::Failure)))); + } + assert_eq!(before, tokio::fs::read(temp.path().join("selected.m3u")).await.unwrap()); + // An authoritatively empty Library must not mask a failed companion either. + ctx.events.0.lock().unwrap().clear(); + tokio::fs::remove_file(temp.path().join("media/Library Movie (2025).mp4")).await.unwrap(); + exec_processing(rescan_run(&ctx, UpdateGuard::new())).await; + { + let events = ctx.events.0.lock().unwrap(); + assert!(events.iter().any(|event| matches!(event, EventMessage::PlaylistUpdate(summary) if summary.state == PlaylistUpdateState::Failure))); + assert!(events.iter().any(|event| matches!(event, EventMessage::PlaylistUpdateProgress(progress) if progress.input_id == Some(3) && progress.state == Some(PlaylistUpdateState::Failure)))); + } + assert_eq!(before, tokio::fs::read(temp.path().join("selected.m3u")).await.unwrap()); + } + + #[derive(Clone, Copy, Debug)] + enum LibraryFilterBoundary { + Processing, + Persist, + M3u, + Strm, + } + + async fn strm_artifacts(root: &Path) -> Vec<(PathBuf, String)> { + let mut directories = vec![root.to_path_buf()]; + let mut artifacts = Vec::new(); + while let Some(directory) = directories.pop() { + let mut entries = tokio::fs::read_dir(directory).await.unwrap(); + while let Some(entry) = entries.next_entry().await.unwrap() { + if entry.file_type().await.unwrap().is_dir() { + directories.push(entry.path()); + } else if entry.path().extension().is_some_and(|extension| extension == "strm") { + artifacts.push((entry.path(), tokio::fs::read_to_string(entry.path()).await.unwrap())); + } + } + } + artifacts.sort(); + artifacts + } + + async fn assert_library_filtered_empty_publication(boundary: LibraryFilterBoundary) { + for disk in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let ctx = library_context(temp.path(), ProcessingOrder::Frm).await; + let mut config = (**ctx.config.config.load()).clone(); + config.disk_based_processing = disk; + ctx.config.config.store(Arc::new(config)); + tokio::fs::write( + temp.path().join("companion.m3u"), + "#EXTM3U\n#EXTINF:-1 type=\"vod\",Companion\nhttp://stream.example/companion.mp4\n", + ) + .await + .unwrap(); + let mut sources = (**ctx.config.sources.load()).clone(); + let target = Arc::make_mut(&mut sources.sources[0].targets[0]); + target.rename = None; + target.mapping = Arc::default(); + let library_only = get_filter(r#"Name ~ "^Library Movie.*""#, None).unwrap(); + if matches!(boundary, LibraryFilterBoundary::Processing) { + target.filter = library_only.clone().into(); + } + if matches!(boundary, LibraryFilterBoundary::Persist) { + target.filter.persist = Some(library_only.clone()); + } + if let TargetOutput::M3u(output) = &mut target.output[0] { + output.filter = matches!(boundary, LibraryFilterBoundary::M3u).then(|| library_only.clone()); + } + let strm_root = temp.path().join("strm"); + let mut strm = tuliprox_core::model::StrmTargetOutput::from(&shared::model::StrmTargetOutputDto { + directory: strm_root.to_string_lossy().into_owned(), + flat: true, + cleanup: true, + ..shared::model::StrmTargetOutputDto::default() + }); + strm.filter = matches!(boundary, LibraryFilterBoundary::Strm).then_some(library_only); + target.output.push(TargetOutput::Strm(strm)); + ctx.config.sources.store(Arc::new(sources)); + + exec_processing(rescan_run(&ctx, UpdateGuard::new())).await; + assert_run_state(&ctx, PlaylistUpdateState::Success); + let m3u_path = temp.path().join("selected.m3u"); + let before_m3u = tokio::fs::read_to_string(&m3u_path).await.unwrap(); + let before_strm = strm_artifacts(&strm_root).await; + let m3u_keeps_companion = matches!(boundary, LibraryFilterBoundary::Strm); + let strm_keeps_companion = matches!(boundary, LibraryFilterBoundary::M3u); + assert_eq!(before_m3u.matches("#EXTINF").count(), 1 + usize::from(m3u_keeps_companion)); + assert_eq!(before_strm.len(), 1 + usize::from(strm_keeps_companion)); + assert_eq!(before_m3u.contains("http://stream.example/companion"), m3u_keeps_companion); + assert_eq!( + before_strm.iter().any(|(_, body)| body.contains("http://stream.example/companion")), + strm_keeps_companion + ); + + ctx.events.0.lock().unwrap().clear(); + tokio::fs::remove_file(temp.path().join("media/Library Movie (2025).mp4")).await.unwrap(); + exec_processing(rescan_run(&ctx, UpdateGuard::new())).await; + let after_m3u = tokio::fs::read_to_string(&m3u_path).await.unwrap(); + let after_strm = strm_artifacts(&strm_root).await; + println!("Library filter {boundary:?}, disk={disk}: M3U before={before_m3u:?}, after={after_m3u:?}; STRM before={before_strm:?}, after={after_strm:?}"); + assert_run_state(&ctx, PlaylistUpdateState::Success); + assert_eq!( + scan_results(&ctx), + vec![LibraryScanResult { + files_scanned: 0, + groups_scanned: 0, + files_added: 0, + files_updated: 0, + files_removed: 1, + errors: 0 + }] + ); + assert_ne!(before_m3u, after_m3u, "old Library entries must really be replaced"); + assert_eq!(after_m3u.matches("#EXTINF").count(), usize::from(m3u_keeps_companion)); + assert_eq!(after_m3u.contains("http://stream.example/companion"), m3u_keeps_companion); + assert_eq!(after_strm.len(), usize::from(strm_keeps_companion)); + for (path, contents) in &before_strm { + if contents.contains("http://stream.example/companion") { + assert!( + after_strm.contains(&(path.clone(), contents.clone())), + "valid companion must be retained byte-for-byte" + ); + } else { + assert!(!path.exists(), "cleanup must remove old Library artifact {path:?}"); + } + } + // The existing M3U database is refreshed too, not only the client text file. + let target_path = + tuliprox_repository::get_target_storage_path(&ctx.config.config.load(), "selected").unwrap(); + let mut db = tuliprox_repository::BPlusTreeQuery::::try_new( + &tuliprox_repository::m3u_get_file_path_for_db(&target_path), + ) + .unwrap(); + assert_eq!(db.iter().count(), usize::from(m3u_keeps_companion)); + assert!(!temp.path().join("unselected.m3u").exists()); + } + } + + #[tokio::test] + async fn manual_update_library_processing_filter_replaces_old_output_after_last_movie_removed() { + assert_library_filtered_empty_publication(LibraryFilterBoundary::Processing).await; + } + + #[tokio::test] + async fn manual_update_library_persist_filter_replaces_old_output_after_last_movie_removed() { + assert_library_filtered_empty_publication(LibraryFilterBoundary::Persist).await; + } + + #[tokio::test] + async fn manual_update_library_m3u_output_filter_replaces_old_output_and_preserves_unfiltered_strm() { + assert_library_filtered_empty_publication(LibraryFilterBoundary::M3u).await; + } + + #[tokio::test] + async fn manual_update_library_strm_output_filter_cleans_old_files_and_preserves_unfiltered_m3u() { + assert_library_filtered_empty_publication(LibraryFilterBoundary::Strm).await; + } + + #[tokio::test] + async fn manual_update_library_empty_does_not_authorize_foreign_forced_empty_target() { + let temp = tempfile::tempdir().unwrap(); + let ctx = library_context(temp.path(), ProcessingOrder::Frm).await; + let sources = ctx.config.sources.load_full(); + let mut playlists = sources + .inputs + .iter() + .map(|input| FetchedPlaylist { input, source: MemoryPlaylistSource::default().into_source(), epg: None }) + .collect::>(); + let prepared = prepare_playlist_for_target( + &ctx, + &mut playlists, + &sources.sources[0].targets[0], + &mut HashMap::new(), + &mut Vec::new(), + ClusterFlags::Vod, + false, + ) + .await + .unwrap(); + assert_eq!(prepared.library_empty, tuliprox_repository::LibraryEmptyPublication::Contribution); + let (result, _) = finalize_prepared_target(Arc::new(ctx), prepared).await; + assert!(result.is_err(), "Library must not relax the existing non-Xtream force-empty output guard"); + assert!(!temp.path().join("selected.m3u").exists()); + } + + #[tokio::test] + async fn manual_update_library_incomplete_discovery_has_no_invented_scan_result_or_rebuild() { + for failure in ["missing", "not-directory", "partial", "no-directories", "corrupt-catalog"] { + let temp = tempfile::tempdir().unwrap(); + let ctx = library_context(temp.path(), ProcessingOrder::Frm).await; + exec_processing(rescan_run(&ctx, UpdateGuard::new())).await; + let before = tokio::fs::read(temp.path().join("selected.m3u")).await.unwrap(); + let status_path = input_cache::resolve_input_storage_path(&temp.path().to_string_lossy(), "local").await; + let cache_before = input_cache::load_input_status(&status_path).clusters; + ctx.events.0.lock().unwrap().clear(); + let mut config = (**ctx.config.config.load()).clone(); + let library = config.library.as_mut().unwrap(); + match failure { + "missing" => { + library.scan_directories[0].path = temp.path().join("missing").to_string_lossy().into_owned(); + } + "not-directory" => { + library.scan_directories[0].path = + temp.path().join("media/Library Movie (2025).mp4").to_string_lossy().into_owned(); + } + "partial" => { + let mut missing = library.scan_directories[0].clone(); + missing.path = temp.path().join("missing").to_string_lossy().into_owned(); + library.scan_directories.push(missing); + } + "no-directories" => library.scan_directories.clear(), + "corrupt-catalog" => { + let path = tuliprox_library::library::resolve_metadata_storage_path( + config.metadata_update.as_ref(), + &config.storage_dir, + ) + .join("library/broken.json"); + tokio::fs::write(path, "invalid json").await.unwrap(); + } + _ => unreachable!(), + } + ctx.config.config.store(Arc::new(config)); + exec_processing(rescan_run(&ctx, UpdateGuard::new())).await; + assert_run_state(&ctx, PlaylistUpdateState::Failure); + assert!(scan_results(&ctx).is_empty(), "{failure}"); + let status = input_cache::load_input_status(&status_path); + assert_eq!(status.last_input_update.unwrap().state, PlaylistUpdateState::Failure, "{failure}"); + assert_eq!(status.clusters, cache_before, "failed rescan must not mutate cache validity"); + assert_eq!(before, tokio::fs::read(temp.path().join("selected.m3u")).await.unwrap()); + } + } + + #[tokio::test] + async fn manual_update_library_selected_target_keeps_companion_and_real_processing_order() { + for (order, expected) in [(ProcessingOrder::Frm, "MAPPED"), (ProcessingOrder::Fmr, "RENAMED")] { + let temp = tempfile::tempdir().unwrap(); + let ctx = library_context(temp.path(), order).await; + exec_processing(rescan_run(&ctx, UpdateGuard::new())).await; + let output = tokio::fs::read_to_string(temp.path().join("selected.m3u")).await.unwrap(); + assert_eq!(output.matches("#EXTINF").count(), 2, "{output}"); + assert!(output.contains("http://stream.example/companion"), "{output}"); + assert!(output.contains(expected), "{output}"); + assert!(!temp.path().join("unselected.m3u").exists()); + let events = ctx.events.0.lock().unwrap(); + let scan = events + .iter() + .position(|event| { + matches!(event, EventMessage::PlaylistUpdateProgress(progress) + if progress.input_id == Some(2) && progress.message.starts_with("Library rescan completed")) + }) + .unwrap(); + let acquired = events + .iter() + .position(|event| { + matches!(event, EventMessage::PlaylistUpdateProgress(progress) + if progress.input_id == Some(2) && progress.state == Some(PlaylistUpdateState::Success)) + }) + .unwrap(); + let published = events + .iter() + .position(|event| { + matches!(event, EventMessage::PlaylistUpdate(summary) + if summary.state == PlaylistUpdateState::Success) + }) + .unwrap(); + assert!(scan < acquired && acquired < published); + assert!(events.iter().all(|event| !matches!(event, EventMessage::PlaylistUpdateProgress(progress) + if progress.input_id == Some(2) && progress.input_telemetry.as_ref().is_some_and(|telemetry| + !telemetry.clusters.is_empty() || telemetry.refresh_policy != InputRefreshPolicy::NORMAL)))); + } + } + + #[tokio::test] + async fn manual_update_library_unreadable_catalog_and_busy_scan_do_not_publish() { + for busy in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let ctx = library_context(temp.path(), ProcessingOrder::Frm).await; + let guard = UpdateGuard::new(); + let _busy = busy.then(|| guard.try_library().unwrap()); + if !busy { + let mut config = (**ctx.config.config.load()).clone(); + config.library.as_mut().unwrap().scan_directories[0].path = + temp.path().join("missing").to_string_lossy().into_owned(); + ctx.config.config.store(Arc::new(config)); + } + exec_processing(rescan_run(&ctx, guard.clone())).await; + assert!(!temp.path().join("selected.m3u").exists()); + assert!(!temp.path().join("unselected.m3u").exists()); + assert!(ctx + .events + .0 + .lock() + .unwrap() + .iter() + .any(|event| matches!(event, EventMessage::PlaylistUpdate(summary) + if summary.state == PlaylistUpdateState::Failure))); + assert!(guard.try_playlist().is_some()); + } + } + + #[tokio::test] + async fn manual_update_library_input_success_remains_separate_from_target_publish_failure() { + let temp = tempfile::tempdir().unwrap(); + let ctx = library_context(temp.path(), ProcessingOrder::Frm).await; + tokio::fs::create_dir(temp.path().join("selected.m3u")).await.unwrap(); + exec_processing(rescan_run(&ctx, UpdateGuard::new())).await; + let events = ctx.events.0.lock().unwrap(); + assert!(events.iter().any(|event| matches!(event, EventMessage::PlaylistUpdateProgress(progress) + if progress.input_id == Some(2) && progress.state == Some(PlaylistUpdateState::Success)))); + assert!(events.iter().any(|event| matches!(event, EventMessage::PlaylistUpdate(summary) + if summary.state == PlaylistUpdateState::Failure))); + } + + #[tokio::test] + async fn manual_update_library_rescan_reloads_valid_cache_but_bulk_keeps_catalog() { + let temp = tempfile::tempdir().unwrap(); + let ctx = library_context(temp.path(), ProcessingOrder::Frm).await; + exec_processing(rescan_run(&ctx, UpdateGuard::new())).await; + tokio::fs::write(temp.path().join("media/Second Movie (2025).mp4"), b"fixture").await.unwrap(); + // Ordinary bulk does not discover the new file and does not acquire a library permit. + let guard = UpdateGuard::new(); + let standalone_scan = guard.try_library().unwrap(); + exec_processing( + ProcessingRun::new(ctx.client.clone(), ctx.config.clone(), ctx.user_targets.clone(), ctx.events.clone()) + .with_update_guard(guard.clone()), + ) + .await; + let bulk = tokio::fs::read_to_string(temp.path().join("selected.m3u")).await.unwrap(); + assert_eq!(bulk.matches("#EXTINF").count(), 2); + drop(standalone_scan); + exec_processing(rescan_run(&ctx, guard)).await; + let rescanned = tokio::fs::read_to_string(temp.path().join("selected.m3u")).await.unwrap(); + assert_eq!(rescanned.matches("#EXTINF").count(), 3, "{rescanned}"); + } +} diff --git a/backend/processing/src/processor/playlist/mod.rs b/backend/processing/src/processor/playlist/mod.rs index 079df0181..5531e32f0 100644 --- a/backend/processing/src/processor/playlist/mod.rs +++ b/backend/processing/src/processor/playlist/mod.rs @@ -2,7 +2,6 @@ use super::providers::{LibraryProvider, PlexProvider, StalkerProvider, XmltvEpgP use crate::{ fetched_playlist::FetchedPlaylist, input_cache, - input_cache::ClusterState, metadata_sink::{MetadataUpdateSink, NoopMetadataSink}, parser::xmltv::{flatten_tvguide, merge_epg_trees, EpgMergeAccumulator, TVGuide}, playlist_watch::{process_group_watch, process_target_groups_watch}, @@ -24,10 +23,14 @@ use shared::{ error::{get_errors_notify_message, TuliproxError}, foundation::{get_field_value, set_field_value, Filter, ValueAccessor, ValueProvider}, model::{ - ClusterFlags, ConfigTargetOptions, CounterModifier, EventMessage, EventSink, FieldGet, FieldSet, InputStats, - InputType, MappingStage, PipelineStats, PlaylistGroup, PlaylistItem, PlaylistItemType, PlaylistStats, - PlaylistUpdateProgressEvent, PlaylistUpdateSummary, ProviderFetchFailure, SourceStats, StreamProperties, - TargetStats, UUIDType, WatchDisabled, WatchDisabledReason, WatchUnmatched, XtreamCluster, + ClusterFlags, ConfigTargetOptions, CounterModifier, EventMessage, EventSink, FieldGet, FieldSet, + InputRefreshOverride, InputRefreshPolicy, InputStats, InputType, InputUpdateAction, InputUpdateRequest, + MappingStage, PersistedPlaylistUpdateClusterSnapshot, PersistedPlaylistUpdateQualityDecision, + PersistedPlaylistUpdateQualitySnapshot, PersistedPlaylistUpdateTechnicalState, PipelineStats, PlaylistGroup, + PlaylistItem, PlaylistItemType, PlaylistStats, PlaylistUpdateClusterDecision, PlaylistUpdateClusterTelemetry, + PlaylistUpdateDataSource, PlaylistUpdateInputTelemetry, PlaylistUpdateProgressEvent, PlaylistUpdateRunId, + PlaylistUpdateRunOrder, PlaylistUpdateState, PlaylistUpdateSummary, ProviderFetchFailure, SourceStats, + StreamProperties, TargetStats, UUIDType, WatchDisabled, WatchDisabledReason, WatchUnmatched, XtreamCluster, }, utils::{create_alias_uuid, interner_gc, sanitize_sensitive_info, Internable}, }; @@ -35,7 +38,7 @@ use std::{ collections::{HashMap, HashSet}, future::Future, path::PathBuf, - sync::{Arc, Weak}, + sync::{atomic::AtomicU64, Arc, Weak}, time::{Duration, Instant}, }; use tokio::{ @@ -44,10 +47,10 @@ use tokio::{ }; use tuliprox_core::{ model::{ - is_valid, retain_filtered_playlist, AppConfig, CompiledMapping, ConfigFavourites, ConfigInput, - ConfigInputFlags, ConfigInputOptions, ConfigRename, ConfigTarget, Epg, FilterOutcome, MappingProgram, - ProcessTargets, ProviderIdType, ResolveReason, ReverseProxyDisabledHeaderConfig, TransformStage, UpdateGuard, - UpdateTask, + is_valid, retain_filtered_playlist, AppConfig, ClusterForceUpdate, ClusterUpdateRejection, CompiledMapping, + ConfigFavourites, ConfigInput, ConfigInputFlags, ConfigInputOptions, ConfigRename, ConfigTarget, Epg, + FilterOutcome, MappingProgram, ProcessTargets, ProviderIdType, ResolveReason, ReverseProxyDisabledHeaderConfig, + TransformStage, UpdateGuard, UpdateTask, }, utils::{debug_if_enabled, log_memory_snapshot, trace_if_enabled, StepMeasure, StepMeasureCallback}, }; @@ -62,15 +65,28 @@ use tuliprox_iptv::{ xtream, }; use tuliprox_repository::{ - load_input_playlist, persist_input_playlist, persist_playlist, CategoryKey, MemoryPlaylistSource, PlaylistSource, - PlaylistStorageState, + load_input_playlist, persist_input_playlist_with_options, persist_playlist, CategoryKey, + InputPlaylistPersistOptions, MemoryPlaylistSource, PlaylistSource, PlaylistStorageState, + TargetPlaylistPersistOptions, }; use tuliprox_session::ActiveProviderManager; const PLAYLIST_UPDATE_MAX_DURATION_SECS: u64 = 3600; const MAX_CONCURRENT_TARGET_FINALIZERS: usize = 2; +static NEXT_PLAYLIST_UPDATE_RUN_ORDER: AtomicU64 = AtomicU64::new(1); +/// Allocates the order at which a run entered the serial execution boundary. +#[must_use] +pub fn next_playlist_update_run_order() -> PlaylistUpdateRunOrder { + PlaylistUpdateRunOrder::from(NEXT_PLAYLIST_UPDATE_RUN_ORDER.fetch_add(1, std::sync::atomic::Ordering::SeqCst)) +} + +mod fetch_outcome; mod ingest; +mod input_status; +mod m3u_quality; +mod manual_update; +use manual_update::LibraryUpdateMode; mod target; mod transform; @@ -123,6 +139,7 @@ pub struct ProcessingRun< M: MetadataUpdateSink = NoopMetadataSink, > { client: reqwest::Client, + run_id: PlaylistUpdateRunId, app_config: Arc, targets: Arc, events: E, @@ -134,12 +151,27 @@ pub struct ProcessingRun< metadata_manager: Option>, pre_processed_inputs: Option>>, acquired_permit: Option, + input_refresh: Option, + library_update_mode: LibraryUpdateMode, } impl ProcessingRun { pub fn new(client: reqwest::Client, app_config: Arc, targets: Arc, events: E) -> Self { + Self::for_run(PlaylistUpdateRunId::generate(), client, app_config, targets, events) + } + + /// Creates a run for an identity already allocated by an accepted queue request. + #[must_use] + pub fn for_run( + run_id: PlaylistUpdateRunId, + client: reqwest::Client, + app_config: Arc, + targets: Arc, + events: E, + ) -> Self { Self { client, + run_id, app_config, targets, events, @@ -151,6 +183,8 @@ impl ProcessingRun pub fn with_bootstrap(self, bootstrap: B2) -> ProcessingRun { ProcessingRun { client: self.client, + run_id: self.run_id, app_config: self.app_config, targets: self.targets, events: self.events, @@ -175,6 +210,8 @@ impl metadata_manager: self.metadata_manager, pre_processed_inputs: self.pre_processed_inputs, acquired_permit: self.acquired_permit, + input_refresh: self.input_refresh, + library_update_mode: self.library_update_mode, } } @@ -211,6 +248,7 @@ impl pub fn with_metadata_manager(self, manager: Arc) -> ProcessingRun { ProcessingRun { client: self.client, + run_id: self.run_id, app_config: self.app_config, targets: self.targets, events: self.events, @@ -222,6 +260,8 @@ impl metadata_manager: Some(manager), pre_processed_inputs: self.pre_processed_inputs, acquired_permit: self.acquired_permit, + input_refresh: self.input_refresh, + library_update_mode: self.library_update_mode, } } @@ -240,6 +280,30 @@ impl self.acquired_permit = permit.into(); self } + + /// Carries a validated card action through the existing serial run. + #[must_use] + pub fn with_manual_input_update(mut self, request: Option) -> Self { + if let Some(request) = request { + match request.action { + InputUpdateAction::Provider(policy) => { + self.input_refresh = Some(InputRefreshOverride { input_id: request.input_id, policy }); + } + InputUpdateAction::Rescan => { + self.library_update_mode = LibraryUpdateMode::Rescan { input_id: request.input_id }; + self.input_refresh = None; + } + } + } + self + } + + /// Applies one manual policy override without changing the input configuration. + #[must_use] + pub fn with_input_refresh(mut self, input_refresh: impl Into>) -> Self { + self.input_refresh = input_refresh.into(); + self + } } #[allow(clippy::too_many_lines)] @@ -248,6 +312,7 @@ pub async fn exec_processing (stats, errors), Ok(Err(_)) => { error!("Playlist processing panicked"); - events.emit(EventMessage::PlaylistUpdate(PlaylistUpdateSummary::state_only( + events.emit(EventMessage::PlaylistUpdate(PlaylistUpdateSummary::for_run( + run_id, + execution_order, shared::model::PlaylistUpdateState::Failure, ))); return; @@ -343,7 +426,9 @@ pub async fn exec_processing PlaylistUpdateState { + if self.has_error { + PlaylistUpdateState::Failure + } else if self.has_pending_stalker_refresh || self.has_quality_rejections { + PlaylistUpdateState::Partial + } else { + PlaylistUpdateState::Success + } + } +} + #[cfg(test)] mod tests; diff --git a/backend/processing/src/processor/playlist/staged_completion_tests.rs b/backend/processing/src/processor/playlist/staged_completion_tests.rs new file mode 100644 index 000000000..1ca760e69 --- /dev/null +++ b/backend/processing/src/processor/playlist/staged_completion_tests.rs @@ -0,0 +1,166 @@ +mod staged_completion { + use super::*; + + pub(super) fn input_completions(events: &CollectSink, input_id: u16) -> Vec { + events + .0 + .lock() + .unwrap() + .iter() + .filter_map(|event| match event { + EventMessage::PlaylistUpdateProgress(progress) + if progress.input_id == Some(input_id) && progress.state.is_some() => + { + Some(progress.clone()) + } + _ => None, + }) + .collect() + } + + async fn staged_run_context( + root: &Path, + base_url: &str, + ) -> (PlaylistProcessingContext, Arc) { + let ctx = processing_context_with_events(root, CollectSink::default()); + let parent_file = root.join("parent.m3u"); + std::fs::write(&parent_file, "#EXTM3U\n#EXTINF:-1,Parent\nhttp://stream.example/parent\n").unwrap(); + let parent = Arc::new(ConfigInput { + id: 7, + name: "parent".intern(), + input_type: InputType::M3u, + url: parent_file.to_string_lossy().into_owned(), + enabled: true, + ..ConfigInput::default() + }); + let mut staged = (*test_input( + base_url, + ConfigInputUpdateQualityDto { live: 95, vod: 0, series: 0 }, + 0, + &[XtreamCluster::Video, XtreamCluster::Series], + )) + .clone(); + staged.id = 8; + staged.name = "staged".intern(); + staged.input_type = InputType::Staged; + staged.staged_type = shared::model::StagedInputType::Xtream; + staged.staged = Some(tuliprox_core::model::ConfigInputStaged { + for_input: Some(parent.name.clone()), + clusters: ClusterFlags::Live, + }); + staged.resolve_staged_download_type(); + let staged = Arc::new(staged); + seed_live_baseline(&ctx, &staged, 100).await; + let failed_companion = Arc::new(ConfigInput { + id: 9, + name: "failed-companion".intern(), + input_type: InputType::M3u, + url: root.join("missing.m3u").to_string_lossy().into_owned(), + enabled: true, + ..ConfigInput::default() + }); + ctx.config.sources.store(Arc::new(SourcesConfig { + inputs: vec![parent.clone(), staged.clone(), failed_companion.clone()], + sources: vec![ConfigSource { + inputs: vec![parent.name.clone(), failed_companion.name.clone()], + targets: vec![xtream_target(true)], + }], + ..SourcesConfig::default() + })); + (ctx, staged) + } + + fn normalized_trace( + events: &[EventMessage], + persisted: shared::model::PersistedPlaylistUpdateInputResult, + ) -> serde_json::Value { + let summary = events + .iter() + .find_map(|event| match event { + EventMessage::PlaylistUpdate(summary) => Some(summary), + _ => None, + }) + .unwrap(); + assert_eq!(summary.run_id, Some("staged-completion-run".into())); + assert!(summary.execution_order.is_some()); + assert_eq!(summary.state, PlaylistUpdateState::Failure); + let progress: Vec<_> = events + .iter() + .filter_map(|event| match event { + EventMessage::PlaylistUpdateProgress(progress) + if progress.input_id == Some(8) || progress.state.is_some() => + { + assert_eq!(progress.run_id, summary.run_id); + assert_eq!(progress.execution_order, summary.execution_order); + let mut progress = progress.clone(); + // Only the globally allocated test-run order is normalized. All + // input identities, states, messages and telemetry remain exact. + progress.execution_order = Some(1.into()); + Some(progress) + } + _ => None, + }) + .collect(); + assert!(persisted.timestamp > 0); + serde_json::json!({ + "progress": progress, + "completed": {"run_id": summary.run_id, "execution_order": 1, "state": summary.state}, + "persisted_input_result": {"state": persisted.state, "timestamp": 1} + }) + } + + #[tokio::test] + async fn playlist_update_status_staged_reporter_trace_matches_persistence_before_foreign_run_failure() { + let mut traces = Vec::new(); + for expected in [PlaylistUpdateState::Partial, PlaylistUpdateState::Success, PlaylistUpdateState::Failure] { + let temp = tempfile::tempdir().unwrap(); + let mut responses = + fixture_responses([if expected == PlaylistUpdateState::Success { 100 } else { 1 }, 0, 0]); + if expected == PlaylistUpdateState::Failure { + responses.insert("get_live_streams".into(), "not JSON".into()); + } + let server = TestXtreamServer::start_with_responses(responses); + let (ctx, staged) = staged_run_context(temp.path(), &server.base_url).await; + exec_processing(ProcessingRun::for_run( + "staged-completion-run".into(), + ctx.client.clone(), + ctx.config.clone(), + ctx.user_targets.clone(), + ctx.events.clone(), + )) + .await; + let path = input_storage_path(&ctx, &staged).await; + let persisted = input_cache::load_input_status(&path); + assert!(!persisted.clusters.contains_key("default")); + assert_eq!(persisted.last_input_update.unwrap().state, expected); + let completions = input_completions(&ctx.events, staged.id); + assert_eq!(completions.len(), 1, "real indirect staged input must emit one correlated completion"); + assert_eq!(completions[0].state, Some(expected)); + let telemetry = completions[0].input_telemetry.as_ref().unwrap(); + assert_eq!(telemetry.refresh_policy, InputRefreshPolicy::NORMAL); + assert_eq!(telemetry.source, None); + assert_eq!(telemetry.clusters[0].source, Some(PlaylistUpdateDataSource::Provider)); + assert_eq!(telemetry.clusters[0].threshold, Some(95)); + if expected == PlaylistUpdateState::Partial { + assert_eq!(telemetry.clusters[0].decision, Some(PlaylistUpdateClusterDecision::Rejected)); + assert_eq!(telemetry.clusters[0].active_count, Some(100)); + } + let emitted = ctx.events.0.lock().unwrap().clone(); + let own_index = emitted.iter().position(|event| matches!(event, EventMessage::PlaylistUpdateProgress(p) if p.input_id == Some(8) && p.state.is_some())).unwrap(); + let parent_index = emitted.iter().position(|event| matches!(event, EventMessage::PlaylistUpdateProgress(p) if p.input_id == Some(7) && p.state.is_some())).unwrap(); + let completed_index = + emitted.iter().position(|event| matches!(event, EventMessage::PlaylistUpdate(_))).unwrap(); + assert!(own_index < parent_index && parent_index < completed_index); + assert_eq!(input_completions(&ctx.events, 9)[0].state, Some(PlaylistUpdateState::Failure)); + traces.push(normalized_trace(&emitted, persisted.last_input_update.unwrap())); + assert_eq!(server.finish().iter().filter(|action| action.as_str() == "get_live_streams").count(), 1); + } + let actual = serde_json::Value::Array(traces); + let fixture: serde_json::Value = serde_json::from_str(include_str!(concat!( + env!("CARGO_MANIFEST_DIR"), + "/../../frontend/src/model/test_data/staged_completions.json" + ))) + .unwrap(); + assert_eq!(actual, fixture, "actual reporter trace: {}", serde_json::to_string_pretty(&actual).unwrap()); + } +} diff --git a/backend/processing/src/processor/playlist/target.rs b/backend/processing/src/processor/playlist/target.rs index c96ef0a3a..e4346fd0e 100644 --- a/backend/processing/src/processor/playlist/target.rs +++ b/backend/processing/src/processor/playlist/target.rs @@ -93,14 +93,23 @@ pub(crate) async fn process_targets], input_stats: &mut HashMap, InputStats>, errors: &mut Vec, + accepted_empty_clusters: ClusterFlags, process_parallel: bool, ) -> Vec { if !process_parallel { let mut target_stats = Vec::with_capacity(targets.len()); for (index, target) in targets.iter().enumerate() { let consume_input_source = index + 1 == targets.len(); - let result = - prepare_playlist_for_target(ctx, playlists, target, input_stats, errors, consume_input_source).await; + let result = prepare_playlist_for_target( + ctx, + playlists, + target, + input_stats, + errors, + accepted_empty_clusters, + consume_input_source, + ) + .await; match result { Ok(prepared) => { let processing = prepared.processing.clone(); @@ -142,7 +151,9 @@ pub(crate) async fn process_targets { let processing = prepared.processing.clone(); let task_ctx = Arc::clone(ctx); @@ -232,10 +243,12 @@ pub(crate) fn apply_persist_filter(target: &ConfigTarget, groups: &mut Vec, pub(crate) epg: Vec, pub(crate) processing: PipelineStats, + pub(crate) accepted_empty_clusters: ClusterFlags, } #[allow(clippy::too_many_arguments)] @@ -245,6 +258,7 @@ pub(crate) async fn prepare_playlist_for_target, InputStats>, errors: &mut Vec, + accepted_empty_clusters: ClusterFlags, consume_input_source: bool, ) -> Result> { debug_if_enabled!("Processing order is {}", &target.processing_order); @@ -254,9 +268,15 @@ pub(crate) async fn prepare_playlist_for_target = vec![]; let mut aggregate_outcome = PipelineOutcome::default(); + let mut has_empty_library = false; + let mut all_inputs_empty = !playlists.is_empty(); + let mut only_library_inputs = true; + // The caller has already required Ready for every input. An empty foreign input + // still needs its own Force contract; it cannot borrow Library's filter permission. + let mut non_library_inputs_populated = true; debug!("Executing processing pipes"); - let broadcast_step = create_broadcast_callback(&ctx.events); + let broadcast_step = create_broadcast_callback(&ctx.events, &ctx.run_id, ctx.execution_order); let bouquet_file = tuliprox_repository::load_target_bouquet(&ctx.config, &target.name).await.map_err(|err| vec![err])?; @@ -270,6 +290,11 @@ pub(crate) async fn prepare_playlist_for_target tuliprox_repository::LibraryEmptyPublication::CompleteTarget, + (true, false) if non_library_inputs_populated && accepted_empty_clusters.is_empty() => { + tuliprox_repository::LibraryEmptyPublication::FilterableContribution + } + (true, false) => tuliprox_repository::LibraryEmptyPublication::Contribution, + (false, _) => tuliprox_repository::LibraryEmptyPublication::None, + }, target: target.clone(), playlist: new_playlist, epg: new_epg, processing: aggregate_outcome.to_stats(), + accepted_empty_clusters, }) } @@ -373,7 +407,7 @@ pub(crate) async fn finalize_prepared_target>>); + +impl EventSink for PlaylistRunCollectSink { + fn emit(&self, event: EventMessage) { + self.0.lock().unwrap_or_else(std::sync::PoisonError::into_inner).push(event); + } +} + +fn playlist_update_run_result( + input_id: u16, + state: InputJobState, + errors: Vec, + had_quality_rejections: bool, +) -> InputJobResult { + let input_name = format!("input-{input_id}").intern(); + InputJobResult { + index: usize::from(input_id), + input_id, + input_name: Arc::clone(&input_name), + state, + source: None, + epg: None, + stat: create_input_stat(0, 0, errors.len(), InputType::Xtream, &input_name, 0), + errors, + accepted_empty_clusters: ClusterFlags::empty(), + had_quality_rejections, + input_telemetry: None, + } +} + +#[test] +fn playlist_update_run_input_completion_uses_existing_success_partial_failure_classification() { + let success = playlist_update_run_result(1, InputJobState::Ready, Vec::new(), false); + let rejection = playlist_update_run_result(2, InputJobState::Ready, Vec::new(), true); + let resumable = playlist_update_run_result(3, InputJobState::Pending, Vec::new(), false); + let failure = playlist_update_run_result( + 4, + InputJobState::Ready, + vec![TuliproxError::RepositoryPlaylist("technical".to_string())], + false, + ); + + assert_eq!(success.update_state(), PlaylistUpdateState::Success); + assert_eq!(rejection.update_state(), PlaylistUpdateState::Partial); + assert_eq!(resumable.update_state(), PlaylistUpdateState::Partial); + assert_eq!(failure.update_state(), PlaylistUpdateState::Failure); +} + +#[test] +fn pipeline_transparency_input_completion_reuses_correlated_progress_event_for_typed_telemetry() { + let events = PlaylistRunCollectSink::default(); + let run_id = PlaylistUpdateRunId::from("pipeline-transparency-run"); + let execution_order = PlaylistUpdateRunOrder::from(19); + let mut result = playlist_update_run_result(7, InputJobState::Ready, Vec::new(), true); + let telemetry = PlaylistUpdateInputTelemetry { + refresh_policy: InputRefreshPolicy::NORMAL, + source: None, + clusters: vec![PlaylistUpdateClusterTelemetry { + cluster: XtreamCluster::Video, + requested: true, + source: Some(PlaylistUpdateDataSource::Provider), + baseline_count: Some(12_543), + candidate_count: Some(217), + active_count: Some(12_543), + threshold: Some(90), + quality: Some(1), + decision: Some(PlaylistUpdateClusterDecision::Rejected), + technical_state: None, + }], + }; + result.input_telemetry = Some(telemetry.clone()); + + report_input_job_completion(&events, &run_id, execution_order, &result); + + let emitted = events.0.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + let EventMessage::PlaylistUpdateProgress(progress) = &emitted[0] else { + panic!("expected existing playlist progress event"); + }; + assert_eq!(progress.run_id.as_ref(), Some(&run_id)); + assert_eq!(progress.execution_order, Some(execution_order)); + assert_eq!(progress.input_id, Some(7)); + assert_eq!(progress.state, Some(PlaylistUpdateState::Partial)); + assert_eq!(progress.input_telemetry, Some(telemetry)); +} + +#[test] +fn playlist_update_run_technical_failure_progress_is_correlated_sanitized_and_input_scoped() { + let events = PlaylistRunCollectSink::default(); + let run_id = PlaylistUpdateRunId::from("processing-run"); + let execution_order = PlaylistUpdateRunOrder::from(17); + let failed = playlist_update_run_result( + 7, + InputJobState::Failed, + vec![TuliproxError::RepositoryPlaylist("https://user:secret@provider.example/playlist".to_string())], + false, + ); + let succeeded = playlist_update_run_result(8, InputJobState::Ready, Vec::new(), false); + + report_input_job_completion(&events, &run_id, execution_order, &failed); + report_input_job_completion(&events, &run_id, execution_order, &succeeded); + + let emitted = events.0.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + let progress = emitted + .iter() + .filter_map(|event| match event { + EventMessage::PlaylistUpdateProgress(progress) => Some(progress), + _ => None, + }) + .collect::>(); + assert_eq!(progress.len(), 2); + assert!(progress.iter().all(|event| event.run_id.as_ref() == Some(&run_id))); + assert!(progress.iter().all(|event| event.execution_order == Some(execution_order))); + assert_eq!(progress[0].input_id, Some(7)); + assert_eq!(progress[0].state, Some(PlaylistUpdateState::Failure)); + assert_eq!(progress[0].message, "Input 'input-7' failed during update"); + assert!(!progress[0].message.contains("secret")); + assert!(!progress[0].message.contains("previous accepted data")); + assert_eq!(progress[1].input_id, Some(8)); + assert_eq!(progress[1].state, Some(PlaylistUpdateState::Success)); + assert!(!progress[1].message.contains("failed during update")); +} + +#[test] +fn playlist_update_run_failure_text_makes_no_unverified_previous_data_claim() { + let events = PlaylistRunCollectSink::default(); + let run_id = PlaylistUpdateRunId::from("neutral-failure-run"); + let execution_order = PlaylistUpdateRunOrder::from(18); + let failed_first_run = playlist_update_run_result( + 7, + InputJobState::Failed, + vec![TuliproxError::RepositoryPlaylist("https://first:secret@provider.example/playlist".to_string())], + false, + ); + let persisted_then_failed = playlist_update_run_result( + 8, + InputJobState::Ready, + vec![TuliproxError::RepositoryPlaylist("https://epg:secret@provider.example/guide".to_string())], + false, + ); + + report_input_job_completion(&events, &run_id, execution_order, &failed_first_run); + report_input_job_completion(&events, &run_id, execution_order, &persisted_then_failed); + + let emitted = events.0.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + let progress = emitted + .iter() + .filter_map(|event| match event { + EventMessage::PlaylistUpdateProgress(progress) => Some(progress), + _ => None, + }) + .collect::>(); + assert_eq!(progress.len(), 2); + assert_eq!(progress[0].input_id, Some(7)); + assert_eq!(progress[0].message, "Input 'input-7' failed during update"); + assert_eq!(progress[1].input_id, Some(8)); + assert_eq!(progress[1].message, "Input 'input-8' failed during update"); + assert!(progress.iter().all(|event| event.run_id.as_ref() == Some(&run_id))); + assert!(progress.iter().all(|event| event.execution_order == Some(execution_order))); + assert!(progress.iter().all(|event| event.state == Some(PlaylistUpdateState::Failure))); + assert!(progress.iter().all(|event| !event.message.contains("secret"))); + assert!(progress.iter().all(|event| !event.message.contains("previous accepted data"))); +} + fn serialize_without_trailing_fields(value: &T, trailing_fields: &[u8]) -> Vec { let mut encoded = rmp_serde::to_vec(value).expect("playlist item should serialize"); for expected in trailing_fields { @@ -512,7 +676,7 @@ fn test_group(cluster: XtreamCluster, item_name: &str, input_name: &str) -> Play } #[test] -fn staged_overlay_replaces_selected_clusters_and_rewrites_input_name() { +fn pipeline_transparency_staged_overlay_replaces_groups_and_neutralizes_overlaid_runtime_facts() { let provider_name = "provider".intern(); let provider_groups = vec![ test_group(XtreamCluster::Live, "provider-live", "provider"), @@ -530,6 +694,59 @@ fn staged_overlay_replaces_selected_clusters_and_rewrites_input_name() { assert_eq!(groups[0].channels[0].header.input_name.as_ref(), "provider"); assert_eq!(groups[1].title.as_ref(), "staged-live"); assert_eq!(groups[1].channels[0].header.input_name.as_ref(), "provider"); + + let mut telemetry = PlaylistUpdateInputTelemetry { + refresh_policy: InputRefreshPolicy::NORMAL, + source: None, + clusters: vec![ + PlaylistUpdateClusterTelemetry { + cluster: XtreamCluster::Live, + requested: true, + source: Some(PlaylistUpdateDataSource::Provider), + baseline_count: Some(100), + candidate_count: Some(95), + active_count: Some(95), + threshold: Some(90), + quality: Some(95), + decision: Some(PlaylistUpdateClusterDecision::Accepted), + technical_state: None, + }, + PlaylistUpdateClusterTelemetry { + cluster: XtreamCluster::Video, + requested: true, + source: Some(PlaylistUpdateDataSource::Provider), + baseline_count: Some(200), + candidate_count: Some(200), + active_count: Some(200), + threshold: Some(100), + quality: Some(100), + decision: Some(PlaylistUpdateClusterDecision::Accepted), + technical_state: None, + }, + ], + }; + + neutralize_overlaid_cluster_facts(&mut telemetry, ClusterFlags::Live); + + let live = + telemetry.clusters.iter().find(|cluster| cluster.cluster == XtreamCluster::Live).expect("Live telemetry"); + assert!(live.requested); + assert_eq!(live.threshold, Some(90)); + assert_eq!(live.source, None); + assert_eq!(live.baseline_count, None); + assert_eq!(live.candidate_count, None); + assert_eq!(live.active_count, None); + assert_eq!(live.quality, None); + assert_eq!(live.decision, None); + + let video = + telemetry.clusters.iter().find(|cluster| cluster.cluster == XtreamCluster::Video).expect("Video telemetry"); + assert_eq!(video.source, Some(PlaylistUpdateDataSource::Provider)); + assert_eq!(video.baseline_count, Some(200)); + assert_eq!(video.candidate_count, Some(200)); + assert_eq!(video.active_count, Some(200)); + assert_eq!(video.quality, Some(100)); + assert_eq!(video.decision, Some(PlaylistUpdateClusterDecision::Accepted)); } #[test] @@ -539,6 +756,46 @@ fn staged_overlay_is_skipped_when_provider_playlist_is_cached() { assert!(!should_apply_staged_overlay(&result)); } +#[test] +fn quality_rejection_keeps_a_usable_input_ready() { + let mut result = InputDownloadResult { + authoritative_library: false, + errors: Vec::new(), + source: MemoryPlaylistSource::new(vec![test_group(XtreamCluster::Video, "retained-vod", "provider-a")]) + .into_source(), + storage_error: None, + partial: false, + quality_rejections: vec![ClusterUpdateRejection { + cluster: XtreamCluster::Video, + current_count: 12_543, + candidate_count: 217, + threshold: 90, + quality: 1, + }], + accepted_empty_clusters: ClusterFlags::empty(), + input_telemetry: None, + }; + + assert_eq!(result.job_state(), InputJobState::Ready); + assert!(result.errors.is_empty()); + assert!(!result.partial); + assert_eq!(result.quality_rejections.len(), 1); +} + +#[test] +fn quality_rejection_marks_the_run_partial_without_reusing_stalker_partial() { + let quality_rejection = PlaylistRunSignals { has_quality_rejections: true, ..PlaylistRunSignals::default() }; + assert!(!quality_rejection.has_pending_stalker_refresh); + assert_eq!(quality_rejection.state(), PlaylistUpdateState::Partial); + + let stalker_partial = PlaylistRunSignals { has_pending_stalker_refresh: true, ..PlaylistRunSignals::default() }; + assert!(!stalker_partial.has_quality_rejections); + assert_eq!(stalker_partial.state(), PlaylistUpdateState::Partial); + + let technical_failure = PlaylistRunSignals { has_error: true, ..quality_rejection }; + assert_eq!(technical_failure.state(), PlaylistUpdateState::Failure); +} + fn make_test_item(name: &str, item_type: PlaylistItemType) -> PlaylistItem { let header = PlaylistItemHeader { name: name.into(), group: "Test Group".intern(), item_type, ..Default::default() }; @@ -919,7 +1176,7 @@ mod mapping_stage { utils::FileLockManager, }; - fn build_mapping(id: &str, stage: MappingStage, script: &str) -> CompiledMapping { + pub(super) fn build_mapping(id: &str, stage: MappingStage, script: &str) -> CompiledMapping { CompiledMapping { id: id.to_string(), match_as_ascii: false, @@ -934,7 +1191,7 @@ mod mapping_stage { } } - fn build_target(mappings: Vec, remove_duplicates: bool) -> ConfigTarget { + pub(super) fn build_target(mappings: Vec, remove_duplicates: bool) -> ConfigTarget { let dto = ConfigTargetDto { options: if remove_duplicates { Some(ConfigTargetOptions { remove_duplicates, ..Default::default() }) @@ -981,6 +1238,8 @@ mod mapping_stage { }; PlaylistProcessingContext { client: reqwest::Client::new(), + run_id: "m3u-alias-test-run".into(), + execution_order: PlaylistUpdateRunOrder::from(1), config: Arc::new(config), user_targets: Arc::new(ProcessTargets { enabled: false, @@ -992,12 +1251,16 @@ mod mapping_stage { playlist_state: None, disabled_headers: None, processed_inputs: Arc::new(Mutex::new(HashSet::new())), + input_completions: Arc::new(Mutex::new(HashMap::new())), input_locks: Arc::new(Mutex::new(HashMap::new())), provider_manager: None, metadata_manager: None, pre_processed_inputs: None, + input_refresh: None, + library_update_mode: LibraryUpdateMode::ExistingCatalog, stalker_refresh_mode: StalkerRefreshMode::Complete, partial_refresh: Arc::new(std::sync::atomic::AtomicBool::new(false)), + had_quality_rejections: Arc::new(std::sync::atomic::AtomicBool::new(false)), } } @@ -1044,11 +1307,19 @@ mod mapping_stage { ..ConfigInput::default() }); - let (errors, mut primary_playlist, storage_error, partial) = download_input(&ctx, &input, false).await; + let InputDownloadResult { + errors, + source: mut primary_playlist, + storage_error, + partial, + quality_rejections, + .. + } = download_input(&ctx, &input, false).await; assert!(errors.is_empty(), "unexpected download errors: {errors:?}"); assert!(storage_error.is_none(), "unexpected primary storage error: {storage_error:?}"); assert!(!partial); + assert!(quality_rejections.is_empty()); assert!(!primary_playlist.is_empty()); let alias_url = tuliprox_repository::load_input_m3u_stream_url( &ctx.config, @@ -1097,12 +1368,19 @@ mod mapping_stage { ..ConfigInput::default() }); - let (first_errors, mut first_playlist, first_storage_error, first_partial) = - download_input(&ctx, &input, false).await; + let InputDownloadResult { + errors: first_errors, + source: mut first_playlist, + storage_error: first_storage_error, + partial: first_partial, + quality_rejections: first_quality_rejections, + .. + } = download_input(&ctx, &input, false).await; assert!(!first_errors.is_empty(), "missing alias should report an error"); assert!(first_storage_error.is_none(), "primary storage should succeed: {first_storage_error:?}"); assert!(!first_partial); + assert!(first_quality_rejections.is_empty()); assert!(!first_playlist.is_empty()); assert!(ctx.is_input_downloaded("main-account").await); assert!(!ctx.is_input_downloaded("retry-account").await); @@ -1114,12 +1392,19 @@ mod mapping_stage { .await .expect("alias fixture should be written"); - let (second_errors, mut second_playlist, second_storage_error, second_partial) = - download_input(&ctx, &input, false).await; + let InputDownloadResult { + errors: second_errors, + source: mut second_playlist, + storage_error: second_storage_error, + partial: second_partial, + quality_rejections: second_quality_rejections, + .. + } = download_input(&ctx, &input, false).await; assert!(second_errors.is_empty(), "unexpected retry errors: {second_errors:?}"); assert!(second_storage_error.is_none(), "primary storage should remain readable: {second_storage_error:?}"); assert!(!second_partial); + assert!(second_quality_rejections.is_empty()); assert!(!second_playlist.is_empty()); assert!(ctx.is_input_downloaded("retry-account").await); let alias_url = tuliprox_repository::load_input_m3u_stream_url( @@ -1169,6 +1454,7 @@ mod mapping_stage { &target, &mut stats, &mut errors, + ClusterFlags::empty(), false, ) .await @@ -1306,6 +1592,7 @@ match { &target, &mut stats, &mut errors, + ClusterFlags::empty(), false, ) .await @@ -1395,6 +1682,7 @@ match { &target, &mut stats, &mut errors, + ClusterFlags::empty(), false, ) .await @@ -1471,6 +1759,7 @@ match { &target, &mut stats, &mut errors, + ClusterFlags::empty(), false, ) .await @@ -1494,6 +1783,1678 @@ async fn trakt_finalization_stage_is_a_noop_without_xtream_configuration() { assert!(playlist.is_empty()); } +#[cfg(test)] +mod quality_rejection_fallback { + include!("m3u_quality_tests.rs"); + include!("manual_update_integration_tests.rs"); + include!("input_status_tests.rs"); + include!("staged_completion_tests.rs"); + use super::*; + use arc_swap::{ArcSwap, ArcSwapOption}; + use shared::model::{ + provider_saturation::build_group_lookup, xtream_const::XTREAM_CLUSTER, ConfigInputOptionsDto, + ConfigInputUpdateQualityDto, ConfigPaths, EventMessage, EventSink, PlaylistUpdateState, ProcessingOrder, + SeriesStreamDetailEpisodeProperties, SeriesStreamDetailProperties, SeriesStreamProperties, UpdateQualityPolicy, + }; + use std::{ + io::{ErrorKind, Read, Write}, + net::{TcpListener, TcpStream}, + path::Path, + sync::{ + atomic::{AtomicBool, Ordering}, + Mutex as StdMutex, + }, + thread::{self, JoinHandle}, + time::{Duration, SystemTime, UNIX_EPOCH}, + }; + use tuliprox_core::{ + model::{ + ConfigSource, MediaToolCapabilities, SourcesConfig, StagedFilter, TargetExecutionPlan, TargetOutput, + XtreamTargetFlagsSet, XtreamTargetOutput, + }, + utils::FileLockManager, + }; + use tuliprox_repository::{ + count_input_xtream_cluster, get_input_storage_path, get_live_cat_collection_path, + get_series_cat_collection_path, get_vod_cat_collection_path, load_xtream_target_storage, persist_playlist, + xtream_get_item_for_stream_id, xtream_get_storage_path, TargetPlaylistPersistOptions, + }; + + struct TestXtreamServer { + base_url: String, + stop: Arc, + requests: Arc>>, + errors: Arc>>, + worker: Option>, + } + + impl TestXtreamServer { + fn start(candidate_counts: [usize; 3]) -> Self { + Self::start_with_responses(fixture_responses(candidate_counts)) + } + + fn start_with_responses(responses: HashMap) -> Self { + Self::start_with_statuses(responses, HashMap::new()) + } + + fn start_with_statuses( + responses: HashMap, + statuses: HashMap, + ) -> Self { + let listener = TcpListener::bind("127.0.0.1:0").expect("bind Xtream fixture server"); + listener.set_nonblocking(true).expect("configure Xtream fixture listener"); + let address = listener.local_addr().expect("Xtream fixture address"); + let stop = Arc::new(AtomicBool::new(false)); + let requests = Arc::new(StdMutex::new(Vec::new())); + let errors = Arc::new(StdMutex::new(Vec::new())); + let worker_stop = Arc::clone(&stop); + let worker_requests = Arc::clone(&requests); + let worker_errors = Arc::clone(&errors); + let worker = thread::spawn(move || { + while !worker_stop.load(Ordering::Acquire) { + match listener.accept() { + Ok((stream, _)) => match serve_fixture_request(stream, &responses, &statuses) { + Ok(action) => worker_requests.lock().expect("request log lock").push(action), + Err(err) => worker_errors.lock().expect("server error lock").push(err.to_string()), + }, + Err(err) if err.kind() == ErrorKind::WouldBlock => thread::sleep(Duration::from_millis(1)), + Err(err) => { + worker_errors.lock().expect("server error lock").push(err.to_string()); + break; + } + } + } + }); + + Self { base_url: format!("http://{address}"), stop, requests, errors, worker: Some(worker) } + } + + fn finish(mut self) -> Vec { + self.stop_worker(); + let errors = self.errors.lock().expect("server error lock").clone(); + assert!(errors.is_empty(), "Xtream fixture server errors: {errors:?}"); + self.requests.lock().expect("request log lock").clone() + } + + fn stop_worker(&mut self) { + self.stop.store(true, Ordering::Release); + if let Some(worker) = self.worker.take() { + worker.join().expect("Xtream fixture server should stop cleanly"); + } + } + } + + impl Drop for TestXtreamServer { + fn drop(&mut self) { self.stop_worker(); } + } + + #[derive(Clone, Default)] + struct CollectSink(Arc>>); + + impl EventSink for CollectSink { + fn emit(&self, event: EventMessage) { self.0.lock().expect("event sink lock").push(event); } + } + + fn fixture_responses(candidate_counts: [usize; 3]) -> HashMap { + let mut responses = + HashMap::from([("login".to_string(), serde_json::json!({"user_info": {"status": "Active"}}).to_string())]); + for (cluster, count) in XTREAM_CLUSTER.into_iter().zip(candidate_counts) { + let (category_action, stream_action, category_id, id_field, id_base) = match cluster { + XtreamCluster::Live => ("get_live_categories", "get_live_streams", 1_u32, "stream_id", 1_000_u32), + XtreamCluster::Video => ("get_vod_categories", "get_vod_streams", 2, "stream_id", 2_000), + XtreamCluster::Series => ("get_series_categories", "get_series", 3, "series_id", 3_000), + }; + responses.insert( + category_action.to_string(), + serde_json::json!([{"category_id": category_id, "category_name": format!("candidate-{cluster}")}]) + .to_string(), + ); + let streams: Vec<_> = (0..count) + .map(|offset| { + serde_json::json!({ + "name": format!("candidate-{cluster}-{offset}"), + id_field: id_base + u32::try_from(offset).expect("fixture count fits u32"), + "category_id": category_id, + }) + }) + .collect(); + responses.insert(stream_action.to_string(), serde_json::Value::Array(streams).to_string()); + } + responses + } + + fn live_fixture_responses( + categories: &serde_json::Value, + streams: Vec, + ) -> HashMap { + HashMap::from([ + ("login".to_string(), serde_json::json!({"user_info": {"status": "Active"}}).to_string()), + ("get_live_categories".to_string(), categories.to_string()), + ("get_live_streams".to_string(), serde_json::Value::Array(streams).to_string()), + ]) + } + + fn live_fixture_stream(provider_id: u32, category_id: u32, name: &str) -> serde_json::Value { + serde_json::json!({ + "name": name, + "stream_id": provider_id, + "category_id": category_id, + }) + } + + fn serve_fixture_request( + mut stream: TcpStream, + responses: &HashMap, + statuses: &HashMap, + ) -> std::io::Result { + stream.set_nonblocking(false)?; + stream.set_read_timeout(Some(Duration::from_secs(1)))?; + let mut request = Vec::with_capacity(1_024); + let mut buffer = [0_u8; 1_024]; + loop { + let read = stream.read(&mut buffer)?; + if read == 0 { + break; + } + request.extend_from_slice(&buffer[..read]); + if request.windows(4).any(|window| window == b"\r\n\r\n") { + break; + } + } + let request = String::from_utf8_lossy(&request); + let action = request + .lines() + .next() + .and_then(|line| line.split_whitespace().nth(1)) + .and_then(|target| target.split_once("action=").map(|(_, value)| value)) + .map_or_else(|| "login".to_string(), |value| value.split('&').next().unwrap_or(value).to_string()); + let body = responses + .get(&action) + .ok_or_else(|| std::io::Error::new(ErrorKind::InvalidInput, format!("unexpected action {action}")))?; + let status = statuses.get(&action).copied().unwrap_or(reqwest::StatusCode::OK); + let response = format!( + "HTTP/1.1 {status}\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len() + ); + stream.write_all(response.as_bytes())?; + Ok(action) + } + + fn processing_context(storage_dir: &Path) -> PlaylistProcessingContext { + processing_context_with_events(storage_dir, shared::model::NoopSink) + } + + fn processing_context_with_events(storage_dir: &Path, events: E) -> PlaylistProcessingContext { + let paths = ConfigPaths { + home_path: String::new(), + config_path: String::new(), + storage_path: String::new(), + config_file_path: String::new(), + sources_file_path: String::new(), + mapping_file_path: None, + mapping_files_used: None, + template_file_path: None, + template_files_used: None, + api_proxy_file_path: String::new(), + custom_stream_response_path: None, + }; + let config = AppConfig { + config: Arc::new(ArcSwap::from_pointee(Config { + storage_dir: storage_dir.to_string_lossy().into_owned(), + disk_based_processing: false, + ..Config::default() + })), + sources: Arc::new(ArcSwap::from_pointee(SourcesConfig::default())), + hdhomerun: Arc::new(ArcSwapOption::default()), + api_proxy: Arc::new(ArcSwapOption::default()), + file_locks: Arc::new(FileLockManager::default()), + paths: Arc::new(ArcSwap::from_pointee(paths)), + custom_stream_response: Arc::new(ArcSwapOption::default()), + access_token_secret: [0; 32], + encrypt_secret: [0; 16], + media_tools: Arc::new(MediaToolCapabilities::new()), + }; + PlaylistProcessingContext { + client: reqwest::Client::new(), + run_id: "xtream-quality-test-run".into(), + execution_order: PlaylistUpdateRunOrder::from(1), + config: Arc::new(config), + user_targets: Arc::new(ProcessTargets { + enabled: false, + inputs: Vec::new(), + targets: Vec::new(), + target_names: Vec::new(), + }), + events, + playlist_state: None, + disabled_headers: None, + processed_inputs: Arc::new(Mutex::new(HashSet::new())), + input_completions: Arc::new(Mutex::new(HashMap::new())), + input_locks: Arc::new(Mutex::new(HashMap::new())), + provider_manager: None, + metadata_manager: None, + pre_processed_inputs: None, + input_refresh: None, + library_update_mode: LibraryUpdateMode::ExistingCatalog, + stalker_refresh_mode: StalkerRefreshMode::Complete, + partial_refresh: Arc::new(AtomicBool::new(false)), + had_quality_rejections: Arc::new(AtomicBool::new(false)), + } + } + + #[test] + fn playlist_update_run_constructor_preserves_queued_identity_and_generates_unique_automatic_ids() { + let temp = tempfile::tempdir().expect("temp dir"); + let ctx = processing_context(temp.path()); + let queued_id = PlaylistUpdateRunId::from("queued-run"); + let queued = ProcessingRun::for_run( + queued_id.clone(), + ctx.client.clone(), + Arc::clone(&ctx.config), + Arc::clone(&ctx.user_targets), + shared::model::NoopSink, + ); + let automatic_a = ProcessingRun::new( + ctx.client.clone(), + Arc::clone(&ctx.config), + Arc::clone(&ctx.user_targets), + shared::model::NoopSink, + ); + let automatic_b = ProcessingRun::new(ctx.client, ctx.config, ctx.user_targets, shared::model::NoopSink); + + assert_eq!(queued.run_id, queued_id); + assert_ne!(automatic_a.run_id, automatic_b.run_id); + } + + #[tokio::test] + async fn playlist_update_run_manual_and_automatic_execution_order_is_generated_and_preserved() { + let temp = tempfile::tempdir().expect("temp dir"); + let events = CollectSink::default(); + let ctx = processing_context_with_events(temp.path(), events.clone()); + let manual_run_id = PlaylistUpdateRunId::from("manual-run"); + + exec_processing(ProcessingRun::for_run( + manual_run_id.clone(), + ctx.client.clone(), + Arc::clone(&ctx.config), + Arc::clone(&ctx.user_targets), + events.clone(), + )) + .await; + let automatic_run = ProcessingRun::new(ctx.client, ctx.config, ctx.user_targets, events.clone()); + let automatic_run_id = automatic_run.run_id.clone(); + exec_processing(automatic_run).await; + + let emitted = events.0.lock().expect("event sink lock"); + let terminal_for = |run_id: &PlaylistUpdateRunId| { + emitted.iter().find_map(|event| match event { + EventMessage::PlaylistUpdate(summary) if summary.run_id.as_ref() == Some(run_id) => Some(summary), + _ => None, + }) + }; + let manual_terminal = terminal_for(&manual_run_id).expect("manual terminal event"); + let automatic_terminal = terminal_for(&automatic_run_id).expect("automatic terminal event"); + let manual_order = manual_terminal.execution_order.expect("manual execution order"); + let automatic_order = automatic_terminal.execution_order.expect("automatic execution order"); + assert!(automatic_order > manual_order); + + let progress_for = |run_id: &PlaylistUpdateRunId| { + emitted + .iter() + .filter_map(|event| match event { + EventMessage::PlaylistUpdateProgress(progress) if progress.run_id.as_ref() == Some(run_id) => { + Some(progress) + } + _ => None, + }) + .collect::>() + }; + let manual_progress = progress_for(&manual_run_id); + let automatic_progress = progress_for(&automatic_run_id); + assert!(!manual_progress.is_empty()); + assert!(!automatic_progress.is_empty()); + assert!(manual_progress.iter().all(|progress| progress.execution_order == Some(manual_order))); + assert!(automatic_progress.iter().all(|progress| progress.execution_order == Some(automatic_order))); + } + + fn test_input( + base_url: &str, + update_quality: ConfigInputUpdateQualityDto, + cache_duration_seconds: u64, + skipped_clusters: &[XtreamCluster], + ) -> Arc { + let options = ConfigInputOptionsDto { + skip_live: skipped_clusters.contains(&XtreamCluster::Live), + skip_vod: skipped_clusters.contains(&XtreamCluster::Video), + skip_series: skipped_clusters.contains(&XtreamCluster::Series), + update_quality, + ..ConfigInputOptionsDto::default() + }; + Arc::new(ConfigInput { + id: 1, + name: "quality-provider".intern(), + input_type: InputType::Xtream, + url: base_url.to_string(), + username: Some("user".to_string()), + password: Some("password".to_string()), + enabled: true, + options: Some(ConfigInputOptions::from(&options)), + cache_duration_seconds, + ..ConfigInput::default() + }) + } + + fn baseline_group(cluster: XtreamCluster, category_id: u32, title: &str, first_item_id: u32) -> PlaylistGroup { + baseline_group_with_count(cluster, category_id, title, first_item_id, 2) + } + + fn baseline_group_with_count( + cluster: XtreamCluster, + category_id: u32, + title: &str, + first_item_id: u32, + count: usize, + ) -> PlaylistGroup { + let title = title.intern(); + let item_type = PlaylistItemType::from(cluster); + let channels = (0..count) + .map(|offset| { + let offset = u32::try_from(offset).expect("baseline fixture count fits u32"); + let item_id = (first_item_id + offset).to_string().intern(); + let mut header = PlaylistItemHeader { + id: Arc::clone(&item_id), + input_stream_id: item_id, + name: format!("{title}-{offset}").intern(), + title: format!("{title}-{offset}").intern(), + group: Arc::clone(&title), + url: format!("http://old.example/{cluster}/{first_item_id}/{offset}").intern(), + input_name: "quality-provider".intern(), + item_type, + xtream_cluster: cluster, + category_id, + ..PlaylistItemHeader::default() + }; + header.gen_uuid(); + PlaylistItem { header } + }) + .collect(); + PlaylistGroup { id: category_id, title, channels, xtream_cluster: cluster } + } + + async fn seed_live_baseline( + ctx: &PlaylistProcessingContext, + input: &ConfigInput, + first_item_id: u32, + ) { + let baseline = baseline_group_with_count(XtreamCluster::Live, 1, "old-live", first_item_id, 100); + let (persisted, error) = tuliprox_repository::persist_input_playlist(&ctx.config, input, vec![baseline]).await; + assert!(error.is_none(), "Live baseline persistence failed: {error:?}"); + assert_eq!(persisted.iter().map(|group| group.channels.len()).sum::(), 100); + } + + fn baseline_groups() -> Vec { + vec![ + baseline_group(XtreamCluster::Live, 1, "old-live", 100), + baseline_group(XtreamCluster::Video, 2, "old-vod", 200), + baseline_group(XtreamCluster::Series, 3, "old-series", 300), + ] + } + + async fn seed_baseline(ctx: &PlaylistProcessingContext, input: &ConfigInput) { + let (persisted, error) = + tuliprox_repository::persist_input_playlist(&ctx.config, input, baseline_groups()).await; + assert!(error.is_none(), "baseline persistence failed: {error:?}"); + assert_baseline(&persisted); + } + + fn assert_baseline(groups: &[PlaylistGroup]) { + for (cluster, category_id, title, item_ids) in [ + (XtreamCluster::Live, 1, "old-live", ["100", "101"]), + (XtreamCluster::Video, 2, "old-vod", ["200", "201"]), + (XtreamCluster::Series, 3, "old-series", ["300", "301"]), + ] { + let group = groups + .iter() + .find(|group| group.xtream_cluster == cluster && group.id == category_id) + .unwrap_or_else(|| panic!("missing persisted {cluster} group")); + assert_eq!(group.title.as_ref(), title); + assert_eq!(group.channels.len(), 2); + assert!(group.channels.iter().all(|item| item.header.xtream_cluster == cluster)); + assert_eq!(group.channels[0].header.id.as_ref(), item_ids[0]); + assert_eq!(group.channels[1].header.id.as_ref(), item_ids[1]); + } + } + + async fn input_storage_path(ctx: &PlaylistProcessingContext, input: &ConfigInput) -> PathBuf { + let storage_dir = ctx.config.config.load().storage_dir.clone(); + get_input_storage_path(&input.name, &storage_dir).await.expect("input storage path") + } + + async fn seed_valid_cluster_cache(ctx: &PlaylistProcessingContext, input: &ConfigInput) -> u64 { + let storage_path = input_storage_path(ctx, input).await; + let now = SystemTime::now().duration_since(UNIX_EPOCH).expect("current time").as_secs(); + let mut status = input_cache::InputStatus::default(); + for cluster in XTREAM_CLUSTER { + status.clusters.insert( + cluster.as_ref().to_string(), + input_cache::ClusterStatus { status: input_cache::ClusterState::Ok, timestamp: now, last_update: None }, + ); + } + input_cache::save_input_status(&storage_path, &status); + now + } + + fn set_refresh_policy( + ctx: &mut PlaylistProcessingContext, + input: &ConfigInput, + policy: InputRefreshPolicy, + ) { + ctx.input_refresh = Some(InputRefreshOverride { input_id: input.id, policy }); + } + + fn xtream_target(use_memory_cache: bool) -> Arc { + Arc::new(ConfigTarget { + id: 1, + enabled: true, + name: "quality-target".to_string(), + options: None, + sort: None, + filter: StagedFilter::default(), + output: vec![TargetOutput::Xtream(XtreamTargetOutput { + flags: XtreamTargetFlagsSet::new(), + trakt: None, + filter: None, + })], + rename: None, + mapping_ids: None, + mapping: Arc::new(ArcSwapOption::new(None)), + favourites: None, + processing_order: ProcessingOrder::default(), + execution_plan: TargetExecutionPlan::default(), + watch: None, + use_memory_cache, + }) + } + + fn install_source( + ctx: &PlaylistProcessingContext, + input: &Arc, + target: &Arc, + ) { + let inputs = vec![Arc::clone(input)]; + ctx.config.sources.store(Arc::new(SourcesConfig { + batch_files: Vec::new(), + templates: None, + provider: Vec::new(), + group_lookup: build_group_lookup(&inputs), + inputs, + sources: vec![ConfigSource { inputs: vec![Arc::clone(&input.name)], targets: vec![Arc::clone(target)] }], + })); + } + + async fn seed_xtream_target( + ctx: &PlaylistProcessingContext, + target: &ConfigTarget, + playlist_state: &Arc, + ) -> u32 { + let mut groups = baseline_groups(); + persist_playlist( + &ctx.config, + &mut groups, + None, + target, + Some(playlist_state), + TargetPlaylistPersistOptions::default(), + ) + .await + .expect("target baseline should persist"); + groups + .iter() + .find(|group| group.xtream_cluster == XtreamCluster::Video) + .and_then(|group| group.channels.first()) + .map(|item| item.header.virtual_id.get()) + .expect("target baseline VOD id") + } + + async fn assert_target_counts(app_config: &Arc, target: &ConfigTarget, expected: [usize; 3]) { + let storage = load_xtream_target_storage(app_config, target).await.expect("target storage should load"); + assert_eq!([storage.live.len(), storage.vod.len(), storage.series.len()], expected); + } + + async fn assert_empty_target_categories( + app_config: &Arc, + target: &ConfigTarget, + clusters: &[XtreamCluster], + ) { + let storage_path = { + let config = app_config.config.load(); + xtream_get_storage_path(&config, &target.name).expect("Xtream target storage path") + }; + for cluster in clusters { + let path = match cluster { + XtreamCluster::Live => get_live_cat_collection_path(&storage_path), + XtreamCluster::Video => get_vod_cat_collection_path(&storage_path), + XtreamCluster::Series => get_series_cat_collection_path(&storage_path), + }; + let categories: Vec = serde_json::from_slice( + &tokio::fs::read(&path).await.unwrap_or_else(|error| panic!("read {}: {error}", path.display())), + ) + .unwrap_or_else(|error| panic!("parse {}: {error}", path.display())); + assert!(categories.is_empty(), "{} should contain no categories", path.display()); + } + } + + #[test] + fn refresh_policy_override_is_scoped_to_the_selected_input_id() { + let temp = tempfile::tempdir().expect("temporary storage"); + let mut ctx = processing_context(temp.path()); + let selected = ConfigInput { id: 17, ..ConfigInput::default() }; + set_refresh_policy(&mut ctx, &selected, InputRefreshPolicy::FORCE); + + assert_eq!(ctx.refresh_policy(17), InputRefreshPolicy::FORCE); + assert_eq!(ctx.refresh_policy(18), InputRefreshPolicy::NORMAL); + } + + fn assert_requested_actions(mut actual: Vec, expected: &[&str]) { + actual.sort(); + let mut expected: Vec<_> = expected.iter().map(ToString::to_string).collect(); + expected.sort(); + assert_eq!(actual, expected); + } + + #[tokio::test] + async fn mixed_in_memory_update_keeps_rejected_vod_ready_and_marks_the_run_partial() { + let temp = tempfile::tempdir().expect("temporary storage"); + let server = TestXtreamServer::start([2, 1, 2]); + let events = CollectSink::default(); + let ctx = processing_context_with_events(temp.path(), events.clone()); + let input = + test_input(&server.base_url, ConfigInputUpdateQualityDto { live: 100, vod: 100, series: 100 }, 0, &[]); + seed_baseline(&ctx, &input).await; + + let mut result = process_input_job_inner(0, &ctx, &input).await; + let requests = server.finish(); + + assert_eq!(result.state, InputJobState::Ready); + assert!(result.errors.is_empty(), "quality rejection must not become an error: {:?}", result.errors); + assert!(ctx.had_quality_rejections.load(Ordering::Acquire)); + assert_eq!( + PlaylistRunSignals { has_quality_rejections: true, ..PlaylistRunSignals::default() }.state(), + PlaylistUpdateState::Partial + ); + + let groups = result.source.take().expect("ready input source").take_groups(); + let live = + groups.iter().find(|group| group.xtream_cluster == XtreamCluster::Live).expect("accepted Live group"); + let vod = groups.iter().find(|group| group.xtream_cluster == XtreamCluster::Video).expect("retained VOD group"); + let series = + groups.iter().find(|group| group.xtream_cluster == XtreamCluster::Series).expect("accepted Series group"); + assert_eq!(live.title.as_ref(), "candidate-live"); + assert_eq!(live.channels.len(), 2); + assert_eq!(vod.title.as_ref(), "old-vod"); + assert_eq!(vod.channels.len(), 2); + assert_eq!(series.title.as_ref(), "candidate-series"); + assert_eq!(series.channels.len(), 2); + + let storage_path = input_storage_path(&ctx, &input).await; + let status = input_cache::load_input_status(&storage_path); + assert_eq!( + status.clusters.get(XtreamCluster::Live.as_ref()).map(|entry| &entry.status), + Some(&input_cache::ClusterState::Ok) + ); + assert_eq!( + status.clusters.get(XtreamCluster::Video.as_ref()).map(|entry| &entry.status), + Some(&input_cache::ClusterState::Failed) + ); + assert_eq!( + status.clusters.get(XtreamCluster::Series.as_ref()).map(|entry| &entry.status), + Some(&input_cache::ClusterState::Ok) + ); + + let emitted = events.0.lock().expect("event sink lock"); + let rejection_events = emitted + .iter() + .filter_map(|event| match event { + EventMessage::PlaylistUpdateProgress(progress) + if progress.message.contains("cluster 'vod' rejected") => + { + Some(progress) + } + _ => None, + }) + .collect::>(); + assert_eq!(rejection_events.len(), 1); + assert_eq!(rejection_events[0].target, "quality-provider"); + assert_eq!( + rejection_events[0].message, + "Input 'quality-provider' cluster 'vod' rejected: current=2 candidate=1 threshold=100 quality=50; retaining previous cluster" + ); + assert_requested_actions( + requests, + &[ + "login", + "get_live_categories", + "get_live_streams", + "get_vod_categories", + "get_vod_streams", + "get_series_categories", + "get_series", + ], + ); + } + + #[tokio::test] + async fn coderabbit_stalker_global_failures_mark_only_requested_non_skipped_clusters() { + use crate::processor::stalker::{download_stalker_playlist, StalkerCluster}; + + #[derive(Clone, Copy, Debug)] + enum FailurePoint { + Portal, + Client, + Storage, + Handshake, + } + + for failure in [FailurePoint::Portal, FailurePoint::Client, FailurePoint::Storage, FailurePoint::Handshake] { + for requested in [None, Some(vec![StalkerCluster::Live, StalkerCluster::Series])] { + let temp = tempfile::tempdir().unwrap(); + let ctx = processing_context(temp.path()); + let server = TestXtreamServer::start_with_responses(HashMap::from([("handshake".into(), "{}".into())])); + let mut input = + (*test_input(&server.base_url, ConfigInputUpdateQualityDto::default(), 0, &[XtreamCluster::Live])) + .clone(); + input.input_type = InputType::Stalker; + input.stalker = Some(tuliprox_core::model::StalkerInputConfig::default()); + match failure { + FailurePoint::Portal => input.url.clear(), + FailurePoint::Client => input.url = "://invalid-url".into(), + FailurePoint::Storage => { + let path = input_storage_path(&ctx, &input).await; + tokio::fs::write(path.join("stalker"), b"not a directory").await.unwrap(); + } + FailurePoint::Handshake => {} + } + let fetch = download_stalker_playlist( + &ctx.config, + &ctx.client, + &input, + requested.as_deref(), + StalkerRefreshMode::Complete, + true, + UpdateQualityPolicy::Enforce, + ) + .await; + let expected = if requested.is_some() { + vec![XtreamCluster::Series] + } else { + vec![XtreamCluster::Video, XtreamCluster::Series] + }; + assert_eq!(fetch.failed_clusters, expected, "{failure:?}"); + assert_eq!(fetch.errors.len(), 1, "{failure:?}"); + assert!(!fetch.is_ok()); + assert!(fetch.quality_acceptances.is_empty() && fetch.quality_rejections.is_empty()); + assert!(fetch.force_updates.is_empty()); + let requests = server.finish(); + match failure { + FailurePoint::Handshake => assert!(!requests.is_empty()), + FailurePoint::Portal | FailurePoint::Client | FailurePoint::Storage => assert!(requests.is_empty()), + } + } + } + } + + #[tokio::test] + async fn coderabbit_xtream_http_failure_preserves_successful_disk_clusters() { + for quality in [UpdateQualityPolicy::Enforce, UpdateQualityPolicy::Bypass] { + let temp = tempfile::tempdir().unwrap(); + let server = TestXtreamServer::start_with_statuses( + fixture_responses([2, 2, 2]), + HashMap::from([("get_vod_streams".to_owned(), reqwest::StatusCode::BAD_REQUEST)]), + ); + let ctx = processing_context(temp.path()); + let mut config = (**ctx.config.config.load()).clone(); + config.disk_based_processing = true; + ctx.config.config.store(Arc::new(config)); + let input = + test_input(&server.base_url, ConfigInputUpdateQualityDto { live: 100, vod: 100, series: 100 }, 0, &[]); + seed_baseline(&ctx, &input).await; + + let fetch = tuliprox_iptv::xtream::download_xtream_playlist( + &ctx.config, + &ctx.client, + &ctx.events, + &input, + None, + quality, + ) + .await; + assert_eq!(fetch.failed_clusters, vec![XtreamCluster::Video]); + assert!(!fetch.is_ok(), "the failed cluster must remain a technical failure"); + assert!(fetch.persisted); + assert!(fetch.quality_rejections.is_empty()); + let accepted: Vec<_> = match quality { + UpdateQualityPolicy::Enforce => fetch.quality_acceptances.iter().map(|value| value.cluster).collect(), + UpdateQualityPolicy::Bypass => fetch.force_updates.iter().map(|value| value.cluster).collect(), + }; + assert_eq!(accepted, vec![XtreamCluster::Live, XtreamCluster::Series]); + + let storage = input_storage_path(&ctx, &input).await; + let groups = + tuliprox_repository::load_input_xtream_playlist(&ctx.config, &storage, &XTREAM_CLUSTER).await.unwrap(); + for (cluster, title) in [ + (XtreamCluster::Live, "candidate-live"), + (XtreamCluster::Video, "old-vod"), + (XtreamCluster::Series, "candidate-series"), + ] { + let group = groups.iter().find(|group| group.xtream_cluster == cluster).unwrap(); + assert_eq!(group.title.as_ref(), title); + assert_eq!(group.channels.len(), 2); + } + assert_eq!(server.finish().len(), 7); + } + } + + #[tokio::test] + async fn pipeline_transparency_shows_failure_survives_xtream_fetch_and_reload() { + for disk in [false, true] { + let temp = tempfile::tempdir().unwrap(); + let mut responses = fixture_responses([2, 2, 2]); + responses.insert("get_series".to_owned(), "{".to_owned()); + let server = TestXtreamServer::start_with_responses(responses); + let mut ctx = processing_context(temp.path()); + let mut config = ctx.config.config.load().as_ref().clone(); + config.disk_based_processing = disk; + ctx.config.config.store(Arc::new(config)); + let input = + test_input(&server.base_url, ConfigInputUpdateQualityDto { live: 100, vod: 100, series: 0 }, 0, &[]); + seed_baseline(&ctx, &input).await; + set_refresh_policy(&mut ctx, &input, InputRefreshPolicy::FORCE); + let mut result = download_input(&ctx, &input, false).await; + assert_eq!(result.update_state(), PlaylistUpdateState::Failure); + assert!(!result.errors.is_empty()); + let telemetry = result.input_telemetry.as_ref().unwrap(); + for cluster in &telemetry.clusters { + if cluster.cluster == XtreamCluster::Series { + assert_eq!(cluster.technical_state, Some(PersistedPlaylistUpdateTechnicalState::Failed)); + assert_eq!(cluster.decision, None); + assert_eq!(cluster.quality, None); + } else { + assert_eq!(cluster.technical_state, None, "do not attribute Shows failure to a sibling"); + assert_eq!(cluster.decision, Some(PlaylistUpdateClusterDecision::Accepted)); + } + assert_eq!(cluster.active_count, None, "failed batch does not prove final activation"); + } + let storage = input_storage_path(&ctx, &input).await; + let saved = input_cache::load_input_status(&storage); + let shows = saved.clusters["series"].last_update.unwrap(); + assert_eq!(shows.technical_state, Some(PersistedPlaylistUpdateTechnicalState::Failed)); + assert_eq!(shows.quality, None); + assert_eq!(shows.quality_guard_threshold, Some(0)); + assert_eq!(shows.policy, Some(InputRefreshPolicy::FORCE)); + assert!(server.finish().iter().any(|action| action == "get_series")); + } + } + + #[tokio::test] + async fn pipeline_transparency_normal_policy_reuses_a_valid_cluster_cache() { + let temp = tempfile::tempdir().expect("temporary storage"); + let server = TestXtreamServer::start([2, 2, 2]); + let mut ctx = processing_context(temp.path()); + let input = + test_input(&server.base_url, ConfigInputUpdateQualityDto { live: 100, vod: 100, series: 100 }, 3_600, &[]); + seed_baseline(&ctx, &input).await; + seed_valid_cluster_cache(&ctx, &input).await; + set_refresh_policy(&mut ctx, &input, InputRefreshPolicy::NORMAL); + + let mut result = download_input(&ctx, &input, false).await; + let requests = server.finish(); + + assert!(result.errors.is_empty()); + assert!(result.storage_error.is_none()); + assert!(result.quality_rejections.is_empty()); + assert_eq!(result.job_state(), InputJobState::Ready); + assert_baseline(&result.source.take_groups()); + assert!(requests.is_empty(), "valid cache should avoid provider requests: {requests:?}"); + let telemetry = result.input_telemetry.as_ref().expect("real cache acquisition telemetry"); + assert_eq!(telemetry.refresh_policy, InputRefreshPolicy::NORMAL); + assert!(telemetry + .clusters + .iter() + .all(|cluster| { cluster.requested && cluster.source == Some(PlaylistUpdateDataSource::Cache) })); + } + + async fn assert_provider_acquisition_survives_in_run_reuse(policy: InputRefreshPolicy) { + let temp = tempfile::tempdir().expect("temporary storage"); + let server = TestXtreamServer::start([2, 2, 2]); + let mut ctx = processing_context(temp.path()); + let input = test_input(&server.base_url, ConfigInputUpdateQualityDto { live: 95, vod: 95, series: 95 }, 0, &[]); + seed_baseline(&ctx, &input).await; + set_refresh_policy(&mut ctx, &input, policy); + + let mut first = download_input(&ctx, &input, false).await; + + assert!(first.errors.is_empty(), "first provider acquisition failed: {:?}", first.errors); + assert!(first.storage_error.is_none()); + assert_eq!(first.source.get_channel_count(), 6); + let first_telemetry = first.input_telemetry.as_ref().expect("provider acquisition telemetry"); + assert_eq!(first_telemetry.refresh_policy, policy); + assert!(first_telemetry + .clusters + .iter() + .all(|cluster| { cluster.requested && cluster.source == Some(PlaylistUpdateDataSource::Provider) })); + + let storage_path = input_storage_path(&ctx, &input).await; + let status_after_acquisition = input_cache::load_input_status(&storage_path); + assert!(XTREAM_CLUSTER.iter().all(|cluster| { + status_after_acquisition.clusters[cluster.as_ref()].last_update.as_ref().is_some_and(|snapshot| { + snapshot.policy == Some(policy) && snapshot.source == Some(PlaylistUpdateDataSource::Provider) + }) + })); + + // A second source/target in the same run reuses the persisted input. It is not a + // second acquisition and therefore must not emit replacement cache telemetry. + let mut reused = download_input(&ctx, &input, false).await; + + assert!(reused.errors.is_empty(), "in-run reuse failed: {:?}", reused.errors); + assert!(reused.storage_error.is_none()); + assert_eq!(reused.source.get_channel_count(), 6); + assert_eq!(reused.input_telemetry, None); + assert_eq!(input_cache::load_input_status(&storage_path), status_after_acquisition); + assert_requested_actions( + server.finish(), + &[ + "login", + "get_live_categories", + "get_live_streams", + "get_vod_categories", + "get_vod_streams", + "get_series_categories", + "get_series", + ], + ); + } + + #[tokio::test] + async fn in_run_reuse_after_force_keeps_provider_acquisition_and_snapshot() { + assert_provider_acquisition_survives_in_run_reuse(InputRefreshPolicy::FORCE).await; + } + + #[tokio::test] + async fn in_run_reuse_after_refresh_keeps_provider_acquisition_and_snapshot() { + assert_provider_acquisition_survives_in_run_reuse(InputRefreshPolicy::REFRESH).await; + } + + #[tokio::test] + async fn in_run_reuse_after_normal_provider_fetch_keeps_provider_acquisition_and_snapshot() { + assert_provider_acquisition_survives_in_run_reuse(InputRefreshPolicy::NORMAL).await; + } + + #[tokio::test] + async fn in_run_reuse_for_parallel_sources_emits_exactly_one_provider_acquisition() { + let temp = tempfile::tempdir().expect("temporary storage"); + let server = TestXtreamServer::start([2, 2, 2]); + let mut ctx = processing_context(temp.path()); + let input = test_input(&server.base_url, ConfigInputUpdateQualityDto { live: 95, vod: 95, series: 95 }, 0, &[]); + seed_baseline(&ctx, &input).await; + set_refresh_policy(&mut ctx, &input, InputRefreshPolicy::FORCE); + + let (mut first_source, mut second_source) = + tokio::join!(download_input(&ctx, &input, false), download_input(&ctx, &input, false)); + + assert!(first_source.errors.is_empty(), "first source failed: {:?}", first_source.errors); + assert!(second_source.errors.is_empty(), "second source failed: {:?}", second_source.errors); + assert_eq!(first_source.source.get_channel_count(), 6); + assert_eq!(second_source.source.get_channel_count(), 6); + let acquisition_telemetry = [first_source.input_telemetry.as_ref(), second_source.input_telemetry.as_ref()]; + assert_eq!(acquisition_telemetry.iter().filter(|telemetry| telemetry.is_some()).count(), 1); + let acquisition = acquisition_telemetry.into_iter().flatten().next().expect("one provider acquisition"); + assert_eq!(acquisition.refresh_policy, InputRefreshPolicy::FORCE); + assert!(acquisition + .clusters + .iter() + .all(|cluster| { cluster.requested && cluster.source == Some(PlaylistUpdateDataSource::Provider) })); + assert_requested_actions( + server.finish(), + &[ + "login", + "get_live_categories", + "get_live_streams", + "get_vod_categories", + "get_vod_streams", + "get_series_categories", + "get_series", + ], + ); + } + + #[tokio::test] + async fn refresh_policy_bypasses_valid_cache_and_keeps_quality_enforced() { + let temp = tempfile::tempdir().expect("temporary storage"); + let server = TestXtreamServer::start([2, 1, 2]); + let mut ctx = processing_context(temp.path()); + let input = + test_input(&server.base_url, ConfigInputUpdateQualityDto { live: 100, vod: 100, series: 100 }, 3_600, &[]); + seed_baseline(&ctx, &input).await; + seed_valid_cluster_cache(&ctx, &input).await; + set_refresh_policy(&mut ctx, &input, InputRefreshPolicy::REFRESH); + + let mut result = download_input(&ctx, &input, false).await; + let requests = server.finish(); + + assert!(result.errors.is_empty()); + assert!(result.storage_error.is_none()); + assert_eq!(result.job_state(), InputJobState::Ready); + assert_eq!(result.quality_rejections.len(), 1); + assert_eq!(result.quality_rejections[0].cluster, XtreamCluster::Video); + let groups = result.source.take_groups(); + assert_eq!( + groups.iter().find(|group| group.xtream_cluster == XtreamCluster::Video).map(|group| group.title.as_ref()), + Some("old-vod") + ); + assert_requested_actions( + requests, + &[ + "login", + "get_live_categories", + "get_live_streams", + "get_vod_categories", + "get_vod_streams", + "get_series_categories", + "get_series", + ], + ); + } + + #[tokio::test] + async fn force_policy_bypasses_valid_cache_and_quality_and_accepts_an_empty_cluster() { + let temp = tempfile::tempdir().expect("temporary storage"); + let server = TestXtreamServer::start([2, 0, 1]); + let events = CollectSink::default(); + let mut ctx = processing_context_with_events(temp.path(), events.clone()); + let input = + test_input(&server.base_url, ConfigInputUpdateQualityDto { live: 100, vod: 95, series: 100 }, 3_600, &[]); + seed_baseline(&ctx, &input).await; + seed_valid_cluster_cache(&ctx, &input).await; + set_refresh_policy(&mut ctx, &input, InputRefreshPolicy::FORCE); + + let mut result = process_input_job_inner(0, &ctx, &input).await; + let requests = server.finish(); + + assert!(result.errors.is_empty(), "forced update failed: {:?}", result.errors); + assert_eq!(result.state, InputJobState::Ready); + let groups = result.source.take().expect("forced input source").take_groups(); + assert_eq!( + groups + .iter() + .filter(|group| group.xtream_cluster == XtreamCluster::Live) + .map(|group| group.channels.len()) + .sum::(), + 2 + ); + assert!(groups.iter().all(|group| group.xtream_cluster != XtreamCluster::Video)); + assert_eq!( + groups + .iter() + .filter(|group| group.xtream_cluster == XtreamCluster::Series) + .map(|group| group.channels.len()) + .sum::(), + 1 + ); + assert_eq!( + PlaylistRunSignals { + has_error: !result.errors.is_empty(), + has_pending_stalker_refresh: ctx.partial_refresh.load(Ordering::Acquire), + has_quality_rejections: ctx.had_quality_rejections.load(Ordering::Acquire), + } + .state(), + PlaylistUpdateState::Success + ); + + let storage_path = input_storage_path(&ctx, &input).await; + assert_eq!( + count_input_xtream_cluster(&ctx.config, &input, XtreamCluster::Video) + .await + .expect("forced empty VOD should remain countable"), + Some(0) + ); + let status = input_cache::load_input_status(&storage_path); + assert!(XTREAM_CLUSTER.iter().all(|cluster| { + status.clusters.get(cluster.as_ref()).map(|entry| &entry.status) == Some(&input_cache::ClusterState::Ok) + })); + let messages = events + .0 + .lock() + .expect("event sink lock") + .iter() + .filter_map(|event| match event { + EventMessage::PlaylistUpdateProgress(progress) => Some(progress.message.clone()), + _ => None, + }) + .collect::>(); + assert!(messages.iter().any(|message| { + message + == "Input 'quality-provider': forced update requested; cache and update-quality bypassed for live,vod,series" + })); + assert!(messages.iter().any(|message| { + message == "Input 'quality-provider' cluster 'vod' force-published: candidate=0 configured_threshold=95" + })); + assert_requested_actions( + requests, + &[ + "login", + "get_live_categories", + "get_live_streams", + "get_vod_categories", + "get_vod_streams", + "get_series_categories", + "get_series", + ], + ); + } + + #[tokio::test] + async fn force_empty_vod_is_published_to_xtream_target_and_memory_cache() { + let temp = tempfile::tempdir().expect("temporary storage"); + let server = TestXtreamServer::start([2, 0, 1]); + let mut ctx = processing_context(temp.path()); + let input = + test_input(&server.base_url, ConfigInputUpdateQualityDto { live: 100, vod: 95, series: 100 }, 0, &[]); + let target = xtream_target(true); + let playlist_state = Arc::new(PlaylistStorageState::new()); + ctx.playlist_state = Some(Arc::clone(&playlist_state)); + seed_baseline(&ctx, &input).await; + let old_vod_id = seed_xtream_target(&ctx, &target, &playlist_state).await; + install_source(&ctx, &input, &target); + set_refresh_policy(&mut ctx, &input, InputRefreshPolicy::FORCE); + let app_config = Arc::clone(&ctx.config); + let pending_refresh = Arc::clone(&ctx.partial_refresh); + let quality_rejections = Arc::clone(&ctx.had_quality_rejections); + + let (_, target_stats, errors) = process_source(0, Arc::new(ctx)).await; + server.finish(); + + assert!(errors.is_empty(), "forced target update failed: {errors:?}"); + assert_eq!(target_stats.len(), 1); + assert!(target_stats[0].success); + assert_eq!( + PlaylistRunSignals { + has_error: false, + has_pending_stalker_refresh: pending_refresh.load(Ordering::Acquire), + has_quality_rejections: quality_rejections.load(Ordering::Acquire), + } + .state(), + PlaylistUpdateState::Success + ); + assert_target_counts(&app_config, &target, [2, 0, 1]).await; + assert_empty_target_categories(&app_config, &target, &[XtreamCluster::Video]).await; + assert!( + xtream_get_item_for_stream_id( + old_vod_id, + &app_config, + &playlist_state, + &target, + Some(XtreamCluster::Video), + ) + .await + .is_err(), + "old VOD id must not remain client-visible" + ); + let cache = playlist_state.data.read().await; + let cached = + cache.get(&target.name).and_then(|storage| storage.xtream.as_ref()).expect("Xtream target memory cache"); + assert_eq!([cached.live.len(), cached.vod.len(), cached.series.len()], [2, 0, 1]); + } + + #[tokio::test] + async fn force_fully_empty_result_clears_the_xtream_target() { + let temp = tempfile::tempdir().expect("temporary storage"); + let server = TestXtreamServer::start([0, 0, 0]); + let mut ctx = processing_context(temp.path()); + let input = test_input(&server.base_url, ConfigInputUpdateQualityDto { live: 90, vod: 90, series: 90 }, 0, &[]); + let target = xtream_target(true); + let playlist_state = Arc::new(PlaylistStorageState::new()); + ctx.playlist_state = Some(Arc::clone(&playlist_state)); + seed_baseline(&ctx, &input).await; + let old_vod_id = seed_xtream_target(&ctx, &target, &playlist_state).await; + install_source(&ctx, &input, &target); + set_refresh_policy(&mut ctx, &input, InputRefreshPolicy::FORCE); + let app_config = Arc::clone(&ctx.config); + let pending_refresh = Arc::clone(&ctx.partial_refresh); + let quality_rejections = Arc::clone(&ctx.had_quality_rejections); + + let (_, target_stats, errors) = process_source(0, Arc::new(ctx)).await; + server.finish(); + + assert!(errors.is_empty(), "fully empty forced target update failed: {errors:?}"); + assert_eq!(target_stats.len(), 1); + assert!(target_stats[0].success); + assert_eq!( + PlaylistRunSignals { + has_error: false, + has_pending_stalker_refresh: pending_refresh.load(Ordering::Acquire), + has_quality_rejections: quality_rejections.load(Ordering::Acquire), + } + .state(), + PlaylistUpdateState::Success + ); + assert_target_counts(&app_config, &target, [0, 0, 0]).await; + assert_empty_target_categories(&app_config, &target, &XTREAM_CLUSTER).await; + assert!( + xtream_get_item_for_stream_id( + old_vod_id, + &app_config, + &playlist_state, + &target, + Some(XtreamCluster::Video), + ) + .await + .is_err(), + "fully empty target must not resolve an old VOD id" + ); + let cache = playlist_state.data.read().await; + let cached = + cache.get(&target.name).and_then(|storage| storage.xtream.as_ref()).expect("Xtream target memory cache"); + assert_eq!([cached.live.len(), cached.vod.len(), cached.series.len()], [0, 0, 0]); + } + + #[tokio::test] + async fn force_technical_parse_failure_retains_the_target_cluster() { + let temp = tempfile::tempdir().expect("temporary storage"); + let mut responses = fixture_responses([2, 1, 2]); + responses.insert("get_vod_streams".to_string(), "{".to_string()); + let server = TestXtreamServer::start_with_responses(responses); + let mut ctx = processing_context(temp.path()); + let input = + test_input(&server.base_url, ConfigInputUpdateQualityDto { live: 100, vod: 100, series: 100 }, 0, &[]); + let target = xtream_target(true); + let playlist_state = Arc::new(PlaylistStorageState::new()); + ctx.playlist_state = Some(Arc::clone(&playlist_state)); + seed_baseline(&ctx, &input).await; + let old_vod_id = seed_xtream_target(&ctx, &target, &playlist_state).await; + install_source(&ctx, &input, &target); + set_refresh_policy(&mut ctx, &input, InputRefreshPolicy::FORCE); + let app_config = Arc::clone(&ctx.config); + + let (_, target_stats, errors) = process_source(0, Arc::new(ctx)).await; + server.finish(); + + assert!(!errors.is_empty(), "malformed provider response must remain a technical error"); + assert_eq!(target_stats.len(), 1); + assert!(target_stats[0].success, "usable fallback should still be published"); + assert_target_counts(&app_config, &target, [2, 2, 2]).await; + let retained_vod = xtream_get_item_for_stream_id( + old_vod_id, + &app_config, + &playlist_state, + &target, + Some(XtreamCluster::Video), + ) + .await + .expect("old VOD should remain client-visible after technical failure"); + assert!(retained_vod.name.starts_with("old-vod")); + } + + #[tokio::test] + async fn normal_empty_candidate_retains_the_xtream_target_without_empty_authorization() { + let temp = tempfile::tempdir().expect("temporary storage"); + let server = TestXtreamServer::start([0, 0, 0]); + let mut ctx = processing_context(temp.path()); + let input = + test_input(&server.base_url, ConfigInputUpdateQualityDto { live: 100, vod: 100, series: 100 }, 0, &[]); + let target = xtream_target(true); + let playlist_state = Arc::new(PlaylistStorageState::new()); + ctx.playlist_state = Some(Arc::clone(&playlist_state)); + seed_baseline(&ctx, &input).await; + let old_vod_id = seed_xtream_target(&ctx, &target, &playlist_state).await; + install_source(&ctx, &input, &target); + let app_config = Arc::clone(&ctx.config); + let quality_rejections = Arc::clone(&ctx.had_quality_rejections); + + let (_, target_stats, errors) = process_source(0, Arc::new(ctx)).await; + server.finish(); + + assert!(errors.is_empty(), "quality rejection must not become a technical error: {errors:?}"); + assert!(quality_rejections.load(Ordering::Acquire)); + assert_eq!(target_stats.len(), 1); + assert!(target_stats[0].success); + assert_target_counts(&app_config, &target, [2, 2, 2]).await; + let retained_vod = xtream_get_item_for_stream_id( + old_vod_id, + &app_config, + &playlist_state, + &target, + Some(XtreamCluster::Video), + ) + .await + .expect("normal empty candidate must retain the target VOD"); + assert!(retained_vod.name.starts_with("old-vod")); + } + + #[tokio::test] + async fn force_policy_retains_a_cluster_after_a_technical_parse_failure() { + let temp = tempfile::tempdir().expect("temporary storage"); + let mut responses = fixture_responses([2, 1, 2]); + responses.insert("get_vod_streams".to_string(), "{".to_string()); + let server = TestXtreamServer::start_with_responses(responses); + let mut ctx = processing_context(temp.path()); + let input = + test_input(&server.base_url, ConfigInputUpdateQualityDto { live: 100, vod: 100, series: 100 }, 0, &[]); + seed_baseline(&ctx, &input).await; + set_refresh_policy(&mut ctx, &input, InputRefreshPolicy::FORCE); + + let mut result = download_input(&ctx, &input, false).await; + let requests = server.finish(); + + assert!(!result.errors.is_empty(), "malformed VOD response must remain a technical error"); + assert!(result.storage_error.is_none()); + assert!(result.quality_rejections.is_empty()); + let groups = result.source.take_groups(); + let vod = groups.iter().find(|group| group.xtream_cluster == XtreamCluster::Video).expect("retained VOD"); + assert_eq!(vod.title.as_ref(), "old-vod"); + assert_eq!(vod.channels.len(), 2); + assert_eq!( + groups.iter().find(|group| group.xtream_cluster == XtreamCluster::Live).map(|group| group.title.as_ref()), + Some("candidate-live") + ); + assert_eq!( + groups.iter().find(|group| group.xtream_cluster == XtreamCluster::Series).map(|group| group.title.as_ref()), + Some("candidate-series") + ); + assert_requested_actions( + requests, + &[ + "login", + "get_live_categories", + "get_live_streams", + "get_vod_categories", + "get_vod_streams", + "get_series_categories", + "get_series", + ], + ); + } + + #[tokio::test] + async fn in_memory_series_quality_compares_catalog_rows_with_an_enriched_baseline() { + let temp = tempfile::tempdir().expect("temporary storage"); + let server = TestXtreamServer::start([0, 0, 1]); + let ctx = processing_context(temp.path()); + let input = test_input( + &server.base_url, + ConfigInputUpdateQualityDto { series: 100, ..ConfigInputUpdateQualityDto::default() }, + 0, + &[XtreamCluster::Live, XtreamCluster::Video], + ); + let mut baseline = baseline_group(XtreamCluster::Series, 3, "old-series", 3_000); + baseline.channels.truncate(1); + let episodes = (0..200_u32) + .map(|id| SeriesStreamDetailEpisodeProperties { id, ..SeriesStreamDetailEpisodeProperties::default() }) + .collect(); + baseline.channels[0].header.additional_properties = + Some(StreamProperties::Series(Box::new(SeriesStreamProperties { + series_id: 3_000, + details: Some(SeriesStreamDetailProperties::new(None, Vec::new(), Some(episodes))), + ..SeriesStreamProperties::default() + }))); + let (persisted, error) = tuliprox_repository::persist_input_playlist(&ctx.config, &input, vec![baseline]).await; + assert!(error.is_none(), "enriched baseline persistence failed: {error:?}"); + assert_eq!(persisted.iter().map(|group| group.channels.len()).sum::(), 1); + assert_eq!( + count_input_xtream_cluster(&ctx.config, &input, XtreamCluster::Series) + .await + .expect("enriched baseline should be countable"), + Some(1) + ); + + let mut result = download_input(&ctx, &input, false).await; + let requests = server.finish(); + + assert!(result.errors.is_empty(), "Series update failed: {:?}", result.errors); + assert!(result.quality_rejections.is_empty()); + let groups = result.source.take_groups(); + assert_eq!(groups.len(), 1); + assert_eq!(groups[0].title.as_ref(), "candidate-series"); + assert_eq!(groups[0].channels.len(), 1); + let episode_count = groups[0].channels[0] + .header + .additional_properties + .as_ref() + .and_then(|properties| match properties { + StreamProperties::Series(series) => series.details.as_ref(), + _ => None, + }) + .and_then(|details| details.episodes.as_ref()) + .map(Vec::len); + assert_eq!(episode_count, Some(200)); + assert_requested_actions(requests, &["login", "get_series_categories", "get_series"]); + } + + #[tokio::test] + async fn in_memory_quality_rejection_counts_duplicate_provider_ids_once_and_retains_the_baseline() { + let temp = tempfile::tempdir().expect("temporary storage"); + let first_provider_id = 1_000_u32; + let streams = (0..50_u32) + .flat_map(|offset| { + let provider_id = first_provider_id + offset; + [ + live_fixture_stream(provider_id, 1, &format!("candidate-{provider_id}")), + live_fixture_stream(provider_id, 1, &format!("duplicate-{provider_id}")), + ] + }) + .collect(); + let responses = live_fixture_responses( + &serde_json::json!([{"category_id": 1, "category_name": "candidate-live"}]), + streams, + ); + let server = TestXtreamServer::start_with_responses(responses); + let ctx = processing_context(temp.path()); + let input = test_input( + &server.base_url, + ConfigInputUpdateQualityDto { live: 100, ..ConfigInputUpdateQualityDto::default() }, + 0, + &[XtreamCluster::Video, XtreamCluster::Series], + ); + seed_live_baseline(&ctx, &input, first_provider_id).await; + + let mut result = download_input(&ctx, &input, false).await; + let requests = server.finish(); + + assert!(result.errors.is_empty(), "duplicate rejection must not become an error: {:?}", result.errors); + assert!(result.storage_error.is_none()); + assert_eq!( + result.quality_rejections, + vec![ClusterUpdateRejection { + cluster: XtreamCluster::Live, + current_count: 100, + candidate_count: 50, + threshold: 100, + quality: 50, + }] + ); + let groups = result.source.take_groups(); + assert_eq!(groups.iter().map(|group| group.channels.len()).sum::(), 100); + assert_eq!(groups[0].title.as_ref(), "old-live"); + + let mut reloaded = load_input_playlist(&ctx.config, &input, None).await.expect("reload retained Live baseline"); + let reloaded_groups = reloaded.take_groups(); + assert_eq!(reloaded_groups.iter().map(|group| group.channels.len()).sum::(), 100); + assert_eq!(reloaded_groups[0].title.as_ref(), "old-live"); + assert_eq!( + count_input_xtream_cluster(&ctx.config, &input, XtreamCluster::Live) + .await + .expect("retained baseline should be countable"), + Some(100) + ); + assert_requested_actions(requests, &["login", "get_live_categories", "get_live_streams"]); + } + + #[tokio::test] + async fn in_memory_quality_accepts_unique_population_and_persists_the_last_duplicate_row() { + let temp = tempfile::tempdir().expect("temporary storage"); + let first_provider_id = 1_000_u32; + let winning_provider_id = first_provider_id + 42; + let mut streams = (0..100_u32) + .map(|offset| { + let provider_id = first_provider_id + offset; + live_fixture_stream(provider_id, 1, &format!("candidate-{provider_id}")) + }) + .collect::>(); + streams.push(live_fixture_stream(winning_provider_id, 2, "winning-duplicate")); + let responses = live_fixture_responses( + &serde_json::json!([ + {"category_id": 1, "category_name": "candidate-live"}, + {"category_id": 2, "category_name": "winning-category"} + ]), + streams, + ); + let server = TestXtreamServer::start_with_responses(responses); + let ctx = processing_context(temp.path()); + let input = test_input( + &server.base_url, + ConfigInputUpdateQualityDto { live: 100, ..ConfigInputUpdateQualityDto::default() }, + 0, + &[XtreamCluster::Video, XtreamCluster::Series], + ); + seed_live_baseline(&ctx, &input, first_provider_id).await; + + let mut result = download_input(&ctx, &input, false).await; + let requests = server.finish(); + + assert!(result.errors.is_empty(), "deduplicated acceptance failed: {:?}", result.errors); + assert!(result.storage_error.is_none()); + assert!(result.quality_rejections.is_empty()); + let groups = result.source.take_groups(); + assert_eq!(groups.iter().map(|group| group.channels.len()).sum::(), 100); + let winner_group = groups.iter().find(|group| group.id == 2).expect("winning category"); + assert_eq!(winner_group.title.as_ref(), "winning-category"); + assert_eq!(winner_group.channels.len(), 1); + assert_eq!(winner_group.channels[0].header.id.as_ref(), winning_provider_id.to_string()); + assert_eq!(winner_group.channels[0].header.name.as_ref(), "winning-duplicate"); + assert_eq!(winner_group.channels[0].header.category_id, 2); + + let mut reloaded = + load_input_playlist(&ctx.config, &input, None).await.expect("reload accepted Live candidate"); + let reloaded_groups = reloaded.take_groups(); + assert_eq!(reloaded_groups.iter().map(|group| group.channels.len()).sum::(), 100); + let reloaded_winner = reloaded_groups.iter().find(|group| group.id == 2).expect("persisted winning category"); + assert_eq!(reloaded_winner.channels.len(), 1); + assert_eq!(reloaded_winner.channels[0].header.id.as_ref(), winning_provider_id.to_string()); + assert_eq!(reloaded_winner.channels[0].header.name.as_ref(), "winning-duplicate"); + assert_eq!(reloaded_winner.channels[0].header.category_id, 2); + assert_eq!( + count_input_xtream_cluster(&ctx.config, &input, XtreamCluster::Live) + .await + .expect("accepted candidate should be countable"), + Some(100) + ); + assert_requested_actions(requests, &["login", "get_live_categories", "get_live_streams"]); + } + + #[tokio::test] + async fn m3u_update_quality_enforces_the_existing_cluster_threshold() { + let temp = tempfile::tempdir().expect("temporary storage"); + let playlist_path = temp.path().join("input.m3u"); + tokio::fs::write( + &playlist_path, + "#EXTM3U\n#EXTINF:-1,Old One\nhttp://old.example/1\n#EXTINF:-1,Old Two\nhttp://old.example/2\n", + ) + .await + .expect("initial M3U fixture"); + let options = ConfigInputOptionsDto { + update_quality: ConfigInputUpdateQualityDto { live: 100, vod: 0, series: 0 }, + ..ConfigInputOptionsDto::default() + }; + let input = Arc::new(ConfigInput { + id: 2, + name: "m3u-quality-guard".intern(), + input_type: InputType::M3u, + url: playlist_path.to_string_lossy().into_owned(), + enabled: true, + options: Some(ConfigInputOptions::from(&options)), + ..ConfigInput::default() + }); + let initial_ctx = processing_context(temp.path()); + + let mut initial = download_input(&initial_ctx, &input, false).await; + + assert!(initial.errors.is_empty(), "initial M3U download failed: {:?}", initial.errors); + assert!(initial.quality_rejections.is_empty()); + assert_eq!(initial.source.get_channel_count(), 2); + + tokio::fs::write(&playlist_path, "#EXTM3U\n#EXTINF:-1,New\nhttp://new.example/1\n") + .await + .expect("replacement M3U fixture"); + let refreshed_ctx = processing_context(temp.path()); + let mut refreshed = download_input(&refreshed_ctx, &input, false).await; + + assert!(refreshed.errors.is_empty(), "replacement M3U download failed: {:?}", refreshed.errors); + assert!(refreshed.storage_error.is_none()); + assert_eq!(refreshed.quality_rejections.len(), 1); + assert_eq!(refreshed.quality_rejections[0].cluster, XtreamCluster::Live); + assert_eq!(refreshed.source.get_channel_count(), 2); + let mut reloaded = load_input_playlist(&refreshed_ctx.config, &input, None) + .await + .expect("replacement M3U should be persisted"); + assert_eq!(reloaded.get_channel_count(), 2); + } + + #[tokio::test] + async fn quality_rejection_for_only_requested_vod_loads_complete_persisted_input() { + let temp = tempfile::tempdir().expect("temporary storage"); + let server = TestXtreamServer::start([0, 1, 0]); + let ctx = processing_context(temp.path()); + let input = test_input( + &server.base_url, + ConfigInputUpdateQualityDto { vod: 100, ..ConfigInputUpdateQualityDto::default() }, + 3_600, + &[], + ); + seed_baseline(&ctx, &input).await; + let storage_path = input_storage_path(&ctx, &input).await; + let now = SystemTime::now().duration_since(UNIX_EPOCH).expect("current time").as_secs(); + let mut status = input_cache::InputStatus::default(); + status.clusters.insert( + XtreamCluster::Live.as_ref().to_string(), + input_cache::ClusterStatus { status: input_cache::ClusterState::Ok, timestamp: now, last_update: None }, + ); + status.clusters.insert( + XtreamCluster::Video.as_ref().to_string(), + input_cache::ClusterStatus { + status: input_cache::ClusterState::Ok, + timestamp: now.saturating_sub(3_601), + last_update: None, + }, + ); + status.clusters.insert( + XtreamCluster::Series.as_ref().to_string(), + input_cache::ClusterStatus { status: input_cache::ClusterState::Ok, timestamp: now, last_update: None }, + ); + input_cache::save_input_status(&storage_path, &status); + + let mut result = download_input(&ctx, &input, false).await; + let requests = server.finish(); + + assert!(result.errors.is_empty(), "quality rejection must not become an error: {:?}", result.errors); + assert!(result.storage_error.is_none()); + assert!(!result.partial); + assert_eq!(result.job_state(), InputJobState::Ready); + assert_eq!(result.quality_rejections.len(), 1); + assert_eq!( + result.quality_rejections[0], + ClusterUpdateRejection { + cluster: XtreamCluster::Video, + current_count: 2, + candidate_count: 1, + threshold: 100, + quality: 50, + } + ); + assert_eq!( + PlaylistRunSignals { + has_quality_rejections: !result.quality_rejections.is_empty(), + ..PlaylistRunSignals::default() + } + .state(), + PlaylistUpdateState::Partial + ); + assert_baseline(&result.source.take_groups()); + + let status = input_cache::load_input_status(&storage_path); + let live = status.clusters.get(XtreamCluster::Live.as_ref()).expect("Live cache status"); + let vod = status.clusters.get(XtreamCluster::Video.as_ref()).expect("VOD cache status"); + let series = status.clusters.get(XtreamCluster::Series.as_ref()).expect("Series cache status"); + assert_eq!(live.status, input_cache::ClusterState::Ok); + assert_eq!(live.timestamp, now); + assert_eq!(vod.status, input_cache::ClusterState::Failed); + assert_eq!(series.status, input_cache::ClusterState::Ok); + assert_eq!(series.timestamp, now); + assert_requested_actions(requests, &["login", "get_vod_categories", "get_vod_streams"]); + } + + #[tokio::test] + async fn all_requested_clusters_rejected_load_the_complete_unchanged_baseline() { + let temp = tempfile::tempdir().expect("temporary storage"); + let server = TestXtreamServer::start([1, 1, 1]); + let ctx = processing_context(temp.path()); + let input = + test_input(&server.base_url, ConfigInputUpdateQualityDto { live: 100, vod: 100, series: 100 }, 0, &[]); + seed_baseline(&ctx, &input).await; + + let mut result = download_input(&ctx, &input, false).await; + let requests = server.finish(); + + assert!(result.errors.is_empty(), "quality rejections must not become errors: {:?}", result.errors); + assert!(result.storage_error.is_none()); + assert_eq!(result.job_state(), InputJobState::Ready); + assert_eq!( + result.quality_rejections.iter().map(|rejection| rejection.cluster).collect::>(), + XTREAM_CLUSTER + ); + assert!(result.quality_rejections.iter().all(|rejection| { + rejection.current_count == 2 + && rejection.candidate_count == 1 + && rejection.threshold == 100 + && rejection.quality == 50 + })); + assert_baseline(&result.source.take_groups()); + + let mut reloaded = load_input_playlist(&ctx.config, &input, None).await.expect("reload persisted baseline"); + assert_baseline(&reloaded.take_groups()); + let storage_path = input_storage_path(&ctx, &input).await; + let status = input_cache::load_input_status(&storage_path); + assert!(XTREAM_CLUSTER.iter().all(|cluster| { + status.clusters.get(cluster.as_ref()).map(|entry| &entry.status) == Some(&input_cache::ClusterState::Failed) + })); + assert_requested_actions( + requests, + &[ + "login", + "get_live_categories", + "get_live_streams", + "get_vod_categories", + "get_vod_streams", + "get_series_categories", + "get_series", + ], + ); + } + + #[tokio::test] + async fn quality_rejection_for_empty_bootstrap_does_not_invent_a_baseline() { + let temp = tempfile::tempdir().expect("temporary storage"); + let server = TestXtreamServer::start([0, 0, 0]); + let ctx = processing_context(temp.path()); + let input = test_input( + &server.base_url, + ConfigInputUpdateQualityDto { vod: 100, ..ConfigInputUpdateQualityDto::default() }, + 0, + &[XtreamCluster::Live, XtreamCluster::Series], + ); + + let mut result = download_input(&ctx, &input, false).await; + let requests = server.finish(); + + assert!(result.errors.is_empty(), "unexpected bootstrap errors: {:?}", result.errors); + assert!(result.storage_error.is_none()); + assert_eq!(result.quality_rejections.len(), 1); + assert_eq!( + result.quality_rejections[0], + ClusterUpdateRejection { + cluster: XtreamCluster::Video, + current_count: 0, + candidate_count: 0, + threshold: 100, + quality: 0, + } + ); + assert!(result.source.is_empty()); + assert_eq!(result.job_state(), InputJobState::Failed); + let mut reloaded = load_input_playlist(&ctx.config, &input, None).await.expect("empty bootstrap reload"); + assert!(reloaded.take_groups().is_empty()); + assert_requested_actions(requests, &["login", "get_vod_categories", "get_vod_streams"]); + } +} + #[cfg(test)] mod disk_epg_wireup_tests { use super::spill_epg_to_disk; diff --git a/backend/processing/src/processor/providers.rs b/backend/processing/src/processor/providers.rs index 311ca244d..786994e12 100644 --- a/backend/processing/src/processor/providers.rs +++ b/backend/processing/src/processor/providers.rs @@ -41,16 +41,16 @@ impl PlaylistProvider for StalkerProvider { fn name(&self) -> &'static str { "stalker" } async fn fetch(&self, request: &PlaylistFetchRequest<'_>) -> PlaylistFetch { - let (groups, errors, persisted, partial) = download_stalker_playlist( + download_stalker_playlist( request.app_config, request.client, request.input, None, self.refresh_mode, self.materialize_active, + request.update_quality, ) - .await; - PlaylistFetch::groups(groups).with_errors(errors).persisted(persisted).partial(partial) + .await } } diff --git a/backend/processing/src/processor/stalker.rs b/backend/processing/src/processor/stalker.rs index d249c8e0c..c61e95439 100644 --- a/backend/processing/src/processor/stalker.rs +++ b/backend/processing/src/processor/stalker.rs @@ -10,15 +10,15 @@ #![allow(clippy::too_many_lines, clippy::needless_pass_by_value)] -use super::stalker_refresh::{ - advance_stalker_refresh, StalkerClusterSelection, StalkerRefreshMode, StalkerRefreshOutcome, -}; +use super::stalker_refresh::{advance_stalker_refresh, StalkerRefreshMode, StalkerRefreshOutcome, StalkerRefreshPlan}; use log::{debug, info, warn}; use lru::LruCache; use parking_lot::Mutex; use shared::{ error::TuliproxError, - model::{stalker::StalkerStreamKind, stalker_item::StalkerPlaylistItem, PlaylistGroup, PlaylistItem}, + model::{ + stalker::StalkerStreamKind, stalker_item::StalkerPlaylistItem, PlaylistGroup, PlaylistItem, UpdateQualityPolicy, + }, utils::Internable, }; use std::{ @@ -29,7 +29,10 @@ use std::{ time::{Duration, Instant}, }; use tuliprox_core::model::{AppConfig, ConfigInput, ConfigInputFlags, StalkerInputConfig}; -use tuliprox_iptv::stalker::{client::StalkerApiClient, error::StalkerError, parser}; +use tuliprox_iptv::{ + provider::PlaylistFetch, + stalker::{client::StalkerApiClient, error::StalkerError, parser}, +}; use tuliprox_repository::{ stalker_generation_repository::{load_active_manifest, load_checkpoint}, stalker_repository::{ensure_stalker_storage_path, load_stalker_items_at, read_stalker_item_at}, @@ -87,6 +90,24 @@ async fn try_acquire_stalker_refresh(input_name: &Arc) -> Option PlaylistFetch { + if matches!(quality_policy, UpdateQualityPolicy::Bypass) { + PlaylistFetch::failed(TuliproxError::ProviderConnection(format!( + "Stalker refresh for input '{input_name}' is busy; retry the forced update" + ))) + .persisted(persisted) + } else { + PlaylistFetch::nothing_to_do().persisted(persisted).partial(true) + } +} + +fn failed_stalker_acquisition(error: TuliproxError, clusters: &[StalkerCluster]) -> PlaylistFetch { + PlaylistFetch { + failed_clusters: clusters.iter().copied().map(xtream_cluster).collect(), + ..PlaylistFetch::failed(error) + } +} + fn cached_resolved_link(key: RuntimeLinkKey, force_refresh: bool) -> Option> { let mut cache = RUNTIME_STALKER_LINKS.lock(); let expired = cache.peek(&key).is_some_and(|entry| entry.expires_at <= Instant::now()); @@ -111,19 +132,15 @@ pub async fn download_stalker_playlist( clusters: Option<&[StalkerCluster]>, refresh_mode: StalkerRefreshMode, materialize_active: bool, -) -> (Vec, Vec, bool, bool) { + quality_policy: UpdateQualityPolicy, +) -> PlaylistFetch { let stalker_cfg = match input.stalker.as_ref() { Some(cfg) => cfg.clone(), None => { - return ( - vec![], - vec![TuliproxError::ConfigInput(format!( - "Stalker input '{}' has no stalker configuration block", - input.name - ))], - false, - false, - ); + return PlaylistFetch::failed(TuliproxError::ConfigInput(format!( + "Stalker input '{}' has no stalker configuration block", + input.name + ))); } }; @@ -136,27 +153,22 @@ pub async fn download_stalker_playlist( if resolved_clusters.is_empty() { info!("Stalker input '{}' has all clusters skipped", input.name); - return (vec![], vec![], false, false); + return PlaylistFetch::nothing_to_do(); } - let refresh_selection = StalkerClusterSelection::requested(input, &resolved_clusters); + let refresh_plan = StalkerRefreshPlan::requested(input, &resolved_clusters, quality_policy); let portal_url = match resolve_stalker_portal_url(input) { Ok(url) => url, - Err(err) => return (vec![], vec![err], false, false), + Err(err) => return failed_stalker_acquisition(err, &resolved_clusters), }; let identity_fingerprint = stalker_identity_fingerprint(&portal_url, &stalker_cfg); let api_client = match cached_runtime_stalker_client(client, portal_url, &stalker_cfg) { Ok(client) => client, Err(err) => { - return ( - vec![], - vec![TuliproxError::ConfigInput(format!( - "failed to build Stalker client for input '{}': {err}", - input.name - ))], - false, - false, + return failed_stalker_acquisition( + TuliproxError::ConfigInput(format!("failed to build Stalker client for input '{}': {err}", input.name)), + &resolved_clusters, ); } }; @@ -164,11 +176,9 @@ pub async fn download_stalker_playlist( let storage_path = match ensure_stalker_storage_path(app_config, &input.name).await { Ok(p) => p, Err(err) => { - return ( - vec![], - vec![TuliproxError::Io(format!("could not prepare Stalker storage for input '{}': {err}", input.name))], - false, - false, + return failed_stalker_acquisition( + TuliproxError::Io(format!("could not prepare Stalker storage for input '{}': {err}", input.name)), + &resolved_clusters, ); } }; @@ -178,14 +188,12 @@ pub async fn download_stalker_playlist( let handshake = match api_client.handshake().await { Ok(handshake) => handshake, Err(err) => { - return ( - vec![], - vec![TuliproxError::ProviderConnection(format!( + return failed_stalker_acquisition( + TuliproxError::ProviderConnection(format!( "Stalker handshake for input '{}' failed: {err}", input.name - ))], - false, - false, + )), + &resolved_clusters, ); } }; @@ -194,7 +202,7 @@ pub async fn download_stalker_playlist( app_config, api_client.as_ref(), &handshake, - refresh_selection, + refresh_plan, &storage_path, identity_fingerprint, refresh_mode.budget(), @@ -205,7 +213,7 @@ pub async fn download_stalker_playlist( } else { let checkpoint = match load_checkpoint(&storage_path, identity_fingerprint).await { Ok(checkpoint) => checkpoint, - Err(err) => return (Vec::new(), vec![err], false, false), + Err(err) => return PlaylistFetch::failed(err), }; break StalkerRefreshOutcome::Yielded { phase: checkpoint.as_ref().map_or( @@ -227,18 +235,40 @@ pub async fn download_stalker_playlist( tokio::task::yield_now().await; } Ok(outcome) => break outcome, - Err(err) => return (Vec::new(), vec![err], false, false), + Err(err) => return PlaylistFetch::failed(err), } } } else { - return (Vec::new(), Vec::new(), app_config.config.load().disk_based_processing, true); + return stalker_refresh_busy_fetch(&input.name, app_config.config.load().disk_based_processing, quality_policy); }; let yielded = matches!(&outcome, StalkerRefreshOutcome::Yielded { .. }); let terminal = matches!(&outcome, StalkerRefreshOutcome::Terminal(_)); let mut errors = Vec::new(); + let mut quality_acceptances = Vec::new(); + let mut quality_rejections = Vec::new(); + let mut force_updates = Vec::new(); match outcome { - StalkerRefreshOutcome::Complete => {} + StalkerRefreshOutcome::Complete { + quality_acceptances: completed_acceptances, + quality_rejections: completed_rejections, + force_updates: completed_force_updates, + } => { + quality_acceptances = completed_acceptances; + quality_rejections = completed_rejections; + force_updates = completed_force_updates; + } + StalkerRefreshOutcome::Failed { + quality_acceptances: completed_acceptances, + quality_rejections: completed_rejections, + force_updates: completed_force_updates, + error, + } => { + return PlaylistFetch::failed(error) + .with_quality_acceptances(completed_acceptances) + .with_quality_rejections(completed_rejections) + .with_force_updates(completed_force_updates); + } StalkerRefreshOutcome::Yielded { phase, processed, skipped, error } => { info!( "Stalker input '{}' yielded in phase {phase:?} after {processed} records ({skipped} skipped)", @@ -255,7 +285,12 @@ pub async fn download_stalker_playlist( Ok(manifest) => manifest, Err(err) => { errors.push(err); - return (Vec::new(), errors, false, yielded); + return PlaylistFetch::groups(Vec::new()) + .with_errors(errors) + .with_quality_acceptances(quality_acceptances) + .with_quality_rejections(quality_rejections) + .with_force_updates(force_updates) + .partial(yielded); } }; if terminal { @@ -268,7 +303,13 @@ pub async fn download_stalker_playlist( } let use_disk_based_processing = app_config.config.load().disk_based_processing; if !materialize_active { - return (Vec::new(), errors, use_disk_based_processing, yielded); + return PlaylistFetch::groups(Vec::new()) + .with_errors(errors) + .with_quality_acceptances(quality_acceptances) + .with_quality_rejections(quality_rejections) + .with_force_updates(force_updates) + .persisted(use_disk_based_processing) + .partial(yielded); } let mut groups = Vec::new(); let mut counts = [0_usize; 3]; @@ -324,7 +365,13 @@ pub async fn download_stalker_playlist( } } parser::log_stalker_download_summary(&input.name, counts[0], counts[1], counts[2]); - (groups, errors, use_disk_based_processing, yielded) + PlaylistFetch::groups(groups) + .with_errors(errors) + .with_quality_acceptances(quality_acceptances) + .with_quality_rejections(quality_rejections) + .with_force_updates(force_updates) + .persisted(use_disk_based_processing) + .partial(yielded) } /// Resolve a per-input portal URL using the standard `provider://` resolution pipeline. @@ -584,4 +631,31 @@ mod tests { drop(first); assert!(try_acquire_stalker_refresh(&name).await.is_some()); } + + #[tokio::test] + async fn force_stalker_refresh_busy_is_visible_and_does_not_mutate_checkpoint() { + use tuliprox_repository::stalker_generation_repository::{save_checkpoint, StalkerCheckpoint}; + + let temp = tempfile::tempdir().expect("temporary Stalker storage"); + let input_name: Arc = "force-busy-portal".into(); + let identity = 17; + save_checkpoint(temp.path(), &StalkerCheckpoint::new(identity, 23, 0b1111, 123)) + .await + .expect("checkpoint should persist"); + + let refresh_permit = try_acquire_stalker_refresh(&input_name).await.expect("first refresh owns permit"); + let fetch = stalker_refresh_busy_fetch(&input_name, true, UpdateQualityPolicy::Bypass); + + assert_eq!(fetch.errors.len(), 1); + assert!(!fetch.partial); + assert!(fetch.persisted); + let checkpoint = load_checkpoint(temp.path(), identity) + .await + .expect("checkpoint should load") + .expect("checkpoint should remain present"); + assert_eq!(checkpoint.quality_bypass_mask, 0); + + drop(refresh_permit); + assert!(try_acquire_stalker_refresh(&input_name).await.is_some()); + } } diff --git a/backend/processing/src/processor/stalker_refresh.rs b/backend/processing/src/processor/stalker_refresh.rs index 76ec317fe..dd9167f03 100644 --- a/backend/processing/src/processor/stalker_refresh.rs +++ b/backend/processing/src/processor/stalker_refresh.rs @@ -1,7 +1,7 @@ use super::stalker::StalkerCluster; use shared::{ error::TuliproxError, - model::{stalker::StalkerStreamKind, stalker_item::StalkerPlaylistItem}, + model::{stalker::StalkerStreamKind, stalker_item::StalkerPlaylistItem, UpdateQualityPolicy, XtreamCluster}, }; use std::{ collections::HashMap, @@ -9,7 +9,10 @@ use std::{ sync::Arc, time::{Instant, SystemTime, UNIX_EPOCH}, }; -use tuliprox_core::model::{AppConfig, ConfigInput, ConfigInputFlags}; +use tuliprox_core::model::{ + evaluate_update_quality, AppConfig, ClusterForceUpdate, ClusterUpdateAcceptance, ClusterUpdateRejection, + ConfigInput, ConfigInputFlags, ConfigInputUpdateQuality, +}; use tuliprox_iptv::stalker::{ catalog::{StalkerCategory, StalkerRawItem}, client::StalkerApiClient, @@ -23,13 +26,19 @@ use tuliprox_repository::{ publish_selection, save_checkpoint, StalkerCheckpoint, StalkerGenerationData, StalkerRefreshPhase, }, stalker_repository::{ - load_stalker_items_after, prepare_stalker_episode_series_at, promote_stalker_file, remove_stalker_file, - snapshot_stalker_epg_at, snapshot_stalker_items_at, upsert_stalker_epg_at, upsert_stalker_items_at, + count_stalker_items_at, load_stalker_items_after, prepare_stalker_episode_series_at, promote_stalker_file, + remove_stalker_file, snapshot_stalker_epg_at, snapshot_stalker_items_at, upsert_stalker_epg_at, + upsert_stalker_items_at, }, }; const MAX_RETRIES: u8 = 3; const SKIPPED_SAMPLE_LIMIT: usize = 32; +const LIVE_SELECTION: u8 = 0b0001; +const VOD_SELECTION: u8 = 0b0010; +const SERIES_SELECTION: u8 = 0b0100; +const EPG_SELECTION: u8 = 0b1000; +const MEDIA_SELECTION: u8 = LIVE_SELECTION | VOD_SELECTION | SERIES_SELECTION; pub enum StalkerRefreshLimit { Deadline(Instant), @@ -118,6 +127,26 @@ impl StalkerClusterSelection { } } +/// Requested clusters and the publication policy applied once their generation is complete. +#[derive(Clone, Copy)] +pub struct StalkerRefreshPlan { + selection: StalkerClusterSelection, + update_quality: ConfigInputUpdateQuality, + quality_bypass_mask: u8, +} + +impl StalkerRefreshPlan { + pub fn requested(input: &ConfigInput, requested: &[StalkerCluster], quality_policy: UpdateQualityPolicy) -> Self { + let selection = StalkerClusterSelection::requested(input, requested); + let update_quality = + input.options.as_ref().map_or_else(ConfigInputUpdateQuality::default, |options| options.update_quality); + let quality_bypass_mask = matches!(quality_policy, UpdateQualityPolicy::Bypass) + .then_some(selection.mask() & MEDIA_SELECTION) + .unwrap_or(0); + Self { selection, update_quality, quality_bypass_mask } + } +} + fn first_phase(selection: StalkerClusterSelection) -> StalkerRefreshPhase { if selection.live { StalkerRefreshPhase::LiveBulk @@ -154,8 +183,23 @@ fn next_phase_after_series(selection: StalkerClusterSelection) -> StalkerRefresh #[derive(Debug)] pub enum StalkerRefreshOutcome { - Complete, - Yielded { phase: StalkerRefreshPhase, processed: u64, skipped: u64, error: Option }, + Complete { + quality_acceptances: Vec, + quality_rejections: Vec, + force_updates: Vec, + }, + Failed { + quality_acceptances: Vec, + quality_rejections: Vec, + force_updates: Vec, + error: TuliproxError, + }, + Yielded { + phase: StalkerRefreshPhase, + processed: u64, + skipped: u64, + error: Option, + }, Terminal(TuliproxError), } @@ -169,9 +213,15 @@ async fn load_or_start_checkpoint( storage_path: &Path, identity_fingerprint: u64, selection: StalkerClusterSelection, + quality_bypass_mask: u8, ) -> Result { - if let Some(state) = load_checkpoint(storage_path, identity_fingerprint).await? { + if let Some(mut state) = load_checkpoint(storage_path, identity_fingerprint).await? { if state.selection_mask == selection.mask() { + let combined_bypass_mask = state.quality_bypass_mask | quality_bypass_mask; + if combined_bypass_mask != state.quality_bypass_mask { + state.quality_bypass_mask = combined_bypass_mask; + save_checkpoint(storage_path, &state).await?; + } return Ok(state); } } @@ -179,30 +229,70 @@ async fn load_or_start_checkpoint( let mut state = StalkerCheckpoint::new(identity_fingerprint, generation_id(), selection.mask(), chrono::Utc::now().timestamp()); state.phase = first_phase(selection); + state.quality_bypass_mask = quality_bypass_mask; save_checkpoint(storage_path, &state).await?; Ok(state) } -async fn finish_completed_refresh_checked( +async fn evaluate_completed_refresh_publication( app_config: &Arc, storage_path: &Path, identity_fingerprint: u64, checkpoint: &StalkerCheckpoint, + update_quality: ConfigInputUpdateQuality, +) -> Result { + let needs_quality_evaluation = + selection_needs_quality_evaluation(checkpoint.selection_mask, checkpoint.quality_bypass_mask, update_quality); + let has_quality_bypass = checkpoint.quality_bypass_mask & MEDIA_SELECTION != 0; + if needs_quality_evaluation || has_quality_bypass { + let active_manifest = if needs_quality_evaluation { + Some(load_active_manifest(storage_path, identity_fingerprint).await?) + } else { + None + }; + evaluate_generation_publication( + app_config, + storage_path, + checkpoint.generation, + checkpoint.selection_mask, + checkpoint.quality_bypass_mask, + update_quality, + active_manifest.as_ref(), + ) + .await + } else { + Ok(StalkerGenerationPublication { + accepted_selection_mask: checkpoint.selection_mask, + quality_acceptances: Vec::new(), + quality_rejections: Vec::new(), + force_updates: Vec::new(), + }) + } +} + +async fn ensure_completed_refresh_has_usable_playlist( + app_config: &Arc, + storage_path: &Path, + identity_fingerprint: u64, + checkpoint: &StalkerCheckpoint, + accepted_selection_mask: u8, ) -> Result<(), TuliproxError> { - if checkpoint.processed == 0 { + let accepted_media_selection = accepted_selection_mask & MEDIA_SELECTION; + let accepted_enforced_selection = accepted_media_selection & !checkpoint.quality_bypass_mask; + if checkpoint.processed == 0 && accepted_enforced_selection != 0 { let active = load_active_manifest(storage_path, identity_fingerprint).await?; let mut retained_has_items = false; - if checkpoint.selection_mask & 0b0001 == 0 { + if accepted_selection_mask & LIVE_SELECTION == 0 { if let Some(files) = active.live.as_ref() { retained_has_items = !load_stalker_items_after(app_config, &files.data, None, 1).await?.is_empty(); } } - if !retained_has_items && checkpoint.selection_mask & 0b0010 == 0 { + if !retained_has_items && accepted_selection_mask & VOD_SELECTION == 0 { if let Some(files) = active.vod.as_ref() { retained_has_items = !load_stalker_items_after(app_config, &files.data, None, 1).await?.is_empty(); } } - if !retained_has_items && checkpoint.selection_mask & 0b0100 == 0 { + if !retained_has_items && accepted_selection_mask & SERIES_SELECTION == 0 { if let Some(files) = active.series.as_ref() { retained_has_items = !load_stalker_items_after(app_config, &files.roots, None, 1).await?.is_empty() || !load_stalker_items_after(app_config, &files.episodes, None, 1).await?.is_empty(); @@ -216,18 +306,242 @@ async fn finish_completed_refresh_checked( )); } } - finish_completed_refresh(storage_path, identity_fingerprint, checkpoint).await + Ok(()) } -async fn finish_completed_refresh( +async fn publish_completed_refresh( storage_path: &Path, identity_fingerprint: u64, checkpoint: &StalkerCheckpoint, + accepted_selection_mask: u8, ) -> Result<(), TuliproxError> { let manifest = - publish_selection(storage_path, identity_fingerprint, checkpoint.generation, checkpoint.selection_mask).await?; + publish_selection(storage_path, identity_fingerprint, checkpoint.generation, accepted_selection_mask).await?; clear_checkpoint(storage_path).await?; - cleanup_obsolete_generations(storage_path, &manifest).await + cleanup_obsolete_generations(storage_path, &manifest).await?; + Ok(()) +} + +struct StalkerRefreshReports { + quality_acceptances: Vec, + quality_rejections: Vec, + force_updates: Vec, + error: Option, +} + +impl StalkerRefreshReports { + fn into_outcome(self) -> StalkerRefreshOutcome { + let Self { quality_acceptances, quality_rejections, force_updates, error } = self; + if let Some(error) = error { + StalkerRefreshOutcome::Failed { quality_acceptances, quality_rejections, force_updates, error } + } else { + StalkerRefreshOutcome::Complete { quality_acceptances, quality_rejections, force_updates } + } + } +} + +#[cfg(test)] +async fn finish_completed_refresh( + app_config: &Arc, + storage_path: &Path, + identity_fingerprint: u64, + checkpoint: &StalkerCheckpoint, + update_quality: ConfigInputUpdateQuality, +) -> Result, TuliproxError> { + let publication = evaluate_completed_refresh_publication( + app_config, + storage_path, + identity_fingerprint, + checkpoint, + update_quality, + ) + .await?; + let StalkerGenerationPublication { + accepted_selection_mask, + quality_rejections, + quality_acceptances: _, + force_updates: _, + } = publication; + publish_completed_refresh(storage_path, identity_fingerprint, checkpoint, accepted_selection_mask).await?; + Ok(quality_rejections) +} + +async fn finish_completed_refresh_checked( + app_config: &Arc, + storage_path: &Path, + identity_fingerprint: u64, + checkpoint: &StalkerCheckpoint, + update_quality: ConfigInputUpdateQuality, +) -> Result { + let publication = evaluate_completed_refresh_publication( + app_config, + storage_path, + identity_fingerprint, + checkpoint, + update_quality, + ) + .await?; + let StalkerGenerationPublication { + accepted_selection_mask, + quality_acceptances, + quality_rejections, + force_updates, + } = publication; + let completion_result = async { + ensure_completed_refresh_has_usable_playlist( + app_config, + storage_path, + identity_fingerprint, + checkpoint, + accepted_selection_mask, + ) + .await?; + publish_completed_refresh(storage_path, identity_fingerprint, checkpoint, accepted_selection_mask).await + } + .await; + Ok(StalkerRefreshReports { quality_acceptances, quality_rejections, force_updates, error: completion_result.err() }) +} + +struct StalkerGenerationPublication { + accepted_selection_mask: u8, + quality_acceptances: Vec, + quality_rejections: Vec, + force_updates: Vec, +} + +fn selection_needs_quality_evaluation( + requested_selection_mask: u8, + quality_bypass_mask: u8, + update_quality: ConfigInputUpdateQuality, +) -> bool { + [StalkerCluster::Live, StalkerCluster::Vod, StalkerCluster::Series].into_iter().any(|cluster| { + let bit = selection_bit(cluster); + requested_selection_mask & bit != 0 + && quality_bypass_mask & bit == 0 + && update_quality.threshold(quality_cluster(cluster)) != 0 + }) +} + +const fn selection_bit(cluster: StalkerCluster) -> u8 { + match cluster { + StalkerCluster::Live => LIVE_SELECTION, + StalkerCluster::Vod => VOD_SELECTION, + StalkerCluster::Series => SERIES_SELECTION, + } +} + +const fn quality_cluster(cluster: StalkerCluster) -> XtreamCluster { + match cluster { + StalkerCluster::Live => XtreamCluster::Live, + StalkerCluster::Vod => XtreamCluster::Video, + StalkerCluster::Series => XtreamCluster::Series, + } +} + +async fn count_stalker_cluster_files( + app_config: &Arc, + primary: &Path, + secondary: Option<&Path>, +) -> Result { + let primary_count = count_stalker_items_at(app_config, primary).await?.unwrap_or_default(); + let secondary_count = match secondary { + Some(path) => count_stalker_items_at(app_config, path).await?.unwrap_or_default(), + None => 0, + }; + primary_count.checked_add(secondary_count).ok_or_else(|| { + TuliproxError::RepositoryStalker("Stalker cluster item count exceeds platform capacity".to_string()) + }) +} + +async fn count_candidate_cluster( + app_config: &Arc, + storage_path: &Path, + generation: u64, + cluster: StalkerCluster, +) -> Result { + let primary_data = match cluster { + StalkerCluster::Live => StalkerGenerationData::Live, + StalkerCluster::Vod => StalkerGenerationData::Vod, + StalkerCluster::Series => StalkerGenerationData::SeriesRoots, + }; + let primary = generation_data_path(storage_path, generation, primary_data); + let secondary = (cluster == StalkerCluster::Series) + .then(|| generation_data_path(storage_path, generation, StalkerGenerationData::SeriesEpisodes)); + count_stalker_cluster_files(app_config, &primary, secondary.as_deref()).await +} + +async fn count_active_cluster( + app_config: &Arc, + manifest: &tuliprox_repository::stalker_generation_repository::StalkerActiveManifest, + cluster: StalkerCluster, +) -> Result, TuliproxError> { + match cluster { + StalkerCluster::Live => match manifest.live.as_ref() { + Some(files) => count_stalker_cluster_files(app_config, &files.data, None).await.map(Some), + None => Ok(None), + }, + StalkerCluster::Vod => match manifest.vod.as_ref() { + Some(files) => count_stalker_cluster_files(app_config, &files.data, None).await.map(Some), + None => Ok(None), + }, + StalkerCluster::Series => match manifest.series.as_ref() { + Some(files) => count_stalker_cluster_files(app_config, &files.roots, Some(&files.episodes)).await.map(Some), + None => Ok(None), + }, + } +} + +async fn evaluate_generation_publication( + app_config: &Arc, + storage_path: &Path, + generation: u64, + requested_selection_mask: u8, + quality_bypass_mask: u8, + update_quality: ConfigInputUpdateQuality, + active_manifest: Option<&tuliprox_repository::stalker_generation_repository::StalkerActiveManifest>, +) -> Result { + let mut accepted_selection_mask = requested_selection_mask; + let mut quality_acceptances = Vec::new(); + let mut quality_rejections = Vec::new(); + let mut force_updates = Vec::new(); + for cluster in [StalkerCluster::Live, StalkerCluster::Vod, StalkerCluster::Series] { + let selection_bit = selection_bit(cluster); + if requested_selection_mask & selection_bit == 0 { + continue; + } + let report_cluster = quality_cluster(cluster); + let threshold = update_quality.threshold(report_cluster); + if quality_bypass_mask & selection_bit != 0 { + let candidate_count = count_candidate_cluster(app_config, storage_path, generation, cluster).await?; + force_updates.push(ClusterForceUpdate { + cluster: report_cluster, + candidate_count, + configured_threshold: threshold, + }); + continue; + } + if threshold == 0 { + continue; + } + let candidate_count = count_candidate_cluster(app_config, storage_path, generation, cluster).await?; + let Some(active_manifest) = active_manifest else { + return Err(TuliproxError::RepositoryStalker(format!( + "Missing active manifest for enforced {report_cluster} quality evaluation" + ))); + }; + let current_count = count_active_cluster(app_config, active_manifest, cluster).await?; + let decision = evaluate_update_quality(current_count, candidate_count, threshold); + if let Some(acceptance) = decision.acceptance(report_cluster) { + quality_acceptances.push(acceptance); + } else if let Some(rejection) = decision.rejection(report_cluster) { + accepted_selection_mask &= !selection_bit; + if cluster == StalkerCluster::Live { + accepted_selection_mask &= !EPG_SELECTION; + } + quality_rejections.push(rejection); + } + } + Ok(StalkerGenerationPublication { accepted_selection_mask, quality_acceptances, quality_rejections, force_updates }) } fn category_map(categories: Vec) -> HashMap { @@ -317,12 +631,14 @@ pub async fn advance_stalker_refresh( app_config: &Arc, api_client: &StalkerApiClient, handshake: &StalkerHandshake, - selection: StalkerClusterSelection, + refresh_plan: StalkerRefreshPlan, storage_path: &Path, identity_fingerprint: u64, mut budget: StalkerRefreshBudget, ) -> Result { - let mut checkpoint = load_or_start_checkpoint(storage_path, identity_fingerprint, selection).await?; + let StalkerRefreshPlan { selection, update_quality, quality_bypass_mask } = refresh_plan; + let mut checkpoint = + load_or_start_checkpoint(storage_path, identity_fingerprint, selection, quality_bypass_mask).await?; if checkpoint.phase == StalkerRefreshPhase::Terminal { return finish_terminal_refresh(storage_path).await; } @@ -568,8 +884,15 @@ pub async fn advance_stalker_refresh( checkpoint.retry_count = 0; } StalkerRefreshPhase::Complete => { - finish_completed_refresh_checked(app_config, storage_path, identity_fingerprint, &checkpoint).await?; - return Ok(StalkerRefreshOutcome::Complete); + let reports = finish_completed_refresh_checked( + app_config, + storage_path, + identity_fingerprint, + &checkpoint, + update_quality, + ) + .await?; + return Ok(reports.into_outcome()); } StalkerRefreshPhase::Terminal => { return finish_terminal_refresh(storage_path).await; @@ -584,11 +907,17 @@ pub async fn advance_stalker_refresh( mod tests { use super::*; use arc_swap::{ArcSwap, ArcSwapOption}; - use shared::model::ConfigPaths; + use shared::{ + model::{ConfigInputUpdateQualityDto, ConfigPaths}, + utils::Internable, + }; use tuliprox_core::{ model::{ApiProxyConfig, Config, CustomStreamResponse, HdHomeRunConfig, MediaToolCapabilities, SourcesConfig}, utils::FileLockManager, }; + use tuliprox_repository::stalker_generation_repository::{ + load_active_manifest, save_active_manifest, ClusterFiles, SeriesFiles, StalkerActiveManifest, + }; fn test_app_config(storage_dir: &Path) -> Arc { Arc::new(AppConfig { @@ -620,6 +949,78 @@ mod tests { }) } + fn update_quality(live: u8, vod: u8, series: u8) -> ConfigInputUpdateQuality { + ConfigInputUpdateQuality::from(&ConfigInputUpdateQualityDto { live, vod, series }) + } + + fn stalker_items(count: usize, kind: StalkerStreamKind, series_roots: bool) -> Vec { + (0..count) + .map(|index| StalkerPlaylistItem { + stream_id: u32::try_from(index + 1).unwrap_or(u32::MAX), + name: format!("item-{index}").intern(), + stream_kind: kind, + is_series: series_roots, + ..StalkerPlaylistItem::default() + }) + .collect() + } + + async fn write_generation_cluster( + app_config: &Arc, + storage_path: &Path, + generation: u64, + cluster: StalkerCluster, + primary_count: usize, + episode_count: usize, + ) -> Result<(), TuliproxError> { + let (primary_data, kind, series_roots) = match cluster { + StalkerCluster::Live => (StalkerGenerationData::Live, StalkerStreamKind::Live, false), + StalkerCluster::Vod => (StalkerGenerationData::Vod, StalkerStreamKind::Movie, false), + StalkerCluster::Series => (StalkerGenerationData::SeriesRoots, StalkerStreamKind::Episode, true), + }; + let primary_path = generation_data_path(storage_path, generation, primary_data); + let primary_items = stalker_items(primary_count, kind, series_roots); + if primary_items.is_empty() { + upsert_stalker_items_at(app_config, &primary_path, &primary_items).await?; + } else { + snapshot_stalker_items_at(app_config, primary_path, &primary_items).await?; + } + if cluster == StalkerCluster::Series { + let episode_path = generation_data_path(storage_path, generation, StalkerGenerationData::SeriesEpisodes); + snapshot_stalker_items_at( + app_config, + episode_path, + &stalker_items(episode_count, StalkerStreamKind::Episode, false), + ) + .await?; + } + Ok(()) + } + + fn complete_manifest(storage_path: &Path, identity_fingerprint: u64, generation: u64) -> StalkerActiveManifest { + StalkerActiveManifest { + schema: StalkerActiveManifest::empty(identity_fingerprint).schema, + identity_fingerprint, + live: Some(ClusterFiles { + generation, + data: generation_data_path(storage_path, generation, StalkerGenerationData::Live), + }), + vod: Some(ClusterFiles { + generation, + data: generation_data_path(storage_path, generation, StalkerGenerationData::Vod), + }), + series: Some(SeriesFiles { + generation, + roots: generation_data_path(storage_path, generation, StalkerGenerationData::SeriesRoots), + episodes: generation_data_path(storage_path, generation, StalkerGenerationData::SeriesEpisodes), + }), + epg: Some(ClusterFiles { + generation, + data: generation_data_path(storage_path, generation, StalkerGenerationData::Epg), + }), + } + } + #[tokio::test] async fn terminal_checkpoint_is_cleared_after_restart() -> Result<(), Box> { let temp = tempfile::tempdir()?; @@ -643,13 +1044,33 @@ mod tests { checkpoint.phase = StalkerRefreshPhase::Complete; save_checkpoint(temp.path(), &checkpoint).await?; - let resumed = load_or_start_checkpoint(temp.path(), 17, selection).await?; + let resumed = load_or_start_checkpoint(temp.path(), 17, selection, 0).await?; assert_eq!(resumed.generation, 23); assert_eq!(resumed.phase, StalkerRefreshPhase::Complete); Ok(()) } + #[tokio::test] + async fn resumed_checkpoint_preserves_and_extends_the_quality_bypass_mask() -> Result<(), Box> + { + let temp = tempfile::tempdir()?; + let selection = StalkerClusterSelection { live: true, vod: true, series: true, epg: false }; + let mut checkpoint = StalkerCheckpoint::new(17, 23, selection.mask(), 123); + checkpoint.phase = StalkerRefreshPhase::Vod { page: 4 }; + checkpoint.quality_bypass_mask = LIVE_SELECTION; + save_checkpoint(temp.path(), &checkpoint).await?; + + let resumed = load_or_start_checkpoint(temp.path(), 17, selection, SERIES_SELECTION).await?; + + assert_eq!(resumed.generation, 23); + assert_eq!(resumed.phase, StalkerRefreshPhase::Vod { page: 4 }); + assert_eq!(resumed.quality_bypass_mask, LIVE_SELECTION | SERIES_SELECTION); + let resumed_without_new_force = load_or_start_checkpoint(temp.path(), 17, selection, 0).await?; + assert_eq!(resumed_without_new_force.quality_bypass_mask, LIVE_SELECTION | SERIES_SELECTION); + Ok(()) + } + #[tokio::test] async fn empty_complete_refresh_keeps_the_active_manifest() -> Result<(), Box> { let temp = tempfile::tempdir()?; @@ -666,9 +1087,16 @@ mod tests { checkpoint.phase = StalkerRefreshPhase::Complete; save_checkpoint(temp.path(), &checkpoint).await?; - let result = finish_completed_refresh_checked(&app_config, temp.path(), 17, &checkpoint).await; + let reports = finish_completed_refresh_checked( + &app_config, + temp.path(), + 17, + &checkpoint, + ConfigInputUpdateQuality::default(), + ) + .await?; - assert!(result.is_err()); + assert!(reports.error.is_some()); assert_eq!(load_active_manifest(temp.path(), 17).await?, active); assert!(load_checkpoint(temp.path(), 17).await?.is_none()); Ok(()) @@ -677,22 +1105,368 @@ mod tests { #[tokio::test] async fn complete_publication_is_idempotent_after_restart() -> Result<(), Box> { let temp = tempfile::tempdir()?; + let app_config = test_app_config(temp.path()); let mut checkpoint = StalkerCheckpoint::new(17, 23, 0b0011, 123); checkpoint.phase = StalkerRefreshPhase::Complete; save_checkpoint(temp.path(), &checkpoint).await?; - finish_completed_refresh(temp.path(), 17, &checkpoint).await?; + finish_completed_refresh(&app_config, temp.path(), 17, &checkpoint, ConfigInputUpdateQuality::default()) + .await?; save_checkpoint(temp.path(), &checkpoint).await?; - finish_completed_refresh(temp.path(), 17, &checkpoint).await?; + finish_completed_refresh(&app_config, temp.path(), 17, &checkpoint, ConfigInputUpdateQuality::default()) + .await?; - let manifest = - tuliprox_repository::stalker_generation_repository::load_active_manifest(temp.path(), 17).await?; + let manifest = load_active_manifest(temp.path(), 17).await?; assert_eq!(manifest.live.as_ref().map(|files| files.generation), Some(23)); assert_eq!(manifest.vod.as_ref().map(|files| files.generation), Some(23)); assert!(load_checkpoint(temp.path(), 17).await?.is_none()); Ok(()) } + #[tokio::test] + async fn pipeline_transparency_stalker_preserves_accepted_quality_facts_after_publication( + ) -> Result<(), Box> { + let temp = tempfile::tempdir()?; + let app_config = test_app_config(temp.path()); + let identity = 17; + let active_generation = 41; + let candidate_generation = 42; + write_generation_cluster(&app_config, temp.path(), active_generation, StalkerCluster::Live, 100, 0).await?; + write_generation_cluster(&app_config, temp.path(), active_generation, StalkerCluster::Vod, 100, 0).await?; + write_generation_cluster(&app_config, temp.path(), active_generation, StalkerCluster::Series, 40, 60).await?; + let active = complete_manifest(temp.path(), identity, active_generation); + save_active_manifest(temp.path(), &active).await?; + write_generation_cluster(&app_config, temp.path(), candidate_generation, StalkerCluster::Live, 90, 0).await?; + write_generation_cluster(&app_config, temp.path(), candidate_generation, StalkerCluster::Vod, 89, 0).await?; + write_generation_cluster(&app_config, temp.path(), candidate_generation, StalkerCluster::Series, 45, 65) + .await?; + let checkpoint = StalkerCheckpoint::new(identity, candidate_generation, 0b1111, 123); + + let reports = finish_completed_refresh_checked( + &app_config, + temp.path(), + identity, + &checkpoint, + update_quality(90, 90, 90), + ) + .await?; + + assert!(reports.error.is_none()); + assert_eq!( + reports.quality_acceptances, + vec![ + ClusterUpdateAcceptance { + cluster: XtreamCluster::Live, + current_count: Some(100), + candidate_count: 90, + threshold: 90, + quality: Some(90), + }, + ClusterUpdateAcceptance { + cluster: XtreamCluster::Series, + current_count: Some(100), + candidate_count: 110, + threshold: 90, + quality: Some(90), + }, + ] + ); + assert_eq!( + reports.quality_rejections, + vec![ClusterUpdateRejection { + cluster: XtreamCluster::Video, + current_count: 100, + candidate_count: 89, + threshold: 90, + quality: 89, + }] + ); + let published = load_active_manifest(temp.path(), identity).await?; + assert_eq!(published.live.as_ref().map(|files| files.generation), Some(candidate_generation)); + assert_eq!(published.vod, active.vod); + assert_eq!(published.series.as_ref().map(|files| files.generation), Some(candidate_generation)); + assert_eq!(published.epg.as_ref().map(|files| files.generation), Some(candidate_generation)); + Ok(()) + } + + #[tokio::test] + async fn pipeline_transparency_stalker_keeps_acceptance_when_pre_publish_validation_fails( + ) -> Result<(), Box> { + let temp = tempfile::tempdir()?; + let app_config = test_app_config(temp.path()); + let identity = 18; + let active_generation = 51; + let candidate_generation = 52; + write_generation_cluster(&app_config, temp.path(), active_generation, StalkerCluster::Live, 100, 0).await?; + let mut active = StalkerActiveManifest::empty(identity); + active.live = Some(ClusterFiles { + generation: active_generation, + data: generation_data_path(temp.path(), active_generation, StalkerGenerationData::Live), + }); + save_active_manifest(temp.path(), &active).await?; + write_generation_cluster(&app_config, temp.path(), candidate_generation, StalkerCluster::Live, 90, 0).await?; + let mut checkpoint = StalkerCheckpoint::new(identity, candidate_generation, LIVE_SELECTION, 123); + checkpoint.phase = StalkerRefreshPhase::Complete; + save_checkpoint(temp.path(), &checkpoint).await?; + + let reports = + finish_completed_refresh_checked(&app_config, temp.path(), identity, &checkpoint, update_quality(90, 0, 0)) + .await?; + + let StalkerRefreshOutcome::Failed { quality_acceptances, error: _, .. } = reports.into_outcome() else { + return Err("expected failed Stalker completion with retained reports".into()); + }; + assert_eq!( + quality_acceptances, + vec![ClusterUpdateAcceptance { + cluster: XtreamCluster::Live, + current_count: Some(100), + candidate_count: 90, + threshold: 90, + quality: Some(90), + }] + ); + assert_eq!(load_active_manifest(temp.path(), identity).await?, active); + assert!(load_checkpoint(temp.path(), identity).await?.is_none()); + Ok(()) + } + + #[tokio::test] + async fn pipeline_transparency_stalker_keeps_acceptance_when_cleanup_fails_after_publish( + ) -> Result<(), Box> { + let temp = tempfile::tempdir()?; + let app_config = test_app_config(temp.path()); + let identity = 19; + let active_generation = 61; + let candidate_generation = 62; + write_generation_cluster(&app_config, temp.path(), active_generation, StalkerCluster::Live, 100, 0).await?; + let mut active = StalkerActiveManifest::empty(identity); + active.live = Some(ClusterFiles { + generation: active_generation, + data: generation_data_path(temp.path(), active_generation, StalkerGenerationData::Live), + }); + save_active_manifest(temp.path(), &active).await?; + write_generation_cluster(&app_config, temp.path(), candidate_generation, StalkerCluster::Live, 90, 0).await?; + tokio::fs::create_dir(generation_data_path(temp.path(), 1, StalkerGenerationData::Live)).await?; + tokio::fs::write(generation_data_path(temp.path(), 2, StalkerGenerationData::Live), b"obsolete").await?; + let mut checkpoint = StalkerCheckpoint::new(identity, candidate_generation, LIVE_SELECTION, 123); + checkpoint.phase = StalkerRefreshPhase::Complete; + checkpoint.processed = 90; + save_checkpoint(temp.path(), &checkpoint).await?; + + let reports = + finish_completed_refresh_checked(&app_config, temp.path(), identity, &checkpoint, update_quality(90, 0, 0)) + .await?; + + let StalkerRefreshOutcome::Failed { quality_acceptances, error: _, .. } = reports.into_outcome() else { + return Err("expected failed Stalker completion with retained reports".into()); + }; + assert_eq!(quality_acceptances.len(), 1); + assert_eq!(quality_acceptances[0].cluster, XtreamCluster::Live); + let published = load_active_manifest(temp.path(), identity).await?; + assert_eq!(published.live.as_ref().map(|files| files.generation), Some(candidate_generation)); + assert!(load_checkpoint(temp.path(), identity).await?.is_none()); + Ok(()) + } + + #[tokio::test] + async fn rejected_live_generation_keeps_active_live_and_epg() -> Result<(), Box> { + let temp = tempfile::tempdir()?; + let app_config = test_app_config(temp.path()); + let identity = 23; + let active_generation = 7; + let candidate_generation = 8; + write_generation_cluster(&app_config, temp.path(), active_generation, StalkerCluster::Live, 100, 0).await?; + let mut active = StalkerActiveManifest::empty(identity); + active.live = Some(ClusterFiles { + generation: active_generation, + data: generation_data_path(temp.path(), active_generation, StalkerGenerationData::Live), + }); + active.epg = Some(ClusterFiles { + generation: active_generation, + data: generation_data_path(temp.path(), active_generation, StalkerGenerationData::Epg), + }); + save_active_manifest(temp.path(), &active).await?; + write_generation_cluster(&app_config, temp.path(), candidate_generation, StalkerCluster::Live, 89, 0).await?; + let checkpoint = StalkerCheckpoint::new(identity, candidate_generation, LIVE_SELECTION | EPG_SELECTION, 123); + + let rejections = + finish_completed_refresh(&app_config, temp.path(), identity, &checkpoint, update_quality(90, 0, 0)).await?; + + assert_eq!(rejections.len(), 1); + assert_eq!(rejections[0].cluster, XtreamCluster::Live); + let published = load_active_manifest(temp.path(), identity).await?; + assert_eq!(published.live, active.live); + assert_eq!(published.epg, active.epg); + Ok(()) + } + + #[tokio::test] + async fn nonempty_bootstrap_generation_is_published() -> Result<(), Box> { + let temp = tempfile::tempdir()?; + let app_config = test_app_config(temp.path()); + let identity = 29; + let generation = 3; + write_generation_cluster(&app_config, temp.path(), generation, StalkerCluster::Series, 2, 3).await?; + let checkpoint = StalkerCheckpoint::new(identity, generation, SERIES_SELECTION, 123); + + let rejections = + finish_completed_refresh(&app_config, temp.path(), identity, &checkpoint, update_quality(0, 0, 100)) + .await?; + + assert!(rejections.is_empty()); + let published = load_active_manifest(temp.path(), identity).await?; + assert_eq!(published.series.as_ref().map(|files| files.generation), Some(generation)); + Ok(()) + } + + #[tokio::test] + async fn empty_bootstrap_generation_is_rejected() -> Result<(), Box> { + let temp = tempfile::tempdir()?; + let app_config = test_app_config(temp.path()); + let identity = 30; + let generation = 4; + write_generation_cluster(&app_config, temp.path(), generation, StalkerCluster::Live, 0, 0).await?; + let checkpoint = StalkerCheckpoint::new(identity, generation, LIVE_SELECTION | EPG_SELECTION, 123); + + let rejections = + finish_completed_refresh(&app_config, temp.path(), identity, &checkpoint, update_quality(90, 0, 0)).await?; + + assert_eq!( + rejections, + vec![ClusterUpdateRejection { + cluster: XtreamCluster::Live, + current_count: 0, + candidate_count: 0, + threshold: 90, + quality: 0, + }] + ); + let published = load_active_manifest(temp.path(), identity).await?; + assert!(published.live.is_none()); + assert!(published.epg.is_none()); + Ok(()) + } + + #[tokio::test] + async fn forced_empty_stalker_generation_is_published_without_a_quality_rejection( + ) -> Result<(), Box> { + let temp = tempfile::tempdir()?; + let app_config = test_app_config(temp.path()); + let identity = 40; + let active_generation = 5; + let candidate_generation = 6; + write_generation_cluster(&app_config, temp.path(), active_generation, StalkerCluster::Vod, 10, 0).await?; + let mut active = complete_manifest(temp.path(), identity, active_generation); + active.live = None; + active.series = None; + active.epg = None; + save_active_manifest(temp.path(), &active).await?; + write_generation_cluster(&app_config, temp.path(), candidate_generation, StalkerCluster::Vod, 0, 0).await?; + let mut checkpoint = StalkerCheckpoint::new(identity, candidate_generation, VOD_SELECTION, 123); + checkpoint.phase = StalkerRefreshPhase::Complete; + checkpoint.quality_bypass_mask = VOD_SELECTION; + save_checkpoint(temp.path(), &checkpoint).await?; + + let reports = finish_completed_refresh_checked( + &app_config, + temp.path(), + identity, + &checkpoint, + update_quality(0, 100, 0), + ) + .await?; + + assert!(reports.quality_rejections.is_empty()); + assert_eq!( + reports.force_updates, + vec![ClusterForceUpdate { cluster: XtreamCluster::Video, candidate_count: 0, configured_threshold: 100 }] + ); + let published = load_active_manifest(temp.path(), identity).await?; + assert_eq!(published.vod.as_ref().map(|files| files.generation), Some(candidate_generation)); + assert!(load_checkpoint(temp.path(), identity).await?.is_none()); + Ok(()) + } + + #[tokio::test] + async fn forced_live_generation_publishes_its_epg_with_the_same_generation( + ) -> Result<(), Box> { + let temp = tempfile::tempdir()?; + let app_config = test_app_config(temp.path()); + let identity = 41; + let active_generation = 7; + let candidate_generation = 8; + write_generation_cluster(&app_config, temp.path(), active_generation, StalkerCluster::Live, 10, 0).await?; + let mut active = StalkerActiveManifest::empty(identity); + active.live = Some(ClusterFiles { + generation: active_generation, + data: generation_data_path(temp.path(), active_generation, StalkerGenerationData::Live), + }); + active.epg = Some(ClusterFiles { + generation: active_generation, + data: generation_data_path(temp.path(), active_generation, StalkerGenerationData::Epg), + }); + save_active_manifest(temp.path(), &active).await?; + write_generation_cluster(&app_config, temp.path(), candidate_generation, StalkerCluster::Live, 1, 0).await?; + let mut checkpoint = + StalkerCheckpoint::new(identity, candidate_generation, LIVE_SELECTION | EPG_SELECTION, 123); + checkpoint.quality_bypass_mask = LIVE_SELECTION; + + let reports = finish_completed_refresh_checked( + &app_config, + temp.path(), + identity, + &checkpoint, + update_quality(100, 0, 0), + ) + .await?; + + assert!(reports.quality_rejections.is_empty()); + assert_eq!(reports.force_updates.len(), 1); + assert_eq!(reports.force_updates[0].cluster, XtreamCluster::Live); + let published = load_active_manifest(temp.path(), identity).await?; + assert_eq!(published.live.as_ref().map(|files| files.generation), Some(candidate_generation)); + assert_eq!(published.epg.as_ref().map(|files| files.generation), Some(candidate_generation)); + Ok(()) + } + + #[tokio::test] + async fn rejected_series_counts_roots_and_episodes_without_blocking_accepted_live_generation( + ) -> Result<(), Box> { + let temp = tempfile::tempdir()?; + let app_config = test_app_config(temp.path()); + let identity = 31; + let active_generation = 11; + let candidate_generation = 12; + write_generation_cluster(&app_config, temp.path(), active_generation, StalkerCluster::Live, 10, 0).await?; + write_generation_cluster(&app_config, temp.path(), active_generation, StalkerCluster::Series, 2, 8).await?; + let mut active = complete_manifest(temp.path(), identity, active_generation); + active.vod = None; + save_active_manifest(temp.path(), &active).await?; + write_generation_cluster(&app_config, temp.path(), candidate_generation, StalkerCluster::Live, 10, 0).await?; + write_generation_cluster(&app_config, temp.path(), candidate_generation, StalkerCluster::Series, 2, 7).await?; + let checkpoint = StalkerCheckpoint::new(identity, candidate_generation, LIVE_SELECTION | SERIES_SELECTION, 123); + + assert_eq!(count_active_cluster(&app_config, &active, StalkerCluster::Series).await?, Some(10)); + assert_eq!( + count_candidate_cluster(&app_config, temp.path(), candidate_generation, StalkerCluster::Series).await?, + 9 + ); + + let rejections = + finish_completed_refresh(&app_config, temp.path(), identity, &checkpoint, update_quality(100, 0, 100)) + .await?; + + assert_eq!(rejections.len(), 1); + assert_eq!(rejections[0].cluster, XtreamCluster::Series); + assert_eq!(rejections[0].current_count, 10); + assert_eq!(rejections[0].candidate_count, 9); + let published = load_active_manifest(temp.path(), identity).await?; + assert_eq!(published.live.as_ref().map(|files| files.generation), Some(candidate_generation)); + assert_eq!(published.series, active.series); + Ok(()) + } + #[test] fn unit_budget_yields_after_completed_unit() { let mut budget = StalkerRefreshBudget::units(1); diff --git a/backend/repository/src/library_repository.rs b/backend/repository/src/library_repository.rs index e571236d7..38533732b 100644 --- a/backend/repository/src/library_repository.rs +++ b/backend/repository/src/library_repository.rs @@ -16,10 +16,6 @@ pub async fn persist_input_library_playlist( library_path: &Path, playlist: Vec, ) -> (Vec, Result<(), TuliproxError>) { - if playlist.is_empty() { - return (playlist, Ok(())); - } - let file_lock = app_config.file_locks.write_lock(library_path).await; let library_path = library_path.to_path_buf(); let library_path_err = library_path.clone(); @@ -169,7 +165,7 @@ pub async fn load_input_local_library_playlist( return Ok(groups); } - Ok(Vec::new()) + Err(TuliproxError::RepositoryLibrary(format!("Library input is missing: {}", lib_path.display()))) } #[cfg(test)] diff --git a/backend/repository/src/m3u_repository.rs b/backend/repository/src/m3u_repository.rs index df4956aa6..6836b61d7 100644 --- a/backend/repository/src/m3u_repository.rs +++ b/backend/repository/src/m3u_repository.rs @@ -264,8 +264,9 @@ pub async fn m3u_write_playlist( target_output: &M3uTargetOutput, target_path: &Path, new_playlist: &[PlaylistGroup], + library_empty: crate::LibraryEmptyPublication, ) -> Result<(), TuliproxError> { - if new_playlist.is_empty() { + if new_playlist.is_empty() && !library_empty.replaces_empty_target() { return Ok(()); } diff --git a/backend/repository/src/playlist_mem_cache.rs b/backend/repository/src/playlist_mem_cache.rs index d28f88cf3..d0d64644e 100644 --- a/backend/repository/src/playlist_mem_cache.rs +++ b/backend/repository/src/playlist_mem_cache.rs @@ -36,6 +36,11 @@ impl Default for PlaylistStorageState { impl PlaylistStorageState { pub fn new() -> Self { Self { data: RwLock::new(HashMap::new()) } } + /// Replaces one target only after its complete persisted state has been loaded. + pub(crate) async fn replace_target(&self, target_name: &str, storage: TargetPlaylistStorage) { + self.data.write().await.insert(target_name.to_string(), storage); + } + pub async fn update_target_id_mapping(&self, target: &ConfigTarget, mapping: Vec) { if target.use_memory_cache { if let Some(storage) = self.data.write().await.get_mut(&target.name) { diff --git a/backend/repository/src/playlist_repository.rs b/backend/repository/src/playlist_repository.rs index ab6a43ee0..27e2bc58e 100644 --- a/backend/repository/src/playlist_repository.rs +++ b/backend/repository/src/playlist_repository.rs @@ -1,20 +1,23 @@ -use super::playlist_mem_cache::{PlaylistM3uStorage, PlaylistStorage, PlaylistStorageState, PlaylistXtreamStorage}; +use super::playlist_mem_cache::{ + PlaylistM3uStorage, PlaylistStorageState, PlaylistXtreamStorage, TargetPlaylistStorage, +}; use crate::{ ensure_target_storage_path, epg_write_for_target, get_input_storage_path, get_target_id_mapping_file, get_target_storage_path, load_input_local_library_playlist, load_input_m3u_playlist, load_input_xtream_playlist, m3u_get_file_path_for_db, m3u_write_playlist, persist_input_library_playlist, persist_input_m3u_playlist, - persist_input_xtream_playlist, stalker_repository::get_stalker_storage_path, write_strm_playlist, - xtream_get_file_path, xtream_get_storage_path, xtream_write_playlist, BPlusTree, LocalLibraryDiskPlaylistSource, - M3uDiskPlaylistSource, MediaServerDiskPlaylistSource, MemoryPlaylistSource, PlaylistSource, - StalkerDiskPlaylistSource, TargetIdMapping, VirtualIdRecord, XtreamDiskPlaylistSource, FILE_SUFFIX_DB, + stalker_repository::get_stalker_storage_path, write_strm_playlist, xtream_get_file_path, xtream_get_storage_path, + xtream_write_playlist, BPlusTree, LocalLibraryDiskPlaylistSource, M3uDiskPlaylistSource, + MediaServerDiskPlaylistSource, MemoryPlaylistSource, PlaylistSource, StalkerDiskPlaylistSource, TargetIdMapping, + VirtualIdRecord, XtreamDiskPlaylistSource, FILE_SUFFIX_DB, }; use log::{debug, warn}; use shared::{ error::TuliproxError, model::{ - xtream_const::XTREAM_CLUSTER, ConfigTargetOptions, InputPersistence, M3uPlaylistItem, PlaylistEntry, - PlaylistGroup, PlaylistItem, PlaylistItemHeader, PlaylistItemType, SeriesStreamDetailEpisodeProperties, - SeriesStreamDetailProperties, StreamProperties, UUIDType, VirtualId, XtreamCluster, XtreamPlaylistItem, + xtream_const::XTREAM_CLUSTER, ClusterFlags, ConfigTargetOptions, InputPersistence, M3uPlaylistItem, + PlaylistEntry, PlaylistGroup, PlaylistItem, PlaylistItemHeader, PlaylistItemType, + SeriesStreamDetailEpisodeProperties, SeriesStreamDetailProperties, StreamProperties, UUIDType, VirtualId, + XtreamCluster, XtreamPlaylistItem, }, utils::{is_dash_url, is_hls_url, Internable}, }; @@ -35,6 +38,172 @@ struct LocalEpisodeKey { fn playlist_has_items(playlist: &[PlaylistGroup]) -> bool { playlist.iter().any(|group| !group.channels.is_empty()) } +#[derive(Debug, Clone, Copy, Default)] +enum TargetPersistenceMode { + #[default] + Persist, + #[cfg(test)] + FailCacheReloadAt(TargetCacheReloadStage), + #[cfg(test)] + FailEmptyReplacementAt(crate::TargetEmptyReplacementFailure), +} + +#[cfg(test)] +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum TargetCacheReloadStage { + IdMapping, + XtreamStorage, +} + +/// Scoped persistence behavior derived from technically successful forced clusters. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct InputPlaylistPersistOptions { + pub accepted_empty_clusters: ClusterFlags, +} + +impl Default for InputPlaylistPersistOptions { + fn default() -> Self { Self { accepted_empty_clusters: ClusterFlags::empty() } } +} + +/// Empty Library contribution proven by successful input jobs, before target transforms. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub enum LibraryEmptyPublication { + #[default] + None, + /// Empty Library contribution without permission to clear the complete target/output. + Contribution, + /// All inputs are ready, non-Library inputs are populated, and no forced-empty cluster is involved. + /// Configured transformations may remove the remaining entries from a target or individual output. + FilterableContribution, + /// All required inputs are empty Library catalogs, or the proven complete inputs + /// have been reduced to an empty target/output by the configured transformations. + CompleteTarget, +} + +impl LibraryEmptyPublication { + #[must_use] + pub const fn replaces_empty_target(self) -> bool { matches!(self, Self::CompleteTarget) } + + /// Resolve the input-level authorization against the actual prepared target/output. + /// A plain contribution never authorizes a fully empty result from a foreign input. + #[must_use] + pub fn for_filtered_playlist(self, playlist: &[PlaylistGroup]) -> Self { + match self { + Self::FilterableContribution if !playlist_has_items(playlist) => Self::CompleteTarget, + Self::None | Self::Contribution | Self::FilterableContribution | Self::CompleteTarget => self, + } + } + + fn replacement_clusters(self) -> ClusterFlags { + match self { + Self::None => ClusterFlags::empty(), + Self::Contribution | Self::FilterableContribution => ClusterFlags::Vod | ClusterFlags::Series, + Self::CompleteTarget => ClusterFlags::all(), + } + } +} + +/// Target persistence behavior authorized by successful input results, not by Quality inference. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct TargetPlaylistPersistOptions { + pub accepted_empty_clusters: ClusterFlags, + pub library_empty: LibraryEmptyPublication, +} + +impl Default for TargetPlaylistPersistOptions { + fn default() -> Self { + Self { accepted_empty_clusters: ClusterFlags::empty(), library_empty: LibraryEmptyPublication::None } + } +} + +fn validate_target_playlist_persistence( + target: &ConfigTarget, + playlist_is_empty: bool, + options: TargetPlaylistPersistOptions, +) -> Result<(), TuliproxError> { + if !playlist_is_empty || options.library_empty.replaces_empty_target() { + return Ok(()); + } + if options.accepted_empty_clusters.is_empty() { + return Err(TuliproxError::RepositoryPlaylist(format!( + "Refusing to persist empty playlist for target '{}'; existing data was retained", + target.name + ))); + } + if target.output.iter().any(|output| !matches!(output, TargetOutput::Xtream(_))) { + return Err(TuliproxError::RepositoryPlaylist(format!( + "Target '{}' has non-Xtream outputs that cannot safely publish a fully empty forced result; existing data was retained", + target.name + ))); + } + Ok(()) +} + +fn prepare_target_output_playlists( + target: &ConfigTarget, + playlist: &[PlaylistGroup], +) -> Vec>> { + target + .output + .iter() + .map(|output| output.filter().map(|filter| apply_filter_to_playlist(playlist, filter))) + .collect() +} + +fn validate_force_empty_output_filters( + target: &ConfigTarget, + prepared_outputs: &[Option>], + options: TargetPlaylistPersistOptions, +) -> Result<(), TuliproxError> { + if options.accepted_empty_clusters.is_empty() { + return Ok(()); + } + + debug_assert_eq!(target.output.len(), prepared_outputs.len()); + for (output, prepared_output) in target.output.iter().zip(prepared_outputs) { + let output_name = match output { + TargetOutput::M3u(_) => "M3U", + TargetOutput::Strm(_) => "STRM", + TargetOutput::Xtream(_) | TargetOutput::HdHomeRun(_) => continue, + }; + let Some(filtered_playlist) = prepared_output else { + continue; + }; + if !playlist_has_items(filtered_playlist) { + return Err(TuliproxError::RepositoryPlaylist(format!( + "Refusing to publish force-empty {output_name} output for target '{}' after its output filter; existing data was retained", + target.name + ))); + } + } + + Ok(()) +} + +async fn persist_xtream_target_playlist( + app_config: &Arc, + target: &ConfigTarget, + playlist: &mut [PlaylistGroup], + accepted_empty_clusters: ClusterFlags, + mode: TargetPersistenceMode, +) -> Result<(), TuliproxError> { + #[cfg(test)] + if let TargetPersistenceMode::FailEmptyReplacementAt(failure) = mode { + return crate::xtream_write_playlist_with_injected_empty_replacement_failure( + app_config, + target, + playlist, + accepted_empty_clusters, + failure, + ) + .await; + } + #[cfg(not(test))] + let _ = mode; + + xtream_write_playlist(app_config, target, playlist, accepted_empty_clusters).await +} + pub struct ProviderEpisodeKey { pub provider_id: u32, pub virtual_id: u32, @@ -66,12 +235,33 @@ pub async fn persist_playlist( epg: Option<&Epg>, target: &ConfigTarget, playlist_state: Option<&Arc>, + options: TargetPlaylistPersistOptions, ) -> Result<(), Vec> { - if !playlist_has_items(playlist) { - return Err(vec![TuliproxError::RepositoryPlaylist(format!( - "Refusing to persist empty playlist for target '{}'; existing data was retained", - target.name - ))]); + persist_playlist_with_mode( + app_config, + playlist, + epg, + target, + playlist_state, + options, + TargetPersistenceMode::Persist, + ) + .await +} + +#[allow(clippy::too_many_lines)] +async fn persist_playlist_with_mode( + app_config: &Arc, + playlist: &mut [PlaylistGroup], + epg: Option<&Epg>, + target: &ConfigTarget, + playlist_state: Option<&Arc>, + options: TargetPlaylistPersistOptions, + persistence_mode: TargetPersistenceMode, +) -> Result<(), Vec> { + let playlist_is_empty = !playlist_has_items(playlist); + if let Err(error) = validate_target_playlist_persistence(target, playlist_is_empty, options) { + return Err(vec![error]); } let mut errors = vec![]; let config = &app_config.config.load(); @@ -148,17 +338,39 @@ pub async fn persist_playlist( normalize_target_playlist_epg_ids(playlist, target.options.as_ref()); - for output in &target.output { - let mut filtered: Option> = - output.filter().map(|flt| apply_filter_to_playlist(playlist, flt)); + let mut prepared_outputs = prepare_target_output_playlists(target, playlist); + if let Err(error) = validate_force_empty_output_filters(target, &prepared_outputs, options) { + target_id_mapping.discard_unpersisted_changes(); + drop(target_id_mapping); + drop(file_lock); + return Err(vec![error]); + } - let pl: &mut [PlaylistGroup] = - if let Some(filtered_playlist) = filtered.as_mut() { filtered_playlist.as_mut_slice() } else { playlist }; + for (output, prepared_output) in target.output.iter().zip(&mut prepared_outputs) { + let pl: &mut [PlaylistGroup] = if let Some(filtered_playlist) = prepared_output.as_mut() { + filtered_playlist.as_mut_slice() + } else { + &mut *playlist + }; + let library_empty = options.library_empty.for_filtered_playlist(pl); let result = match output { - TargetOutput::Xtream(_xtream_output) => xtream_write_playlist(app_config, target, pl).await, - TargetOutput::M3u(m3u_output) => m3u_write_playlist(app_config, target, m3u_output, &target_path, pl).await, - TargetOutput::Strm(strm_output) => write_strm_playlist(app_config, target, strm_output, pl).await, + TargetOutput::Xtream(_xtream_output) => { + persist_xtream_target_playlist( + app_config, + target, + pl, + options.accepted_empty_clusters | library_empty.replacement_clusters(), + persistence_mode, + ) + .await + } + TargetOutput::M3u(m3u_output) => { + m3u_write_playlist(app_config, target, m3u_output, &target_path, pl, library_empty).await + } + TargetOutput::Strm(strm_output) => { + write_strm_playlist(app_config, target, strm_output, pl, library_empty).await + } TargetOutput::HdHomeRun(_hdhomerun_output) => Ok(()), }; @@ -186,30 +398,18 @@ pub async fn persist_playlist( drop(target_id_mapping); drop(file_lock); - if target.use_memory_cache { - if let Some(playlist_storage) = playlist_state { - if let Ok(id_mapping) = load_id_mapping_target_storage(app_config, target).await { - playlist_storage.cache_id_mapping(&target.name, id_mapping).await; - } - for output in &target.output { - match output { - TargetOutput::Xtream(_) => { - if let Ok(storage) = load_xtream_target_storage(app_config, target).await { - playlist_storage - .cache_playlist(&target.name, PlaylistStorage::XtreamPlaylist(Box::new(storage))) - .await; - } - } - TargetOutput::M3u(_) => { - if let Ok(storage) = load_m3u_target_storage(app_config, target).await { - playlist_storage - .cache_playlist(&target.name, PlaylistStorage::M3uPlaylist(Box::new(storage))) - .await; - } - } - _ => {} + if errors.is_empty() && target.use_memory_cache { + match playlist_state { + Some(playlist_storage) => { + match load_target_memory_cache_snapshot(app_config, target, persistence_mode).await { + Ok(storage) => playlist_storage.replace_target(&target.name, storage).await, + Err(error) => errors.push(error), } } + None => errors.push(TuliproxError::RepositoryPlaylist(format!( + "Target '{}' was persisted but its configured memory cache is unavailable", + target.name + ))), } } @@ -637,6 +837,56 @@ pub async fn load_m3u_target_storage( } } +fn target_cache_reload_error(target: &ConfigTarget, component: &str, error: impl std::fmt::Display) -> TuliproxError { + TuliproxError::RepositoryPlaylist(format!( + "Target '{}' was persisted but its {component} could not be reloaded into the memory cache: {error}", + target.name + )) +} + +async fn load_target_memory_cache_snapshot( + app_config: &AppConfig, + target: &ConfigTarget, + mode: TargetPersistenceMode, +) -> Result { + #[cfg(not(test))] + let _ = mode; + + #[cfg(test)] + if matches!(mode, TargetPersistenceMode::FailCacheReloadAt(TargetCacheReloadStage::IdMapping)) { + return Err(target_cache_reload_error(target, "ID mapping", "injected reload failure")); + } + let id_mapping = load_id_mapping_target_storage(app_config, target) + .await + .map_err(|error| target_cache_reload_error(target, "ID mapping", error))?; + + let xtream = if target.output.iter().any(|output| matches!(output, TargetOutput::Xtream(_))) { + #[cfg(test)] + if matches!(mode, TargetPersistenceMode::FailCacheReloadAt(TargetCacheReloadStage::XtreamStorage)) { + return Err(target_cache_reload_error(target, "Xtream storage", "injected reload failure")); + } + Some( + load_xtream_target_storage(app_config, target) + .await + .map_err(|error| target_cache_reload_error(target, "Xtream storage", error))?, + ) + } else { + None + }; + + let m3u = if target.output.iter().any(|output| matches!(output, TargetOutput::M3u(_))) { + Some( + load_m3u_target_storage(app_config, target) + .await + .map_err(|error| target_cache_reload_error(target, "M3U storage", error))?, + ) + } else { + None + }; + + Ok(TargetPlaylistStorage { xtream, m3u, id_mapping: Some(id_mapping) }) +} + async fn publish_all_cluster_catalogs( app_config: &AppConfig, storage_path: &Path, @@ -670,9 +920,22 @@ async fn publish_all_cluster_catalogs( pub async fn persist_input_playlist( app_config: &Arc, input: &ConfigInput, - mut playlist: Vec, + playlist: Vec, ) -> (Vec, Option) { - if !playlist_has_items(&playlist) { + persist_input_playlist_with_options(app_config, input, playlist, InputPlaylistPersistOptions::default()).await +} + +pub async fn persist_input_playlist_with_options( + app_config: &Arc, + input: &ConfigInput, + mut playlist: Vec, + options: InputPlaylistPersistOptions, +) -> (Vec, Option) { + let persistence = input.get_download_input_type().persistence(); + let accepts_empty = persistence == InputPersistence::Library + || (!options.accepted_empty_clusters.is_empty() + && matches!(persistence, InputPersistence::Xtream | InputPersistence::Stalker)); + if !playlist_has_items(&playlist) && !accepts_empty { let empty_error = TuliproxError::RepositoryPlaylist(format!( "Refusing to persist empty playlist for input '{}'; existing data was retained", input.name @@ -694,7 +957,7 @@ pub async fn persist_input_playlist( ), }; } - if input.get_download_input_type().persistence() == InputPersistence::Stalker { + if persistence == InputPersistence::Stalker { // The Stalker processor (`processor::stalker::download_stalker_playlist`) // is the single writer of the per-cluster B+Tree and raw group catalogs. // Re-encoding the `PlaylistItem` runtime projection back into `StalkerPlaylistItem` @@ -719,8 +982,16 @@ pub async fn persist_input_playlist( } }; - let (persisted_playlist, err) = match input.get_download_input_type().persistence() { - InputPersistence::Xtream => persist_input_xtream_playlist(app_config, &storage_path, playlist).await, + let (persisted_playlist, err) = match persistence { + InputPersistence::Xtream => { + crate::persist_input_xtream_playlist_with_empty_replacements( + app_config, + &storage_path, + playlist, + options.accepted_empty_clusters, + ) + .await + } InputPersistence::M3u => { // Persist M3U @@ -934,21 +1205,409 @@ mod tests { use super::{ assign_local_series_info_episode_key, assign_media_server_series_info_episode, get_input_media_server_playlist_file_path, materialize_media_server_series_info_episodes, - normalize_target_playlist_epg_ids, playlist_has_items, rewrite_local_series_info_episode_virtual_id, - rewrite_series_episode_parent_virtual_ids, rewrite_series_info_episode_virtual_id, skipped_clusters, - LocalEpisodeKey, ProviderEpisodeKey, + normalize_target_playlist_epg_ids, persist_playlist_with_mode, playlist_has_items, + rewrite_local_series_info_episode_virtual_id, rewrite_series_episode_parent_virtual_ids, + rewrite_series_info_episode_virtual_id, skipped_clusters, validate_target_playlist_persistence, + LocalEpisodeKey, ProviderEpisodeKey, TargetCacheReloadStage, TargetPersistenceMode, + TargetPlaylistPersistOptions, }; - use crate::{BPlusTreeQuery, TargetIdMapping, VirtualIdRecord}; + use crate::{load_xtream_target_storage, BPlusTreeQuery, PlaylistStorageState, TargetIdMapping, VirtualIdRecord}; + use arc_swap::{ArcSwap, ArcSwapOption}; use shared::{ + foundation::get_filter, model::{ - ConfigTargetOptions, EpgOutputOptions, EpisodeStreamProperties, M3uPlaylistItem, PlaylistEntry, - PlaylistGroup, PlaylistItem, PlaylistItemHeader, PlaylistItemType, SeriesStreamDetailEpisodeProperties, - SeriesStreamDetailProperties, SeriesStreamDetailSeasonProperties, SeriesStreamProperties, StreamProperties, - UUIDType, VirtualId, XtreamCluster, XtreamPlaylistItem, + ClusterFlags, ConfigPaths, ConfigTargetOptions, EpgOutputOptions, EpisodeStreamProperties, M3uPlaylistItem, + PlaylistEntry, PlaylistGroup, PlaylistItem, PlaylistItemHeader, PlaylistItemType, ProcessingOrder, + SeriesStreamDetailEpisodeProperties, SeriesStreamDetailProperties, SeriesStreamDetailSeasonProperties, + SeriesStreamProperties, StreamProperties, StrmExportStyle, UUIDType, VirtualId, XtreamCluster, + XtreamPlaylistItem, }, utils::Internable, }; - use std::{collections::HashMap, sync::Arc}; + use std::{collections::HashMap, path::Path, sync::Arc}; + use tuliprox_core::{ + model::{ + ApiProxyConfig, AppConfig, Config, ConfigTarget, CustomStreamResponse, HdHomeRunConfig, M3uTargetOutput, + MediaToolCapabilities, SourcesConfig, StagedFilter, StrmTargetFlagsSet, StrmTargetOutput, + TargetExecutionPlan, TargetOutput, XtreamTargetFlagsSet, XtreamTargetOutput, + }, + utils::FileLockManager, + }; + + fn target_with_outputs(output: Vec) -> ConfigTarget { + target_with_options("target-empty-guard", output, false) + } + + fn target_with_options(name: &str, output: Vec, use_memory_cache: bool) -> ConfigTarget { + ConfigTarget { + id: 1, + enabled: true, + name: name.to_string(), + options: None, + sort: None, + filter: StagedFilter::default(), + output, + rename: None, + mapping_ids: None, + mapping: Arc::new(ArcSwapOption::new(None)), + favourites: None, + processing_order: ProcessingOrder::default(), + execution_plan: TargetExecutionPlan::default(), + watch: None, + use_memory_cache, + } + } + + fn test_app_config(storage_dir: &Path) -> Arc { + Arc::new(AppConfig { + config: Arc::new(ArcSwap::from_pointee(Config { + storage_dir: storage_dir.to_string_lossy().into_owned(), + ..Config::default() + })), + sources: Arc::new(ArcSwap::from_pointee(SourcesConfig::default())), + hdhomerun: Arc::new(ArcSwapOption::::default()), + api_proxy: Arc::new(ArcSwapOption::::default()), + file_locks: Arc::new(FileLockManager::default()), + paths: Arc::new(ArcSwap::from_pointee(ConfigPaths { + home_path: String::new(), + config_path: String::new(), + storage_path: String::new(), + config_file_path: String::new(), + sources_file_path: String::new(), + mapping_file_path: None, + mapping_files_used: None, + template_file_path: None, + template_files_used: None, + api_proxy_file_path: String::new(), + custom_stream_response_path: None, + })), + custom_stream_response: Arc::new(ArcSwapOption::::default()), + access_token_secret: [0; 32], + encrypt_secret: [0; 16], + media_tools: Arc::new(MediaToolCapabilities::new()), + }) + } + + fn target_group(cluster: XtreamCluster, id: u32, name: &str) -> PlaylistGroup { + let name = name.intern(); + let mut header = PlaylistItemHeader { + id: id.to_string().intern(), + input_stream_id: id.to_string().intern(), + name: Arc::clone(&name), + title: Arc::clone(&name), + group: Arc::clone(&name), + input_name: "target-test-input".intern(), + item_type: PlaylistItemType::from(cluster), + xtream_cluster: cluster, + category_id: id, + ..PlaylistItemHeader::default() + }; + header.gen_uuid(); + PlaylistGroup { id, title: name, channels: vec![PlaylistItem { header }], xtream_cluster: cluster } + } + + fn xtream_target(name: &str, use_memory_cache: bool) -> ConfigTarget { + target_with_options( + name, + vec![TargetOutput::Xtream(XtreamTargetOutput { + flags: XtreamTargetFlagsSet::new(), + trakt: None, + filter: None, + })], + use_memory_cache, + ) + } + + async fn cached_target_signature( + playlist_state: &PlaylistStorageState, + target_name: &str, + ) -> (usize, Vec<(XtreamCluster, u32, Arc)>) { + let cache = playlist_state.data.read().await; + let target = cache.get(target_name).expect("target cache"); + let mapping_len = target.id_mapping.as_ref().expect("cached ID mapping").len(); + let xtream = target.xtream.as_ref().expect("cached Xtream storage"); + let mut items = Vec::new(); + for (cluster, storage) in [ + (XtreamCluster::Live, &xtream.live), + (XtreamCluster::Video, &xtream.vod), + (XtreamCluster::Series, &xtream.series), + ] { + for virtual_id in 1..=16 { + if let Some(item) = storage.query(&virtual_id) { + items.push((cluster, virtual_id, Arc::clone(&item.name))); + } + } + } + (mapping_len, items) + } + + async fn seed_memory_cached_xtream_target( + app_config: &Arc, + playlist_state: &Arc, + target: &ConfigTarget, + ) { + let mut baseline = vec![ + target_group(XtreamCluster::Live, 101, "old-live"), + target_group(XtreamCluster::Video, 201, "old-vod"), + target_group(XtreamCluster::Series, 301, "old-series"), + ]; + persist_playlist_with_mode( + app_config, + &mut baseline, + None, + target, + Some(playlist_state), + TargetPlaylistPersistOptions::default(), + TargetPersistenceMode::Persist, + ) + .await + .expect("baseline target persistence"); + } + + #[test] + fn target_force_empty_authorization_is_limited_to_xtream_outputs() { + let xtream_target = target_with_outputs(vec![TargetOutput::Xtream(XtreamTargetOutput { + flags: XtreamTargetFlagsSet::new(), + trakt: None, + filter: None, + })]); + let options = TargetPlaylistPersistOptions { + accepted_empty_clusters: ClusterFlags::Vod, + ..TargetPlaylistPersistOptions::default() + }; + assert!(validate_target_playlist_persistence(&xtream_target, true, options).is_ok()); + + let mixed_target = target_with_outputs(vec![ + TargetOutput::Xtream(XtreamTargetOutput { flags: XtreamTargetFlagsSet::new(), trakt: None, filter: None }), + TargetOutput::M3u(M3uTargetOutput { + filename: None, + include_type_in_url: false, + mask_redirect_url: false, + filter: None, + }), + ]); + let error = validate_target_playlist_persistence(&mixed_target, true, options) + .expect_err("fully empty mixed output must fail before persistence"); + assert!(error.to_string().contains("non-Xtream outputs")); + } + + #[test] + fn target_empty_playlist_without_force_authorization_remains_rejected() { + let target = target_with_outputs(vec![TargetOutput::Xtream(XtreamTargetOutput { + flags: XtreamTargetFlagsSet::new(), + trakt: None, + filter: None, + })]); + let error = validate_target_playlist_persistence(&target, true, TargetPlaylistPersistOptions::default()) + .expect_err("unauthorized empty target must remain rejected"); + assert!(error.to_string().contains("Refusing to persist empty playlist")); + } + + #[tokio::test] + async fn target_cache_reload_failures_return_errors_without_mutating_the_previous_cache() { + for stage in [TargetCacheReloadStage::IdMapping, TargetCacheReloadStage::XtreamStorage] { + let directory = tempdir().expect("temporary storage"); + let app_config = test_app_config(directory.path()); + let playlist_state = Arc::new(PlaylistStorageState::new()); + let target = xtream_target(&format!("cache-reload-{stage:?}"), true); + seed_memory_cached_xtream_target(&app_config, &playlist_state, &target).await; + let before = cached_target_signature(&playlist_state, &target.name).await; + let mut candidate = vec![ + target_group(XtreamCluster::Live, 102, "new-live"), + target_group(XtreamCluster::Series, 302, "new-series"), + ]; + + let errors = persist_playlist_with_mode( + &app_config, + &mut candidate, + None, + &target, + Some(&playlist_state), + TargetPlaylistPersistOptions { + accepted_empty_clusters: ClusterFlags::Vod, + ..TargetPlaylistPersistOptions::default() + }, + TargetPersistenceMode::FailCacheReloadAt(stage), + ) + .await + .expect_err("injected target cache reload must fail"); + + assert!(errors.iter().any(|error| error.to_string().contains("could not be reloaded"))); + assert_eq!(cached_target_signature(&playlist_state, &target.name).await, before); + let persisted = load_xtream_target_storage(&app_config, &target).await.expect("persisted target storage"); + assert_eq!([persisted.live.len(), persisted.vod.len(), persisted.series.len()], [1, 0, 1]); + } + } + + #[tokio::test] + async fn target_force_empty_reload_replaces_the_complete_memory_cache() { + let directory = tempdir().expect("temporary storage"); + let app_config = test_app_config(directory.path()); + let playlist_state = Arc::new(PlaylistStorageState::new()); + let target = xtream_target("force-empty-cache-success", true); + seed_memory_cached_xtream_target(&app_config, &playlist_state, &target).await; + let mut candidate = vec![ + target_group(XtreamCluster::Live, 102, "new-live"), + target_group(XtreamCluster::Series, 302, "new-series"), + ]; + + persist_playlist_with_mode( + &app_config, + &mut candidate, + None, + &target, + Some(&playlist_state), + TargetPlaylistPersistOptions { + accepted_empty_clusters: ClusterFlags::Vod, + ..TargetPlaylistPersistOptions::default() + }, + TargetPersistenceMode::Persist, + ) + .await + .expect("force-empty target persistence and cache reload"); + + let (_, cached_items) = cached_target_signature(&playlist_state, &target.name).await; + assert_eq!(cached_items.iter().filter(|(cluster, _, _)| *cluster == XtreamCluster::Live).count(), 1); + assert!(!cached_items.iter().any(|(cluster, _, _)| *cluster == XtreamCluster::Video)); + assert_eq!(cached_items.iter().filter(|(cluster, _, _)| *cluster == XtreamCluster::Series).count(), 1); + assert!(cached_items.iter().any(|(_, _, name)| name.as_ref() == "new-live")); + assert!(cached_items.iter().any(|(_, _, name)| name.as_ref() == "new-series")); + } + + #[tokio::test] + async fn target_empty_replacement_failures_leave_the_memory_cache_unchanged() { + for failure in [ + crate::TargetEmptyReplacementFailure::CategoryPersistence, + crate::TargetEmptyReplacementFailure::BTreePersistence, + crate::TargetEmptyReplacementFailure::Publication, + ] { + let directory = tempdir().expect("temporary storage"); + let app_config = test_app_config(directory.path()); + let playlist_state = Arc::new(PlaylistStorageState::new()); + let target = xtream_target(&format!("empty-persist-{failure:?}"), true); + seed_memory_cached_xtream_target(&app_config, &playlist_state, &target).await; + let before = cached_target_signature(&playlist_state, &target.name).await; + let mut candidate = vec![ + target_group(XtreamCluster::Live, 102, "new-live"), + target_group(XtreamCluster::Series, 302, "new-series"), + ]; + + let errors = persist_playlist_with_mode( + &app_config, + &mut candidate, + None, + &target, + Some(&playlist_state), + TargetPlaylistPersistOptions { + accepted_empty_clusters: ClusterFlags::Vod, + ..TargetPlaylistPersistOptions::default() + }, + TargetPersistenceMode::FailEmptyReplacementAt(failure), + ) + .await + .expect_err("injected empty replacement must fail"); + + assert!(errors.iter().any(|error| error.to_string().contains("target cluster failed"))); + assert_eq!(cached_target_signature(&playlist_state, &target.name).await, before); + let persisted = load_xtream_target_storage(&app_config, &target).await.expect("retained target storage"); + assert_eq!(persisted.vod.len(), 1); + } + } + + #[tokio::test] + async fn force_empty_output_filters_reject_m3u_and_strm_before_mutation() { + let filter = get_filter(r#"EpgId ~ "^Mixed\.Case$""#, None).expect("case-sensitive EPG-ID filter"); + let cases = [ + ( + "filtered-m3u", + TargetOutput::M3u(M3uTargetOutput { + filename: Some("client.m3u".to_string()), + include_type_in_url: false, + mask_redirect_url: false, + filter: Some(filter.clone()), + }), + "client.m3u", + ), + ( + "filtered-strm", + TargetOutput::Strm(StrmTargetOutput { + directory: "client-strm".to_string(), + username: None, + style: StrmExportStyle::Jellyfin, + flags: StrmTargetFlagsSet::new(), + strm_props: None, + filter: Some(filter), + probe_probe_size_bytes: None, + probe_analyze_duration: None, + }), + "client-strm/old.strm", + ), + ]; + + for (target_name, output, artifact) in cases { + let directory = tempdir().expect("temporary storage"); + let app_config = test_app_config(directory.path()); + let artifact_path = directory.path().join(artifact); + tokio::fs::create_dir_all(artifact_path.parent().expect("artifact parent")) + .await + .expect("artifact directory"); + tokio::fs::write(&artifact_path, b"previous-client-state").await.expect("previous client artifact"); + let mut target = target_with_options(target_name, vec![output], false); + target.options = Some(epg_normalization_options(true)); + let target_path = { + let config = app_config.config.load(); + crate::get_target_storage_path(&config, &target.name).expect("target storage path") + }; + tokio::fs::create_dir_all(&target_path).await.expect("target storage directory"); + let mapping_path = crate::get_target_id_mapping_file(&target_path); + let mapping_uuid_path = mapping_path.with_extension("uuid.db"); + let seed_group = target_group(XtreamCluster::Live, 100, "mapping-seed"); + { + let seed_header = &seed_group.channels[0].header; + let mut mapping = TargetIdMapping::new(&mapping_path, false).expect("seed target ID mapping"); + mapping.get_and_update_virtual_id( + seed_header.get_uuid(), + 0, + seed_header.item_type, + VirtualId::default(), + ); + mapping.persist().expect("persist target ID mapping seed"); + } + let mapping_before = [ + tokio::fs::read(&mapping_path).await.expect("target ID mapping"), + tokio::fs::read(&mapping_uuid_path).await.expect("target UUID mapping"), + ]; + let mut candidate = vec![target_group(XtreamCluster::Live, 101, "filtered-after-normalization")]; + candidate[0].channels[0].header.epg_channel_id = Some("Mixed.Case".intern()); + + let errors = persist_playlist_with_mode( + &app_config, + &mut candidate, + None, + &target, + None, + TargetPlaylistPersistOptions { + accepted_empty_clusters: ClusterFlags::Vod, + ..TargetPlaylistPersistOptions::default() + }, + TargetPersistenceMode::Persist, + ) + .await + .expect_err("force-empty filtered output must be rejected"); + + assert!(errors.iter().any(|error| error.to_string().contains("after its output filter"))); + assert_eq!(candidate[0].channels[0].header.epg_channel_id.as_deref(), Some("mixed.case")); + assert_eq!( + tokio::fs::read(&artifact_path).await.expect("retained client artifact"), + b"previous-client-state" + ); + assert_eq!(tokio::fs::read(&mapping_path).await.expect("retained target ID mapping"), mapping_before[0]); + assert_eq!( + tokio::fs::read(&mapping_uuid_path).await.expect("retained target UUID mapping"), + mapping_before[1] + ); + } + } #[test] fn playlist_without_channels_is_empty_for_persistence() { diff --git a/backend/repository/src/stalker_generation_repository.rs b/backend/repository/src/stalker_generation_repository.rs index e5e5811d2..1c759f389 100644 --- a/backend/repository/src/stalker_generation_repository.rs +++ b/backend/repository/src/stalker_generation_repository.rs @@ -67,6 +67,9 @@ pub struct StalkerCheckpoint { pub processed: u64, pub skipped_count: u64, pub skipped_sample: Vec, + /// Requested media clusters whose quality check is bypassed for this generation. + #[serde(default)] + pub quality_bypass_mask: u8, } impl StalkerCheckpoint { @@ -83,6 +86,7 @@ impl StalkerCheckpoint { processed: 0, skipped_count: 0, skipped_sample: Vec::new(), + quality_bypass_mask: 0, } } } @@ -398,12 +402,25 @@ mod tests { let mut checkpoint = StalkerCheckpoint::new(7, 3, 1, 123); checkpoint.phase = StalkerRefreshPhase::Vod { page: 9 }; checkpoint.processed = 42; + checkpoint.quality_bypass_mask = 5; save_checkpoint(temp.path(), &checkpoint).await?; assert_eq!(load_checkpoint(temp.path(), 7).await?, Some(checkpoint)); Ok(()) } + #[test] + fn legacy_checkpoint_defaults_quality_bypass_mask_to_zero() { + let mut serialized = + serde_json::to_value(StalkerCheckpoint::new(7, 3, 1, 123)).expect("checkpoint should serialize"); + serialized.as_object_mut().expect("checkpoint should serialize as an object").remove("quality_bypass_mask"); + + let checkpoint: StalkerCheckpoint = + serde_json::from_value(serialized).expect("legacy checkpoint should deserialize"); + + assert_eq!(checkpoint.quality_bypass_mask, 0); + } + #[test] fn checkpoint_keeps_generation_start_time() { let checkpoint = StalkerCheckpoint::new(7, 3, 1, 1_723_456_789); diff --git a/backend/repository/src/stalker_repository.rs b/backend/repository/src/stalker_repository.rs index e16c1f1df..e6c11e96a 100644 --- a/backend/repository/src/stalker_repository.rs +++ b/backend/repository/src/stalker_repository.rs @@ -13,7 +13,7 @@ //! takes a write lock for each file it removes. use crate::{ - bplustree::{BPlusTree, BPlusTreeQuery, BPlusTreeUpdate, FlushPolicy}, + bplustree::{BPlusTree, BPlusTreeError, BPlusTreeQuery, BPlusTreeUpdate, FlushPolicy}, storage::ensure_input_storage_path, storage_const, }; @@ -28,6 +28,7 @@ use shared::{ }; use std::{ collections::HashSet, + io::ErrorKind, ops::Bound, path::{Path, PathBuf}, sync::Arc, @@ -259,6 +260,33 @@ pub async fn load_stalker_items_at( load_stalker_items_after(app_config, file_path, None, usize::MAX).await } +/// Counts one Stalker generation file without materializing its records. +/// +/// The read lock is acquired before opening the tree, so a missing file is a +/// stable `None` result rather than an existence-check race. +pub async fn count_stalker_items_at( + app_config: &Arc, + file_path: &Path, +) -> Result, TuliproxError> { + let file_lock = app_config.file_locks.read_lock(file_path).await; + let blocking_path = file_path.to_path_buf(); + tokio::task::spawn_blocking(move || { + let _guard = file_lock; + let mut query = match BPlusTreeQuery::::try_new(&blocking_path) { + Ok(query) => query, + Err(err) if err.kind() == ErrorKind::NotFound => return Ok(None), + Err(err) => return Err(repo_err!("open {} failed: {err}", blocking_path.display())), + }; + query + .len() + .map(Some) + .map_err(BPlusTreeError::to_io) + .map_err(|err| repo_err!("count {} failed: {err}", blocking_path.display())) + }) + .await + .map_err(|err| repo_err!("blocking task join error: {err}"))? +} + pub async fn prepare_stalker_episode_series_at( app_config: &Arc, file_path: &Path, diff --git a/backend/repository/src/strm_repository.rs b/backend/repository/src/strm_repository.rs index 763eb9f63..ea2d2b156 100644 --- a/backend/repository/src/strm_repository.rs +++ b/backend/repository/src/strm_repository.rs @@ -807,8 +807,9 @@ pub async fn write_strm_playlist( target: &ConfigTarget, target_output: &StrmTargetOutput, new_playlist: &mut [PlaylistGroup], + library_empty: crate::LibraryEmptyPublication, ) -> Result<(), TuliproxError> { - if new_playlist.is_empty() { + if new_playlist.is_empty() && !library_empty.replaces_empty_target() { return Ok(()); } diff --git a/backend/repository/src/target_id_mapping.rs b/backend/repository/src/target_id_mapping.rs index 5d709d90e..094249068 100644 --- a/backend/repository/src/target_id_mapping.rs +++ b/backend/repository/src/target_id_mapping.rs @@ -232,6 +232,14 @@ impl TargetIdMapping { !self.pending_virtual_id_upserts.is_empty() || !self.pending_uuid_upserts.is_empty() } + /// Discards the in-memory batch when target persistence is rejected before + /// any output is written. This prevents [`Drop`] from committing mapping + /// changes for a target state that never became client-visible. + pub(crate) fn discard_unpersisted_changes(&mut self) { + self.pending_virtual_id_upserts.clear(); + self.pending_uuid_upserts.clear(); + } + pub fn find_virtual_ids(&self, provider_id: u32) -> Vec { self.mem_by_virtual_id .values() diff --git a/backend/repository/src/user_repository.rs b/backend/repository/src/user_repository.rs index 349273b3b..bdd1e506a 100644 --- a/backend/repository/src/user_repository.rs +++ b/backend/repository/src/user_repository.rs @@ -254,7 +254,7 @@ fn user_get_series_bouquet_path(user_storage_path: &Path, target: TargetType) -> } async fn save_xtream_user_bouquet_for_target( - config: &Config, + app_config: &AppConfig, target_name: &str, storage_path: &Path, cluster: XtreamCluster, @@ -268,7 +268,7 @@ async fn save_xtream_user_bouquet_for_target( match bouquet { Some(bouquet_categories) => { - if let Some(xtream_categories) = xtream_get_playlist_categories(config, target_name, cluster).await { + if let Some(xtream_categories) = xtream_get_playlist_categories(app_config, target_name, cluster).await { let filtered: Vec = xtream_categories.iter().filter(|p| bouquet_categories.contains(&p.name)).cloned().collect(); if filtered.is_empty() { @@ -321,7 +321,7 @@ async fn save_m3u_user_bouquet_for_target( } async fn save_user_bouquet_for_target( - config: &Config, + app_config: &AppConfig, target_name: &str, storage_path: &Path, target: TargetType, @@ -329,7 +329,7 @@ async fn save_user_bouquet_for_target( ) -> Result<(), Error> { if target == TargetType::Xtream { save_xtream_user_bouquet_for_target( - config, + app_config, target_name, storage_path, XtreamCluster::Live, @@ -337,7 +337,7 @@ async fn save_user_bouquet_for_target( ) .await?; save_xtream_user_bouquet_for_target( - config, + app_config, target_name, storage_path, XtreamCluster::Video, @@ -345,7 +345,7 @@ async fn save_user_bouquet_for_target( ) .await?; save_xtream_user_bouquet_for_target( - config, + app_config, target_name, storage_path, XtreamCluster::Series, @@ -379,15 +379,26 @@ async fn save_user_bouquet_for_target( } pub async fn save_user_bouquet( - cfg: &Config, + app_config: &AppConfig, target_name: &str, username: &str, bouquet: &PlaylistBouquetDto, ) -> Result<(), Error> { - if let Some(storage_path) = ensure_user_storage_path(cfg, username) { - save_user_bouquet_for_target(cfg, target_name, &storage_path, TargetType::Xtream, bouquet.xtream.as_ref()) + let storage_path = { + let config = app_config.config.load(); + ensure_user_storage_path(&config, username) + }; + if let Some(storage_path) = storage_path { + save_user_bouquet_for_target( + app_config, + target_name, + &storage_path, + TargetType::Xtream, + bouquet.xtream.as_ref(), + ) + .await?; + save_user_bouquet_for_target(app_config, target_name, &storage_path, TargetType::M3u, bouquet.m3u.as_ref()) .await?; - save_user_bouquet_for_target(cfg, target_name, &storage_path, TargetType::M3u, bouquet.m3u.as_ref()).await?; Ok(()) } else { Err(Error::new(std::io::ErrorKind::NotFound, format!("User config path not found for user {username}"))) @@ -641,7 +652,32 @@ mod tests { let bouquet_path = user_get_live_bouquet_path(dir.path(), TargetType::Xtream); tokio::fs::write(&bouquet_path, "[]").await.expect("test bouquet file should be created"); - save_xtream_user_bouquet_for_target(&Config::default(), "target", dir.path(), XtreamCluster::Live, None) + let app_config = AppConfig { + config: Arc::new(ArcSwapAny::default()), + sources: Arc::new(ArcSwapAny::default()), + hdhomerun: Arc::new(ArcSwapAny::default()), + api_proxy: Arc::new(ArcSwapAny::default()), + paths: Arc::new(ArcSwap::from(Arc::new(ConfigPaths { + home_path: String::new(), + config_path: String::new(), + storage_path: String::new(), + config_file_path: String::new(), + sources_file_path: String::new(), + mapping_file_path: None, + mapping_files_used: None, + template_file_path: None, + template_files_used: None, + api_proxy_file_path: String::new(), + custom_stream_response_path: None, + }))), + file_locks: Arc::new(FileLockManager::default()), + custom_stream_response: Arc::new(ArcSwapAny::default()), + access_token_secret: Default::default(), + encrypt_secret: Default::default(), + media_tools: Arc::new(MediaToolCapabilities::new()), + }; + + save_xtream_user_bouquet_for_target(&app_config, "target", dir.path(), XtreamCluster::Live, None) .await .expect("saving empty xtream bouquet should succeed"); diff --git a/backend/repository/src/xtream_repository.rs b/backend/repository/src/xtream_repository.rs index 185f0dde2..576849b8b 100644 --- a/backend/repository/src/xtream_repository.rs +++ b/backend/repository/src/xtream_repository.rs @@ -1,8 +1,8 @@ use super::playlist_mem_cache::PlaylistStorageState; use crate::{ bplustree::{ - ensure_distinct_sidecar_lock_domains, publish_staged_database, BPlusTree, BPlusTreeError, BPlusTreeQuery, - BPlusTreeStagingArtifacts, BPlusTreeUpdate, FlushPolicy, + ensure_distinct_sidecar_lock_domains, get_file_path_for_db_index, publish_staged_database, BPlusTree, + BPlusTreeError, BPlusTreeQuery, BPlusTreeStagingArtifacts, BPlusTreeUpdate, FlushPolicy, }, error_macros::{cant_read_result, cant_write_result}, playlist_backend::{ensure_storage_path, iter_raw_playlist, PlaylistBackend, PlaylistKey, Xtream}, @@ -19,18 +19,21 @@ use crate::{ use bytes::Bytes; use futures::{stream, Stream, StreamExt}; use indexmap::IndexMap; -use log::error; +use log::{error, warn}; use serde::{Deserialize, Serialize}; use serde_json::{json, Value}; use shared::{ concat_string, error::{string_to_io_error, TuliproxError}, model::{ - xtream_const::XTREAM_CLUSTER, LiveStreamProperties, PlaylistGroup, PlaylistItem, PlaylistItemType, ProviderId, - SeriesStreamProperties, StreamProperties, VideoStreamProperties, VirtualId, XtreamCluster, XtreamPlaylistItem, + xtream_const::XTREAM_CLUSTER, ClusterFlags, LiveStreamProperties, PlaylistGroup, PlaylistItem, + PlaylistItemType, ProviderId, SeriesStreamProperties, StreamProperties, VideoStreamProperties, VirtualId, + XtreamCluster, XtreamPlaylistItem, }, utils::{arc_str_serde, get_u32_from_serde_value, Internable}, }; +#[cfg(unix)] +use std::fs; use std::{ collections::{HashMap, HashSet}, ffi::OsString, @@ -39,14 +42,17 @@ use std::{ path::{Path, PathBuf}, sync::Arc, }; +#[cfg(not(windows))] +use tuliprox_core::utils::parent_or_dot; use tuliprox_core::{ model::{ - AppConfig, Config, ConfigInput, ConfigInputFlags, ConfigTarget, PlaylistXtreamCategory, ProxyUserCredentials, - XtreamCategory, + evaluate_update_quality, AppConfig, ClusterForceUpdate, ClusterUpdateAcceptance, ClusterUpdateRejection, + Config, ConfigInput, ConfigInputFlags, ConfigTarget, PlaylistXtreamCategory, ProxyUserCredentials, + UpdateQualityDecision, XtreamCategory, }, utils::{ - file_exists_async, file_reader, json_write_documents_to_file, parent_or_dot, remove_file_if_exists, - request::DynReader, require_same_parent_directory, FileReadGuard, FileWriteGuard, + file_exists_async, file_reader, json_write_documents_to_file, remove_file_if_exists, request::DynReader, + require_same_parent_directory, FileReadGuard, FileWriteGuard, }, }; use tuliprox_parser::xtream; @@ -79,6 +85,10 @@ pub fn get_series_cat_collection_path(path: &Path) -> PathBuf { get_collection_path(path, storage_const::COL_CAT_SERIES) } +fn target_category_lock_path(category_path: &Path) -> PathBuf { + category_path.with_extension("json.target-category.lock") +} + #[inline] pub async fn ensure_xtream_storage_path(cfg: &Config, target_name: &str) -> Result { ensure_storage_path::(cfg, target_name).await @@ -98,13 +108,14 @@ async fn write_playlists_to_file( with_index: bool, key_of: F, collections: Vec<(XtreamCluster, Vec)>, + replace_empty_clusters: ClusterFlags, ) -> Result<(), TuliproxError> where K: PlaylistKey, F: Fn(&XtreamPlaylistItem) -> K + Copy + Send + 'static, { for (cluster, playlist) in collections { - if playlist.is_empty() { + if playlist.is_empty() && !replace_empty_clusters.contains(cluster_flag(cluster)) { continue; } let xtream_path = xtream_get_file_path(storage_path, cluster); @@ -136,6 +147,394 @@ where Ok(()) } +#[cfg(test)] +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum TargetEmptyReplacementFailure { + CategoryPersistence, + BTreePersistence, + Publication, +} + +#[derive(Debug, Clone, Default)] +enum TargetEmptyReplacementMode { + #[default] + Persist, + #[cfg(test)] + FailAt(TargetEmptyReplacementFailure), + #[cfg(test)] + PauseDuringPublication(TargetEmptyPublicationHook), +} + +#[cfg(test)] +#[derive(Debug, Clone)] +struct TargetEmptyPublicationHook { + backup_window_entered: Arc, + resume_publication: Arc, +} + +#[cfg(test)] +impl TargetEmptyPublicationHook { + fn new() -> Self { + Self { + backup_window_entered: Arc::new(std::sync::Barrier::new(2)), + resume_publication: Arc::new(std::sync::Barrier::new(2)), + } + } +} + +struct TargetEmptyClusterPaths { + published_database: PathBuf, + published_index: PathBuf, + published_categories: PathBuf, + staging_database: PathBuf, + staging_index: PathBuf, + staging_categories: PathBuf, +} + +impl TargetEmptyClusterPaths { + fn new(storage_path: &Path, cluster: XtreamCluster) -> Self { + let token = Uuid::new_v4().simple(); + let cluster_name = cluster.as_str().to_lowercase(); + let published_database = xtream_get_file_path(storage_path, cluster); + let staging_database = storage_path.join(format!(".{cluster_name}.force-empty-{token}.db")); + Self { + published_index: get_file_path_for_db_index(&published_database), + published_categories: get_collection_path(storage_path, xtream_cluster_category_collection(cluster)), + staging_index: get_file_path_for_db_index(&staging_database), + staging_categories: storage_path.join(format!(".{cluster_name}.force-empty-{token}.json")), + published_database, + staging_database, + } + } +} + +struct TargetFileReplacement { + published: PathBuf, + staging: PathBuf, + backup: PathBuf, + previous_moved: bool, + replacement_published: bool, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum TargetFileMoveMode { + PreserveDestination, + ReplaceDestination, +} + +#[cfg(unix)] +fn move_target_file_platform(source: &Path, destination: &Path, mode: TargetFileMoveMode) -> io::Result<()> { + require_same_parent_directory(source, destination)?; + if mode == TargetFileMoveMode::PreserveDestination && destination.exists() { + return Err(io::Error::new( + io::ErrorKind::AlreadyExists, + format!("target transaction destination already exists: {}", destination.display()), + )); + } + fs::rename(source, destination) +} + +#[cfg(windows)] +fn move_target_file_platform(source: &Path, destination: &Path, mode: TargetFileMoveMode) -> io::Result<()> { + use windows_sys::Win32::Storage::FileSystem::{MoveFileExW, MOVEFILE_REPLACE_EXISTING, MOVEFILE_WRITE_THROUGH}; + + require_same_parent_directory(source, destination)?; + let source_encoded = encode_windows_path(source)?; + let destination_encoded = encode_windows_path(destination)?; + let flags = MOVEFILE_WRITE_THROUGH + | if mode == TargetFileMoveMode::ReplaceDestination { MOVEFILE_REPLACE_EXISTING } else { 0 }; + + // SAFETY: both buffers are live, immutable, and NUL-terminated for the + // duration of the call. The same-directory check prevents a cross-volume + // move from degrading into a copy. + let result = unsafe { MoveFileExW(source_encoded.as_ptr(), destination_encoded.as_ptr(), flags) }; + if result == 0 { + let error = io::Error::last_os_error(); + return Err(io::Error::new( + error.kind(), + format!( + "failed to move target transaction file {} to {} with Windows write-through semantics: {error}", + source.display(), + destination.display() + ), + )); + } + Ok(()) +} + +#[cfg(all(not(unix), not(windows)))] +fn move_target_file_platform(source: &Path, destination: &Path, _mode: TargetFileMoveMode) -> io::Result<()> { + Err(io::Error::new( + io::ErrorKind::Unsupported, + format!( + "durable target transaction moves are unsupported on this platform: {} -> {}", + source.display(), + destination.display() + ), + )) +} + +impl TargetFileReplacement { + fn new(published: &Path, staging: &Path, token: uuid::fmt::Simple) -> Self { + let filename = + published.file_name().map_or_else(|| "target-artifact".into(), |name| name.to_string_lossy().into_owned()); + Self { + published: published.to_path_buf(), + staging: staging.to_path_buf(), + backup: published.with_file_name(format!(".{filename}.force-empty-backup-{token}")), + previous_moved: false, + replacement_published: false, + } + } +} + +fn cleanup_target_empty_staging( + staging_artifacts: &BPlusTreeStagingArtifacts, + staging_categories: &Path, +) -> io::Result<()> { + let database_result = staging_artifacts.remove_owned_staging_artifacts(); + let category_result = remove_file_if_exists(staging_categories); + match (database_result, category_result) { + (Ok(()), Ok(())) => Ok(()), + (Err(error), Ok(())) | (Ok(()), Err(error)) => Err(error), + (Err(database_error), Err(category_error)) => Err(io::Error::new( + database_error.kind(), + format!("{database_error}; category staging cleanup also failed: {category_error}"), + )), + } +} + +fn rollback_target_file_replacements(replacements: &mut [TargetFileReplacement]) -> io::Result<()> { + let mut errors = Vec::new(); + for replacement in replacements.iter_mut().rev() { + if replacement.previous_moved { + if let Err(error) = move_target_file_platform( + &replacement.backup, + &replacement.published, + TargetFileMoveMode::ReplaceDestination, + ) { + errors.push(format!( + "failed to restore {} from {}: {error}", + replacement.published.display(), + replacement.backup.display() + )); + } + } else if replacement.replacement_published { + if let Err(error) = move_target_file_platform( + &replacement.published, + &replacement.backup, + TargetFileMoveMode::ReplaceDestination, + ) { + errors.push(format!( + "failed to withdraw replacement {} during rollback: {error}", + replacement.published.display() + )); + } else if let Err(error) = remove_file_if_exists(&replacement.backup) { + errors + .push(format!("failed to remove withdrawn replacement {}: {error}", replacement.backup.display())); + } + } + } + + if errors.is_empty() { + Ok(()) + } else { + Err(io::Error::other(errors.join("; "))) + } +} + +fn publish_target_file_replacements( + paths: &TargetEmptyClusterPaths, + mode: &TargetEmptyReplacementMode, +) -> io::Result<()> { + #[cfg(not(test))] + let _ = &mode; + let staging_artifacts = BPlusTreeStagingArtifacts::new(&paths.published_database, &paths.staging_database)?; + let token = Uuid::new_v4().simple(); + let mut replacements = vec![ + TargetFileReplacement::new(&paths.published_database, &paths.staging_database, token), + TargetFileReplacement::new(&paths.published_index, &paths.staging_index, token), + TargetFileReplacement::new(&paths.published_categories, &paths.staging_categories, token), + ]; + + let publication = (|| -> io::Result<()> { + for replacement in &replacements { + require_same_parent_directory(&replacement.staging, &replacement.published)?; + if !replacement.staging.is_file() { + return Err(io::Error::new( + io::ErrorKind::NotFound, + format!("missing prepared target artifact {}", replacement.staging.display()), + )); + } + } + + for replacement in &mut replacements { + if replacement.published.exists() { + move_target_file_platform( + &replacement.published, + &replacement.backup, + TargetFileMoveMode::PreserveDestination, + )?; + replacement.previous_moved = true; + } + } + + #[cfg(test)] + if let TargetEmptyReplacementMode::PauseDuringPublication(hook) = mode { + hook.backup_window_entered.wait(); + hook.resume_publication.wait(); + } + + for (index, replacement) in replacements.iter_mut().enumerate() { + move_target_file_platform( + &replacement.staging, + &replacement.published, + TargetFileMoveMode::ReplaceDestination, + )?; + replacement.replacement_published = true; + #[cfg(test)] + if matches!(mode, TargetEmptyReplacementMode::FailAt(TargetEmptyReplacementFailure::Publication)) + && index == 0 + { + return Err(io::Error::other("injected target empty-replacement publication failure")); + } + #[cfg(not(test))] + let _ = index; + } + sync_published_file_parent(&paths.published_database)?; + cleanup_target_empty_staging(&staging_artifacts, &paths.staging_categories) + })(); + + if let Err(publication_error) = publication { + let rollback_result = rollback_target_file_replacements(&mut replacements); + let durability_result = sync_published_file_parent(&paths.published_database); + return match (rollback_result, durability_result) { + (Ok(()), Ok(())) => Err(publication_error), + (rollback, durability) => Err(io::Error::new( + publication_error.kind(), + format!( + "{publication_error}; rollback result: {}; rollback directory sync result: {}", + rollback.map_or_else(|error| error.to_string(), |()| "ok".to_string()), + durability.map_or_else(|error| error.to_string(), |()| "ok".to_string()) + ), + )), + }; + } + + for replacement in replacements { + if replacement.previous_moved { + if let Err(error) = remove_file_if_exists(&replacement.backup) { + warn!( + "Target empty replacement was published, but backup cleanup failed for {}: {error}", + replacement.backup.display() + ); + } + } + } + Ok(()) +} + +async fn replace_target_xtream_cluster_with_empty( + app_config: &Arc, + storage_path: &Path, + cluster: XtreamCluster, + mode: TargetEmptyReplacementMode, +) -> Result<(), TuliproxError> { + #[cfg(not(test))] + let _ = &mode; + let paths = TargetEmptyClusterPaths::new(storage_path, cluster); + let staging_artifacts = BPlusTreeStagingArtifacts::new(&paths.published_database, &paths.staging_database) + .map_err(|error| { + TuliproxError::RepositoryXtream(format!("Failed to prepare empty {cluster} target: {error}")) + })?; + + let operation = async { + #[cfg(test)] + if matches!(&mode, TargetEmptyReplacementMode::FailAt(TargetEmptyReplacementFailure::CategoryPersistence)) { + return Err(TuliproxError::RepositoryXtream("injected target category persistence failure".to_string())); + } + json_write_documents_to_file(&paths.staging_categories, &Vec::::new()).await.map_err( + |error| { + TuliproxError::RepositoryXtream(format!( + "Failed to prepare empty {cluster} target categories {}: {error}", + paths.staging_categories.display() + )) + }, + )?; + let staging_category_file = tokio::fs::File::open(&paths.staging_categories).await.map_err(|error| { + TuliproxError::RepositoryXtream(format!( + "Failed to reopen empty {cluster} target categories {}: {error}", + paths.staging_categories.display() + )) + })?; + staging_category_file.sync_all().await.map_err(|error| { + TuliproxError::RepositoryXtream(format!( + "Failed to synchronize empty {cluster} target categories {}: {error}", + paths.staging_categories.display() + )) + })?; + + #[cfg(test)] + if matches!(&mode, TargetEmptyReplacementMode::FailAt(TargetEmptyReplacementFailure::BTreePersistence)) { + return Err(TuliproxError::RepositoryXtream("injected target BTree/index persistence failure".to_string())); + } + let staging_database = paths.staging_database.clone(); + tokio::task::spawn_blocking(move || { + BPlusTree::::new().store_with_index(&staging_database, |item| item.source_ordinal) + }) + .await + .map_err(|error| TuliproxError::RepositoryXtream(format!("Empty target BTree task failed: {error}")))? + .map_err(|error| { + TuliproxError::RepositoryXtream(format!( + "Failed to prepare empty {cluster} target BTree/index {}: {error}", + paths.staging_database.display() + )) + })?; + + let category_lock_path = target_category_lock_path(&paths.published_categories); + let mut lock_paths = [&paths.published_database, &category_lock_path]; + lock_paths.sort_unstable(); + let mut file_locks = Vec::with_capacity(lock_paths.len()); + for path in lock_paths { + file_locks.push(app_config.file_locks.write_lock(path).await); + } + let publication_paths = TargetEmptyClusterPaths { + published_database: paths.published_database.clone(), + published_index: paths.published_index.clone(), + published_categories: paths.published_categories.clone(), + staging_database: paths.staging_database.clone(), + staging_index: paths.staging_index.clone(), + staging_categories: paths.staging_categories.clone(), + }; + let publication_mode = mode.clone(); + tokio::task::spawn_blocking(move || { + let result = publish_target_file_replacements(&publication_paths, &publication_mode); + drop(file_locks); + result + }) + .await + .map_err(|error| TuliproxError::RepositoryXtream(format!("Empty target publish task failed: {error}")))? + .map_err(|error| { + TuliproxError::RepositoryXtream(format!( + "Failed to publish empty {cluster} target cluster atomically: {error}" + )) + }) + } + .await; + + let cleanup = cleanup_target_empty_staging(&staging_artifacts, &paths.staging_categories); + match (operation, cleanup) { + (Ok(()), Ok(())) => Ok(()), + (Err(error), Ok(())) => Err(error), + (Ok(()), Err(cleanup_error)) => Err(TuliproxError::RepositoryXtream(format!( + "Empty {cluster} target cluster was published, but staging cleanup failed: {cleanup_error}" + ))), + (Err(error), Err(cleanup_error)) => { + Err(TuliproxError::RepositoryXtream(format!("{error}; staging cleanup also failed: {cleanup_error}"))) + } + } +} + pub async fn write_playlist_item_update( app_config: &Arc, target_name: &str, @@ -315,6 +714,42 @@ pub async fn xtream_write_playlist( app_cfg: &Arc, target: &ConfigTarget, playlist: &mut [PlaylistGroup], + replace_empty_clusters: ClusterFlags, +) -> Result<(), TuliproxError> { + xtream_write_playlist_with_mode( + app_cfg, + target, + playlist, + replace_empty_clusters, + TargetEmptyReplacementMode::Persist, + ) + .await +} + +#[cfg(test)] +pub(crate) async fn xtream_write_playlist_with_injected_empty_replacement_failure( + app_cfg: &Arc, + target: &ConfigTarget, + playlist: &mut [PlaylistGroup], + replace_empty_clusters: ClusterFlags, + failure: TargetEmptyReplacementFailure, +) -> Result<(), TuliproxError> { + xtream_write_playlist_with_mode( + app_cfg, + target, + playlist, + replace_empty_clusters, + TargetEmptyReplacementMode::FailAt(failure), + ) + .await +} + +async fn xtream_write_playlist_with_mode( + app_cfg: &Arc, + target: &ConfigTarget, + playlist: &mut [PlaylistGroup], + replace_empty_clusters: ClusterFlags, + empty_replacement_mode: TargetEmptyReplacementMode, ) -> Result<(), TuliproxError> { let path = { let config = app_cfg.config.load(); @@ -357,12 +792,21 @@ pub async fn xtream_write_playlist( let root_path = path.clone(); let app_config = app_cfg.clone(); - for (col_path, data) in [ - (get_live_cat_collection_path(&root_path), &cat_live_col), - (get_vod_cat_collection_path(&root_path), &cat_vod_col), - (get_series_cat_collection_path(&root_path), &cat_series_col), + for (cluster, col_path, data) in [ + (XtreamCluster::Live, get_live_cat_collection_path(&root_path), &cat_live_col), + (XtreamCluster::Video, get_vod_cat_collection_path(&root_path), &cat_vod_col), + (XtreamCluster::Series, get_series_cat_collection_path(&root_path), &cat_series_col), ] { - let lock = app_config.file_locks.write_lock(&col_path).await; + if data.is_empty() { + if replace_empty_clusters.contains(cluster_flag(cluster)) { + continue; + } + if file_exists_async(&col_path).await { + continue; + } + } + let category_lock_path = target_category_lock_path(&col_path); + let lock = app_config.file_locks.write_lock(&category_lock_path).await; match json_write_documents_to_file(&col_path, data).await { Ok(()) => {} Err(err) => { @@ -377,12 +821,24 @@ pub async fn xtream_write_playlist( for (cluster, col) in [(XtreamCluster::Live, &live_col), (XtreamCluster::Video, &vod_col), (XtreamCluster::Series, &series_col)] { - if col.is_empty() { + if col.is_empty() && replace_empty_clusters.contains(cluster_flag(cluster)) { + if let Err(error) = + replace_target_xtream_cluster_with_empty(app_cfg, &path, cluster, empty_replacement_mode.clone()).await + { + errors.push(format!("Persisting empty {cluster} target cluster failed: {error}")); + } continue; } let data = col.iter().map(|item| XtreamPlaylistItem::from(&**item)).collect::>(); - if let Err(err) = - write_playlists_to_file(app_cfg, &path, true, |item| item.virtual_id, vec![(cluster, data)]).await + if let Err(err) = write_playlists_to_file( + app_cfg, + &path, + true, + |item| item.virtual_id, + vec![(cluster, data)], + replace_empty_clusters, + ) + .await { errors.push(format!("Persisting collection failed:{err}")); } @@ -720,6 +1176,35 @@ pub async fn iter_raw_xtream_input_playlist( iter_raw_playlist::(app_config, &xtream_path, |_| true).await } +/// Counts entries in one active persisted raw input cluster without materializing them. +pub async fn count_input_xtream_cluster( + app_config: &AppConfig, + input: &ConfigInput, + cluster: XtreamCluster, +) -> Result, TuliproxError> { + let storage_dir = app_config.config.load().storage_dir.clone(); + let storage_path = get_input_storage_path(&input.name, &storage_dir).await.map_err(|err| { + TuliproxError::RepositoryXtream(format!("Failed to resolve active input storage for {}: {err}", input.name)) + })?; + let xtream_path = xtream_get_file_path(&storage_path, cluster); + let file_lock = app_config.file_locks.read_lock(&xtream_path).await; + let query_path = xtream_path.clone(); + let count = tokio::task::spawn_blocking(move || -> io::Result> { + let _guard = file_lock; + let mut query = match BPlusTreeQuery::::try_new(&query_path) { + Ok(query) => query, + Err(err) if err.kind() == ErrorKind::NotFound => return Ok(None), + Err(err) => return Err(err), + }; + query.len().map(Some).map_err(BPlusTreeError::to_io) + }) + .await + .map_err(|err| cant_read_result!(RepositoryXtream, "xtream", &xtream_path, err))? + .map_err(|err| cant_read_result!(RepositoryXtream, "xtream", &xtream_path, err))?; + + Ok(count) +} + pub fn playlist_iter_to_stream(channels: Option<(FileReadGuard, I)>) -> impl Stream> where I: Iterator + 'static, @@ -744,21 +1229,100 @@ where } pub async fn xtream_get_playlist_categories( - config: &Config, + app_config: &AppConfig, target_name: &str, cluster: XtreamCluster, ) -> Option> { - let path = xtream_get_collection_path(config, target_name, xtream_cluster_category_collection(cluster)); - if let Ok(file_path) = path { - if let Ok(content) = tokio::fs::read_to_string(&file_path).await { - return serde_json::from_str::>(&content).ok(); - } - } - None + let file_path = { + let config = app_config.config.load(); + let storage_path = xtream_get_storage_path(&config, target_name)?; + get_collection_path(&storage_path, xtream_cluster_category_collection(cluster)) + }; + let category_lock_path = target_category_lock_path(&file_path); + let _file_lock = app_config.file_locks.read_lock(&category_lock_path).await; + let content = tokio::fs::read_to_string(&file_path).await.ok()?; + serde_json::from_str::>(&content).ok() } const BATCH_SIZE: usize = 1000; +/// Result of publishing one fully staged Xtream input cluster. +#[derive(Debug, Clone, Copy, Eq, PartialEq)] +pub enum XtreamClusterPublishOutcome { + /// The staged database and categories replaced the active cluster. + Published, + /// The staged candidate passed its configured Quality guard and was published. + QualityAccepted(ClusterUpdateAcceptance), + /// The staged candidate was published through a request-local quality bypass. + ForcePublished(ClusterForceUpdate), + /// The staged candidate was rejected and the active cluster was retained. + RetainedPrevious(ClusterUpdateRejection), +} + +/// Decision reports, completed publications, and technical errors from one +/// ordered disk-based Xtream batch. +/// +/// Quality reports are recorded when the existing guard evaluates a cluster; +/// `outcomes` records only the later publication/retention result. Keeping the +/// two facts separate lets callers retain an evaluated decision when a +/// subsequent technical step fails. +#[derive(Debug, Default)] +pub struct XtreamClusterPublishBatchResult { + /// Clusters whose publication or retention completed. + pub outcomes: Vec, + /// Quality acceptances evaluated before any later technical failure. + pub quality_acceptances: Vec, + /// Quality rejections evaluated before any later technical failure. + pub quality_rejections: Vec, + /// Request-local Quality bypasses evaluated before any later technical failure. + pub force_updates: Vec, + /// Technical failures that stopped the ordered batch. + pub errors: Vec, + /// Known failing cluster; batch-wide failures remain unscoped. + pub failed_cluster: Option, +} + +impl XtreamClusterPublishBatchResult { + fn record_cluster_error(&mut self, cluster: XtreamCluster, error: TuliproxError) { + self.failed_cluster = Some(cluster); + self.errors.push(error); + } + + fn record_evaluation(&mut self, evaluation: XtreamClusterEvaluationReport) { + if let Some(acceptance) = evaluation.quality_acceptance { + self.quality_acceptances.push(acceptance); + } + if let Some(rejection) = evaluation.quality_rejection { + self.quality_rejections.push(rejection); + } + if let Some(force_update) = evaluation.force_update { + self.force_updates.push(force_update); + } + } +} + +/// Quality behavior for one staged Xtream cluster publication. +#[derive(Debug, Clone, Copy, Eq, PartialEq)] +pub enum XtreamClusterQualityPolicy { + Enforce { threshold: u8 }, + Bypass { configured_threshold: u8 }, +} + +/// Readers and publication policy for one fully independent Xtream cluster refresh. +pub struct XtreamClusterRefreshRequest { + pub cluster: XtreamCluster, + pub quality: XtreamClusterQualityPolicy, + pub categories: DynReader, + pub streams: DynReader, +} + +#[derive(Clone, Copy, Debug, Default)] +struct XtreamClusterEvaluationReport { + quality_acceptance: Option, + quality_rejection: Option, + force_update: Option, +} + #[derive(Debug, Clone, Eq, PartialEq)] struct XtreamRefreshPaths { generation: Uuid, @@ -1026,6 +1590,15 @@ fn preserve_details_with_injected_operation_failure( }) } +#[derive(Clone, Copy)] +struct XtreamClusterStageOperations { + preserve_details: fn(&Path, &Path) -> Result, +} + +impl Default for XtreamClusterStageOperations { + fn default() -> Self { Self { preserve_details: preserve_details_input_xtream_playlist_cluster_to_disk } } +} + struct StagedXtreamClusterRefresh { refresh_lease: XtreamRefreshLease, publish_lock: Arc, @@ -1034,16 +1607,78 @@ struct StagedXtreamClusterRefresh { cluster: XtreamCluster, raw_groups: Vec, item_count: usize, + evaluation: XtreamClusterEvaluationReport, +} + +enum StagedXtreamClusterOutcome { + Ready(StagedXtreamClusterRefresh), + RetainedPrevious(ClusterUpdateRejection), + Failed { evaluation: XtreamClusterEvaluationReport, error: TuliproxError }, +} + +impl StagedXtreamClusterOutcome { + const fn evaluation(&self) -> XtreamClusterEvaluationReport { + match self { + Self::Ready(refresh) => refresh.evaluation, + Self::RetainedPrevious(rejection) => XtreamClusterEvaluationReport { + quality_acceptance: None, + quality_rejection: Some(*rejection), + force_update: None, + }, + Self::Failed { evaluation, .. } => *evaluation, + } + } +} + +fn count_xtream_tree_entries(path: &Path) -> io::Result> { + let mut query = match BPlusTreeQuery::::try_new(path) { + Ok(query) => query, + Err(error) if error.kind() == ErrorKind::NotFound => return Ok(None), + Err(error) => return Err(error), + }; + query.len().map(Some).map_err(BPlusTreeError::to_io) +} + +fn evaluate_staged_xtream_cluster_quality( + paths: &XtreamRefreshPaths, + cluster: XtreamCluster, + threshold: u8, +) -> Result { + if threshold == 0 { + return Ok(UpdateQualityDecision::Disabled); + } + + let candidate_count = count_xtream_tree_entries(&paths.staging_database) + .map_err(|error| { + TuliproxError::RepositoryXtream(format!( + "Failed to count staging Xtream tree {} for {cluster} quality evaluation: {error}", + paths.staging_database.display() + )) + })? + .ok_or_else(|| { + TuliproxError::RepositoryXtream(format!( + "Staging Xtream tree {} disappeared before {cluster} quality evaluation", + paths.staging_database.display() + )) + })?; + let current_count = count_xtream_tree_entries(&paths.published_database).map_err(|error| { + TuliproxError::RepositoryXtream(format!( + "Failed to count published Xtream tree {} for {cluster} quality evaluation: {error}", + paths.published_database.display() + )) + })?; + + Ok(evaluate_update_quality(current_count, candidate_count, threshold)) } #[allow(clippy::too_many_lines)] async fn stage_input_xtream_playlist_cluster_to_disk( app_config: &Arc, input: &ConfigInput, - cluster: XtreamCluster, - categories: DynReader, - streams: DynReader, -) -> Result { + request: XtreamClusterRefreshRequest, + operations: XtreamClusterStageOperations, +) -> Result { + let XtreamClusterRefreshRequest { cluster, quality: quality_policy, categories, streams } = request; let cfg = app_config.config.load(); let storage_path = ensure_input_storage_path(&cfg, &input.name).await?; drop(cfg); @@ -1182,21 +1817,88 @@ async fn stage_input_xtream_playlist_cluster_to_disk( // handle escapes its blocking closure, so none is held while this async lock is acquired. let publish_lock = Arc::new(app_config.file_locks.write_lock(&refresh_lease.paths().published_database).await); + let quality_lease = refresh_lease.clone(); + let quality_lock = Arc::clone(&publish_lock); + let (evaluation, rejection_cleanup_error) = tokio::task::spawn_blocking(move || { + let _publish_guard = quality_lock; + let evaluation = match quality_policy { + XtreamClusterQualityPolicy::Enforce { threshold } => { + let decision = evaluate_staged_xtream_cluster_quality(quality_lease.paths(), cluster, threshold)?; + XtreamClusterEvaluationReport { + quality_acceptance: decision.acceptance(cluster), + quality_rejection: decision.rejection(cluster), + force_update: None, + } + } + XtreamClusterQualityPolicy::Bypass { configured_threshold } => { + let candidate_count = count_xtream_tree_entries(&quality_lease.paths().staging_database) + .map_err(|error| { + TuliproxError::RepositoryXtream(format!( + "Failed to count staging Xtream tree {} for forced {cluster} publication: {error}", + quality_lease.paths().staging_database.display() + )) + })? + .ok_or_else(|| { + TuliproxError::RepositoryXtream(format!( + "Staging Xtream tree {} disappeared before forced {cluster} publication", + quality_lease.paths().staging_database.display() + )) + })?; + XtreamClusterEvaluationReport { + quality_acceptance: None, + quality_rejection: None, + force_update: Some(ClusterForceUpdate { cluster, candidate_count, configured_threshold }), + } + } + }; + let rejection_cleanup_error = if evaluation.quality_rejection.is_some() { + quality_lease.cleanup_staging_artifacts().err().map(|error| { + TuliproxError::RepositoryXtream(format!( + "Failed to clean rejected Xtream staging artifacts for {cluster}: {error}" + )) + }) + } else { + None + }; + Ok::<_, TuliproxError>((evaluation, rejection_cleanup_error)) + }) + .await + .map_err(|error| { + TuliproxError::RepositoryXtream(format!( + "Quality-evaluation task failed to join during {cluster} refresh: {error}" + )) + })??; + + if let Some(error) = rejection_cleanup_error { + return Ok(StagedXtreamClusterOutcome::Failed { evaluation, error }); + } + + if let Some(rejection) = evaluation.quality_rejection { + drop(publish_lock); + log::debug!( + "Xtream cluster candidate rejected; retained active cluster: cluster={cluster} generation={} rejection={rejection:?}", + refresh_lease.paths().generation + ); + return Ok(StagedXtreamClusterOutcome::RetainedPrevious(rejection)); + } + let merge_lease = refresh_lease.clone(); let merge_lock = Arc::clone(&publish_lock); - let merge_outcome = tokio::task::spawn_blocking(move || { + let preserve_details = operations.preserve_details; + let merge_result = tokio::task::spawn_blocking(move || { let _publish_guard = merge_lock; - preserve_details_input_xtream_playlist_cluster_to_disk( - &merge_lease.paths().published_database, - &merge_lease.paths().staging_database, - ) + preserve_details(&merge_lease.paths().published_database, &merge_lease.paths().staging_database) }) .await .map_err(|error| { TuliproxError::RepositoryXtream(format!( "Detail-preservation task failed to join during {cluster} refresh: {error}" )) - })??; + }); + let merge_outcome = match merge_result { + Ok(Ok(outcome)) => outcome, + Ok(Err(error)) | Err(error) => return Ok(StagedXtreamClusterOutcome::Failed { evaluation, error }), + }; log::debug!( "Xtream cluster detail preservation completed: cluster={cluster} generation={} outcome={merge_outcome:?}", refresh_lease.paths().generation @@ -1204,7 +1906,7 @@ async fn stage_input_xtream_playlist_cluster_to_disk( let compact_lease = refresh_lease.clone(); let compact_lock = Arc::clone(&publish_lock); - tokio::task::spawn_blocking(move || { + let compact_result = tokio::task::spawn_blocking(move || { let _publish_guard = compact_lock; let staging_path = &compact_lease.paths().staging_database; let mut tree = @@ -1224,9 +1926,13 @@ async fn stage_input_xtream_playlist_cluster_to_disk( .await .map_err(|error| { TuliproxError::RepositoryXtream(format!("Compaction task failed to join during {cluster} refresh: {error}")) - })??; + }); + match compact_result { + Ok(Ok(())) => {} + Ok(Err(error)) | Err(error) => return Ok(StagedXtreamClusterOutcome::Failed { evaluation, error }), + } - Ok(StagedXtreamClusterRefresh { + Ok(StagedXtreamClusterOutcome::Ready(StagedXtreamClusterRefresh { refresh_lease, publish_lock, storage_path, @@ -1234,7 +1940,8 @@ async fn stage_input_xtream_playlist_cluster_to_disk( cluster, raw_groups, item_count, - }) + evaluation, + })) } async fn publish_staged_xtream_cluster( @@ -1249,6 +1956,7 @@ async fn publish_staged_xtream_cluster( cluster, raw_groups, item_count: _, + evaluation: _, } = staged; let database_publish_lease = refresh_lease.clone(); @@ -1361,52 +2069,152 @@ async fn published_xtream_cluster_has_items( pub async fn persist_input_xtream_playlist_clusters_to_disk( app_config: &Arc, input: &ConfigInput, - cluster_readers: Vec<(XtreamCluster, DynReader, DynReader)>, -) -> Result<(), TuliproxError> { + cluster_readers: Vec, +) -> XtreamClusterPublishBatchResult { + persist_input_xtream_playlist_clusters_to_disk_with_operations( + app_config, + input, + cluster_readers, + XtreamClusterStageOperations::default(), + ) + .await +} + +async fn persist_input_xtream_playlist_clusters_to_disk_with_operations( + app_config: &Arc, + input: &ConfigInput, + cluster_readers: Vec, + operations: XtreamClusterStageOperations, +) -> XtreamClusterPublishBatchResult { + let mut result = XtreamClusterPublishBatchResult::default(); let mut staged = Vec::with_capacity(cluster_readers.len()); - for (cluster, categories, streams) in cluster_readers { - staged - .push(stage_input_xtream_playlist_cluster_to_disk(app_config, input, cluster, categories, streams).await?); + for request in cluster_readers { + let cluster = request.cluster; + let outcome = match stage_input_xtream_playlist_cluster_to_disk(app_config, input, request, operations).await { + Ok(outcome) => outcome, + Err(error) => { + result.record_cluster_error(cluster, error); + return result; + } + }; + result.record_evaluation(outcome.evaluation()); + match outcome { + StagedXtreamClusterOutcome::Failed { error, .. } => { + result.record_cluster_error(cluster, error); + return result; + } + ready_or_retained => staged.push(ready_or_retained), + } } - let staged_clusters: HashSet = staged.iter().map(|refresh| refresh.cluster).collect(); - let mut has_items = staged.iter().any(|refresh| refresh.item_count > 0); - if !has_items { + let staged_clusters: HashSet = staged + .iter() + .filter_map(|outcome| match outcome { + StagedXtreamClusterOutcome::Ready(refresh) => Some(refresh.cluster), + StagedXtreamClusterOutcome::RetainedPrevious(_) | StagedXtreamClusterOutcome::Failed { .. } => None, + }) + .collect(); + let has_publishable_clusters = !staged_clusters.is_empty(); + let mut has_items = staged + .iter() + .any(|outcome| matches!(outcome, StagedXtreamClusterOutcome::Ready(refresh) if refresh.item_count > 0)); + if has_publishable_clusters && !has_items { let cfg = app_config.config.load(); - let storage_path = ensure_input_storage_path(&cfg, &input.name).await?; + let storage_path = match ensure_input_storage_path(&cfg, &input.name).await { + Ok(storage_path) => storage_path, + Err(error) => { + drop(cfg); + result.errors.push(error); + return result; + } + }; drop(cfg); for cluster in XTREAM_CLUSTER { - if input_cluster_enabled(input, cluster) - && !staged_clusters.contains(&cluster) - && published_xtream_cluster_has_items(app_config, &storage_path, cluster).await? - { - has_items = true; - break; + if input_cluster_enabled(input, cluster) && !staged_clusters.contains(&cluster) { + match published_xtream_cluster_has_items(app_config, &storage_path, cluster).await { + Ok(true) => { + has_items = true; + break; + } + Ok(false) => {} + Err(error) => { + result.errors.push(error); + return result; + } + } } } } - if !has_items { - return Err(TuliproxError::RepositoryPlaylist(format!( + let all_publishable_clusters_are_forced = has_publishable_clusters + && staged.iter().all( + |outcome| matches!(outcome, StagedXtreamClusterOutcome::Ready(refresh) if refresh.evaluation.force_update.is_some()), + ); + if has_publishable_clusters && !has_items && !all_publishable_clusters_are_forced { + result.errors.push(TuliproxError::RepositoryPlaylist(format!( "Refusing to publish empty disk-based Xtream playlist for input '{}'; existing data was retained", input.name ))); + return result; } - for refresh in staged { - publish_staged_xtream_cluster(app_config, refresh).await?; + for outcome in staged { + match outcome { + StagedXtreamClusterOutcome::Ready(refresh) => { + let evaluation = refresh.evaluation; + let cluster = refresh.cluster; + if let Err(error) = publish_staged_xtream_cluster(app_config, refresh).await { + result.record_cluster_error(cluster, error); + break; + } + result.outcomes.push(if let Some(force_update) = evaluation.force_update { + XtreamClusterPublishOutcome::ForcePublished(force_update) + } else if let Some(quality_acceptance) = evaluation.quality_acceptance { + XtreamClusterPublishOutcome::QualityAccepted(quality_acceptance) + } else { + XtreamClusterPublishOutcome::Published + }); + } + StagedXtreamClusterOutcome::RetainedPrevious(rejection) => { + result.outcomes.push(XtreamClusterPublishOutcome::RetainedPrevious(rejection)); + } + StagedXtreamClusterOutcome::Failed { error, .. } => { + result.errors.push(error); + break; + } + } } - Ok(()) + result } pub async fn persist_input_xtream_playlist_cluster_to_disk( app_config: &Arc, input: &ConfigInput, cluster: XtreamCluster, + quality_threshold: u8, categories: DynReader, streams: DynReader, -) -> Result<(), TuliproxError> { - persist_input_xtream_playlist_clusters_to_disk(app_config, input, vec![(cluster, categories, streams)]).await +) -> Result { + let mut result = persist_input_xtream_playlist_clusters_to_disk( + app_config, + input, + vec![XtreamClusterRefreshRequest { + cluster, + quality: XtreamClusterQualityPolicy::Enforce { threshold: quality_threshold }, + categories, + streams, + }], + ) + .await; + if let Some(error) = result.errors.pop() { + return Err(error); + } + result.outcomes.pop().ok_or_else(|| { + TuliproxError::RepositoryXtream(format!( + "Missing Xtream publish outcome for input '{}' cluster {cluster}", + input.name + )) + }) } fn publish_staged_file_same_directory(staging: &Path, published: &Path) -> io::Result<()> { @@ -1446,45 +2254,7 @@ fn publish_staged_file_with_parent_sync( #[cfg(windows)] fn publish_staged_file_platform(mut staging_path: tempfile::TempPath, published: &Path) -> io::Result<()> { - use std::os::windows::ffi::OsStrExt; - use windows_sys::Win32::Storage::FileSystem::{MoveFileExW, MOVEFILE_REPLACE_EXISTING, MOVEFILE_WRITE_THROUGH}; - - fn encode_path(path: &Path) -> io::Result> { - let mut encoded = path.as_os_str().encode_wide().collect::>(); - if encoded.contains(&0) { - return Err(io::Error::new( - io::ErrorKind::InvalidInput, - format!("Windows path contains an embedded NUL: {}", path.display()), - )); - } - encoded.push(0); - Ok(encoded) - } - - let staging_encoded = encode_path(staging_path.as_ref())?; - let published_encoded = encode_path(published)?; - - // SAFETY: both buffers are live, immutable, and NUL-terminated for the - // duration of the call. The same-directory check above ensures that the - // operation cannot degrade into a cross-volume copy. - let result = unsafe { - MoveFileExW( - staging_encoded.as_ptr(), - published_encoded.as_ptr(), - MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH, - ) - }; - if result == 0 { - let error = io::Error::last_os_error(); - return Err(io::Error::new( - error.kind(), - format!( - "failed to atomically publish staging file {} as {} with a Windows write-through rename: {error}", - staging_path.display(), - published.display() - ), - )); - } + move_target_file_platform(staging_path.as_ref(), published, TargetFileMoveMode::ReplaceDestination)?; // MoveFileExW consumed the source path. Prevent TempPath from issuing a // redundant delete for a path that no longer exists. @@ -1492,9 +2262,30 @@ fn publish_staged_file_platform(mut staging_path: tempfile::TempPath, published: Ok(()) } +#[cfg(windows)] +fn encode_windows_path(path: &Path) -> io::Result> { + use std::os::windows::ffi::OsStrExt; + + let mut encoded = path.as_os_str().encode_wide().collect::>(); + if encoded.contains(&0) { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + format!("Windows path contains an embedded NUL: {}", path.display()), + )); + } + encoded.push(0); + Ok(encoded) +} + #[cfg(unix)] fn sync_published_file_parent(path: &Path) -> io::Result<()> { File::open(parent_or_dot(path))?.sync_all() } +/// Every Windows transaction move uses `MOVEFILE_WRITE_THROUGH`; reaching +/// this barrier therefore means all preceding backup or publication moves +/// have completed durably without a second raw rename. +#[cfg(windows)] +fn sync_published_file_parent(_path: &Path) -> io::Result<()> { Ok(()) } + /// There is no supported directory durability barrier for other targets. /// Callers report this only after the atomic rename has completed. #[cfg(all(not(unix), not(windows)))] @@ -1567,11 +2358,21 @@ async fn save_xtream_categories_to_file( .map_err(|e| TuliproxError::RepositoryXtream(format!("Spawn error {e}")))? } -#[allow(clippy::too_many_lines)] pub async fn persist_input_xtream_playlist( app_config: &Arc, storage_path: &Path, playlist: Vec, +) -> (Vec, Option) { + persist_input_xtream_playlist_with_empty_replacements(app_config, storage_path, playlist, ClusterFlags::empty()) + .await +} + +#[allow(clippy::too_many_lines)] +pub async fn persist_input_xtream_playlist_with_empty_replacements( + app_config: &Arc, + storage_path: &Path, + playlist: Vec, + replace_empty_clusters: ClusterFlags, ) -> (Vec, Option) { let mut errors = Vec::new(); @@ -1644,7 +2445,7 @@ pub async fn persist_input_xtream_playlist( } fetched_col.push(pli); } - groups.insert(plg.id, plg); + groups.insert((plg.xtream_cluster, plg.id), plg); } } @@ -1652,7 +2453,10 @@ pub async fn persist_input_xtream_playlist( for xc in XTREAM_CLUSTER { processed_scratch.set( xc, - if !stored_scratch.is_empty(xc) && fetched_scratch.is_empty(xc) { + if !replace_empty_clusters.contains(cluster_flag(xc)) + && !stored_scratch.is_empty(xc) + && fetched_scratch.is_empty(xc) + { stored_scratch.take(xc).iter().map(|(_, item)| PlaylistItem::from(item)).collect::>() } else { fetched_scratch.take(xc) @@ -1668,7 +2472,10 @@ pub async fn persist_input_xtream_playlist( let col_path = get_collection_path(&root_path, xtream_cluster_category_collection(cluster)); let data = fetched_categories.get_mut(cluster); // if there is no data save only if no file exists! Prevent data loss from failed download attempt - if !data.is_empty() || !file_exists_async(&col_path).await { + if !data.is_empty() + || replace_empty_clusters.contains(cluster_flag(cluster)) + || !file_exists_async(&col_path).await + { let lock = app_cfg.file_locks.write_lock(&col_path).await; if let Err(err) = json_write_documents_to_file(&col_path, data).await { errors.push(format!("Persisting collection failed: {}: {err}", col_path.display())); @@ -1687,6 +2494,7 @@ pub async fn persist_input_xtream_playlist( false, |item| ProviderId::new(item.provider_id), vec![(cluster, col.iter().map(Into::into).collect::>())], + replace_empty_clusters, ) .await { @@ -1694,8 +2502,9 @@ pub async fn persist_input_xtream_playlist( } for item in col { + let group_key = (item.header.xtream_cluster, item.header.category_id); groups - .entry(item.header.category_id) + .entry(group_key) .or_insert_with(|| PlaylistGroup { id: item.header.category_id, title: item.header.group.clone(), @@ -1707,13 +2516,21 @@ pub async fn persist_input_xtream_playlist( } } - let result = groups.into_iter().map(|(_, group)| group).collect(); + let result = groups.into_values().collect(); let err = if errors.is_empty() { None } else { Some(TuliproxError::RepositoryXtream(errors.join("\n"))) }; (result, err) } +const fn cluster_flag(cluster: XtreamCluster) -> ClusterFlags { + match cluster { + XtreamCluster::Live => ClusterFlags::Live, + XtreamCluster::Video => ClusterFlags::Vod, + XtreamCluster::Series => ClusterFlags::Series, + } +} + // Checks if the info has changed after the last update pub fn needs_update_info_details(new_stream_props: &StreamProperties, old_stream_props: &StreamProperties) -> bool { let new_modified = new_stream_props.get_last_modified(); @@ -2052,21 +2869,31 @@ mod tests { #[cfg(unix)] use super::refresh_staging_path; use super::{ - merge_preserved_stream_properties, needs_update_info_details, persist_input_xtream_playlist_cluster_to_disk, - preserve_details_input_xtream_playlist_cluster_to_disk, preserve_details_with_injected_operation_failure, - publish_staged_file_same_directory, DetailPreservationOperation, PreserveDetailsOutcome, XtreamRefreshLease, - XtreamRefreshPaths, + count_input_xtream_cluster, get_collection_path, load_input_xtream_playlist, merge_preserved_stream_properties, + needs_update_info_details, persist_input_xtream_playlist, persist_input_xtream_playlist_cluster_to_disk, + persist_input_xtream_playlist_clusters_to_disk, persist_input_xtream_playlist_clusters_to_disk_with_operations, + persists_input_series_info, preserve_details_input_xtream_playlist_cluster_to_disk, + preserve_details_with_injected_operation_failure, publish_staged_file_same_directory, + target_category_lock_path, xtream_cluster_category_collection, xtream_get_playlist_categories, + xtream_write_playlist, xtream_write_playlist_with_injected_empty_replacement_failure, + xtream_write_playlist_with_mode, DetailPreservationOperation, PreserveDetailsOutcome, + TargetEmptyPublicationHook, TargetEmptyReplacementFailure, TargetEmptyReplacementMode, + XtreamClusterPublishOutcome, XtreamClusterQualityPolicy, XtreamClusterRefreshRequest, + XtreamClusterStageOperations, XtreamRefreshLease, XtreamRefreshPaths, }; use crate::{ bplustree::{ensure_distinct_sidecar_lock_domains, sidecar_lock_path}, build_input_storage_path, cleanup_orphaned_staging_artifacts, get_file_path_for_db_index, - refresh_generation_guard_path, BPlusTreeQuery, BPlusTreeUpdate, + get_input_storage_path, refresh_generation_guard_path, BPlusTreeQuery, BPlusTreeUpdate, }; use arc_swap::{ArcSwap, ArcSwapOption}; use shared::{ + error::TuliproxError, model::{ - CatchupProperties, ConfigPaths, InputType, LiveStreamProperties, SeriesStreamProperties, StreamProperties, - VideoStreamProperties, VirtualId, XtreamCluster, XtreamPlaylistItem, + CatchupProperties, ClusterFlags, ConfigPaths, InputType, LiveStreamProperties, PlaylistGroup, PlaylistItem, + PlaylistItemHeader, PlaylistItemType, ProcessingOrder, SeriesStreamDetailEpisodeProperties, + SeriesStreamDetailProperties, SeriesStreamProperties, StreamProperties, VideoStreamProperties, VirtualId, + XtreamCluster, XtreamPlaylistItem, }, utils::Internable, }; @@ -2082,8 +2909,10 @@ mod tests { use tokio::io::AsyncWriteExt; use tuliprox_core::{ model::{ - ApiProxyConfig, AppConfig, Config, ConfigInput, CustomStreamResponse, HdHomeRunConfig, - MediaToolCapabilities, SourcesConfig, + ApiProxyConfig, AppConfig, ClusterForceUpdate, ClusterUpdateAcceptance, ClusterUpdateRejection, Config, + ConfigInput, ConfigTarget, CustomStreamResponse, HdHomeRunConfig, MediaToolCapabilities, SourcesConfig, + StagedFilter, TargetExecutionPlan, TargetOutput, UpdateQualityDecision, XtreamTargetFlagsSet, + XtreamTargetOutput, }, utils::{request::DynReader, FileLockManager}, }; @@ -2119,10 +2948,239 @@ mod tests { }) } - fn json_reader(content: &'static str) -> DynReader { + fn target_writer_config() -> ConfigTarget { + ConfigTarget { + id: 1, + enabled: true, + name: "target-empty-cluster-guard".to_string(), + options: None, + sort: None, + filter: StagedFilter::default(), + output: vec![TargetOutput::Xtream(XtreamTargetOutput { + flags: XtreamTargetFlagsSet::new(), + trakt: None, + filter: None, + })], + rename: None, + mapping_ids: None, + mapping: Arc::new(ArcSwapOption::new(None)), + favourites: None, + processing_order: ProcessingOrder::default(), + execution_plan: TargetExecutionPlan::default(), + watch: None, + use_memory_cache: false, + } + } + + fn target_writer_group(cluster: XtreamCluster, category_id: u32, virtual_id: u32) -> PlaylistGroup { + let title = format!("target-{cluster}").intern(); + PlaylistGroup { + id: category_id, + title: Arc::clone(&title), + channels: vec![PlaylistItem { + header: PlaylistItemHeader { + id: virtual_id.to_string().intern(), + input_stream_id: virtual_id.to_string().intern(), + virtual_id: VirtualId::new(virtual_id), + name: Arc::clone(&title), + title: Arc::clone(&title), + group: title, + input_name: "target-writer-input".intern(), + item_type: PlaylistItemType::from(cluster), + xtream_cluster: cluster, + category_id, + ..PlaylistItemHeader::default() + }, + }], + xtream_cluster: cluster, + } + } + + #[tokio::test] + async fn target_writer_creates_missing_empty_category_files_without_force() -> Result<(), Box> + { + let directory = tempfile::tempdir()?; + let app_config = test_app_config(directory.path()); + let target = target_writer_config(); + let mut live_only = vec![target_writer_group(XtreamCluster::Live, 1, 101)]; + + xtream_write_playlist(&app_config, &target, &mut live_only, ClusterFlags::empty()).await?; + + let storage_path = { + let config = app_config.config.load(); + super::xtream_get_storage_path(&config, &target.name).expect("target Xtream storage") + }; + for cluster in [XtreamCluster::Video, XtreamCluster::Series] { + let categories = get_collection_path(&storage_path, xtream_cluster_category_collection(cluster)); + assert_eq!(tokio::fs::read(categories).await?, b"[]"); + assert!(!super::xtream_get_file_path(&storage_path, cluster).exists()); + } + Ok(()) + } + + #[tokio::test] + async fn target_writer_preserves_an_unauthorized_empty_cluster() -> Result<(), Box> { + let directory = tempfile::tempdir()?; + let app_config = test_app_config(directory.path()); + let target = target_writer_config(); + let mut baseline = vec![ + target_writer_group(XtreamCluster::Live, 1, 101), + target_writer_group(XtreamCluster::Video, 2, 201), + target_writer_group(XtreamCluster::Series, 3, 301), + ]; + xtream_write_playlist(&app_config, &target, &mut baseline, ClusterFlags::empty()).await?; + + let storage_path = { + let config = app_config.config.load(); + super::xtream_get_storage_path(&config, &target.name).expect("target Xtream storage") + }; + let retained_paths = [XtreamCluster::Video, XtreamCluster::Series].map(|cluster| { + ( + super::xtream_get_file_path(&storage_path, cluster), + get_collection_path(&storage_path, xtream_cluster_category_collection(cluster)), + ) + }); + let mut retained_before = Vec::new(); + for (database, categories) in &retained_paths { + retained_before.push((tokio::fs::read(database).await?, tokio::fs::read(categories).await?)); + } + + let mut live_only = vec![target_writer_group(XtreamCluster::Live, 1, 102)]; + xtream_write_playlist(&app_config, &target, &mut live_only, ClusterFlags::empty()).await?; + + for ((database, categories), (database_before, categories_before)) in retained_paths.iter().zip(retained_before) + { + assert_eq!(tokio::fs::read(database).await?, database_before); + assert_eq!(tokio::fs::read(categories).await?, categories_before); + } + Ok(()) + } + + #[tokio::test] + async fn target_force_empty_failures_restore_database_index_and_categories( + ) -> Result<(), Box> { + for failure in [ + TargetEmptyReplacementFailure::CategoryPersistence, + TargetEmptyReplacementFailure::BTreePersistence, + TargetEmptyReplacementFailure::Publication, + ] { + let directory = tempfile::tempdir()?; + let app_config = test_app_config(directory.path()); + let target = target_writer_config(); + let mut baseline = vec![ + target_writer_group(XtreamCluster::Live, 1, 101), + target_writer_group(XtreamCluster::Video, 2, 201), + target_writer_group(XtreamCluster::Series, 3, 301), + ]; + xtream_write_playlist(&app_config, &target, &mut baseline, ClusterFlags::empty()).await?; + let storage_path = { + let config = app_config.config.load(); + super::xtream_get_storage_path(&config, &target.name).expect("target Xtream storage") + }; + let database = super::xtream_get_file_path(&storage_path, XtreamCluster::Video); + let index = get_file_path_for_db_index(&database); + let categories = super::get_vod_cat_collection_path(&storage_path); + let before = [ + tokio::fs::read(&database).await?, + tokio::fs::read(&index).await?, + tokio::fs::read(&categories).await?, + ]; + let mut candidate = vec![ + target_writer_group(XtreamCluster::Live, 1, 102), + target_writer_group(XtreamCluster::Series, 3, 302), + ]; + + let error = xtream_write_playlist_with_injected_empty_replacement_failure( + &app_config, + &target, + &mut candidate, + ClusterFlags::Vod, + failure, + ) + .await + .expect_err("injected empty replacement must fail"); + + assert!(error.to_string().contains("target cluster failed")); + assert_eq!(tokio::fs::read(&database).await?, before[0]); + assert_eq!(tokio::fs::read(&index).await?, before[1]); + assert_eq!(tokio::fs::read(&categories).await?, before[2]); + let leaked = fs::read_dir(&storage_path)? + .filter_map(Result::ok) + .map(|entry| entry.file_name().to_string_lossy().into_owned()) + .filter(|name| name.contains("force-empty")) + .collect::>(); + assert!(leaked.is_empty(), "staging or backup files leaked after {failure:?}: {leaked:?}"); + } + Ok(()) + } + + #[tokio::test] + async fn target_force_empty_category_reader_waits_through_the_backup_window( + ) -> Result<(), Box> { + let directory = tempfile::tempdir()?; + let app_config = test_app_config(directory.path()); + let target = target_writer_config(); + let mut baseline = vec![ + target_writer_group(XtreamCluster::Live, 1, 101), + target_writer_group(XtreamCluster::Video, 2, 201), + target_writer_group(XtreamCluster::Series, 3, 301), + ]; + xtream_write_playlist(&app_config, &target, &mut baseline, ClusterFlags::empty()).await?; + let storage_path = { + let config = app_config.config.load(); + super::xtream_get_storage_path(&config, &target.name).expect("target Xtream storage") + }; + let category_path = super::get_vod_cat_collection_path(&storage_path); + let hook = TargetEmptyPublicationHook::new(); + let entered = Arc::clone(&hook.backup_window_entered); + let resume = Arc::clone(&hook.resume_publication); + let writer_app_config = Arc::clone(&app_config); + let writer_target = target.clone(); + let writer = tokio::spawn(async move { + let mut candidate = vec![ + target_writer_group(XtreamCluster::Live, 1, 102), + target_writer_group(XtreamCluster::Series, 3, 302), + ]; + xtream_write_playlist_with_mode( + &writer_app_config, + &writer_target, + &mut candidate, + ClusterFlags::Vod, + TargetEmptyReplacementMode::PauseDuringPublication(hook), + ) + .await + }); + + tokio::task::spawn_blocking(move || entered.wait()).await?; + let category_was_temporarily_backed_up = !category_path.exists(); + let category_lock_path = target_category_lock_path(&category_path); + let category_write_lock_was_held = app_config.file_locks.try_write_lock(&category_lock_path).await.is_err(); + + let category_read = xtream_get_playlist_categories(&app_config, &target.name, XtreamCluster::Video); + tokio::pin!(category_read); + let category_read_poll = futures::poll!(category_read.as_mut()); + let category_reader_state = match &category_read_poll { + std::task::Poll::Pending => "waiting", + std::task::Poll::Ready(None) => "missing", + std::task::Poll::Ready(Some(categories)) if categories.is_empty() => "new", + std::task::Poll::Ready(Some(_)) => "old-or-partial", + }; + + tokio::task::spawn_blocking(move || resume.wait()).await?; + writer.await??; + assert!(category_was_temporarily_backed_up, "failure hook must expose the internal backup window"); + assert!(category_write_lock_was_held, "writer must hold the target category lock during publication"); + assert_eq!(category_reader_state, "waiting", "the category reader must wait on the writer's category lock"); + let categories = category_read.await.expect("published VOD category catalog"); + assert!(categories.is_empty(), "reader must observe the complete force-empty category catalog"); + Ok(()) + } + + fn json_reader(content: &str) -> DynReader { let (mut writer, reader) = tokio::io::duplex(4096); + let content = content.as_bytes().to_vec(); tokio::spawn(async move { - writer.write_all(content.as_bytes()).await.expect("fixture should fit into duplex reader"); + writer.write_all(&content).await.expect("fixture should fit into duplex reader"); writer.shutdown().await.expect("fixture writer should shut down"); }); Box::pin(reader) @@ -2138,6 +3196,7 @@ mod tests { &app_config, &input, XtreamCluster::Live, + 0, json_reader(r#"[{"category_id":"1","category_name":"News"}]"#), json_reader(r#"[{"name":"Channel","stream_id":7,"category_id":"1","added":"0"}]"#), ) @@ -2153,6 +3212,7 @@ mod tests { &app_config, &input, XtreamCluster::Live, + 0, json_reader("[]"), json_reader("[]"), ) @@ -2187,6 +3247,7 @@ mod tests { &app_config, &input, cluster, + 0, json_reader(category), json_reader(stream), ) @@ -2197,6 +3258,7 @@ mod tests { &app_config, &input, XtreamCluster::Live, + 0, json_reader("[]"), json_reader("[]"), ) @@ -2211,6 +3273,723 @@ mod tests { Ok(()) } + fn disk_test_input(name: &str) -> ConfigInput { + ConfigInput { + name: name.intern(), + input_type: InputType::Xtream, + url: "http://provider.example".to_string(), + username: Some("user".to_string()), + password: Some("password".to_string()), + ..ConfigInput::default() + } + } + + fn cluster_fixture_readers( + cluster: XtreamCluster, + category_name: &str, + count: usize, + first_provider_id: u32, + ) -> (DynReader, DynReader) { + let category_id = match cluster { + XtreamCluster::Live => 1_u32, + XtreamCluster::Video => 2, + XtreamCluster::Series => 3, + }; + let categories = serde_json::json!([{ + "category_id": category_id, + "category_name": category_name, + }]) + .to_string(); + let streams = (0..count) + .map(|offset| { + let provider_id = + first_provider_id + u32::try_from(offset).expect("fixture item count should fit into u32"); + let name = format!("{category_name}-{provider_id}"); + match cluster { + XtreamCluster::Live => serde_json::json!({ + "name": name, + "stream_id": provider_id, + "category_id": category_id, + "added": "0", + }), + XtreamCluster::Video => serde_json::json!({ + "name": name, + "stream_id": provider_id, + "category_id": category_id, + "added": "0", + "container_extension": "mp4", + }), + XtreamCluster::Series => serde_json::json!({ + "name": name, + "series_id": provider_id, + "category_id": category_id, + "last_modified": "0", + }), + } + }) + .collect::>(); + let streams = serde_json::Value::Array(streams).to_string(); + (json_reader(&categories), json_reader(&streams)) + } + + fn live_fixture_stream(provider_id: u32, category_id: u32, name: &str) -> serde_json::Value { + serde_json::json!({ + "name": name, + "stream_id": provider_id, + "category_id": category_id, + "added": "0", + }) + } + + async fn publish_live_fixture_rows( + app_config: &Arc, + input: &ConfigInput, + threshold: u8, + categories: serde_json::Value, + streams: Vec, + ) -> XtreamClusterPublishOutcome { + persist_input_xtream_playlist_cluster_to_disk( + app_config, + input, + XtreamCluster::Live, + threshold, + json_reader(&categories.to_string()), + json_reader(&serde_json::Value::Array(streams).to_string()), + ) + .await + .expect("Live fixture refresh should complete") + } + + async fn publish_test_cluster( + app_config: &Arc, + input: &ConfigInput, + cluster: XtreamCluster, + threshold: u8, + category_name: &str, + count: usize, + first_provider_id: u32, + ) -> XtreamClusterPublishOutcome { + let (categories, streams) = cluster_fixture_readers(cluster, category_name, count, first_provider_id); + persist_input_xtream_playlist_cluster_to_disk(app_config, input, cluster, threshold, categories, streams) + .await + .expect("test cluster refresh should complete") + } + + fn active_cluster_count(storage_path: &Path, cluster: XtreamCluster) -> usize { + super::count_xtream_tree_entries(&super::xtream_get_file_path(storage_path, cluster)) + .expect("active cluster should be countable") + .expect("active cluster should exist") + } + + fn active_category_bytes(storage_path: &Path, cluster: XtreamCluster) -> Vec { + fs::read(get_collection_path(storage_path, xtream_cluster_category_collection(cluster))) + .expect("active categories should be readable") + } + + #[derive(Debug, Eq, PartialEq)] + struct ActiveClusterSnapshot { + database: Vec, + categories: Vec, + } + + fn active_cluster_snapshot(storage_path: &Path, cluster: XtreamCluster) -> ActiveClusterSnapshot { + let database_path = super::xtream_get_file_path(storage_path, cluster); + ActiveClusterSnapshot { + database: fs::read(&database_path).expect("active database should be readable"), + categories: active_category_bytes(storage_path, cluster), + } + } + + fn fail_detail_preservation_after_quality( + published_path: &Path, + staging_path: &Path, + ) -> Result { + preserve_details_with_injected_operation_failure( + published_path, + staging_path, + DetailPreservationOperation::Commit, + ) + } + + fn read_series_props(path: &Path, provider_id: u32) -> SeriesStreamProperties { + let mut query = BPlusTreeQuery::::try_new(path).expect("query open should succeed"); + let item = query.query_zero_copy(&provider_id).expect("query should succeed").expect("item should exist"); + match item.additional_properties { + Some(StreamProperties::Series(series)) => *series, + other => panic!("expected series stream properties, got {other:?}"), + } + } + + fn assert_no_refresh_artifacts(storage_path: &Path) { + let entries = fs::read_dir(storage_path) + .expect("input storage should be readable") + .collect::>>() + .expect("input storage entries should be readable"); + let refresh_artifacts = entries + .into_iter() + .map(|entry| entry.file_name().to_string_lossy().into_owned()) + .filter(|name| name.contains("refresh-")) + .collect::>(); + assert!(refresh_artifacts.is_empty(), "staging artifacts survived: {refresh_artifacts:?}"); + } + + #[test] + fn disk_quality_disabled_does_not_read_staging_or_baseline() { + let directory = tempdir().expect("temp directory"); + let paths = fixed_refresh_paths(directory.path(), 18); + + let decision = super::evaluate_staged_xtream_cluster_quality(&paths, XtreamCluster::Live, 0) + .expect("disabled quality evaluation should not access missing files"); + + assert_eq!(decision, UpdateQualityDecision::Disabled); + } + + #[tokio::test] + async fn disk_force_publishes_an_empty_cluster_and_cleans_staging() { + let directory = tempdir().expect("temporary directory"); + let app_config = test_app_config(directory.path()); + let input = disk_test_input("force-empty"); + assert_eq!( + publish_test_cluster(&app_config, &input, XtreamCluster::Video, 0, "old-vod", 3, 2_000).await, + XtreamClusterPublishOutcome::Published + ); + let storage_path = get_input_storage_path(&input.name, directory.path().to_string_lossy().as_ref()) + .await + .expect("force storage"); + let before = active_cluster_snapshot(&storage_path, XtreamCluster::Video); + + let result = persist_input_xtream_playlist_clusters_to_disk( + &app_config, + &input, + vec![XtreamClusterRefreshRequest { + cluster: XtreamCluster::Video, + quality: XtreamClusterQualityPolicy::Bypass { configured_threshold: 95 }, + categories: json_reader("[]"), + streams: json_reader("[]"), + }], + ) + .await; + + assert!(result.errors.is_empty()); + assert_eq!( + result.outcomes, + vec![XtreamClusterPublishOutcome::ForcePublished(ClusterForceUpdate { + cluster: XtreamCluster::Video, + candidate_count: 0, + configured_threshold: 95, + })] + ); + assert_eq!(active_cluster_count(&storage_path, XtreamCluster::Video), 0); + assert_ne!(active_cluster_snapshot(&storage_path, XtreamCluster::Video), before); + assert_eq!( + serde_json::from_slice::(&active_category_bytes(&storage_path, XtreamCluster::Video,)) + .expect("empty category JSON"), + serde_json::json!([]) + ); + assert_no_refresh_artifacts(&storage_path); + } + + #[tokio::test] + async fn disk_force_keeps_active_files_after_a_technical_staging_failure() { + let directory = tempdir().expect("temporary directory"); + let app_config = test_app_config(directory.path()); + let input = disk_test_input("force-technical-error"); + assert_eq!( + publish_test_cluster(&app_config, &input, XtreamCluster::Live, 0, "old-live", 3, 1_000).await, + XtreamClusterPublishOutcome::Published + ); + let storage_path = get_input_storage_path(&input.name, directory.path().to_string_lossy().as_ref()) + .await + .expect("force storage"); + let before = active_cluster_snapshot(&storage_path, XtreamCluster::Live); + + let result = persist_input_xtream_playlist_clusters_to_disk( + &app_config, + &input, + vec![XtreamClusterRefreshRequest { + cluster: XtreamCluster::Live, + quality: XtreamClusterQualityPolicy::Bypass { configured_threshold: 100 }, + categories: json_reader(r#"[{"category_id":"1","category_name":"new-live"}]"#), + streams: json_reader("{"), + }], + ) + .await; + + assert_eq!(result.errors.len(), 1); + assert!(result.outcomes.is_empty()); + assert!(result.force_updates.is_empty()); + assert_eq!(active_cluster_snapshot(&storage_path, XtreamCluster::Live), before); + assert_no_refresh_artifacts(&storage_path); + } + + #[tokio::test] + async fn pipeline_transparency_disk_quality_survives_post_evaluation_staging_failure() { + let directory = tempdir().expect("temporary directory"); + let app_config = test_app_config(directory.path()); + let input = disk_test_input("accepted-then-detail-error"); + assert_eq!( + publish_test_cluster(&app_config, &input, XtreamCluster::Live, 0, "old-live", 100, 1_000).await, + XtreamClusterPublishOutcome::Published + ); + let storage_path = get_input_storage_path(&input.name, directory.path().to_string_lossy().as_ref()) + .await + .expect("storage path"); + let before = active_cluster_snapshot(&storage_path, XtreamCluster::Live); + let (categories, streams) = cluster_fixture_readers(XtreamCluster::Live, "new-live", 95, 10_000); + + let result = persist_input_xtream_playlist_clusters_to_disk_with_operations( + &app_config, + &input, + vec![XtreamClusterRefreshRequest { + cluster: XtreamCluster::Live, + quality: XtreamClusterQualityPolicy::Enforce { threshold: 90 }, + categories, + streams, + }], + XtreamClusterStageOperations { preserve_details: fail_detail_preservation_after_quality }, + ) + .await; + + assert_eq!( + result.quality_acceptances, + vec![ClusterUpdateAcceptance { + cluster: XtreamCluster::Live, + current_count: Some(100), + candidate_count: 95, + threshold: 90, + quality: Some(95), + }] + ); + assert!(result.outcomes.is_empty()); + assert_eq!(result.errors.len(), 1); + assert_eq!(result.failed_cluster, Some(XtreamCluster::Live)); + assert_eq!(active_cluster_snapshot(&storage_path, XtreamCluster::Live), before); + assert_no_refresh_artifacts(&storage_path); + } + + #[tokio::test] + async fn pipeline_transparency_disk_batch_keeps_prior_publish_and_all_evaluated_quality_on_later_error() { + let directory = tempdir().expect("temporary directory"); + let app_config = test_app_config(directory.path()); + let input = disk_test_input("later-cluster-publish-error"); + assert_eq!( + publish_test_cluster(&app_config, &input, XtreamCluster::Live, 0, "old-live", 100, 1_000).await, + XtreamClusterPublishOutcome::Published + ); + assert_eq!( + publish_test_cluster(&app_config, &input, XtreamCluster::Video, 0, "old-vod", 100, 2_000).await, + XtreamClusterPublishOutcome::Published + ); + let storage_path = get_input_storage_path(&input.name, directory.path().to_string_lossy().as_ref()) + .await + .expect("storage path"); + let blocked_category_path = + get_collection_path(&storage_path, xtream_cluster_category_collection(XtreamCluster::Video)); + fs::remove_file(&blocked_category_path).expect("replace published VOD categories with a directory"); + fs::create_dir(&blocked_category_path).expect("create category publication blocker"); + let (live_categories, live_streams) = cluster_fixture_readers(XtreamCluster::Live, "new-live", 90, 10_000); + let (vod_categories, vod_streams) = cluster_fixture_readers(XtreamCluster::Video, "new-vod", 90, 20_000); + + let result = persist_input_xtream_playlist_clusters_to_disk( + &app_config, + &input, + vec![ + XtreamClusterRefreshRequest { + cluster: XtreamCluster::Live, + quality: XtreamClusterQualityPolicy::Enforce { threshold: 90 }, + categories: live_categories, + streams: live_streams, + }, + XtreamClusterRefreshRequest { + cluster: XtreamCluster::Video, + quality: XtreamClusterQualityPolicy::Enforce { threshold: 90 }, + categories: vod_categories, + streams: vod_streams, + }, + ], + ) + .await; + + assert_eq!(result.quality_acceptances.len(), 2); + assert_eq!(result.quality_acceptances[0].cluster, XtreamCluster::Live); + assert_eq!(result.quality_acceptances[1].cluster, XtreamCluster::Video); + assert_eq!(result.outcomes, vec![XtreamClusterPublishOutcome::QualityAccepted(result.quality_acceptances[0])]); + assert_eq!(result.errors.len(), 1); + assert_eq!(result.failed_cluster, Some(XtreamCluster::Video)); + assert_eq!(active_cluster_count(&storage_path, XtreamCluster::Live), 90); + assert_no_refresh_artifacts(&storage_path); + } + + #[tokio::test] + async fn disk_quality_bootstrap_publishes_nonempty_candidate_and_rejects_empty_without_baseline() { + let populated_directory = tempdir().expect("populated bootstrap directory"); + let populated_config = test_app_config(populated_directory.path()); + let populated_input = disk_test_input("bootstrap-populated"); + + let populated_outcome = publish_test_cluster( + &populated_config, + &populated_input, + XtreamCluster::Live, + 90, + "bootstrap-live", + 3, + 1_000, + ) + .await; + let populated_storage = + get_input_storage_path(&populated_input.name, populated_directory.path().to_string_lossy().as_ref()) + .await + .expect("populated bootstrap storage"); + + assert_eq!( + populated_outcome, + XtreamClusterPublishOutcome::QualityAccepted(ClusterUpdateAcceptance { + cluster: XtreamCluster::Live, + current_count: None, + candidate_count: 3, + threshold: 90, + quality: None, + }) + ); + assert_eq!(active_cluster_count(&populated_storage, XtreamCluster::Live), 3); + assert!(String::from_utf8_lossy(&active_category_bytes(&populated_storage, XtreamCluster::Live)) + .contains("bootstrap-live")); + assert_no_refresh_artifacts(&populated_storage); + + let empty_directory = tempdir().expect("empty bootstrap directory"); + let empty_config = test_app_config(empty_directory.path()); + let empty_input = disk_test_input("bootstrap-empty"); + let empty_outcome = + publish_test_cluster(&empty_config, &empty_input, XtreamCluster::Video, 90, "empty-vod", 0, 2_000).await; + let empty_storage = + get_input_storage_path(&empty_input.name, empty_directory.path().to_string_lossy().as_ref()) + .await + .expect("empty bootstrap storage"); + + assert_eq!( + empty_outcome, + XtreamClusterPublishOutcome::RetainedPrevious(ClusterUpdateRejection { + cluster: XtreamCluster::Video, + current_count: 0, + candidate_count: 0, + threshold: 90, + quality: 0, + }) + ); + assert!(!super::xtream_get_file_path(&empty_storage, XtreamCluster::Video).exists()); + assert!(!get_collection_path(&empty_storage, xtream_cluster_category_collection(XtreamCluster::Video)).exists()); + assert_no_refresh_artifacts(&empty_storage); + } + + #[tokio::test] + async fn disk_quality_enforces_exact_90_percent_boundaries_and_retains_rejected_files() { + for (name, candidate_count, accepted) in [ + ("lower-boundary", 90, true), + ("below-lower-boundary", 89, false), + ("upper-boundary", 110, true), + ("above-upper-boundary", 111, false), + ] { + let directory = tempdir().expect("boundary directory"); + let app_config = test_app_config(directory.path()); + let input = disk_test_input(name); + assert_eq!( + publish_test_cluster(&app_config, &input, XtreamCluster::Live, 0, "old-live", 100, 10_000,).await, + XtreamClusterPublishOutcome::Published, + "baseline publish failed for {name}" + ); + let storage_path = get_input_storage_path(&input.name, directory.path().to_string_lossy().as_ref()) + .await + .expect("boundary storage"); + let before = active_cluster_snapshot(&storage_path, XtreamCluster::Live); + + let outcome = + publish_test_cluster(&app_config, &input, XtreamCluster::Live, 90, "new-live", candidate_count, 20_000) + .await; + + if accepted { + assert_eq!( + outcome, + XtreamClusterPublishOutcome::QualityAccepted(ClusterUpdateAcceptance { + cluster: XtreamCluster::Live, + current_count: Some(100), + candidate_count, + threshold: 90, + quality: Some(90), + }), + "case: {name}" + ); + assert_eq!(active_cluster_count(&storage_path, XtreamCluster::Live), candidate_count); + assert!(String::from_utf8_lossy(&active_category_bytes(&storage_path, XtreamCluster::Live)) + .contains("new-live")); + } else { + assert_eq!( + outcome, + XtreamClusterPublishOutcome::RetainedPrevious(ClusterUpdateRejection { + cluster: XtreamCluster::Live, + current_count: 100, + candidate_count, + threshold: 90, + quality: 89, + }), + "case: {name}" + ); + assert_eq!(active_cluster_snapshot(&storage_path, XtreamCluster::Live), before, "case: {name}"); + } + assert_no_refresh_artifacts(&storage_path); + } + } + + #[tokio::test] + async fn disk_quality_rejects_duplicate_rows_that_represent_only_half_the_provider_ids() { + let directory = tempdir().expect("duplicate rejection directory"); + let app_config = test_app_config(directory.path()); + let input = disk_test_input("duplicate-rejection"); + let first_provider_id = 1_000_u32; + assert_eq!( + publish_test_cluster(&app_config, &input, XtreamCluster::Live, 0, "old-live", 100, first_provider_id,) + .await, + XtreamClusterPublishOutcome::Published + ); + let storage_path = get_input_storage_path(&input.name, directory.path().to_string_lossy().as_ref()) + .await + .expect("duplicate rejection storage"); + let before = active_cluster_snapshot(&storage_path, XtreamCluster::Live); + let streams = (0..50_u32) + .flat_map(|offset| { + let provider_id = first_provider_id + offset; + [ + live_fixture_stream(provider_id, 1, &format!("candidate-{provider_id}")), + live_fixture_stream(provider_id, 1, &format!("duplicate-{provider_id}")), + ] + }) + .collect(); + + let outcome = publish_live_fixture_rows( + &app_config, + &input, + 100, + serde_json::json!([{"category_id": 1, "category_name": "candidate-live"}]), + streams, + ) + .await; + + assert_eq!( + outcome, + XtreamClusterPublishOutcome::RetainedPrevious(ClusterUpdateRejection { + cluster: XtreamCluster::Live, + current_count: 100, + candidate_count: 50, + threshold: 100, + quality: 50, + }) + ); + assert_eq!(active_cluster_snapshot(&storage_path, XtreamCluster::Live), before); + assert_no_refresh_artifacts(&storage_path); + } + + #[tokio::test] + async fn pipeline_transparency_disk_quality_preserves_accepted_publish_facts() { + let directory = tempdir().expect("duplicate acceptance directory"); + let app_config = test_app_config(directory.path()); + let input = disk_test_input("duplicate-acceptance"); + let first_provider_id = 1_000_u32; + let winning_provider_id = first_provider_id + 42; + assert_eq!( + publish_test_cluster(&app_config, &input, XtreamCluster::Live, 0, "old-live", 100, first_provider_id,) + .await, + XtreamClusterPublishOutcome::Published + ); + let storage_path = get_input_storage_path(&input.name, directory.path().to_string_lossy().as_ref()) + .await + .expect("duplicate acceptance storage"); + let mut streams = (0..100_u32) + .map(|offset| { + let provider_id = first_provider_id + offset; + live_fixture_stream(provider_id, 1, &format!("candidate-{provider_id}")) + }) + .collect::>(); + streams.push(live_fixture_stream(winning_provider_id, 2, "winning-duplicate")); + + let outcome = publish_live_fixture_rows( + &app_config, + &input, + 100, + serde_json::json!([ + {"category_id": 1, "category_name": "candidate-live"}, + {"category_id": 2, "category_name": "winning-category"} + ]), + streams, + ) + .await; + + assert_eq!( + outcome, + XtreamClusterPublishOutcome::QualityAccepted(ClusterUpdateAcceptance { + cluster: XtreamCluster::Live, + current_count: Some(100), + candidate_count: 100, + threshold: 100, + quality: Some(100), + }) + ); + assert_eq!(active_cluster_count(&storage_path, XtreamCluster::Live), 100); + let active_path = super::xtream_get_file_path(&storage_path, XtreamCluster::Live); + let mut query = BPlusTreeQuery::::try_new(&active_path) + .expect("accepted Live cluster should open"); + let winner = query + .query_zero_copy(&winning_provider_id) + .expect("winner lookup should succeed") + .expect("winner should exist"); + assert_eq!(winner.name.as_ref(), "winning-duplicate"); + assert_eq!(winner.category_id, 2); + assert!(String::from_utf8_lossy(&active_category_bytes(&storage_path, XtreamCluster::Live)) + .contains("winning-category")); + assert_no_refresh_artifacts(&storage_path); + } + + #[tokio::test] + async fn disk_series_quality_compares_catalog_rows_and_ignores_embedded_episode_details() { + let directory = tempdir().expect("series population directory"); + let app_config = test_app_config(directory.path()); + let input = disk_test_input("series-logical-population"); + let provider_id = 30_000; + assert_eq!( + publish_test_cluster(&app_config, &input, XtreamCluster::Series, 0, "old-series", 1, provider_id,).await, + XtreamClusterPublishOutcome::Published + ); + let storage_path = get_input_storage_path(&input.name, directory.path().to_string_lossy().as_ref()) + .await + .expect("series population storage"); + let episodes = (0..200_u32) + .map(|id| SeriesStreamDetailEpisodeProperties { id, ..SeriesStreamDetailEpisodeProperties::default() }) + .collect(); + let enriched = SeriesStreamProperties { + series_id: provider_id, + details: Some(SeriesStreamDetailProperties::new(None, Vec::new(), Some(episodes))), + ..SeriesStreamProperties::default() + }; + persists_input_series_info( + &app_config, + &storage_path, + XtreamCluster::Series, + &input.name, + provider_id, + &enriched, + ) + .await + .expect("series enrichment should persist"); + let active_path = super::xtream_get_file_path(&storage_path, XtreamCluster::Series); + assert_eq!( + read_series_props(&active_path, provider_id) + .details + .and_then(|details| details.episodes) + .map(|episodes| episodes.len()), + Some(200) + ); + + let equal_catalog = + publish_test_cluster(&app_config, &input, XtreamCluster::Series, 100, "equal-series", 1, provider_id).await; + + assert_eq!( + equal_catalog, + XtreamClusterPublishOutcome::QualityAccepted(ClusterUpdateAcceptance { + cluster: XtreamCluster::Series, + current_count: Some(1), + candidate_count: 1, + threshold: 100, + quality: Some(100), + }) + ); + assert_eq!(active_cluster_count(&storage_path, XtreamCluster::Series), 1); + let accepted_snapshot = active_cluster_snapshot(&storage_path, XtreamCluster::Series); + + let different_catalog = + publish_test_cluster(&app_config, &input, XtreamCluster::Series, 100, "different-series", 2, 40_000).await; + + assert_eq!( + different_catalog, + XtreamClusterPublishOutcome::RetainedPrevious(ClusterUpdateRejection { + cluster: XtreamCluster::Series, + current_count: 1, + candidate_count: 2, + threshold: 100, + quality: 0, + }) + ); + assert_eq!(active_cluster_snapshot(&storage_path, XtreamCluster::Series), accepted_snapshot); + assert_no_refresh_artifacts(&storage_path); + } + + #[tokio::test] + async fn disk_quality_publishes_accepted_clusters_and_retains_rejected_cluster_independently() { + let directory = tempdir().expect("mixed cluster directory"); + let app_config = test_app_config(directory.path()); + let input = disk_test_input("mixed-clusters"); + + for (cluster, category_name, first_provider_id) in [ + (XtreamCluster::Live, "old-live", 10_000), + (XtreamCluster::Video, "old-vod", 20_000), + (XtreamCluster::Series, "old-series", 30_000), + ] { + assert_eq!( + publish_test_cluster(&app_config, &input, cluster, 0, category_name, 100, first_provider_id).await, + XtreamClusterPublishOutcome::Published + ); + } + let storage_path = get_input_storage_path(&input.name, directory.path().to_string_lossy().as_ref()) + .await + .expect("mixed cluster storage"); + let previous_vod = active_cluster_snapshot(&storage_path, XtreamCluster::Video); + + let live_outcome = + publish_test_cluster(&app_config, &input, XtreamCluster::Live, 90, "new-live", 90, 40_000).await; + let vod_outcome = + publish_test_cluster(&app_config, &input, XtreamCluster::Video, 90, "new-vod", 89, 50_000).await; + let series_outcome = + publish_test_cluster(&app_config, &input, XtreamCluster::Series, 90, "new-series", 110, 60_000).await; + + assert_eq!( + live_outcome, + XtreamClusterPublishOutcome::QualityAccepted(ClusterUpdateAcceptance { + cluster: XtreamCluster::Live, + current_count: Some(100), + candidate_count: 90, + threshold: 90, + quality: Some(90), + }) + ); + assert_eq!( + vod_outcome, + XtreamClusterPublishOutcome::RetainedPrevious(ClusterUpdateRejection { + cluster: XtreamCluster::Video, + current_count: 100, + candidate_count: 89, + threshold: 90, + quality: 89, + }) + ); + assert_eq!( + series_outcome, + XtreamClusterPublishOutcome::QualityAccepted(ClusterUpdateAcceptance { + cluster: XtreamCluster::Series, + current_count: Some(100), + candidate_count: 110, + threshold: 90, + quality: Some(90), + }) + ); + assert_eq!(active_cluster_count(&storage_path, XtreamCluster::Live), 90); + assert_eq!(active_cluster_count(&storage_path, XtreamCluster::Video), 100); + assert_eq!(active_cluster_count(&storage_path, XtreamCluster::Series), 110); + assert!( + String::from_utf8_lossy(&active_category_bytes(&storage_path, XtreamCluster::Live)).contains("new-live") + ); + assert_eq!(active_cluster_snapshot(&storage_path, XtreamCluster::Video), previous_vod); + assert!(String::from_utf8_lossy(&active_category_bytes(&storage_path, XtreamCluster::Series)) + .contains("new-series")); + assert_no_refresh_artifacts(&storage_path); + } + fn wait_for_child(mut child: Child, timeout: Duration) -> io::Result { let deadline = Instant::now() + timeout; loop { @@ -2460,6 +4239,158 @@ mod tests { tree.commit().expect("tree commit should succeed"); } + fn make_input_group( + cluster: XtreamCluster, + category_id: u32, + category_name: &str, + provider_id: u32, + ) -> PlaylistGroup { + let stream_id = provider_id.to_string().intern(); + let category_name = category_name.intern(); + PlaylistGroup { + id: category_id, + title: Arc::clone(&category_name), + channels: vec![PlaylistItem { + header: PlaylistItemHeader { + id: Arc::clone(&stream_id), + input_stream_id: stream_id, + name: format!("stream-{provider_id}").intern(), + title: format!("stream-{provider_id}").intern(), + group: category_name, + url: format!("http://provider.example/{cluster}/{provider_id}").intern(), + item_type: PlaylistItemType::from(cluster), + xtream_cluster: cluster, + category_id, + input_name: "provider-a".intern(), + ..PlaylistItemHeader::default() + }, + }], + xtream_cluster: cluster, + } + } + + #[tokio::test] + async fn input_cluster_count_returns_none_for_missing_baseline() { + let directory = tempdir().expect("temp directory"); + let app_config = test_app_config(directory.path()); + let input = + ConfigInput { name: "provider-a".intern(), input_type: InputType::Xtream, ..ConfigInput::default() }; + + assert_eq!( + count_input_xtream_cluster(&app_config, &input, XtreamCluster::Live) + .await + .expect("missing baseline should be readable"), + None + ); + } + + #[tokio::test] + async fn input_cluster_count_reads_the_canonical_active_raw_cluster() { + let directory = tempdir().expect("temp directory"); + let app_config = test_app_config(directory.path()); + let input = + ConfigInput { name: "provider-a".intern(), input_type: InputType::Xtream, ..ConfigInput::default() }; + + let storage_path = get_input_storage_path(&input.name, directory.path().to_string_lossy().as_ref()) + .await + .expect("canonical input storage"); + write_single_item( + &super::xtream_get_file_path(&storage_path, XtreamCluster::Live), + &make_live_item(700, None, None, None, None, 0), + ); + + assert_eq!( + count_input_xtream_cluster(&app_config, &input, XtreamCluster::Live) + .await + .expect("active baseline should be readable"), + Some(1) + ); + assert_eq!( + count_input_xtream_cluster(&app_config, &input, XtreamCluster::Video) + .await + .expect("other cluster should remain absent"), + None + ); + } + + #[tokio::test] + async fn in_memory_fallback_keeps_colliding_category_ids_separate_by_cluster() { + let directory = tempdir().expect("temp directory"); + let app_config = test_app_config(directory.path()); + let storage_path = get_input_storage_path("provider-a", directory.path().to_string_lossy().as_ref()) + .await + .expect("canonical input storage"); + + let (_, seed_error) = persist_input_xtream_playlist( + &app_config, + &storage_path, + vec![ + make_input_group(XtreamCluster::Live, 1, "Previous Live", 100), + make_input_group(XtreamCluster::Video, 2, "Previous VOD", 200), + make_input_group(XtreamCluster::Series, 3, "Previous Series", 300), + ], + ) + .await; + assert!(seed_error.is_none(), "failed to seed persisted VOD: {seed_error:?}"); + let previous_vod = load_input_xtream_playlist(&app_config, &storage_path, &[XtreamCluster::Video]) + .await + .expect("seeded VOD should load"); + let category_path = + get_collection_path(&storage_path, xtream_cluster_category_collection(XtreamCluster::Video)); + let previous_categories = fs::read(&category_path).expect("persisted VOD categories"); + + let (merged, persist_error) = persist_input_xtream_playlist( + &app_config, + &storage_path, + vec![ + make_input_group(XtreamCluster::Live, 1, "New Live", 101), + make_input_group(XtreamCluster::Series, 2, "New Series", 301), + ], + ) + .await; + + assert!(persist_error.is_none(), "failed to persist accepted clusters: {persist_error:?}"); + assert_eq!(fs::read(&category_path).expect("retained VOD categories"), previous_categories); + assert!(merged + .iter() + .all(|group| { group.channels.iter().all(|item| item.header.xtream_cluster == group.xtream_cluster) })); + + let retained_vod = merged + .iter() + .find(|group| group.xtream_cluster == XtreamCluster::Video && group.id == 2) + .expect("persisted VOD fallback"); + assert_eq!(retained_vod.title.as_ref(), "Previous VOD"); + assert_eq!(retained_vod.channels.len(), 1); + assert_eq!(retained_vod.channels[0].header.id.as_ref(), "200"); + + let accepted_live = merged + .iter() + .find(|group| group.xtream_cluster == XtreamCluster::Live && group.id == 1) + .expect("accepted Live candidate"); + assert_eq!(accepted_live.title.as_ref(), "New Live"); + assert_eq!(accepted_live.channels[0].header.id.as_ref(), "101"); + + let accepted_series = merged + .iter() + .find(|group| group.xtream_cluster == XtreamCluster::Series && group.id == 2) + .expect("accepted Series candidate sharing VOD category id"); + assert_eq!(accepted_series.title.as_ref(), "New Series"); + assert_eq!(accepted_series.channels[0].header.id.as_ref(), "301"); + + let loaded = load_input_xtream_playlist(&app_config, &storage_path, &[XtreamCluster::Video]) + .await + .expect("retained VOD should load"); + assert_eq!(loaded.len(), previous_vod.len()); + assert_eq!(loaded[0].id, previous_vod[0].id); + assert_eq!(loaded[0].title, previous_vod[0].title); + assert_eq!(loaded[0].xtream_cluster, previous_vod[0].xtream_cluster); + assert_eq!(loaded[0].channels.len(), previous_vod[0].channels.len()); + assert_eq!(loaded[0].channels[0].header.id, previous_vod[0].channels[0].header.id); + assert_eq!(loaded[0].channels[0].header.name, previous_vod[0].channels[0].header.name); + assert_eq!(loaded[0].channels[0].header.category_id, previous_vod[0].channels[0].header.category_id); + assert_eq!(loaded[0].channels[0].header.xtream_cluster, previous_vod[0].channels[0].header.xtream_cluster); + } + fn read_live_props(path: &Path, provider_id: u32) -> LiveStreamProperties { let mut query = BPlusTreeQuery::::try_new(path).expect("query open should succeed"); let item = query.query_zero_copy(&provider_id).expect("query should succeed").expect("item should exist"); @@ -2869,6 +4800,7 @@ mod tests { &app_config, &input, XtreamCluster::Live, + 0, json_reader(r#"[{"category_id":"1","category_name":"Sports"}]"#), json_reader(r#"[{"name":"Live","stream_id":700,"category_id":"1","added":"0"}]"#), ) diff --git a/backend/session/src/event_manager.rs b/backend/session/src/event_manager.rs index 9fe19b43c..ff8bb9ba9 100644 --- a/backend/session/src/event_manager.rs +++ b/backend/session/src/event_manager.rs @@ -7,12 +7,14 @@ //! [`StreamMeterRegistry`] now; `EventManager` owns one so the composition //! root still constructs a single handle. +mod playlist_update; + use crate::{ meter_registry::{MeterQos, StreamMeterRegistry}, StreamMeterHandle, }; use log::trace; -use shared::model::{EventKind, EventKindMask, EventMessage, EventSink, StreamMeterEntry}; +use shared::model::{EventKind, EventKindMask, EventMessage, EventSink, PlaylistUpdateProgressEvent, StreamMeterEntry}; use std::{ collections::{HashMap, VecDeque}, sync::{ @@ -146,6 +148,12 @@ fn lock(mutex: &Mutex) -> std::sync::MutexGuard<'_, T> { mutex.lock().unwrap_or_else(std::sync::PoisonError::into_inner) } +#[derive(Default)] +struct LatchedEvents { + messages: HashMap, + playlist_updates: playlist_update::PlaylistUpdates, +} + pub struct EventManager { channel_tx: tokio::sync::broadcast::Sender, meters: StreamMeterRegistry, @@ -156,7 +164,7 @@ pub struct EventManager { last_nudge: Mutex>, /// The newest message of each latched kind, for subscribers that connect /// after it was published. - latched: Mutex>, + latched: Mutex, /// The last [`RECENT_EVENT_CAPACITY`] events, newest last. recent: Mutex>, started: Instant, @@ -191,7 +199,7 @@ impl EventManager { meters: StreamMeterRegistry::new(), stats: Arc::new(EventBusStats::default()), last_nudge: Mutex::new(HashMap::new()), - latched: Mutex::new(HashMap::new()), + latched: Mutex::new(LatchedEvents::default()), recent: Mutex::new(VecDeque::with_capacity(RECENT_EVENT_CAPACITY)), started: Instant::now(), } @@ -235,8 +243,12 @@ impl EventManager { return EmitOutcome::Coalesced; } - if kind.is_latched() { - lock(&self.latched).insert(kind, event.clone()); + if kind.is_latched() || matches!(kind, EventKind::PlaylistUpdateProgress | EventKind::PlaylistUpdate) { + let mut latched = lock(&self.latched); + if kind.is_latched() { + latched.messages.insert(kind, event.clone()); + } + latched.playlist_updates.record(&event); } let outcome = match self.channel_tx.send(event) { @@ -267,7 +279,14 @@ impl EventManager { /// resync-on-lag path re-requested a status snapshot for the same /// reason. Both are answered by handing over what the bus already has. #[must_use] - pub fn snapshot(&self) -> Vec { lock(&self.latched).values().cloned().collect() } + pub fn snapshot(&self) -> Vec { lock(&self.latched).messages.values().cloned().collect() } + + /// Current correlated input facts, removed at their run's completion. + /// This read projection is volatile and never writes input status/history. + #[must_use] + pub fn playlist_update_snapshot(&self) -> Vec { + lock(&self.latched).playlist_updates.snapshot() + } /// The last [`RECENT_EVENT_CAPACITY`] events, oldest first. /// @@ -615,10 +634,8 @@ mod coalescing_tests { let _rx = manager.get_event_channel(); for _ in 0..5 { - let outcome = manager.send_event(EventMessage::PlaylistUpdateProgress(PlaylistUpdateProgressEvent { - target: "t".to_string(), - message: "m".to_string(), - })); + let outcome = + manager.send_event(EventMessage::PlaylistUpdateProgress(PlaylistUpdateProgressEvent::global("t", "m"))); assert!(outcome.is_delivered(), "dropping a progress tick would lose the message it carried"); } assert_eq!(manager.stats().coalesced(), 0); diff --git a/backend/session/src/event_manager/playlist_update.rs b/backend/session/src/event_manager/playlist_update.rs new file mode 100644 index 000000000..a293d5ee3 --- /dev/null +++ b/backend/session/src/event_manager/playlist_update.rs @@ -0,0 +1,192 @@ +use shared::model::{ + EventMessage, PlaylistUpdateProgressEvent, PlaylistUpdateRunId, PlaylistUpdateRunOrder, PlaylistUpdateState, +}; +use std::collections::{hash_map::Entry, HashMap}; + +struct InputUpdate { + run_id: PlaylistUpdateRunId, + order: PlaylistUpdateRunOrder, + progress: Option, +} + +/// Latest active facts per input, not a log. Completion drops the payload; +/// only its identity/order remain to reject late events from that execution. +#[derive(Default)] +pub(super) struct PlaylistUpdates { + inputs: HashMap, +} + +impl PlaylistUpdates { + pub(super) fn record(&mut self, event: &EventMessage) { + match event { + EventMessage::PlaylistUpdateProgress(progress) => { + let (Some(run_id), Some(order), Some(input_id)) = + (&progress.run_id, progress.execution_order, progress.input_id) + else { + return; + }; + match self.inputs.entry(input_id) { + Entry::Vacant(entry) => { + entry.insert(InputUpdate { run_id: run_id.clone(), order, progress: Some(progress.clone()) }); + } + Entry::Occupied(mut entry) => { + let current = entry.get_mut(); + if order > current.order { + *current = InputUpdate { run_id: run_id.clone(), order, progress: Some(progress.clone()) }; + } else if order == current.order && *run_id == current.run_id { + if let Some(previous) = current.progress.as_mut() { + let mut next = progress.clone(); + // A later text-only/reuse event is not a new acquisition + // and must not erase proven facts or a stronger result. + next.state = match (previous.state, next.state) { + (Some(PlaylistUpdateState::Failure), _) + | (_, Some(PlaylistUpdateState::Failure)) => Some(PlaylistUpdateState::Failure), + (Some(PlaylistUpdateState::Partial), _) + | (_, Some(PlaylistUpdateState::Partial)) => Some(PlaylistUpdateState::Partial), + (Some(PlaylistUpdateState::Success), _) + | (_, Some(PlaylistUpdateState::Success)) => Some(PlaylistUpdateState::Success), + (None, None) => None, + }; + next.input_telemetry = + next.input_telemetry.or_else(|| previous.input_telemetry.clone()); + next.library_scan_result = + next.library_scan_result.or_else(|| previous.library_scan_result.clone()); + *previous = next; + } + } + } + } + } + EventMessage::PlaylistUpdate(completed) => { + if let (Some(run_id), Some(order)) = (&completed.run_id, completed.execution_order) { + for input in self.inputs.values_mut() { + if input.run_id == *run_id && input.order == order { + input.progress = None; + } + } + } + } + _ => {} + } + } + + pub(super) fn snapshot(&self) -> Vec { + let mut progress: Vec<_> = self.inputs.values().filter_map(|input| input.progress.clone()).collect(); + progress.sort_by_key(|event| (event.execution_order, event.input_id)); + progress + } +} + +#[cfg(test)] +mod tests { + use crate::EventManager; + use shared::model::{ + EventMessage, InputRefreshPolicy, LibraryScanResult, PlaylistUpdateDataSource, PlaylistUpdateInputTelemetry, + PlaylistUpdateProgressEvent, PlaylistUpdateState, PlaylistUpdateSummary, + }; + + fn progress(run: &str, order: u64, input: u16) -> PlaylistUpdateProgressEvent { + PlaylistUpdateProgressEvent::for_run_input(run.into(), order.into(), input, "input", "working") + } + + fn completed(run: &str, order: u64) -> EventMessage { + EventMessage::PlaylistUpdate(PlaylistUpdateSummary::for_run( + run.into(), + order.into(), + PlaylistUpdateState::Success, + )) + } + + #[test] + fn playlist_update_status_snapshot_survives_late_subscription_until_correlated_run_completion() { + let manager = EventManager::new(); + let mut own = progress("r1", 1, 7); + manager.send_event(EventMessage::PlaylistUpdateProgress(own.clone())); + assert_eq!(manager.playlist_update_snapshot(), vec![own.clone()]); + own.state = Some(PlaylistUpdateState::Partial); + manager.send_event(EventMessage::PlaylistUpdateProgress(own.clone())); + assert_eq!( + manager.playlist_update_snapshot(), + vec![own.clone()], + "input completion does not finish target rebuild" + ); + manager.send_event(completed("unrelated", 1)); + manager.send_event(completed("r1", 2)); + assert_eq!(manager.playlist_update_snapshot(), vec![own]); + manager.send_event(completed("r1", 1)); + assert!(manager.playlist_update_snapshot().is_empty()); + manager.send_event(EventMessage::PlaylistUpdateProgress(progress("r1", 1, 7))); + assert!(manager.playlist_update_snapshot().is_empty(), "late progress cannot resurrect a finished input/run"); + let next = progress("r2", 2, 7); + manager.send_event(EventMessage::PlaylistUpdateProgress(next.clone())); + assert_eq!(manager.playlist_update_snapshot(), vec![next]); + assert!(EventManager::new().playlist_update_snapshot().is_empty(), "restart retains no transient update"); + } + + #[test] + fn playlist_update_status_snapshot_is_input_scoped_ordered_and_not_a_progress_log() { + let manager = EventManager::new(); + for input in [9, 7, 8] { + for _ in 0..300 { + manager.send_event(EventMessage::PlaylistUpdateProgress(progress("r1", 1, input))); + } + } + assert_eq!( + manager.playlist_update_snapshot().iter().map(|event| event.input_id).collect::>(), + vec![Some(7), Some(8), Some(9)] + ); + manager.send_event(EventMessage::PlaylistUpdateProgress(progress("r2", 2, 7))); + manager.send_event(EventMessage::PlaylistUpdateProgress(progress("r0", 0, 7))); + manager.send_event(completed("r1", 1)); + assert_eq!(manager.playlist_update_snapshot(), vec![progress("r2", 2, 7)]); + } + + #[test] + fn playlist_update_status_snapshot_preserves_own_facts_on_reuse_and_clears_them_for_a_new_run() { + let manager = EventManager::new(); + let mut own = progress("r1", 1, 7); + own.state = Some(PlaylistUpdateState::Partial); + own.input_telemetry = Some(PlaylistUpdateInputTelemetry { + refresh_policy: InputRefreshPolicy::REFRESH, + source: Some(PlaylistUpdateDataSource::Cache), + clusters: Vec::new(), + }); + own.library_scan_result = Some(LibraryScanResult { + files_scanned: 9, + groups_scanned: 3, + files_added: 1, + files_updated: 2, + files_removed: 3, + errors: 4, + }); + manager.send_event(EventMessage::PlaylistUpdateProgress(own.clone())); + let mut reused = progress("r1", 1, 7); + reused.state = Some(PlaylistUpdateState::Success); + manager.send_event(EventMessage::PlaylistUpdateProgress(reused)); + assert_eq!(manager.playlist_update_snapshot(), vec![own.clone()]); + let mut failure = progress("r1", 1, 7); + failure.state = Some(PlaylistUpdateState::Failure); + manager.send_event(EventMessage::PlaylistUpdateProgress(failure)); + own.state = Some(PlaylistUpdateState::Failure); + assert_eq!(manager.playlist_update_snapshot(), vec![own]); + manager.send_event(EventMessage::PlaylistUpdateProgress(progress("r2", 2, 7))); + assert_eq!(manager.playlist_update_snapshot(), vec![progress("r2", 2, 7)]); + } + + #[test] + fn playlist_update_status_snapshot_ignores_global_legacy_and_ambiguous_events() { + let manager = EventManager::new(); + let valid = progress("r1", 1, 7); + let mut no_order = valid.clone(); + no_order.execution_order = None; + let mut no_run = valid.clone(); + no_run.run_id = None; + for event in [PlaylistUpdateProgressEvent::global("target", "working"), no_order, no_run] { + manager.send_event(EventMessage::PlaylistUpdateProgress(event)); + } + assert!(manager.playlist_update_snapshot().is_empty()); + manager.send_event(EventMessage::PlaylistUpdateProgress(valid.clone())); + manager.send_event(EventMessage::PlaylistUpdateProgress(progress("conflicting", 1, 7))); + assert_eq!(manager.playlist_update_snapshot(), vec![valid]); + } +} diff --git a/bin/build_local.sh b/bin/build_local.sh index 0c7937be0..aa1e60793 100755 --- a/bin/build_local.sh +++ b/bin/build_local.sh @@ -102,6 +102,12 @@ for arg in "$@"; do --debug) BUILD_DEBUG_SYMBOLS=true ;; + --dev) + BUILD_PROFILE="dev" + ;; + --dev-slim) + BUILD_PROFILE="dev-slim" + ;; -h|--help) usage exit 0 @@ -139,9 +145,7 @@ if [ "$BUILD_BACKEND" = true ]; then export CARGO_PROFILE_RELEASE_STRIP=false fi - if [ -z "${SOURCE_DATE_EPOCH:-}" ]; then - SOURCE_DATE_EPOCH="$(date -u +%s)" - fi + SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:-$(git log -1 --format=%ct)}" export SOURCE_DATE_EPOCH echo "==> Backend build SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH}" @@ -158,29 +162,29 @@ if [ "$BUILD_BACKEND" = true ]; then exit 1 fi rustup target add x86_64-unknown-linux-musl - cargo zigbuild -p tuliprox --release --target x86_64-unknown-linux-musl + cargo zigbuild -p tuliprox --profile ${BUILD_PROFILE:-release} --target x86_64-unknown-linux-musl else - cross build -p tuliprox --release --target x86_64-unknown-linux-musl + cross build -p tuliprox --profile ${BUILD_PROFILE:-release} --target x86_64-unknown-linux-musl fi ;; linux-gnu) if [ "$(uname)" = "Linux" ]; then - cargo build -p tuliprox --release + cargo build -p tuliprox --profile ${BUILD_PROFILE:-release} else - cross build -p tuliprox --release --target x86_64-unknown-linux-gnu + cross build -p tuliprox --profile ${BUILD_PROFILE:-release} --target x86_64-unknown-linux-gnu fi ;; armv7) - cross build -p tuliprox --release --target armv7-unknown-linux-musleabihf + cross build -p tuliprox --profile ${BUILD_PROFILE:-release} --target armv7-unknown-linux-musleabihf ;; aarch64) - cross build -p tuliprox --release --target aarch64-unknown-linux-musl + cross build -p tuliprox --profile ${BUILD_PROFILE:-release} --target aarch64-unknown-linux-musl ;; macos) - cross build -p tuliprox --release --target x86_64-apple-darwin + cross build -p tuliprox --profile ${BUILD_PROFILE:-release} --target x86_64-apple-darwin ;; windows) - cargo build -p tuliprox --release --target x86_64-pc-windows-gnu + cargo build -p tuliprox --profile ${BUILD_PROFILE:-release} --target x86_64-pc-windows-gnu ;; esac fi diff --git a/docs/src/configuration/source.md b/docs/src/configuration/source.md index 2bc99ce05..2042d4d13 100644 --- a/docs/src/configuration/source.md +++ b/docs/src/configuration/source.md @@ -245,6 +245,7 @@ specific provider. | Parameter | Type | Default | Technical Impact & Background | |:-------------------------------------------|:---------|:--------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `skip_live` / `skip_vod` / `skip_series` | Bool | `false` | Immediately ignores entire categories during Xtream or Stalker ingestion. Saves massive amounts of RAM and runtime if you only want specific clusters from a provider. | +| `update_quality.live` / `.vod` / `.series` | Int | `0` | Rejects an Xtream, Stalker, or parsed M3U cluster when its item count differs too much from the last accepted cluster. Values are percentages from `0` through `100`; `0` disables the guard. | | `xtream_live_stream_without_extension` | Bool | `false` | Strips `.ts` from generated stream URLs. | | `xtream_live_stream_use_prefix` | Bool | `true` | Injects the `/live/` prefix into URLs. | | `disable_hls_streaming` | Bool | `false` | Rewrites live `.m3u8` requests to `.ts` and bypasses Tuliprox HLS handling. | @@ -265,6 +266,40 @@ specific provider. > **Note:** For `resolve_vod` and `resolve_series`, data is cached per input and only new or changed entries are > updated. +#### Update quality guard + +`update_quality` protects Live, VOD, and Series clusters from unexpectedly small or large provider responses. It is +available for Xtream, expanded Xtream batch inputs, staged Xtream inputs, Stalker, expanded Stalker batch inputs, and +M3U inputs. For M3U, the downloaded document is partitioned by the parsed content cluster before the guard is evaluated; +accepted candidate clusters and retained previous clusters are combined before persistence. + +```yaml +inputs: + - name: guarded-provider + type: xtream + url: https://provider.example + username: user + password: password + options: + update_quality: + live: 90 + vod: 90 + series: 100 +``` + +Each value is a minimum percentage of similarity between the previous and candidate item counts: + +* `0` disables the additional check and preserves the behavior of existing configurations. +* `90` accepts counts from 90% through 110% of the previous count, including both boundaries. +* `100` accepts only an identical item count. + +With no previous cluster, the first non-empty candidate is accepted as the baseline; an empty candidate is not. A +rejected Xtream cluster retains its active cluster database and associated categories. A rejected Stalker cluster keeps +its previous active manifest entry and generation. Accepted clusters from the same update continue to publication, so +target processing receives newly accepted clusters together with retained clusters. The input remains usable and the +overall update is reported as partial. For Xtream, only the rejected cluster is marked for retry in the input cache; +Stalker continues to use its existing requested selection and refresh lifecycle. + #### Minimal Xtream MPEG-TS Example ```yaml diff --git a/frontend/Cargo.toml b/frontend/Cargo.toml index 3c43c29b0..f80af3249 100644 --- a/frontend/Cargo.toml +++ b/frontend/Cargo.toml @@ -42,7 +42,7 @@ strum_macros.workspace = true [dependencies.web-sys] version = "0.3.104" -features = ["AbortController", "AbortSignal", "ReadableStream", "ReadableStreamDefaultReader", "Document", "Element", "Node", "Window", "HtmlCollection", "HtmlImageElement", "HtmlInputElement", "DomRect", "HtmlDialogElement", "CssStyleDeclaration", "Event", "BeforeUnloadEvent", "MutationObserver", "MutationObserverInit", "WebGlRenderingContext", "WebGlProgram", "WebGlShader", "WebGlBuffer", "WebGlUniformLocation", "HtmlCanvasElement", "HtmlSelectElement", "DomTokenList", "Selection"] +features = ["AbortController", "AbortSignal", "ReadableStream", "ReadableStreamDefaultReader", "Document", "Element", "Node", "Window", "VisualViewport", "HtmlCollection", "HtmlImageElement", "HtmlInputElement", "DomRect", "HtmlDialogElement", "CssStyleDeclaration", "Event", "BeforeUnloadEvent", "MutationObserver", "MutationObserverInit", "WebGlRenderingContext", "WebGlProgram", "WebGlShader", "WebGlBuffer", "WebGlUniformLocation", "HtmlCanvasElement", "HtmlSelectElement", "DomTokenList", "Selection"] [dev-dependencies] wasm-bindgen-test = "0.3.77" diff --git a/frontend/public/assets/i18n/ar.json b/frontend/public/assets/i18n/ar.json index 840cd7429..613c95b09 100644 --- a/frontend/public/assets/i18n/ar.json +++ b/frontend/public/assets/i18n/ar.json @@ -505,6 +505,7 @@ }, "STAGED_CLUSTERS": "المجموعات التي يتجاوزها هذا الإدخال المُرحّل في إدخال المزود المتصل.", "STAGED_PERSIST": "مسار ملف اختياري يُستخدم لحفظ قائمة التشغيل المُرحّلة التي تم تنزيلها أو إعادة استخدامها.", + "UPDATE_QUALITY": "الحد الأدنى المقبول لجودة تحديث المجموعة بالنسبة المئوية. تؤدي القيمة 0 إلى تعطيل فحص الجودة.", "URL": "عنوان URL المصدر لهذا الإدخال.", "SERIES_SOURCE": "محدد المصدر لمجموعة المسلسلات عند تفعيل الإدخال المُرحّل (`default`، `staged`، `input`، `skip`).", "VOD_SOURCE": "محدد المصدر لمجموعة الفيديو حسب الطلب عند تفعيل الإدخال المُرحّل (`default`، `staged`، `input`، `skip`)." @@ -1156,6 +1157,7 @@ "IPv4": "IPv4", "IPv6": "IPv6", "ISSUER": "المُصدر", + "EMBY": "Emby", "JELLYFIN": "Jellyfin", "JOIN_ON_DISCORD": "انضم إلينا على Discord", "KICK": "طرد", @@ -1521,6 +1523,8 @@ "SPECIFIC_TEMPLATES": "قوالب محددة", "SSDP_DISCOVERY": "اكتشاف SSDP", "STAGED": "مُرحّل", + "STALKER": "بوابة ستالكر", + "STALKER_BATCH": "دفعة ستالكر", "CHILD": "فرع", "LIVE_SOURCE": "مصدر البث المباشر", "VOD_SOURCE": "مصدر VOD", @@ -1623,10 +1627,35 @@ "TYPE": "النوع", "UI_ENABLED": "واجهة المستخدم", "UNDERSCORE_WHITESPACE": "شرطة سفلية للمسافات", + "ACTION": "الإجراء", + "AFFECTED_TARGETS": "الأهداف المتأثرة", "UPDATE": "تحديث", + "UPDATE_ALL_INPUTS": "تحديث جميع المدخلات", + "UPDATE_START": "بدء التحديث", + "RESCAN": "إعادة الفحص", + "START_RESCAN": "بدء إعادة الفحص", + "BULK_ONLY": "التحديث الشامل فقط", + "UPDATE_ACTIONS": "إجراءات التحديث", + "UPDATE_STATUS_READY": "جاهز", + "UPDATE_STATUS_QUEUED": "في قائمة الانتظار", + "UPDATE_STATUS_UPDATING": "جارٍ التحديث…", + "UPDATE_STATUS_SUCCESS": "ناجح", + "UPDATE_STATUS_PARTIAL": "جزئي", + "UPDATE_STATUS_FAILED": "فشل", + "REFRESH_START": "بدء التحديث دون استخدام ذاكرة التخزين المؤقت", + "FORCE_UPDATE": "تحديث إجباري", + "FORCE_UPDATE_START": "بدء التحديث الإجباري", + "UPDATE_QUALITY": "جودة التحديث", "UPDATE_GEOIP_DB": "تحديث قاعدة بيانات Geo-IP", "UPDATE_LOCAL_LIBRARY": "تحديث المكتبة المحلية", "UPDATE_ON_BOOT": "التحديث عند الإقلاع", + "LAST_UPDATE": "آخر تحديث", + "LAST_UPDATE_NEVER": "أبدًا", + "LAST_UPDATE_JUST_NOW": "الآن للتو", + "LAST_UPDATE_MINUTE_AGO": "قبل دقيقة واحدة", + "LAST_UPDATE_MINUTES_AGO": "قبل {count} دقيقة", + "LAST_UPDATE_HOUR_AGO": "قبل ساعة واحدة", + "LAST_UPDATE_HOURS_AGO": "قبل {count} ساعة", "URL": "URL", "URL_IPV4": "URL IPv4", "URL_IPV6": "URL IPv6", @@ -1954,8 +1983,74 @@ "WRITE_WITHOUT_READ": "الكتابة مفعّلة بدون قراءة" }, "PLAYLIST_UPDATE": { + "CONTENT_LIVE": "مباشر", + "CONTENT_SERIES": "مسلسلات", + "CONTENT_MOVIES": "أفلام", + "ACTIVE_POPULATION": "المجموعة النشطة", + "BASELINE": "المجموعة السابقة", + "BULK_BUSY": "يوجد تحديث آخر لقائمة التشغيل قيد التشغيل أو الانتظار. لم يبدأ التحديث الشامل.", + "BULK_ONLY_DESCRIPTION": "لا يتوفر تحديث منفرد لهذا الإدخال. استخدم «تحديث جميع الإدخالات».", + "PROVIDER_IMPORT_UNAVAILABLE": "استيراد المزود غير متاح", + "CACHE_BYPASSED_FORCE": "تم تجاوزه بواسطة Force", + "CACHE_BYPASSED_REFRESH": "تم تجاوزه بواسطة Refresh", + "CACHE_NOT_USED": "لم يُستخدم (المزوّد)", + "CACHE_REFRESH_DESCRIPTION": "إعادة بناء الأهداف المحددة بجميع بيانات الإدخالات المطلوبة. تجاوز ذاكرة التخزين المؤقت لهذا الإدخال وتنزيله من جديد.", + "CACHE_UPDATE_DESCRIPTION": "إعادة بناء الأهداف المحددة بجميع بيانات الإدخالات المطلوبة. لهذا الإدخال، استخدم ذاكرة التخزين المؤقت الصالحة أو نزّل البيانات من جديد.", + "CACHE_USED": "مستخدم", + "CANDIDATE": "المرشح", + "CLUSTER_ACCEPTED": "تم الاعتماد", + "CLUSTER_CACHED": "من التخزين المؤقت", + "CLUSTER_NOT_REQUESTED": "لم يُطلب / غير ذي صلة", + "CLUSTER_REJECTED": "مرفوض", + "CLUSTER_TECHNICAL_ERROR": "خطأ تقني", + "CONFIGURED_COUNT": "تم التكوين: {count}", + "DATA_UNAVAILABLE": "البيانات غير متاحة", + "DECISION": "القرار", + "DETAIL_FORCED_UPDATE_REQUESTED": "طُلب تحديث إجباري؛ تم تجاوز التخزين المؤقت وفحوصات الجودة.", + "DETAIL_INPUT_COMPLETED_PARTIALLY": "اكتمل تحديث الإدخال جزئيًا.", + "DETAIL_INPUT_COMPLETED_SUCCESSFULLY": "اكتمل تحديث الإدخال بنجاح.", + "DETAIL_INPUT_FAILED": "فشل تحديث الإدخال.", + "DETAIL_LIBRARY_RESCAN_COMPLETED": "اكتملت إعادة فحص المكتبة؛ تجري إعادة بناء الأهداف المحددة.", + "DETAIL_LIBRARY_RESCAN_FAILED": "فشلت إعادة فحص المكتبة؛ لم تُعَد بناء الأهداف المحددة.", + "DETAIL_LIBRARY_RESCAN_STARTED": "بدأت إعادة فحص المكتبة؛ إعادة بناء الأهداف المحددة قيد الانتظار.", + "DETAIL_LIBRARY_SCAN_RESULT": "تم استلام نتيجة فحص المكتبة.", "FAIL": "فشل تحديث قائمة التشغيل!", + "FORCE_CONFIRM": "سيُعاد تحميل الإدخال بغض النظر عن ذاكرة التخزين المؤقت. سيتم تجاوز جودة التحديث المكوّنة لهذا التشغيل. قد تؤدي نتيجة موفر فارغة وناجحة تقنيًا إلى إفراغ المجموعات المحددة. تبقى البيانات الحالية نشطة عند حدوث أخطاء في الشبكة أو التحليل أو التخزين أو النشر.", + "IDENTICAL_POPULATION": "مجموعة متطابقة", + "INPUT_ACCEPTED": "تم بدء {action} للإدخال {input}.", + "INPUT_BUSY": "يوجد تحديث آخر قيد التشغيل أو الانتظار. لم يبدأ {action} للإدخال {input}.", + "INPUT_DISABLED": "هذا الإدخال معطّل في الإعدادات ولا يمكن تحديثه.", + "INPUT_FAILED": "تعذر بدء {action} للإدخال {input}.", + "LIBRARY_DESCRIPTION": "إعادة فحص المكتبة المحلية وإعادة بناء الأهداف المحددة بجميع بيانات الإدخالات المطلوبة.", + "LIBRARY_CATALOG": "كتالوج المكتبة", + "LIBRARY_MOVIES": "الأفلام", + "LIBRARY_SERIES": "المسلسلات", + "LIBRARY_EPISODES": "الحلقات", + "LIBRARY_TOTAL_ITEMS": "إجمالي العناصر", + "LIBRARY_LAST_RESCAN": "آخر إعادة فحص", + "LIBRARY_FILES_SCANNED": "ملفات الفيديو المفحوصة", + "LIBRARY_GROUPS_SCANNED": "مجموعات الوسائط", + "LIBRARY_ADDED": "المضافة", + "LIBRARY_UPDATED": "المحدثة", + "LIBRARY_REMOVED": "المحذوفة", + "LIBRARY_ERRORS": "الأخطاء", + "MANUAL_UNAVAILABLE": "لا يتوفر التحديث اليدوي المنفرد لهذا النوع من الإدخالات.", + "MODE": "الوضع", "NO_TARGETS": "لم يتم تكوين قوائم تشغيل. أنشئ المصادر والأهداف في محرر المصادر أولاً.", + "NO_TARGETS_ASSIGNED": "لم يتم تعيين أهداف.", + "POLICY_FORCE_DESCRIPTION": "إعادة بناء الأهداف المحددة بجميع بيانات الإدخالات المطلوبة. لهذا الإدخال فقط، تجاوز ذاكرة التخزين المؤقت وجودة التحديث المُعدّة لهذا التشغيل.", + "POLICY_LABEL": "{action} للإدخال {input}", + "POLICY_REFRESH_DESCRIPTION": "إعادة بناء الأهداف المحددة بجميع بيانات الإدخالات المطلوبة. إعادة تحميل هذا الإدخال دون ذاكرة التخزين المؤقت وتطبيق جودة التحديث المُعدّة.", + "POLICY_UPDATE_DESCRIPTION": "إعادة بناء الأهداف المحددة بجميع بيانات الإدخالات المطلوبة. لهذا الإدخال، استخدم ذاكرة التخزين المؤقت الصالحة وطبّق جودة التحديث المُعدّة.", + "UPDATE_ONLY_DESCRIPTION": "إعادة بناء الأهداف المحددة بجميع بيانات الإدخالات المطلوبة. لا يتوفر التحديث مع إعادة التحميل أو التحديث الإجباري لهذا الإدخال.", + "QUALITY_ACHIEVED": "الجودة المحققة", + "QUALITY_BYPASSED": "تم التجاوز", + "QUALITY_GUARD": "حارس الجودة", + "QUALITY_THRESHOLD": "الحد", + "REQUIRED": "المطلوب", + "RUN_DETAILS": "تفاصيل التشغيل", + "START_LABEL": "{action} للإدخال {input}", + "UPDATE_DETAILS": "تفاصيل التحديث", "FAIL_FINISH": "فشل تحديث قائمة التشغيل!", "PARTIAL_FINISH": "تحديث قائمة التشغيل غير مكتمل وسيستمر من نقطة الحفظ.", "SUCCESS": "تم بدء تحديث قائمة التشغيل بنجاح!", diff --git a/frontend/public/assets/i18n/en.json b/frontend/public/assets/i18n/en.json index 87bc00511..68d3ee7cc 100644 --- a/frontend/public/assets/i18n/en.json +++ b/frontend/public/assets/i18n/en.json @@ -509,6 +509,7 @@ }, "STAGED_CLUSTERS": "Clusters this staged input overrides on the connected provider input.", "STAGED_PERSIST": "Optional file path used to persist or reuse the downloaded staged playlist.", + "UPDATE_QUALITY": "Minimum accepted cluster update quality in percent. A value of 0 disables the quality check.", "URL": "Source URL for this input.", "SERIES_SOURCE": "Source selector for the Series cluster when staged input is enabled (`default`, `staged`, `input`, `skip`).", "VOD_SOURCE": "Source selector for the VOD cluster when staged input is enabled (`default`, `staged`, `input`, `skip`)." @@ -1163,6 +1164,7 @@ "IPv4": "IPv4", "IPv6": "IPv6", "ISSUER": "Issuer", + "EMBY": "Emby", "JELLYFIN": "Jellyfin", "JOIN_ON_DISCORD": "Join us on Discord", "KICK": "Kick", @@ -1709,10 +1711,35 @@ "TYPE": "Type", "UI_ENABLED": "Ui", "UNDERSCORE_WHITESPACE": "Underscore whitespace", + "ACTION": "Action", + "AFFECTED_TARGETS": "Affected targets", "UPDATE": "Update", + "UPDATE_ALL_INPUTS": "Update all inputs", + "UPDATE_START": "Start update", + "RESCAN": "Rescan", + "START_RESCAN": "Start rescan", + "BULK_ONLY": "Bulk only", + "UPDATE_ACTIONS": "Update actions", + "UPDATE_STATUS_READY": "Ready", + "UPDATE_STATUS_QUEUED": "Queued", + "UPDATE_STATUS_UPDATING": "Updating…", + "UPDATE_STATUS_SUCCESS": "Successful", + "UPDATE_STATUS_PARTIAL": "Partial", + "UPDATE_STATUS_FAILED": "Failed", + "REFRESH_START": "Start refresh", + "FORCE_UPDATE": "Force Update", + "FORCE_UPDATE_START": "Start force update", + "UPDATE_QUALITY": "Update quality", "UPDATE_GEOIP_DB": "Update Geo-IP db", "UPDATE_LOCAL_LIBRARY": "Update Local Library", "UPDATE_ON_BOOT": "Update on boot", + "LAST_UPDATE": "Last update", + "LAST_UPDATE_NEVER": "Never", + "LAST_UPDATE_JUST_NOW": "just now", + "LAST_UPDATE_MINUTE_AGO": "1 minute ago", + "LAST_UPDATE_MINUTES_AGO": "{count} minutes ago", + "LAST_UPDATE_HOUR_AGO": "1 hour ago", + "LAST_UPDATE_HOURS_AGO": "{count} hours ago", "UPTIME": "Uptime", "URL": "Url", "URL_IPV4": "Url IPv4", @@ -1989,8 +2016,74 @@ "WRITE_WITHOUT_READ": "Write is enabled without Read" }, "PLAYLIST_UPDATE": { + "CONTENT_LIVE": "Live", + "CONTENT_SERIES": "Series", + "CONTENT_MOVIES": "Movies", + "ACTIVE_POPULATION": "Active population", + "BASELINE": "Previous population", + "BULK_BUSY": "Another playlist update is already running or queued. The bulk update was not started.", + "BULK_ONLY_DESCRIPTION": "Single-input updates are not available for this input. Use ‘Update all inputs’.", + "PROVIDER_IMPORT_UNAVAILABLE": "Provider import not available", + "CACHE_BYPASSED_FORCE": "Bypassed by Force", + "CACHE_BYPASSED_REFRESH": "Bypassed by Refresh", + "CACHE_NOT_USED": "Not used (provider)", + "CACHE_REFRESH_DESCRIPTION": "Rebuild the selected targets with all required input data. Bypass this input's cache and download it again.", + "CACHE_UPDATE_DESCRIPTION": "Rebuild the selected targets with all required input data. For this input, reuse a valid cache or download it again.", + "CACHE_USED": "Used", + "CANDIDATE": "Candidate", + "CLUSTER_ACCEPTED": "Accepted", + "CLUSTER_CACHED": "From cache", + "CLUSTER_NOT_REQUESTED": "Not requested / not relevant", + "CLUSTER_REJECTED": "Rejected", + "CLUSTER_TECHNICAL_ERROR": "Technical error", + "CONFIGURED_COUNT": "{count} configured", + "DATA_UNAVAILABLE": "Data unavailable", + "DECISION": "Decision", + "DETAIL_FORCED_UPDATE_REQUESTED": "Forced update requested; cache and quality checks were bypassed.", + "DETAIL_INPUT_COMPLETED_PARTIALLY": "Input update completed partially.", + "DETAIL_INPUT_COMPLETED_SUCCESSFULLY": "Input update completed successfully.", + "DETAIL_INPUT_FAILED": "Input update failed.", + "DETAIL_LIBRARY_RESCAN_COMPLETED": "Library rescan completed; rebuilding selected targets.", + "DETAIL_LIBRARY_RESCAN_FAILED": "Library rescan failed; selected targets were not rebuilt.", + "DETAIL_LIBRARY_RESCAN_STARTED": "Library rescan started; selected target rebuild is waiting.", + "DETAIL_LIBRARY_SCAN_RESULT": "Library scan result received.", "FAIL": "Playlist update failed!", + "FORCE_CONFIRM": "The input will be reloaded regardless of its cache. The configured update quality is bypassed for this run. A technically successful empty provider result can clear the selected clusters. Existing data remains active after network, parsing, persistence, or publication errors.", + "IDENTICAL_POPULATION": "Identical population", + "INPUT_ACCEPTED": "Started {action} for {input}.", + "INPUT_BUSY": "Another update is already running or queued. {action} for {input} was not started.", + "INPUT_DISABLED": "This input is disabled in configuration and cannot be updated.", + "INPUT_FAILED": "{action} for {input} could not be started.", + "LIBRARY_DESCRIPTION": "Rescan the local library and rebuild the selected targets with all required input data.", + "LIBRARY_CATALOG": "Library catalog", + "LIBRARY_MOVIES": "Movies", + "LIBRARY_SERIES": "Series", + "LIBRARY_EPISODES": "Episodes", + "LIBRARY_TOTAL_ITEMS": "Total items", + "LIBRARY_LAST_RESCAN": "Last rescan", + "LIBRARY_FILES_SCANNED": "Video files scanned", + "LIBRARY_GROUPS_SCANNED": "Media groups", + "LIBRARY_ADDED": "Added", + "LIBRARY_UPDATED": "Updated", + "LIBRARY_REMOVED": "Removed", + "LIBRARY_ERRORS": "Errors", + "MANUAL_UNAVAILABLE": "Manual single-input updates are not available for this input type.", + "MODE": "Mode", "NO_TARGETS": "No playlists configured. Create sources and targets in the source editor first.", + "NO_TARGETS_ASSIGNED": "No targets assigned.", + "POLICY_FORCE_DESCRIPTION": "Rebuild the selected targets with all required input data. For this input only, bypass cache and the configured update quality for this run.", + "POLICY_LABEL": "{action} for {input}", + "POLICY_REFRESH_DESCRIPTION": "Rebuild the selected targets with all required input data. Reload this input without its cache and enforce the configured update quality.", + "POLICY_UPDATE_DESCRIPTION": "Rebuild the selected targets with all required input data. For this input, reuse a valid cache and enforce the configured update quality.", + "UPDATE_ONLY_DESCRIPTION": "Rebuild the selected targets with all required input data. Refresh and Force are not available for this input.", + "QUALITY_ACHIEVED": "Achieved quality", + "QUALITY_BYPASSED": "Bypassed", + "QUALITY_GUARD": "Quality Guard", + "QUALITY_THRESHOLD": "Threshold", + "REQUIRED": "Required", + "RUN_DETAILS": "Run details", + "START_LABEL": "{action} for {input}", + "UPDATE_DETAILS": "Update Details", "FAIL_FINISH": "Playlist update failed!", "PARTIAL_FINISH": "Playlist update is incomplete and will continue from its saved checkpoint.", "SUCCESS": "Successfully started playlist update!", diff --git a/frontend/public/assets/i18n/ru.json b/frontend/public/assets/i18n/ru.json index 65955b918..ce646ba60 100644 --- a/frontend/public/assets/i18n/ru.json +++ b/frontend/public/assets/i18n/ru.json @@ -481,6 +481,7 @@ }, "STAGED_CLUSTERS": "Кластеры, которые этот промежуточный ввод переопределяет для подключенного ввода провайдера.", "STAGED_PERSIST": "Необязательный путь к файлу для сохранения или повторного использования загруженного промежуточного плейлиста.", + "UPDATE_QUALITY": "Минимально допустимое качество обновления кластера в процентах. Значение 0 отключает проверку качества.", "URL": "URL источника для этого ввода.", "SERIES_SOURCE": "Селектор источника для кластера Series, когда включен промежуточный ввод (`default`, `staged`, `input`, `skip`).", "VOD_SOURCE": "Селектор источника для кластера VOD, когда включен промежуточный ввод (`default`, `staged`, `input`, `skip`)." @@ -1106,6 +1107,7 @@ "IPv4": "IPv4", "IPv6": "IPv6", "ISSUER": "Эмитент", + "EMBY": "Emby", "JELLYFIN": "Jellyfin", "JOIN_ON_DISCORD": "Присоединяйтесь к нам в Discord", "KICK": "Выкинуть", @@ -1521,10 +1523,35 @@ "TYPE": "Тип", "UI_ENABLED": "Ui", "UNDERSCORE_WHITESPACE": "Подчеркивание пробелов", + "ACTION": "Действие", + "AFFECTED_TARGETS": "Затрагиваемые цели", "UPDATE": "Обновить", + "UPDATE_ALL_INPUTS": "Обновить все источники", + "UPDATE_START": "Запустить обновление", + "RESCAN": "Повторное сканирование", + "START_RESCAN": "Начать сканирование", + "BULK_ONLY": "Только общее обновление", + "UPDATE_ACTIONS": "Действия обновления", + "UPDATE_STATUS_READY": "Готово", + "UPDATE_STATUS_QUEUED": "В очереди", + "UPDATE_STATUS_UPDATING": "Обновляется…", + "UPDATE_STATUS_SUCCESS": "Успешно", + "UPDATE_STATUS_PARTIAL": "Частично", + "UPDATE_STATUS_FAILED": "Ошибка", + "REFRESH_START": "Запустить обновление без кэша", + "FORCE_UPDATE": "Принудительно обновить", + "FORCE_UPDATE_START": "Запустить принудительное обновление", + "UPDATE_QUALITY": "Качество обновления", "UPDATE_GEOIP_DB": "Обновить Geo-IP базу", "UPDATE_LOCAL_LIBRARY": "Обновить локальную библиотеку", "UPDATE_ON_BOOT": "Обновлять при загрузке", + "LAST_UPDATE": "Последнее обновление", + "LAST_UPDATE_NEVER": "Никогда", + "LAST_UPDATE_JUST_NOW": "только что", + "LAST_UPDATE_MINUTE_AGO": "1 минуту назад", + "LAST_UPDATE_MINUTES_AGO": "{count} мин. назад", + "LAST_UPDATE_HOUR_AGO": "1 час назад", + "LAST_UPDATE_HOURS_AGO": "{count} ч. назад", "URL": "Url", "URL_IPV4": "Url IPv4", "URL_IPV6": "Url IPv6", @@ -1855,8 +1882,74 @@ "WRITE_WITHOUT_READ": "Запись включена без чтения" }, "PLAYLIST_UPDATE": { + "CONTENT_LIVE": "Прямой эфир", + "CONTENT_SERIES": "Сериалы", + "CONTENT_MOVIES": "Фильмы", + "ACTIVE_POPULATION": "Активный набор", + "BASELINE": "Предыдущий набор", + "BULK_BUSY": "Другое обновление плейлиста уже выполняется или ожидает. Массовое обновление не запущено.", + "BULK_ONLY_DESCRIPTION": "Отдельное обновление этого источника недоступно. Используйте «Обновить все источники».", + "PROVIDER_IMPORT_UNAVAILABLE": "Импорт провайдера недоступен", + "CACHE_BYPASSED_FORCE": "Обойден режимом Force", + "CACHE_BYPASSED_REFRESH": "Обойден режимом Refresh", + "CACHE_NOT_USED": "Не использован (провайдер)", + "CACHE_REFRESH_DESCRIPTION": "Пересобрать выбранные цели со всеми необходимыми данными источников. Обойти кэш этого источника и загрузить его заново.", + "CACHE_UPDATE_DESCRIPTION": "Пересобрать выбранные цели со всеми необходимыми данными источников. Для этого источника использовать действующий кэш или загрузить данные заново.", + "CACHE_USED": "Использован", + "CANDIDATE": "Кандидат", + "CLUSTER_ACCEPTED": "Принято", + "CLUSTER_CACHED": "Из кэша", + "CLUSTER_NOT_REQUESTED": "Не запрошено / неприменимо", + "CLUSTER_REJECTED": "Отклонено", + "CLUSTER_TECHNICAL_ERROR": "Техническая ошибка", + "CONFIGURED_COUNT": "Настроено: {count}", + "DATA_UNAVAILABLE": "Данные недоступны", + "DECISION": "Решение", + "DETAIL_FORCED_UPDATE_REQUESTED": "Запрошено принудительное обновление; кэш и проверка качества обойдены.", + "DETAIL_INPUT_COMPLETED_PARTIALLY": "Обновление источника завершено частично.", + "DETAIL_INPUT_COMPLETED_SUCCESSFULLY": "Источник успешно обновлён.", + "DETAIL_INPUT_FAILED": "Не удалось обновить источник.", + "DETAIL_LIBRARY_RESCAN_COMPLETED": "Повторное сканирование библиотеки завершено; выбранные цели перестраиваются.", + "DETAIL_LIBRARY_RESCAN_FAILED": "Повторное сканирование библиотеки не удалось; выбранные цели не перестроены.", + "DETAIL_LIBRARY_RESCAN_STARTED": "Начато повторное сканирование библиотеки; перестроение выбранных целей ожидает.", + "DETAIL_LIBRARY_SCAN_RESULT": "Получен результат сканирования библиотеки.", "FAIL": "Обновление плейлиста не удалось!", + "FORCE_CONFIRM": "Источник будет загружен заново независимо от кэша. Настроенная проверка качества обновления будет пропущена для этого запуска. Технически успешный пустой ответ провайдера может очистить выбранные кластеры. При ошибках сети, разбора, сохранения или публикации текущие данные останутся активными.", + "IDENTICAL_POPULATION": "Идентичный набор", + "INPUT_ACCEPTED": "Запущено действие «{action}» для «{input}».", + "INPUT_BUSY": "Другое обновление уже выполняется или ожидает. Действие «{action}» для «{input}» не запущено.", + "INPUT_DISABLED": "Этот источник отключён в конфигурации и не может быть обновлён.", + "INPUT_FAILED": "Не удалось запустить действие «{action}» для «{input}».", + "LIBRARY_DESCRIPTION": "Повторно просканировать локальную библиотеку и пересобрать выбранные цели со всеми необходимыми данными источников.", + "LIBRARY_CATALOG": "Каталог библиотеки", + "LIBRARY_MOVIES": "Фильмы", + "LIBRARY_SERIES": "Сериалы", + "LIBRARY_EPISODES": "Эпизоды", + "LIBRARY_TOTAL_ITEMS": "Всего записей", + "LIBRARY_LAST_RESCAN": "Последнее сканирование", + "LIBRARY_FILES_SCANNED": "Просканировано видеофайлов", + "LIBRARY_GROUPS_SCANNED": "Группы медиа", + "LIBRARY_ADDED": "Добавлено", + "LIBRARY_UPDATED": "Обновлено", + "LIBRARY_REMOVED": "Удалено", + "LIBRARY_ERRORS": "Ошибки", + "MANUAL_UNAVAILABLE": "Ручное отдельное обновление для этого типа источника недоступно.", + "MODE": "Режим", "NO_TARGETS": "Плейлисты не настроены. Сначала создайте источники и цели в редакторе источников.", + "NO_TARGETS_ASSIGNED": "Цели не назначены.", + "POLICY_FORCE_DESCRIPTION": "Пересобрать выбранные цели со всеми необходимыми данными источников. Только для этого источника обойти кэш и настроенную проверку качества обновления в этом запуске.", + "POLICY_LABEL": "{action} для «{input}»", + "POLICY_REFRESH_DESCRIPTION": "Пересобрать выбранные цели со всеми необходимыми данными источников. Загрузить этот источник заново без кэша и применить настроенную проверку качества обновления.", + "POLICY_UPDATE_DESCRIPTION": "Пересобрать выбранные цели со всеми необходимыми данными источников. Для этого источника использовать действующий кэш и применить настроенную проверку качества обновления.", + "UPDATE_ONLY_DESCRIPTION": "Пересобрать выбранные цели со всеми необходимыми данными источников. Обновление с перезагрузкой и принудительное обновление для этого источника недоступны.", + "QUALITY_ACHIEVED": "Достигнутое качество", + "QUALITY_BYPASSED": "Обойден", + "QUALITY_GUARD": "Контроль качества", + "QUALITY_THRESHOLD": "Порог", + "REQUIRED": "Требование", + "RUN_DETAILS": "Детали запуска", + "START_LABEL": "{action} для «{input}»", + "UPDATE_DETAILS": "Детали обновления", "FAIL_FINISH": "Обновление плейлиста не удалось!", "PARTIAL_FINISH": "Обновление плейлиста не завершено и продолжится с сохранённой контрольной точки.", "SUCCESS": "Обновление плейлиста успешно запущено!", diff --git a/frontend/scss/app/_component.scss b/frontend/scss/app/_component.scss index 4c4b97068..3a68e2f92 100644 --- a/frontend/scss/app/_component.scss +++ b/frontend/scss/app/_component.scss @@ -62,6 +62,7 @@ @forward "components/userlist/proxy_user_credentials_form"; @forward "components/userlist/user_table"; @forward "components/radio_button_group"; +@forward "components/range_slider"; @forward "components/no_content"; @forward "components/search"; @forward "components/loading_indicator"; diff --git a/frontend/scss/app/components/_custom_dialog.scss b/frontend/scss/app/components/_custom_dialog.scss index befc735de..933f71395 100644 --- a/frontend/scss/app/components/_custom_dialog.scss +++ b/frontend/scss/app/components/_custom_dialog.scss @@ -18,6 +18,7 @@ } .tp__custom-dialog { + --dialog-padding: var(--padding-default); display: flex; flex-direction: column; overflow: hidden; @@ -37,16 +38,39 @@ animation: slideIn 0.2s ease-out; &.tp__confirm-dialog { - min-width: size.$confirm-dialog-min-width; - max-width: size.$confirm-dialog-max-width; - padding: var(--padding-default); + --dialog-padding: var(--padding-default); + + min-width: min(size.$confirm-dialog-min-width, 90vw); + max-width: min(size.$confirm-dialog-max-width, 90vw); + max-height: 80vh; + box-sizing: border-box; + padding: var(--dialog-padding); + gap: var(--gap-large); border-radius: var(--border-radius); + + @media (max-width: size.$mobile-breakpoint) { + --dialog-padding: var(--padding-larger); + } + + .tp__confirm-dialog__message { + margin: 0; + min-height: 0; + font-size: var(--font-size-norm); + line-height: 1.5; + white-space: pre-wrap; + overflow-wrap: anywhere; + overflow-y: auto; + } + + .tp__dialog__toolbar { + flex-shrink: 0; + } } &.tp__content-dialog { min-width: size.$content-dialog-min-width; max-width: size.$content-dialog-max-width; - padding: var(--padding-default); + padding: var(--dialog-padding); border-radius: var(--border-radius); } } @@ -77,7 +101,7 @@ display: flex; justify-content: flex-end; border-top: 1px solid var(--border-color); - padding-top: var(--padding-small); + padding-top: var(--dialog-padding, var(--padding-small)); gap: var(--gap-default); &-left { diff --git a/frontend/scss/app/components/_dialog.scss b/frontend/scss/app/components/_dialog.scss index 1a8db7cca..ccf2a5540 100644 --- a/frontend/scss/app/components/_dialog.scss +++ b/frontend/scss/app/components/_dialog.scss @@ -1,8 +1,9 @@ dialog { + --dialog-padding: var(--padding-larger); display: flex; flex-flow: column; gap: var(--gap-large); - padding: var(--padding-larger); + padding: var(--dialog-padding); background-color: var(--card-background-color); color: var(--card-color); border: none; diff --git a/frontend/scss/app/components/_drop_down_icon_button.scss b/frontend/scss/app/components/_drop_down_icon_button.scss index 3a45016ab..6528070ff 100644 --- a/frontend/scss/app/components/_drop_down_icon_button.scss +++ b/frontend/scss/app/components/_drop_down_icon_button.scss @@ -14,10 +14,15 @@ } &__option { + box-sizing: border-box; + width: 100%; + height: auto; + min-height: 1.3rem; cursor: pointer; } &__option-item { + min-width: 0; pointer-events: none; } -} \ No newline at end of file +} diff --git a/frontend/scss/app/components/_popup_menu.scss b/frontend/scss/app/components/_popup_menu.scss index 19612b474..955ce39b4 100644 --- a/frontend/scss/app/components/_popup_menu.scss +++ b/frontend/scss/app/components/_popup_menu.scss @@ -8,6 +8,7 @@ max-width: min(24rem, calc(100vw - 1rem)); max-height: 70vh; max-height: min(70vh, calc(100vh - 1rem)); + box-sizing: border-box; padding: var(--padding-mini); border: 1px solid var(--border-color); border-radius: var(--border-radius); @@ -19,6 +20,11 @@ backdrop-filter: blur(6px); -webkit-backdrop-filter: blur(6px); + &--match-anchor { + width: var(--popup-anchor-width); + max-width: calc(100vw - 1rem); + } + ul { margin: 0; padding: 0; diff --git a/frontend/scss/app/components/_range_slider.scss b/frontend/scss/app/components/_range_slider.scss new file mode 100644 index 000000000..fd42a2274 --- /dev/null +++ b/frontend/scss/app/components/_range_slider.scss @@ -0,0 +1,94 @@ +.tp__range-slider { + display: grid; + grid-template-columns: minmax(0, 1fr) 3.25rem; + align-items: center; + gap: var(--gap-small); + width: 100%; + + &--disabled { + opacity: 0.6; + } + + &__range { + --tp-range-slider-fill: 0%; + + width: 100%; + height: 0.55rem; + margin: 0; + border-radius: var(--border-radius); + appearance: none; + -webkit-appearance: none; + background: linear-gradient( + to right, + var(--text-button-primary-background-color) 0%, + var(--text-button-primary-background-color) var(--tp-range-slider-fill), + var(--card-border-color) var(--tp-range-slider-fill), + var(--card-border-color) 100% + ); + cursor: pointer; + + &:disabled { + cursor: not-allowed; + } + } + + &__range::-webkit-slider-thumb { + width: 1.15rem; + height: 1.15rem; + border: 1px solid var(--card-border-color); + border-radius: 50%; + appearance: none; + -webkit-appearance: none; + background-color: var(--text-button-primary-color); + box-shadow: var(--toggle-switch-toggle-shadow); + } + + &__range::-moz-range-thumb { + width: 1.15rem; + height: 1.15rem; + border: 1px solid var(--card-border-color); + border-radius: 50%; + background-color: var(--text-button-primary-color); + box-shadow: var(--toggle-switch-toggle-shadow); + } + + &__range::-moz-range-track { + height: 0.55rem; + border-radius: var(--border-radius); + background: none; + } + + &__value { + box-sizing: border-box; + width: 3.25rem; + min-width: 3.25rem; + min-height: 2.5rem; + max-height: 2.5rem; + padding: var(--padding-small) var(--padding-mini); + border: 1px solid var(--input-border-color); + border-radius: var(--border-radius); + background-color: var(--input-background-color); + color: var(--modest-text-color); + font-family: inherit; + font-size: var(--font-size-md); + line-height: 1.5rem; + text-align: center; + appearance: textfield; + + &:focus { + border-color: var(--input-focus-border-color); + outline: none; + } + + &:disabled { + cursor: not-allowed; + } + + &::-webkit-inner-spin-button, + &::-webkit-outer-spin-button { + margin: 0; + appearance: none; + -webkit-appearance: none; + } + } +} diff --git a/frontend/scss/app/components/_text_button.scss b/frontend/scss/app/components/_text_button.scss index ef854fc97..360cd9c47 100644 --- a/frontend/scss/app/components/_text_button.scss +++ b/frontend/scss/app/components/_text_button.scss @@ -1,3 +1,18 @@ +// Touch browsers can retain :hover and :focus after a tap. Keep that +// highlight for mouse pointers, while preserving keyboard focus everywhere. +@mixin text-button-highlight { + &:focus-visible { + @content; + } + + @media (hover: hover) and (pointer: fine) { + &:focus, + &:hover { + @content; + } + } +} + .tp__text-button { display: flex; justify-content: center; @@ -28,8 +43,7 @@ pointer-events: none; } - &:focus, - &:hover { + @include text-button-highlight { background-color: var(--text-button-hover-background-color); color: var(--text-button-hover-color); outline: none; @@ -74,8 +88,7 @@ border-color: var(--text-button-primary-border-color); color: var(--text-button-primary-color); fill: var(--text-button-primary-color); - &:focus, - &:hover { + @include text-button-highlight { background-color: var(--text-button-primary-hover-background-color); color: var(--text-button-primary-hover-color); fill: var(--text-button-primary-hover-color); @@ -88,8 +101,7 @@ border-color: var(--text-button-secondary-border-color); color: var(--text-button-secondary-color); fill: var(--text-button-secondary-color); - &:focus, - &:hover { + @include text-button-highlight { background-color: var(--text-button-secondary-hover-background-color); color: var(--text-button-secondary-hover-color); fill: var(--text-button-secondary-hover-color); @@ -102,8 +114,7 @@ border-color: var(--text-button-tertiary-border-color); color: var(--text-button-tertiary-color); fill: var(--text-button-tertiary-color); - &:focus, - &:hover { + @include text-button-highlight { background-color: var(--text-button-tertiary-hover-background-color); color: var(--text-button-tertiary-hover-color); fill: var(--text-button-tertiary-hover-color); @@ -119,8 +130,7 @@ border-color: var(--text-button-active-border-color); color: var(--text-button-active-color); fill: var(--text-button-active-color); - &:focus, - &:hover { + @include text-button-highlight { background-color: var(--text-button-active-hover-background-color); border-color: var(--text-button-active-border-color); color: var(--text-button-active-hover-color); diff --git a/frontend/scss/app/components/config/_config_view.scss b/frontend/scss/app/components/config/_config_view.scss index cc268395c..349d83f10 100644 --- a/frontend/scss/app/components/config/_config_view.scss +++ b/frontend/scss/app/components/config/_config_view.scss @@ -149,65 +149,6 @@ } } -.tp__hls-repair-slider { - display: grid; - grid-template-columns: minmax(0, 1fr) 2.35rem; - align-items: center; - gap: var(--gap-small); - width: 100%; - - &__range { - --tp-hls-repair-slider-fill: 0%; - - width: 100%; - height: 0.55rem; - margin: 0; - border-radius: var(--border-radius); - appearance: none; - -webkit-appearance: none; - background: linear-gradient( - to right, - var(--text-button-primary-background-color) 0%, - var(--text-button-primary-background-color) var(--tp-hls-repair-slider-fill), - var(--card-border-color) var(--tp-hls-repair-slider-fill), - var(--card-border-color) 100% - ); - cursor: pointer; - } - - &__range::-webkit-slider-thumb { - width: 1.15rem; - height: 1.15rem; - border: 1px solid var(--card-border-color); - border-radius: 50%; - appearance: none; - -webkit-appearance: none; - background-color: var(--text-button-primary-color); - box-shadow: var(--toggle-switch-toggle-shadow); - } - - &__range::-moz-range-thumb { - width: 1.15rem; - height: 1.15rem; - border: 1px solid var(--card-border-color); - border-radius: 50%; - background-color: var(--text-button-primary-color); - box-shadow: var(--toggle-switch-toggle-shadow); - } - - &__range::-moz-range-track { - height: 0.55rem; - border-radius: var(--border-radius); - background: transparent; - } - - &__value { - box-sizing: border-box; - min-width: 2.35rem; - text-align: center; - } -} - .tp__schedules-config-view { &__body { display: flex; diff --git a/frontend/scss/app/components/playlist/_playlist_update_view.scss b/frontend/scss/app/components/playlist/_playlist_update_view.scss index 597c15b7d..d577aa437 100644 --- a/frontend/scss/app/components/playlist/_playlist_update_view.scss +++ b/frontend/scss/app/components/playlist/_playlist_update_view.scss @@ -1,10 +1,16 @@ +@use "../../../size" as size; + .tp__playlist-update-view { display: flex; flex-flow: column; width: 100%; - gap: var(--gap-default); + height: 100%; + min-height: 0; + gap: max(var(--gap-large), 1rem); padding: var(--padding-default) 0; box-sizing: border-box; + max-width: 100%; + overflow: hidden; &__header { display: flex; @@ -16,39 +22,427 @@ gap: var(--gap-default); } - &__body { + &__library-status { + padding: var(--gap-small) var(--gap-default); + border-left: 2px solid var(--tag-active-color); + background-color: var(--background-color); + color: var(--modest-text-color); + + &--error { + border-left-color: var(--error-color); + } + } + + &__cards { + display: flex; + flex: 1 1 auto; + flex-flow: column; + gap: max(var(--gap-large), 1rem); + box-sizing: border-box; + width: 100%; + min-height: 0; + overflow-y: auto; + overflow-x: hidden; + overscroll-behavior: contain; + -webkit-overflow-scrolling: touch; + } + + &__input-card { + box-sizing: border-box; + width: 100%; + max-width: 100%; + min-width: 0; + } + + &__input-card article { + display: flex; + flex-flow: column; + gap: max(var(--gap-large), 1rem); + min-width: 0; + } + + &__input-card-header { + display: flex; + align-items: center; + justify-content: space-between; + gap: var(--gap-default); + min-width: 0; + } + + &__input-identity { + display: flex; + align-items: baseline; + flex-flow: row wrap; + gap: max(var(--gap-large), 1rem); + min-width: 0; + + h2 { + margin: 0; + overflow-wrap: anywhere; + } + } + + &__input-type { + color: var(--modest-text-color); + text-transform: capitalize; + } + + &__input-card-body { + display: grid; + grid-template-columns: minmax(0, 1fr) 20rem; + gap: max(var(--gap-large), 1rem); + min-width: 0; + } + + &__input-content { + display: flex; + flex-flow: column; + gap: max(var(--gap-large), 1rem); + min-width: 0; + } + + &__pipeline-summary { + display: flex; + flex-flow: column; + gap: max(var(--gap-default), 0.5rem); + min-width: 0; + + h3 { + margin: 0; + font-size: inherit; + } + } + + &__cluster-chips { display: flex; flex-flow: row wrap; - gap: var(--gap-default); - box-sizing: border-box; - overflow-y: auto; - width: 100%; - -webkit-overflow-scrolling: touch; - touch-action: pan-y; - overscroll-behavior: contain; + gap: max(var(--gap-default), 0.5rem); + min-width: 0; } - &__log { - width: 100%; - box-sizing: border-box; - max-height: 320px; - min-height: 160px; - overflow-y: auto; - background-color: var(--card-background-color); - color: var(--modest-text-color); - font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, "Liberation Mono", monospace; - font-size: 0.85rem; - line-height: 1.35; - padding: var(--gap-default); - border-radius: var(--border-radius); + &__pipeline-status { + display: inline-flex; + align-items: center; + gap: var(--padding-mini); + width: fit-content; + max-width: 100%; + padding: var(--padding-micro) var(--padding-mini); border: 1px solid var(--border-color); - -webkit-overflow-scrolling: touch; - touch-action: pan-y; - overscroll-behavior: contain; + border-radius: var(--tag-border-radius); + color: var(--modest-text-color); + background: none; + font-size: var(--font-size-sm); + + // A fixed line box keeps the warning/checkmark glyphs from changing badge height. + line-height: 1.5; + font-weight: normal; + white-space: nowrap; + + &--accepted, + &--persisted-ok { + color: var(--success-color); + background-color: var(--success-background-color); + border-color: var(--ok-color); + } + + &--rejected { + color: var(--warn-color); + background-color: var(--warn-background-color); + border-color: var(--warn-color); + } + + &--cached, + &--updating { + color: var(--active-color); + background-color: var(--success-background-color); + border-color: var(--active-color); + } + + &--queued { + color: var(--tag-status-pending-color); + background-color: var(--tag-status-pending-background-color); + border-color: var(--tag-status-pending-border-color); + } + + &--error { + color: var(--error-color); + background-color: var(--error-background-color); + border-color: var(--error-color); + } + + &--neutral { + color: var(--modest-text-color); + border-color: var(--modest-text-color); + } } - &__log-line { - white-space: pre-wrap; + &__input-targets { + display: flex; + flex-flow: column; + gap: max(var(--gap-default), 0.5rem); + min-width: 0; + + h3 { + margin: 0; + font-size: inherit; + } + } + + &__target-toggles { + display: flex; + flex-flow: row wrap; + gap: max(var(--gap-default), 0.5rem); + min-width: 0; + } + + &__target-toggle { + max-width: 100%; + max-height: none; + min-width: 0; + + span:last-child { + min-width: 0; + overflow-wrap: anywhere; + } + + &.mixed { + border-style: dashed; + } + + &:focus-visible { + outline: 2px solid var(--input-focus-border-color); + outline-offset: 2px; + } + } + + &__empty-targets { + margin: 0; + color: var(--modest-text-color); + } + + &__input-actions { + display: flex; + flex-flow: column; + gap: max(var(--gap-large), 1rem); + min-width: 0; + padding-inline: var(--padding-large) calc(var(--padding-large) - var(--padding-default)); + border-inline-start: 1px solid var(--border-color); + } + + &__policy-control { + display: flex; + flex-flow: column; + gap: max(var(--gap-default), 0.5rem); + min-width: 0; + } + + &__action-label { + font-weight: bold; + } + + &__policy-select, + &__input-actions .tp__text-button { + width: 100%; + } + + &__policy-notes { + padding-block-start: max(var(--padding-mini), 0.5rem); + min-width: 0; + } + + &__policy-description { + margin: 0; + color: var(--modest-text-color); + font-size: var(--font-size-sm); + line-height: 1.35; overflow-wrap: anywhere; } -} \ No newline at end of file + + &__update-details { + min-width: 0; + border-block-start: 1px solid var(--border-color); + + summary { + width: fit-content; + padding-block: max(var(--padding-default), 0.75rem); + color: var(--text-color); + cursor: pointer; + font-weight: bold; + + &:focus-visible { + outline: 2px solid var(--input-focus-border-color); + outline-offset: 2px; + } + } + } + + &__update-details-body { + display: flex; + flex-flow: column; + gap: max(var(--gap-large), 1rem); + padding-block-end: var(--padding-default); + } + + &__run-context, + &__cluster-detail dl, + &__target-detail dl { + display: flex; + flex-flow: column; + gap: max(var(--gap-default), 0.5rem); + margin: 0; + } + + &__cluster-detail { + display: flex; + flex-flow: column; + gap: max(var(--gap-default), 0.5rem); + padding-inline-start: var(--gap-default); + border-inline-start: 2px solid var(--border-color); + + header { + display: flex; + align-items: center; + justify-content: space-between; + flex-flow: row wrap; + gap: var(--gap-default); + } + + h4 { + margin: 0; + } + } + + &__target-details { + display: flex; + flex-flow: column; + gap: max(var(--gap-default), 0.5rem); + padding-block-start: var(--gap-default); + border-block-start: 1px solid var(--border-color); + + > h4 { + margin: 0; + } + } + + &__target-details-list { + display: flex; + flex-flow: column; + gap: max(var(--gap-large), 1rem); + } + + &__target-detail { + display: flex; + flex-flow: column; + gap: max(var(--gap-default), 0.5rem); + min-width: 0; + padding-inline-start: var(--gap-default); + border-inline-start: 2px solid var(--border-color); + + header h5 { + margin: 0; + overflow-wrap: anywhere; + } + } + + &__target-processing-order { + display: flex; + align-items: center; + justify-content: space-between; + flex-flow: row wrap; + gap: var(--gap-default); + + > span { + color: var(--modest-text-color); + } + + .tp__playlist-processing { + justify-content: flex-start; + } + } + + &__target-pipeline-unavailable { + margin: 0; + color: var(--modest-text-color); + } + + &__detail-row { + display: grid; + grid-template-columns: minmax(8rem, 0.4fr) minmax(0, 1fr); + gap: var(--gap-default); + + dt { + color: var(--modest-text-color); + } + + dd { + min-width: 0; + margin: 0; + overflow-wrap: anywhere; + } + } + + &__progress-details { + display: flex; + flex-flow: column; + gap: max(var(--gap-default), 0.5rem); + padding-inline-start: var(--gap-default); + border-inline-start: 2px solid var(--border-color); + + h4 { + margin: 0; + } + + ul { + display: flex; + flex-flow: column; + gap: var(--padding-micro); + margin: 0; + padding-inline-start: var(--gap-default); + } + + li { + overflow-wrap: anywhere; + } + } + + &__input-card-footer { + display: flex; + flex-flow: row wrap; + gap: max(var(--gap-default), 0.5rem); + color: var(--modest-text-color); + font-size: var(--font-size-sm); + } + + &__last-update { + color: var(--text-color); + } +} + +@media (max-width: size.$mobile-breakpoint) { + .tp__playlist-update-view { + &__input-card { + padding: max(var(--padding-default), 0.75rem); + } + + &__input-card-body { + grid-template-columns: minmax(0, 1fr); + } + + &__input-actions { + width: 100%; + padding-block-start: max(var(--padding-default), 1rem); + padding-inline: 0; + border-block-start: 1px solid var(--border-color); + border-inline-start: 0; + } + + &__input-actions .tp__text-button { + width: 100%; + } + + &__detail-row { + grid-template-columns: minmax(0, 1fr); + gap: var(--padding-micro); + } + } +} diff --git a/frontend/src/app/components/config/reverse_proxy_config_view.rs b/frontend/src/app/components/config/reverse_proxy_config_view.rs index 864431f54..c51da871c 100644 --- a/frontend/src/app/components/config/reverse_proxy_config_view.rs +++ b/frontend/src/app/components/config/reverse_proxy_config_view.rs @@ -14,7 +14,8 @@ use crate::{ }, dto_field_id, number_input::NumberInput, - Card, Chip, DropDownOption, DropDownSelection, IconButton, RadioButtonGroup, Select, TextButton, + Card, Chip, DropDownOption, DropDownSelection, IconButton, RadioButtonGroup, RangeSlider, Select, + TextButton, }, context::ConfigContext, }, @@ -37,7 +38,6 @@ use shared::{ }; use std::{rc::Rc, str::FromStr}; use strum::IntoEnumIterator; -use web_sys::HtmlInputElement; use yew::prelude::*; const LABEL_CACHE: &str = "LABEL.CACHE"; @@ -1050,56 +1050,27 @@ pub fn ReverseProxyConfigView() -> Html { hls_cache_state.dispatch(HlsCacheConfigFormAction::SegmentRepair(segment_repair)); }) }; - let render_slider_control = |name: String, value: u8, max_value: u8, on_change: Callback| -> Html { - let value_string = value.to_string(); - let max_string = max_value.to_string(); - let fill = if max_value == 0 { 0 } else { (u16::from(value) * 100) / u16::from(max_value) }; - let slider_style = format!("--tp-hls-repair-slider-fill: {fill}%;"); - let oninput = Callback::from(move |event: InputEvent| { - let input: HtmlInputElement = event.target_unchecked_into(); - if let Ok(value) = input.value().parse::() { - on_change.emit(value.min(max_value)); - } - }); - html! { -
- - {value_string} -
- } - }; - let render_slider = - |label: String, info_key: &'static str, name: String, value: u8, max_value: u8, on_change: Callback| { - let control = render_slider_control(name, value, max_value, on_change); - config_field_child!(label, info_key, { - html! { - { control } - } - }) - }; let size_increase_editor = if let Some(size_increase_percent) = hls_segment_repair_size_increase_percent(&segment_repair) { let hls_cache_state = hls_cache_state.clone(); - render_slider( + let on_change = Callback::from(move |value| { + let mut segment_repair = hls_cache_state.form.segment_repair.clone(); + set_hls_segment_repair_size_increase_percent(&mut segment_repair, value); + hls_cache_state.dispatch(HlsCacheConfigFormAction::SegmentRepair(segment_repair)); + }); + config_field_child!( hls_segment_repair_size_increase_label(&translate, segment_repair.max_level), "HLS_CACHE_CONFIG.SEGMENT_REPAIR_SIZE_INCREASE", - "hls_segment_repair_size_increase".to_string(), - size_increase_percent, - 100, - Callback::from(move |value| { - let mut segment_repair = hls_cache_state.form.segment_repair.clone(); - set_hls_segment_repair_size_increase_percent(&mut segment_repair, value); - hls_cache_state.dispatch(HlsCacheConfigFormAction::SegmentRepair(segment_repair)); - }), + { + html! { + + } + } ) } else { config_field_child!( @@ -1625,4 +1596,36 @@ mod tests { assert_eq!(parsed, policy); } } + + #[test] + fn range_slider_value_updates_only_the_active_segment_repair_mode() { + for (mode, expected) in [ + (HlsSegmentRepairMode::Off, (11, 22, 33)), + (HlsSegmentRepairMode::Low, (100, 22, 33)), + (HlsSegmentRepairMode::Medium, (11, 100, 33)), + (HlsSegmentRepairMode::High, (11, 22, 100)), + ] { + let mut segment_repair = HlsSegmentRepairConfigDto { + max_level: mode, + size_increase: shared::model::HlsSegmentRepairSizeIncreaseConfigDto { + low_percent: 11, + medium_percent: 22, + high_percent: 33, + }, + ..HlsSegmentRepairConfigDto::default() + }; + + set_hls_segment_repair_size_increase_percent(&mut segment_repair, 100); + + assert_eq!( + ( + segment_repair.size_increase.low_percent, + segment_repair.size_increase.medium_percent, + segment_repair.size_increase.high_percent, + ), + expected, + "mode: {mode:?}" + ); + } + } } diff --git a/frontend/src/app/components/confirm_dialog.rs b/frontend/src/app/components/confirm_dialog.rs index d3fe6730d..8f0df66de 100644 --- a/frontend/src/app/components/confirm_dialog.rs +++ b/frontend/src/app/components/confirm_dialog.rs @@ -44,17 +44,6 @@ pub fn ConfirmDialog(props: &ConfirmDialogProps) -> Html { Callback::from(move |()| on_result(DialogResult::Cancel)) }; - let title_html: Html = props - .title - .split('\n') - .map(|line| { - html! { <> - { line } -
- } - }) - .collect::(); - html! { Html { close_on_backdrop_click={props.close_on_backdrop_click} on_close={Some(on_close)} > -

{ title_html }

+ {confirm_dialog_message(&props.title)}
@@ -72,3 +61,68 @@ pub fn ConfirmDialog(props: &ConfirmDialogProps) -> Html { } } + +fn confirm_dialog_message(message: &str) -> Html { + html! {

{message.to_owned()}

} +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn confirm_dialog_renders_short_multiline_and_long_messages_as_body_text() { + for message in [ + "Delete this item?".to_owned(), + "First line\nSecond line\n".to_owned(), + "A long confirmation message. ".repeat(30), + "".to_owned(), + ] { + let rendered = confirm_dialog_message(&message); + let Html::VTag(paragraph) = &rendered else { + panic!("confirmation message must be a paragraph"); + }; + assert_eq!(paragraph.tag(), "p"); + assert_eq!(rendered, html! {

{message}

}); + } + } + + #[test] + fn confirm_dialog_uses_standard_desktop_padding_and_preserves_mobile_spacing() { + let styles = include_str!("../../../scss/app/components/_custom_dialog.scss"); + let native_dialog = include_str!("../../../scss/app/components/_dialog.scss"); + let confirmation = styles.split("&.tp__confirm-dialog {").nth(1).expect("confirmation styles"); + let confirmation = confirmation.split("&.tp__content-dialog {").next().expect("confirmation styles"); + + let (desktop, mobile_and_body) = + confirmation.split_once("@media (max-width: size.$mobile-breakpoint)").expect("existing mobile breakpoint"); + assert!(desktop.contains("--dialog-padding: var(--padding-default);")); + assert!(!desktop.contains("--padding-larger")); + let mobile = mobile_and_body.split('}').next().expect("mobile spacing"); + assert!(mobile.contains("--dialog-padding: var(--padding-larger);")); + + for shared_spacing in ["padding: var(--dialog-padding);", "gap: var(--gap-large);"] { + assert!(native_dialog.contains(shared_spacing)); + assert!(confirmation.contains(shared_spacing)); + } + assert!(confirmation.contains("max-width: min(size.$confirm-dialog-max-width, 90vw);")); + assert!(confirmation.contains("box-sizing: border-box;")); + assert!(confirmation.contains("font-size: var(--font-size-norm);")); + assert!(confirmation.contains("white-space: pre-wrap;")); + assert!(confirmation.contains("overflow-y: auto;")); + assert!(confirmation.contains("flex-shrink: 0;")); + } + + #[test] + fn confirm_dialog_toolbar_spacing_matches_each_shared_dialog_padding() { + let styles = include_str!("../../../scss/app/components/_custom_dialog.scss"); + let native_dialog = include_str!("../../../scss/app/components/_dialog.scss"); + + assert!(styles.contains(".tp__custom-dialog {\n --dialog-padding: var(--padding-default);")); + assert!(styles.contains("&.tp__confirm-dialog {\n --dialog-padding: var(--padding-default);")); + assert_eq!(styles.matches("padding: var(--dialog-padding);").count(), 2); + assert!(native_dialog.contains("--dialog-padding: var(--padding-larger);")); + assert!(native_dialog.contains("padding: var(--dialog-padding);")); + assert!(styles.contains("padding-top: var(--dialog-padding, var(--padding-small));")); + } +} diff --git a/frontend/src/app/components/drop_down_icon_button.rs b/frontend/src/app/components/drop_down_icon_button.rs index dcd433fb4..e3c8cadc5 100644 --- a/frontend/src/app/components/drop_down_icon_button.rs +++ b/frontend/src/app/components/drop_down_icon_button.rs @@ -1,5 +1,9 @@ use crate::{ - app::components::{button_utils::prevent_default_and_stop, popup_menu::PopupMenu, AppIcon, IconButton}, + app::components::{ + button_utils::prevent_default_and_stop, + popup_menu::{PopupMenu, PopupMenuPlacement, PopupMenuWidth}, + AppIcon, IconButton, + }, html_if, }; use std::{collections::HashSet, rc::Rc}; @@ -38,6 +42,12 @@ pub struct DropDownIconButtonProps { #[prop_or_default] pub button_ref: Option, #[prop_or_default] + pub popup_width: PopupMenuWidth, + #[prop_or_default] + pub popup_placement: PopupMenuPlacement, + #[prop_or_default] + pub width_anchor_ref: Option, + #[prop_or_default] pub aria_label: Option, #[prop_or_default] pub aria_required: Option, @@ -64,8 +74,12 @@ pub fn DropDownIconButton(props: &DropDownIconButtonProps) -> Html { let handle_popup_close = { let set_is_open = popup_is_open.clone(); + let button_ref = button_ref.clone(); Callback::from(move |()| { set_is_open.set(false); + if let Some(button) = button_ref.cast::() { + let _ = button.focus(); + } }) }; @@ -87,6 +101,7 @@ pub fn DropDownIconButton(props: &DropDownIconButtonProps) -> Html { let selections = selections.clone(); let onselect = props.on_select.clone(); let set_is_open = popup_is_open.clone(); + let button_ref = button_ref.clone(); Callback::from(move |(id, _event): (String, MouseEvent)| { let selected_options = if multi_select { if selections.current().contains(&id) { @@ -107,6 +122,9 @@ pub fn DropDownIconButton(props: &DropDownIconButtonProps) -> Html { onselect.emit((name.clone(), selected_options)); if !multi_select { set_is_open.set(false); + if let Some(button) = button_ref.cast::() { + let _ = button.focus(); + } } }) }; @@ -121,7 +139,9 @@ pub fn DropDownIconButton(props: &DropDownIconButtonProps) -> Html { aria_required={props.aria_required} aria_invalid={props.aria_invalid} aria_describedby={props.aria_describedby.clone()} /> - + { for props.options.iter().map(|o| { let checkbox_id = o.id.clone(); @@ -131,7 +151,7 @@ pub fn DropDownIconButton(props: &DropDownIconButtonProps) -> Html { move |event| checkbox_handler.emit((id.clone(), event)) }); html! { -
+
+ }}) } diff --git a/frontend/src/app/components/home.rs b/frontend/src/app/components/home.rs index f3a8f0637..e91751dbf 100644 --- a/frontend/src/app/components/home.rs +++ b/frontend/src/app/components/home.rs @@ -219,7 +219,7 @@ pub fn Home() -> Html { ToastOptions { close_mode: ToastCloseMode::Manual }, ); } - EventMessage::PlaylistUpdate(update_state) => match update_state { + EventMessage::PlaylistUpdate(update) => match update.state { PlaylistUpdateState::Success => { services_ctx_clone.toastr.success(translate_clone.t("MESSAGES.PLAYLIST_UPDATE.SUCCESS_FINISH")); } diff --git a/frontend/src/app/components/mod.rs b/frontend/src/app/components/mod.rs index 33f0d7bac..cd4ac86d3 100644 --- a/frontend/src/app/components/mod.rs +++ b/frontend/src/app/components/mod.rs @@ -39,6 +39,7 @@ mod panel; mod playlist; mod popup_menu; mod radio_button_group; +mod range_slider; mod rbac; mod reveal_content; mod role_based_content; @@ -83,9 +84,10 @@ pub(crate) use self::{ date_input_action::*, datetime_input::*, downloads::DownloadsView, drop_down_icon_button::*, error_boundary::*, field_explanation::*, field_id::*, field_wrapper::*, filter::*, health_banner::*, hide_content::*, home::*, icon_button::*, key_value_editor::*, language_picker::*, loading_indicator::*, loading_screen::*, login::*, - no_access::*, no_content::*, panel::*, particle_flow_background::*, playlist::*, radio_button_group::*, rbac::*, - reveal_content::*, role_based_content::*, search::*, select::*, select_helpers::*, setup::*, sidebar::*, - source_editor::*, svg_icon::*, table::*, tabset::*, tag_list::*, task_status_badge::*, text_button::*, textarea::*, - theme_picker::*, title_card::*, toastr::*, toggle_switch::*, userlist::*, websocket_status::*, + no_access::*, no_content::*, panel::*, particle_flow_background::*, playlist::*, radio_button_group::*, + range_slider::*, rbac::*, reveal_content::*, role_based_content::*, search::*, select::*, select_helpers::*, + setup::*, sidebar::*, source_editor::*, svg_icon::*, table::*, tabset::*, tag_list::*, task_status_badge::*, + text_button::*, textarea::*, theme_picker::*, title_card::*, toastr::*, toggle_switch::*, userlist::*, + websocket_status::*, }; pub use self::{confirm_dialog::*, content_dialog::*}; diff --git a/frontend/src/app/components/playlist/input/input_type_view.rs b/frontend/src/app/components/playlist/input/input_type_view.rs index 81b0d36fe..d7dcc3432 100644 --- a/frontend/src/app/components/playlist/input/input_type_view.rs +++ b/frontend/src/app/components/playlist/input/input_type_view.rs @@ -7,11 +7,9 @@ pub struct InputTypeViewProps { pub input_type: InputType, } -#[component] -pub fn InputTypeView(props: &InputTypeViewProps) -> Html { - let translate = use_translation(); - - let label = match props.input_type { +#[must_use] +pub const fn input_type_label_key(input_type: InputType) -> &'static str { + match input_type { InputType::M3u => "LABEL.M3U", InputType::Xtream => "LABEL.XTREAM", InputType::M3uBatch => "LABEL.M3U_BATCH", @@ -23,9 +21,47 @@ pub fn InputTypeView(props: &InputTypeViewProps) -> Html { InputType::Jellyfin => "LABEL.JELLYFIN", InputType::Plex => "LABEL.PLEX", InputType::Staged => "LABEL.STAGED", - }; - - html! { - + } +} + +#[component] +pub fn InputTypeView(props: &InputTypeViewProps) -> Html { + let translate = use_translation(); + + html! { + + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn every_input_type_label_exists_in_each_supported_locale() { + let input_types = [ + InputType::M3u, + InputType::Xtream, + InputType::M3uBatch, + InputType::XtreamBatch, + InputType::Stalker, + InputType::StalkerBatch, + InputType::Library, + InputType::Emby, + InputType::Jellyfin, + InputType::Plex, + InputType::Staged, + ]; + for locale in [ + include_str!("../../../../../public/assets/i18n/en.json"), + include_str!("../../../../../public/assets/i18n/ar.json"), + include_str!("../../../../../public/assets/i18n/ru.json"), + ] { + let translations: serde_json::Value = serde_json::from_str(locale).unwrap(); + for input_type in input_types { + let pointer = format!("/{}", input_type_label_key(input_type).replace('.', "/")); + assert!(translations.pointer(&pointer).and_then(serde_json::Value::as_str).is_some(), "{pointer}"); + } + } } } diff --git a/frontend/src/app/components/playlist/input_update_card.rs b/frontend/src/app/components/playlist/input_update_card.rs new file mode 100644 index 000000000..9541e5cb8 --- /dev/null +++ b/frontend/src/app/components/playlist/input_update_card.rs @@ -0,0 +1,2369 @@ +use super::{ + input_type_label_key, + processing::{processing_stages, PlaylistProcessingStage}, +}; +use crate::{ + app::components::{ + popup_menu::{PopupMenuPlacement, PopupMenuWidth}, + Card, DropDownOption, DropDownSelection, PlaylistProcessing, Select, TextButton, TitledCard, + }, + error::Error, + hooks::use_service_context, + i18n::use_translation, + model::{ + last_update_view_model, DialogResult, InputClusterOutcomeView, InputClusterRunView, InputDataSourceView, + InputUpdateCapabilities, InputUpdateCapabilitiesExt, InputUpdateCardModel, InputUpdateCardState, + InputUpdateCardStatus, InputUpdateRunView, LastUpdateRelative, PlaylistUpdateAcceptedScope, RuleStageSummary, + TargetPipelineView, TargetRunView, + }, + services::{DialogService, WebSocketConnectionContext}, + utils::content_cluster_presentation, +}; +#[cfg(not(target_arch = "wasm32"))] +use chrono::{Local, TimeZone}; +use shared::model::{ + InputRefreshPolicy, OperationRunAccepted, PersistedPlaylistUpdateClusterState, + PersistedPlaylistUpdateTechnicalState, PlaylistUpdateProgressDetail, PlaylistUpdateRunId, XtreamCluster, +}; +use std::rc::Rc; +#[cfg(target_arch = "wasm32")] +use wasm_bindgen::JsValue; +use yew::{platform::spawn_local, prelude::*}; + +const POLICY_NORMAL: &str = "normal"; +const POLICY_REFRESH: &str = "refresh"; +const POLICY_FORCE: &str = "force"; + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum TargetSelectionState { + Empty, + Mixed, + All, +} + +impl TargetSelectionState { + const fn aria_pressed(self) -> &'static str { + match self { + Self::Empty => "false", + Self::Mixed => "mixed", + Self::All => "true", + } + } +} + +#[derive(Clone, Debug, PartialEq, Eq)] +enum PlaylistUpdateActionOutcome { + Accepted(Option), + Conflict, + Failed, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum ConfirmationSubmissionDecision { + Continue, + Cancel, + IgnoreStale, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +struct InputUpdateCardInteractionState { + controls: InputUpdateCardState, + submitting_for: Option, +} + +impl InputUpdateCardInteractionState { + fn from_model(model: &InputUpdateCardModel) -> Self { + Self { + controls: InputUpdateCardState { + selected_target_ids: model.targets.iter().map(|target| target.id).collect(), + policy: InputRefreshPolicy::NORMAL, + }, + submitting_for: None, + } + } + + const fn is_submitting(&self) -> bool { self.submitting_for.is_some() } + + fn is_submitting_for(&self, connection_context: WebSocketConnectionContext) -> bool { + self.submitting_for == Some(connection_context) + } + + // Normalize the current card selection independently of the cloned request_state. + // Without provider policies, NORMAL is inert; capabilities still gate every action. + fn policy_for_capabilities(&self, capabilities: InputUpdateCapabilities) -> InputRefreshPolicy { + if capabilities.supports(self.controls.policy) { + self.controls.policy + } else { + capabilities.policies().first().copied().unwrap_or(InputRefreshPolicy::NORMAL) + } + } +} + +enum InputUpdateCardAction { + ToggleTarget(u16), + ToggleAll(Vec), + SelectPolicy(InputRefreshPolicy), + SyncPolicy(InputUpdateCapabilities), + BeginSubmission(WebSocketConnectionContext), + CancelSubmission(WebSocketConnectionContext), + ConnectionChanged, + FinishSubmission { connection_context: WebSocketConnectionContext, outcome: PlaylistUpdateActionOutcome }, +} + +fn target_selection_state(selected_target_ids: &[u16], target_ids: &[u16]) -> TargetSelectionState { + if target_ids.is_empty() { + return TargetSelectionState::Empty; + } + let selected_count = target_ids.iter().filter(|target_id| selected_target_ids.contains(target_id)).count(); + match selected_count { + 0 => TargetSelectionState::Empty, + count if count == target_ids.len() => TargetSelectionState::All, + _ => TargetSelectionState::Mixed, + } +} + +fn reduce_input_update_card_state( + mut state: InputUpdateCardInteractionState, + action: InputUpdateCardAction, +) -> InputUpdateCardInteractionState { + match action { + InputUpdateCardAction::ToggleTarget(target_id) if !state.is_submitting() => { + if let Some(index) = state.controls.selected_target_ids.iter().position(|selected| *selected == target_id) { + state.controls.selected_target_ids.remove(index); + } else { + state.controls.selected_target_ids.push(target_id); + } + } + InputUpdateCardAction::ToggleAll(target_ids) if !state.is_submitting() => { + state.controls.selected_target_ids = if target_selection_state( + &state.controls.selected_target_ids, + &target_ids, + ) == TargetSelectionState::All + { + Vec::new() + } else { + target_ids + }; + } + InputUpdateCardAction::SelectPolicy(policy) if !state.is_submitting() => state.controls.policy = policy, + InputUpdateCardAction::SyncPolicy(capabilities) => { + state.controls.policy = state.policy_for_capabilities(capabilities); + } + InputUpdateCardAction::BeginSubmission(connection_context) if !state.is_submitting() => { + state.submitting_for = Some(connection_context); + } + InputUpdateCardAction::CancelSubmission(connection_context) if state.is_submitting_for(connection_context) => { + state.submitting_for = None; + } + InputUpdateCardAction::ConnectionChanged => state.submitting_for = None, + InputUpdateCardAction::FinishSubmission { connection_context, outcome } + if state.is_submitting_for(connection_context) => + { + state.submitting_for = None; + if matches!(outcome, PlaylistUpdateActionOutcome::Accepted(_)) { + state.controls.policy = InputRefreshPolicy::NORMAL; + } + } + InputUpdateCardAction::ToggleTarget(_) + | InputUpdateCardAction::ToggleAll(_) + | InputUpdateCardAction::SelectPolicy(_) + | InputUpdateCardAction::BeginSubmission(_) + | InputUpdateCardAction::CancelSubmission(_) + | InputUpdateCardAction::FinishSubmission { .. } => {} + } + state +} + +impl Reducible for InputUpdateCardInteractionState { + type Action = InputUpdateCardAction; + + fn reduce(self: Rc, action: Self::Action) -> Rc { + Rc::new(reduce_input_update_card_state((*self).clone(), action)) + } +} + +fn policy_from_selection(selection: &DropDownSelection) -> Option { + let DropDownSelection::Single(policy) = selection else { + return None; + }; + match policy.as_str() { + POLICY_NORMAL => Some(InputRefreshPolicy::NORMAL), + POLICY_REFRESH => Some(InputRefreshPolicy::REFRESH), + POLICY_FORCE => Some(InputRefreshPolicy::FORCE), + _ => None, + } +} + +const fn policy_id(policy: InputRefreshPolicy) -> &'static str { + if policy.bypasses_quality() { + POLICY_FORCE + } else if policy.bypasses_cache() { + POLICY_REFRESH + } else { + POLICY_NORMAL + } +} + +const fn policy_label_key(policy: InputRefreshPolicy) -> &'static str { + if policy.bypasses_quality() { + "LABEL.FORCE_UPDATE" + } else if policy.bypasses_cache() { + "LABEL.REFRESH" + } else { + "LABEL.UPDATE" + } +} + +const fn start_button_label_key(policy: InputRefreshPolicy) -> &'static str { + if policy.bypasses_quality() { + "LABEL.FORCE_UPDATE_START" + } else if policy.bypasses_cache() { + "LABEL.REFRESH_START" + } else { + "LABEL.UPDATE_START" + } +} + +const fn requires_confirmation(policy: InputRefreshPolicy) -> bool { policy.bypasses_quality() } + +fn confirmation_allows_submission(policy: InputRefreshPolicy, result: Option) -> bool { + !requires_confirmation(policy) || result == Some(DialogResult::Ok) +} + +fn confirmation_submission_decision( + request_connection_context: WebSocketConnectionContext, + current_connection_context: WebSocketConnectionContext, + policy: InputRefreshPolicy, + result: Option, +) -> ConfirmationSubmissionDecision { + if !request_connection_context.is_same_live_connection(current_connection_context) { + ConfirmationSubmissionDecision::IgnoreStale + } else if confirmation_allows_submission(policy, result) { + ConfirmationSubmissionDecision::Continue + } else { + ConfirmationSubmissionDecision::Cancel + } +} + +fn classify_update_outcome(result: Result) -> PlaylistUpdateActionOutcome { + match result { + Ok(accepted) => PlaylistUpdateActionOutcome::Accepted(accepted.run_id), + Err(Error::Conflict(_)) => PlaylistUpdateActionOutcome::Conflict, + Err(_) => PlaylistUpdateActionOutcome::Failed, + } +} + +fn current_submission_outcome( + request_connection_context: WebSocketConnectionContext, + current_connection_context: WebSocketConnectionContext, + outcome: PlaylistUpdateActionOutcome, +) -> Option { + request_connection_context.is_same_live_connection(current_connection_context).then_some(outcome) +} + +fn selected_current_target_ids(model: &InputUpdateCardModel, selected_target_ids: &[u16]) -> Vec { + model.targets.iter().map(|target| target.id).filter(|target_id| selected_target_ids.contains(target_id)).collect() +} + +fn can_start_update(state: &InputUpdateCardInteractionState, model: &InputUpdateCardModel, can_submit: bool) -> bool { + can_submit + && model.capabilities.supports_action(model.capabilities.selected_action(state.controls.policy)) + && (model.capabilities != InputUpdateCapabilities::Rescan + || state.controls.policy == InputRefreshPolicy::NORMAL) + && !state.is_submitting() + && !selected_current_target_ids(model, &state.controls.selected_target_ids).is_empty() +} + +fn action_message(template: String, input_name: &str, action: &str) -> String { + template.replace("{input}", input_name).replace("{action}", action) +} + +fn count_message(template: String, count: impl std::fmt::Display) -> String { + template.replace("{count}", &count.to_string()) +} + +const fn progress_detail_label_key(detail: PlaylistUpdateProgressDetail) -> &'static str { + match detail { + PlaylistUpdateProgressDetail::InputCompletedSuccessfully => { + "MESSAGES.PLAYLIST_UPDATE.DETAIL_INPUT_COMPLETED_SUCCESSFULLY" + } + PlaylistUpdateProgressDetail::InputCompletedPartially => { + "MESSAGES.PLAYLIST_UPDATE.DETAIL_INPUT_COMPLETED_PARTIALLY" + } + PlaylistUpdateProgressDetail::InputFailed => "MESSAGES.PLAYLIST_UPDATE.DETAIL_INPUT_FAILED", + PlaylistUpdateProgressDetail::ForcedUpdateRequested => { + "MESSAGES.PLAYLIST_UPDATE.DETAIL_FORCED_UPDATE_REQUESTED" + } + PlaylistUpdateProgressDetail::LibraryRescanStarted => "MESSAGES.PLAYLIST_UPDATE.DETAIL_LIBRARY_RESCAN_STARTED", + PlaylistUpdateProgressDetail::LibraryScanResult => "MESSAGES.PLAYLIST_UPDATE.DETAIL_LIBRARY_SCAN_RESULT", + PlaylistUpdateProgressDetail::LibraryRescanCompleted => { + "MESSAGES.PLAYLIST_UPDATE.DETAIL_LIBRARY_RESCAN_COMPLETED" + } + PlaylistUpdateProgressDetail::LibraryRescanFailed => "MESSAGES.PLAYLIST_UPDATE.DETAIL_LIBRARY_RESCAN_FAILED", + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum ClusterSummaryStatus { + Accepted, + Rejected, + Cached, + TechnicalError, + PersistedOk, + PersistedFailed, + NotRequested, + Unavailable, +} + +impl ClusterSummaryStatus { + const fn label_key(self) -> &'static str { + match self { + Self::Accepted => "MESSAGES.PLAYLIST_UPDATE.CLUSTER_ACCEPTED", + Self::Rejected => "MESSAGES.PLAYLIST_UPDATE.CLUSTER_REJECTED", + Self::Cached => "MESSAGES.PLAYLIST_UPDATE.CLUSTER_CACHED", + Self::TechnicalError | Self::PersistedFailed => "LABEL.UPDATE_STATUS_FAILED", + Self::PersistedOk => "LABEL.UPDATE_STATUS_SUCCESS", + Self::NotRequested => "MESSAGES.PLAYLIST_UPDATE.CLUSTER_NOT_REQUESTED", + Self::Unavailable => "MESSAGES.PLAYLIST_UPDATE.DATA_UNAVAILABLE", + } + } + + const fn symbol(self) -> &'static str { + match self { + Self::Accepted => "✓", + Self::Rejected => "⚠", + Self::Cached => "●", + Self::TechnicalError | Self::PersistedFailed => "✕", + Self::PersistedOk => "✓", + Self::NotRequested | Self::Unavailable => "–", + } + } + + const fn modifier(self) -> &'static str { + match self { + Self::Accepted => "tp__playlist-update-view__pipeline-status--accepted", + Self::Rejected => "tp__playlist-update-view__pipeline-status--rejected", + Self::Cached => "tp__playlist-update-view__pipeline-status--cached", + Self::TechnicalError | Self::PersistedFailed => "tp__playlist-update-view__pipeline-status--error", + Self::PersistedOk => "tp__playlist-update-view__pipeline-status--persisted-ok", + Self::NotRequested | Self::Unavailable => "tp__playlist-update-view__pipeline-status--neutral", + } + } +} + +fn cluster_summary_status(cluster: &InputClusterRunView) -> ClusterSummaryStatus { + if !cluster.requested.unwrap_or(cluster.enabled_by_configuration) { + return ClusterSummaryStatus::NotRequested; + } + if cluster.technical_state == Some(PersistedPlaylistUpdateTechnicalState::Failed) { + return ClusterSummaryStatus::TechnicalError; + } + if cluster.outcome == Some(InputClusterOutcomeView::Rejected) { + return ClusterSummaryStatus::Rejected; + } + if cluster.persisted_status.is_some_and(|persisted| { + persisted.last_update.is_some() + && persisted.status == PersistedPlaylistUpdateClusterState::Failed + && persisted.last_update.is_some_and(|snapshot| snapshot.technical_state.is_none()) + }) { + return ClusterSummaryStatus::PersistedFailed; + } + match (cluster.outcome, cluster.source) { + (Some(InputClusterOutcomeView::TechnicalError), _) => ClusterSummaryStatus::TechnicalError, + (Some(InputClusterOutcomeView::Rejected), _) => ClusterSummaryStatus::Rejected, + (_, Some(InputDataSourceView::Cache)) => ClusterSummaryStatus::Cached, + (Some(InputClusterOutcomeView::Accepted), _) => ClusterSummaryStatus::Accepted, + (None, Some(InputDataSourceView::Provider)) + if cluster.technical_state == Some(PersistedPlaylistUpdateTechnicalState::Succeeded) => + { + ClusterSummaryStatus::Accepted + } + (None, Some(InputDataSourceView::Provider)) => ClusterSummaryStatus::Unavailable, + (None, None) => match cluster.persisted_status.map(|persisted| persisted.status) { + Some(PersistedPlaylistUpdateClusterState::Ok) => ClusterSummaryStatus::PersistedOk, + Some(PersistedPlaylistUpdateClusterState::Failed) => ClusterSummaryStatus::PersistedFailed, + None => ClusterSummaryStatus::Unavailable, + }, + } +} + +const fn cluster_label_key(cluster: XtreamCluster) -> &'static str { content_cluster_presentation(cluster).label_key } + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum QualityPresentation { + Bypassed, + Disabled, + Unavailable, + ConfiguredThreshold { threshold: u8, identical_population_required: bool }, + Evaluated { threshold: u8, quality: u8, identical_population_required: bool }, +} + +fn quality_presentation( + cluster: &InputClusterRunView, + policy: Option, +) -> Option { + if !cluster.requested.unwrap_or(cluster.enabled_by_configuration) { + return None; + } + if cluster.persisted_status.is_some_and(|persisted| persisted.last_update.is_none()) { + return Some(cluster.threshold.map_or(QualityPresentation::Disabled, |threshold| { + QualityPresentation::ConfiguredThreshold { threshold, identical_population_required: threshold == 100 } + })); + } + let Some(threshold) = cluster.threshold else { + return Some(QualityPresentation::Disabled); + }; + if let Some(quality) = cluster.quality { + return Some(QualityPresentation::Evaluated { + threshold, + quality, + identical_population_required: threshold == 100, + }); + } + if cluster + .persisted_status + .and_then(|persisted| persisted.last_update) + .is_some_and(|snapshot| snapshot.quality.is_some()) + { + return Some(QualityPresentation::ConfiguredThreshold { + threshold, + identical_population_required: threshold == 100, + }); + } + if policy.is_some_and(InputRefreshPolicy::bypasses_quality) { + return Some(QualityPresentation::Bypassed); + } + Some(QualityPresentation::Unavailable) +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum CachePresentation { + Used, + Provider, + BypassedRefresh, + BypassedForce, +} + +impl CachePresentation { + const fn label_key(self) -> &'static str { + match self { + Self::Used => "MESSAGES.PLAYLIST_UPDATE.CACHE_USED", + Self::Provider => "MESSAGES.PLAYLIST_UPDATE.CACHE_NOT_USED", + Self::BypassedRefresh => "MESSAGES.PLAYLIST_UPDATE.CACHE_BYPASSED_REFRESH", + Self::BypassedForce => "MESSAGES.PLAYLIST_UPDATE.CACHE_BYPASSED_FORCE", + } + } +} + +fn cache_presentation( + policy: Option, + source: Option, +) -> Option { + Some(match source? { + InputDataSourceView::Cache => CachePresentation::Used, + InputDataSourceView::Provider if policy.is_some_and(InputRefreshPolicy::bypasses_quality) => { + CachePresentation::BypassedForce + } + InputDataSourceView::Provider if policy.is_some_and(InputRefreshPolicy::bypasses_cache) => { + CachePresentation::BypassedRefresh + } + InputDataSourceView::Provider => CachePresentation::Provider, + }) +} + +const fn source_label_key(source: InputDataSourceView) -> &'static str { + match source { + InputDataSourceView::Cache => "LABEL.CACHE", + InputDataSourceView::Provider => "LABEL.PROVIDER", + } +} + +const fn input_status_symbol(status: InputUpdateCardStatus) -> &'static str { + match status { + InputUpdateCardStatus::Ready => "–", + InputUpdateCardStatus::Queued | InputUpdateCardStatus::Updating => "●", + InputUpdateCardStatus::Success => "✓", + InputUpdateCardStatus::Partial => "⚠", + InputUpdateCardStatus::Failed => "✕", + } +} + +fn card_status_badge(view: &InputUpdateRunView, translate: impl Fn(&str) -> String) -> Html { + // A persisted input result does not imply a completed target publication. + // Only a correlated run supplies the current action status. + let status = if view.run_id.is_none() && view.input_state != InputUpdateCardStatus::Ready { + view.input_state + } else { + view.overall_state + }; + html! { + + {translate(status.label_key())} + + + } +} + +fn detail_row(label: String, value: String) -> Html { + html! { +
+
{label}
+
{value}
+
+ } +} + +fn quality_details( + cluster: &InputClusterRunView, + policy: Option, + translate: &impl Fn(&str) -> String, +) -> Html { + match quality_presentation(cluster, policy) { + None => Html::default(), + Some(QualityPresentation::Bypassed) => detail_row( + translate("MESSAGES.PLAYLIST_UPDATE.QUALITY_GUARD"), + translate("MESSAGES.PLAYLIST_UPDATE.QUALITY_BYPASSED"), + ), + Some(QualityPresentation::Disabled) => { + detail_row(translate("MESSAGES.PLAYLIST_UPDATE.QUALITY_GUARD"), translate("LABEL.DISABLED")) + } + Some(QualityPresentation::Unavailable) => detail_row( + translate("MESSAGES.PLAYLIST_UPDATE.QUALITY_GUARD"), + translate("MESSAGES.PLAYLIST_UPDATE.DATA_UNAVAILABLE"), + ), + Some(QualityPresentation::ConfiguredThreshold { threshold, identical_population_required }) => { + quality_threshold_details(threshold, identical_population_required, translate) + } + Some(QualityPresentation::Evaluated { threshold, quality, identical_population_required }) => html! { + <> + {quality_threshold_details(threshold, identical_population_required, translate)} + {detail_row( + translate("MESSAGES.PLAYLIST_UPDATE.QUALITY_ACHIEVED"), + format!("{quality}%"), + )} + + }, + } +} + +fn quality_threshold_details( + threshold: u8, + identical_population_required: bool, + translate: &impl Fn(&str) -> String, +) -> Html { + html! { + <> + {detail_row( + translate("MESSAGES.PLAYLIST_UPDATE.QUALITY_THRESHOLD"), + format!("{threshold}%"), + )} + if identical_population_required { + {detail_row( + translate("MESSAGES.PLAYLIST_UPDATE.REQUIRED"), + translate("MESSAGES.PLAYLIST_UPDATE.IDENTICAL_POPULATION"), + )} + } + + } +} + +fn cache_details( + source: Option, + policy: Option, + translate: &impl Fn(&str) -> String, +) -> Html { + html! { + <> + if let Some(source) = source { + {detail_row(translate("LABEL.SOURCE"), translate(source_label_key(source)))} + } + if let Some(cache) = cache_presentation(policy, source) { + {detail_row(translate("LABEL.CACHE"), translate(cache.label_key()))} + } + + } +} + +fn cluster_details( + input_id: u16, + cluster: &InputClusterRunView, + policy: Option, + translate: &impl Fn(&str) -> String, +) -> Html { + let status = cluster_summary_status(cluster); + let heading_id = format!("input-update-card-{input_id}-{}-details", cluster.cluster.as_stream_type()); + let outcome = cluster.outcome.map(|outcome| match outcome { + InputClusterOutcomeView::Accepted => ClusterSummaryStatus::Accepted, + InputClusterOutcomeView::Rejected => ClusterSummaryStatus::Rejected, + InputClusterOutcomeView::TechnicalError => ClusterSummaryStatus::TechnicalError, + }); + + html! { +
+
+

{translate(cluster_label_key(cluster.cluster))}

+ + + {translate(status.label_key())} + +
+
+ if cluster.persisted_status.is_some_and(|persisted| persisted.last_update.is_some()) { + if let Some(policy) = policy { + {detail_row( + translate("MESSAGES.PLAYLIST_UPDATE.MODE"), + translate(policy_label_key(policy)), + )} + } + } + if let Some(count) = cluster.baseline_count { + {detail_row(translate("MESSAGES.PLAYLIST_UPDATE.BASELINE"), count.to_string())} + } + if let Some(count) = cluster.candidate_count { + {detail_row(translate("MESSAGES.PLAYLIST_UPDATE.CANDIDATE"), count.to_string())} + } + if let Some(count) = cluster.active_count { + {detail_row(translate("MESSAGES.PLAYLIST_UPDATE.ACTIVE_POPULATION"), count.to_string())} + } + {quality_details(cluster, policy, translate)} + if let Some(outcome) = outcome { + {detail_row( + translate(if outcome == ClusterSummaryStatus::TechnicalError { + "LABEL.STATUS" + } else { + "MESSAGES.PLAYLIST_UPDATE.DECISION" + }), + translate(outcome.label_key()), + )} + } + if cluster.technical_state == Some(PersistedPlaylistUpdateTechnicalState::Failed) + && outcome != Some(ClusterSummaryStatus::TechnicalError) { + {detail_row( + translate("LABEL.STATUS"), + translate(ClusterSummaryStatus::TechnicalError.label_key()), + )} + } + {cache_details(cluster.source, policy, translate)} +
+
+ } +} + +const fn target_stage_label_key(stage: PlaylistProcessingStage) -> &'static str { + match stage { + PlaylistProcessingStage::Filter => "LABEL.FILTER", + PlaylistProcessingStage::Rename => "LABEL.RENAME", + PlaylistProcessingStage::Mapping => "LABEL.MAPPING", + } +} + +const fn target_stage_summary(stage: PlaylistProcessingStage, pipeline: TargetPipelineView) -> RuleStageSummary { + match stage { + PlaylistProcessingStage::Filter => pipeline.filter_summary, + PlaylistProcessingStage::Rename => pipeline.rename_summary, + PlaylistProcessingStage::Mapping => pipeline.mapping_summary, + } +} + +fn target_pipeline_details(input_id: u16, target: &TargetRunView, translate: &impl Fn(&str) -> String) -> Html { + let heading_id = format!("input-update-card-{input_id}-target-{}-pipeline", target.target_id); + + html! { +
+
+
{target.target_name.as_str()}
+
+ if let Some(pipeline) = target.pipeline { +
+ {translate("LABEL.PROCESSING_ORDER")} + +
+
+ {for processing_stages(pipeline.processing_order).into_iter().map(|stage| { + let summary = target_stage_summary(stage, pipeline); + detail_row( + translate(target_stage_label_key(stage)), + count_message( + translate("MESSAGES.PLAYLIST_UPDATE.CONFIGURED_COUNT"), + summary.configured_count, + ), + ) + })} +
+ } else { +

+ {translate("MESSAGES.PLAYLIST_UPDATE.DATA_UNAVAILABLE")} +

+ } +
+ } +} + +fn policy_options( + capabilities: InputUpdateCapabilities, + selected_policy: InputRefreshPolicy, + label: impl Fn(&str) -> String, +) -> Rc> { + if capabilities == InputUpdateCapabilities::Rescan { + return Rc::new(vec![DropDownOption::new("rescan", Html::from(label("LABEL.RESCAN")), true)]); + } + Rc::new( + capabilities + .policies() + .iter() + .copied() + .map(|policy| { + DropDownOption::new( + policy_id(policy), + Html::from(label(policy_label_key(policy))), + policy == selected_policy, + ) + }) + .collect(), + ) +} + +#[cfg(target_arch = "wasm32")] +fn format_local_timestamp(timestamp: u64) -> String { + let date = js_sys::Date::new(&JsValue::from_f64(timestamp as f64 * 1_000.0)); + date.to_locale_string("default", &JsValue::UNDEFINED).into() +} + +#[cfg(not(target_arch = "wasm32"))] +fn format_local_timestamp(timestamp: u64) -> String { + i64::try_from(timestamp) + .ok() + .and_then(|seconds| Local.timestamp_opt(seconds, 0).single()) + .map_or_else(|| timestamp.to_string(), |date| date.format("%Y-%m-%d %H:%M:%S").to_string()) +} + +// Render the complete content hierarchy independently of interaction hooks. +// Target controls and Last update keep their existing owners and callbacks. +fn input_content( + view: &InputUpdateRunView, + capabilities: InputUpdateCapabilities, + last_update: Html, + targets: Html, + translate: impl Fn(&str) -> String, +) -> Html { + let pipeline_summary_heading_id = format!("input-update-card-{}-pipeline-summary", view.input_id); + let target_pipeline_heading_id = format!("input-update-card-{}-target-pipelines", view.input_id); + let mut catalog_clusters = view.catalog_content_clusters(); + catalog_clusters.sort_by_key(|cluster| content_cluster_presentation(*cluster).rank); + let mut cluster_results: Vec<_> = view.cluster_results.iter().collect(); + cluster_results.sort_by_key(|cluster| content_cluster_presentation(cluster.cluster).rank); + html! { +
+ {last_update} +
+

+ {translate(if view.cluster_results.is_empty() && catalog_clusters.is_empty() { "LABEL.INPUT" } else { "LABEL.CLUSTER" })} +

+
+ if view.cluster_results.is_empty() && !catalog_clusters.is_empty() { + {for catalog_clusters.iter().map(|cluster| html! { + + {translate(cluster_label_key(*cluster))} + + })} + } else if view.cluster_results.is_empty() { + {"—"} + } else { + {for cluster_results.iter().map(|cluster| { + let cluster_status = cluster_summary_status(cluster); + let cluster_label = translate(cluster_label_key(cluster.cluster)); + let state_label = translate(cluster_status.label_key()); + html! { + + {cluster_label} + + + } + })} + } +
+
+ {targets} +
+ {translate("MESSAGES.PLAYLIST_UPDATE.UPDATE_DETAILS")} +
+ if let Some(policy) = view.refresh_policy.filter(|_| capabilities != InputUpdateCapabilities::Rescan) { +
+ {detail_row( + translate("MESSAGES.PLAYLIST_UPDATE.MODE"), + translate(policy_label_key(policy)), + )} +
+ } + if view.source_type == shared::model::InputType::Library { + {library_update_details::library_details(view, &translate)} + } else if view.cluster_results.is_empty() { +
+
+

{translate("LABEL.INPUT")}

+ + {translate(view.input_state.label_key())} + + +
+
+ {cache_details(view.cache_source, view.refresh_policy, &translate)} +
+
+ } else { + {for cluster_results.iter().map(|cluster| { + cluster_details( + view.input_id, + cluster, + cluster.refresh_policy.or(view.refresh_policy), + &translate, + ) + })} + } + if !view.localized_progress_details.is_empty() { +
+

{translate("MESSAGES.PLAYLIST_UPDATE.RUN_DETAILS")}

+
    + {for view.localized_progress_details.iter().map(|detail| { + html! {
  • {translate(progress_detail_label_key(*detail))}
    • } + })} +
+
+ } + if !view.target_results.is_empty() { +
+

{translate("LABEL.TARGETS")}

+
+ {for view.target_results.iter().map(|target| { + target_pipeline_details(view.input_id, target, &translate) + })} +
+
+ } +
+
+
+ } +} + +#[derive(Properties, Clone, PartialEq)] +pub struct InputUpdateCardProps { + pub model: Rc, + pub view: Rc, + pub now: u64, + pub connection_context: WebSocketConnectionContext, + #[prop_or_default] + pub on_request_accepted: Callback, + #[prop_or(true)] + pub can_submit: bool, +} + +#[component] +pub fn InputUpdateCard(props: &InputUpdateCardProps) -> Html { + let translate = use_translation(); + let services = use_service_context(); + let dialog = use_context::(); + let model = Rc::clone(&props.model); + let view = Rc::clone(&props.view); + let state = use_reducer(|| InputUpdateCardInteractionState::from_model(&model)); + let target_ids = view.target_results.iter().map(|target| target.target_id).collect::>(); + let all_selection = target_selection_state(&state.controls.selected_target_ids, &target_ids); + + { + let state = state.clone(); + use_effect_with(props.connection_context, move |_| { + state.dispatch(InputUpdateCardAction::ConnectionChanged); + || () + }); + } + + { + let state = state.clone(); + use_effect_with( + (model.capabilities, state.controls.policy, state.is_submitting()), + move |(capabilities, _, _)| { + if state.controls.policy != state.policy_for_capabilities(*capabilities) { + state.dispatch(InputUpdateCardAction::SyncPolicy(*capabilities)); + } + || () + }, + ); + } + + // Render consistently while the effect synchronizes the retained reducer state. + let effective_policy = state.policy_for_capabilities(model.capabilities); + let policy_options = policy_options(model.capabilities, effective_policy, |key| translate.t(key)); + + let on_policy_select = { + let state = state.clone(); + let capabilities = model.capabilities; + Callback::from(move |(_name, selection): (String, DropDownSelection)| { + if let Some(policy) = policy_from_selection(&selection).filter(|policy| capabilities.supports(*policy)) { + state.dispatch(InputUpdateCardAction::SelectPolicy(policy)); + } + }) + }; + + let on_toggle_all = { + let state = state.clone(); + let target_ids = target_ids.clone(); + Callback::from(move |_| state.dispatch(InputUpdateCardAction::ToggleAll(target_ids.clone()))) + }; + + let on_start = { + let dialog = dialog.clone(); + let model = Rc::clone(&model); + let services = services.clone(); + let state = state.clone(); + let translate = translate.clone(); + let can_submit = props.can_submit; + let connection_context = props.connection_context; + let on_request_accepted = props.on_request_accepted.clone(); + Callback::from(move |_: String| { + if !can_start_update(&state, &model, can_submit) + || !services.websocket.is_current_connection(connection_context) + { + return; + } + let request_state = state.controls.clone(); + let target_ids = selected_current_target_ids(&model, &request_state.selected_target_ids); + if target_ids.is_empty() { + return; + } + state.dispatch(InputUpdateCardAction::BeginSubmission(connection_context)); + let dialog = dialog.clone(); + let model = Rc::clone(&model); + let services = services.clone(); + let state = state.clone(); + let translate = translate.clone(); + let on_request_accepted = on_request_accepted.clone(); + spawn_local(async move { + let confirmation = if requires_confirmation(request_state.policy) { + let Some(dialog) = dialog else { + log::error!("Dialog service is unavailable"); + state.dispatch(InputUpdateCardAction::CancelSubmission(connection_context)); + return; + }; + Some(dialog.confirm(&translate.t("MESSAGES.PLAYLIST_UPDATE.FORCE_CONFIRM")).await) + } else { + None + }; + match confirmation_submission_decision( + connection_context, + services.websocket.connection_context(), + request_state.policy, + confirmation, + ) { + ConfirmationSubmissionDecision::Continue => {} + ConfirmationSubmissionDecision::Cancel => { + state.dispatch(InputUpdateCardAction::CancelSubmission(connection_context)); + return; + } + ConfirmationSubmissionDecision::IgnoreStale => return, + } + + let result = services + .playlist + .update_input(&target_ids, model.input_id, model.capabilities.selected_action(request_state.policy)) + .await; + let Some(outcome) = current_submission_outcome( + connection_context, + services.websocket.connection_context(), + classify_update_outcome(result), + ) else { + return; + }; + state + .dispatch(InputUpdateCardAction::FinishSubmission { connection_context, outcome: outcome.clone() }); + if let PlaylistUpdateActionOutcome::Accepted(Some(run_id)) = &outcome { + on_request_accepted.emit(PlaylistUpdateAcceptedScope { + connection_context, + run_id: run_id.clone(), + input_ids: vec![model.input_id], + refresh_policy: request_state.policy, + }); + } + + let action = translate.t(if model.capabilities == InputUpdateCapabilities::Rescan { + "LABEL.RESCAN" + } else { + policy_label_key(request_state.policy) + }); + let message_key = match outcome { + PlaylistUpdateActionOutcome::Accepted(_) => "MESSAGES.PLAYLIST_UPDATE.INPUT_ACCEPTED", + PlaylistUpdateActionOutcome::Conflict => "MESSAGES.PLAYLIST_UPDATE.INPUT_BUSY", + PlaylistUpdateActionOutcome::Failed => "MESSAGES.PLAYLIST_UPDATE.INPUT_FAILED", + }; + let message = action_message(translate.t(message_key), &model.input_name, &action); + match outcome { + PlaylistUpdateActionOutcome::Accepted(_) => services.toastr.success(message), + PlaylistUpdateActionOutcome::Conflict => services.toastr.warning(message), + PlaylistUpdateActionOutcome::Failed => services.toastr.error(message), + } + }); + }) + }; + + let heading_id = format!("input-update-card-{}-title", view.input_id); + let target_group_heading_id = format!("input-update-card-{}-targets", view.input_id); + let policy_description_id = format!("input-update-card-{}-policy-description", view.input_id); + let start_label = translate.t(if model.capabilities == InputUpdateCapabilities::Rescan { + "LABEL.START_RESCAN" + } else { + start_button_label_key(effective_policy) + }); + let action_label = translate.t("LABEL.ACTION"); + let select_label = + action_message(translate.t("MESSAGES.PLAYLIST_UPDATE.POLICY_LABEL"), &view.input_name, &action_label); + let start_aria_label = + action_message(translate.t("MESSAGES.PLAYLIST_UPDATE.START_LABEL"), &view.input_name, &start_label); + let policy_description = translate.t(model.capabilities.description_key(effective_policy)); + let last_update = last_update_view_model(view.last_update_at, props.now); + let last_update_text = match last_update.relative { + LastUpdateRelative::Never => translate.t("LABEL.LAST_UPDATE_NEVER"), + LastUpdateRelative::JustNow => translate.t("LABEL.LAST_UPDATE_JUST_NOW"), + LastUpdateRelative::Minutes(1) => translate.t("LABEL.LAST_UPDATE_MINUTE_AGO"), + LastUpdateRelative::Minutes(count) => count_message(translate.t("LABEL.LAST_UPDATE_MINUTES_AGO"), count), + LastUpdateRelative::Hours(1) => translate.t("LABEL.LAST_UPDATE_HOUR_AGO"), + LastUpdateRelative::Hours(count) => count_message(translate.t("LABEL.LAST_UPDATE_HOURS_AGO"), count), + LastUpdateRelative::LocalDateTime(timestamp) => format_local_timestamp(timestamp), + }; + let last_update_title = last_update.title_timestamp.map(format_local_timestamp).unwrap_or_default(); + + html! { + +
+
+
+

{view.input_name.as_ref()}

+ + {translate.t(input_type_label_key(view.source_type))} + +
+ {card_status_badge(&view, |key| translate.t(key))} +
+
+ {input_content( + &view, + model.capabilities, + html! { +
+ {translate.t("LABEL.LAST_UPDATE")}{":"} + + {last_update_text} + +
+ }, + html! { +
+

{translate.t("LABEL.AFFECTED_TARGETS")}

+ if view.target_results.is_empty() { +

+ {translate.t("MESSAGES.PLAYLIST_UPDATE.NO_TARGETS_ASSIGNED")} +

+ } else { +
+ + {for view.target_results.iter().map(|target| { + let selected = state.controls.selected_target_ids.contains(&target.target_id); + let selection = if selected { TargetSelectionState::All } else { TargetSelectionState::Empty }; + let state = state.clone(); + let target_id = target.target_id; + html! { + + } + })} +
+ } +
+ }, + |key| translate.t(key), + )} +
+
+ {action_label} + if model.capabilities.rebuilds_targets() { + + +
+ } +} + +#[cfg(test)] +mod tests { + use super::{emit_slider_value, normalize_range, parse_slider_value, NormalizedRange}; + use std::{cell::Cell, rc::Rc}; + use yew::Callback; + + #[test] + fn range_normalization_clamps_values_and_calculates_fill() { + let cases = [ + (0, 100, 90, NormalizedRange { min: 0, max: 100, value: 90, fill_percent: 90 }), + (0, 100, 101, NormalizedRange { min: 0, max: 100, value: 100, fill_percent: 100 }), + (20, 60, 10, NormalizedRange { min: 20, max: 60, value: 20, fill_percent: 0 }), + (20, 60, 40, NormalizedRange { min: 20, max: 60, value: 40, fill_percent: 50 }), + (50, 50, 50, NormalizedRange { min: 50, max: 50, value: 50, fill_percent: 0 }), + (100, 0, 25, NormalizedRange { min: 0, max: 100, value: 25, fill_percent: 25 }), + ]; + + for (min, max, value, expected) in cases { + assert_eq!(normalize_range(min, max, value), expected); + } + } + + #[test] + fn parsed_values_are_bounded_to_the_normalized_range() { + assert_eq!(parse_slider_value("101", 0, 100), Some(100)); + assert_eq!(parse_slider_value("-1", 0, 100), Some(0)); + assert_eq!(parse_slider_value("42", 20, 60), Some(42)); + assert_eq!(parse_slider_value("", 0, 100), None); + assert_eq!(parse_slider_value("invalid", 0, 100), None); + } + + #[test] + fn range_slider_controls_emit_direct_and_bounded_values_only_while_active() { + let emitted = Rc::new(Cell::new(None)); + let captured = Rc::clone(&emitted); + let callback = Callback::from(move |value| captured.set(Some(value))); + let range = normalize_range(0, 100, 0); + + emit_slider_value(false, range, || "42".to_string(), &callback); + assert_eq!(emitted.get(), Some(42)); + + emit_slider_value(false, range, || "101".to_string(), &callback); + assert_eq!(emitted.get(), Some(100)); + + emitted.set(None); + emit_slider_value(false, range, String::new, &callback); + assert_eq!(emitted.get(), None); + + emitted.set(None); + emit_slider_value(true, range, || panic!("disabled slider must not read the event value"), &callback); + assert_eq!(emitted.get(), None); + } +} diff --git a/frontend/src/app/components/select.rs b/frontend/src/app/components/select.rs index b34e80064..ffc7d0bcb 100644 --- a/frontend/src/app/components/select.rs +++ b/frontend/src/app/components/select.rs @@ -1,4 +1,7 @@ -use crate::app::components::{DropDownIconButton, DropDownOption, DropDownSelection}; +use crate::app::components::{ + popup_menu::{PopupMenuPlacement, PopupMenuWidth}, + DropDownIconButton, DropDownOption, DropDownSelection, +}; use std::rc::Rc; use yew::prelude::*; @@ -14,6 +17,10 @@ pub struct SelectProps { pub on_select: Callback<(String, DropDownSelection)>, #[prop_or_default] pub class: String, + #[prop_or_default] + pub popup_width: PopupMenuWidth, + #[prop_or_default] + pub popup_placement: PopupMenuPlacement, pub options: Rc>, #[prop_or_default] pub multi_select: bool, @@ -21,12 +28,21 @@ pub struct SelectProps { pub required: bool, #[prop_or_default] pub error: Option, + #[prop_or_default] + pub aria_describedby: Option, } #[component] pub fn Select(props: &SelectProps) -> Html { let button_ref = use_node_ref(); + let field_ref = use_node_ref(); let error_id = props.error.as_ref().map(|_| format!("{}-select-error", props.name)); + let aria_describedby = match (&props.aria_describedby, &error_id) { + (Some(description), Some(error)) => Some(format!("{description} {error}")), + (Some(description), None) => Some(description.clone()), + (None, Some(error)) => Some(error.clone()), + (None, None) => None, + }; let selected_options = use_state(Vec::new); { @@ -56,12 +72,15 @@ pub fn Select(props: &SelectProps) -> Html { props.required.then_some("tp__input--required"), props.error.as_ref().map(|_| "tp__input--error"), props.class.clone())}> -
+
{(*selected_options).clone()}
Html { aria_label={props.name.clone()} aria_required={props.required.then_some(true)} aria_invalid={props.error.as_ref().map(|_| true)} - aria_describedby={error_id.clone()} + aria_describedby={aria_describedby} on_select={props.on_select.clone()} />
{ props.error.as_ref().map_or_else(Html::default, |error| html! { @@ -78,3 +97,28 @@ pub fn Select(props: &SelectProps) -> Html {
} } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn playlist_update_dropdown_select_width_is_opt_in_and_passed_to_the_popup() { + let props = yew::props!(SelectProps { + name: "default-select", + options: Rc::new(Vec::new()), + on_select: Callback::noop(), + }); + assert_eq!(props.popup_width, PopupMenuWidth::Content); + assert_eq!(props.popup_placement, PopupMenuPlacement::Adaptive); + let source = include_str!("select.rs").split("#[cfg(test)]").next().unwrap(); + let dropdown = include_str!("drop_down_icon_button.rs"); + assert!(source.contains("ref={field_ref.clone()}")); + assert!(source.contains("popup_width={props.popup_width}")); + assert!(source.contains("popup_placement={props.popup_placement}")); + assert!(source.contains("width_anchor_ref={Some(field_ref)}")); + assert!(dropdown.contains("width={props.popup_width}")); + assert!(dropdown.contains("placement={props.popup_placement}")); + assert!(dropdown.contains("width_anchor_ref={props.width_anchor_ref.clone()}")); + } +} diff --git a/frontend/src/app/components/source_editor/input_form.rs b/frontend/src/app/components/source_editor/input_form.rs index d371f03fd..9ae4e3e8c 100644 --- a/frontend/src/app/components/source_editor/input_form.rs +++ b/frontend/src/app/components/source_editor/input_form.rs @@ -33,10 +33,10 @@ use shared::{ concat_string, error::TuliproxError, model::{ - ConfigInputAliasDto, ConfigInputDto, ConfigInputOptionsDto, ConfigInputStagedDto, ConfigProviderDto, - EpgSmartMatchConfigDto, EpgSourceDto, InputFetchMethod, MediaServerInputConfigDto, MediaServerLibrarySelector, - OnConnectErrorPolicy, ProviderUrlSelectionPolicy, StagedInputType, StalkerDeviceProfileDto, - StalkerInputConfigDto, + ConfigInputAliasDto, ConfigInputDto, ConfigInputOptionsDto, ConfigInputStagedDto, ConfigInputUpdateQualityDto, + ConfigProviderDto, EpgSmartMatchConfigDto, EpgSourceDto, InputFetchMethod, InputType, + MediaServerInputConfigDto, MediaServerLibrarySelector, OnConnectErrorPolicy, ProviderUrlSelectionPolicy, + StagedInputType, StalkerDeviceProfileDto, StalkerInputConfigDto, }, utils::{Internable, BATCH_SCHEME_PREFIX}, }; @@ -80,9 +80,9 @@ const LABEL_ADD_ALIAS: &str = "LABEL.ADD_ALIAS"; const LABEL_ADD_PROVIDER: &str = "LABEL.ADD_PROVIDER"; const LABEL_PROVIDERS: &str = "LABEL.PROVIDER"; const LABEL_SKIP: &str = "LABEL.SKIP"; -const LABEL_SKIP_LIVE: &str = "LABEL.LIVE"; -const LABEL_SKIP_VOD: &str = "LABEL.VOD"; -const LABEL_SKIP_SERIES: &str = "LABEL.SERIES"; +const LABEL_LIVE: &str = "LABEL.LIVE"; +const LABEL_VOD: &str = "LABEL.VOD"; +const LABEL_SERIES: &str = "LABEL.SERIES"; const LABEL_XTREAM_LIVE_STREAM_USE_PREFIX: &str = "LABEL.LIVE_STREAM_USE_PREFIX"; const LABEL_XTREAM_LIVE_STREAM_WITHOUT_EXTENSION: &str = "LABEL.LIVE_STREAM_WITHOUT_EXTENSION"; const LABEL_DISABLE_HLS_STREAMING: &str = "LABEL.DISABLE_HLS_STREAMING"; @@ -102,6 +102,7 @@ const LABEL_PROBE_SERIES: &str = "LABEL.PROBE_SERIES"; const LABEL_RESOLVE_VOD: &str = "LABEL.RESOLVE_VOD"; const LABEL_RESOLVE_SERIES: &str = "LABEL.RESOLVE_SERIES"; const LABEL_METADATA: &str = "LABEL.METADATA"; +const LABEL_UPDATE_QUALITY: &str = "LABEL.UPDATE_QUALITY"; const LABEL_CACHE_DURATION: &str = "LABEL.CACHE_DURATION"; const LABEL_MAIN: &str = "LABEL.MAIN_CONFIG"; const LABEL_OPTIONS: &str = "LABEL.OPTIONS"; @@ -144,6 +145,19 @@ fn staged_type_from_selection(selections: &[String]) -> StagedInputType { selections.first().and_then(|value| value.parse::().ok()).unwrap_or_default() } +fn options_kind_for(input_type: InputType, staged_type: StagedInputType) -> OptionsKind { + match input_type { + InputType::Xtream | InputType::XtreamBatch => OptionsKind::Xtream, + InputType::Stalker | InputType::StalkerBatch => OptionsKind::Stalker, + InputType::Staged if staged_type == StagedInputType::Xtream => OptionsKind::Xtream, + InputType::M3u | InputType::M3uBatch => OptionsKind::M3u, + InputType::Staged if staged_type == StagedInputType::M3u => OptionsKind::M3u, + InputType::Library | InputType::Emby | InputType::Jellyfin | InputType::Plex | InputType::Staged => { + OptionsKind::Basic + } + } +} + fn libraries_to_text(libraries: &[MediaServerLibrarySelector]) -> String { libraries .iter() @@ -348,6 +362,7 @@ generate_form_reducer!( SkipLive => skip_live: bool, SkipVod => skip_vod: bool, SkipSeries => skip_series: bool, + UpdateQuality => update_quality: ConfigInputUpdateQualityDto, XtreamLiveStreamUsePrefix => xtream_live_stream_use_prefix: bool, XtreamLiveStreamWithoutExtension => xtream_live_stream_without_extension: bool, DisableHlsStreaming => disable_hls_streaming: bool, @@ -843,10 +858,8 @@ pub fn ConfigInputView(props: &ConfigInputViewProps) -> Html { let library_input = input_form_state.form.input_type.is_library(); let media_server_input = input_form_state.form.input_type.is_media_server(); - let xtream_input = input_form_state.form.input_type.is_xtream(); let stalker_input = input_form_state.form.input_type.is_stalker(); let staged_input = input_form_state.form.input_type.is_staged(); - let staged_xtream_input = staged_input && input_form_state.form.staged_type == StagedInputType::Xtream; let options_input = !library_input && !media_server_input; let render_main = || match input_form_state.form.input_type { @@ -871,13 +884,7 @@ pub fn ConfigInputView(props: &ConfigInputViewProps) -> Html { }; let render_options = || { - let kind = if stalker_input { - OptionsKind::Stalker - } else if xtream_input || staged_xtream_input { - OptionsKind::Xtream - } else { - OptionsKind::Basic - }; + let kind = options_kind_for(input_form_state.form.input_type, input_form_state.form.staged_type); let extra = if stalker_input { stalker_options_fields(&stalker_config_state, props.allow_write, &translate) } else { @@ -1419,6 +1426,7 @@ pub fn ConfigInputView(props: &ConfigInputViewProps) -> Html { mod tests { use super::*; use shared::model::{MediaServerLibraryKind, MediaServerLibrarySelectorDetailsDto, StalkerAuthMode}; + use yew::Reducible; #[test] fn media_server_libraries_page_round_trips() { @@ -1475,6 +1483,71 @@ mod tests { assert_eq!(input_persist_hint_key(false), "INPUT_FORM.PERSIST"); } + #[test] + fn m3u_update_quality_is_visible_only_for_cluster_capable_input_types() { + let cases = [ + (InputType::Xtream, StagedInputType::M3u, true), + (InputType::XtreamBatch, StagedInputType::M3u, true), + (InputType::Stalker, StagedInputType::M3u, true), + (InputType::StalkerBatch, StagedInputType::M3u, true), + (InputType::Staged, StagedInputType::Xtream, true), + (InputType::M3u, StagedInputType::M3u, true), + (InputType::M3uBatch, StagedInputType::M3u, true), + (InputType::Staged, StagedInputType::M3u, true), + (InputType::Library, StagedInputType::M3u, false), + (InputType::Emby, StagedInputType::M3u, false), + (InputType::Jellyfin, StagedInputType::M3u, false), + (InputType::Plex, StagedInputType::M3u, false), + ]; + + for (input_type, staged_type, expected) in cases { + assert_eq!(options_kind_for(input_type, staged_type).supports_update_quality(), expected); + } + } + + #[test] + fn update_quality_reducer_preserves_loaded_and_applied_values() { + let loaded = ConfigInputOptionsDto { + update_quality: ConfigInputUpdateQualityDto { live: 91, vod: 82, series: 73 }, + ..ConfigInputOptionsDto::default() + }; + let state = Rc::new(ConfigInputOptionsDtoFormState { form: ConfigInputOptionsDto::default(), modified: false }) + .reduce(ConfigInputOptionsFormAction::SetAll(loaded.clone())); + assert_eq!(state.form, loaded); + assert!(!state.modified()); + + let updated = ConfigInputUpdateQualityDto { live: 95, vod: 90, series: 85 }; + let state = state.reduce(ConfigInputOptionsFormAction::UpdateQuality(updated)); + + assert_eq!(state.form.update_quality, updated); + assert!(state.modified()); + let applied = state.form.clone(); + assert_eq!(applied.update_quality, updated); + } + + #[test] + fn update_quality_translations_exist_in_every_frontend_locale() -> Result<(), serde_json::Error> { + let locales = [ + ("en", include_str!("../../../../public/assets/i18n/en.json")), + ("ru", include_str!("../../../../public/assets/i18n/ru.json")), + ("ar", include_str!("../../../../public/assets/i18n/ar.json")), + ]; + + for (locale, source) in locales { + let translations: serde_json::Value = serde_json::from_str(source)?; + for pointer in ["/LABEL/UPDATE_QUALITY", "/EXPLANATION/INPUT_FORM/UPDATE_QUALITY"] { + assert!( + translations + .pointer(pointer) + .and_then(serde_json::Value::as_str) + .is_some_and(|value| !value.is_empty()), + "missing {pointer} translation for {locale}" + ); + } + } + Ok(()) + } + #[test] fn libraries_from_text_preserves_existing_detailed_selector() { let detailed = MediaServerLibrarySelector::Detailed(MediaServerLibrarySelectorDetailsDto { diff --git a/frontend/src/app/components/source_editor/input_form/common.rs b/frontend/src/app/components/source_editor/input_form/common.rs index 79da84dfd..1ddac05a6 100644 --- a/frontend/src/app/components/source_editor/input_form/common.rs +++ b/frontend/src/app/components/source_editor/input_form/common.rs @@ -1,23 +1,27 @@ use super::{ input_persist_hint_key, input_url_hint_key, ConfigInputFormAction, ConfigInputFormState, ConfigInputOptionsDtoFormState, ConfigInputOptionsFormAction, LABEL_CACHE_DURATION, LABEL_DISABLE_HLS_STREAMING, - LABEL_ENABLED, LABEL_EXP_DATE, LABEL_FETCH_METHOD, LABEL_HEADERS, LABEL_LIVE_STREAMS, LABEL_MAX_CONNECTIONS, - LABEL_METADATA, LABEL_NAME, LABEL_PASSWORD, LABEL_PERSIST, LABEL_PRIORITY, LABEL_PROBE, LABEL_PROBE_DELAY_SEC, - LABEL_PROBE_FILTER, LABEL_PROBE_LIVE, LABEL_PROBE_LIVE_INTERVAL_HOURS, LABEL_PROBE_SERIES, LABEL_PROBE_VOD, - LABEL_RESOLVE, LABEL_RESOLVE_BACKGROUND, LABEL_RESOLVE_DELAY_SEC, LABEL_RESOLVE_FILTER, LABEL_RESOLVE_SERIES, - LABEL_RESOLVE_TMDB, LABEL_RESOLVE_VOD, LABEL_SEQUENTIAL_GROUP, LABEL_SKIP, LABEL_SKIP_LIVE, LABEL_SKIP_SERIES, - LABEL_SKIP_VOD, LABEL_URL, LABEL_USERNAME, LABEL_USER_AGENT_STREAM_INDEX, LABEL_XTREAM_LIVE_STREAM_USE_PREFIX, - LABEL_XTREAM_LIVE_STREAM_WITHOUT_EXTENSION, + LABEL_ENABLED, LABEL_EXP_DATE, LABEL_FETCH_METHOD, LABEL_HEADERS, LABEL_LIVE, LABEL_LIVE_STREAMS, + LABEL_MAX_CONNECTIONS, LABEL_METADATA, LABEL_NAME, LABEL_PASSWORD, LABEL_PERSIST, LABEL_PRIORITY, LABEL_PROBE, + LABEL_PROBE_DELAY_SEC, LABEL_PROBE_FILTER, LABEL_PROBE_LIVE, LABEL_PROBE_LIVE_INTERVAL_HOURS, LABEL_PROBE_SERIES, + LABEL_PROBE_VOD, LABEL_RESOLVE, LABEL_RESOLVE_BACKGROUND, LABEL_RESOLVE_DELAY_SEC, LABEL_RESOLVE_FILTER, + LABEL_RESOLVE_SERIES, LABEL_RESOLVE_TMDB, LABEL_RESOLVE_VOD, LABEL_SEQUENTIAL_GROUP, LABEL_SERIES, LABEL_SKIP, + LABEL_UPDATE_QUALITY, LABEL_URL, LABEL_USERNAME, LABEL_USER_AGENT_STREAM_INDEX, LABEL_VOD, + LABEL_XTREAM_LIVE_STREAM_USE_PREFIX, LABEL_XTREAM_LIVE_STREAM_WITHOUT_EXTENSION, }; use crate::{ - app::components::{Card, FilterInput, KeyValueEditor, RadioButtonGroup, TitledCard, ToolAction}, + app::components::{Card, FilterInput, KeyValueEditor, RadioButtonGroup, RangeSlider, TitledCard, ToolAction}, config_field, config_field_bool, config_field_child, config_field_custom, config_field_optional, config_field_optional_hide, edit_field_bool, edit_field_exp_date, edit_field_number_i16, edit_field_number_option_u32, edit_field_number_u16, edit_field_number_u32, edit_field_text, edit_field_text_option, html_if, i18n::{use_translation, YewI18n}, + utils::content_cluster_presentation, +}; +use shared::{ + model::{ConfigInputUpdateQualityDto, InputFetchMethod, XtreamCluster}, + utils::BATCH_SCHEME_PREFIX, }; -use shared::{model::InputFetchMethod, utils::BATCH_SCHEME_PREFIX}; use std::{collections::HashMap, rc::Rc}; use yew::{component, html, use_memo, Callback, Html, Properties, UseReducerHandle, UseStateHandle}; @@ -170,10 +174,86 @@ pub(super) fn CommonInputForm(props: &CommonInputFormProps) -> Html { #[derive(Debug, Copy, Clone, PartialEq, Eq)] pub(super) enum OptionsKind { Basic, + M3u, Xtream, Stalker, } +impl OptionsKind { + pub(super) const fn supports_update_quality(self) -> bool { !matches!(self, Self::Basic) } +} + +fn update_quality_value(update_quality: ConfigInputUpdateQualityDto, cluster: XtreamCluster) -> u8 { + match cluster { + XtreamCluster::Live => update_quality.live, + XtreamCluster::Video => update_quality.vod, + XtreamCluster::Series => update_quality.series, + } +} + +fn set_update_quality_value(update_quality: &mut ConfigInputUpdateQualityDto, cluster: XtreamCluster, value: u8) { + match cluster { + XtreamCluster::Live => update_quality.live = value, + XtreamCluster::Video => update_quality.vod = value, + XtreamCluster::Series => update_quality.series = value, + } +} + +const fn update_quality_field(cluster: XtreamCluster) -> (&'static str, &'static str) { + let field = match cluster { + XtreamCluster::Live => "update_quality_live", + XtreamCluster::Video => "update_quality_vod", + XtreamCluster::Series => "update_quality_series", + }; + (field, content_cluster_presentation(cluster).label_key) +} + +fn update_quality_content( + quality: ConfigInputUpdateQualityDto, + translate: impl Fn(&str) -> String, + allow_write: bool, + on_change: impl Fn(XtreamCluster) -> Callback, +) -> Html { + let mut clusters = shared::model::xtream_const::XTREAM_CLUSTER; + clusters.sort_by_key(|cluster| content_cluster_presentation(*cluster).rank); + html! { + + {for clusters.into_iter().map(|cluster| { + let (name, label) = update_quality_field(cluster); + let value = update_quality_value(quality, cluster); + let on_change = on_change(cluster); + config_field_child!(translate(label), "INPUT_FORM.UPDATE_QUALITY", { + html! { } + }) + })} + + } +} + +fn update_quality_options( + state: &UseReducerHandle, + translate: &YewI18n, + allow_write: bool, +) -> Html { + update_quality_content( + state.form.update_quality, + |key| translate.t(key), + allow_write, + |cluster| { + if allow_write { + let state = state.clone(); + Callback::from(move |value| { + let mut update_quality = state.form.update_quality; + set_update_quality_value(&mut update_quality, cluster, value); + state.dispatch(ConfigInputOptionsFormAction::UpdateQuality(update_quality)); + }) + } else { + Callback::noop() + } + }, + ) +} + #[derive(Properties, Clone)] pub(super) struct InputOptionsFormProps { pub state: UseReducerHandle, @@ -193,36 +273,48 @@ pub(super) fn InputOptionsForm(props: &InputOptionsFormProps) -> Html { let translate = use_translation(); let state = props.state.clone(); let headers = props.headers.clone(); + let update_quality = if props.kind.supports_update_quality() { + update_quality_options(&state, &translate, props.allow_write) + } else { + Html::default() + }; let type_options = match (props.allow_write, props.kind) { (_, OptionsKind::Basic) => Html::default(), + (_, OptionsKind::M3u) => update_quality, (false, OptionsKind::Stalker) => html! { - + <> +
- { config_field_bool!(state.form, translate.t(LABEL_SKIP_LIVE), skip_live) } - { config_field_bool!(state.form, translate.t(LABEL_SKIP_VOD), skip_vod) } - { config_field_bool!(state.form, translate.t(LABEL_SKIP_SERIES), skip_series) } + { config_field_bool!(state.form, translate.t(LABEL_LIVE), skip_live) } + { config_field_bool!(state.form, translate.t(LABEL_VOD), skip_vod) } + { config_field_bool!(state.form, translate.t(LABEL_SERIES), skip_series) }
+ + {update_quality} + { config_field_bool!(state.form, translate.t("LABEL.STALKER_BULK_EPG"), stalker_bulk_epg) } {props.extra.clone()} + }, (true, OptionsKind::Stalker) => html! { <>
- { edit_field_bool!(state, translate.t(LABEL_SKIP_LIVE), skip_live, ConfigInputOptionsFormAction::SkipLive) } - { edit_field_bool!(state, translate.t(LABEL_SKIP_VOD), skip_vod, ConfigInputOptionsFormAction::SkipVod) } - { edit_field_bool!(state, translate.t(LABEL_SKIP_SERIES), skip_series, ConfigInputOptionsFormAction::SkipSeries) } + { edit_field_bool!(state, translate.t(LABEL_LIVE), skip_live, ConfigInputOptionsFormAction::SkipLive) } + { edit_field_bool!(state, translate.t(LABEL_VOD), skip_vod, ConfigInputOptionsFormAction::SkipVod) } + { edit_field_bool!(state, translate.t(LABEL_SERIES), skip_series, ConfigInputOptionsFormAction::SkipSeries) }
+ {update_quality} { edit_field_bool!(state, translate.t("LABEL.STALKER_BULK_EPG"), stalker_bulk_epg, ConfigInputOptionsFormAction::StalkerBulkEpg) } {props.extra.clone()} }, - (false, OptionsKind::Xtream) => xtream_options_readonly(&state, &translate), - (true, OptionsKind::Xtream) => xtream_options_editable(&state, &translate), + (false, OptionsKind::Xtream) => xtream_options_readonly(&state, &translate, update_quality), + (true, OptionsKind::Xtream) => xtream_options_editable(&state, &translate, update_quality), }; html! { @@ -251,13 +343,18 @@ pub(super) fn InputOptionsForm(props: &InputOptionsFormProps) -> Html { } } -fn xtream_options_readonly(state: &UseReducerHandle, translate: &YewI18n) -> Html { +fn xtream_options_readonly( + state: &UseReducerHandle, + translate: &YewI18n, + update_quality: Html, +) -> Html { html! { <>
- { config_field_bool!(state.form, translate.t(LABEL_SKIP_LIVE), skip_live) } - { config_field_bool!(state.form, translate.t(LABEL_SKIP_VOD), skip_vod) } - { config_field_bool!(state.form, translate.t(LABEL_SKIP_SERIES), skip_series) } + { config_field_bool!(state.form, translate.t(LABEL_LIVE), skip_live) } + { config_field_bool!(state.form, translate.t(LABEL_VOD), skip_vod) } + { config_field_bool!(state.form, translate.t(LABEL_SERIES), skip_series) }
+ {update_quality}
{ config_field_bool!(state.form, translate.t(LABEL_XTREAM_LIVE_STREAM_USE_PREFIX), xtream_live_stream_use_prefix) } { config_field_bool!(state.form, translate.t(LABEL_XTREAM_LIVE_STREAM_WITHOUT_EXTENSION), xtream_live_stream_without_extension) } @@ -289,13 +386,18 @@ fn xtream_options_readonly(state: &UseReducerHandle } } -fn xtream_options_editable(state: &UseReducerHandle, translate: &YewI18n) -> Html { +fn xtream_options_editable( + state: &UseReducerHandle, + translate: &YewI18n, + update_quality: Html, +) -> Html { html! { <>
- { edit_field_bool!(state, translate.t(LABEL_SKIP_LIVE), skip_live, ConfigInputOptionsFormAction::SkipLive) } - { edit_field_bool!(state, translate.t(LABEL_SKIP_VOD), skip_vod, ConfigInputOptionsFormAction::SkipVod) } - { edit_field_bool!(state, translate.t(LABEL_SKIP_SERIES), skip_series, ConfigInputOptionsFormAction::SkipSeries) } + { edit_field_bool!(state, translate.t(LABEL_LIVE), skip_live, ConfigInputOptionsFormAction::SkipLive) } + { edit_field_bool!(state, translate.t(LABEL_VOD), skip_vod, ConfigInputOptionsFormAction::SkipVod) } + { edit_field_bool!(state, translate.t(LABEL_SERIES), skip_series, ConfigInputOptionsFormAction::SkipSeries) }
+ {update_quality}
{ edit_field_bool!(state, translate.t(LABEL_XTREAM_LIVE_STREAM_USE_PREFIX), xtream_live_stream_use_prefix, ConfigInputOptionsFormAction::XtreamLiveStreamUsePrefix) } { edit_field_bool!(state, translate.t(LABEL_XTREAM_LIVE_STREAM_WITHOUT_EXTENSION), xtream_live_stream_without_extension, ConfigInputOptionsFormAction::XtreamLiveStreamWithoutExtension) } @@ -332,3 +434,42 @@ fn xtream_options_editable(state: &UseReducerHandle } } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn update_quality_slider_uses_content_taxonomy_without_renaming_config_fields() { + let translations: serde_json::Value = + serde_json::from_str(include_str!("../../../../../public/assets/i18n/en.json")).unwrap(); + let translate = |key: &str| { + translations + .pointer(&format!("/{}", key.replace('.', "/"))) + .and_then(serde_json::Value::as_str) + .unwrap() + .to_owned() + }; + let quality = ConfigInputUpdateQualityDto { live: 11, vod: 22, series: 33 }; + let callback = Callback::noop(); + for allow_write in [false, true] { + let actual = update_quality_content(quality, translate, allow_write, |_| callback.clone()); + // Compare the complete production VNode, including FieldLabel and RangeSlider + // properties, in exact order. No source-text or set/contains assertion. + let expected = html! { + + {for [("Live", "update_quality_live", 11u8), ("Series", "update_quality_series", 33u8), ("Movies", "update_quality_vod", 22u8)].into_iter().map(|(label, name, value)| { + config_field_child!(label, "INPUT_FORM.UPDATE_QUALITY", { + html! { } + }) + })} + + }; + assert_eq!(actual, expected, "visible labels/order, values, field names and readonly state"); + } + let mut changed = quality; + set_update_quality_value(&mut changed, XtreamCluster::Series, 94); + assert_eq!(serde_json::to_value(changed).unwrap(), serde_json::json!({"live": 11, "vod": 22, "series": 94})); + assert_eq!(update_quality_value(changed, XtreamCluster::Video), 22); + } +} diff --git a/frontend/src/app/components/text_button.rs b/frontend/src/app/components/text_button.rs index 75473d390..e52ffff0f 100644 --- a/frontend/src/app/components/text_button.rs +++ b/frontend/src/app/components/text_button.rs @@ -19,6 +19,9 @@ pub struct TextButtonProps { /// each one deletes; the visible label stays short. #[prop_or(None)] pub aria_label: Option, + /// Optional toggle state (`true`, `false` or `mixed`); absent for ordinary actions. + #[prop_or(None)] + pub aria_pressed: Option, #[prop_or(None)] pub hint: Option, } @@ -33,6 +36,10 @@ pub fn TextButton(props: &TextButtonProps) -> Html { }) }; + text_button_view(props, handle_click) +} + +fn text_button_view(props: &TextButtonProps, handle_click: Callback) -> Html { html! { } } + +#[cfg(test)] +mod tests { + use super::*; + use std::collections::HashMap; + + #[test] + fn text_button_touch_does_not_keep_mouse_highlights_and_preserves_keyboard_focus() { + let styles = include_str!("../../../scss/app/components/_text_button.scss"); + let (highlight, variants) = styles.split_once(".tp__text-button {").expect("shared button styles"); + let (keyboard, mouse) = + highlight.split_once("@media (hover: hover) and (pointer: fine)").expect("hover capability"); + + assert!(keyboard.contains("&:focus-visible {\n @content;")); + assert!(mouse.contains("&:focus,\n &:hover {\n @content;")); + assert_eq!(variants.matches("@include text-button-highlight {").count(), 5); + assert!(!variants.contains("&:hover")); + assert!(!variants.contains("&:focus")); + assert!(variants.contains("font-weight: bold;")); + } + + #[test] + fn text_button_preserves_optional_toggle_accessibility_and_disabled_state() { + for pressed in [None, Some("false"), Some("true"), Some("mixed")] { + for disabled in [false, true] { + let props = yew::props!(TextButtonProps { + name: "target-7", + title: "Target seven", + onclick: Callback::noop(), + aria_label: Some("Target seven".to_string()), + aria_pressed: pressed.map(str::to_string), + disabled, + }); + let Html::VTag(button) = text_button_view(&props, Callback::noop()) else { + panic!("TextButton must render a native button"); + }; + let attributes = button.attributes.iter().collect::>(); + + assert_eq!(button.tag(), "button"); + assert_eq!(attributes.get("type"), Some(&"button")); + assert_eq!(attributes.get("class"), Some(&"tp__text-button")); + assert_eq!(attributes.get("aria-label"), Some(&"Target seven")); + assert_eq!(attributes.get("aria-pressed").copied(), pressed); + assert_eq!(attributes.contains_key("disabled"), disabled); + } + } + } +} diff --git a/frontend/src/app/mod.rs b/frontend/src/app/mod.rs index ed32e62e1..cecc8d683 100644 --- a/frontend/src/app/mod.rs +++ b/frontend/src/app/mod.rs @@ -12,6 +12,7 @@ use crate::{ services::request_get, utils::{get_local_storage_item, set_local_storage_item}, }; +pub(crate) use components::{map_sources_to_playlist_rows, InputRow}; pub use context::*; use futures::future::join_all; use log::error; diff --git a/frontend/src/model/event_message.rs b/frontend/src/model/event_message.rs index cc852fd99..388b0ced5 100644 --- a/frontend/src/model/event_message.rs +++ b/frontend/src/model/event_message.rs @@ -1,7 +1,8 @@ use crate::model::BusyStatus; use shared::model::{ ActiveUserConnectionChange, ConfigType, DownloadsDelta, DownloadsResponse, LibraryScanProgressEvent, - PlaylistUpdateProgressEvent, PlaylistUpdateState, StatusCheck, StreamMeterEntry, SystemInfo, TransferTaskDto, + PlaylistUpdateProgressEvent, PlaylistUpdateRunStateEvent, StatusCheck, StreamMeterEntry, SystemInfo, + TransferTaskDto, }; use std::{rc::Rc, sync::Arc}; @@ -16,7 +17,7 @@ pub enum EventMessage { ActiveProviderCount(usize), // all provider ConfigChange(ConfigType), Busy(BusyStatus), - PlaylistUpdate(PlaylistUpdateState), + PlaylistUpdate(PlaylistUpdateRunStateEvent), PlaylistUpdateProgress(PlaylistUpdateProgressEvent), WebSocketStatus(bool), SystemInfoUpdate(SystemInfo), diff --git a/frontend/src/model/input_update_capabilities.rs b/frontend/src/model/input_update_capabilities.rs new file mode 100644 index 000000000..51e342003 --- /dev/null +++ b/frontend/src/model/input_update_capabilities.rs @@ -0,0 +1,116 @@ +pub use shared::model::InputUpdateCapabilities; +use shared::model::{ConfigInputDto, InputRefreshPolicy, InputUpdateAction}; + +/// Presentation adapters for the shared capability contract, without provider decisions. +pub trait InputUpdateCapabilitiesExt { + fn for_input(input: &ConfigInputDto) -> Self; + fn selected_action(self, policy: InputRefreshPolicy) -> InputUpdateAction; + fn description_key(self, policy: InputRefreshPolicy) -> &'static str; +} + +impl InputUpdateCapabilitiesExt for InputUpdateCapabilities { + fn for_input(input: &ConfigInputDto) -> Self { Self::for_input_type(input.input_type, input.enabled) } + + fn selected_action(self, policy: InputRefreshPolicy) -> InputUpdateAction { + match self { + Self::Rescan => InputUpdateAction::Rescan, + Self::Manual { .. } | Self::BulkOnly | Self::ImportUnavailable | Self::Disabled => { + InputUpdateAction::Provider(policy) + } + } + } + + fn description_key(self, policy: InputRefreshPolicy) -> &'static str { + match self { + Self::Manual { .. } if !self.supports(policy) => "MESSAGES.PLAYLIST_UPDATE.MANUAL_UNAVAILABLE", + Self::Manual { .. } => { + if policy == InputRefreshPolicy::FORCE { + "MESSAGES.PLAYLIST_UPDATE.POLICY_FORCE_DESCRIPTION" + } else if self.supports(InputRefreshPolicy::FORCE) { + if policy == InputRefreshPolicy::REFRESH { + "MESSAGES.PLAYLIST_UPDATE.POLICY_REFRESH_DESCRIPTION" + } else { + "MESSAGES.PLAYLIST_UPDATE.POLICY_UPDATE_DESCRIPTION" + } + } else if policy == InputRefreshPolicy::REFRESH { + "MESSAGES.PLAYLIST_UPDATE.CACHE_REFRESH_DESCRIPTION" + } else { + "MESSAGES.PLAYLIST_UPDATE.CACHE_UPDATE_DESCRIPTION" + } + } + Self::Rescan => "MESSAGES.PLAYLIST_UPDATE.LIBRARY_DESCRIPTION", + Self::BulkOnly => "MESSAGES.PLAYLIST_UPDATE.BULK_ONLY_DESCRIPTION", + Self::ImportUnavailable => "MESSAGES.PLAYLIST_UPDATE.PROVIDER_IMPORT_UNAVAILABLE", + Self::Disabled => "MESSAGES.PLAYLIST_UPDATE.INPUT_DISABLED", + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use shared::model::InputType; + + #[test] + fn input_update_capabilities_uses_shared_contract_for_every_type() { + for input_type in [ + InputType::Xtream, + InputType::Stalker, + InputType::M3u, + InputType::Plex, + InputType::Library, + InputType::Jellyfin, + InputType::Emby, + InputType::M3uBatch, + InputType::XtreamBatch, + InputType::StalkerBatch, + InputType::Staged, + ] { + for enabled in [false, true] { + let input = ConfigInputDto { input_type, enabled, ..ConfigInputDto::default() }; + assert_eq!( + InputUpdateCapabilities::for_input(&input), + InputUpdateCapabilities::for_input_type(input_type, enabled) + ); + } + } + } + + #[test] + fn input_update_capabilities_rescan_is_not_a_provider_policy_and_uses_targets() { + let capability = InputUpdateCapabilities::Rescan; + assert!(capability.policies().is_empty()); + assert!(capability.rebuilds_targets()); + assert_eq!(capability.selected_action(InputRefreshPolicy::NORMAL), InputUpdateAction::Rescan); + for unavailable in [ + InputUpdateCapabilities::BulkOnly, + InputUpdateCapabilities::ImportUnavailable, + InputUpdateCapabilities::Disabled, + ] { + assert!(!unavailable.rebuilds_targets()); + } + } + + #[test] + fn input_update_capabilities_notes_are_translated_and_explain_complete_rescan() -> Result<(), serde_json::Error> { + for source in [ + include_str!("../../public/assets/i18n/en.json"), + include_str!("../../public/assets/i18n/ru.json"), + include_str!("../../public/assets/i18n/ar.json"), + ] { + let locale: serde_json::Value = serde_json::from_str(source)?; + for key in ["LIBRARY_DESCRIPTION", "PROVIDER_IMPORT_UNAVAILABLE", "BULK_ONLY_DESCRIPTION"] { + assert!(locale["MESSAGES"]["PLAYLIST_UPDATE"][key].as_str().is_some_and(|text| !text.is_empty())); + } + for key in ["RESCAN", "START_RESCAN"] { + assert!(locale["LABEL"][key].as_str().is_some_and(|text| !text.is_empty())); + } + } + let en: serde_json::Value = serde_json::from_str(include_str!("../../public/assets/i18n/en.json"))?; + assert!(en["MESSAGES"]["PLAYLIST_UPDATE"]["LIBRARY_DESCRIPTION"] + .as_str() + .unwrap() + .contains("selected targets")); + Ok(()) + } +} diff --git a/frontend/src/model/input_update_card_model.rs b/frontend/src/model/input_update_card_model.rs new file mode 100644 index 000000000..e6ca9b1da --- /dev/null +++ b/frontend/src/model/input_update_card_model.rs @@ -0,0 +1,404 @@ +use crate::{ + app::{map_sources_to_playlist_rows, InputRow}, + model::{InputUpdateCapabilities, InputUpdateCapabilitiesExt}, +}; +use shared::model::{InputRefreshPolicy, InputType, PlaylistUpdateStatusDto, SourcesConfigDto}; +use std::{ + collections::{HashMap, HashSet}, + sync::Arc, +}; + +/// Target rendered in one input-update card. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct InputUpdateCardTarget { + pub id: u16, + pub name: String, +} + +/// Frontend read model for one configured input, including inputs without a manual action. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct InputUpdateCardModel { + pub input_id: u16, + pub input_name: Arc, + pub input_type: InputType, + pub capabilities: InputUpdateCapabilities, + pub targets: Vec, + pub last_update_at: Option, +} + +impl InputUpdateCardModel { + /// Builds the deterministic last-update presentation for a supplied clock value. + #[must_use] + pub fn last_update(&self, now: u64) -> LastUpdateViewModel { last_update_view_model(self.last_update_at, now) } +} + +/// Local, non-persisted controls for one input card. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct InputUpdateCardState { + pub selected_target_ids: Vec, + pub policy: InputRefreshPolicy, +} + +/// Localization-ready relative representation of a persisted update timestamp. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum LastUpdateRelative { + Never, + JustNow, + Minutes(u64), + Hours(u64), + LocalDateTime(u64), +} + +/// Relative value plus the source timestamp for a separately formatted local-time title. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct LastUpdateViewModel { + pub relative: LastUpdateRelative, + pub title_timestamp: Option, +} + +/// Classifies a persisted timestamp without reading the clock or browser locale. +#[must_use] +pub fn last_update_view_model(last_update_at: Option, now: u64) -> LastUpdateViewModel { + let relative = match last_update_at { + None => LastUpdateRelative::Never, + Some(timestamp) => { + let elapsed = now.saturating_sub(timestamp); + match elapsed { + 0..=59 => LastUpdateRelative::JustNow, + 60..=3_599 => LastUpdateRelative::Minutes(elapsed / 60), + 3_600..=86_399 => LastUpdateRelative::Hours(elapsed / 3_600), + _ => LastUpdateRelative::LocalDateTime(timestamp), + } + } + }; + LastUpdateViewModel { relative, title_timestamp: last_update_at } +} + +/// Builds one card per configured input ID in configuration order. +/// Aliases are accounts of their root input, not independently configured inputs. +#[must_use] +pub fn build_input_update_card_models( + sources: &SourcesConfigDto, + update_status: &PlaylistUpdateStatusDto, +) -> Vec { + let rows = map_sources_to_playlist_rows(sources); + let mut targets_by_input_id = HashMap::>::new(); + let mut target_ids_by_input_id = HashMap::>::new(); + + for (source_inputs, source_targets) in rows.iter() { + for row in source_inputs { + let InputRow::Input(input) = &**row else { + continue; + }; + let targets = targets_by_input_id.entry(input.id).or_default(); + let seen_target_ids = target_ids_by_input_id.entry(input.id).or_default(); + for target in source_targets { + if seen_target_ids.insert(target.id) { + targets.push(InputUpdateCardTarget { id: target.id, name: target.name.clone() }); + } + } + } + } + + let last_update_by_input_id = + update_status.inputs.iter().map(|status| (status.input_id, status.last_update)).collect::>(); + + let mut cards = Vec::with_capacity(sources.inputs.len()); + let mut seen_input_ids = HashSet::with_capacity(sources.inputs.len()); + for input in &sources.inputs { + if !seen_input_ids.insert(input.id) { + continue; + } + cards.push(InputUpdateCardModel { + input_id: input.id, + input_name: Arc::clone(&input.name), + input_type: input.input_type, + capabilities: InputUpdateCapabilities::for_input(input), + targets: targets_by_input_id.remove(&input.id).unwrap_or_default(), + last_update_at: last_update_by_input_id.get(&input.id).copied().flatten(), + }); + } + cards +} + +#[cfg(test)] +mod tests { + use super::{ + build_input_update_card_models, last_update_view_model, InputUpdateCapabilities, InputUpdateCapabilitiesExt, + InputUpdateCardModel, LastUpdateRelative, + }; + use shared::{ + model::{ + ConfigInputAliasDto, ConfigInputDto, ConfigSourceDto, ConfigTargetDto, InputPlaylistUpdateStatusDto, + InputRefreshPolicy, InputType, PlaylistUpdateStatusDto, SourcesConfigDto, + }, + utils::Internable, + }; + use std::sync::Arc; + + fn input(id: u16, name: &str, input_type: InputType) -> ConfigInputDto { + ConfigInputDto { id, name: name.intern(), input_type, enabled: true, ..ConfigInputDto::default() } + } + + fn target(id: u16, name: &str) -> ConfigTargetDto { + ConfigTargetDto { id, name: name.to_string(), ..ConfigTargetDto::default() } + } + + fn card_fixture() -> (SourcesConfigDto, PlaylistUpdateStatusDto) { + let sources = SourcesConfigDto { + inputs: vec![ + input(30, "zulu", InputType::Xtream), + input(10, "alpha", InputType::Stalker), + input(40, "without-targets", InputType::Xtream), + input(50, "m3u", InputType::M3u), + ], + sources: vec![ + ConfigSourceDto { + inputs: vec!["alpha".intern()], + targets: vec![target(2, "second"), target(1, "shared-name")], + }, + ConfigSourceDto { inputs: vec!["zulu".intern()], targets: vec![target(3, "third")] }, + ConfigSourceDto { + inputs: vec!["alpha".intern()], + targets: vec![target(2, "duplicate-id"), target(4, "shared-name")], + }, + ConfigSourceDto { inputs: vec!["without-targets".intern()], targets: Vec::new() }, + ConfigSourceDto { inputs: vec!["m3u".intern()], targets: vec![target(5, "m3u-target")] }, + ], + ..SourcesConfigDto::default() + }; + let status = PlaylistUpdateStatusDto { + inputs: vec![ + InputPlaylistUpdateStatusDto { + input_id: 10, + last_update: Some(700), + last_input_update: None, + clusters: Vec::new(), + }, + InputPlaylistUpdateStatusDto { + input_id: 30, + last_update: Some(500), + last_input_update: None, + clusters: Vec::new(), + }, + ], + ..PlaylistUpdateStatusDto::default() + }; + (sources, status) + } + + #[test] + fn input_update_card_model_preserves_input_and_target_configuration_order() { + let (sources, status) = card_fixture(); + + let cards = build_input_update_card_models(&sources, &status); + + assert_eq!(cards.iter().map(|card| card.input_id).collect::>(), vec![30, 10, 40, 50]); + assert_eq!(cards[1].targets.iter().map(|target| target.id).collect::>(), vec![2, 1, 4]); + assert_eq!(cards[0].input_type, InputType::Xtream); + assert_eq!(cards[1].input_type, InputType::Stalker); + assert_eq!(cards[3].input_type, InputType::M3u); + assert_eq!(cards[3].targets.iter().map(|target| target.id).collect::>(), vec![5]); + } + + #[test] + fn input_update_card_model_deduplicates_only_by_stable_target_id() { + let (sources, status) = card_fixture(); + + let cards = build_input_update_card_models(&sources, &status); + let alpha = cards.iter().find(|card| card.input_id == 10).expect("alpha card"); + + assert_eq!( + alpha.targets.iter().map(|target| target.name.as_str()).collect::>(), + ["second", "shared-name", "shared-name",] + ); + } + + #[test] + fn input_update_card_model_keeps_updateable_input_from_source_without_targets() { + let (sources, status) = card_fixture(); + + let cards = build_input_update_card_models(&sources, &status); + let card = cards.iter().find(|card| card.input_id == 40).expect("card without targets"); + + assert!(card.targets.is_empty()); + } + + #[test] + fn input_update_card_model_keeps_updateable_input_without_source_reference() { + let sources = SourcesConfigDto { + inputs: vec![input(7, "unreferenced", InputType::Xtream)], + sources: Vec::new(), + ..SourcesConfigDto::default() + }; + + let cards = build_input_update_card_models(&sources, &PlaylistUpdateStatusDto::default()); + + assert_eq!(cards.len(), 1); + assert_eq!(cards[0].input_id, 7); + assert_eq!(cards[0].input_type, InputType::Xtream); + assert!(cards[0].targets.is_empty()); + } + + #[test] + fn input_update_card_model_visibility_is_independent_of_action_capabilities() { + let mut disabled = input(2, "disabled", InputType::Xtream); + disabled.enabled = false; + let mut sources = SourcesConfigDto { + inputs: vec![ + input(1, "xtream", InputType::Xtream), + disabled, + input(3, "xtream-batch", InputType::XtreamBatch), + input(4, "stalker-batch", InputType::StalkerBatch), + input(5, "m3u", InputType::M3u), + input(6, "stalker", InputType::Stalker), + input(7, "m3u-batch", InputType::M3uBatch), + input(8, "staged", InputType::Staged), + input(9, "library", InputType::Library), + input(10, "plex", InputType::Plex), + input(11, "jellyfin", InputType::Jellyfin), + input(12, "emby", InputType::Emby), + ], + sources: Vec::new(), + ..SourcesConfigDto::default() + }; + sources.sources.push(ConfigSourceDto { + inputs: sources.inputs.iter().map(|input| Arc::clone(&input.name)).collect(), + targets: vec![target(2, "same-name"), target(1, "same-name"), target(2, "duplicate-id")], + }); + + let cards = build_input_update_card_models(&sources, &PlaylistUpdateStatusDto::default()); + + assert_eq!(cards.iter().map(|card| card.input_id).collect::>(), (1..=12).collect::>()); + assert_eq!(cards[1].capabilities, InputUpdateCapabilities::Disabled); + for card in &cards { + assert_eq!(card.targets.iter().map(|target| target.id).collect::>(), vec![2, 1]); + assert_eq!( + card.capabilities, + InputUpdateCapabilities::for_input(&sources.inputs[usize::from(card.input_id - 1)]) + ); + } + assert_eq!( + cards + .iter() + .filter(|card| card.capabilities.supports(InputRefreshPolicy::NORMAL)) + .map(|card| card.input_id) + .collect::>(), + vec![1, 5, 6, 10] + ); + } + + #[test] + fn input_update_card_model_keeps_duplicate_names_but_not_duplicate_ids_or_aliases() { + let first = ConfigInputDto { + aliases: Some(vec![ConfigInputAliasDto { + id: 99, + name: "alias".intern(), + ..ConfigInputAliasDto::default() + }]), + ..input(42, "same", InputType::M3u) + }; + let second = input(7, "same", InputType::Plex); + let duplicate = ConfigInputDto { name: "repeat".intern(), ..first.clone() }; + let sources = SourcesConfigDto { + inputs: vec![first, second, duplicate], + sources: vec![ + ConfigSourceDto { inputs: vec!["same".intern(), "same".intern()], targets: vec![target(8, "one")] }, + ConfigSourceDto { inputs: vec!["repeat".intern()], targets: vec![target(9, "two")] }, + ], + ..SourcesConfigDto::default() + }; + let status = PlaylistUpdateStatusDto { + inputs: vec![ + InputPlaylistUpdateStatusDto { + input_id: 7, + last_update: Some(700), + last_input_update: None, + clusters: Vec::new(), + }, + InputPlaylistUpdateStatusDto { + input_id: 42, + last_update: Some(500), + last_input_update: None, + clusters: Vec::new(), + }, + ], + ..PlaylistUpdateStatusDto::default() + }; + let cards = build_input_update_card_models(&sources, &status); + assert_eq!(cards.iter().map(|card| card.input_id).collect::>(), vec![42, 7]); + assert_eq!(cards[0].input_name, cards[1].input_name); + assert_eq!(cards[0].targets.iter().map(|target| target.id).collect::>(), vec![8, 9]); + assert_eq!(cards[1].targets.iter().map(|target| target.id).collect::>(), vec![8]); + assert_eq!(cards[0].last_update_at, Some(500)); + assert_eq!(cards[1].last_update_at, Some(700)); + } + + #[test] + fn input_update_card_model_maps_status_by_stable_input_id() { + let (sources, status) = card_fixture(); + let cards = build_input_update_card_models(&sources, &status); + + assert_eq!(cards[0].last_update_at, Some(500)); + assert_eq!(cards[1].last_update_at, Some(700)); + assert_eq!(cards[2].last_update_at, None); + } + + #[test] + fn last_update_covers_relative_time_boundaries() { + let now = 100_000; + let cases = [ + (None, LastUpdateRelative::Never), + (Some(now), LastUpdateRelative::JustNow), + (Some(now - 59), LastUpdateRelative::JustNow), + (Some(now - 60), LastUpdateRelative::Minutes(1)), + (Some(now - 3_599), LastUpdateRelative::Minutes(59)), + (Some(now - 3_600), LastUpdateRelative::Hours(1)), + (Some(now - 86_399), LastUpdateRelative::Hours(23)), + (Some(now - 86_400), LastUpdateRelative::LocalDateTime(now - 86_400)), + ]; + + for (timestamp, expected) in cases { + assert_eq!(last_update_view_model(timestamp, now).relative, expected); + } + } + + #[test] + fn last_update_treats_future_clock_skew_as_just_now() { + assert_eq!(last_update_view_model(Some(101), 100).relative, LastUpdateRelative::JustNow); + } + + #[test] + fn last_update_exposes_full_timestamp_separately_for_local_title_formatting() { + let view_model = last_update_view_model(Some(12_345), 12_405); + + assert_eq!(view_model.relative, LastUpdateRelative::Minutes(1)); + assert_eq!(view_model.title_timestamp, Some(12_345)); + assert_eq!(last_update_view_model(None, 12_405).title_timestamp, None); + } + + #[test] + fn playlist_update_view_reclassifies_last_update_from_one_shared_clock_value() { + let first = InputUpdateCardModel { + input_id: 1, + input_name: "first".intern(), + input_type: InputType::Xtream, + capabilities: InputUpdateCapabilities::for_input(&input(1, "first", InputType::Xtream)), + targets: Vec::new(), + last_update_at: Some(1_000), + }; + let second = InputUpdateCardModel { + input_id: 2, + input_name: "second".intern(), + input_type: InputType::Stalker, + capabilities: InputUpdateCapabilities::for_input(&input(2, "second", InputType::Stalker)), + targets: Vec::new(), + last_update_at: Some(940), + }; + + assert_eq!(first.last_update(1_059).relative, LastUpdateRelative::JustNow); + assert_eq!(second.last_update(1_059).relative, LastUpdateRelative::Minutes(1)); + assert_eq!(first.last_update(1_060).relative, LastUpdateRelative::Minutes(1)); + assert_eq!(second.last_update(1_060).relative, LastUpdateRelative::Minutes(2)); + } +} diff --git a/frontend/src/model/mod.rs b/frontend/src/model/mod.rs index e70b35aac..b0c3f966e 100644 --- a/frontend/src/model/mod.rs +++ b/frontend/src/model/mod.rs @@ -3,9 +3,24 @@ mod busy_status; mod dialog; mod event_message; mod explorer_source_type; +mod input_update_capabilities; +mod input_update_card_model; +mod pipeline_transparency_model; +mod playlist_update_card_status; +#[cfg(test)] +mod staged_completion_tests; mod web_config; pub use self::{ - background_transfer::*, busy_status::*, dialog::*, event_message::*, explorer_source_type::*, web_config::*, + background_transfer::*, + busy_status::*, + dialog::*, + event_message::*, + explorer_source_type::*, + input_update_capabilities::{InputUpdateCapabilities, InputUpdateCapabilitiesExt}, + input_update_card_model::*, + pipeline_transparency_model::*, + playlist_update_card_status::*, + web_config::*, }; pub use shared::model::view_type::*; diff --git a/frontend/src/model/pipeline_transparency_model.rs b/frontend/src/model/pipeline_transparency_model.rs new file mode 100644 index 000000000..33b19a458 --- /dev/null +++ b/frontend/src/model/pipeline_transparency_model.rs @@ -0,0 +1,1713 @@ +use crate::model::{InputUpdateCardModel, InputUpdateCardStatus, PlaylistUpdateCardStatuses}; +use shared::model::{ + ConfigInputDto, ConfigTargetDto, InputRefreshPolicy, InputType, LibraryScanResult, LibraryStatus, + PersistedPlaylistUpdateClusterSnapshot, PersistedPlaylistUpdateClusterState, + PersistedPlaylistUpdateClusterStatusDto, PersistedPlaylistUpdateQualityDecision, + PersistedPlaylistUpdateTechnicalState, PlaylistUpdateClusterDecision, PlaylistUpdateDataSource, + PlaylistUpdateInputTelemetry, PlaylistUpdateProgressDetail, PlaylistUpdateRunId, PlaylistUpdateRunOrder, + PlaylistUpdateStatusDto, ProcessingOrder, SourcesConfigDto, XtreamCluster, +}; +use std::{collections::HashMap, sync::Arc}; + +/// Frontend read model assembled from the existing configuration and update-run lifecycle. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct InputUpdateRunView { + /// Current canonical catalog, independent of the correlated run's scan metrics. + pub library_catalog: Option, + /// Real result for this effective run/input only; never reconstructed after reload. + pub library_scan_result: Option, + pub input_id: u16, + pub input_name: Arc, + pub run_id: Option, + pub execution_order: Option, + pub overall_state: InputUpdateCardStatus, + /// Input acquisition outcome; an input success does not prove target publication. + pub input_state: InputUpdateCardStatus, + pub refresh_policy: Option, + pub source_type: InputType, + /// Actual source of a non-cluster input, when existing runtime data proves it. + pub cache_source: Option, + pub last_update_at: Option, + pub progress_details: Vec, + /// Structured details safe for localized presentation. + pub localized_progress_details: Vec, + pub cluster_results: Vec, + pub target_results: Vec, +} + +impl InputUpdateRunView { + /// Content proven by the canonical catalog, not a fabricated per-cluster run outcome. + #[must_use] + pub fn catalog_content_clusters(&self) -> Vec { + self.library_catalog.as_ref().map_or_else(Vec::new, |catalog| { + [(XtreamCluster::Series, catalog.series), (XtreamCluster::Video, catalog.movies)] + .into_iter() + .filter_map(|(cluster, count)| (count > 0).then_some(cluster)) + .collect() + }) + } + /// Attaches the existing Library status read without changing run correlation. + #[must_use] + pub fn with_library_catalog(mut self, catalog: Option<&LibraryStatus>) -> Self { + self.library_catalog = + catalog.filter(|catalog| self.source_type == InputType::Library && catalog.enabled).cloned(); + self + } +} + +/// Available configuration and optional population facts for one real input cluster. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct InputClusterRunView { + pub cluster: XtreamCluster, + pub enabled_by_configuration: bool, + /// Actual run selection when reported by the processing lifecycle. + pub requested: Option, + /// Current run policy or the request-local policy retained by this cluster's last snapshot. + pub refresh_policy: Option, + /// `None` means that the configured quality guard is disabled for this cluster. + pub threshold: Option, + pub baseline_count: Option, + pub candidate_count: Option, + pub active_count: Option, + pub quality: Option, + /// Actual source used for this cluster; never inferred from the selected policy. + pub source: Option, + /// Typed runtime outcome; never recalculated from `quality` and `threshold`. + pub outcome: Option, + /// Persisted technical completion, kept separate from the Quality outcome. + pub technical_state: Option, + /// Last canonical persisted result, present only when no correlated run determines this card. + pub persisted_status: Option, +} + +/// Minimal reload fallback projected from the canonical persisted input status. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct PersistedClusterStatusView { + pub status: PersistedPlaylistUpdateClusterState, + pub timestamp: u64, + pub last_update: Option, +} + +/// Proven origin of the input population used by a run. +pub type InputDataSourceView = PlaylistUpdateDataSource; + +/// Proven cluster result supplied by the run lifecycle. +pub type InputClusterOutcomeView = PlaylistUpdateClusterDecision; + +/// Target configuration related to one input strictly through stable IDs. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct TargetRunView { + pub target_id: u16, + pub target_name: String, + pub input_ids: Vec, + pub pipeline: Option, +} + +/// Existing target processing configuration; no execution telemetry is inferred. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct TargetPipelineView { + pub processing_order: ProcessingOrder, + pub filter_summary: RuleStageSummary, + pub rename_summary: RuleStageSummary, + pub mapping_summary: RuleStageSummary, +} + +/// Count of configured entries for one target stage. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct RuleStageSummary { + pub configured_count: usize, +} + +/// Aggregates existing frontend runtime and configuration data without retaining another run history. +#[must_use] +pub fn build_input_update_run_views( + sources: &SourcesConfigDto, + cards: &[InputUpdateCardModel], + statuses: &PlaylistUpdateCardStatuses, + persisted_statuses: &PlaylistUpdateStatusDto, +) -> Vec { + let input_config_by_id = sources.inputs.iter().map(|input| (input.id, input)).collect::>(); + let target_config_by_id = sources + .sources + .iter() + .flat_map(|source| &source.targets) + .map(|target| (target.id, target)) + .collect::>(); + let input_ids_by_target_id = input_ids_by_target_id(cards); + let persisted_by_input_id = + persisted_statuses.inputs.iter().map(|status| (status.input_id, status)).collect::>(); + + cards + .iter() + .map(|card| { + let runtime = statuses.for_input(card.input_id); + let persisted = persisted_by_input_id.get(&card.input_id).filter(|_| runtime.run_id().is_none()); + let persisted_clusters = persisted.map_or(&[][..], |status| status.clusters.as_slice()); + let input_state = persisted + .and_then(|status| status.last_input_update) + .map_or(runtime.status(), |result| super::playlist_update_card_status::status_from_state(result.state)); + let cluster_results = input_config_by_id.get(&card.input_id).map_or_else(Vec::new, |input| { + cluster_views(input, card.input_type, runtime.input_telemetry(), persisted_clusters) + }); + let target_results = card + .targets + .iter() + .map(|target| TargetRunView { + target_id: target.id, + target_name: target.name.clone(), + input_ids: input_ids_by_target_id.get(&target.id).cloned().unwrap_or_default(), + pipeline: target_config_by_id.get(&target.id).map(|config| target_pipeline(config)), + }) + .collect(); + + InputUpdateRunView { + library_catalog: None, + library_scan_result: (card.input_type == InputType::Library) + .then(|| runtime.library_scan_result().cloned()) + .flatten(), + input_id: card.input_id, + input_name: Arc::clone(&card.input_name), + run_id: runtime.run_id().cloned(), + execution_order: runtime.execution_order(), + overall_state: runtime.action_status(), + input_state, + refresh_policy: runtime.refresh_policy(), + source_type: card.input_type, + cache_source: runtime.input_telemetry().and_then(|telemetry| telemetry.source), + last_update_at: card.last_update_at, + progress_details: runtime.details().to_vec(), + localized_progress_details: runtime.localized_details().to_vec(), + cluster_results, + target_results, + } + }) + .collect() +} + +fn input_ids_by_target_id(cards: &[InputUpdateCardModel]) -> HashMap> { + let mut input_ids = HashMap::>::new(); + for card in cards { + for target in &card.targets { + let target_input_ids = input_ids.entry(target.id).or_default(); + if !target_input_ids.contains(&card.input_id) { + target_input_ids.push(card.input_id); + } + } + } + input_ids +} + +fn cluster_views( + input: &ConfigInputDto, + input_type: InputType, + input_telemetry: Option<&PlaylistUpdateInputTelemetry>, + persisted_statuses: &[PersistedPlaylistUpdateClusterStatusDto], +) -> Vec { + let observed_clusters_only = input_type.is_m3u() || input_type == InputType::Staged; + if !(input_type.is_xtream() || input_type.is_stalker() || observed_clusters_only) { + return Vec::new(); + } + + let options = input.options.as_ref(); + let configured = [ + ( + XtreamCluster::Live, + options.is_none_or(|value| !value.skip_live), + options.map_or(0, |value| value.update_quality.live), + ), + ( + XtreamCluster::Video, + options.is_none_or(|value| !value.skip_vod), + options.map_or(0, |value| value.update_quality.vod), + ), + ( + XtreamCluster::Series, + options.is_none_or(|value| !value.skip_series), + options.map_or(0, |value| value.update_quality.series), + ), + ]; + configured + .into_iter() + .filter(|(cluster, _, _)| { + !observed_clusters_only + || input_telemetry + .is_some_and(|telemetry| telemetry.clusters.iter().any(|fact| fact.cluster == *cluster)) + || persisted_statuses.iter().any(|fact| fact.cluster == *cluster) + }) + .map(|(cluster, enabled_by_configuration, threshold)| { + let runtime = input_telemetry.and_then(|telemetry| { + telemetry.clusters.iter().find(|runtime_cluster| runtime_cluster.cluster == cluster) + }); + let persisted_status = if runtime.is_none() { + persisted_statuses.iter().find(|persisted| persisted.cluster == cluster).map(|persisted| { + PersistedClusterStatusView { + status: persisted.status, + timestamp: persisted.timestamp, + last_update: persisted.last_update, + } + }) + } else { + None + }; + let snapshot = persisted_status.and_then(|persisted| persisted.last_update); + let configured_threshold = (threshold > 0).then_some(threshold); + let ( + refresh_policy, + threshold, + baseline_count, + candidate_count, + active_count, + quality, + source, + outcome, + technical_state, + ) = if let Some(runtime) = runtime { + ( + input_telemetry.map(|telemetry| telemetry.refresh_policy), + runtime.threshold, + runtime.baseline_count, + runtime.candidate_count, + runtime.active_count, + runtime.quality, + runtime.source, + runtime.decision, + runtime.technical_state, + ) + } else if let Some(snapshot) = snapshot { + let quality = snapshot.quality; + let historical_threshold = quality.map(|quality| quality.threshold).or_else(|| { + snapshot + .quality_guard_threshold + .map_or(configured_threshold, |threshold| (threshold > 0).then_some(threshold)) + }); + ( + snapshot.policy, + historical_threshold, + quality.and_then(|quality| quality.baseline_count), + quality.and_then(|quality| quality.candidate_count), + snapshot.active_count, + quality.and_then(|quality| quality.achieved_quality), + snapshot.source, + quality.map(|quality| match quality.decision { + PersistedPlaylistUpdateQualityDecision::Accepted => PlaylistUpdateClusterDecision::Accepted, + PersistedPlaylistUpdateQualityDecision::Rejected => PlaylistUpdateClusterDecision::Rejected, + }), + snapshot.technical_state, + ) + } else { + (None, configured_threshold, None, None, None, None, None, None, None) + }; + InputClusterRunView { + cluster, + enabled_by_configuration, + requested: runtime.map(|runtime| runtime.requested), + refresh_policy, + threshold, + baseline_count, + candidate_count, + active_count, + quality, + source, + outcome, + technical_state, + persisted_status, + } + }) + .collect() +} + +fn target_pipeline(target: &ConfigTargetDto) -> TargetPipelineView { + TargetPipelineView { + processing_order: target.processing_order, + filter_summary: RuleStageSummary { configured_count: configured_filter_count(target) }, + rename_summary: RuleStageSummary { configured_count: target.rename.as_ref().map_or(0, Vec::len) }, + mapping_summary: RuleStageSummary { configured_count: target.mapping.as_ref().map_or(0, Vec::len) }, + } +} + +fn configured_filter_count(target: &ConfigTargetDto) -> usize { + [target.filter.processing.as_deref(), target.filter.persist.as_deref()] + .into_iter() + .flatten() + .filter(|filter| !filter.trim().is_empty()) + .count() +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::{ + model::{ + InputUpdateCapabilities, InputUpdateCapabilitiesExt, InputUpdateCardTarget, PlaylistUpdateAcceptedScope, + PlaylistUpdateCardStatusAction, + }, + services::WebSocketConnectionContext, + }; + use shared::{ + model::{ + ConfigInputOptionsDto, ConfigInputUpdateQualityDto, ConfigRenameDto, ConfigSourceDto, + ConfigTargetFilterDto, InputPlaylistUpdateStatusDto, ItemField, PersistedPlaylistUpdateQualitySnapshot, + PlaylistUpdateClusterTelemetry, PlaylistUpdateProgressEvent, PlaylistUpdateRunStateEvent, + PlaylistUpdateState, + }, + utils::Internable, + }; + use std::rc::Rc; + use yew::Reducible; + + fn input(id: u16, name: &str, input_type: InputType) -> ConfigInputDto { + ConfigInputDto { id, name: name.intern(), input_type, enabled: true, ..ConfigInputDto::default() } + } + + #[test] + fn update_overview_badges_m3u_reload_uses_typed_quality_and_live_run_wins() { + let input = input(27, "m3u", InputType::M3u); + let cards = vec![card(&input, &[], Some(123))]; + let sources = SourcesConfigDto { inputs: vec![input], ..SourcesConfigDto::default() }; + let mut persisted = + persisted_status(27, &[(XtreamCluster::Series, PersistedPlaylistUpdateClusterState::Failed, 123)]); + persisted.inputs[0].clusters[0].last_update = Some(PersistedPlaylistUpdateClusterSnapshot { + policy: Some(InputRefreshPolicy::REFRESH), + source: Some(InputDataSourceView::Provider), + quality: Some(PersistedPlaylistUpdateQualitySnapshot { + threshold: 90, + baseline_count: Some(10), + candidate_count: Some(0), + achieved_quality: Some(0), + decision: PersistedPlaylistUpdateQualityDecision::Rejected, + }), + active_count: Some(10), + technical_state: Some(PersistedPlaylistUpdateTechnicalState::Succeeded), + ..PersistedPlaylistUpdateClusterSnapshot::default() + }); + let project = |state: &PlaylistUpdateCardStatuses| { + build_input_update_run_views(&sources, &cards, state, &persisted).remove(0) + }; + let reload = project(&PlaylistUpdateCardStatuses::default()); + assert_eq!(reload.cluster_results.len(), 1, "Do not invent other content from the input type"); + let series = reload.cluster_results[0]; + assert_eq!(series.outcome, Some(InputClusterOutcomeView::Rejected)); + assert_eq!((series.threshold, series.quality, series.active_count), (Some(90), Some(0), Some(10))); + assert_eq!(series.technical_state, Some(PersistedPlaylistUpdateTechnicalState::Succeeded)); + let connection = WebSocketConnectionContext::new(1, true); + let queued = Rc::new(PlaylistUpdateCardStatuses::for_connection(connection)).reduce( + PlaylistUpdateCardStatusAction::Accepted(PlaylistUpdateAcceptedScope { + connection_context: connection, + run_id: "new-m3u".into(), + input_ids: vec![27], + refresh_policy: InputRefreshPolicy::NORMAL, + }), + ); + assert_eq!(project(&queued).overall_state, InputUpdateCardStatus::Queued); + assert!(project(&queued).cluster_results.is_empty(), "A queued run inherits no prior cluster outcome"); + let active = queued.reduce(PlaylistUpdateCardStatusAction::Progress { + connection_context: connection, + progress: PlaylistUpdateProgressEvent::for_run_input("new-m3u".into(), 2.into(), 27, "m3u", "current") + .with_input_telemetry(PlaylistUpdateInputTelemetry { + refresh_policy: InputRefreshPolicy::NORMAL, + source: None, + clusters: vec![PlaylistUpdateClusterTelemetry { + cluster: XtreamCluster::Series, + requested: true, + source: Some(InputDataSourceView::Cache), + baseline_count: None, + candidate_count: None, + active_count: None, + threshold: None, + quality: None, + decision: None, + technical_state: None, + }], + }), + }); + let current = project(&active); + assert_eq!(current.run_id.as_ref().map(AsRef::as_ref), Some("new-m3u")); + assert_eq!(current.overall_state, InputUpdateCardStatus::Updating); + assert_eq!(current.cluster_results[0].source, Some(InputDataSourceView::Cache)); + assert_eq!(current.cluster_results[0].persisted_status, None); + assert_eq!(current.cluster_results[0].outcome, None); + } + + #[test] + fn pipeline_transparency_playlist_update_status_delayed_sources_agree_before_and_after_reload() { + use shared::model::PersistedPlaylistUpdateInputResult; + for weaker in [PlaylistUpdateState::Success, PlaylistUpdateState::Partial] { + for failure_first in [false, true] { + let input = input(7, "renamed-shared-input", InputType::M3u); + let cards = vec![card(&input, &[], Some(123))]; + let sources = SourcesConfigDto { inputs: vec![input], ..SourcesConfigDto::default() }; + let connection = WebSocketConnectionContext::new(1, true); + let mut state = Rc::new(PlaylistUpdateCardStatuses::for_connection(connection)); + let results = if failure_first { + [PlaylistUpdateState::Failure, weaker] + } else { + [weaker, PlaylistUpdateState::Failure] + }; + for result in results { + state = state.reduce(PlaylistUpdateCardStatusAction::Progress { + connection_context: connection, + progress: PlaylistUpdateProgressEvent::input_completed( + "shared-run".into(), + 1.into(), + 7, + result, + "old-name", + "source completion", + ), + }); + } + let live = build_input_update_run_views(&sources, &cards, &state, &PlaylistUpdateStatusDto::default()) + .remove(0); + assert_eq!(live.input_state, InputUpdateCardStatus::Failed); + assert_eq!(live.run_id, Some("shared-run".into())); + // The canonical writer's order-independent result, carried through + // the existing DTO; a reload has no run or target facts to infer. + let persisted = PlaylistUpdateStatusDto { + inputs: vec![InputPlaylistUpdateStatusDto { + input_id: 7, + last_update: Some(123), + clusters: Vec::new(), + last_input_update: Some(PersistedPlaylistUpdateInputResult { + state: PlaylistUpdateState::Failure, + timestamp: 456, + }), + }], + ..PlaylistUpdateStatusDto::default() + }; + let wire = serde_json::to_vec(&persisted).unwrap(); + let reloaded = build_input_update_run_views( + &sources, + &cards, + &PlaylistUpdateCardStatuses::default(), + &serde_json::from_slice(&wire).unwrap(), + ) + .remove(0); + assert_eq!(reloaded.input_state, live.input_state); + assert_eq!(reloaded.overall_state, InputUpdateCardStatus::Ready); + assert_eq!(reloaded.run_id, None); + assert_eq!(reloaded.cache_source, None); + assert!(reloaded.cluster_results.is_empty()); + } + } + } + + #[test] + fn pipeline_transparency_playlist_update_status_staged_reload_uses_own_id_not_parent_result() { + use shared::model::PersistedPlaylistUpdateInputResult; + let parent = input(7, "parent", InputType::Xtream); + let staged = input(8, "renamed-staged", InputType::Staged); + let cards = vec![card(&staged, &[], Some(123)), card(&parent, &[], Some(123))]; + let sources = SourcesConfigDto { inputs: vec![parent, staged], ..SourcesConfigDto::default() }; + for (own, parent_state, expected) in [ + (Some(PlaylistUpdateState::Success), PlaylistUpdateState::Failure, InputUpdateCardStatus::Success), + (Some(PlaylistUpdateState::Failure), PlaylistUpdateState::Success, InputUpdateCardStatus::Failed), + (None, PlaylistUpdateState::Success, InputUpdateCardStatus::Ready), + ] { + let persisted = PlaylistUpdateStatusDto { + inputs: vec![ + InputPlaylistUpdateStatusDto { + input_id: 7, + last_update: Some(123), + clusters: Vec::new(), + last_input_update: Some(PersistedPlaylistUpdateInputResult { + state: parent_state, + timestamp: 456, + }), + }, + InputPlaylistUpdateStatusDto { + input_id: 8, + last_update: Some(123), + clusters: vec![PersistedPlaylistUpdateClusterStatusDto { + cluster: XtreamCluster::Live, + status: PersistedPlaylistUpdateClusterState::Ok, + timestamp: 123, + last_update: Some(PersistedPlaylistUpdateClusterSnapshot { + policy: Some(InputRefreshPolicy::REFRESH), + source: Some(PlaylistUpdateDataSource::Cache), + quality_guard_threshold: Some(95), + ..PersistedPlaylistUpdateClusterSnapshot::default() + }), + }], + last_input_update: own + .map(|state| PersistedPlaylistUpdateInputResult { state, timestamp: 456 }), + }, + ], + ..PlaylistUpdateStatusDto::default() + }; + let view = + build_input_update_run_views(&sources, &cards, &PlaylistUpdateCardStatuses::default(), &persisted) + .remove(0); + assert_eq!(view.input_id, 8); + assert_eq!(view.input_state, expected); + assert_eq!(view.cluster_results.len(), 1); + assert_eq!(view.cluster_results[0].cluster, XtreamCluster::Live); + assert_eq!(view.cluster_results[0].source, Some(PlaylistUpdateDataSource::Cache)); + assert_eq!(view.cache_source, None); + assert_eq!(view.refresh_policy, None); + assert_eq!(view.overall_state, InputUpdateCardStatus::Ready); + } + } + + #[test] + fn pipeline_transparency_persisted_input_status_restores_all_types_without_runtime_or_target_inference() { + use shared::model::PersistedPlaylistUpdateInputResult; + for input_type in [ + InputType::Xtream, + InputType::XtreamBatch, + InputType::Stalker, + InputType::StalkerBatch, + InputType::M3u, + InputType::M3uBatch, + InputType::Library, + InputType::Plex, + InputType::Emby, + InputType::Jellyfin, + InputType::Staged, + ] { + for enabled in [false, true] { + let mut input = input(7, "renamed-display-name", input_type); + input.enabled = enabled; + let cards = vec![card(&input, &[], Some(123))]; + let sources = SourcesConfigDto { inputs: vec![input], ..SourcesConfigDto::default() }; + for (state, expected) in [ + (PlaylistUpdateState::Success, InputUpdateCardStatus::Success), + (PlaylistUpdateState::Partial, InputUpdateCardStatus::Partial), + (PlaylistUpdateState::Failure, InputUpdateCardStatus::Failed), + ] { + let persisted = PlaylistUpdateStatusDto { + inputs: vec![InputPlaylistUpdateStatusDto { + input_id: 7, + last_update: Some(123), + last_input_update: Some(PersistedPlaylistUpdateInputResult { state, timestamp: 456 }), + clusters: Vec::new(), + }], + ..PlaylistUpdateStatusDto::default() + }; + let view = build_input_update_run_views( + &sources, + &cards, + &PlaylistUpdateCardStatuses::default(), + &persisted, + ) + .remove(0); + assert_eq!(view.input_state, expected, "{input_type}"); + assert_eq!( + view.overall_state, + InputUpdateCardStatus::Ready, + "input completion does not prove target publication" + ); + assert_eq!(view.last_update_at, Some(123)); + assert_eq!(view.run_id, None); + assert_eq!(view.cache_source, None); + assert_eq!(view.refresh_policy, None); + assert_eq!(view.library_scan_result, None); + assert!(view.progress_details.is_empty()); + assert_eq!(view.cluster_results.is_empty(), !(input_type.is_xtream() || input_type.is_stalker())); + assert!(view.cluster_results.iter().all(|cluster| cluster.outcome.is_none() + && cluster.quality.is_none() + && cluster.source.is_none())); + let unrelated = PlaylistUpdateStatusDto { + inputs: vec![InputPlaylistUpdateStatusDto { input_id: 99, ..persisted.inputs[0].clone() }], + ..PlaylistUpdateStatusDto::default() + }; + assert_eq!( + build_input_update_run_views( + &sources, + &cards, + &PlaylistUpdateCardStatuses::default(), + &unrelated + )[0] + .input_state, + InputUpdateCardStatus::Ready + ); + } + assert_eq!( + build_input_update_run_views( + &sources, + &cards, + &PlaylistUpdateCardStatuses::default(), + &PlaylistUpdateStatusDto::default() + )[0] + .input_state, + InputUpdateCardStatus::Ready + ); + } + } + } + + #[test] + fn pipeline_transparency_persisted_input_status_never_supplements_current_run_and_reloads_after_reset() { + use shared::model::PersistedPlaylistUpdateInputResult; + for input_type in [InputType::M3u, InputType::Library, InputType::Plex, InputType::Xtream, InputType::Stalker] { + let input = input(7, "input", input_type); + let cards = vec![card(&input, &[], Some(123))]; + let sources = SourcesConfigDto { inputs: vec![input], ..SourcesConfigDto::default() }; + let mut persisted = PlaylistUpdateStatusDto { + inputs: vec![InputPlaylistUpdateStatusDto { + input_id: 7, + last_update: Some(123), + clusters: Vec::new(), + last_input_update: Some(PersistedPlaylistUpdateInputResult { + state: PlaylistUpdateState::Failure, + timestamp: 123, + }), + }], + ..PlaylistUpdateStatusDto::default() + }; + let connection = WebSocketConnectionContext::new(1, true); + let mut state = Rc::new(PlaylistUpdateCardStatuses::for_connection(connection)).reduce( + PlaylistUpdateCardStatusAction::Accepted(PlaylistUpdateAcceptedScope { + connection_context: connection, + run_id: "current".into(), + input_ids: vec![7], + refresh_policy: InputRefreshPolicy::REFRESH, + }), + ); + for expected in + [InputUpdateCardStatus::Queued, InputUpdateCardStatus::Updating, InputUpdateCardStatus::Success] + { + let view = build_input_update_run_views(&sources, &cards, &state, &persisted).remove(0); + assert_eq!(view.input_state, expected); + assert_eq!(view.run_id, Some("current".into())); + assert_eq!(view.refresh_policy, Some(InputRefreshPolicy::REFRESH)); + if expected == InputUpdateCardStatus::Queued { + state = state.reduce(PlaylistUpdateCardStatusAction::Progress { + connection_context: connection, + progress: PlaylistUpdateProgressEvent::for_run_input( + "current".into(), + 1.into(), + 7, + "input", + "running", + ), + }); + } else if expected == InputUpdateCardStatus::Updating { + state = state.reduce(PlaylistUpdateCardStatusAction::Progress { + connection_context: connection, + progress: PlaylistUpdateProgressEvent::input_completed( + "current".into(), + 1.into(), + 7, + PlaylistUpdateState::Success, + "input", + "completed", + ), + }); + } + } + state = state.reduce(PlaylistUpdateCardStatusAction::Completed { + connection_context: connection, + completed: PlaylistUpdateRunStateEvent::correlated( + "current".into(), + 1.into(), + PlaylistUpdateState::Failure, + ), + }); + let view = build_input_update_run_views(&sources, &cards, &state, &persisted).remove(0); + assert_eq!(view.input_state, InputUpdateCardStatus::Success); + assert_eq!(view.overall_state, InputUpdateCardStatus::Failed); + persisted.inputs[0].last_input_update = + Some(PersistedPlaylistUpdateInputResult { state: PlaylistUpdateState::Success, timestamp: 456 }); + let reloaded = state + .reduce(PlaylistUpdateCardStatusAction::ConnectionChanged(WebSocketConnectionContext::new(2, true))); + let view = build_input_update_run_views(&sources, &cards, &reloaded, &persisted).remove(0); + assert_eq!(view.input_state, InputUpdateCardStatus::Success); + assert_eq!(view.run_id, None); + assert!(view.progress_details.is_empty()); + assert_eq!(view.refresh_policy, None); + } + } + + #[test] + fn pipeline_transparency_playlist_update_status_library_scan_stays_with_rebuilding_run_and_reload_is_unknown() { + let input = input(17, "library", InputType::Library); + let cards = vec![card(&input, &[], Some(1234))]; + let sources = SourcesConfigDto { inputs: vec![input], ..SourcesConfigDto::default() }; + let connection = WebSocketConnectionContext::new(1, true); + let catalog = LibraryStatus { enabled: true, movies: 2, series: 3, episodes: 19, total_items: 5, path: None }; + let scan = LibraryScanResult { + files_scanned: 12, + groups_scanned: 7, + files_added: 3, + files_updated: 2, + files_removed: 1, + errors: 0, + }; + let project = |state: &PlaylistUpdateCardStatuses| { + build_input_update_run_views(&sources, &cards, state, &PlaylistUpdateStatusDto::default()) + .remove(0) + .with_library_catalog(Some(&catalog)) + }; + let accepted = |run: &str| { + PlaylistUpdateCardStatusAction::Accepted(PlaylistUpdateAcceptedScope { + connection_context: connection, + run_id: run.into(), + input_ids: vec![17], + refresh_policy: InputRefreshPolicy::NORMAL, + }) + }; + let progress = + |event| PlaylistUpdateCardStatusAction::Progress { connection_context: connection, progress: event }; + let completed = |run: &str, order: u64, state| PlaylistUpdateCardStatusAction::Completed { + connection_context: connection, + completed: PlaylistUpdateRunStateEvent { + run_id: Some(run.into()), + execution_order: Some(order.into()), + state, + }, + }; + for terminal in [PlaylistUpdateState::Success, PlaylistUpdateState::Partial, PlaylistUpdateState::Failure] { + let fresh = Rc::new(PlaylistUpdateCardStatuses::for_connection(connection)); + let reload = project(&fresh); + assert_eq!(reload.library_catalog, Some(catalog.clone())); + assert_eq!(reload.library_scan_result, None); + assert!(reload.cluster_results.is_empty()); + assert_eq!(reload.last_update_at, Some(1234)); + let state = fresh + .reduce(accepted("r1")) + .reduce(progress( + PlaylistUpdateProgressEvent::for_run_input("r1".into(), 1.into(), 17, "library", "scan finished") + .with_library_scan_result(scan.clone()), + )) + .reduce(progress(PlaylistUpdateProgressEvent::input_completed( + "r1".into(), + 1.into(), + 17, + PlaylistUpdateState::Success, + "library", + "input finished", + ))); + let state = state.reduce(accepted("r2")); + let rebuilding = project(&state); + assert_eq!(rebuilding.run_id, Some("r1".into())); + assert_eq!(rebuilding.input_state, InputUpdateCardStatus::Success); + assert_eq!(rebuilding.overall_state, InputUpdateCardStatus::Updating); + assert_eq!(rebuilding.library_scan_result, Some(scan.clone())); + let state = state.reduce(completed("r1", 1_u64, terminal)); + let queued = project(&state); + assert_eq!(queued.run_id, Some("r2".into())); + assert_eq!(queued.overall_state, InputUpdateCardStatus::Queued); + assert_eq!(queued.library_scan_result, None); + assert!(queued.progress_details.is_empty()); + assert_eq!(project(&state.clone().reduce(accepted("r1"))), queued); + let state = state.reduce(progress(PlaylistUpdateProgressEvent::for_run_input( + "r2".into(), + 2.into(), + 17, + "library", + "scan started", + ))); + assert_eq!(project(&state).library_scan_result, None); + let failed_scan = LibraryScanResult { errors: 4, ..scan.clone() }; + let state = state + .reduce(progress( + PlaylistUpdateProgressEvent::for_run_input("r2".into(), 2.into(), 17, "library", "scan result") + .with_library_scan_result(failed_scan.clone()), + )) + .reduce(progress(PlaylistUpdateProgressEvent::input_completed( + "r2".into(), + 2.into(), + 17, + PlaylistUpdateState::Failure, + "library", + "failed", + ))) + .reduce(completed("r2", 2, PlaylistUpdateState::Failure)); + let failed = project(&state); + assert_eq!(failed.library_scan_result, Some(failed_scan)); + assert_eq!(failed.overall_state, InputUpdateCardStatus::Failed); + assert_eq!(failed.refresh_policy, Some(InputRefreshPolicy::NORMAL)); + assert_eq!(failed.library_catalog, Some(catalog.clone())); + let reset = state + .reduce(PlaylistUpdateCardStatusAction::ConnectionChanged(WebSocketConnectionContext::new(2, true))); + assert_eq!(project(&reset).library_scan_result, None); + } + } + + #[test] + fn pipeline_transparency_library_projection_does_not_replace_other_provider_facts() { + for input_type in [InputType::Xtream, InputType::Stalker, InputType::M3u, InputType::Plex] { + let input = input(7, "provider", input_type); + let cards = vec![card(&input, &[], None)]; + let sources = SourcesConfigDto { inputs: vec![input], ..SourcesConfigDto::default() }; + let connection = WebSocketConnectionContext::new(1, true); + let telemetry = PlaylistUpdateInputTelemetry { + refresh_policy: InputRefreshPolicy::REFRESH, + source: Some(PlaylistUpdateDataSource::Provider), + clusters: vec![], + }; + let state = Rc::new(PlaylistUpdateCardStatuses::for_connection(connection)).reduce( + PlaylistUpdateCardStatusAction::Progress { + connection_context: connection, + progress: PlaylistUpdateProgressEvent::for_run_input( + "provider".into(), + 1.into(), + 7, + "provider", + "loaded", + ) + .with_input_telemetry(telemetry), + }, + ); + let view = build_input_update_run_views(&sources, &cards, &state, &PlaylistUpdateStatusDto::default()) + .remove(0) + .with_library_catalog(Some(&LibraryStatus { enabled: true, ..LibraryStatus::default() })); + assert_eq!(view.cache_source, Some(PlaylistUpdateDataSource::Provider)); + assert_eq!(view.refresh_policy, Some(InputRefreshPolicy::REFRESH)); + assert_eq!(view.library_catalog, None); + assert_eq!(view.library_scan_result, None); + } + } + + fn card(input: &ConfigInputDto, targets: &[(u16, &str)], last_update_at: Option) -> InputUpdateCardModel { + InputUpdateCardModel { + input_id: input.id, + input_name: Arc::clone(&input.name), + input_type: input.input_type, + capabilities: InputUpdateCapabilities::for_input(input), + targets: targets + .iter() + .map(|(id, name)| InputUpdateCardTarget { id: *id, name: (*name).to_string() }) + .collect(), + last_update_at, + } + } + + fn target(id: u16, name: &str, processing_order: ProcessingOrder) -> ConfigTargetDto { + ConfigTargetDto { id, name: name.to_string(), processing_order, ..ConfigTargetDto::default() } + } + + fn rename() -> ConfigRenameDto { + ConfigRenameDto { + field: ItemField::Name, + pattern: "old".to_string(), + new_name: "new".to_string(), + t_pattern: None, + } + } + + fn persisted_status( + input_id: u16, + clusters: &[(XtreamCluster, PersistedPlaylistUpdateClusterState, u64)], + ) -> PlaylistUpdateStatusDto { + PlaylistUpdateStatusDto { + inputs: vec![InputPlaylistUpdateStatusDto { + input_id, + last_update: clusters.iter().map(|(_, _, timestamp)| *timestamp).max(), + last_input_update: None, + clusters: clusters + .iter() + .map(|(cluster, status, timestamp)| PersistedPlaylistUpdateClusterStatusDto { + cluster: *cluster, + status: *status, + timestamp: *timestamp, + last_update: None, + }) + .collect(), + }], + ..PlaylistUpdateStatusDto::default() + } + } + + fn statuses_with_manual_multi_input_run() -> Rc { + let connection = WebSocketConnectionContext::new(4, true); + Rc::new(PlaylistUpdateCardStatuses::for_connection(connection)) + .reduce(PlaylistUpdateCardStatusAction::Accepted(PlaylistUpdateAcceptedScope { + connection_context: connection, + run_id: "run-stable".into(), + input_ids: vec![7], + refresh_policy: InputRefreshPolicy::FORCE, + })) + .reduce(PlaylistUpdateCardStatusAction::Progress { + connection_context: connection, + progress: PlaylistUpdateProgressEvent::for_run_input( + "run-stable".into(), + PlaylistUpdateRunOrder::from(17), + 7, + "display-name", + "accepted input progress", + ), + }) + .reduce(PlaylistUpdateCardStatusAction::Progress { + connection_context: connection, + progress: PlaylistUpdateProgressEvent::for_run_input( + "run-stable".into(), + PlaylistUpdateRunOrder::from(17), + 9, + "same-display-name", + "observed dependency progress", + ), + }) + } + + #[test] + fn persisted_cluster_status_restores_reload_fallback_without_runtime_inference() { + let mut xtream = input(7, "xtream", InputType::Xtream); + xtream.options = Some(ConfigInputOptionsDto { + update_quality: ConfigInputUpdateQualityDto { live: 90, vod: 80, series: 100 }, + ..ConfigInputOptionsDto::default() + }); + let m3u = input(8, "m3u", InputType::M3u); + let sources = SourcesConfigDto { + inputs: vec![xtream.clone(), m3u.clone()], + sources: Vec::new(), + ..SourcesConfigDto::default() + }; + let cards = vec![card(&xtream, &[], Some(303)), card(&m3u, &[], Some(404))]; + let persisted = persisted_status( + 7, + &[ + (XtreamCluster::Live, PersistedPlaylistUpdateClusterState::Ok, 101), + (XtreamCluster::Video, PersistedPlaylistUpdateClusterState::Failed, 202), + (XtreamCluster::Series, PersistedPlaylistUpdateClusterState::Ok, 303), + ], + ); + + let views = build_input_update_run_views(&sources, &cards, &PlaylistUpdateCardStatuses::default(), &persisted); + + assert_eq!(views[0].run_id, None); + assert_eq!( + views[0] + .cluster_results + .iter() + .map(|cluster| cluster.persisted_status.map(|status| status.status)) + .collect::>(), + [ + Some(PersistedPlaylistUpdateClusterState::Ok), + Some(PersistedPlaylistUpdateClusterState::Failed), + Some(PersistedPlaylistUpdateClusterState::Ok), + ] + ); + assert_eq!( + views[0] + .cluster_results + .iter() + .map(|cluster| cluster.persisted_status.map(|status| status.timestamp)) + .collect::>(), + [Some(101), Some(202), Some(303)] + ); + assert_eq!( + views[0].cluster_results.iter().map(|cluster| cluster.threshold).collect::>(), + [Some(90), Some(80), Some(100)] + ); + assert!(views[0].cluster_results.iter().all(|cluster| { + cluster.requested.is_none() + && cluster.baseline_count.is_none() + && cluster.candidate_count.is_none() + && cluster.active_count.is_none() + && cluster.quality.is_none() + && cluster.source.is_none() + && cluster.outcome.is_none() + })); + assert!(views[1].cluster_results.is_empty(), "M3U must not receive persisted synthetic clusters"); + } + + #[test] + fn persisted_cluster_status_snapshot_restores_typed_facts_and_keeps_technical_failure_separate() { + let mut xtream = input(7, "xtream", InputType::Xtream); + xtream.options = Some(ConfigInputOptionsDto { + update_quality: ConfigInputUpdateQualityDto { live: 80, vod: 80, series: 80 }, + ..ConfigInputOptionsDto::default() + }); + let sources = + SourcesConfigDto { inputs: vec![xtream.clone()], sources: Vec::new(), ..SourcesConfigDto::default() }; + let cards = vec![card(&xtream, &[], Some(303))]; + let accepted_quality = PersistedPlaylistUpdateQualitySnapshot { + threshold: 90, + baseline_count: Some(1_000), + candidate_count: Some(950), + achieved_quality: Some(95), + decision: PersistedPlaylistUpdateQualityDecision::Accepted, + }; + let persisted = PlaylistUpdateStatusDto { + inputs: vec![InputPlaylistUpdateStatusDto { + input_id: 7, + last_update: Some(303), + last_input_update: None, + clusters: vec![ + PersistedPlaylistUpdateClusterStatusDto { + cluster: XtreamCluster::Live, + status: PersistedPlaylistUpdateClusterState::Ok, + timestamp: 101, + last_update: Some(PersistedPlaylistUpdateClusterSnapshot { + policy: Some(InputRefreshPolicy::REFRESH), + source: Some(PlaylistUpdateDataSource::Provider), + quality_guard_threshold: None, + quality: Some(accepted_quality), + active_count: Some(950), + technical_state: Some(PersistedPlaylistUpdateTechnicalState::Succeeded), + }), + }, + PersistedPlaylistUpdateClusterStatusDto { + cluster: XtreamCluster::Video, + status: PersistedPlaylistUpdateClusterState::Failed, + timestamp: 202, + last_update: Some(PersistedPlaylistUpdateClusterSnapshot { + policy: None, + source: Some(PlaylistUpdateDataSource::Provider), + quality_guard_threshold: None, + quality: Some(PersistedPlaylistUpdateQualitySnapshot { + threshold: 95, + baseline_count: Some(500), + candidate_count: Some(300), + achieved_quality: Some(60), + decision: PersistedPlaylistUpdateQualityDecision::Rejected, + }), + active_count: Some(500), + technical_state: Some(PersistedPlaylistUpdateTechnicalState::Succeeded), + }), + }, + PersistedPlaylistUpdateClusterStatusDto { + cluster: XtreamCluster::Series, + status: PersistedPlaylistUpdateClusterState::Failed, + timestamp: 303, + last_update: Some(PersistedPlaylistUpdateClusterSnapshot { + policy: None, + source: Some(PlaylistUpdateDataSource::Provider), + quality_guard_threshold: None, + quality: Some(accepted_quality), + active_count: None, + technical_state: Some(PersistedPlaylistUpdateTechnicalState::Failed), + }), + }, + ], + }], + ..PlaylistUpdateStatusDto::default() + }; + + let views = build_input_update_run_views(&sources, &cards, &PlaylistUpdateCardStatuses::default(), &persisted); + let live = views[0].cluster_results[0]; + assert_eq!(live.refresh_policy, Some(InputRefreshPolicy::REFRESH)); + assert_eq!(live.source, Some(PlaylistUpdateDataSource::Provider)); + assert_eq!(live.outcome, Some(PlaylistUpdateClusterDecision::Accepted)); + assert_eq!((live.threshold, live.quality), (Some(90), Some(95))); + assert_eq!((live.baseline_count, live.candidate_count, live.active_count), (Some(1_000), Some(950), Some(950))); + assert_eq!(live.technical_state, Some(PersistedPlaylistUpdateTechnicalState::Succeeded)); + + let video = views[0].cluster_results[1]; + assert_eq!(video.outcome, Some(PlaylistUpdateClusterDecision::Rejected)); + assert_eq!(video.active_count, Some(500)); + assert_eq!(video.technical_state, Some(PersistedPlaylistUpdateTechnicalState::Succeeded)); + + let series = views[0].cluster_results[2]; + assert_eq!(series.outcome, Some(PlaylistUpdateClusterDecision::Accepted)); + assert_eq!(series.technical_state, Some(PersistedPlaylistUpdateTechnicalState::Failed)); + assert_eq!(series.active_count, None); + } + + #[test] + fn persisted_cluster_status_snapshot_keeps_bootstrap_cache_and_disabled_force_facts_sparse() { + let mut xtream = input(7, "xtream", InputType::Xtream); + xtream.options = Some(ConfigInputOptionsDto { + update_quality: ConfigInputUpdateQualityDto { live: 90, vod: 90, series: 0 }, + ..ConfigInputOptionsDto::default() + }); + let sources = + SourcesConfigDto { inputs: vec![xtream.clone()], sources: Vec::new(), ..SourcesConfigDto::default() }; + let cards = vec![card(&xtream, &[], Some(303))]; + let persisted = PlaylistUpdateStatusDto { + inputs: vec![InputPlaylistUpdateStatusDto { + input_id: 7, + last_update: Some(303), + last_input_update: None, + clusters: vec![ + PersistedPlaylistUpdateClusterStatusDto { + cluster: XtreamCluster::Live, + status: PersistedPlaylistUpdateClusterState::Ok, + timestamp: 101, + last_update: Some(PersistedPlaylistUpdateClusterSnapshot { + policy: None, + source: Some(PlaylistUpdateDataSource::Provider), + quality_guard_threshold: None, + quality: Some(PersistedPlaylistUpdateQualitySnapshot { + threshold: 90, + baseline_count: None, + candidate_count: Some(42), + achieved_quality: None, + decision: PersistedPlaylistUpdateQualityDecision::Accepted, + }), + active_count: Some(42), + technical_state: Some(PersistedPlaylistUpdateTechnicalState::Succeeded), + }), + }, + PersistedPlaylistUpdateClusterStatusDto { + cluster: XtreamCluster::Video, + status: PersistedPlaylistUpdateClusterState::Ok, + timestamp: 202, + last_update: Some(PersistedPlaylistUpdateClusterSnapshot { + policy: Some(InputRefreshPolicy::REFRESH), + source: Some(PlaylistUpdateDataSource::Cache), + quality_guard_threshold: Some(90), + quality: None, + active_count: None, + technical_state: Some(PersistedPlaylistUpdateTechnicalState::Succeeded), + }), + }, + PersistedPlaylistUpdateClusterStatusDto { + cluster: XtreamCluster::Series, + status: PersistedPlaylistUpdateClusterState::Ok, + timestamp: 303, + last_update: Some(PersistedPlaylistUpdateClusterSnapshot { + policy: Some(InputRefreshPolicy::FORCE), + source: Some(PlaylistUpdateDataSource::Provider), + quality_guard_threshold: Some(0), + quality: None, + active_count: Some(12), + technical_state: Some(PersistedPlaylistUpdateTechnicalState::Succeeded), + }), + }, + ], + }], + ..PlaylistUpdateStatusDto::default() + }; + + let views = build_input_update_run_views(&sources, &cards, &PlaylistUpdateCardStatuses::default(), &persisted); + let live = views[0].cluster_results[0]; + assert_eq!(live.outcome, Some(PlaylistUpdateClusterDecision::Accepted)); + assert_eq!((live.baseline_count, live.candidate_count), (None, Some(42))); + assert_eq!((live.threshold, live.quality), (Some(90), None)); + assert_eq!(live.refresh_policy, None, "an automatic run must not acquire a default policy"); + + let video = views[0].cluster_results[1]; + assert_eq!(video.source, Some(PlaylistUpdateDataSource::Cache)); + assert_eq!(video.refresh_policy, Some(InputRefreshPolicy::REFRESH)); + assert_eq!(video.outcome, None); + + let series = views[0].cluster_results[2]; + assert_eq!(series.refresh_policy, Some(InputRefreshPolicy::FORCE)); + assert_eq!(series.threshold, None); + assert_eq!(series.quality, None); + assert_eq!(series.active_count, Some(12)); + } + + #[test] + fn persisted_cluster_status_snapshot_uses_historical_guard_before_current_configuration() { + let mut xtream = input(7, "xtream", InputType::Xtream); + xtream.options = Some(ConfigInputOptionsDto { + update_quality: ConfigInputUpdateQualityDto { live: 0, vod: 95, series: 80 }, + ..ConfigInputOptionsDto::default() + }); + let sources = + SourcesConfigDto { inputs: vec![xtream.clone()], sources: Vec::new(), ..SourcesConfigDto::default() }; + let cards = vec![card(&xtream, &[], Some(303))]; + let force_snapshot = |quality_guard_threshold| PersistedPlaylistUpdateClusterSnapshot { + policy: Some(InputRefreshPolicy::FORCE), + source: Some(PlaylistUpdateDataSource::Provider), + quality_guard_threshold, + quality: None, + active_count: Some(42), + technical_state: Some(PersistedPlaylistUpdateTechnicalState::Succeeded), + }; + let persisted = PlaylistUpdateStatusDto { + inputs: vec![InputPlaylistUpdateStatusDto { + input_id: 7, + last_update: Some(303), + last_input_update: None, + clusters: vec![ + PersistedPlaylistUpdateClusterStatusDto { + cluster: XtreamCluster::Live, + status: PersistedPlaylistUpdateClusterState::Ok, + timestamp: 101, + last_update: Some(force_snapshot(Some(95))), + }, + PersistedPlaylistUpdateClusterStatusDto { + cluster: XtreamCluster::Video, + status: PersistedPlaylistUpdateClusterState::Ok, + timestamp: 202, + last_update: Some(force_snapshot(Some(0))), + }, + PersistedPlaylistUpdateClusterStatusDto { + cluster: XtreamCluster::Series, + status: PersistedPlaylistUpdateClusterState::Ok, + timestamp: 303, + last_update: Some(force_snapshot(None)), + }, + ], + }], + ..PlaylistUpdateStatusDto::default() + }; + + let views = build_input_update_run_views(&sources, &cards, &PlaylistUpdateCardStatuses::default(), &persisted); + assert_eq!(views[0].cluster_results[0].threshold, Some(95), "historical active guard beats disabled config"); + assert_eq!(views[0].cluster_results[1].threshold, None, "historical disabled guard beats active config"); + assert_eq!( + views[0].cluster_results[2].threshold, + Some(80), + "only a legacy Block-22a snapshot falls back to current configuration" + ); + } + + #[test] + fn persisted_cluster_status_missing_entry_remains_unknown() { + let xtream = input(7, "xtream", InputType::Xtream); + let sources = + SourcesConfigDto { inputs: vec![xtream.clone()], sources: Vec::new(), ..SourcesConfigDto::default() }; + let cards = vec![card(&xtream, &[], None)]; + + let views = build_input_update_run_views( + &sources, + &cards, + &PlaylistUpdateCardStatuses::default(), + &PlaylistUpdateStatusDto::default(), + ); + + assert!(views[0].cluster_results.iter().all(|cluster| cluster.persisted_status.is_none())); + } + + #[test] + fn persisted_cluster_status_never_augments_queued_or_active_run() { + let xtream = input(7, "xtream", InputType::Xtream); + let sources = + SourcesConfigDto { inputs: vec![xtream.clone()], sources: Vec::new(), ..SourcesConfigDto::default() }; + let cards = vec![card(&xtream, &[], Some(101))]; + let mut persisted = + persisted_status(7, &[(XtreamCluster::Live, PersistedPlaylistUpdateClusterState::Failed, 101)]); + persisted.inputs[0].clusters[0].last_update = Some(PersistedPlaylistUpdateClusterSnapshot { + policy: Some(InputRefreshPolicy::FORCE), + source: Some(PlaylistUpdateDataSource::Provider), + quality_guard_threshold: Some(95), + quality: None, + active_count: Some(100), + technical_state: Some(PersistedPlaylistUpdateTechnicalState::Succeeded), + }); + let connection = WebSocketConnectionContext::new(10, true); + let queued = Rc::new(PlaylistUpdateCardStatuses::for_connection(connection)).reduce( + PlaylistUpdateCardStatusAction::Accepted(PlaylistUpdateAcceptedScope { + connection_context: connection, + run_id: "new-run".into(), + input_ids: vec![7], + refresh_policy: InputRefreshPolicy::NORMAL, + }), + ); + + let queued_views = build_input_update_run_views(&sources, &cards, &queued, &persisted); + assert_eq!(queued_views[0].overall_state, InputUpdateCardStatus::Queued); + assert!(queued_views[0].cluster_results.iter().all(|cluster| { + cluster.persisted_status.is_none() + && cluster.outcome.is_none() + && cluster.source.is_none() + && cluster.quality.is_none() + && cluster.threshold.is_none() + && cluster.refresh_policy.is_none() + && cluster.technical_state.is_none() + })); + + let active = queued.reduce(PlaylistUpdateCardStatusAction::Progress { + connection_context: connection, + progress: PlaylistUpdateProgressEvent::for_run_input( + "new-run".into(), + PlaylistUpdateRunOrder::from(1), + 7, + "xtream", + "updating", + ), + }); + let active_views = build_input_update_run_views(&sources, &cards, &active, &persisted); + assert_eq!(active_views[0].overall_state, InputUpdateCardStatus::Updating); + assert!(active_views[0].cluster_results.iter().all(|cluster| cluster.persisted_status.is_none())); + } + + #[test] + fn persisted_cluster_status_reload_is_adopted_after_current_run_lifecycle_resets() { + let xtream = input(7, "xtream", InputType::Xtream); + let sources = + SourcesConfigDto { inputs: vec![xtream.clone()], sources: Vec::new(), ..SourcesConfigDto::default() }; + let cards = vec![card(&xtream, &[], Some(202))]; + let refreshed = persisted_status(7, &[(XtreamCluster::Live, PersistedPlaylistUpdateClusterState::Failed, 202)]); + let connection = WebSocketConnectionContext::new(11, true); + let telemetry = PlaylistUpdateInputTelemetry { + refresh_policy: InputRefreshPolicy::NORMAL, + source: None, + clusters: vec![PlaylistUpdateClusterTelemetry { + cluster: XtreamCluster::Live, + requested: true, + source: Some(InputDataSourceView::Provider), + baseline_count: Some(100), + candidate_count: Some(100), + active_count: Some(100), + threshold: Some(80), + quality: Some(100), + decision: Some(InputClusterOutcomeView::Accepted), + technical_state: None, + }], + }; + let current = Rc::new(PlaylistUpdateCardStatuses::for_connection(connection)) + .reduce(PlaylistUpdateCardStatusAction::Progress { + connection_context: connection, + progress: PlaylistUpdateProgressEvent::input_completed( + "current-run".into(), + PlaylistUpdateRunOrder::from(1), + 7, + PlaylistUpdateState::Success, + "xtream", + "completed", + ) + .with_input_telemetry(telemetry), + }) + .reduce(PlaylistUpdateCardStatusAction::Completed { + connection_context: connection, + completed: PlaylistUpdateRunStateEvent::correlated( + "current-run".into(), + PlaylistUpdateRunOrder::from(1), + PlaylistUpdateState::Success, + ), + }); + + let current_views = build_input_update_run_views(&sources, &cards, ¤t, &refreshed); + let current_live = current_views[0].cluster_results[0]; + assert_eq!(current_live.outcome, Some(InputClusterOutcomeView::Accepted)); + assert_eq!(current_live.quality, Some(100)); + assert_eq!(current_live.persisted_status, None); + + let reset = current + .reduce(PlaylistUpdateCardStatusAction::ConnectionChanged(WebSocketConnectionContext::new(12, true))); + let reloaded_views = build_input_update_run_views(&sources, &cards, &reset, &refreshed); + let reloaded_live = reloaded_views[0].cluster_results[0]; + assert_eq!(reloaded_live.outcome, None); + assert_eq!(reloaded_live.quality, None); + assert_eq!( + reloaded_live.persisted_status, + Some(PersistedClusterStatusView { + status: PersistedPlaylistUpdateClusterState::Failed, + timestamp: 202, + last_update: None, + }) + ); + } + + #[test] + fn pipeline_transparency_model_aggregates_stable_ids_and_preserves_typed_refresh_policy() { + let input_a = input(7, "same-name", InputType::Xtream); + let input_b = input(9, "same-name", InputType::M3u); + let target_a = target(41, "same-target-name", ProcessingOrder::Rmf); + let target_b = target(42, "same-target-name", ProcessingOrder::Frm); + let sources = SourcesConfigDto { + inputs: vec![input_a.clone(), input_b.clone()], + sources: vec![ConfigSourceDto { inputs: Vec::new(), targets: vec![target_a, target_b] }], + ..SourcesConfigDto::default() + }; + let cards = vec![ + card(&input_a, &[(41, "same-target-name"), (42, "same-target-name")], Some(500)), + card(&input_b, &[(41, "same-target-name")], None), + ]; + + let views = build_input_update_run_views( + &sources, + &cards, + &statuses_with_manual_multi_input_run(), + &PlaylistUpdateStatusDto::default(), + ); + + assert_eq!(views.iter().map(|view| view.input_id).collect::>(), [7, 9]); + assert_eq!(views[0].run_id.as_ref().map(PlaylistUpdateRunId::as_ref), Some("run-stable")); + assert_eq!(views[0].execution_order.map(PlaylistUpdateRunOrder::get), Some(17)); + assert_eq!(views[0].overall_state, InputUpdateCardStatus::Updating); + assert_eq!(views[0].refresh_policy, Some(InputRefreshPolicy::FORCE)); + assert_eq!(views[0].cache_source, None); + assert!(views[0].refresh_policy.is_some_and(InputRefreshPolicy::bypasses_cache)); + assert!(views[0].refresh_policy.is_some_and(InputRefreshPolicy::bypasses_quality)); + assert_eq!(views[0].progress_details, ["accepted input progress"]); + assert_eq!(views[0].target_results.iter().map(|target| target.target_id).collect::>(), [41, 42]); + assert_eq!(views[0].target_results[0].input_ids, [7, 9]); + assert_eq!(views[0].target_results[1].input_ids, [7]); + assert_eq!(views[1].run_id.as_ref().map(PlaylistUpdateRunId::as_ref), Some("run-stable")); + assert_eq!(views[1].execution_order.map(PlaylistUpdateRunOrder::get), Some(17)); + assert_eq!(views[1].overall_state, InputUpdateCardStatus::Updating); + assert_eq!(views[1].refresh_policy, None); + assert_eq!(views[1].progress_details, ["observed dependency progress"]); + assert_eq!(views[1].last_update_at, None); + } + + #[test] + fn pipeline_transparency_model_maps_cluster_configuration_and_quality_without_redeciding() { + let mut xtream = input(7, "xtream", InputType::Xtream); + xtream.options = Some(ConfigInputOptionsDto { + skip_vod: true, + update_quality: ConfigInputUpdateQualityDto { live: 90, vod: 95, series: 100 }, + ..ConfigInputOptionsDto::default() + }); + let stalker = input(9, "stalker", InputType::Stalker); + let m3u = input(8, "m3u", InputType::M3u); + let sources = SourcesConfigDto { + inputs: vec![xtream.clone(), stalker.clone(), m3u.clone()], + sources: Vec::new(), + ..SourcesConfigDto::default() + }; + let cards = vec![card(&xtream, &[], None), card(&stalker, &[], None), card(&m3u, &[], None)]; + + let views = build_input_update_run_views( + &sources, + &cards, + &PlaylistUpdateCardStatuses::default(), + &PlaylistUpdateStatusDto::default(), + ); + + assert_eq!( + views[0].cluster_results.iter().map(|view| view.cluster).collect::>(), + [XtreamCluster::Live, XtreamCluster::Video, XtreamCluster::Series] + ); + assert_eq!( + views[0].cluster_results.iter().map(|view| view.enabled_by_configuration).collect::>(), + [true, false, true] + ); + assert_eq!( + views[0].cluster_results.iter().map(|view| view.threshold).collect::>(), + [Some(90), Some(95), Some(100)] + ); + assert!(views[0].cluster_results.iter().all(|view| { + view.baseline_count.is_none() + && view.candidate_count.is_none() + && view.active_count.is_none() + && view.quality.is_none() + && view.source.is_none() + && view.outcome.is_none() + })); + assert_eq!( + views[1].cluster_results.iter().map(|view| view.cluster).collect::>(), + [XtreamCluster::Live, XtreamCluster::Video, XtreamCluster::Series] + ); + assert!(views[2].cluster_results.is_empty(), "M3U must not receive synthetic clusters"); + } + + #[test] + fn pipeline_transparency_target_uses_real_orders_and_only_configured_rule_counts() { + let input = input(7, "provider", InputType::Stalker); + let mut configured_target = target(41, "target", ProcessingOrder::Mrf); + configured_target.filter = ConfigTargetFilterDto { + processing: Some("Group ~ .*".to_string()), + persist: Some("EpgId ~ .+".to_string()), + ..ConfigTargetFilterDto::default() + }; + configured_target.rename = Some(vec![rename(), rename()]); + configured_target.mapping = Some(vec!["first".to_string(), "second".to_string(), "third".to_string()]); + let mut target_without_rules = target(42, "empty-target", ProcessingOrder::Fmr); + target_without_rules.filter.processing = Some(" ".to_string()); + let sources = SourcesConfigDto { + inputs: vec![input.clone()], + sources: vec![ConfigSourceDto { + inputs: Vec::new(), + targets: vec![configured_target, target_without_rules], + }], + ..SourcesConfigDto::default() + }; + let cards = vec![card(&input, &[(41, "target"), (42, "empty-target")], None)]; + + let views = build_input_update_run_views( + &sources, + &cards, + &PlaylistUpdateCardStatuses::default(), + &PlaylistUpdateStatusDto::default(), + ); + let configured = views[0].target_results[0].pipeline.expect("configured target pipeline"); + let empty = views[0].target_results[1].pipeline.expect("empty configured target pipeline"); + + assert_eq!(configured.processing_order, ProcessingOrder::Mrf); + assert_eq!(configured.filter_summary.configured_count, 2); + assert_eq!(configured.rename_summary.configured_count, 2); + assert_eq!(configured.mapping_summary.configured_count, 3); + assert_eq!(empty.processing_order, ProcessingOrder::Fmr); + assert_eq!(empty.filter_summary.configured_count, 0); + assert_eq!(empty.rename_summary.configured_count, 0); + assert_eq!(empty.mapping_summary.configured_count, 0); + } + + #[test] + fn pipeline_transparency_model_keeps_unavailable_runtime_values_optional() { + let input = input(7, "provider", InputType::Xtream); + let sources = + SourcesConfigDto { inputs: vec![input.clone()], sources: Vec::new(), ..SourcesConfigDto::default() }; + let cards = vec![card(&input, &[(99, "removed-target")], None)]; + + let views = build_input_update_run_views( + &sources, + &cards, + &PlaylistUpdateCardStatuses::default(), + &PlaylistUpdateStatusDto::default(), + ); + let view = &views[0]; + + assert_eq!(view.run_id, None); + assert_eq!(view.execution_order, None); + assert_eq!(view.overall_state, InputUpdateCardStatus::Ready); + assert_eq!(view.refresh_policy, None); + assert_eq!(view.cache_source, None); + assert_eq!(view.last_update_at, None); + assert!(view.progress_details.is_empty()); + assert!(view.cluster_results.iter().all(|cluster| cluster.threshold.is_none())); + assert_eq!(view.target_results[0].target_id, 99); + assert_eq!(view.target_results[0].pipeline, None); + } + + #[test] + fn pipeline_transparency_input_keeps_typed_source_and_outcome_without_redeciding() { + let xtream = input(7, "xtream", InputType::Xtream); + let m3u = input(8, "m3u", InputType::M3u); + let sources = SourcesConfigDto { + inputs: vec![xtream.clone(), m3u.clone()], + sources: Vec::new(), + ..SourcesConfigDto::default() + }; + let cards = vec![card(&xtream, &[], None), card(&m3u, &[], None)]; + let connection = WebSocketConnectionContext::new(8, true); + let statuses = Rc::new(PlaylistUpdateCardStatuses::for_connection(connection)) + .reduce(PlaylistUpdateCardStatusAction::Progress { + connection_context: connection, + progress: PlaylistUpdateProgressEvent::for_run_input( + "runtime-telemetry".into(), + PlaylistUpdateRunOrder::from(23), + 7, + "xtream", + "typed runtime facts", + ) + .with_input_telemetry(PlaylistUpdateInputTelemetry { + refresh_policy: InputRefreshPolicy::NORMAL, + source: None, + clusters: vec![PlaylistUpdateClusterTelemetry { + cluster: XtreamCluster::Live, + requested: true, + source: Some(InputDataSourceView::Provider), + baseline_count: Some(12_543), + candidate_count: Some(217), + active_count: Some(12_543), + threshold: Some(90), + quality: Some(1), + decision: Some(InputClusterOutcomeView::Rejected), + technical_state: None, + }], + }), + }) + .reduce(PlaylistUpdateCardStatusAction::Progress { + connection_context: connection, + progress: PlaylistUpdateProgressEvent::for_run_input( + "runtime-telemetry".into(), + PlaylistUpdateRunOrder::from(23), + 8, + "m3u", + "typed cache source", + ) + .with_input_telemetry(PlaylistUpdateInputTelemetry { + refresh_policy: InputRefreshPolicy::NORMAL, + source: Some(InputDataSourceView::Cache), + clusters: Vec::new(), + }), + }); + + let views = build_input_update_run_views(&sources, &cards, &statuses, &PlaylistUpdateStatusDto::default()); + + let cluster = views[0].cluster_results[0]; + assert_eq!(views[0].refresh_policy, Some(InputRefreshPolicy::NORMAL)); + assert_eq!(cluster.requested, Some(true)); + assert_eq!(cluster.source, Some(InputDataSourceView::Provider)); + assert_eq!(cluster.outcome, Some(InputClusterOutcomeView::Rejected)); + assert_eq!((cluster.threshold, cluster.quality), (Some(90), Some(1))); + assert_eq!( + (cluster.baseline_count, cluster.candidate_count, cluster.active_count), + (Some(12_543), Some(217), Some(12_543)) + ); + assert_eq!(views[1].cache_source, Some(InputDataSourceView::Cache)); + assert!(views[1].cluster_results.is_empty(), "M3U must remain a single input pipeline"); + } + + #[test] + fn pipeline_transparency_integration_keeps_partial_target_and_follow_up_run_facts_separate() { + let mut xtream = input(7, "provider", InputType::Xtream); + xtream.options = Some(ConfigInputOptionsDto { + update_quality: ConfigInputUpdateQualityDto { live: 90, vod: 0, series: 100 }, + ..ConfigInputOptionsDto::default() + }); + let mut configured_target = target(41, "output", ProcessingOrder::Rmf); + configured_target.filter.processing = Some("Group ~ .*".to_string()); + configured_target.rename = Some(vec![rename()]); + configured_target.mapping = Some(vec!["mapping".to_string()]); + let sources = SourcesConfigDto { + inputs: vec![xtream.clone()], + sources: vec![ConfigSourceDto { inputs: Vec::new(), targets: vec![configured_target] }], + ..SourcesConfigDto::default() + }; + let cards = vec![card(&xtream, &[(41, "output")], Some(500))]; + let connection = WebSocketConnectionContext::new(9, true); + let partial_telemetry = PlaylistUpdateInputTelemetry { + refresh_policy: InputRefreshPolicy::NORMAL, + source: None, + clusters: vec![PlaylistUpdateClusterTelemetry { + cluster: XtreamCluster::Live, + requested: true, + source: Some(InputDataSourceView::Provider), + baseline_count: Some(100), + candidate_count: Some(80), + active_count: Some(100), + threshold: Some(90), + quality: Some(80), + decision: Some(InputClusterOutcomeView::Rejected), + technical_state: None, + }], + }; + let statuses = Rc::new(PlaylistUpdateCardStatuses::for_connection(connection)) + .reduce(PlaylistUpdateCardStatusAction::Accepted(PlaylistUpdateAcceptedScope { + connection_context: connection, + run_id: "partial-run".into(), + input_ids: vec![7], + refresh_policy: InputRefreshPolicy::NORMAL, + })) + .reduce(PlaylistUpdateCardStatusAction::Progress { + connection_context: connection, + progress: PlaylistUpdateProgressEvent::input_completed( + "partial-run".into(), + PlaylistUpdateRunOrder::from(1), + 7, + PlaylistUpdateState::Partial, + "provider", + "live quality rejected", + ) + .with_input_telemetry(partial_telemetry), + }) + .reduce(PlaylistUpdateCardStatusAction::Completed { + connection_context: connection, + completed: PlaylistUpdateRunStateEvent::correlated( + "partial-run".into(), + PlaylistUpdateRunOrder::from(1), + PlaylistUpdateState::Partial, + ), + }); + + let partial_views = + build_input_update_run_views(&sources, &cards, &statuses, &PlaylistUpdateStatusDto::default()); + let partial = &partial_views[0]; + assert_eq!(partial.overall_state, InputUpdateCardStatus::Partial); + assert_eq!(partial.progress_details, ["live quality rejected"]); + let live = partial.cluster_results[0]; + assert_eq!(live.outcome, Some(InputClusterOutcomeView::Rejected)); + assert_eq!(live.source, Some(InputDataSourceView::Provider)); + assert_eq!((live.threshold, live.quality), (Some(90), Some(80))); + let target = &partial.target_results[0]; + let pipeline = target.pipeline.expect("configured target pipeline"); + assert_eq!(target.target_id, 41); + assert_eq!(target.input_ids, [7]); + assert_eq!(pipeline.processing_order, ProcessingOrder::Rmf); + assert_eq!(pipeline.filter_summary.configured_count, 1); + assert_eq!(pipeline.rename_summary.configured_count, 1); + assert_eq!(pipeline.mapping_summary.configured_count, 1); + + let statuses = statuses.reduce(PlaylistUpdateCardStatusAction::Accepted(PlaylistUpdateAcceptedScope { + connection_context: connection, + run_id: "force-run".into(), + input_ids: vec![7], + refresh_policy: InputRefreshPolicy::FORCE, + })); + let queued_views = + build_input_update_run_views(&sources, &cards, &statuses, &PlaylistUpdateStatusDto::default()); + let queued = &queued_views[0]; + + assert_eq!(queued.run_id.as_ref().map(PlaylistUpdateRunId::as_ref), Some("force-run")); + assert_eq!(queued.overall_state, InputUpdateCardStatus::Queued); + assert_eq!(queued.refresh_policy, Some(InputRefreshPolicy::FORCE)); + assert!(queued.progress_details.is_empty()); + assert_eq!(queued.target_results, partial.target_results); + let live = queued.cluster_results[0]; + assert_eq!(live.requested, None); + assert_eq!(live.source, None); + assert_eq!(live.outcome, None); + assert_eq!((live.baseline_count, live.candidate_count, live.active_count), (None, None, None)); + assert_eq!((live.threshold, live.quality), (Some(90), None)); + } +} diff --git a/frontend/src/model/playlist_update_card_status.rs b/frontend/src/model/playlist_update_card_status.rs new file mode 100644 index 000000000..3fd329825 --- /dev/null +++ b/frontend/src/model/playlist_update_card_status.rs @@ -0,0 +1,1600 @@ +use crate::{model::InputUpdateCardModel, services::WebSocketConnectionContext}; +use indexmap::{IndexMap, IndexSet}; +use shared::model::{ + InputRefreshPolicy, LibraryScanResult, PlaylistUpdateInputTelemetry, PlaylistUpdateProgressDetail, + PlaylistUpdateProgressEvent, PlaylistUpdateRunId, PlaylistUpdateRunOrder, PlaylistUpdateRunStateEvent, + PlaylistUpdateState, SourcesConfigDto, +}; +use std::{collections::HashSet, rc::Rc}; +use yew::Reducible; + +const MAX_CARD_DETAIL_LINES: usize = 100; +const MAX_TRACKED_RUNS: usize = 32; + +/// Effective status shown in an input update card. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub enum InputUpdateCardStatus { + #[default] + Ready, + Queued, + Updating, + Success, + Partial, + Failed, +} + +impl InputUpdateCardStatus { + #[must_use] + pub const fn label_key(self) -> &'static str { + match self { + Self::Ready => "LABEL.UPDATE_STATUS_READY", + Self::Queued => "LABEL.UPDATE_STATUS_QUEUED", + Self::Updating => "LABEL.UPDATE_STATUS_UPDATING", + Self::Success => "LABEL.UPDATE_STATUS_SUCCESS", + Self::Partial => "LABEL.UPDATE_STATUS_PARTIAL", + Self::Failed => "LABEL.UPDATE_STATUS_FAILED", + } + } + + #[must_use] + pub const fn modifier(self) -> &'static str { + match self { + Self::Ready => "tp__playlist-update-view__pipeline-status--neutral", + Self::Queued => "tp__playlist-update-view__pipeline-status--queued", + Self::Updating => "tp__playlist-update-view__pipeline-status--updating", + Self::Success => "tp__playlist-update-view__pipeline-status--accepted", + Self::Partial => "tp__playlist-update-view__pipeline-status--rejected", + Self::Failed => "tp__playlist-update-view__pipeline-status--error", + } + } +} + +/// Request/progress state and details for one stable input ID. +#[derive(Clone, Debug, PartialEq)] +pub struct InputUpdateCardRuntime { + library_scan_result: Option, + status: InputUpdateCardStatus, + action_status: InputUpdateCardStatus, + details: Vec, + localized_details: Vec, + run_id: Option, + execution_order: Option, + refresh_policy: Option, + input_telemetry: Option, +} + +impl Default for InputUpdateCardRuntime { + fn default() -> Self { + Self { + library_scan_result: None, + status: InputUpdateCardStatus::Ready, + action_status: InputUpdateCardStatus::Ready, + details: Vec::new(), + localized_details: Vec::new(), + run_id: None, + execution_order: None, + refresh_policy: None, + input_telemetry: None, + } + } +} + +impl InputUpdateCardRuntime { + #[must_use] + pub const fn status(&self) -> InputUpdateCardStatus { self.status } + + /// Completion of the correlated input-to-target run, independent of input success. + #[must_use] + pub const fn action_status(&self) -> InputUpdateCardStatus { self.action_status } + + #[must_use] + pub fn details(&self) -> &[String] { &self.details } + + #[must_use] + pub fn localized_details(&self) -> &[PlaylistUpdateProgressDetail] { &self.localized_details } + + /// Stable identity of the run that currently determines this card's status. + #[must_use] + pub fn run_id(&self) -> Option<&PlaylistUpdateRunId> { self.run_id.as_ref() } + + /// Execution order of the run that currently determines this card's status. + #[must_use] + pub const fn execution_order(&self) -> Option { self.execution_order } + + /// Effective policy proven by request acceptance or typed backend telemetry. + #[must_use] + pub const fn refresh_policy(&self) -> Option { self.refresh_policy } + + /// Typed input facts emitted by the backend for this effective run. + #[must_use] + pub const fn input_telemetry(&self) -> Option<&PlaylistUpdateInputTelemetry> { self.input_telemetry.as_ref() } + + /// Scanner result belonging exclusively to this effective run and input. + #[must_use] + pub const fn library_scan_result(&self) -> Option<&LibraryScanResult> { self.library_scan_result.as_ref() } +} + +struct EffectiveInputRun<'a> { + status: InputUpdateCardStatus, + run_id: &'a PlaylistUpdateRunId, + execution_order: Option, + refresh_policy: Option, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +struct InputTerminalState { + run_id: PlaylistUpdateRunId, + execution_order: PlaylistUpdateRunOrder, + state: PlaylistUpdateState, + action_state: Option, + refresh_policy: Option, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +struct InputDetails { + library_scan_result: Option, + run_id: PlaylistUpdateRunId, + execution_order: PlaylistUpdateRunOrder, + lines: Vec, + localized_lines: Vec, + input_telemetry: Option, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +struct PlaylistUpdateRunRuntime { + execution_order: Option, + accepted: bool, + accepted_input_ids: IndexSet, + observed_input_ids: IndexSet, + input_states: IndexMap, + terminal: Option, + refresh_policy_by_input_id: IndexMap, +} + +impl PlaylistUpdateRunRuntime { + fn new() -> Self { + Self { + execution_order: None, + accepted: false, + accepted_input_ids: IndexSet::new(), + observed_input_ids: IndexSet::new(), + input_states: IndexMap::new(), + terminal: None, + refresh_policy_by_input_id: IndexMap::new(), + } + } + + fn scope(&self) -> IndexSet { + self.accepted_input_ids.iter().chain(&self.observed_input_ids).copied().collect() + } + + fn refresh_policy(&self, input_id: u16) -> Option { + self.refresh_policy_by_input_id.get(&input_id).copied() + } + + fn assign_execution_order(&mut self, execution_order: PlaylistUpdateRunOrder) -> bool { + if let Some(current) = self.execution_order { + current == execution_order + } else { + self.execution_order = Some(execution_order); + true + } + } +} + +/// Stable scope returned by one accepted card or bulk request. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct PlaylistUpdateAcceptedScope { + pub connection_context: WebSocketConnectionContext, + pub run_id: PlaylistUpdateRunId, + pub input_ids: Vec, + pub refresh_policy: InputRefreshPolicy, +} + +/// Card runtime state correlated by update-run and stable input identity. +#[derive(Clone, Debug, PartialEq)] +pub struct PlaylistUpdateCardStatuses { + connection_context: WebSocketConnectionContext, + runs: IndexMap, + terminal_by_input_id: IndexMap, + details_by_input_id: IndexMap, +} + +impl Default for PlaylistUpdateCardStatuses { + fn default() -> Self { Self::for_connection(WebSocketConnectionContext::default()) } +} + +impl PlaylistUpdateCardStatuses { + #[must_use] + pub fn for_connection(connection_context: WebSocketConnectionContext) -> Self { + Self { + connection_context, + runs: IndexMap::new(), + terminal_by_input_id: IndexMap::new(), + details_by_input_id: IndexMap::new(), + } + } + + #[must_use] + pub fn for_input(&self, input_id: u16) -> InputUpdateCardRuntime { + let effective_run = self.effective_run_for_input(input_id); + let status = effective_run.as_ref().map_or(InputUpdateCardStatus::Ready, |run| run.status); + let action_status = effective_run.as_ref().map_or(InputUpdateCardStatus::Ready, |effective| { + if let Some(run) = self.runs.get(effective.run_id) { + return run.terminal.map_or_else( + || { + if effective.status == InputUpdateCardStatus::Queued { + InputUpdateCardStatus::Queued + } else { + InputUpdateCardStatus::Updating + } + }, + status_from_state, + ); + } + self.terminal_by_input_id + .get(&input_id) + .and_then(|terminal| terminal.action_state) + .map_or(effective.status, status_from_state) + }); + let run_id = effective_run.as_ref().map(|run| run.run_id.clone()); + let refresh_policy = effective_run.as_ref().and_then(|run| run.refresh_policy); + let execution_order = effective_run.as_ref().and_then(|run| run.execution_order); + let details = effective_run + .and_then(|run| self.details_by_input_id.get(&input_id).filter(|details| details.run_id == *run.run_id)); + InputUpdateCardRuntime { + library_scan_result: details.and_then(|details| details.library_scan_result.clone()), + status, + action_status, + details: details.map_or_else(Vec::new, |details| details.lines.clone()), + localized_details: details.map_or_else(Vec::new, |details| details.localized_lines.clone()), + run_id, + execution_order, + refresh_policy, + input_telemetry: details.and_then(|details| details.input_telemetry.clone()), + } + } + + fn effective_run_for_input(&self, input_id: u16) -> Option> { + let terminal = self.terminal_by_input_id.get(&input_id); + let terminal_order = terminal.map(|terminal| terminal.execution_order); + let running = self + .runs + .iter() + .filter(|(run_id, run)| { + run.terminal.is_none() + && run.observed_input_ids.contains(&input_id) + // Input completion does not finish its run's target rebuild. Only the + // same run may retain priority at the recorded input-terminal order. + && run.execution_order.is_some_and(|order| { + terminal.is_none_or(|terminal| { + order > terminal.execution_order + || (order == terminal.execution_order && *run_id == &terminal.run_id) + }) + }) + }) + .max_by_key(|(_, run)| run.execution_order); + if let Some((run_id, run)) = running { + return Some(EffectiveInputRun { + status: run + .input_states + .get(&input_id) + .copied() + .map_or(InputUpdateCardStatus::Updating, status_from_state), + run_id, + execution_order: run.execution_order, + refresh_policy: run.refresh_policy(input_id), + }); + } + + let queued = self.runs.iter().rev().find(|(_, run)| { + run.terminal.is_none() + && run.accepted + && run.accepted_input_ids.contains(&input_id) + && !run.observed_input_ids.contains(&input_id) + && run.execution_order.is_none_or(|order| terminal_order.is_none_or(|terminal| order > terminal)) + }); + if let Some((run_id, run)) = queued { + return Some(EffectiveInputRun { + status: InputUpdateCardStatus::Queued, + run_id, + execution_order: run.execution_order, + refresh_policy: run.refresh_policy(input_id), + }); + } + + self.terminal_by_input_id.get(&input_id).map(|terminal| EffectiveInputRun { + status: status_from_state(terminal.state), + run_id: &terminal.run_id, + execution_order: Some(terminal.execution_order), + refresh_policy: terminal.refresh_policy, + }) + } + + fn ensure_run(&mut self, run_id: &PlaylistUpdateRunId) { + if self.runs.contains_key(run_id) { + return; + } + self.runs.insert(run_id.clone(), PlaylistUpdateRunRuntime::new()); + self.prune_runs(); + } + + fn connection_changed(&mut self, connection_context: WebSocketConnectionContext) { + if self.connection_context == connection_context { + return; + } + self.connection_context = connection_context; + self.runs.clear(); + self.terminal_by_input_id.clear(); + self.details_by_input_id.clear(); + } + + fn is_current_live_connection(&self, connection_context: WebSocketConnectionContext) -> bool { + self.connection_context.is_same_live_connection(connection_context) + } + + fn prune_runs(&mut self) { + while self.runs.len() > MAX_TRACKED_RUNS { + let removable = self.runs.iter().position(|(_, run)| run.terminal.is_some()); + let Some(index) = removable else { + break; + }; + self.runs.shift_remove_index(index); + } + } + + fn record_terminal( + &mut self, + input_id: u16, + run_id: &PlaylistUpdateRunId, + execution_order: PlaylistUpdateRunOrder, + state: PlaylistUpdateState, + refresh_policy: Option, + ) { + let action_state = self.runs.get(run_id).and_then(|run| run.terminal); + match self.terminal_by_input_id.get_mut(&input_id) { + Some(current) if current.execution_order > execution_order => {} + Some(current) if current.execution_order == execution_order && current.run_id == *run_id => { + current.state = strongest_state(current.state, state); + current.action_state = action_state.or(current.action_state); + if current.refresh_policy.is_none() { + current.refresh_policy = refresh_policy; + } + } + Some(current) if current.execution_order == execution_order => {} + Some(current) => { + *current = + InputTerminalState { run_id: run_id.clone(), execution_order, state, action_state, refresh_policy }; + } + None => { + self.terminal_by_input_id.insert( + input_id, + InputTerminalState { run_id: run_id.clone(), execution_order, state, action_state, refresh_policy }, + ); + } + } + } + + fn activate_details_run( + &mut self, + input_id: u16, + run_id: &PlaylistUpdateRunId, + execution_order: PlaylistUpdateRunOrder, + ) -> bool { + match self.details_by_input_id.get_mut(&input_id) { + Some(current) if current.execution_order > execution_order => false, + Some(current) if current.execution_order == execution_order => current.run_id == *run_id, + Some(current) => { + *current = InputDetails { + run_id: run_id.clone(), + execution_order, + lines: Vec::new(), + localized_lines: Vec::new(), + input_telemetry: None, + library_scan_result: None, + }; + true + } + None => { + self.details_by_input_id.insert( + input_id, + InputDetails { + run_id: run_id.clone(), + execution_order, + lines: Vec::new(), + localized_lines: Vec::new(), + input_telemetry: None, + library_scan_result: None, + }, + ); + true + } + } + } + + fn accept(&mut self, accepted: PlaylistUpdateAcceptedScope) { + let PlaylistUpdateAcceptedScope { connection_context, run_id, input_ids, refresh_policy } = accepted; + if !self.is_current_live_connection(connection_context) { + return; + } + self.ensure_run(&run_id); + let (execution_order, terminal, explicit_states, input_policies) = { + let Some(run) = self.runs.get_mut(&run_id) else { + return; + }; + run.accepted = true; + for input_id in &input_ids { + run.accepted_input_ids.insert(*input_id); + run.refresh_policy_by_input_id.entry(*input_id).or_insert(refresh_policy); + } + let input_policies = input_ids + .iter() + .filter_map(|input_id| { + run.refresh_policy_by_input_id.get(input_id).copied().map(|policy| (*input_id, policy)) + }) + .collect::>(); + (run.execution_order, run.terminal, run.input_states.clone(), input_policies) + }; + for (input_id, refresh_policy) in input_policies { + if let Some(current) = self + .terminal_by_input_id + .get_mut(&input_id) + .filter(|current| current.run_id == run_id && current.refresh_policy.is_none()) + { + current.refresh_policy = Some(refresh_policy); + } + if let (Some(execution_order), Some(terminal)) = (execution_order, terminal) { + let state = explicit_states.get(&input_id).copied().unwrap_or(terminal); + self.activate_details_run(input_id, &run_id, execution_order); + self.record_terminal(input_id, &run_id, execution_order, state, Some(refresh_policy)); + } + } + } + + fn restore_active_updates( + &mut self, + connection_context: WebSocketConnectionContext, + progress: Vec, + ) { + for event in progress { + let (Some(run_id), Some(order), Some(input_id)) = (&event.run_id, event.execution_order, event.input_id) + else { + continue; + }; + // A response can arrive after live progress or even run completion. + // Hydrate missing inputs only; never roll those newer facts back. + if self + .runs + .get(run_id) + .is_some_and(|run| run.terminal.is_some() || run.observed_input_ids.contains(&input_id)) + || self + .terminal_by_input_id + .get(&input_id) + .is_some_and(|terminal| terminal.action_state.is_some() && terminal.execution_order >= order) + { + continue; + } + self.progress(connection_context, event); + } + } + + fn progress(&mut self, connection_context: WebSocketConnectionContext, progress: PlaylistUpdateProgressEvent) { + if !self.is_current_live_connection(connection_context) { + return; + } + let PlaylistUpdateProgressEvent { + run_id: Some(run_id), + execution_order: Some(execution_order), + input_id: Some(input_id), + state, + input_telemetry, + library_scan_result, + detail, + message, + .. + } = progress + else { + return; + }; + self.ensure_run(&run_id); + let (input_state, refresh_policy) = { + let Some(run) = self.runs.get_mut(&run_id) else { + return; + }; + if run.terminal.is_some() || !run.assign_execution_order(execution_order) { + return; + } + run.observed_input_ids.insert(input_id); + if let Some(input_telemetry) = input_telemetry.as_ref() { + run.refresh_policy_by_input_id.entry(input_id).or_insert(input_telemetry.refresh_policy); + } + let input_state = state.map(|state| { + let entry = run.input_states.entry(input_id).or_insert(state); + *entry = strongest_state(*entry, state); + *entry + }); + (input_state, run.refresh_policy(input_id)) + }; + + if self.activate_details_run(input_id, &run_id, execution_order) { + if let Some(details) = self.details_by_input_id.get_mut(&input_id) { + details.lines.push(message); + if details.lines.len() > MAX_CARD_DETAIL_LINES { + details.lines.remove(0); + } + if let Some(detail) = detail { + details.localized_lines.push(detail); + if details.localized_lines.len() > MAX_CARD_DETAIL_LINES { + details.localized_lines.remove(0); + } + } + if let Some(input_telemetry) = input_telemetry { + details.input_telemetry = Some(input_telemetry); + } + if let Some(result) = library_scan_result { + details.library_scan_result = Some(result); + } + } + } + if let Some(state) = input_state { + self.record_terminal(input_id, &run_id, execution_order, state, refresh_policy); + } + } + + fn complete(&mut self, connection_context: WebSocketConnectionContext, completed: PlaylistUpdateRunStateEvent) { + if !self.is_current_live_connection(connection_context) { + return; + } + let Some(run_id) = completed.run_id else { + return; + }; + self.ensure_run(&run_id); + let (execution_order, scope, input_states, input_policies, terminal, was_known) = { + let Some(run) = self.runs.get_mut(&run_id) else { + return; + }; + if let Some(execution_order) = completed.execution_order { + if !run.assign_execution_order(execution_order) { + return; + } + } + let was_known = run.accepted || !run.observed_input_ids.is_empty(); + if run.terminal.is_none() { + run.terminal = Some(completed.state); + } + ( + run.execution_order, + run.scope(), + run.input_states.clone(), + run.refresh_policy_by_input_id.clone(), + run.terminal.unwrap_or(completed.state), + was_known, + ) + }; + let Some(execution_order) = execution_order else { + return; + }; + if !was_known { + return; + } + for input_id in scope { + let state = input_states.get(&input_id).copied().unwrap_or(terminal); + let refresh_policy = input_policies.get(&input_id).copied(); + self.activate_details_run(input_id, &run_id, execution_order); + self.record_terminal(input_id, &run_id, execution_order, state, refresh_policy); + } + } + + fn apply(mut self, action: PlaylistUpdateCardStatusAction) -> Self { + match action { + PlaylistUpdateCardStatusAction::ConnectionChanged(connection_context) => { + self.connection_changed(connection_context); + } + PlaylistUpdateCardStatusAction::Accepted(accepted) => self.accept(accepted), + PlaylistUpdateCardStatusAction::Progress { connection_context, progress } => { + self.progress(connection_context, progress); + } + PlaylistUpdateCardStatusAction::RestoreActive { connection_context, progress } => { + self.restore_active_updates(connection_context, progress); + } + PlaylistUpdateCardStatusAction::Completed { connection_context, completed } => { + self.complete(connection_context, completed); + } + } + self + } +} + +fn strongest_state(current: PlaylistUpdateState, candidate: PlaylistUpdateState) -> PlaylistUpdateState { + match (current, candidate) { + (PlaylistUpdateState::Failure, _) | (_, PlaylistUpdateState::Failure) => PlaylistUpdateState::Failure, + (PlaylistUpdateState::Partial, _) | (_, PlaylistUpdateState::Partial) => PlaylistUpdateState::Partial, + _ => PlaylistUpdateState::Success, + } +} + +pub(super) const fn status_from_state(state: PlaylistUpdateState) -> InputUpdateCardStatus { + match state { + PlaylistUpdateState::Success => InputUpdateCardStatus::Success, + PlaylistUpdateState::Partial => InputUpdateCardStatus::Partial, + PlaylistUpdateState::Failure => InputUpdateCardStatus::Failed, + } +} + +pub enum PlaylistUpdateCardStatusAction { + ConnectionChanged(WebSocketConnectionContext), + Accepted(PlaylistUpdateAcceptedScope), + Progress { connection_context: WebSocketConnectionContext, progress: PlaylistUpdateProgressEvent }, + RestoreActive { connection_context: WebSocketConnectionContext, progress: Vec }, + Completed { connection_context: WebSocketConnectionContext, completed: PlaylistUpdateRunStateEvent }, +} + +impl Reducible for PlaylistUpdateCardStatuses { + type Action = PlaylistUpdateCardStatusAction; + + fn reduce(self: Rc, action: Self::Action) -> Rc { Rc::new((*self).clone().apply(action)) } +} + +/// Two deliberately separate scopes for the global action: server-side +/// executability and the visible cards that can show this run as queued. +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub struct PlaylistUpdateBulkScope { + pub can_submit: bool, + pub visible_input_ids: Vec, +} + +#[must_use] +pub fn playlist_update_bulk_scope( + sources: &SourcesConfigDto, + cards: &[InputUpdateCardModel], +) -> PlaylistUpdateBulkScope { + let enabled_inputs = + sources.inputs.iter().filter(|input| input.enabled).map(|input| input.name.as_ref()).collect::>(); + let enabled_target_ids = sources + .sources + .iter() + .flat_map(|source| &source.targets) + .filter(|target| target.enabled) + .map(|target| target.id) + .collect::>(); + let can_submit = sources.sources.iter().any(|source| { + source.inputs.iter().any(|input| enabled_inputs.contains(input.as_ref())) + && source.targets.iter().any(|target| target.enabled) + }); + // Visible cards include disabled inputs; bulk participation follows configuration IDs, not manual capabilities. + let enabled_input_ids = + sources.inputs.iter().filter(|input| input.enabled).map(|input| input.id).collect::>(); + let visible_input_ids = cards + .iter() + .filter(|card| { + enabled_input_ids.contains(&card.input_id) + && card.targets.iter().any(|target| enabled_target_ids.contains(&target.id)) + }) + .map(|card| card.input_id) + .collect(); + PlaylistUpdateBulkScope { can_submit, visible_input_ids } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::model::{ + build_input_update_card_models, InputUpdateCapabilities, InputUpdateCapabilitiesExt, InputUpdateCardState, + InputUpdateCardTarget, + }; + use shared::{ + model::{ + ConfigInputDto, ConfigSourceDto, ConfigTargetDto, InputRefreshPolicy, InputType, + PlaylistUpdateClusterTelemetry, PlaylistUpdateDataSource, PlaylistUpdateRunId, PlaylistUpdateRunOrder, + XtreamCluster, + }, + utils::Internable, + }; + use std::sync::Arc; + + fn run_id(value: &str) -> PlaylistUpdateRunId { value.into() } + + fn execution_order(value: u64) -> PlaylistUpdateRunOrder { PlaylistUpdateRunOrder::from(value) } + + const fn connection(socket_epoch: u64) -> WebSocketConnectionContext { + WebSocketConnectionContext::new(socket_epoch, true) + } + + const fn disconnected_connection(socket_epoch: u64) -> WebSocketConnectionContext { + WebSocketConnectionContext::new(socket_epoch, false) + } + + fn statuses() -> PlaylistUpdateCardStatuses { PlaylistUpdateCardStatuses::for_connection(connection(1)) } + + const fn connection_changed(connection_context: WebSocketConnectionContext) -> PlaylistUpdateCardStatusAction { + PlaylistUpdateCardStatusAction::ConnectionChanged(connection_context) + } + + fn card(input_id: u16, target_ids: &[u16]) -> InputUpdateCardModel { + InputUpdateCardModel { + input_id, + input_name: Arc::from(format!("input-{input_id}")), + input_type: InputType::Xtream, + capabilities: crate::model::InputUpdateCapabilities::for_input(&ConfigInputDto { + input_type: InputType::Xtream, + ..ConfigInputDto::default() + }), + targets: target_ids + .iter() + .map(|target_id| InputUpdateCardTarget { id: *target_id, name: format!("target-{target_id}") }) + .collect(), + last_update_at: None, + } + } + + fn reduce(state: PlaylistUpdateCardStatuses, action: PlaylistUpdateCardStatusAction) -> PlaylistUpdateCardStatuses { + state.apply(action) + } + + fn accepted_in( + connection_context: WebSocketConnectionContext, + run: &str, + input_ids: &[u16], + ) -> PlaylistUpdateCardStatusAction { + accepted_with_policy_in(connection_context, run, input_ids, InputRefreshPolicy::NORMAL) + } + + fn accepted_with_policy_in( + connection_context: WebSocketConnectionContext, + run: &str, + input_ids: &[u16], + refresh_policy: InputRefreshPolicy, + ) -> PlaylistUpdateCardStatusAction { + PlaylistUpdateCardStatusAction::Accepted(PlaylistUpdateAcceptedScope { + connection_context, + run_id: run_id(run), + input_ids: input_ids.to_vec(), + refresh_policy, + }) + } + + fn accepted(run: &str, input_ids: &[u16]) -> PlaylistUpdateCardStatusAction { + accepted_in(connection(1), run, input_ids) + } + + fn accepted_with_policy( + run: &str, + input_ids: &[u16], + refresh_policy: InputRefreshPolicy, + ) -> PlaylistUpdateCardStatusAction { + accepted_with_policy_in(connection(1), run, input_ids, refresh_policy) + } + + fn progress_in( + connection_context: WebSocketConnectionContext, + run: &str, + order: u64, + input_id: u16, + message: &str, + ) -> PlaylistUpdateCardStatusAction { + PlaylistUpdateCardStatusAction::Progress { + connection_context, + progress: PlaylistUpdateProgressEvent::for_run_input( + run_id(run), + execution_order(order), + input_id, + format!("input-{input_id}"), + message, + ), + } + } + + fn progress(run: &str, order: u64, input_id: u16, message: &str) -> PlaylistUpdateCardStatusAction { + progress_in(connection(1), run, order, input_id, message) + } + + fn input_completed_in( + connection_context: WebSocketConnectionContext, + run: &str, + order: u64, + input_id: u16, + state: PlaylistUpdateState, + message: &str, + ) -> PlaylistUpdateCardStatusAction { + PlaylistUpdateCardStatusAction::Progress { + connection_context, + progress: PlaylistUpdateProgressEvent::input_completed( + run_id(run), + execution_order(order), + input_id, + state, + format!("input-{input_id}"), + message, + ), + } + } + + fn input_completed( + run: &str, + order: u64, + input_id: u16, + state: PlaylistUpdateState, + message: &str, + ) -> PlaylistUpdateCardStatusAction { + input_completed_in(connection(1), run, order, input_id, state, message) + } + + fn completed_in( + connection_context: WebSocketConnectionContext, + run: &str, + order: u64, + state: PlaylistUpdateState, + ) -> PlaylistUpdateCardStatusAction { + PlaylistUpdateCardStatusAction::Completed { + connection_context, + completed: PlaylistUpdateRunStateEvent::correlated(run_id(run), execution_order(order), state), + } + } + + fn completed(run: &str, order: u64, state: PlaylistUpdateState) -> PlaylistUpdateCardStatusAction { + completed_in(connection(1), run, order, state) + } + + #[test] + fn pipeline_transparency_input_action_waits_for_target_completion_and_keeps_input_success() { + for terminal in [PlaylistUpdateState::Success, PlaylistUpdateState::Failure, PlaylistUpdateState::Partial] { + let state = PlaylistUpdateCardStatuses::for_connection(connection(1)) + .apply(accepted("rescan", &[2])) + .apply(progress("rescan", 1, 2, "Rescan started")) + .apply(input_completed("rescan", 1, 2, PlaylistUpdateState::Success, "Input acquired")); + assert_eq!(state.for_input(2).status(), InputUpdateCardStatus::Success); + assert_eq!(state.for_input(2).action_status(), InputUpdateCardStatus::Updating); + let state = state.apply(completed("rescan", 1, terminal)); + assert_eq!(state.for_input(2).status(), InputUpdateCardStatus::Success); + assert_eq!(state.for_input(2).action_status(), super::status_from_state(terminal)); + let state = state.apply(accepted("next-run", &[2])); + assert_eq!(state.for_input(2).action_status(), InputUpdateCardStatus::Queued); + assert!(state.for_input(2).input_telemetry().is_none()); + } + } + + #[test] + fn pipeline_transparency_playlist_update_status_running_rebuild_precedes_queued_follow_up() { + let r1_telemetry = PlaylistUpdateInputTelemetry { + refresh_policy: InputRefreshPolicy::FORCE, + source: Some(PlaylistUpdateDataSource::Provider), + clusters: Vec::new(), + }; + let r2_telemetry = PlaylistUpdateInputTelemetry { + refresh_policy: InputRefreshPolicy::NORMAL, + source: Some(PlaylistUpdateDataSource::Cache), + clusters: Vec::new(), + }; + for terminal in [PlaylistUpdateState::Success, PlaylistUpdateState::Partial, PlaylistUpdateState::Failure] { + let state = statuses() + .apply(accepted_with_policy("r1", &[7], InputRefreshPolicy::FORCE)) + .apply(PlaylistUpdateCardStatusAction::Progress { + connection_context: connection(1), + progress: PlaylistUpdateProgressEvent::for_run_input( + run_id("r1"), + execution_order(1), + 7, + "input-7", + "r1 provider acquisition", + ) + .with_input_telemetry(r1_telemetry.clone()), + }) + .apply(input_completed("r1", 1, 7, PlaylistUpdateState::Success, "r1 input completed")); + let rebuilding = InputUpdateCardRuntime { + library_scan_result: None, + status: InputUpdateCardStatus::Success, + action_status: InputUpdateCardStatus::Updating, + details: vec!["r1 provider acquisition".into(), "r1 input completed".into()], + localized_details: vec![PlaylistUpdateProgressDetail::InputCompletedSuccessfully], + run_id: Some(run_id("r1")), + execution_order: Some(execution_order(1)), + refresh_policy: Some(InputRefreshPolicy::FORCE), + input_telemetry: Some(r1_telemetry.clone()), + }; + assert_eq!(state.for_input(7), rebuilding); + let state = state.apply(accepted("r2", &[7])); + let pending = state.runs[&run_id("r2")].clone(); + assert_eq!(state.for_input(7), rebuilding, "R1 targets still running: {terminal:?}"); + let state = state.apply(completed("r1", 1, terminal)); + let queued = InputUpdateCardRuntime { + status: InputUpdateCardStatus::Queued, + action_status: InputUpdateCardStatus::Queued, + run_id: Some(run_id("r2")), + refresh_policy: Some(InputRefreshPolicy::NORMAL), + ..InputUpdateCardRuntime::default() + }; + assert_eq!(state.for_input(7), queued); + assert_eq!(state.terminal_by_input_id[&7].action_state, Some(terminal)); + assert_eq!(state.runs[&run_id("r2")], pending, "R2 must not be consumed by R1 completion"); + let state = state.apply(accepted_with_policy("r1", &[7], InputRefreshPolicy::FORCE)); + assert_eq!(state.for_input(7), queued, "Late R1 acceptance must not displace queued R2"); + assert_eq!(state.runs[&run_id("r2")], pending); + assert_eq!(state.runs.len(), 2); + + // R2 starts through normal progress, without another Accepted or a resubmission. + let state = state.apply(PlaylistUpdateCardStatusAction::Progress { + connection_context: connection(1), + progress: PlaylistUpdateProgressEvent::for_run_input( + run_id("r2"), + execution_order(2), + 7, + "input-7", + "r2 cache acquisition", + ) + .with_input_telemetry(r2_telemetry.clone()), + }); + let mut r2_runtime = InputUpdateCardRuntime { + library_scan_result: None, + status: InputUpdateCardStatus::Updating, + action_status: InputUpdateCardStatus::Updating, + details: vec!["r2 cache acquisition".into()], + localized_details: Vec::new(), + run_id: Some(run_id("r2")), + execution_order: Some(execution_order(2)), + refresh_policy: Some(InputRefreshPolicy::NORMAL), + input_telemetry: Some(r2_telemetry.clone()), + }; + assert_eq!(state.for_input(7), r2_runtime); + let state = state.apply(input_completed("r2", 2, 7, PlaylistUpdateState::Success, "r2 input completed")); + r2_runtime.status = InputUpdateCardStatus::Success; + r2_runtime.details.push("r2 input completed".into()); + r2_runtime.localized_details.push(PlaylistUpdateProgressDetail::InputCompletedSuccessfully); + assert_eq!(state.for_input(7), r2_runtime); + let state = state.apply(accepted_with_policy("r1", &[7], InputRefreshPolicy::FORCE)); + assert_eq!(state.for_input(7), r2_runtime); + let state = state.apply(completed("r2", 2, PlaylistUpdateState::Success)); + r2_runtime.action_status = InputUpdateCardStatus::Success; + assert_eq!(state.for_input(7), r2_runtime); + let state = state.apply(accepted_with_policy("r1", &[7], InputRefreshPolicy::FORCE)); + assert_eq!(state.for_input(7), r2_runtime); + } + } + + fn input(id: u16, name: &str, input_type: InputType) -> ConfigInputDto { + ConfigInputDto { id, name: name.intern(), input_type, enabled: true, ..ConfigInputDto::default() } + } + + fn target(id: u16, name: &str) -> ConfigTargetDto { + ConfigTargetDto { id, name: name.to_string(), ..ConfigTargetDto::default() } + } + + #[test] + fn playlist_update_bulk_m3u_only_target_is_executable_without_a_manual_card() { + let sources = SourcesConfigDto { + inputs: vec![input(50, "m3u", InputType::M3u)], + sources: vec![ConfigSourceDto { inputs: vec!["m3u".intern()], targets: vec![target(5, "m3u-target")] }], + ..SourcesConfigDto::default() + }; + + assert_eq!( + playlist_update_bulk_scope(&sources, &[]), + PlaylistUpdateBulkScope { can_submit: true, visible_input_ids: Vec::new() } + ); + } + + #[test] + fn playlist_update_bulk_queues_only_visible_targeted_cards_and_ignores_card_policy() { + let sources = SourcesConfigDto { + inputs: vec![input(7, "xtream", InputType::Xtream), input(50, "m3u", InputType::M3u)], + sources: vec![ConfigSourceDto { + inputs: vec!["xtream".intern(), "m3u".intern()], + targets: vec![target(1, "shared")], + }], + ..SourcesConfigDto::default() + }; + let cards = vec![card(7, &[1]), card(8, &[])]; + let local_controls = + InputUpdateCardState { selected_target_ids: Vec::new(), policy: InputRefreshPolicy::FORCE }; + + assert_eq!( + playlist_update_bulk_scope(&sources, &cards), + PlaylistUpdateBulkScope { can_submit: true, visible_input_ids: vec![7] } + ); + assert!(local_controls.selected_target_ids.is_empty()); + assert_eq!(local_controls.policy, InputRefreshPolicy::FORCE); + } + + #[test] + fn playlist_update_bulk_without_an_executable_target_is_disabled() { + let mut disabled_target = target(1, "disabled"); + disabled_target.enabled = false; + let sources = SourcesConfigDto { + inputs: vec![input(7, "xtream", InputType::Xtream)], + sources: vec![ConfigSourceDto { inputs: vec!["xtream".intern()], targets: vec![disabled_target] }], + ..SourcesConfigDto::default() + }; + + assert_eq!(playlist_update_bulk_scope(&sources, &[card(7, &[1])]), PlaylistUpdateBulkScope::default()); + + let no_targets = SourcesConfigDto { + inputs: vec![input(7, "xtream", InputType::Xtream)], + sources: vec![ConfigSourceDto { inputs: vec!["xtream".intern()], targets: Vec::new() }], + ..SourcesConfigDto::default() + }; + assert_eq!(playlist_update_bulk_scope(&no_targets, &[card(7, &[])]), PlaylistUpdateBulkScope::default()); + } + + #[test] + fn playlist_update_bulk_excludes_disabled_visible_inputs_by_id_not_name() { + let sources = SourcesConfigDto { + inputs: vec![ + ConfigInputDto { enabled: false, ..input(7, "same", InputType::Xtream) }, + input(50, "same", InputType::M3u), + ], + sources: vec![ConfigSourceDto { + inputs: vec!["same".intern(), "same".intern()], + targets: vec![target(1, "shared")], + }], + ..SourcesConfigDto::default() + }; + let mut cards = build_input_update_card_models(&sources, &Default::default()); + assert_eq!(cards.iter().map(|card| card.input_id).collect::>(), vec![7, 50]); + assert_eq!(cards[0].capabilities, InputUpdateCapabilities::Disabled); + assert_eq!(cards[0].targets, cards[1].targets); + // A card without a corresponding configured input must not join the accepted scope either. + cards.push(card(99, &[1])); + + assert_eq!( + playlist_update_bulk_scope(&sources, &cards), + PlaylistUpdateBulkScope { can_submit: true, visible_input_ids: vec![50] } + ); + } + + #[test] + fn playlist_update_bulk_includes_enabled_inputs_independently_of_force_capability() { + for (input_type, supports_force) in [(InputType::M3u, true), (InputType::Plex, false)] { + let sources = SourcesConfigDto { + inputs: vec![input(50, "provider", input_type)], + sources: vec![ConfigSourceDto { + inputs: vec!["provider".intern()], + targets: vec![target(5, "target")], + }], + ..SourcesConfigDto::default() + }; + let cards = build_input_update_card_models(&sources, &Default::default()); + assert_eq!(cards[0].capabilities.supports(InputRefreshPolicy::FORCE), supports_force); + assert_eq!( + playlist_update_bulk_scope(&sources, &cards), + PlaylistUpdateBulkScope { can_submit: true, visible_input_ids: vec![50] } + ); + } + } + + #[test] + fn playlist_update_bulk_includes_enabled_update_only_cards_in_configuration_order() { + let sources = SourcesConfigDto { + inputs: vec![ + input(30, "m3u-batch", InputType::M3uBatch), + input(10, "xtream-batch", InputType::XtreamBatch), + input(20, "stalker-batch", InputType::StalkerBatch), + ], + sources: vec![ConfigSourceDto { + inputs: vec!["xtream-batch".intern(), "stalker-batch".intern(), "m3u-batch".intern()], + targets: vec![target(5, "shared")], + }], + ..SourcesConfigDto::default() + }; + let cards = build_input_update_card_models(&sources, &Default::default()); + assert!(cards.iter().all(|card| card.capabilities == InputUpdateCapabilities::BulkOnly)); + assert_eq!( + playlist_update_bulk_scope(&sources, &cards), + PlaylistUpdateBulkScope { can_submit: true, visible_input_ids: vec![30, 10, 20] } + ); + } + + #[test] + fn playlist_update_bulk_excludes_targetless_and_disabled_target_cards_from_an_executable_run() { + let sources = SourcesConfigDto { + inputs: vec![ + input(7, "targetless", InputType::Xtream), + input(8, "disabled-target", InputType::M3u), + input(9, "active-target", InputType::XtreamBatch), + ], + sources: vec![ + ConfigSourceDto { inputs: vec!["targetless".intern()], targets: Vec::new() }, + ConfigSourceDto { + inputs: vec!["disabled-target".intern()], + targets: vec![ConfigTargetDto { enabled: false, ..target(1, "disabled") }], + }, + ConfigSourceDto { inputs: vec!["active-target".intern()], targets: vec![target(2, "active")] }, + ], + ..SourcesConfigDto::default() + }; + let cards = build_input_update_card_models(&sources, &Default::default()); + assert_eq!(cards.len(), 3); + assert_eq!( + playlist_update_bulk_scope(&sources, &cards), + PlaylistUpdateBulkScope { can_submit: true, visible_input_ids: vec![9] } + ); + } + + #[test] + fn playlist_update_bulk_only_disabled_inputs_cannot_submit_or_queue_visible_cards() { + let sources = SourcesConfigDto { + inputs: vec![ConfigInputDto { enabled: false, ..input(7, "disabled", InputType::Xtream) }], + sources: vec![ConfigSourceDto { inputs: vec!["disabled".intern()], targets: vec![target(1, "active")] }], + ..SourcesConfigDto::default() + }; + let cards = build_input_update_card_models(&sources, &Default::default()); + assert_eq!(cards.len(), 1); + assert_eq!(playlist_update_bulk_scope(&sources, &cards), PlaylistUpdateBulkScope::default()); + } + + #[test] + fn playlist_update_bulk_disabled_card_is_unchanged_through_the_accepted_run() { + let sources = SourcesConfigDto { + inputs: vec![ + ConfigInputDto { enabled: false, ..input(7, "disabled", InputType::Xtream) }, + input(8, "enabled", InputType::M3uBatch), + ], + sources: vec![ConfigSourceDto { + inputs: vec!["disabled".intern(), "enabled".intern()], + targets: vec![target(1, "shared")], + }], + ..SourcesConfigDto::default() + }; + let cards = build_input_update_card_models(&sources, &Default::default()); + assert_eq!(cards.len(), 2); + let scope = playlist_update_bulk_scope(&sources, &cards); + assert!(scope.can_submit); + for terminal in [PlaylistUpdateState::Success, PlaylistUpdateState::Partial, PlaylistUpdateState::Failure] { + let state = reduce(statuses(), accepted("bulk", &scope.visible_input_ids)); + assert_eq!(state.for_input(7), InputUpdateCardRuntime::default()); + assert_eq!(state.for_input(8).status(), InputUpdateCardStatus::Queued); + let state = reduce(state, progress("bulk", 1, 8, "started")); + assert_eq!(state.for_input(7), InputUpdateCardRuntime::default()); + assert_eq!(state.for_input(8).status(), InputUpdateCardStatus::Updating); + let state = reduce(state, completed("bulk", 1, terminal)); + assert_eq!(state.for_input(7), InputUpdateCardRuntime::default()); + assert_eq!(state.for_input(8).status(), status_from_state(terminal)); + + // Only an actual input event may bring the excluded ID into a run's observed scope. + let state = reduce(state, progress("observed", 2, 7, "actual input event")); + assert_eq!(state.for_input(7).status(), InputUpdateCardStatus::Updating); + assert_eq!(state.for_input(7).details(), ["actual input event"]); + } + } + + #[test] + fn playlist_update_status_progress_then_same_run_acceptance_does_not_create_a_phantom_queue() { + let state = reduce(statuses(), progress("r", 1, 7, "started")); + let state = reduce(state, accepted("r", &[7])); + assert_eq!(state.for_input(7).status(), InputUpdateCardStatus::Updating); + assert_eq!(state.for_input(7).refresh_policy(), Some(InputRefreshPolicy::NORMAL)); + assert_eq!(state.runs.len(), 1); + let state = reduce(state, completed("r", 1, PlaylistUpdateState::Success)); + + assert_eq!(state.for_input(7).status(), InputUpdateCardStatus::Success); + } + + #[test] + fn playlist_update_status_exposes_effective_run_identity_from_the_existing_lifecycle() { + let state = reduce(statuses(), accepted("stable-run", &[7])); + let queued = state.for_input(7); + assert_eq!(queued.run_id().map(PlaylistUpdateRunId::as_ref), Some("stable-run")); + assert_eq!(queued.execution_order(), None); + assert_eq!(queued.refresh_policy(), Some(InputRefreshPolicy::NORMAL)); + + let state = reduce(state, progress("stable-run", 17, 7, "started")); + let updating = state.for_input(7); + assert_eq!(updating.run_id().map(PlaylistUpdateRunId::as_ref), Some("stable-run")); + assert_eq!(updating.execution_order().map(PlaylistUpdateRunOrder::get), Some(17)); + assert_eq!(updating.refresh_policy(), Some(InputRefreshPolicy::NORMAL)); + + let state = reduce(state, completed("stable-run", 17, PlaylistUpdateState::Partial)); + let terminal = state.for_input(7); + assert_eq!(terminal.status(), InputUpdateCardStatus::Partial); + assert_eq!(terminal.run_id().map(PlaylistUpdateRunId::as_ref), Some("stable-run")); + assert_eq!(terminal.execution_order().map(PlaylistUpdateRunOrder::get), Some(17)); + assert_eq!(terminal.refresh_policy(), Some(InputRefreshPolicy::NORMAL)); + assert_eq!(state.for_input(8), InputUpdateCardRuntime::default()); + } + + #[test] + fn playlist_update_status_manual_policy_applies_only_to_accepted_input_ids() { + let state = reduce(statuses(), accepted_with_policy("manual", &[7], InputRefreshPolicy::FORCE)); + let state = reduce(state, progress("manual", 17, 7, "requested input")); + let state = reduce(state, progress("manual", 17, 9, "target dependency")); + + assert_eq!(state.for_input(7).refresh_policy(), Some(InputRefreshPolicy::FORCE)); + assert_eq!(state.for_input(9).refresh_policy(), None); + assert_eq!(state.for_input(7).status(), InputUpdateCardStatus::Updating); + assert_eq!(state.for_input(9).status(), InputUpdateCardStatus::Updating); + + let state = reduce(state, completed("manual", 17, PlaylistUpdateState::Success)); + assert_eq!(state.for_input(7).refresh_policy(), Some(InputRefreshPolicy::FORCE)); + assert_eq!(state.for_input(9).refresh_policy(), None); + } + + #[test] + fn playlist_update_status_bulk_policy_applies_to_every_explicitly_accepted_input() { + let state = reduce(statuses(), accepted("bulk", &[7, 9])); + + assert_eq!(state.for_input(7).refresh_policy(), Some(InputRefreshPolicy::NORMAL)); + assert_eq!(state.for_input(9).refresh_policy(), Some(InputRefreshPolicy::NORMAL)); + assert_eq!(state.for_input(8).refresh_policy(), None); + } + + #[test] + fn playlist_update_status_late_acceptance_adds_policy_without_changing_terminal_state() { + let state = reduce(statuses(), progress("late", 23, 7, "started before acceptance")); + let state = + reduce(state, input_completed("late", 23, 7, PlaylistUpdateState::Partial, "finished before acceptance")); + let state = reduce(state, completed("late", 23, PlaylistUpdateState::Partial)); + let state = reduce(state, accepted_with_policy("late", &[7], InputRefreshPolicy::REFRESH)); + let runtime = state.for_input(7); + + assert_eq!(runtime.status(), InputUpdateCardStatus::Partial); + assert_eq!(runtime.run_id().map(PlaylistUpdateRunId::as_ref), Some("late")); + assert_eq!(runtime.execution_order().map(PlaylistUpdateRunOrder::get), Some(23)); + assert_eq!(runtime.details(), ["started before acceptance", "finished before acceptance"]); + assert_eq!(runtime.refresh_policy(), Some(InputRefreshPolicy::REFRESH)); + assert_eq!(state.runs.len(), 1); + } + + #[test] + fn playlist_update_status_terminal_policy_survives_run_pruning() { + let state = reduce(statuses(), accepted_with_policy("retained", &[7], InputRefreshPolicy::FORCE)); + let state = + reduce(state, input_completed("retained", 1, 7, PlaylistUpdateState::Success, "retained terminal detail")); + let mut state = reduce(state, completed("retained", 1, PlaylistUpdateState::Success)); + + for offset in 0..MAX_TRACKED_RUNS { + let run = format!("later-{offset}"); + let input_id = 100 + u16::try_from(offset).expect("tracked-run limit fits u16"); + state = reduce(state, accepted(&run, &[input_id])); + state = reduce( + state, + completed( + &run, + u64::try_from(offset).expect("tracked-run limit fits u64") + 2, + PlaylistUpdateState::Success, + ), + ); + } + + assert!(!state.runs.contains_key(&run_id("retained"))); + let runtime = state.for_input(7); + assert_eq!(runtime.run_id().map(PlaylistUpdateRunId::as_ref), Some("retained")); + assert_eq!(runtime.execution_order().map(PlaylistUpdateRunOrder::get), Some(1)); + assert_eq!(runtime.status(), InputUpdateCardStatus::Success); + assert_eq!(runtime.details(), ["retained terminal detail"]); + assert_eq!(runtime.refresh_policy(), Some(InputRefreshPolicy::FORCE)); + } + + #[test] + fn playlist_update_status_progress_only_run_does_not_invent_normal_policy() { + let state = reduce(statuses(), progress("automatic", 5, 7, "automatic progress")); + assert_eq!(state.for_input(7).refresh_policy(), None); + + let state = reduce(state, completed("automatic", 5, PlaylistUpdateState::Success)); + assert_eq!(state.for_input(7).status(), InputUpdateCardStatus::Success); + assert_eq!(state.for_input(7).refresh_policy(), None); + } + + #[test] + fn playlist_update_status_follow_up_run_remains_queued_after_previous_run_completes() { + let state = reduce(statuses(), progress("r1", 1, 7, "first")); + let state = reduce(state, accepted("r2", &[7])); + let state = reduce(state, completed("r1", 1, PlaylistUpdateState::Success)); + + assert_eq!(state.for_input(7).status(), InputUpdateCardStatus::Queued); + } + + #[test] + fn pipeline_transparency_playlist_update_status_hides_previous_run_facts_from_queued_follow_up() { + let previous_telemetry = PlaylistUpdateInputTelemetry { + refresh_policy: InputRefreshPolicy::NORMAL, + source: Some(PlaylistUpdateDataSource::Cache), + clusters: Vec::new(), + }; + let state = reduce(statuses(), accepted("r1", &[7])); + let state = reduce( + state, + PlaylistUpdateCardStatusAction::Progress { + connection_context: connection(1), + progress: PlaylistUpdateProgressEvent::for_run_input( + run_id("r1"), + execution_order(1), + 7, + "input-7", + "r1 cache result", + ) + .with_input_telemetry(previous_telemetry.clone()), + }, + ); + let state = reduce(state, input_completed("r1", 1, 7, PlaylistUpdateState::Success, "r1 completed")); + let state = reduce(state, completed("r1", 1, PlaylistUpdateState::Success)); + let previous = state.for_input(7); + assert_eq!(previous.input_telemetry(), Some(&previous_telemetry)); + assert_eq!(previous.details(), ["r1 cache result", "r1 completed"]); + + let state = reduce(state, accepted_with_policy("r2", &[7], InputRefreshPolicy::FORCE)); + let queued = state.for_input(7); + + assert_eq!(queued.status(), InputUpdateCardStatus::Queued); + assert_eq!(queued.run_id().map(PlaylistUpdateRunId::as_ref), Some("r2")); + assert_eq!(queued.refresh_policy(), Some(InputRefreshPolicy::FORCE)); + assert!(queued.details().is_empty()); + assert_eq!(queued.input_telemetry(), None); + } + + #[test] + fn pipeline_transparency_playlist_update_status_in_run_reuse_preserves_acquisition_telemetry() { + let provider_telemetry = PlaylistUpdateInputTelemetry { + refresh_policy: InputRefreshPolicy::FORCE, + source: None, + clusters: vec![PlaylistUpdateClusterTelemetry { + cluster: XtreamCluster::Live, + requested: true, + source: Some(PlaylistUpdateDataSource::Provider), + baseline_count: None, + candidate_count: None, + active_count: None, + threshold: Some(95), + quality: None, + decision: None, + technical_state: None, + }], + }; + let state = reduce(statuses(), accepted_with_policy("shared", &[7], InputRefreshPolicy::FORCE)); + let state = reduce( + state, + PlaylistUpdateCardStatusAction::Progress { + connection_context: connection(1), + progress: PlaylistUpdateProgressEvent::for_run_input( + run_id("shared"), + execution_order(7), + 7, + "input-7", + "provider acquisition", + ) + .with_input_telemetry(provider_telemetry.clone()), + }, + ); + + let state = + reduce(state, input_completed("shared", 7, 7, PlaylistUpdateState::Success, "in-run reuse completed")); + let runtime = state.for_input(7); + + assert_eq!(runtime.input_telemetry(), Some(&provider_telemetry)); + assert_eq!(runtime.details(), ["provider acquisition", "in-run reuse completed"]); + assert_eq!(runtime.refresh_policy(), Some(InputRefreshPolicy::FORCE)); + } + + #[test] + fn playlist_update_status_accepted_run_failing_before_progress_finishes_as_failed() { + let state = reduce(statuses(), accepted("r", &[7])); + let state = reduce(state, completed("r", 1, PlaylistUpdateState::Failure)); + + assert_eq!(state.for_input(7).status(), InputUpdateCardStatus::Failed); + } + + #[test] + fn playlist_update_status_unknown_terminal_changes_no_active_or_queued_card() { + let state = reduce(statuses(), progress("active", 1, 7, "started")); + let state = reduce(state, accepted("queued", &[8])); + let state = reduce(state, completed("unknown", 2, PlaylistUpdateState::Failure)); + + assert_eq!(state.for_input(7).status(), InputUpdateCardStatus::Updating); + assert_eq!(state.for_input(8).status(), InputUpdateCardStatus::Queued); + assert_eq!(state.for_input(9).status(), InputUpdateCardStatus::Ready); + } + + #[test] + fn playlist_update_status_late_acceptance_after_early_terminal_uses_the_same_run() { + let state = reduce(statuses(), completed("r", 1, PlaylistUpdateState::Failure)); + let state = reduce(state, accepted("r", &[7])); + + assert_eq!(state.for_input(7).status(), InputUpdateCardStatus::Failed); + } + + #[test] + fn playlist_update_status_later_execution_replaces_earlier_status_and_details() { + let state = reduce(statuses(), accepted("r2", &[7])); + assert_eq!(state.for_input(7).status(), InputUpdateCardStatus::Queued); + let state = reduce(state, progress("r1", 1, 7, "r1 started")); + assert_eq!(state.for_input(7).status(), InputUpdateCardStatus::Updating); + let state = reduce(state, input_completed("r1", 1, 7, PlaylistUpdateState::Success, "r1 success")); + let state = reduce(state, completed("r1", 1, PlaylistUpdateState::Success)); + assert_eq!(state.for_input(7).status(), InputUpdateCardStatus::Queued); + + let state = reduce(state, progress("r2", 2, 7, "r2 started")); + assert_eq!(state.for_input(7).status(), InputUpdateCardStatus::Updating); + let state = reduce(state, input_completed("r2", 2, 7, PlaylistUpdateState::Failure, "r2 failed")); + let state = reduce(state, completed("r2", 2, PlaylistUpdateState::Failure)); + + let runtime = state.for_input(7); + assert_eq!(runtime.status(), InputUpdateCardStatus::Failed); + assert_eq!(runtime.details(), ["r2 started", "r2 failed"]); + let terminal = state.terminal_by_input_id.get(&7).expect("terminal state"); + assert_eq!(terminal.run_id, run_id("r2")); + assert_eq!(terminal.execution_order, execution_order(2)); + let details = state.details_by_input_id.get(&7).expect("input details"); + assert_eq!(details.run_id, run_id("r2")); + assert_eq!(details.execution_order, execution_order(2)); + } + + #[test] + fn playlist_update_status_later_execution_early_failure_clears_queue_and_older_details() { + let state = reduce(statuses(), accepted("r2", &[7])); + let state = reduce(state, progress("r1", 1, 7, "r1 started")); + let state = reduce(state, input_completed("r1", 1, 7, PlaylistUpdateState::Success, "r1 success")); + let state = reduce(state, completed("r1", 1, PlaylistUpdateState::Success)); + let state = reduce(state, completed("r2", 2, PlaylistUpdateState::Failure)); + + let runtime = state.for_input(7); + assert_eq!(runtime.status(), InputUpdateCardStatus::Failed); + assert!(runtime.details().is_empty()); + let terminal = state.terminal_by_input_id.get(&7).expect("terminal state"); + assert_eq!(terminal.run_id, run_id("r2")); + assert_eq!(terminal.execution_order, execution_order(2)); + } + + #[test] + fn playlist_update_status_late_older_acceptance_and_terminal_cannot_replace_newer_execution() { + let state = reduce(statuses(), progress("r1", 1, 7, "r1 started")); + let state = reduce(state, input_completed("r1", 1, 7, PlaylistUpdateState::Success, "r1 success")); + let state = reduce(state, accepted("r2", &[7])); + let state = reduce(state, progress("r2", 2, 7, "r2 started")); + let state = reduce(state, input_completed("r2", 2, 7, PlaylistUpdateState::Failure, "r2 failed")); + let state = reduce(state, completed("r2", 2, PlaylistUpdateState::Failure)); + let state = reduce(state, accepted("r1", &[7])); + let state = reduce(state, completed("r1", 1, PlaylistUpdateState::Success)); + + let runtime = state.for_input(7); + assert_eq!(runtime.status(), InputUpdateCardStatus::Failed); + assert_eq!(runtime.details(), ["r2 started", "r2 failed"]); + assert_eq!(state.terminal_by_input_id.get(&7).map(|terminal| &terminal.run_id), Some(&run_id("r2"))); + assert_eq!(state.details_by_input_id.get(&7).map(|details| &details.run_id), Some(&run_id("r2"))); + } + + #[test] + fn playlist_update_status_backend_restart_accepts_lower_order_for_badge_and_details() { + let runtime_a = connection(1); + let runtime_b = connection(2); + let state = PlaylistUpdateCardStatuses::for_connection(runtime_a); + let state = reduce(state, progress_in(runtime_a, "r1", 42, 7, "r1 started")); + let state = + reduce(state, input_completed_in(runtime_a, "r1", 42, 7, PlaylistUpdateState::Success, "r1 success")); + let state = reduce(state, completed_in(runtime_a, "r1", 42, PlaylistUpdateState::Success)); + assert_eq!(state.for_input(7).status(), InputUpdateCardStatus::Success); + assert_eq!(state.for_input(7).details(), ["r1 started", "r1 success"]); + + let state = reduce(state, connection_changed(disconnected_connection(1))); + assert_eq!(state.for_input(7), InputUpdateCardRuntime::default()); + let state = reduce(state, connection_changed(runtime_b)); + let state = reduce(state, accepted_in(runtime_b, "r2", &[7])); + let state = reduce(state, progress_in(runtime_b, "r2", 1, 7, "r2 started")); + assert_eq!(state.for_input(7).status(), InputUpdateCardStatus::Updating); + let state = reduce(state, input_completed_in(runtime_b, "r2", 1, 7, PlaylistUpdateState::Failure, "r2 failed")); + let state = reduce(state, completed_in(runtime_b, "r2", 1, PlaylistUpdateState::Failure)); + + assert_eq!(state.for_input(7).status(), InputUpdateCardStatus::Failed); + assert_eq!(state.for_input(7).details(), ["r2 started", "r2 failed"]); + } + + #[test] + fn playlist_update_status_equal_orders_in_different_connections_are_not_tied() { + let runtime_a = connection(1); + let runtime_b = connection(2); + let state = PlaylistUpdateCardStatuses::for_connection(runtime_a); + let state = + reduce(state, input_completed_in(runtime_a, "r1", 1, 7, PlaylistUpdateState::Success, "r1 success")); + let state = reduce(state, completed_in(runtime_a, "r1", 1, PlaylistUpdateState::Success)); + let state = reduce(state, connection_changed(disconnected_connection(1))); + let state = reduce(state, connection_changed(runtime_b)); + let state = reduce(state, accepted_in(runtime_b, "r2", &[7])); + let state = reduce(state, input_completed_in(runtime_b, "r2", 1, 7, PlaylistUpdateState::Failure, "r2 failed")); + let state = reduce(state, completed_in(runtime_b, "r2", 1, PlaylistUpdateState::Failure)); + + assert_eq!(state.for_input(7).status(), InputUpdateCardStatus::Failed); + assert_eq!(state.for_input(7).details(), ["r2 failed"]); + } + + #[test] + fn playlist_update_status_late_events_from_left_connection_are_ignored_without_queue() { + let runtime_a = connection(1); + let runtime_b = connection(2); + let state = PlaylistUpdateCardStatuses::for_connection(runtime_a); + let state = reduce(state, connection_changed(disconnected_connection(1))); + let state = reduce(state, connection_changed(runtime_b)); + let state = reduce(state, accepted_in(runtime_b, "r2", &[7])); + let state = reduce(state, input_completed_in(runtime_b, "r2", 1, 7, PlaylistUpdateState::Failure, "r2 failed")); + let state = reduce(state, completed_in(runtime_b, "r2", 1, PlaylistUpdateState::Failure)); + let state = reduce(state, accepted_in(runtime_a, "r1", &[7])); + let state = reduce(state, progress_in(runtime_a, "r1", 42, 7, "late r1 detail")); + let state = reduce(state, completed_in(runtime_a, "r1", 42, PlaylistUpdateState::Success)); + + assert_eq!(state.for_input(7).status(), InputUpdateCardStatus::Failed); + assert_eq!(state.for_input(7).details(), ["r2 failed"]); + assert!(!state.runs.contains_key(&run_id("r1"))); + } + + #[test] + fn playlist_update_status_reconnect_clears_orphaned_runs_without_replay_or_terminal_guess() { + let runtime_a = connection(1); + let runtime_b = connection(2); + let state = PlaylistUpdateCardStatuses::for_connection(runtime_a); + let state = reduce(state, accepted_in(runtime_a, "queued", &[7])); + let state = reduce(state, progress_in(runtime_a, "active", 1, 8, "active detail")); + assert_eq!(state.for_input(7).status(), InputUpdateCardStatus::Queued); + assert_eq!(state.for_input(8).status(), InputUpdateCardStatus::Updating); + + let state = reduce(state, connection_changed(disconnected_connection(1))); + let state = reduce(state, connection_changed(runtime_b)); + let state = reduce(state, completed_in(runtime_a, "active", 1, PlaylistUpdateState::Success)); + + assert_eq!(state.for_input(7), InputUpdateCardRuntime::default()); + assert_eq!(state.for_input(8), InputUpdateCardRuntime::default()); + assert!(state.runs.is_empty()); + } + + #[test] + fn playlist_update_status_mixed_bulk_keeps_input_outcomes_distinct_from_global_partial() { + let state = reduce(statuses(), accepted("bulk", &[7, 8])); + let state = reduce(state, input_completed("bulk", 1, 7, PlaylistUpdateState::Success, "success")); + let state = reduce(state, progress("bulk", 1, 8, "vod quality rejected")); + let state = reduce(state, input_completed("bulk", 1, 8, PlaylistUpdateState::Partial, "partial")); + let state = reduce(state, completed("bulk", 1, PlaylistUpdateState::Partial)); + + assert_eq!(state.for_input(7).status(), InputUpdateCardStatus::Success); + assert_eq!(state.for_input(8).status(), InputUpdateCardStatus::Partial); + } + + #[test] + fn playlist_update_status_technical_error_detail_is_scoped_to_the_failed_input() { + let state = reduce(statuses(), accepted("bulk", &[7, 8])); + let state = reduce( + state, + input_completed("bulk", 1, 7, PlaylistUpdateState::Failure, "Input 'a' failed during update"), + ); + let state = reduce(state, input_completed("bulk", 1, 8, PlaylistUpdateState::Success, "Input 'b' completed")); + + assert!(state.for_input(7).details().iter().any(|line| line.contains("failed during update"))); + assert!(!state.for_input(8).details().iter().any(|line| line.contains("failed during update"))); + } + + #[test] + fn playlist_update_status_correlated_legacy_terminal_without_order_cannot_replace_ordered_result() { + let state = reduce(statuses(), accepted("r2", &[7])); + let state = reduce(state, input_completed("r2", 2, 7, PlaylistUpdateState::Failure, "r2 failed")); + let state = reduce(state, completed("r2", 2, PlaylistUpdateState::Failure)); + let state = reduce( + state, + PlaylistUpdateCardStatusAction::Completed { + connection_context: connection(1), + completed: PlaylistUpdateRunStateEvent { + run_id: Some(run_id("r1")), + execution_order: None, + state: PlaylistUpdateState::Success, + }, + }, + ); + let state = reduce(state, accepted("r1", &[7])); + + let runtime = state.for_input(7); + assert_eq!(runtime.status(), InputUpdateCardStatus::Failed); + assert_eq!(runtime.details(), ["r2 failed"]); + } + + #[test] + fn playlist_update_status_legacy_events_without_run_identity_do_not_mutate_cards() { + let state = reduce( + statuses(), + PlaylistUpdateCardStatusAction::Progress { + connection_context: connection(1), + progress: PlaylistUpdateProgressEvent::for_input(7, "legacy", "old"), + }, + ); + let state = reduce( + state, + PlaylistUpdateCardStatusAction::Completed { + connection_context: connection(1), + completed: PlaylistUpdateRunStateEvent::uncorrelated(PlaylistUpdateState::Failure), + }, + ); + + assert_eq!(state.for_input(7), InputUpdateCardRuntime::default()); + } +} diff --git a/frontend/src/model/staged_completion_tests.rs b/frontend/src/model/staged_completion_tests.rs new file mode 100644 index 000000000..1d9bd9d2f --- /dev/null +++ b/frontend/src/model/staged_completion_tests.rs @@ -0,0 +1,232 @@ +use super::{ + build_input_update_card_models, build_input_update_run_views, InputUpdateCardStatus, + PlaylistUpdateCardStatusAction, PlaylistUpdateCardStatuses, +}; +use crate::services::WebSocketConnectionContext; +use serde::Deserialize; +use shared::model::{ + ConfigInputDto, InputPlaylistUpdateStatusDto, InputType, PersistedPlaylistUpdateInputResult, + PlaylistUpdateProgressEvent, PlaylistUpdateRunStateEvent, PlaylistUpdateState, PlaylistUpdateStatusDto, + SourcesConfigDto, +}; +use std::rc::Rc; +use yew::Reducible; + +// This exact fixture is checked against a real parent/staged acquisition, +// canonical status.json and exec_processing's emitted global completion in +// tuliprox-processing::...::staged_completion_tests (playlist_update_status). +#[derive(Deserialize)] +struct StagedTrace { + progress: Vec, + completed: PlaylistUpdateRunStateEvent, + persisted_input_result: PersistedPlaylistUpdateInputResult, +} + +fn traces() -> Vec { + let traces: Vec<_> = serde_json::from_str(include_str!("test_data/staged_completions.json")).unwrap(); + assert_eq!(traces.len(), 3, "processing fixture must cover Partial, Success and Failure"); + traces +} + +fn sources() -> SourcesConfigDto { + SourcesConfigDto { + // Names and order differ from the captured events: identities must carry. + inputs: [(8, "renamed-staged", InputType::Staged), (9, "other", InputType::M3u), (7, "parent", InputType::M3u)] + .into_iter() + .map(|(id, name, input_type)| ConfigInputDto { + id, + name: name.into(), + input_type, + staged_type: shared::model::StagedInputType::Xtream, + enabled: true, + ..ConfigInputDto::default() + }) + .collect(), + ..SourcesConfigDto::default() + } +} + +#[test] +fn playlist_update_status_pipeline_transparency_reload_restores_running_target_rebuild_without_inheriting_follow_up_facts( +) { + for trace in traces() { + let connection = WebSocketConnectionContext::new(2, true); + let own = + trace.progress.iter().find(|event| event.input_id == Some(8) && event.state.is_some()).unwrap().clone(); + let sources = sources(); + let status = PlaylistUpdateStatusDto::default(); + let cards = build_input_update_card_models(&sources, &status); + let mut state = Rc::new(PlaylistUpdateCardStatuses::for_connection(connection)); + state = state.reduce(PlaylistUpdateCardStatusAction::RestoreActive { + connection_context: connection, + progress: vec![own.clone()], + }); + let restored = build_input_update_run_views(&sources, &cards, &state, &status).remove(0); + assert_eq!(restored.overall_state, InputUpdateCardStatus::Updating); + assert_eq!(restored.run_id, own.run_id); + assert_eq!(restored.execution_order, own.execution_order); + assert_eq!(state.for_input(8).input_telemetry(), own.input_telemetry.as_ref()); + state = state.reduce(PlaylistUpdateCardStatusAction::Accepted(super::PlaylistUpdateAcceptedScope { + connection_context: connection, + run_id: "follow-up".into(), + input_ids: vec![8], + refresh_policy: shared::model::InputRefreshPolicy::FORCE, + })); + let rebuilding = build_input_update_run_views(&sources, &cards, &state, &status).remove(0); + assert_eq!(rebuilding.run_id, own.run_id); + assert_eq!(rebuilding.overall_state, InputUpdateCardStatus::Updating); + assert_eq!(rebuilding.input_state, restored.input_state); + state = state.reduce(PlaylistUpdateCardStatusAction::Completed { + connection_context: connection, + completed: trace.completed, + }); + // Even a late HTTP response from before Completed cannot resurrect R1. + state = state.reduce(PlaylistUpdateCardStatusAction::RestoreActive { + connection_context: connection, + progress: vec![own], + }); + let queued = build_input_update_run_views(&sources, &cards, &state, &status).remove(0); + assert_eq!(queued.run_id, Some("follow-up".into())); + assert_eq!(queued.overall_state, InputUpdateCardStatus::Queued); + assert!(queued.progress_details.is_empty()); + assert!(state.for_input(8).input_telemetry().is_none()); + } +} + +#[test] +fn playlist_update_status_reload_restores_updating_but_never_overwrites_live_or_completed_run() { + let connection = WebSocketConnectionContext::new(2, true); + let snapshot = PlaylistUpdateProgressEvent::for_run_input("current".into(), 7.into(), 8, "staged", "snapshot"); + let restore = || PlaylistUpdateCardStatusAction::RestoreActive { + connection_context: connection, + progress: vec![snapshot.clone()], + }; + let empty = || Rc::new(PlaylistUpdateCardStatuses::for_connection(connection)); + let restored = empty().reduce(restore()); + assert_eq!(restored.for_input(8).status(), InputUpdateCardStatus::Updating); + assert_eq!(restored.for_input(8).action_status(), InputUpdateCardStatus::Updating); + let live = PlaylistUpdateProgressEvent::input_completed( + "current".into(), + 7.into(), + 8, + PlaylistUpdateState::Partial, + "staged", + "newer live fact", + ); + let progressed = + empty().reduce(PlaylistUpdateCardStatusAction::Progress { connection_context: connection, progress: live }); + let after = progressed.clone().reduce(restore()); + assert_eq!(*after, *progressed); + let completed = empty().reduce(PlaylistUpdateCardStatusAction::Completed { + connection_context: connection, + completed: PlaylistUpdateRunStateEvent::correlated("current".into(), 7.into(), PlaylistUpdateState::Failure), + }); + let after = completed.clone().reduce(restore()); + assert_eq!(*after, *completed, "terminal received before the snapshot wins even for a previously unknown run"); + let new_socket = + empty().reduce(PlaylistUpdateCardStatusAction::ConnectionChanged(WebSocketConnectionContext::new(3, true))); + let after = new_socket.clone().reduce(restore()); + assert_eq!(*after, *new_socket); +} + +#[test] +fn playlist_update_status_pipeline_transparency_real_staged_trace_keeps_own_result_after_global_failure_and_reload() { + for trace in traces() { + let expected = match trace.persisted_input_result.state { + PlaylistUpdateState::Success => InputUpdateCardStatus::Success, + PlaylistUpdateState::Partial => InputUpdateCardStatus::Partial, + PlaylistUpdateState::Failure => InputUpdateCardStatus::Failed, + }; + let sources = sources(); + let persisted = PlaylistUpdateStatusDto { + inputs: vec![InputPlaylistUpdateStatusDto { + input_id: 8, + last_update: Some(1), + last_input_update: Some(trace.persisted_input_result), + clusters: Vec::new(), + }], + ..PlaylistUpdateStatusDto::default() + }; + let cards = build_input_update_card_models(&sources, &persisted); + let connection = WebSocketConnectionContext::new(1, true); + let mut state = Rc::new(PlaylistUpdateCardStatuses::for_connection(connection)); + for progress in trace.progress { + state = state.reduce(PlaylistUpdateCardStatusAction::Progress { connection_context: connection, progress }); + } + let before_terminal = build_input_update_run_views(&sources, &cards, &state, &persisted).remove(0); + assert_eq!(before_terminal.input_state, expected); + assert_eq!(before_terminal.overall_state, InputUpdateCardStatus::Updating); + assert_eq!(before_terminal.input_id, 8); + let staged_telemetry = state.for_input(8).input_telemetry().cloned().unwrap(); + assert_eq!(staged_telemetry.source, None, "must not inherit the M3U parent's provider label"); + assert_eq!(staged_telemetry.clusters.len(), 3); + + assert_eq!(trace.completed.state, PlaylistUpdateState::Failure); + let run_id = trace.completed.run_id.clone(); + let order = trace.completed.execution_order; + state = state.reduce(PlaylistUpdateCardStatusAction::Completed { + connection_context: connection, + completed: trace.completed, + }); + let live = build_input_update_run_views(&sources, &cards, &state, &persisted).remove(0); + assert_eq!(live.input_state, expected); + assert_eq!(live.overall_state, InputUpdateCardStatus::Failed); + assert_eq!(live.run_id, run_id); + assert_eq!(live.execution_order, order); + assert_eq!(state.for_input(8).input_telemetry(), Some(&staged_telemetry)); + assert_eq!(live.cluster_results.len(), staged_telemetry.clusters.len()); + for (visible, recorded) in live.cluster_results.iter().zip(&staged_telemetry.clusters) { + assert_eq!(visible.cluster, recorded.cluster); + assert_eq!(visible.outcome, recorded.decision); + assert_eq!(visible.active_count, recorded.active_count); + } + + let reloaded = + state.reduce(PlaylistUpdateCardStatusAction::ConnectionChanged(WebSocketConnectionContext::new(2, true))); + let reload = build_input_update_run_views(&sources, &cards, &reloaded, &persisted).remove(0); + assert_eq!(reload.input_state, live.input_state); + assert_eq!( + reload.overall_state, + InputUpdateCardStatus::Ready, + "persisted input result proves no target/run result" + ); + assert_eq!(reload.run_id, None); + assert!(reload.progress_details.is_empty()); + } +} + +#[test] +fn playlist_update_status_pipeline_transparency_unknown_completion_still_uses_global_failure_fallback() { + let trace = + traces().into_iter().find(|trace| trace.persisted_input_result.state == PlaylistUpdateState::Partial).unwrap(); + let connection = WebSocketConnectionContext::new(1, true); + let mut state = Rc::new(PlaylistUpdateCardStatuses::for_connection(connection)); + let mut observed = false; + for progress in trace.progress.into_iter().filter(|event| event.input_id != Some(8) || event.state.is_none()) { + observed |= progress.input_id == Some(8); + state = state.reduce(PlaylistUpdateCardStatusAction::Progress { connection_context: connection, progress }); + } + assert!(observed, "the actual Rejection-progress observes the staged input without completing it"); + state = state.reduce(PlaylistUpdateCardStatusAction::Completed { + connection_context: connection, + completed: trace.completed, + }); + let sources = sources(); + let persisted = PlaylistUpdateStatusDto { + inputs: vec![InputPlaylistUpdateStatusDto { + input_id: 8, + last_update: Some(1), + last_input_update: Some(trace.persisted_input_result), + clusters: Vec::new(), + }], + ..PlaylistUpdateStatusDto::default() + }; + let cards = build_input_update_card_models(&sources, &persisted); + let live = build_input_update_run_views(&sources, &cards, &state, &persisted).remove(0); + assert_eq!( + live.input_state, + InputUpdateCardStatus::Failed, + "do not compensate missing runtime facts with persisted Partial" + ); + assert_eq!(live.overall_state, InputUpdateCardStatus::Failed); +} diff --git a/frontend/src/model/test_data/staged_completions.json b/frontend/src/model/test_data/staged_completions.json new file mode 100644 index 000000000..0eed12bcd --- /dev/null +++ b/frontend/src/model/test_data/staged_completions.json @@ -0,0 +1,370 @@ +[ + { + "progress": [ + { + "run_id": "staged-completion-run", + "execution_order": 1, + "input_id": 8, + "target": "staged", + "message": "Input 'staged' cluster 'live' rejected: current=100 candidate=1 threshold=95 quality=1; retaining previous cluster" + }, + { + "run_id": "staged-completion-run", + "execution_order": 1, + "input_id": 8, + "state": "Partial", + "input_telemetry": { + "refresh_policy": { + "cache": "respect", + "quality": "enforce" + }, + "clusters": [ + { + "cluster": "Live", + "requested": true, + "source": "provider", + "baseline_count": 100, + "candidate_count": 1, + "active_count": 100, + "threshold": 95, + "quality": 1, + "decision": "rejected" + }, + { + "cluster": "Video", + "requested": false + }, + { + "cluster": "Series", + "requested": false + } + ] + }, + "detail": "input_completed_partially", + "target": "staged", + "message": "Input 'staged' completed partially" + }, + { + "run_id": "staged-completion-run", + "execution_order": 1, + "input_id": 7, + "state": "Partial", + "input_telemetry": { + "refresh_policy": { + "cache": "respect", + "quality": "enforce" + }, + "clusters": [ + { + "cluster": "Live", + "requested": true + }, + { + "cluster": "Video", + "requested": true, + "source": "provider", + "candidate_count": 0, + "active_count": 0, + "decision": "accepted" + }, + { + "cluster": "Series", + "requested": true, + "source": "provider", + "candidate_count": 0, + "active_count": 0, + "decision": "accepted" + } + ] + }, + "detail": "input_completed_partially", + "target": "parent", + "message": "Input 'parent' completed partially" + }, + { + "run_id": "staged-completion-run", + "execution_order": 1, + "input_id": 9, + "state": "Failure", + "input_telemetry": { + "refresh_policy": { + "cache": "respect", + "quality": "enforce" + }, + "clusters": [ + { + "cluster": "Live", + "requested": true, + "source": "provider", + "decision": "technical_error" + }, + { + "cluster": "Video", + "requested": true, + "source": "provider", + "decision": "technical_error" + }, + { + "cluster": "Series", + "requested": true, + "source": "provider", + "decision": "technical_error" + } + ] + }, + "detail": "input_failed", + "target": "failed-companion", + "message": "Input 'failed-companion' failed during update" + } + ], + "completed": { + "run_id": "staged-completion-run", + "execution_order": 1, + "state": "Failure" + }, + "persisted_input_result": { + "state": "Partial", + "timestamp": 1 + } + }, + { + "progress": [ + { + "run_id": "staged-completion-run", + "execution_order": 1, + "input_id": 8, + "state": "Success", + "input_telemetry": { + "refresh_policy": { + "cache": "respect", + "quality": "enforce" + }, + "clusters": [ + { + "cluster": "Live", + "requested": true, + "source": "provider", + "baseline_count": 100, + "candidate_count": 100, + "active_count": 100, + "threshold": 95, + "quality": 100, + "decision": "accepted" + }, + { + "cluster": "Video", + "requested": false + }, + { + "cluster": "Series", + "requested": false + } + ] + }, + "detail": "input_completed_successfully", + "target": "staged", + "message": "Input 'staged' completed successfully" + }, + { + "run_id": "staged-completion-run", + "execution_order": 1, + "input_id": 7, + "state": "Success", + "input_telemetry": { + "refresh_policy": { + "cache": "respect", + "quality": "enforce" + }, + "clusters": [ + { + "cluster": "Live", + "requested": true + }, + { + "cluster": "Video", + "requested": true, + "source": "provider", + "candidate_count": 0, + "active_count": 0, + "decision": "accepted" + }, + { + "cluster": "Series", + "requested": true, + "source": "provider", + "candidate_count": 0, + "active_count": 0, + "decision": "accepted" + } + ] + }, + "detail": "input_completed_successfully", + "target": "parent", + "message": "Input 'parent' completed successfully" + }, + { + "run_id": "staged-completion-run", + "execution_order": 1, + "input_id": 9, + "state": "Failure", + "input_telemetry": { + "refresh_policy": { + "cache": "respect", + "quality": "enforce" + }, + "clusters": [ + { + "cluster": "Live", + "requested": true, + "source": "provider", + "decision": "technical_error" + }, + { + "cluster": "Video", + "requested": true, + "source": "provider", + "decision": "technical_error" + }, + { + "cluster": "Series", + "requested": true, + "source": "provider", + "decision": "technical_error" + } + ] + }, + "detail": "input_failed", + "target": "failed-companion", + "message": "Input 'failed-companion' failed during update" + } + ], + "completed": { + "run_id": "staged-completion-run", + "execution_order": 1, + "state": "Failure" + }, + "persisted_input_result": { + "state": "Success", + "timestamp": 1 + } + }, + { + "progress": [ + { + "run_id": "staged-completion-run", + "execution_order": 1, + "input_id": 8, + "state": "Failure", + "input_telemetry": { + "refresh_policy": { + "cache": "respect", + "quality": "enforce" + }, + "clusters": [ + { + "cluster": "Live", + "requested": true, + "source": "provider", + "threshold": 95, + "decision": "technical_error", + "technical_state": "failed" + }, + { + "cluster": "Video", + "requested": false + }, + { + "cluster": "Series", + "requested": false + } + ] + }, + "detail": "input_failed", + "target": "staged", + "message": "Input 'staged' failed during update" + }, + { + "run_id": "staged-completion-run", + "execution_order": 1, + "input_id": 7, + "state": "Failure", + "input_telemetry": { + "refresh_policy": { + "cache": "respect", + "quality": "enforce" + }, + "clusters": [ + { + "cluster": "Live", + "requested": true, + "source": "provider", + "candidate_count": 1, + "decision": "accepted" + }, + { + "cluster": "Video", + "requested": true, + "source": "provider", + "candidate_count": 0, + "decision": "accepted" + }, + { + "cluster": "Series", + "requested": true, + "source": "provider", + "candidate_count": 0, + "decision": "accepted" + } + ] + }, + "detail": "input_failed", + "target": "parent", + "message": "Input 'parent' failed during update" + }, + { + "run_id": "staged-completion-run", + "execution_order": 1, + "input_id": 9, + "state": "Failure", + "input_telemetry": { + "refresh_policy": { + "cache": "respect", + "quality": "enforce" + }, + "clusters": [ + { + "cluster": "Live", + "requested": true, + "source": "provider", + "decision": "technical_error" + }, + { + "cluster": "Video", + "requested": true, + "source": "provider", + "decision": "technical_error" + }, + { + "cluster": "Series", + "requested": true, + "source": "provider", + "decision": "technical_error" + } + ] + }, + "detail": "input_failed", + "target": "failed-companion", + "message": "Input 'failed-companion' failed during update" + } + ], + "completed": { + "run_id": "staged-completion-run", + "execution_order": 1, + "state": "Failure" + }, + "persisted_input_result": { + "state": "Failure", + "timestamp": 1 + } + } +] diff --git a/frontend/src/services/config_service.rs b/frontend/src/services/config_service.rs index 3f19c1506..3e010aea4 100644 --- a/frontend/src/services/config_service.rs +++ b/frontend/src/services/config_service.rs @@ -11,9 +11,8 @@ use log::error; use shared::{ foundation::{get_filter, prepare_templates, MapperScript}, model::{ - permission::Permission, ApiProxyConfigDto, AppConfigDto, ConfigDto, ConfigInputDto, IpCheckDto, - LibraryScanRequest, OperationRunAccepted, PlansConfigDto, SourcesConfigDto, TargetOutputDto, XtreamLoginInfo, - XtreamLoginRequest, + permission::Permission, ApiProxyConfigDto, AppConfigDto, ConfigDto, ConfigInputDto, IpCheckDto, LibraryStatus, + PlansConfigDto, SourcesConfigDto, TargetOutputDto, XtreamLoginInfo, XtreamLoginRequest, }, utils::{ concat_path, concat_path_leading_slash, HEADER_CONFIG_API_PROXY_REVISION, HEADER_CONFIG_MAIN_REVISION, @@ -467,10 +466,8 @@ impl ConfigService { request_get::<()>(&self.geoip_path, None, None).await } - pub async fn update_library(&self, force_rescan: bool) -> Result<(), Error> { - let path = concat_path(&self.library_path, "scan"); - let params = LibraryScanRequest { force_rescan }; - request_post::(&path, params, None, None).await.map(|_| ()) + pub async fn get_library_status(&self) -> Result, Error> { + request_get(&concat_path(&self.library_path, "status"), None, None).await } pub async fn complete_setup(&self, payload: SetupCompleteRequestDto) -> Result<(), Error> { diff --git a/frontend/src/services/playlist_service.rs b/frontend/src/services/playlist_service.rs index c873da9f6..2ca439fbe 100644 --- a/frontend/src/services/playlist_service.rs +++ b/frontend/src/services/playlist_service.rs @@ -1,12 +1,16 @@ -use crate::services::{get_base_href, request_post, Encoding}; +use crate::{ + error::Error, + services::{get_base_href, request_get, request_post, Encoding}, +}; use futures::join; use indexmap::IndexMap; use log::error; use shared::{ model::{ - EpgChannel, EpgTv, OperationRunAccepted, PlaylistEpgRequest, PlaylistRequest, PlaylistUrlResolveRequest, - SeriesStreamProperties, StreamEpgItemRequest, StreamEpgRequest, StreamEpgResponse, UiPlaylistCategories, - UiPlaylistGroup, UiPlaylistItem, XtreamCluster, XtreamSeriesInfoDoc, + EpgChannel, EpgTv, InputRefreshOverride, OperationRunAccepted, PlaylistEpgRequest, PlaylistRequest, + PlaylistUpdateRequestDto, PlaylistUpdateStatusDto, PlaylistUrlResolveRequest, SeriesStreamProperties, + StreamEpgItemRequest, StreamEpgRequest, StreamEpgResponse, UiPlaylistCategories, UiPlaylistGroup, + UiPlaylistItem, XtreamCluster, XtreamSeriesInfoDoc, }, utils::concat_path_leading_slash, }; @@ -14,6 +18,7 @@ use std::rc::Rc; pub struct PlaylistService { target_update_api_path: String, + playlist_update_status_path: String, playlist_api_live_path: String, playlist_api_vod_path: String, playlist_api_series_path: String, @@ -34,6 +39,7 @@ impl PlaylistService { Self { target_update_api_path: api("update"), + playlist_update_status_path: api("update/status"), playlist_api_live_path: api("live"), playlist_api_vod_path: api("vod"), playlist_api_series_path: api("series"), @@ -45,7 +51,38 @@ impl PlaylistService { } } pub async fn update_targets(&self, targets: &[&str]) -> bool { - request_post::<&[&str], OperationRunAccepted>(&self.target_update_api_path, targets, None, None).await.is_ok() + let request = build_playlist_update_request(targets); + self.submit_update_request(&request).await.is_ok() + } + + pub async fn update_all_inputs(&self) -> Result { + self.submit_update_request(&build_playlist_update_bulk_request()).await + } + + pub async fn update_input( + &self, + target_ids: &[u16], + input_id: u16, + action: shared::model::InputUpdateAction, + ) -> Result { + let request = + build_manual_input_update_request(target_ids, shared::model::InputUpdateRequest { input_id, action }); + self.submit_update_request(&request).await + } + + pub async fn get_update_status(&self) -> Result, crate::error::Error> { + request_get(&self.playlist_update_status_path, None, None).await + } + + async fn submit_update_request(&self, request: &PlaylistUpdateRequestDto) -> Result { + request_post::<&PlaylistUpdateRequestDto, OperationRunAccepted>( + &self.target_update_api_path, + request, + None, + None, + ) + .await + .and_then(|response| response.ok_or(Error::RequestError)) } pub async fn get_playlist_categories( @@ -177,6 +214,45 @@ impl PlaylistService { } } +fn build_playlist_update_request(targets: &[&str]) -> PlaylistUpdateRequestDto { + PlaylistUpdateRequestDto { + targets: targets.iter().map(|target| (*target).to_string()).collect(), + target_ids: None, + input_refresh: None, + input_action: None, + } +} + +fn build_playlist_update_bulk_request() -> PlaylistUpdateRequestDto { PlaylistUpdateRequestDto::default() } + +fn build_manual_input_update_request( + target_ids: &[u16], + request: shared::model::InputUpdateRequest, +) -> PlaylistUpdateRequestDto { + match request.action { + shared::model::InputUpdateAction::Provider(policy) => { + build_input_playlist_update_request(target_ids, InputRefreshOverride { input_id: request.input_id, policy }) + } + shared::model::InputUpdateAction::Rescan => PlaylistUpdateRequestDto { + target_ids: Some(target_ids.to_vec()), + input_action: Some(request), + ..PlaylistUpdateRequestDto::default() + }, + } +} + +fn build_input_playlist_update_request( + target_ids: &[u16], + input_refresh: InputRefreshOverride, +) -> PlaylistUpdateRequestDto { + PlaylistUpdateRequestDto { + targets: Vec::new(), + target_ids: Some(target_ids.to_vec()), + input_refresh: Some(input_refresh), + input_action: None, + } +} + fn to_ui_playlist_groups(list: Vec, xtream_cluster: XtreamCluster) -> Vec> { let mut groups = IndexMap::new(); list.into_iter().for_each(|item| { @@ -191,3 +267,153 @@ fn to_ui_playlist_groups(list: Vec, xtream_cluster: XtreamCluste }); groups.into_iter().map(|(_, v)| Rc::new(v)).collect::>() } + +#[cfg(test)] +mod tests { + use super::{ + build_input_playlist_update_request, build_playlist_update_bulk_request, build_playlist_update_request, + }; + use crate::model::{InputUpdateCapabilities, InputUpdateCapabilitiesExt}; + use shared::model::{ConfigInputDto, InputRefreshOverride, InputRefreshPolicy, InputType}; + + #[test] + fn playlist_update_action_rescan_sends_selected_ids_without_a_provider_policy() { + let action = + shared::model::InputUpdateRequest { input_id: 2, action: shared::model::InputUpdateAction::Rescan }; + let request = super::build_manual_input_update_request(&[20, 21], action); + assert_eq!(request.target_ids, Some(vec![20, 21])); + assert_eq!(request.input_action, Some(action)); + assert_eq!(request.input_refresh, None); + assert!(request.targets.is_empty()); + assert_eq!(build_playlist_update_bulk_request().input_action, None); + } + + #[test] + fn playlist_update_view_action_capability_policies_reach_the_existing_id_scoped_request() { + for input_type in [ + InputType::Xtream, + InputType::Stalker, + InputType::M3u, + InputType::M3uBatch, + InputType::XtreamBatch, + InputType::StalkerBatch, + InputType::Staged, + InputType::Plex, + InputType::Jellyfin, + InputType::Emby, + ] { + let input = ConfigInputDto { id: 7, input_type, ..ConfigInputDto::default() }; + for &policy in InputUpdateCapabilities::for_input(&input).policies() { + let request = + build_input_playlist_update_request(&[30, 20], InputRefreshOverride { input_id: input.id, policy }); + assert!(request.targets.is_empty()); + assert_eq!(request.target_ids, Some(vec![30, 20])); + assert_eq!(request.input_refresh, Some(InputRefreshOverride { input_id: 7, policy })); + } + } + } + + #[test] + fn playlist_update_view_action_m3u_cache_refresh_contract_is_request_local() -> Result<(), serde_json::Error> { + let input = ConfigInputDto { id: 7, input_type: InputType::M3u, ..ConfigInputDto::default() }; + let capabilities = InputUpdateCapabilities::for_input(&input); + // Exercise the existing request contract consumed by the default input-cache path. + // A later normal request (including another input) must not inherit refresh or force overrides. + for (input_id, policy, cache, quality) in [ + (7, InputRefreshPolicy::NORMAL, "respect", "enforce"), + (7, InputRefreshPolicy::REFRESH, "bypass", "enforce"), + (7, InputRefreshPolicy::FORCE, "bypass", "bypass"), + (8, InputRefreshPolicy::NORMAL, "respect", "enforce"), + (7, InputRefreshPolicy::NORMAL, "respect", "enforce"), + ] { + assert!(capabilities.supports(policy)); + let request = build_input_playlist_update_request(&[30], InputRefreshOverride { input_id, policy }); + assert_eq!( + serde_json::to_value(request)?, + serde_json::json!({ + "targets": [], "target_ids": [30], + "input_refresh": { "input_id": input_id, "policy": { "cache": cache, "quality": quality } } + }) + ); + assert_eq!(policy.bypasses_cache(), cache == "bypass"); + assert_eq!(policy.bypasses_quality(), quality == "bypass"); + } + Ok(()) + } + + #[test] + fn playlist_update_bulk_request_is_independent_of_every_card_capability_and_selection() { + for input_type in [ + InputType::Xtream, + InputType::Stalker, + InputType::M3u, + InputType::M3uBatch, + InputType::XtreamBatch, + InputType::StalkerBatch, + InputType::Staged, + InputType::Library, + InputType::Plex, + InputType::Jellyfin, + InputType::Emby, + ] { + for enabled in [true, false] { + let input = ConfigInputDto { id: 7, input_type, enabled, ..ConfigInputDto::default() }; + for &policy in InputUpdateCapabilities::for_input(&input).policies() { + let card_request = + build_input_playlist_update_request(&[30], InputRefreshOverride { input_id: input.id, policy }); + assert!(card_request.input_refresh.is_some()); + assert_eq!(build_playlist_update_bulk_request(), Default::default()); + } + // Bulk remains available as the normal server-selected all-targets request, + // including when no single-input policy is offered by this capability. + assert_eq!(build_playlist_update_bulk_request(), Default::default()); + } + } + } + + #[test] + fn playlist_update_action_request_keeps_target_ids_input_id_and_typed_policy() { + let request = build_input_playlist_update_request( + &[30, 20], + InputRefreshOverride { input_id: 7, policy: InputRefreshPolicy::FORCE }, + ); + + assert!(request.targets.is_empty()); + assert_eq!(request.target_ids, Some(vec![30, 20])); + assert_eq!( + request.input_refresh, + Some(InputRefreshOverride { input_id: 7, policy: InputRefreshPolicy::FORCE }) + ); + } + + #[test] + fn playlist_update_action_same_name_targets_create_distinct_id_only_requests() { + for (selected_ids, expected_ids) in [(&[1][..], vec![1]), (&[4][..], vec![4]), (&[1, 4][..], vec![1, 4])] { + let request = build_input_playlist_update_request( + selected_ids, + InputRefreshOverride { input_id: 7, policy: InputRefreshPolicy::NORMAL }, + ); + + assert!(request.targets.is_empty()); + assert_eq!(request.target_ids, Some(expected_ids)); + } + } + + #[test] + fn playlist_update_action_legacy_bulk_request_keeps_target_names_without_ids() { + let request = build_playlist_update_request(&["target-30", "target-20"]); + + assert_eq!(request.targets, vec!["target-30", "target-20"]); + assert_eq!(request.target_ids, None); + assert_eq!(request.input_refresh, None); + } + + #[test] + fn playlist_update_bulk_request_uses_normal_policy_and_complete_server_target_selection() { + let request = build_playlist_update_bulk_request(); + + assert!(request.targets.is_empty()); + assert_eq!(request.target_ids, None); + assert_eq!(request.input_refresh, None); + } +} diff --git a/frontend/src/services/websocket_service.rs b/frontend/src/services/websocket_service.rs index d2fb0411f..364bafd37 100644 --- a/frontend/src/services/websocket_service.rs +++ b/frontend/src/services/websocket_service.rs @@ -23,6 +23,28 @@ const WS_RECONNECT_BASE_MS: u32 = 300; const WS_RECONNECT_MAX_MS: u32 = 2000; const WS_RECONNECT_MAX_ATTEMPTS: u16 = 20; +/// Frontend-local identity of one concrete WebSocket connection lifecycle. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct WebSocketConnectionContext { + socket_epoch: u64, + connected: bool, +} + +impl WebSocketConnectionContext { + #[must_use] + pub const fn new(socket_epoch: u64, connected: bool) -> Self { Self { socket_epoch, connected } } + + #[must_use] + pub const fn is_connected(self) -> bool { self.connected } + + #[must_use] + pub fn is_same_live_connection(self, other: Self) -> bool { self.connected && self == other } +} + +const fn next_socket_epoch(current: u64) -> u64 { current.saturating_add(1) } + +const fn socket_epoch_is_current(current: u64, candidate: u64) -> bool { current == candidate } + fn reconnect_delay(attempt: u16) -> u32 { if attempt < 6 { let d = WS_RECONNECT_BASE_MS * (u32::from(attempt) + 1u32); @@ -39,6 +61,7 @@ type JsOnOpenCallback = Option>; pub struct WebSocketService { connected: Rc>, + connection_epoch: Rc>, attempt_counter: Rc>, ws: Rc>>, pending_messages: Rc>>>, @@ -58,6 +81,7 @@ impl WebSocketService { let base_href = get_base_href(); Self { connected: Rc::new(Cell::new(false)), + connection_epoch: Rc::new(Cell::new(0)), attempt_counter: Rc::new(Cell::new(0)), ws: Rc::new(RefCell::new(None)), pending_messages: Rc::new(RefCell::new(VecDeque::new())), @@ -73,10 +97,21 @@ impl WebSocketService { pub fn is_connected(&self) -> bool { self.connected.get() } + #[must_use] + pub fn connection_context(&self) -> WebSocketConnectionContext { + WebSocketConnectionContext::new(self.connection_epoch.get(), self.connected.get()) + } + + #[must_use] + pub fn is_current_connection(&self, context: WebSocketConnectionContext) -> bool { + self.connection_context().is_same_live_connection(context) + } + /// Helper function to allow cloning the service into JS closures for reconnect fn clone_for_reconnect(&self) -> Self { Self { connected: self.connected.clone(), + connection_epoch: self.connection_epoch.clone(), attempt_counter: self.attempt_counter.clone(), ws: self.ws.clone(), pending_messages: self.pending_messages.clone(), @@ -102,6 +137,8 @@ impl WebSocketService { match WebSocket::new(&self.ws_path) { Err(err) => error!("Failed to open websocket connection: {err:?}"), Ok(socket) => { + let socket_epoch = next_socket_epoch(self.connection_epoch.get()); + self.connection_epoch.set(socket_epoch); socket.set_binary_type(web_sys::BinaryType::Arraybuffer); let ws_clone = self.ws.clone(); *ws_clone.borrow_mut() = Some(socket.clone()); @@ -113,9 +150,13 @@ impl WebSocketService { let event_service = self.event_service.clone(); let attempt_counter = self.attempt_counter.clone(); let connected = self.connected.clone(); + let connection_epoch = self.connection_epoch.clone(); let pending_messages = self.pending_messages.clone(); let onmessage_callback = Closure::::wrap(Box::new(move |event: MessageEvent| { + if !socket_epoch_is_current(connection_epoch.get(), socket_epoch) { + return; + } trace!("WebSocket received message: {event:?}"); if let Some(response) = handle_socket_protocol_msg( event, @@ -138,7 +179,11 @@ impl WebSocketService { let ws_onopen_ref = self.ws_onopen.clone(); let ws_open_clone = ws_clone.clone(); let connected_clone = self.connected.clone(); + let connection_epoch = self.connection_epoch.clone(); let onopen_callback = Closure::::wrap(Box::new(move |_event: Event| { + if !socket_epoch_is_current(connection_epoch.get(), socket_epoch) { + return; + } // on open starts the protocol handshake; application messages wait until authorization. trace!("WebSocket connection opened."); connected_clone.set(false); @@ -162,10 +207,14 @@ impl WebSocketService { let ws_onerror_ref = self.ws_onerror.clone(); let ws_close_rc = self.ws.clone(); let connected_clone = self.connected.clone(); + let connection_epoch = self.connection_epoch.clone(); let event_service_clone = Rc::clone(&self.event_service); let ws_service_reconnect_clone = Rc::clone(&ws_service_reconnect); let onclose_callback = Closure::::wrap(Box::new(move |e: CloseEvent| { + if !socket_epoch_is_current(connection_epoch.get(), socket_epoch) { + return; + } trace!( "WebSocket closed (Code {}, Reason: {}, Clean: {})", e.code(), @@ -203,10 +252,14 @@ impl WebSocketService { { let ws_onerror_ref = self.ws_onerror.clone(); let connected_clone = self.connected.clone(); + let connection_epoch = self.connection_epoch.clone(); let event_service_clone = Rc::clone(&self.event_service); //let ws_service_reconnect_clone = Rc::clone(&ws_service_reconnect); let onerror_callback = Closure::::wrap(Box::new(move |e: ErrorEvent| { + if !socket_epoch_is_current(connection_epoch.get(), socket_epoch) { + return; + } error!("WebSocket error: {e:?}"); connected_clone.set(false); event_service_clone.broadcast(EventMessage::WebSocketStatus(false)); @@ -441,3 +494,30 @@ fn flush_pending_messages(pending_messages: &Rc>>>, ws: *pending = remaining; } + +#[cfg(test)] +mod tests { + use super::{next_socket_epoch, socket_epoch_is_current, WebSocketConnectionContext}; + + #[test] + fn playlist_update_status_socket_epoch_advances_for_each_connection_and_rejects_stale_callbacks() { + let first = next_socket_epoch(0); + let second = next_socket_epoch(first); + + assert_eq!(first, 1); + assert_eq!(second, 2); + assert!(socket_epoch_is_current(second, second)); + assert!(!socket_epoch_is_current(second, first)); + } + + #[test] + fn playlist_update_status_connection_context_matches_only_the_same_live_socket() { + let first = WebSocketConnectionContext::new(1, true); + let first_disconnected = WebSocketConnectionContext::new(1, false); + let second = WebSocketConnectionContext::new(2, true); + + assert!(first.is_same_live_connection(first)); + assert!(!first_disconnected.is_same_live_connection(first_disconnected)); + assert!(!second.is_same_live_connection(first)); + } +} diff --git a/frontend/src/utils/content_cluster.rs b/frontend/src/utils/content_cluster.rs new file mode 100644 index 000000000..a66d3675f --- /dev/null +++ b/frontend/src/utils/content_cluster.rs @@ -0,0 +1,16 @@ +use shared::model::XtreamCluster; + +/// Update UI terminology and order only; never a wire or processing order. +pub(crate) struct ContentClusterPresentation { + pub label_key: &'static str, + pub rank: u8, +} + +pub(crate) const fn content_cluster_presentation(cluster: XtreamCluster) -> ContentClusterPresentation { + let (label_key, rank) = match cluster { + XtreamCluster::Live => ("MESSAGES.PLAYLIST_UPDATE.CONTENT_LIVE", 0), + XtreamCluster::Series => ("MESSAGES.PLAYLIST_UPDATE.CONTENT_SERIES", 1), + XtreamCluster::Video => ("MESSAGES.PLAYLIST_UPDATE.CONTENT_MOVIES", 2), + }; + ContentClusterPresentation { label_key, rank } +} diff --git a/frontend/src/utils/mod.rs b/frontend/src/utils/mod.rs index ec68fef76..f9ec99f4f 100644 --- a/frontend/src/utils/mod.rs +++ b/frontend/src/utils/mod.rs @@ -1,7 +1,9 @@ +mod content_cluster; mod format; mod storage; use crate::i18n::YewI18n; +pub(crate) use content_cluster::content_cluster_presentation; pub use format::*; use shared::model::{PlaylistItemType, StreamInfo}; pub use storage::*; diff --git a/shared/src/defaults/primitives.rs b/shared/src/defaults/primitives.rs index 806fa16c6..f5096aaf4 100644 --- a/shared/src/defaults/primitives.rs +++ b/shared/src/defaults/primitives.rs @@ -2,6 +2,7 @@ use crate::model::{ClusterFlags, RuntimeConfigReportFormat}; +pub const fn is_zero_u8(v: &u8) -> bool { *v == 0 } pub const fn is_zero_u16(v: &u16) -> bool { *v == 0 } pub const fn is_zero_i16(v: &i16) -> bool { *v == 0 } pub const fn is_zero_u32(v: &u32) -> bool { *v == 0 } diff --git a/shared/src/model/config/input.rs b/shared/src/model/config/input.rs index 1dc19d341..1eb3ae0c9 100644 --- a/shared/src/model/config/input.rs +++ b/shared/src/model/config/input.rs @@ -9,14 +9,14 @@ use crate::{ error::TuliproxError, foundation::{get_filter, Filter}, model::{ - config::media_server_catalog::MediaServerInputConfigDto, ClusterFlags, EpgConfigDto, PatternTemplate, Prepare, - StalkerAuthMode, StalkerInputConfigDto, + config::media_server_catalog::MediaServerInputConfigDto, ClusterFlags, ConfigInputUpdateQualityDto, + EpgConfigDto, PatternTemplate, Prepare, StalkerAuthMode, StalkerInputConfigDto, }, utils::{ arc_str_option_serde, arc_str_serde, arc_str_vec_serde, deserialize_timestamp, get_credentials_from_url_str, get_trimmed_string, is_blank_optional_arc_str, is_blank_optional_string, is_non_blank_optional_string, - parse_duration_seconds, parse_provider_scheme_url_parts, sanitize_sensitive_info, - serialize_option_vec_flow_map_items, trim_last_slash, Internable, BATCH_SCHEME_PREFIX, PROVIDER_SCHEME_PREFIX, + parse_duration_seconds, parse_provider_scheme_url_parts, sanitize_sensitive_info, trim_last_slash, Internable, + BATCH_SCHEME_PREFIX, PROVIDER_SCHEME_PREFIX, }, }; use log::warn; @@ -275,6 +275,8 @@ pub struct ConfigInputOptionsDto { pub skip_vod: bool, #[serde(default, alias = "xtream_skip_series", alias = "stalker_skip_series", skip_serializing_if = "is_false")] pub skip_series: bool, + #[serde(default, skip_serializing_if = "ConfigInputUpdateQualityDto::is_disabled")] + pub update_quality: ConfigInputUpdateQualityDto, #[serde(default = "default_xtream_live_stream_use_prefix", skip_serializing_if = "is_true")] pub xtream_live_stream_use_prefix: bool, #[serde(default, skip_serializing_if = "is_false")] @@ -326,6 +328,7 @@ impl Default for ConfigInputOptionsDto { skip_live: false, skip_vod: false, skip_series: false, + update_quality: ConfigInputUpdateQualityDto::default(), xtream_live_stream_use_prefix: default_xtream_live_stream_use_prefix(), xtream_live_stream_without_extension: false, disable_hls_streaming: false, @@ -354,6 +357,7 @@ impl ConfigInputOptionsDto { !self.skip_live && !self.skip_vod && !self.skip_series + && self.update_quality.is_empty() && self.xtream_live_stream_use_prefix && !self.xtream_live_stream_without_extension && !self.disable_hls_streaming @@ -377,6 +381,7 @@ impl ConfigInputOptionsDto { self.skip_live = false; self.skip_vod = false; self.skip_series = false; + self.update_quality.clean(); self.xtream_live_stream_use_prefix = default_as_true(); self.xtream_live_stream_without_extension = false; self.disable_hls_streaming = false; @@ -403,6 +408,7 @@ impl Prepare for ConfigInputOptionsDto { type Ctx<'a> = Option<&'a [PatternTemplate]>; fn prepare(&mut self, templates: Self::Ctx<'_>) -> Result<(), TuliproxError> { + self.update_quality.prepare(())?; if let Some(raw_filter) = &self.resolve_filter { self.t_resolve_filter = Some(get_filter(raw_filter, templates)?); } @@ -496,7 +502,7 @@ pub struct ConfigInputDto { pub cache_duration: Option, #[serde(skip)] pub cache_duration_seconds: u64, - #[serde(default, skip_serializing_if = "Option::is_none", serialize_with = "serialize_option_vec_flow_map_items")] + #[serde(default, skip_serializing_if = "Option::is_none")] pub aliases: Option>, #[serde(default, skip_serializing_if = "is_zero_i16")] pub priority: i16, @@ -1350,6 +1356,44 @@ mod tests { dto.prepare(0, false, &HashSet::new(), None) } + #[test] + fn config_input_alias_serialization_uses_block_default_and_reads_both_styles() { + for aliases in [ + "- {name: alias-one, url: provider://example, username: alias, password: 'pass # with: {}, commas', max_connections: 1, exp_date: 1788779145}\n", + "- name: alias-one\n url: provider://example\n username: alias\n password: 'pass # with: {}, commas'\n max_connections: 1\n exp_date: 1788779145\n", + ] { + let yaml = format!( + "name: provider\ntype: xtream\nurl: provider://example\nusername: root\npassword: pass\noptions:\n update_quality:\n live: 85\n vod: 85\n series: 85\n resolve_background: false\ncache_duration: 20h\nmax_connections: 1\nexp_date: 1788779143\naliases:\n{aliases}panel_api:\n url: http://panel.example\n api_key: synthetic-key\n" + ); + let input: ConfigInputDto = serde_saphyr::from_str(&yaml).expect("both alias styles parse"); + let serialized = serde_saphyr::to_string(&input).expect("serialize input"); + assert!(serialized.contains("- name: alias-one\n")); + assert!(!serialized.contains("- {")); + let reparsed: ConfigInputDto = serde_saphyr::from_str(&serialized).expect("read serialized input"); + assert_eq!(reparsed, input); + let json = serde_json::to_string(&input).expect("JSON serialization"); + assert_eq!(serde_json::from_str::(&json).expect("JSON roundtrip"), input); + + let mut prepared = input; + prepared.prepare(0, false, &HashSet::from(["example".to_string()]), None).expect("prepare input"); + assert_eq!(prepared.cache_duration_seconds, 72_000); + let options = prepared.options.as_ref().expect("options"); + assert_eq!(options.update_quality, ConfigInputUpdateQualityDto { live: 85, vod: 85, series: 85 }); + assert!(!options.resolve_background); + assert_eq!(prepared.aliases.as_ref().expect("aliases").len(), 1); + assert!(prepared.panel_api.is_some()); + } + } + + #[test] + fn config_input_alias_serialization_preserves_empty_and_missing_lists() { + for aliases in [None, Some(Vec::new())] { + let input = ConfigInputDto { name: "provider".intern(), aliases, ..Default::default() }; + let serialized = serde_saphyr::to_string(&input).expect("serialize input"); + assert_eq!(serde_saphyr::from_str::(&serialized).expect("roundtrip"), input); + } + } + #[test] fn sequential_group_round_trips_yaml_and_json() { let yaml = "name: grouped\ntype: m3u\nurl: http://example.com/list.m3u\nsequential_group: 7\n"; @@ -1994,6 +2038,67 @@ mod tests { assert!(value.get("stalker_skip_vod").is_none()); } + #[test] + fn config_input_options_without_update_quality_remains_backward_compatible() -> Result<(), serde_json::Error> { + let options: ConfigInputOptionsDto = serde_json::from_str("{}")?; + let value = serde_json::to_value(&options)?; + + assert_eq!(options.update_quality, ConfigInputUpdateQualityDto::default()); + assert!(options.is_empty()); + assert!(value.get("update_quality").is_none()); + Ok(()) + } + + #[test] + fn config_input_options_update_quality_round_trips_and_keeps_options_non_empty() -> Result<(), serde_json::Error> { + let options: ConfigInputOptionsDto = serde_json::from_value(serde_json::json!({ + "update_quality": { + "live": 95, + "vod": 90, + "series": 100 + } + }))?; + + assert!(!options.is_empty()); + assert_eq!(options.update_quality.live, 95); + assert_eq!(options.update_quality.vod, 90); + assert_eq!(options.update_quality.series, 100); + assert_eq!( + serde_json::to_value(options)?, + serde_json::json!({ + "update_quality": { + "live": 95, + "vod": 90, + "series": 100 + } + }) + ); + Ok(()) + } + + #[test] + fn config_input_options_prepare_validates_update_quality() { + let mut options: ConfigInputOptionsDto = serde_saphyr::from_str("update_quality:\n live: 101\n") + .expect("manually edited update quality should deserialize before validation"); + + let error = options.prepare(None).expect_err("invalid update quality must be rejected"); + + assert!(error.to_string().contains("options.update_quality.live")); + } + + #[test] + fn config_input_options_clean_resets_update_quality() { + let mut options = ConfigInputOptionsDto { + update_quality: ConfigInputUpdateQualityDto { live: 95, vod: 90, series: 85 }, + ..ConfigInputOptionsDto::default() + }; + + options.clean(); + + assert_eq!(options.update_quality, ConfigInputUpdateQualityDto::default()); + assert!(options.is_empty()); + } + #[test] fn disable_hls_streaming_defaults_to_false() -> Result<(), serde_json::Error> { let options: ConfigInputOptionsDto = serde_json::from_str("{}")?; diff --git a/shared/src/model/config/input_update_quality.rs b/shared/src/model/config/input_update_quality.rs new file mode 100644 index 000000000..4e5f8f19a --- /dev/null +++ b/shared/src/model/config/input_update_quality.rs @@ -0,0 +1,102 @@ +use crate::{defaults::is_zero_u8, error::TuliproxError, model::Prepare}; + +/// Per-cluster minimum quality thresholds for accepting input updates. +/// +/// A value of zero disables the additional quality check for that cluster. +#[derive(Default, Debug, Clone, Copy, serde::Serialize, serde::Deserialize, PartialEq, Eq)] +#[serde(deny_unknown_fields)] +pub struct ConfigInputUpdateQualityDto { + #[serde(default, skip_serializing_if = "is_zero_u8")] + pub live: u8, + #[serde(default, skip_serializing_if = "is_zero_u8")] + pub vod: u8, + #[serde(default, skip_serializing_if = "is_zero_u8")] + pub series: u8, +} + +impl ConfigInputUpdateQualityDto { + #[must_use] + pub const fn is_disabled(&self) -> bool { self.live == 0 && self.vod == 0 && self.series == 0 } + + #[must_use] + pub const fn is_empty(&self) -> bool { self.is_disabled() } + + pub fn clean(&mut self) { *self = Self::default(); } +} + +impl Prepare for ConfigInputUpdateQualityDto { + type Ctx<'a> = (); + + fn prepare(&mut self, (): Self::Ctx<'_>) -> Result<(), TuliproxError> { + for (field, value) in [("live", self.live), ("vod", self.vod), ("series", self.series)] { + if value > 100 { + return Err(TuliproxError::ConfigInput(format!( + "options.update_quality.{field} must be in the range 0..=100" + ))); + } + } + Ok(()) + } +} + +#[cfg(test)] +mod tests { + use super::ConfigInputUpdateQualityDto; + use crate::model::Prepare; + + #[test] + fn defaults_are_disabled_and_serialize_as_an_empty_map() -> Result<(), serde_json::Error> { + let policy = ConfigInputUpdateQualityDto::default(); + + assert!(policy.is_disabled()); + assert!(policy.is_empty()); + assert_eq!(serde_json::to_value(policy)?, serde_json::json!({})); + Ok(()) + } + + #[test] + fn serde_round_trip_omits_only_zero_thresholds() -> Result<(), serde_json::Error> { + let policy = ConfigInputUpdateQualityDto { live: 95, vod: 0, series: 100 }; + let value = serde_json::to_value(policy)?; + + assert_eq!(value, serde_json::json!({ "live": 95, "series": 100 })); + assert_eq!(serde_json::from_value::(value)?, policy); + Ok(()) + } + + #[test] + fn deserialization_rejects_unknown_fields() { + let result = serde_json::from_value::(serde_json::json!({ "movie": 90 })); + + assert!(result.is_err()); + } + + #[test] + fn prepare_validates_every_threshold() { + let cases = [ + ("live", ConfigInputUpdateQualityDto { live: 101, ..ConfigInputUpdateQualityDto::default() }), + ("vod", ConfigInputUpdateQualityDto { vod: 101, ..ConfigInputUpdateQualityDto::default() }), + ("series", ConfigInputUpdateQualityDto { series: 101, ..ConfigInputUpdateQualityDto::default() }), + ]; + + for (field, mut policy) in cases { + let error = policy.prepare(()).expect_err("threshold above 100 must be rejected"); + assert!(error.to_string().contains(field), "unexpected error for {field}: {error}"); + } + + for threshold in [0, 100] { + let mut policy = ConfigInputUpdateQualityDto { live: threshold, vod: threshold, series: threshold }; + assert!(policy.prepare(()).is_ok(), "threshold {threshold} should be valid"); + } + } + + #[test] + fn clean_restores_disabled_defaults() { + let mut policy = ConfigInputUpdateQualityDto { live: 95, vod: 90, series: 85 }; + + policy.clean(); + + assert_eq!(policy, ConfigInputUpdateQualityDto::default()); + assert!(policy.is_empty()); + } +} diff --git a/shared/src/model/config/mod.rs b/shared/src/model/config/mod.rs index 61cfb386a..2df4d2f95 100644 --- a/shared/src/model/config/mod.rs +++ b/shared/src/model/config/mod.rs @@ -16,6 +16,7 @@ mod hls_manifest_recovery_burst_level; mod hls_segment_repair_mode; mod hls_strip_mode; mod input; +mod input_update_quality; mod ipcheck; mod log; pub mod macros; @@ -73,6 +74,7 @@ pub use hls_manifest_recovery_burst_level::*; pub use hls_segment_repair_mode::*; pub use hls_strip_mode::*; pub use input::*; +pub use input_update_quality::*; pub use ipcheck::*; pub use library::*; pub use log::*; diff --git a/shared/src/model/config/playlist_update_state.rs b/shared/src/model/config/playlist_update_state.rs index 5fa7ac19f..56224a004 100644 --- a/shared/src/model/config/playlist_update_state.rs +++ b/shared/src/model/config/playlist_update_state.rs @@ -1,6 +1,6 @@ use serde::{Deserialize, Serialize}; -#[derive(Debug, Clone, Copy, PartialEq, Serialize, Deserialize)] +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] pub enum PlaylistUpdateState { Success, Failure, diff --git a/shared/src/model/config/source.rs b/shared/src/model/config/source.rs index 634e6acba..d77e1a59a 100644 --- a/shared/src/model/config/source.rs +++ b/shared/src/model/config/source.rs @@ -286,6 +286,62 @@ mod tests { } } + #[test] + fn legacy_source_yaml_without_update_quality_round_trips_with_disabled_defaults() { + let yaml = r" +inputs: + - name: legacy-provider + type: xtream + url: https://provider.example + username: user + password: password + options: + skip_vod: true +sources: [] +"; + + let parsed: SourcesConfigDto = serde_saphyr::from_str(yaml).expect("legacy source config should parse"); + let options = parsed.inputs[0].options.as_ref().expect("legacy options should be present"); + assert!(options.update_quality.is_disabled()); + + let serialized = serde_saphyr::to_string(&parsed).expect("legacy source config should serialize"); + assert!(!serialized.contains("update_quality")); + let reparsed: SourcesConfigDto = + serde_saphyr::from_str(&serialized).expect("serialized legacy source config should parse"); + assert_eq!(reparsed, parsed); + } + + #[test] + fn source_yaml_round_trip_preserves_all_update_quality_thresholds() { + let yaml = r" +inputs: + - name: guarded-provider + type: stalker + url: https://provider.example + username: user + password: password + options: + update_quality: + live: 90 + vod: 95 + series: 100 +sources: [] +"; + + let parsed: SourcesConfigDto = serde_saphyr::from_str(yaml).expect("guarded source config should parse"); + let quality = parsed.inputs[0] + .options + .as_ref() + .map(|options| options.update_quality) + .expect("update quality should be present"); + assert_eq!((quality.live, quality.vod, quality.series), (90, 95, 100)); + + let serialized = serde_saphyr::to_string(&parsed).expect("guarded source config should serialize"); + let reparsed: SourcesConfigDto = + serde_saphyr::from_str(&serialized).expect("serialized guarded source config should parse"); + assert_eq!(reparsed, parsed); + } + #[test] fn duplicate_credentials_warning_identifies_same_url_account() { let previous = CredentialOwner { kind: "input", name: "primary", url: "provider://example" }; diff --git a/shared/src/model/event.rs b/shared/src/model/event.rs index 7644a7e0d..b7c7415ef 100644 --- a/shared/src/model/event.rs +++ b/shared/src/model/event.rs @@ -15,10 +15,11 @@ use crate::model::{ stats::SourceStats, ActiveUserConnectionChange, ConfigReloadFailure, ConfigType, ConnectionDenied, DiskAlert, DownloadsDelta, DownloadsResponse, LibraryScanProgressEvent, LibraryScanSummaryStatus, MetadataUpdateFailure, MsgKind, - NotificationDeadLetter, Permission, PlaylistGroupsChanged, PlaylistUpdateProgressEvent, PlaylistUpdateState, - ProviderAccountEvent, ProviderAccountState, ProviderFetchFailure, ProviderPoolExhausted, ProviderPriorityFallback, - RecordingLifecycleMessage, ScheduledTaskFailure, ServerLifecycleEvent, ServerLifecycleState, StreamProbeFailure, - SystemInfo, UserLifecycleEvent, UserLifecycleState, WatchChanges, WatchDisabled, WatchUnmatched, + NotificationDeadLetter, Permission, PlaylistGroupsChanged, PlaylistUpdateProgressEvent, PlaylistUpdateRunId, + PlaylistUpdateRunOrder, PlaylistUpdateState, ProviderAccountEvent, ProviderAccountState, ProviderFetchFailure, + ProviderPoolExhausted, ProviderPriorityFallback, RecordingLifecycleMessage, ScheduledTaskFailure, + ServerLifecycleEvent, ServerLifecycleState, StreamProbeFailure, SystemInfo, UserLifecycleEvent, UserLifecycleState, + WatchChanges, WatchDisabled, WatchUnmatched, }; use std::sync::Arc; @@ -694,6 +695,12 @@ impl EventMessage { /// `playlist.update` got a bare enum. #[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)] pub struct PlaylistUpdateSummary { + /// Stable identity of the processing run. Missing only on legacy/test-only events. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub run_id: Option, + /// Actual serial execution order. Missing only on legacy/test-only events. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub execution_order: Option, pub state: PlaylistUpdateState, /// Per-source statistics. Empty when the run produced none. #[serde(default, skip_serializing_if = "Vec::is_empty")] @@ -707,7 +714,18 @@ impl PlaylistUpdateSummary { /// A summary carrying only an outcome - the timeout and panic paths, /// which have no statistics to report. #[must_use] - pub fn state_only(state: PlaylistUpdateState) -> Self { Self { state, stats: Vec::new(), error: None } } + pub fn state_only(state: PlaylistUpdateState) -> Self { + Self { run_id: None, execution_order: None, state, stats: Vec::new(), error: None } + } + + #[must_use] + pub fn for_run( + run_id: PlaylistUpdateRunId, + execution_order: PlaylistUpdateRunOrder, + state: PlaylistUpdateState, + ) -> Self { + Self { run_id: Some(run_id), execution_order: Some(execution_order), state, stats: Vec::new(), error: None } + } } /// A set of [`EventKind`]s, as one word. diff --git a/shared/src/model/input_refresh_policy.rs b/shared/src/model/input_refresh_policy.rs new file mode 100644 index 000000000..40a71c9cf --- /dev/null +++ b/shared/src/model/input_refresh_policy.rs @@ -0,0 +1,197 @@ +use crate::model::{InputUpdateAction, InputUpdateRequest}; +use serde::{Deserialize, Serialize}; + +/// Whether a manual playlist update may reuse a valid input cache entry. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum CacheReadPolicy { + #[default] + Respect, + Bypass, +} + +/// Whether a manual playlist update applies the configured quality thresholds. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum UpdateQualityPolicy { + #[default] + Enforce, + Bypass, +} + +/// Request-local cache and quality behavior for one input update. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct InputRefreshPolicy { + pub cache: CacheReadPolicy, + pub quality: UpdateQualityPolicy, +} + +impl InputRefreshPolicy { + pub const NORMAL: Self = Self { cache: CacheReadPolicy::Respect, quality: UpdateQualityPolicy::Enforce }; + pub const REFRESH: Self = Self { cache: CacheReadPolicy::Bypass, quality: UpdateQualityPolicy::Enforce }; + pub const FORCE: Self = Self { cache: CacheReadPolicy::Bypass, quality: UpdateQualityPolicy::Bypass }; + + #[must_use] + pub const fn bypasses_cache(self) -> bool { matches!(self.cache, CacheReadPolicy::Bypass) } + + #[must_use] + pub const fn bypasses_quality(self) -> bool { matches!(self.quality, UpdateQualityPolicy::Bypass) } +} + +impl Default for InputRefreshPolicy { + fn default() -> Self { Self::NORMAL } +} + +/// Applies a refresh policy to one configured input without changing its persisted configuration. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct InputRefreshOverride { + pub input_id: u16, + pub policy: InputRefreshPolicy, +} + +/// Current request body for a manual playlist update. +#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct PlaylistUpdateRequestDto { + #[serde(default)] + pub targets: Vec, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub target_ids: Option>, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub input_refresh: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub input_action: Option, +} + +impl PlaylistUpdateRequestDto { + /// Resolves the legacy policy field without permitting two conflicting actions. + pub fn manual_input_update(&self) -> Result, &'static str> { + match (self.input_action, self.input_refresh) { + (Some(_), Some(_)) => Err("Specify either input_action or input_refresh, not both"), + (Some(action), None) => Ok(Some(action)), + (None, Some(refresh)) => Ok(Some(InputUpdateRequest { + input_id: refresh.input_id, + action: InputUpdateAction::Provider(refresh.policy), + })), + (None, None) => Ok(None), + } + } +} + +/// Accepted wire formats for the update endpoint. +/// +/// The legacy target-name array remains valid so existing API clients keep +/// their pre-policy behavior. +#[derive(Debug, Clone, PartialEq, Eq, Deserialize)] +#[serde(untagged)] +pub enum PlaylistUpdateRequestPayload { + Current(PlaylistUpdateRequestDto), + LegacyTargets(Vec), +} + +impl PlaylistUpdateRequestPayload { + #[must_use] + pub fn into_request(self) -> PlaylistUpdateRequestDto { + match self { + Self::Current(request) => request, + Self::LegacyTargets(targets) => PlaylistUpdateRequestDto { targets, ..PlaylistUpdateRequestDto::default() }, + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn named_policies_have_the_required_cache_and_quality_semantics() { + assert_eq!(InputRefreshPolicy::NORMAL.cache, CacheReadPolicy::Respect); + assert_eq!(InputRefreshPolicy::NORMAL.quality, UpdateQualityPolicy::Enforce); + assert_eq!(InputRefreshPolicy::REFRESH.cache, CacheReadPolicy::Bypass); + assert_eq!(InputRefreshPolicy::REFRESH.quality, UpdateQualityPolicy::Enforce); + assert_eq!(InputRefreshPolicy::FORCE.cache, CacheReadPolicy::Bypass); + assert_eq!(InputRefreshPolicy::FORCE.quality, UpdateQualityPolicy::Bypass); + } + + #[test] + fn playlist_update_request_with_target_ids_round_trips_without_mutating_policy() -> Result<(), serde_json::Error> { + let request = PlaylistUpdateRequestDto { + targets: Vec::new(), + target_ids: Some(vec![4, 1]), + input_refresh: Some(InputRefreshOverride { input_id: 17, policy: InputRefreshPolicy::FORCE }), + input_action: None, + }; + + let encoded = serde_json::to_string(&request)?; + let decoded = serde_json::from_str::(&encoded)?; + + assert_eq!(decoded, request); + assert_eq!( + serde_json::from_str::(&encoded)?, + serde_json::json!({ + "targets": [], + "target_ids": [4, 1], + "input_refresh": { + "input_id": 17, + "policy": {"cache": "bypass", "quality": "bypass"} + } + }) + ); + Ok(()) + } + + #[test] + fn playlist_update_request_distinguishes_missing_and_explicitly_empty_target_ids() -> Result<(), serde_json::Error> + { + let missing = serde_json::from_str::(r#"{"targets":[]}"#)?; + let explicit_empty = serde_json::from_str::(r#"{"targets":[],"target_ids":[]}"#)?; + + assert_eq!(missing.target_ids, None); + assert_eq!(explicit_empty.target_ids, Some(Vec::new())); + Ok(()) + } + + #[test] + fn playlist_update_request_named_targets_remain_backward_compatible() -> Result<(), serde_json::Error> { + let request = serde_json::from_str::(r#"{"targets":["family"]}"#)?; + + assert_eq!(request.targets, vec!["family"]); + assert_eq!(request.target_ids, None); + assert_eq!(request.input_refresh, None); + Ok(()) + } + + #[test] + fn playlist_update_request_legacy_target_array_maps_to_normal_behavior() -> Result<(), serde_json::Error> { + let payload = serde_json::from_str::(r#"["family","mobile"]"#)?; + + assert_eq!( + payload.into_request(), + PlaylistUpdateRequestDto { + targets: vec!["family".to_string(), "mobile".to_string()], + target_ids: None, + input_refresh: None, + input_action: None, + } + ); + Ok(()) + } + + #[test] + fn playlist_update_request_rescan_round_trips_and_rejects_competing_policy() -> Result<(), serde_json::Error> { + let mut request = PlaylistUpdateRequestDto { + target_ids: Some(vec![20, 21]), + input_action: Some(InputUpdateRequest { input_id: 2, action: InputUpdateAction::Rescan }), + ..PlaylistUpdateRequestDto::default() + }; + let encoded = serde_json::to_string(&request)?; + assert_eq!(serde_json::from_str::(&encoded)?, request); + assert_eq!(request.manual_input_update(), Ok(request.input_action)); + request.input_refresh = Some(InputRefreshOverride { input_id: 2, policy: InputRefreshPolicy::FORCE }); + assert!(request.manual_input_update().is_err()); + assert_eq!(PlaylistUpdateRequestDto::default().manual_input_update(), Ok(None)); + Ok(()) + } +} diff --git a/shared/src/model/input_update_action.rs b/shared/src/model/input_update_action.rs new file mode 100644 index 000000000..ec10f4900 --- /dev/null +++ b/shared/src/model/input_update_action.rs @@ -0,0 +1,100 @@ +use crate::model::{InputRefreshPolicy, InputType}; +use serde::{Deserialize, Serialize}; + +/// One request-local acquisition before the common target pipeline. +/// Provider actions keep the existing policy domain; Rescan is local discovery. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(tag = "kind", content = "policy", rename_all = "snake_case", deny_unknown_fields)] +pub enum InputUpdateAction { + Provider(InputRefreshPolicy), + Rescan, +} + +/// Stable input identity and its manual action. Target selection stays on the update request. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct InputUpdateRequest { + pub input_id: u16, + pub action: InputUpdateAction, +} + +/// The single capability contract shared by cards and backend request validation. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum InputUpdateCapabilities { + Manual { policies: &'static [InputRefreshPolicy] }, + Rescan, + BulkOnly, + ImportUnavailable, + Disabled, +} + +impl InputUpdateCapabilities { + #[must_use] + pub const fn for_input_type(input_type: InputType, enabled: bool) -> Self { + if !enabled { + return Self::Disabled; + } + match input_type { + InputType::Xtream | InputType::Stalker | InputType::M3u => Self::Manual { + policies: &[InputRefreshPolicy::NORMAL, InputRefreshPolicy::REFRESH, InputRefreshPolicy::FORCE], + }, + InputType::Plex => Self::Manual { policies: &[InputRefreshPolicy::NORMAL, InputRefreshPolicy::REFRESH] }, + InputType::Library => Self::Rescan, + InputType::Jellyfin | InputType::Emby => Self::ImportUnavailable, + InputType::M3uBatch | InputType::XtreamBatch | InputType::StalkerBatch | InputType::Staged => { + Self::BulkOnly + } + } + } + + #[must_use] + pub const fn policies(self) -> &'static [InputRefreshPolicy] { + match self { + Self::Manual { policies } => policies, + Self::Rescan | Self::BulkOnly | Self::ImportUnavailable | Self::Disabled => &[], + } + } + + #[must_use] + pub fn supports(self, policy: InputRefreshPolicy) -> bool { self.policies().contains(&policy) } + + #[must_use] + pub fn supports_action(self, action: InputUpdateAction) -> bool { + match action { + InputUpdateAction::Provider(policy) => self.supports(policy), + InputUpdateAction::Rescan => self == Self::Rescan, + } + } + + #[must_use] + pub const fn rebuilds_targets(self) -> bool { matches!(self, Self::Manual { .. } | Self::Rescan) } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn playlist_update_request_capabilities_cover_all_types_and_never_invent_force() { + let cases = [ + (InputType::Xtream, 3), + (InputType::Stalker, 3), + (InputType::M3u, 3), + (InputType::Plex, 2), + (InputType::Library, 0), + (InputType::Jellyfin, 0), + (InputType::Emby, 0), + (InputType::M3uBatch, 0), + (InputType::XtreamBatch, 0), + (InputType::StalkerBatch, 0), + (InputType::Staged, 0), + ]; + for (input_type, count) in cases { + let capabilities = InputUpdateCapabilities::for_input_type(input_type, true); + assert_eq!(capabilities.policies().len(), count, "{input_type}"); + assert_eq!(capabilities.supports(InputRefreshPolicy::FORCE), count == 3); + assert_eq!(capabilities.supports_action(InputUpdateAction::Rescan), input_type == InputType::Library); + assert_eq!(InputUpdateCapabilities::for_input_type(input_type, false), InputUpdateCapabilities::Disabled); + } + } +} diff --git a/shared/src/model/library_request.rs b/shared/src/model/library_request.rs index 5e1d2e943..bb3eea243 100644 --- a/shared/src/model/library_request.rs +++ b/shared/src/model/library_request.rs @@ -9,7 +9,7 @@ pub struct LibraryScanRequest { } // Scan result with statistics -#[derive(Debug, Clone, Deserialize, Serialize, PartialEq)] +#[derive(Debug, Clone, Deserialize, Serialize, PartialEq, Eq)] pub struct LibraryScanResult { pub files_scanned: usize, pub groups_scanned: usize, @@ -33,11 +33,14 @@ pub struct LibraryScanSummary { pub result: Option, } -#[derive(Default, Debug, Clone, Deserialize, Serialize, PartialEq)] +#[derive(Default, Debug, Clone, Deserialize, Serialize, PartialEq, Eq)] pub struct LibraryStatus { pub enabled: bool, pub total_items: usize, pub movies: usize, pub series: usize, + /// Stored episode records; `total_items` still counts movie/series catalog entries. + #[serde(default)] + pub episodes: usize, pub path: Option, } diff --git a/shared/src/model/mod.rs b/shared/src/model/mod.rs index 7cdc79b43..3f8bdbe73 100644 --- a/shared/src/model/mod.rs +++ b/shared/src/model/mod.rs @@ -12,6 +12,8 @@ mod event; mod identity_registry; mod ids; mod info_doc_utils; +mod input_refresh_policy; +mod input_update_action; mod ip_check; mod item_field; mod library_request; @@ -29,6 +31,8 @@ mod playlist_document; mod playlist_groups_changed; mod playlist_info_document; mod playlist_request; +mod playlist_update_run; +mod playlist_update_status; mod prepare; mod processing_order; mod progress; @@ -72,10 +76,11 @@ pub mod xtream_const; pub use self::{ active_user_connection_change::*, auth::*, auth_audit::*, cluster_flags::*, config::*, connection_denied::*, - custom_video_stream_type::*, download::*, epg::*, epg_request::*, event::*, identity_registry::*, ip_check::*, - item_field::*, library_request::*, log::*, mapping::*, media_properties::*, messaging::*, - metadata_update_failure::*, notification::*, notification_dead_letter::*, pagination::*, playlist::*, - playlist_categories::*, playlist_groups_changed::*, playlist_info_document::*, playlist_request::*, + custom_video_stream_type::*, download::*, epg::*, epg_request::*, event::*, identity_registry::*, + input_refresh_policy::*, input_update_action::*, ip_check::*, item_field::*, library_request::*, log::*, + mapping::*, media_properties::*, messaging::*, metadata_update_failure::*, notification::*, + notification_dead_letter::*, pagination::*, playlist::*, playlist_categories::*, playlist_groups_changed::*, + playlist_info_document::*, playlist_request::*, playlist_update_run::*, playlist_update_status::*, processing_order::*, progress::*, provider_fetch_failure::*, provider_pool::*, recording::*, recording_math::*, regex_cache::*, scheduled_task_failure::*, search_fields::*, search_request::*, server_lifecycle::*, short_epg::*, stalker::*, stalker_item::*, stats::*, status_check::*, stream_history::*, stream_history_record::*, diff --git a/shared/src/model/playlist_update_run.rs b/shared/src/model/playlist_update_run.rs new file mode 100644 index 000000000..d5a400a28 --- /dev/null +++ b/shared/src/model/playlist_update_run.rs @@ -0,0 +1,146 @@ +use crate::model::PlaylistUpdateState; +use serde::{Deserialize, Deserializer, Serialize}; +use std::{fmt, sync::Arc}; + +/// Stable identity shared by every event and request belonging to one playlist update run. +#[derive(Clone, Debug, Eq, Hash, PartialEq, Serialize, Deserialize)] +#[serde(transparent)] +pub struct PlaylistUpdateRunId(Arc); + +impl PlaylistUpdateRunId { + /// Generates a process-independent 128-bit run identity. + #[must_use] + pub fn generate() -> Self { + let high = fastrand::u64(..); + let low = fastrand::u64(..); + Self(format!("{high:016x}{low:016x}").into()) + } +} + +impl AsRef for PlaylistUpdateRunId { + fn as_ref(&self) -> &str { &self.0 } +} + +impl fmt::Display for PlaylistUpdateRunId { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { formatter.write_str(&self.0) } +} + +impl From<&str> for PlaylistUpdateRunId { + fn from(value: &str) -> Self { Self(value.into()) } +} + +impl From for PlaylistUpdateRunId { + fn from(value: String) -> Self { Self(value.into()) } +} + +/// Monotonic order assigned when a playlist update enters actual execution. +/// +/// This is deliberately separate from [`PlaylistUpdateRunId`]: request +/// acceptance order is not execution order for the bounded serial queue. +#[derive(Clone, Copy, Debug, Eq, Hash, Ord, PartialEq, PartialOrd, Serialize, Deserialize)] +#[serde(transparent)] +pub struct PlaylistUpdateRunOrder(u64); + +impl PlaylistUpdateRunOrder { + #[must_use] + pub const fn get(self) -> u64 { self.0 } +} + +impl From for PlaylistUpdateRunOrder { + fn from(value: u64) -> Self { Self(value) } +} + +/// Terminal state of one playlist update run. +/// +/// Deserialization also accepts the legacy bare [`PlaylistUpdateState`] payload. +#[derive(Clone, Debug, Eq, PartialEq, Serialize)] +pub struct PlaylistUpdateRunStateEvent { + #[serde(default, skip_serializing_if = "Option::is_none")] + pub run_id: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub execution_order: Option, + pub state: PlaylistUpdateState, +} + +impl PlaylistUpdateRunStateEvent { + #[must_use] + pub fn correlated( + run_id: PlaylistUpdateRunId, + execution_order: PlaylistUpdateRunOrder, + state: PlaylistUpdateState, + ) -> Self { + Self { run_id: Some(run_id), execution_order: Some(execution_order), state } + } + + #[must_use] + pub const fn uncorrelated(state: PlaylistUpdateState) -> Self { + Self { run_id: None, execution_order: None, state } + } +} + +impl<'de> Deserialize<'de> for PlaylistUpdateRunStateEvent { + fn deserialize(deserializer: D) -> Result + where + D: Deserializer<'de>, + { + #[derive(Deserialize)] + #[serde(untagged)] + enum CompatiblePayload { + Legacy(PlaylistUpdateState), + Correlated { + #[serde(default)] + run_id: Option, + #[serde(default)] + execution_order: Option, + state: PlaylistUpdateState, + }, + } + + Ok(match CompatiblePayload::deserialize(deserializer)? { + CompatiblePayload::Legacy(state) => Self::uncorrelated(state), + CompatiblePayload::Correlated { run_id, execution_order, state } => Self { run_id, execution_order, state }, + }) + } +} + +#[cfg(test)] +mod tests { + use super::{PlaylistUpdateRunId, PlaylistUpdateRunOrder, PlaylistUpdateRunStateEvent}; + use crate::model::PlaylistUpdateState; + + #[test] + fn playlist_update_progress_terminal_event_accepts_legacy_state_payload() -> Result<(), serde_json::Error> { + let event = serde_json::from_str::(r#""Success""#)?; + + assert_eq!(event, PlaylistUpdateRunStateEvent::uncorrelated(PlaylistUpdateState::Success)); + Ok(()) + } + + #[test] + fn playlist_update_progress_terminal_event_roundtrips_run_identity() -> Result<(), serde_json::Error> { + let event = PlaylistUpdateRunStateEvent::correlated( + "run-17".into(), + PlaylistUpdateRunOrder::from(17), + PlaylistUpdateState::Partial, + ); + + let encoded = serde_json::to_string(&event)?; + let decoded = serde_json::from_str::(&encoded)?; + + assert_eq!(decoded, event); + assert_eq!(decoded.run_id.as_ref().map(PlaylistUpdateRunId::as_ref), Some("run-17")); + assert_eq!(decoded.execution_order.map(PlaylistUpdateRunOrder::get), Some(17)); + Ok(()) + } + + #[test] + fn playlist_update_progress_terminal_event_accepts_correlated_payload_without_execution_order( + ) -> Result<(), serde_json::Error> { + let event = serde_json::from_str::(r#"{"run_id":"run-12-1","state":"Success"}"#)?; + + assert_eq!(event.run_id.as_ref().map(PlaylistUpdateRunId::as_ref), Some("run-12-1")); + assert_eq!(event.execution_order, None); + assert_eq!(event.state, PlaylistUpdateState::Success); + Ok(()) + } +} diff --git a/shared/src/model/playlist_update_status.rs b/shared/src/model/playlist_update_status.rs new file mode 100644 index 000000000..a1c5583a5 --- /dev/null +++ b/shared/src/model/playlist_update_status.rs @@ -0,0 +1,231 @@ +use crate::model::{ + InputRefreshPolicy, PlaylistUpdateDataSource, PlaylistUpdateProgressEvent, PlaylistUpdateState, XtreamCluster, +}; +use serde::{Deserialize, Serialize}; + +/// Persisted outcome exposed by the canonical input-cache status read path. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum PersistedPlaylistUpdateClusterState { + Ok, + Failed, +} + +/// Persisted result of an already completed Quality evaluation. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum PersistedPlaylistUpdateQualityDecision { + Accepted, + Rejected, +} + +/// Persisted Quality facts copied from the typed runtime decision. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct PersistedPlaylistUpdateQualitySnapshot { + pub threshold: u8, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub baseline_count: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub candidate_count: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub achieved_quality: Option, + pub decision: PersistedPlaylistUpdateQualityDecision, +} + +/// Technical completion of the last cluster update, independent of Quality. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum PersistedPlaylistUpdateTechnicalState { + Succeeded, + Failed, +} + +/// Exactly one last, non-historical runtime snapshot for a persisted cluster. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct PersistedPlaylistUpdateClusterSnapshot { + /// Present only when a request-local policy was typed for this input. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub policy: Option, + /// Actual data source; never reconstructed from `policy`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub source: Option, + /// Guard configured for this run: zero means disabled, a positive value is active. + /// Missing only for legacy or deliberately neutralized snapshots. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub quality_guard_threshold: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub quality: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub active_count: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub technical_state: Option, +} + +/// Read projection of one persisted Xtream-compatible cluster status. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct PersistedPlaylistUpdateClusterStatusDto { + pub cluster: XtreamCluster, + pub status: PersistedPlaylistUpdateClusterState, + pub timestamp: u64, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub last_update: Option, +} + +/// Last completed input job, not the result of the subsequent target publication. +/// Replaced on completion; independent of cache validity and cluster snapshots. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct PersistedPlaylistUpdateInputResult { + pub state: PlaylistUpdateState, + pub timestamp: u64, +} + +/// Persisted playlist-update facts for one configured input. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct InputPlaylistUpdateStatusDto { + pub input_id: u16, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub last_update: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub last_input_update: Option, + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub clusters: Vec, +} + +/// Persisted input results plus optional, volatile facts of still running updates. +#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct PlaylistUpdateStatusDto { + #[serde(default)] + pub inputs: Vec, + /// Existing correlated progress facts, never written into status.json. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub active_updates: Vec, +} + +#[cfg(test)] +mod tests { + use super::{ + InputPlaylistUpdateStatusDto, PersistedPlaylistUpdateClusterSnapshot, PersistedPlaylistUpdateClusterState, + PersistedPlaylistUpdateClusterStatusDto, PersistedPlaylistUpdateQualityDecision, + PersistedPlaylistUpdateQualitySnapshot, PersistedPlaylistUpdateTechnicalState, PlaylistUpdateStatusDto, + }; + use crate::model::{InputRefreshPolicy, PlaylistUpdateDataSource, XtreamCluster}; + + #[test] + fn playlist_update_status_last_input_result_is_optional_terminal_and_round_trips() -> Result<(), serde_json::Error> + { + let legacy = serde_json::from_str::(r#"{"input_id":7,"last_update":23}"#)?; + assert_eq!(legacy.last_input_update, None); + for state in [ + super::PlaylistUpdateState::Success, + super::PlaylistUpdateState::Partial, + super::PlaylistUpdateState::Failure, + ] { + let result = super::PersistedPlaylistUpdateInputResult { state, timestamp: 123 }; + let dto = InputPlaylistUpdateStatusDto { last_input_update: Some(result), ..legacy.clone() }; + assert_eq!(serde_json::from_str::(&serde_json::to_string(&dto)?)?, dto); + } + // A transient queued/updating state must never become a durable terminal result. + assert!(serde_json::from_str::( + r#"{"state":"Queued","timestamp":123}"# + ) + .is_err()); + Ok(()) + } + + #[test] + fn playlist_update_status_dto_round_trips_ids_timestamps_and_cluster_states() -> Result<(), serde_json::Error> { + let status = PlaylistUpdateStatusDto { + inputs: vec![ + InputPlaylistUpdateStatusDto { + input_id: 7, + last_update: Some(1_725_000_000), + last_input_update: None, + clusters: vec![PersistedPlaylistUpdateClusterStatusDto { + cluster: XtreamCluster::Video, + status: PersistedPlaylistUpdateClusterState::Failed, + timestamp: 1_725_000_000, + last_update: Some(PersistedPlaylistUpdateClusterSnapshot { + policy: Some(InputRefreshPolicy::REFRESH), + source: Some(PlaylistUpdateDataSource::Provider), + quality_guard_threshold: None, + quality: Some(PersistedPlaylistUpdateQualitySnapshot { + threshold: 95, + baseline_count: Some(4_812), + candidate_count: Some(3_104), + achieved_quality: Some(64), + decision: PersistedPlaylistUpdateQualityDecision::Rejected, + }), + active_count: Some(4_812), + technical_state: Some(PersistedPlaylistUpdateTechnicalState::Succeeded), + }), + }], + }, + InputPlaylistUpdateStatusDto { + input_id: 9, + last_update: None, + last_input_update: None, + clusters: Vec::new(), + }, + ], + ..PlaylistUpdateStatusDto::default() + }; + + let encoded = serde_json::to_string(&status)?; + let decoded = serde_json::from_str::(&encoded)?; + + assert_eq!(decoded, status); + Ok(()) + } + + #[test] + fn playlist_update_status_dto_accepts_legacy_payload_without_cluster_statuses() -> Result<(), serde_json::Error> { + let decoded = + serde_json::from_str::(r#"{"inputs":[{"input_id":7,"last_update":1725000000}]}"#)?; + + assert!(decoded.inputs[0].clusters.is_empty()); + assert!(decoded.active_updates.is_empty()); + Ok(()) + } + + #[test] + fn playlist_update_status_active_read_facts_round_trip_without_changing_persisted_results() { + let event = super::PlaylistUpdateProgressEvent::for_run_input("r1".into(), 1.into(), 7, "provider", "updating"); + let status = PlaylistUpdateStatusDto { active_updates: vec![event], ..PlaylistUpdateStatusDto::default() }; + let encoded = serde_json::to_string(&status).unwrap(); + assert_eq!(serde_json::from_str::(&encoded).unwrap(), status); + assert!(!serde_json::to_string(&PlaylistUpdateStatusDto::default()).unwrap().contains("active_updates")); + assert!(serde_json::from_str::( + r#"{"state":"Updating","timestamp":123}"# + ) + .is_err()); + } + + #[test] + fn playlist_update_status_dto_accepts_block_22_cluster_without_snapshot() -> Result<(), serde_json::Error> { + let decoded = serde_json::from_str::( + r#"{"inputs":[{"input_id":7,"clusters":[{"cluster":"Live","status":"ok","timestamp":1725000000}]}]}"#, + )?; + + assert_eq!(decoded.inputs[0].clusters[0].last_update, None); + Ok(()) + } + + #[test] + fn playlist_update_status_dto_accepts_block_22a_snapshot_without_historical_guard() -> Result<(), serde_json::Error> + { + let decoded = serde_json::from_str::( + r#"{"inputs":[{"input_id":7,"clusters":[{"cluster":"Live","status":"ok","timestamp":1725000000,"last_update":{"policy":{"cache":"bypass","quality":"bypass"},"source":"provider","active_count":42,"technical_state":"succeeded"}}]}]}"#, + )?; + + let snapshot = decoded.inputs[0].clusters[0].last_update.expect("Block-22a snapshot"); + assert_eq!(snapshot.policy, Some(InputRefreshPolicy::FORCE)); + assert_eq!(snapshot.quality_guard_threshold, None); + Ok(()) + } +} diff --git a/shared/src/model/progress.rs b/shared/src/model/progress.rs index d07d1ce97..016b0c985 100644 --- a/shared/src/model/progress.rs +++ b/shared/src/model/progress.rs @@ -1,18 +1,389 @@ -use crate::model::LibraryScanSummary; +use crate::model::{ + InputRefreshPolicy, LibraryScanResult, LibraryScanSummary, PersistedPlaylistUpdateTechnicalState, + PlaylistUpdateRunId, PlaylistUpdateRunOrder, PlaylistUpdateState, XtreamCluster, +}; use serde::{Deserialize, Serialize}; -/// Marker response for accepted long-running operations. Carries no payload: -/// progress is delivered over the event channel, not through this response. +/// Proven retrieval source selected for an input during one playlist-update run. +#[derive(Debug, Clone, Copy, Deserialize, Serialize, PartialEq, Eq)] +#[serde(rename_all = "snake_case")] +pub enum PlaylistUpdateDataSource { + Cache, + Provider, +} + +/// Backend-owned result for one requested Xtream-compatible cluster. +#[derive(Debug, Clone, Copy, Deserialize, Serialize, PartialEq, Eq)] +#[serde(rename_all = "snake_case")] +pub enum PlaylistUpdateClusterDecision { + Accepted, + Rejected, + TechnicalError, +} + +/// Stable, presentation-neutral details that clients can localize. +#[derive(Debug, Clone, Copy, Deserialize, Serialize, PartialEq, Eq)] +#[serde(rename_all = "snake_case")] +pub enum PlaylistUpdateProgressDetail { + InputCompletedSuccessfully, + InputCompletedPartially, + InputFailed, + ForcedUpdateRequested, + LibraryRescanStarted, + LibraryScanResult, + LibraryRescanCompleted, + LibraryRescanFailed, +} + +/// Runtime facts proven for one cluster during a playlist-update run. +/// +/// Population and quality values remain optional because not every provider +/// boundary exposes them. Consumers must not reconstruct missing values. +#[derive(Debug, Clone, Deserialize, Serialize, PartialEq, Eq)] +pub struct PlaylistUpdateClusterTelemetry { + pub cluster: XtreamCluster, + pub requested: bool, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub source: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub baseline_count: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub candidate_count: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub active_count: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub threshold: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub quality: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub decision: Option, + /// Proven technical result, independent of an earlier Quality decision. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub technical_state: Option, +} + +/// Input-scoped runtime facts carried by the existing progress lifecycle. +#[derive(Debug, Clone, Deserialize, Serialize, PartialEq, Eq)] +pub struct PlaylistUpdateInputTelemetry { + pub refresh_policy: InputRefreshPolicy, + /// Set for inputs without cluster-specific sources, such as M3U. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub source: Option, + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub clusters: Vec, +} + +/// Response for an accepted long-running operation. +/// +/// Playlist updates return the identity of the queued run. Other operations and +/// legacy payloads may leave it empty. #[derive(Debug, Clone, Default, Deserialize, Serialize, PartialEq, Eq)] -pub struct OperationRunAccepted {} +pub struct OperationRunAccepted { + #[serde(default, skip_serializing_if = "Option::is_none")] + pub run_id: Option, +} + +impl OperationRunAccepted { + #[must_use] + pub fn playlist_update(run_id: PlaylistUpdateRunId) -> Self { Self { run_id: Some(run_id) } } +} #[derive(Debug, Clone, Deserialize, Serialize, PartialEq, Eq)] pub struct PlaylistUpdateProgressEvent { + /// Stable update-run identity. Missing only on legacy payloads. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub run_id: Option, + /// Actual serial execution order. Missing on accepted responses and legacy events. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub execution_order: Option, + /// Stable input identity for input-scoped progress. Global and target-scoped + /// processing steps leave this empty because they cannot be assigned to one input. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub input_id: Option, + /// Input-local completion state when this progress line closes an input job. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub state: Option, + /// Optional structured input facts. Missing on legacy and text-only + /// progress frames. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub input_telemetry: Option, + /// Actual scanner result for this run/input, including unsuccessful scans. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub library_scan_result: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub detail: Option, pub target: String, pub message: String, } +impl PlaylistUpdateProgressEvent { + #[must_use] + pub fn for_run_input( + run_id: PlaylistUpdateRunId, + execution_order: PlaylistUpdateRunOrder, + input_id: u16, + target: impl Into, + message: impl Into, + ) -> Self { + Self { + run_id: Some(run_id), + execution_order: Some(execution_order), + input_id: Some(input_id), + state: None, + input_telemetry: None, + library_scan_result: None, + detail: None, + target: target.into(), + message: message.into(), + } + } + + #[must_use] + pub fn input_completed( + run_id: PlaylistUpdateRunId, + execution_order: PlaylistUpdateRunOrder, + input_id: u16, + state: PlaylistUpdateState, + target: impl Into, + message: impl Into, + ) -> Self { + Self { + run_id: Some(run_id), + execution_order: Some(execution_order), + input_id: Some(input_id), + state: Some(state), + input_telemetry: None, + library_scan_result: None, + detail: Some(match state { + PlaylistUpdateState::Success => PlaylistUpdateProgressDetail::InputCompletedSuccessfully, + PlaylistUpdateState::Partial => PlaylistUpdateProgressDetail::InputCompletedPartially, + PlaylistUpdateState::Failure => PlaylistUpdateProgressDetail::InputFailed, + }), + target: target.into(), + message: message.into(), + } + } + + #[must_use] + pub fn for_run_global( + run_id: PlaylistUpdateRunId, + execution_order: PlaylistUpdateRunOrder, + target: impl Into, + message: impl Into, + ) -> Self { + Self { + run_id: Some(run_id), + execution_order: Some(execution_order), + input_id: None, + state: None, + input_telemetry: None, + library_scan_result: None, + detail: None, + target: target.into(), + message: message.into(), + } + } + + #[must_use] + pub fn for_input(input_id: u16, target: impl Into, message: impl Into) -> Self { + Self { + run_id: None, + execution_order: None, + input_id: Some(input_id), + state: None, + input_telemetry: None, + library_scan_result: None, + detail: None, + target: target.into(), + message: message.into(), + } + } + + #[must_use] + pub fn global(target: impl Into, message: impl Into) -> Self { + Self { + run_id: None, + execution_order: None, + input_id: None, + state: None, + input_telemetry: None, + library_scan_result: None, + detail: None, + target: target.into(), + message: message.into(), + } + } + + /// Attaches input telemetry without creating another event family. + #[must_use] + pub fn with_input_telemetry(mut self, input_telemetry: PlaylistUpdateInputTelemetry) -> Self { + self.input_telemetry = Some(input_telemetry); + self + } + + /// Carries the scanner's facts without changing acquisition or Quality telemetry. + #[must_use] + pub fn with_library_scan_result(mut self, result: LibraryScanResult) -> Self { + self.library_scan_result = Some(result); + self + } + + /// Attaches a stable UI detail while retaining the diagnostic log message. + #[must_use] + pub const fn with_detail(mut self, detail: PlaylistUpdateProgressDetail) -> Self { + self.detail = Some(detail); + self + } +} + #[derive(Debug, Clone, Deserialize, Serialize, PartialEq)] pub struct LibraryScanProgressEvent { pub summary: LibraryScanSummary, } + +#[cfg(test)] +mod tests { + #[test] + fn library_scan_result_roundtrips_with_run_identity_and_legacy_remains_unknown() { + let result = crate::model::LibraryScanResult { + files_scanned: 11, + groups_scanned: 7, + files_added: 3, + files_updated: 2, + files_removed: 1, + errors: 4, + }; + let event = + super::PlaylistUpdateProgressEvent::for_run_input("library-run".into(), 8.into(), 17, "Library", "result") + .with_library_scan_result(result.clone()); + let encoded = serde_json::to_string(&event).unwrap(); + let decoded: super::PlaylistUpdateProgressEvent = serde_json::from_str(&encoded).unwrap(); + assert_eq!(decoded, event); + assert_eq!(decoded.library_scan_result, Some(result)); + let old: super::PlaylistUpdateProgressEvent = + serde_json::from_str(r#"{"target":"Library","message":"scanning"}"#).unwrap(); + assert_eq!(old.library_scan_result, None); + assert!(!serde_json::to_string(&old).unwrap().contains("library_scan_result")); + } + + use super::{ + OperationRunAccepted, PlaylistUpdateClusterDecision, PlaylistUpdateClusterTelemetry, PlaylistUpdateDataSource, + PlaylistUpdateInputTelemetry, PlaylistUpdateProgressEvent, + }; + use crate::model::{ + InputRefreshPolicy, PlaylistUpdateRunId, PlaylistUpdateRunOrder, PlaylistUpdateState, XtreamCluster, + }; + + #[test] + fn playlist_update_progress_legacy_payload_defaults_to_unscoped_identity() -> Result<(), serde_json::Error> { + let event: PlaylistUpdateProgressEvent = serde_json::from_str(r#"{"target":"legacy","message":"working"}"#)?; + + assert_eq!(event.input_id, None); + assert_eq!(event.run_id, None); + assert_eq!(event.execution_order, None); + assert_eq!(event.state, None); + assert_eq!(event.input_telemetry, None); + assert_eq!(event.target, "legacy"); + assert_eq!(event.message, "working"); + Ok(()) + } + + #[test] + fn playlist_update_progress_input_identity_roundtrips() -> Result<(), serde_json::Error> { + let event = PlaylistUpdateProgressEvent::input_completed( + "run-17".into(), + PlaylistUpdateRunOrder::from(23), + 17, + PlaylistUpdateState::Partial, + "provider", + "working", + ); + + let encoded = serde_json::to_string(&event)?; + let decoded = serde_json::from_str::(&encoded)?; + + assert_eq!(decoded, event); + assert_eq!(decoded.execution_order.map(PlaylistUpdateRunOrder::get), Some(23)); + Ok(()) + } + + #[test] + fn playlist_update_progress_correlated_payload_without_execution_order_remains_compatible( + ) -> Result<(), serde_json::Error> { + let event = serde_json::from_str::( + r#"{"run_id":"run-12-1","input_id":7,"target":"provider","message":"working"}"#, + )?; + + assert_eq!(event.run_id.as_ref().map(PlaylistUpdateRunId::as_ref), Some("run-12-1")); + assert_eq!(event.execution_order, None); + assert_eq!(event.input_id, Some(7)); + Ok(()) + } + + #[test] + fn playlist_update_progress_accepted_legacy_payload_has_no_run_identity() -> Result<(), serde_json::Error> { + let accepted = serde_json::from_str::("{}")?; + + assert_eq!(accepted.run_id, None); + Ok(()) + } + + #[test] + fn playlist_update_progress_accepted_response_roundtrips_run_identity() -> Result<(), serde_json::Error> { + let accepted = OperationRunAccepted::playlist_update("run-accepted".into()); + + let decoded = serde_json::from_str::(&serde_json::to_string(&accepted)?)?; + + assert_eq!(decoded.run_id.as_ref().map(PlaylistUpdateRunId::as_ref), Some("run-accepted")); + Ok(()) + } + + #[test] + fn pipeline_transparency_cluster_failure_is_optional_and_separate_from_quality() { + let legacy = serde_json::json!({"cluster": XtreamCluster::Series, "requested": true, "decision": "accepted"}); + let mut telemetry: PlaylistUpdateClusterTelemetry = serde_json::from_value(legacy.clone()).unwrap(); + assert_eq!(telemetry.technical_state, None); + assert_eq!(serde_json::to_value(&telemetry).unwrap(), legacy); + telemetry.technical_state = Some(super::PersistedPlaylistUpdateTechnicalState::Failed); + let encoded = serde_json::to_value(&telemetry).unwrap(); + assert_eq!(encoded["decision"], "accepted"); + assert_eq!(encoded["technical_state"], "failed"); + assert_eq!(serde_json::from_value::(encoded).unwrap(), telemetry); + } + + #[test] + fn pipeline_transparency_input_telemetry_roundtrips_on_existing_progress_event() -> Result<(), serde_json::Error> { + let telemetry = PlaylistUpdateInputTelemetry { + refresh_policy: InputRefreshPolicy::NORMAL, + source: None, + clusters: vec![PlaylistUpdateClusterTelemetry { + cluster: XtreamCluster::Video, + requested: true, + source: Some(PlaylistUpdateDataSource::Provider), + baseline_count: Some(12_543), + candidate_count: Some(217), + active_count: Some(12_543), + threshold: Some(90), + quality: Some(1), + decision: Some(PlaylistUpdateClusterDecision::Rejected), + technical_state: None, + }], + }; + let event = PlaylistUpdateProgressEvent::input_completed( + "run-telemetry".into(), + PlaylistUpdateRunOrder::from(31), + 17, + PlaylistUpdateState::Partial, + "provider", + "completed", + ) + .with_input_telemetry(telemetry.clone()); + + let encoded = serde_json::to_string(&event)?; + let decoded = serde_json::from_str::(&encoded)?; + + assert_eq!(decoded.input_telemetry, Some(telemetry)); + assert_eq!(decoded, event); + Ok(()) + } +} diff --git a/shared/src/model/web_socket.rs b/shared/src/model/web_socket.rs index 8220a7cbc..beada104b 100644 --- a/shared/src/model/web_socket.rs +++ b/shared/src/model/web_socket.rs @@ -1,13 +1,15 @@ use crate::model::{ user_command::UserCommand, ActiveUserConnectionChange, ConfigType, DownloadsDelta, DownloadsResponse, - FileDownloadDto, LibraryScanProgressEvent, PermissionSet, PlaylistUpdateProgressEvent, PlaylistUpdateState, + FileDownloadDto, LibraryScanProgressEvent, PermissionSet, PlaylistUpdateProgressEvent, PlaylistUpdateRunStateEvent, QueueRevision, StatusCheck, StreamMeterEntry, SystemInfo, }; use bytes::Bytes; use serde::{Deserialize, Serialize}; use std::{io, sync::Arc}; -pub const PROTOCOL_VERSION: u8 = 3; +// Version 4 carries correlated run objects in PlaylistUpdateResponse. Version 3 +// clients expect a bare state and must reconnect with an updated frontend. +pub const PROTOCOL_VERSION: u8 = 4; #[derive(Default, PartialOrd, PartialEq, Debug, Clone)] pub enum UserRole { @@ -93,7 +95,7 @@ pub enum ProtocolMessage { ActiveProviderCountRequest(String), ActiveProviderCountResponse(usize), ConfigChangeResponse(ConfigType), - PlaylistUpdateResponse(PlaylistUpdateState), + PlaylistUpdateResponse(PlaylistUpdateRunStateEvent), PlaylistUpdateProgressResponse(PlaylistUpdateProgressEvent), UserActionResponse(bool), SystemInfoResponse(SystemInfo), @@ -153,3 +155,40 @@ impl ProtocolMessage { } } } + +#[cfg(test)] +mod tests { + use super::{ProtocolMessage, PROTOCOL_VERSION}; + use crate::model::{PlaylistUpdateRunStateEvent, PlaylistUpdateState}; + use bytes::Bytes; + + #[test] + fn playlist_update_protocol_version_distinguishes_legacy_clients() { + assert_eq!(PROTOCOL_VERSION, 4); + assert_eq!(ProtocolMessage::Version(PROTOCOL_VERSION).to_bytes().unwrap().as_ref(), &[4]); + let ProtocolMessage::Version(legacy) = ProtocolMessage::from_bytes(Bytes::from_static(&[3])).unwrap() else { + panic!("expected a protocol handshake"); + }; + assert_ne!(legacy, PROTOCOL_VERSION, "the server's version gate must reject legacy clients"); + } + + #[test] + fn playlist_update_protocol_preserves_correlated_frames_and_legacy_decoding() { + for state in [PlaylistUpdateState::Success, PlaylistUpdateState::Partial, PlaylistUpdateState::Failure] { + let event = PlaylistUpdateRunStateEvent::correlated("review-run".into(), 17.into(), state); + let bytes = ProtocolMessage::PlaylistUpdateResponse(event.clone()).to_bytes().unwrap(); + let ProtocolMessage::PlaylistUpdateResponse(decoded) = ProtocolMessage::from_bytes(bytes).unwrap() else { + panic!("expected a correlated playlist update frame"); + }; + assert_eq!(decoded, event); + + let legacy = serde_json::to_vec(&serde_json::json!({"PlaylistUpdateResponse": state})).unwrap(); + let ProtocolMessage::PlaylistUpdateResponse(decoded) = + ProtocolMessage::from_bytes(Bytes::from(legacy)).unwrap() + else { + panic!("expected a legacy playlist update frame"); + }; + assert_eq!(decoded, PlaylistUpdateRunStateEvent::uncorrelated(state)); + } + } +}