diff --git a/CHANGELOG.md b/CHANGELOG.md index 769f0c054..645420e19 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ ## ⚠️ Breaking Changes +- `working_dir` in `config.yml` renamed to `storage_dir`. - **Global Input Definitions**: To align input definitions with the SourceEditor, inputs are now defined globally in the `inputs` section of the config file. Each source can reference one or more inputs by their name in the `inputs` attribute. - **Data Format Migration**: Due to heavy refactoring, the old data format is invalid. You need to clean your `data` folder and update the playlists. @@ -38,6 +39,27 @@ fallback_to_filename: true ``` +- **Input Batch URL Scheme**: Batch input URLs now use the `batch://` scheme instead of `file://`. + `file://` is no longer accepted for batch CSV definitions. Update your `source.yml`: + + ```yaml + # Before + inputs: + - type: xtream_batch + url: 'file:///home/tuliprox/config/batch.csv' + + # After + inputs: + - type: xtream_batch + url: 'batch:///home/tuliprox/config/batch.csv' + ``` + + Local paths without a scheme (`/path/file.csv`, `./file.csv`) continue to work. + The `batch://` scheme clearly distinguishes batch alias files from provider `file://` URLs. +- **DNS Resolved Persistence**: `dns.resolved` has been removed from `source.yml` and the `ProviderDnsDto`. + Resolved IPs are now persisted separately in `{storage_dir}/provider_dns_resolved.json`. + This eliminates hot-reload interference caused by DNS refresh cycles writing to `source.yml`. + DNS caches are automatically carried over during config hot-reloads. - **Input Batch Changes**: `name` attribute is now mandatory for input type batch to ensure stable playlist UUIDs. - **Favorites Redesign**: Replaced implicit `create_alias` with explicit `add_favourite(group_name)` script function. - **EpgSmartMatch**: Field `name_prefix` syntax needs to be changed from `name_prefix: !suffix "."` to `name_prefix: { suffix: "." }`. @@ -104,6 +126,7 @@ - **Playlist Caching**: Added `cache_duration` to inputs, allowing configurable provider playlist cache times during subsequent updates (e.g., `60s`, `5m` `12h`, `1d`). - **Database Viewer**: New CLI flags `--dbx` and `--dbm` to inspect internal database content. +- **Home Directory Override**: Added `--home` (`-H`) CLI argument to set the base directory for config, data, backup, and downloads. - **Added `disk_based_processing`**: (boolean, default `false`) to `config.yml`. When enabled, input playlists are processed from disk instead of memory. - **User-Agent `default_user_agent`**: Ensures that outgoing requests always pass a default user agent. diff --git a/README.md b/README.md index 9e45041fd..8dc58da7e 100644 --- a/README.md +++ b/README.md @@ -61,6 +61,7 @@ Define complex filters using expressive logic, e.g.: Usage: tuliprox [OPTIONS] Options: + -H, --home The home directory (base for config, storage, backup, downloads, web) -p, --config-path The config directory -c, --config The config file -i, --source The source config file @@ -83,10 +84,27 @@ Options: ``` +Home directory resolution order: + +1. `--home` +2. `TULIPROX_HOME` environment variable +3. Directory of the `tuliprox` binary + +All relative paths that are resolved against `home` (including `storage_dir` when configured as relative) use this order. + +Default layout under `{home}`: + +- `{home}/config` +- `{home}/data` (`storage_dir` default) +- `{home}/data/backup` (`backup_dir` default = `{storage_dir}/backup`) +- `{home}/downloads` +- `{home}/web` + ## 1. `config.yml` -For running in cli mode, you need to define a `config.yml` file which can be inside config directory next to the executable or provided with the -`-c` cli argument. +For CLI mode, `config.yml` is loaded from `{home}/config/config.yml` by default. +`{home}` is resolved in this order: `--home` -> `TULIPROX_HOME` -> directory of the `tuliprox` binary. +You can override the config file with `-c` (and config directory with `-p`). For running specific targets use the `-t` argument like `tuliprox -t -t `. Target names should be provided in the config. The -t option overrides `enabled` attributes of `input` and `target` elements. @@ -95,7 +113,7 @@ This means, even disabled inputs and targets are processed when the given target Top level entries in the config files are: - `api` -- `working_dir` +- `storage_dir` - `default_user_agent` _optional_, used as fallback for upstream requests when no `User-Agent` is provided by input headers or the client request (client request overrides it). - `process_parallel` _optional_ @@ -132,12 +150,16 @@ If you process the same provider multiple times each thread uses a connection. K `api` contains the `server-mode` settings. To run `tuliprox` in `server-mode` you need to start it with the `-s`cli argument. -`api: {host: localhost, port: 8901, web_root: ./web}` -### 1.3. `working_dir` +`web_root` follows the same home-based resolution rules: +with `web_root: ./web`, the effective path is `{home}/web` (`--home` -> `TULIPROX_HOME` -> binary directory). -`working_dir` is the directory where files are written which are given with relative paths. --`working_dir: ./data` +### 1.3. `storage_dir` -With this configuration, you should create a `data` directory where you execute the binary. +`storage_dir` is the directory where files are written which are given with relative paths. +-`storage_dir: ./data` + +`storage_dir` is resolved relative to `home` (resolved via `--home`, then `TULIPROX_HOME`, then binary directory). +With `storage_dir: ./data`, the effective directory is `{home}/data`. Be aware that different configurations (e.g. user bouquets) along the playlists are stored in this directory. @@ -208,7 +230,6 @@ Each provider can optionally enable a `dns` block: - `overrides`: static host -> IP list (used before DNS lookup) - `on_resolve_error`: `keep_last_good` | `fallback_to_hostname` (default: `keep_last_good`) - `on_connect_error`: `try_next_ip` | `rotate_provider_url` (default: `try_next_ip`) -- `resolved`: runtime-managed resolved snapshot per host Behavior: @@ -219,12 +240,12 @@ Behavior: - `keep_vhost=true`: `Host` header keeps `hostname[:port]`. - On connect/timeout errors and `on_connect_error=try_next_ip`, Tuliprox tries the next IP for the same host before rotating provider URL. -### 1.4.4 `dns.resolved` persistence and visibility +### 1.4.4 DNS resolved IP persistence -- `dns.resolved` is runtime-managed. -- It is exposed in `GET /api/v1/config`. -- It is also written back to `source.yml` and overwritten on each DNS refresh cycle. -- Save endpoints treat `dns.resolved` as managed data and do not accept it as user-controlled input. +Resolved IPs are persisted to `{storage_dir}/provider_dns_resolved.json` (not to `source.yml`). +This file is written atomically after each DNS refresh cycle and read at startup to seed DNS caches +before the background resolver completes its first cycle. On config hot-reloads, DNS caches are +carried over from the previous provider instances so that resolved IPs are available immediately. ### 1.4.5 Provider Failover + DNS Configuration Example @@ -253,10 +274,6 @@ provider: overrides: stable.golden-bridge.con: - 203.0.113.10 - # runtime-managed, written by tuliprox: - resolved: - hello.provider.me: - - 203.0.113.20 inputs: - name: my_input type: xtream @@ -477,7 +494,7 @@ metadata_update: **Group overview:** -- `cache_path` (default `metadata`): Directory where TMDB cache files and metadata are stored. Relative paths are resolved against `working_dir`. +- `cache_path` (default `metadata`): Directory where TMDB cache files and metadata are stored. Relative paths are resolved against `storage_dir`. Used by all metadata resolution paths (Xtream VOD/Series, local library). - `resolve`: Controls retries for metadata lookup tasks (for example title/date/TMDB resolution). - `probe`: Controls retries and cooldown for technical stream probing tasks. @@ -653,12 +670,25 @@ schedules: - xtream - schedule: "0 0 20 * * * *" type: LibraryScan +- schedule: "0 0 4 * * 1 *" + type: GeoIpUpdate +- schedule: "0 0 4 1 * * *" + type: GeoIpUpdate ``` The `type` attribute defines the task to be executed and defaults to `PlaylistUpdate` if omitted. Possible values: - `PlaylistUpdate`: Updates the target playlists (optionally filtered by `targets`). - `LibraryScan`: Triggers a scan of the local media library (requires `library` configuration to be enabled). +- `GeoIpUpdate`: Downloads and rebuilds the Geo-IP database file (requires `reverse_proxy.geoip.enabled: true`). + +Cron fields for Tuliprox schedules always start with **seconds**: + +```yaml +# sec min hour day-of-month month day-of-week year +schedule: "0 0 4 * * 1 *" # every Monday at 04:00:00 +schedule: "0 0 4 1 * * *" # every 1st of the month at 04:00:00 +``` At the given times the update is started. Do not start it every second or minute. You could be banned from your server. Twice a day should be enough. @@ -741,7 +771,7 @@ This is useful for players that might time out or error if they receive data and #### 1.7.2 `cache` -LRU-Cache is for resources. If it is `enabled`, the resources/images are persisted in the given `dir`. If the cache size exceeds `size`, +LRU-Cache is for resources. If it is `enabled`, the resources/images are persisted in the given `directory`. If the cache size exceeds `size`, In an LRU cache, the least recently used items are evicted to make room for new items if the cache `size`is exceeded. #### 1.7.3 `resource_rewrite_disabled` @@ -762,7 +792,7 @@ reverse_proxy: cache: enabled: true size: 1GB - dir: ./cache + directory: ./cache ``` #### 1.7.4 `rate_limit` @@ -983,13 +1013,17 @@ The encrypted pasword needs to be added manually into the users file. ```yaml threads: 4 -working_dir: ./data +storage_dir: ./data api: host: localhost port: 8901 web_root: ./web ``` +Relative paths in this example are resolved under `{home}`. +So `storage_dir: ./data` -> `{home}/data` and `web_root: ./web` -> `{home}/web` +(`--home` -> `TULIPROX_HOME` -> binary directory). + ### 1.12 `user_access_control` The default is `false`. @@ -1463,6 +1497,23 @@ inputs: probe_stream: true ``` +#### Input URL Schemes (`inputs[].url`) + +Tuliprox supports the following URL schemes for input sources: + +- `http://...` / `https://...`: Download from remote provider endpoints. +- `file://...`: Read from a local file. +- `provider:///...`: Resolve the URL via `provider` definitions (supports failover/rotation). +- `batch://...`: CSV source for batch aliases (e.g. `batch:///path/file.csv` or `batch://./file.csv`). + +Notes: + +- If `type` is `m3u` or `xtream` and the URL starts with `batch://`, Tuliprox automatically treats it as `m3u_batch` / `xtream_batch`. +- For `m3u_batch` and `xtream_batch`, only local CSV sources are supported: + - `batch://...` or + - plain local filesystem path (absolute/relative). +- For batch inputs, `provider://...` and other URI schemes such as `http(s)://...` or `file://...` are rejected. + Input alias definition for same provider with same content but different credentials. `max_connections` default is unlimited @@ -1488,7 +1539,7 @@ sources: Input aliases can be defined as batches in csv files with `;` separator. There are 2 batch input types `xtream_batch` and `m3u_batch`. -Batch inputs are file-based only: use a local path (`/path/file.csv`, `./file.csv`) or `file://...`. +Batch inputs use the `batch://` URL scheme: `batch:///path/file.csv` (absolute) or `batch://./file.csv` (relative). #### `XtreamBatch` @@ -1496,7 +1547,7 @@ Batch inputs are file-based only: use a local path (`/path/file.csv`, `./file.cs inputs: - type: xtream_batch name: my_provider # Mandatory: used for playlist UUID generation - url: 'file:///home/tuliprox/config/my_provider_batch.csv' + url: 'batch:///home/tuliprox/config/my_provider_batch.csv' sources: - inputs: - my_provider @@ -1520,7 +1571,7 @@ This is necessary because of playlist uuid generation and assigning same channel ```yaml inputs: - type: m3u_batch - url: 'file:///home/tuliprox/config/my_provider_batch.csv' + url: 'batch:///home/tuliprox/config/my_provider_batch.csv' sources: - inputs: - m3u_batch @@ -2993,25 +3044,30 @@ The provider gives you: - epg_url: `http://fantastic.provider.xyz:8080/xmltv.php?username=tvjunkie&password=junkie.secret` To use `tuliprox` you need to create the configuration. -The configuration consist of 4 files. +The configuration consists of 3 required files and up to 2 optional files. + +Required files: - config.yml - source.yml -- mapping.yml - api-proxy.yml -The file `mapping.yml`is optional and only needed if you want to do something linke renaming titles or changing attributes. +Optional files: + +- mapping.yml (for advanced transformations like renaming titles or changing attributes) +- template.yml (for centralized pattern reuse) Lets start with `config.yml`. An example basic configuration is: ```yaml api: {host: 0.0.0.0, port: 8901, web_root: ./web} -working_dir: ./data +storage_dir: ./data update_on_boot: true ``` -This configuration starts `tuliprox`and listens on the 8901 port. The downloaded playlists are stored inside the `data`-folder in the current working -directory. +This configuration starts `tuliprox`and listens on the 8901 port. The downloaded playlists are stored inside the `data` folder under the resolved +home directory (`--home`, then `TULIPROX_HOME`, then binary directory), i.e. `{home}/data`. +`web_root: ./web` resolves to `{home}/web` with the same order. The property `update_on_boot` is optional and can be helpful in the beginning until you have found a working configuration. I prefer to set it to false. diff --git a/backend/src/api/config_file.rs b/backend/src/api/config_file.rs index ff44ec1c4..4ffdf4e40 100644 --- a/backend/src/api/config_file.rs +++ b/backend/src/api/config_file.rs @@ -223,20 +223,24 @@ impl ConfigFile { // ── Parse ──────────────────────────────────────────────────── let config_dto = read_config_file(config_file.as_str(), true, true)?; - let default_mapping_path = utils::get_default_mappings_path(paths.config_path.as_str()); - let current_mapping_path = paths.mapping_file_path.clone().unwrap_or_else(|| default_mapping_path.clone()); - let next_mapping_path = - config_dto.mapping_path.clone().filter(|path| !path.trim().is_empty()).unwrap_or(default_mapping_path); + let current_mapping_path = paths.mapping_file_path.clone().unwrap_or_else(|| { + utils::resolve_mapping_file_path(paths.config_path.as_str(), None) + }); + let next_mapping_path = paths.mapping_file_path.clone().unwrap_or_else(|| { + utils::resolve_mapping_file_path(paths.config_path.as_str(), config_dto.mapping_path.as_deref()) + }); let mapping_changed = current_mapping_path != next_mapping_path; - let default_template_path = utils::get_default_templates_path(paths.config_path.as_str()); - let current_template_path = paths.template_file_path.clone().unwrap_or_else(|| default_template_path.clone()); - let next_template_path = - config_dto.template_path.clone().filter(|path| !path.trim().is_empty()).unwrap_or(default_template_path); + let current_template_path = paths.template_file_path.clone().unwrap_or_else(|| { + utils::resolve_template_file_path(paths.config_path.as_str(), None) + }); + let next_template_path = paths.template_file_path.clone().unwrap_or_else(|| { + utils::resolve_template_file_path(paths.config_path.as_str(), config_dto.template_path.as_deref()) + }); let template_changed = current_template_path != next_template_path; let mut config: Config = Config::from(config_dto); - config.prepare(paths.config_path.as_str())?; + config.prepare(paths.config_path.as_str(), paths.home_path.as_str())?; // Compute effective runtime paths for the NEW config before apply. // This ensures prepare-phase reads/validates against the same files that will be active after apply. diff --git a/backend/src/api/endpoints/api_playlist_utils.rs b/backend/src/api/endpoints/api_playlist_utils.rs index 1c98e3a1b..6fc866508 100644 --- a/backend/src/api/endpoints/api_playlist_utils.rs +++ b/backend/src/api/endpoints/api_playlist_utils.rs @@ -129,7 +129,7 @@ pub(in crate::api::endpoints) async fn get_playlist_for_custom_provider( let cfg = app_state.app_config.config.load(); match cfg_input { Some(input) => { - let (result, errors) = match input.input_type { + let (result, errors) = match input.get_download_input_type() { InputType::M3u | InputType::M3uBatch => { m3u::download_m3u_playlist(&app_state.app_config, client, &cfg, input).await } diff --git a/backend/src/api/endpoints/library_api.rs b/backend/src/api/endpoints/library_api.rs index 9832f29ba..3dd84a934 100644 --- a/backend/src/api/endpoints/library_api.rs +++ b/backend/src/api/endpoints/library_api.rs @@ -34,10 +34,10 @@ async fn scan_library( }; // Check if Library is enabled - let (lib_config, metadata_update_config, working_dir) = { + let (lib_config, metadata_update_config, storage_dir) = { let config = app_state.app_config.config.load(); match config.library.as_ref() { - Some(lib) if lib.enabled => (lib.clone(), config.metadata_update.clone(), config.working_dir.clone()), + Some(lib) if lib.enabled => (lib.clone(), config.metadata_update.clone(), config.storage_dir.clone()), _ => { let response = LibraryScanSummary { status: LibraryScanSummaryStatus::Error, @@ -61,7 +61,7 @@ async fn scan_library( lib_config, metadata_update_config, client, - LibraryScanTaskOptions { force_rescan: request.force_rescan, message_prefix: "", working_dir }, + LibraryScanTaskOptions { force_rescan: request.force_rescan, message_prefix: "", storage_dir }, permit, ); @@ -78,7 +78,7 @@ async fn get_library_status( let client = app_state.http_client.load_full().as_ref().clone(); // Get statistics from processor let processor = - LibraryProcessor::new(config.clone(), config_snapshot.metadata_update.as_ref(), client, &config_snapshot.working_dir); + LibraryProcessor::new(config.clone(), config_snapshot.metadata_update.as_ref(), client, &config_snapshot.storage_dir); let entries = processor.get_all_entries().await; let movies = entries.iter().filter(|e| e.metadata.is_movie()).count(); @@ -90,7 +90,7 @@ async fn get_library_status( movies, series, path: Some( - resolve_metadata_storage_path(config_snapshot.metadata_update.as_ref(), &config_snapshot.working_dir) + resolve_metadata_storage_path(config_snapshot.metadata_update.as_ref(), &config_snapshot.storage_dir) .to_string_lossy() .to_string(), ), diff --git a/backend/src/api/endpoints/v1_api.rs b/backend/src/api/endpoints/v1_api.rs index d726e7fd2..72a1201ae 100644 --- a/backend/src/api/endpoints/v1_api.rs +++ b/backend/src/api/endpoints/v1_api.rs @@ -10,22 +10,16 @@ use crate::{ model::AppState, }, auth::validator_admin, - model::InputSource, - repository::get_geoip_path, - utils::{ip_checker::get_ips, request::download_text_content, GeoIp}, + processing::geoip::{update_geoip_db, GeoIpUpdateError}, + utils::ip_checker::get_ips, VERSION, }; use axum::response::IntoResponse; -use log::{error, info}; use shared::{ - model::{default_geoip_url, InputFetchMethod, IpCheckDto, StatusCheck}, - utils::{concat_path_leading_slash, Internable}, -}; -use std::{ - collections::{BTreeMap, HashMap}, - io::Cursor, - sync::Arc, + model::{IpCheckDto, StatusCheck}, + utils::concat_path_leading_slash, }; +use std::{collections::BTreeMap, sync::Arc}; async fn create_ipinfo_check(app_state: &Arc) -> Option<(Option, Option)> { let config = app_state.app_config.config.load(); @@ -85,63 +79,16 @@ async fn streams( async fn geoip_update( axum::extract::State(app_state): axum::extract::State>, ) -> axum::response::Response { - let config = app_state.app_config.config.load(); - if let Some(geoip) = config.reverse_proxy.as_ref().and_then(|r| r.geoip.as_ref()) { - if geoip.enabled { - let geoip_db_path = &*get_geoip_path(&config.working_dir); - let _file_lock = app_state.app_config.file_locks.write_lock(geoip_db_path).await; - - let url = if geoip.url.trim().is_empty() { default_geoip_url() } else { geoip.url.clone() }; - let input_source = InputSource { - name: "GeoIP".intern(), - url, - provider: None, - username: None, - password: None, - method: InputFetchMethod::GET, - headers: HashMap::default(), - }; - return match download_text_content( - &app_state.app_config, - &app_state.http_client.load(), - &input_source, - None, - None, - false, - ) - .await - { - Ok((content, _)) => { - let reader = Cursor::new(content); - let mut geoip = GeoIp::new(); - let result = { - match geoip.import_ipv4_from_csv(reader, geoip_db_path) { - Ok(size) => (Some(size), None), - Err(err) => (None, Some(err)), - } - }; - - return match result { - (Some(_), None) => { - info!("GeoIp db updated"); - app_state.geoip.store(Some(Arc::new(geoip))); - axum::http::StatusCode::OK.into_response() - } - (None, Some(err)) => { - error!("Failed to process geoip db: {err}"); - internal_server_error!() - } - _ => internal_server_error!(), - }; - } - Err(err) => { - error!("Failed to download geoip db: {err}"); - axum::http::StatusCode::BAD_REQUEST.into_response() - } - }; + match update_geoip_db(&app_state).await { + Ok(()) => axum::http::StatusCode::OK.into_response(), + Err(GeoIpUpdateError::Disabled | GeoIpUpdateError::DownloadFailed(_)) => { + axum::http::StatusCode::BAD_REQUEST.into_response() + } + Err(err) => { + log::error!("GeoIp update failed: {err}"); + internal_server_error!() } } - axum::http::StatusCode::BAD_REQUEST.into_response() } async fn ipinfo(axum::extract::State(app_state): axum::extract::State>) -> axum::response::Response { diff --git a/backend/src/api/endpoints/v1_api_config.rs b/backend/src/api/endpoints/v1_api_config.rs index 919c7bc4c..410a57774 100644 --- a/backend/src/api/endpoints/v1_api_config.rs +++ b/backend/src/api/endpoints/v1_api_config.rs @@ -16,11 +16,11 @@ use axum::{ response::IntoResponse, Router, }; -use log::{error, warn}; +use log::error; use serde_json::json; use shared::{ error::TuliproxError, - model::{ApiProxyConfigDto, ConfigDto, ConfigProviderDto, ProviderDnsDto, SourcesConfigDto}, + model::{ApiProxyConfigDto, ConfigDto, SourcesConfigDto}, utils::{ HEADER_CONFIG_API_PROXY_REVISION, HEADER_CONFIG_MAIN_REVISION, HEADER_CONFIG_SOURCES_REVISION, HEADER_IF_MATCH, }, @@ -28,112 +28,7 @@ use shared::{ use std::path::Path; use std::sync::Arc; -fn inject_provider_dns_resolved(sources_dto: &mut SourcesConfigDto, runtime_sources: &crate::model::SourcesConfig) { - let Some(provider_dtos) = sources_dto.provider.as_mut() else { - return; - }; - for provider_dto in provider_dtos { - let Some(runtime_provider) = runtime_sources.get_provider_by_name(provider_dto.name.as_ref()) else { - continue; - }; - let Some(dns_dto) = provider_dto.dns.as_mut() else { - continue; - }; - if runtime_provider.get_dns_config().is_none() { - dns_dto.resolved = None; - continue; - } - let snapshot = runtime_provider.snapshot_resolved_ordered(); - dns_dto.resolved = (!snapshot.is_empty()).then_some(snapshot); - } -} -fn strip_provider_dns_resolved(sources_dto: &mut SourcesConfigDto) { - let Some(provider_dtos) = sources_dto.provider.as_mut() else { - return; - }; - for provider_dto in provider_dtos { - if let Some(dns_dto) = provider_dto.dns.as_mut() { - dns_dto.resolved = None; - } - } -} - -fn merge_provider_dns_resolved_from_existing( - sources_dto: &mut SourcesConfigDto, - existing_sources_dto: &SourcesConfigDto, -) { - let Some(provider_dtos) = sources_dto.provider.as_mut() else { - return; - }; - let Some(existing_provider_dtos) = existing_sources_dto.provider.as_ref() else { - return; - }; - - let mut resolved_by_provider_name = std::collections::HashMap::new(); - for provider in existing_provider_dtos { - let Some(dns) = provider.dns.as_ref() else { - continue; - }; - let Some(resolved) = dns.resolved.as_ref() else { - continue; - }; - if !resolved.is_empty() { - resolved_by_provider_name.insert(provider.name.to_string(), resolved.clone()); - } - } - - if resolved_by_provider_name.is_empty() { - return; - } - - for provider in provider_dtos { - let Some(dns) = provider.dns.as_mut() else { - continue; - }; - if !dns.enabled { - continue; - } - if dns.resolved.is_some() { - continue; - } - if let Some(resolved) = resolved_by_provider_name.get(provider.name.as_ref()) { - dns.resolved = Some(resolved.clone()); - } - } -} - -fn build_existing_sources_from_runtime(runtime_sources: &crate::model::SourcesConfig) -> Option { - let providers: Vec = runtime_sources - .provider - .iter() - .filter_map(|runtime_provider| { - let resolved = runtime_provider.snapshot_resolved_ordered(); - if resolved.is_empty() { - return None; - } - - Some(ConfigProviderDto { - name: runtime_provider.name.clone(), - urls: runtime_provider.urls.clone(), - dns: Some(ProviderDnsDto { - enabled: runtime_provider.get_dns_config().is_some_and(|cfg| cfg.enabled), - resolved: Some(resolved), - ..ProviderDnsDto::default() - }), - }) - }) - .collect(); - - if providers.is_empty() { - None - } else { - Some(SourcesConfigDto { - provider: Some(providers), - ..SourcesConfigDto::default() - }) - } -} fn file_revision_from_bytes(bytes: &[u8]) -> String { blake3::hash(bytes).to_hex().to_string() } @@ -273,7 +168,7 @@ async fn save_config_main( async fn save_config_sources( axum::extract::State(app_state): axum::extract::State>, headers: HeaderMap, - axum::extract::Json(mut sources): axum::extract::Json, + axum::extract::Json(sources): axum::extract::Json, ) -> impl axum::response::IntoResponse + Send { let sources_file_path = { let paths = app_state.app_config.paths.load(); @@ -294,27 +189,6 @@ async fn save_config_sources( return response; } - let existing_sources_dto = - match utils::read_sources_file_from_path(Path::new(&sources_file_path), false, false, None).await { - Ok(existing) => Some(existing), - Err(err) => { - warn!("Failed to preload existing source.yml from '{sources_file_path}' before save: {err}"); - let runtime_sources = app_state.app_config.sources.load(); - let runtime_fallback = build_existing_sources_from_runtime(runtime_sources.as_ref()); - if runtime_fallback.is_some() { - warn!("Using runtime provider dns snapshot as fallback for preserving dns.resolved"); - } - runtime_fallback - } - }; - - // `dns.resolved` is runtime-managed and must not be accepted from API input, - // but existing runtime values should survive UI saves until the next DNS refresh updates them. - strip_provider_dns_resolved(&mut sources); - if let Some(existing_sources_dto) = existing_sources_dto.as_ref() { - merge_provider_dns_resolved_from_existing(&mut sources, existing_sources_dto); - } - let templates_to_persist = match utils::validate_source_config_for_persist(&app_state, &sources).await { Ok(value) => value, Err(err) => { @@ -499,8 +373,6 @@ async fn config(axum::extract::State(app_state): axum::extract::State>) -> axum::Router().expect("ip parse should work")], - ); - let runtime_sources = SourcesConfig { - provider: vec![runtime_provider], - ..SourcesConfig::default() - }; - - inject_provider_dns_resolved(&mut dto, &runtime_sources); - - let resolved = dto.provider.as_ref() - .and_then(|providers| providers.first()) - .and_then(|provider| provider.dns.as_ref()) - .and_then(|dns| dns.resolved.as_ref()) - .expect("resolved dns snapshot should be present"); - assert_eq!( - resolved.get("example.com"), - Some(&vec!["203.0.113.10".parse::().expect("ip parse should work")]) - ); - } - - #[test] - fn inject_provider_dns_resolved_clears_value_when_runtime_dns_disabled() { - let mut dto = SourcesConfigDto { - provider: Some(vec![ConfigProviderDto { - name: "p1".into(), - urls: vec!["http://example.com".into()], - dns: Some(ProviderDnsDto { - enabled: true, - resolved: Some(IndexMap::from([( - "example.com".to_string(), - vec!["203.0.113.10".parse::().expect("ip parse should work")], - )])), - ..ProviderDnsDto::default() - }), - }]), - ..SourcesConfigDto::default() - }; - - let runtime_provider = Arc::new(ConfigProvider::from(&ConfigProviderDto { - name: "p1".into(), - urls: vec!["http://example.com".into()], - dns: None, - })); - let runtime_sources = SourcesConfig { - provider: vec![runtime_provider], - ..SourcesConfig::default() - }; - - inject_provider_dns_resolved(&mut dto, &runtime_sources); - - let resolved = dto.provider.as_ref() - .and_then(|providers| providers.first()) - .and_then(|provider| provider.dns.as_ref()) - .and_then(|dns| dns.resolved.as_ref()); - assert!(resolved.is_none(), "resolved output must be empty when runtime dns is disabled"); - } - - #[test] - fn strip_provider_dns_resolved_removes_payload_values() { - let mut dto = SourcesConfigDto { - provider: Some(vec![ConfigProviderDto { - name: "p1".into(), - urls: vec!["http://example.com".into()], - dns: Some(ProviderDnsDto { - enabled: true, - resolved: Some(IndexMap::from([( - "example.com".to_string(), - vec!["203.0.113.10".parse::().expect("ip parse should work")], - )])), - ..ProviderDnsDto::default() - }), - }]), - ..SourcesConfigDto::default() - }; - - strip_provider_dns_resolved(&mut dto); - - let resolved = dto - .provider - .as_ref() - .and_then(|providers| providers.first()) - .and_then(|provider| provider.dns.as_ref()) - .and_then(|dns| dns.resolved.as_ref()); - assert!(resolved.is_none(), "resolved must be stripped from incoming payload"); - } - - #[test] - fn merge_provider_dns_resolved_from_existing_preserves_runtime_snapshot() { - let mut incoming = SourcesConfigDto { - provider: Some(vec![ConfigProviderDto { - name: "p1".into(), - urls: vec!["http://example.com".into()], - dns: Some(ProviderDnsDto { - enabled: true, - ..ProviderDnsDto::default() - }), - }]), - ..SourcesConfigDto::default() - }; - let existing = SourcesConfigDto { - provider: Some(vec![ConfigProviderDto { - name: "p1".into(), - urls: vec!["http://example.com".into()], - dns: Some(ProviderDnsDto { - enabled: true, - resolved: Some(IndexMap::from([( - "example.com".to_string(), - vec!["203.0.113.10".parse::().expect("ip parse should work")], - )])), - ..ProviderDnsDto::default() - }), - }]), - ..SourcesConfigDto::default() - }; - - merge_provider_dns_resolved_from_existing(&mut incoming, &existing); - - let resolved = incoming - .provider - .as_ref() - .and_then(|providers| providers.first()) - .and_then(|provider| provider.dns.as_ref()) - .and_then(|dns| dns.resolved.as_ref()) - .expect("resolved should be merged from existing source dto"); - assert_eq!( - resolved.get("example.com"), - Some(&vec!["203.0.113.10".parse::().expect("ip parse should work")]) - ); - } - - #[test] - fn merge_provider_dns_resolved_from_existing_skips_disabled_dns() { - let mut incoming = SourcesConfigDto { - provider: Some(vec![ConfigProviderDto { - name: "p1".into(), - urls: vec!["http://example.com".into()], - dns: Some(ProviderDnsDto { - enabled: false, - ..ProviderDnsDto::default() - }), - }]), - ..SourcesConfigDto::default() - }; - let existing = SourcesConfigDto { - provider: Some(vec![ConfigProviderDto { - name: "p1".into(), - urls: vec!["http://example.com".into()], - dns: Some(ProviderDnsDto { - enabled: true, - resolved: Some(IndexMap::from([( - "example.com".to_string(), - vec!["203.0.113.10".parse::().expect("ip parse should work")], - )])), - ..ProviderDnsDto::default() - }), - }]), - ..SourcesConfigDto::default() - }; - - merge_provider_dns_resolved_from_existing(&mut incoming, &existing); - - let resolved = incoming - .provider - .as_ref() - .and_then(|providers| providers.first()) - .and_then(|provider| provider.dns.as_ref()) - .and_then(|dns| dns.resolved.as_ref()); - assert!(resolved.is_none(), "resolved should not be merged for disabled dns entries"); - } - - #[test] - fn build_existing_sources_from_runtime_exports_provider_dns_resolved() { - let runtime_provider = Arc::new(ConfigProvider::from(&ConfigProviderDto { - name: "p1".into(), - urls: vec!["http://example.com".into()], - dns: Some(ProviderDnsDto { - enabled: true, - schemes: Some(vec![DnsScheme::Http]), - ..ProviderDnsDto::default() - }), - })); - runtime_provider.store_resolved( - "example.com", - vec!["203.0.113.10".parse::().expect("ip parse should work")], - ); - let runtime_sources = SourcesConfig { - provider: vec![runtime_provider], - ..SourcesConfig::default() - }; - - let exported = - build_existing_sources_from_runtime(&runtime_sources).expect("runtime fallback dto should exist"); - let resolved = exported - .provider - .as_ref() - .and_then(|providers| providers.first()) - .and_then(|provider| provider.dns.as_ref()) - .and_then(|dns| dns.resolved.as_ref()) - .expect("resolved should be present in runtime fallback dto"); - assert_eq!( - resolved.get("example.com"), - Some(&vec!["203.0.113.10".parse::().expect("ip parse should work")]) - ); - } #[test] fn require_matching_revision_rejects_missing_if_match_header() { diff --git a/backend/src/api/library_scan.rs b/backend/src/api/library_scan.rs index 5dfba5930..79874d698 100644 --- a/backend/src/api/library_scan.rs +++ b/backend/src/api/library_scan.rs @@ -10,7 +10,7 @@ use std::sync::Arc; pub(crate) struct LibraryScanTaskOptions { pub force_rescan: bool, pub message_prefix: &'static str, - pub working_dir: String, + pub storage_dir: String, } pub(crate) fn spawn_library_scan( @@ -21,11 +21,11 @@ pub(crate) fn spawn_library_scan( options: LibraryScanTaskOptions, permit: UpdateGuardPermit, ) { - let LibraryScanTaskOptions { force_rescan, message_prefix, working_dir } = options; + let LibraryScanTaskOptions { force_rescan, message_prefix, storage_dir } = options; let prefix = message_prefix.to_string(); tokio::spawn(async move { let _permit = permit; - let processor = LibraryProcessor::new(lib_config, metadata_update_config.as_ref(), client, &working_dir); + let processor = LibraryProcessor::new(lib_config, metadata_update_config.as_ref(), client, &storage_dir); match processor.scan(force_rescan).await { Ok(result) => { info!("{prefix}Library scan completed successfully"); diff --git a/backend/src/api/main_api.rs b/backend/src/api/main_api.rs index f6212f4dd..778a64ead 100644 --- a/backend/src/api/main_api.rs +++ b/backend/src/api/main_api.rs @@ -204,7 +204,7 @@ async fn create_shared_data( let use_geoip = config.is_geoip_enabled(); let geoip = if use_geoip { - let path = get_geoip_path(&config.working_dir); + let path = get_geoip_path(&config.storage_dir); let _file_lock = app_config.file_locks.read_lock(&path).await; match GeoIp::load(&path) { Ok(db) => { diff --git a/backend/src/api/model/active_provider_manager.rs b/backend/src/api/model/active_provider_manager.rs index ab876d605..29c31937d 100644 --- a/backend/src/api/model/active_provider_manager.rs +++ b/backend/src/api/model/active_provider_manager.rs @@ -708,7 +708,9 @@ mod tests { api_proxy: Arc::new(ArcSwapOption::default()), file_locks: Arc::new(FileLockManager::default()), paths: Arc::new(ArcSwap::from_pointee(ConfigPaths { + home_path: String::new(), config_path: String::new(), + storage_path: String::new(), config_file_path: String::new(), sources_file_path: String::new(), mapping_file_path: None, diff --git a/backend/src/api/model/app_state.rs b/backend/src/api/model/app_state.rs index ad4a06f17..819542a74 100644 --- a/backend/src/api/model/app_state.rs +++ b/backend/src/api/model/app_state.rs @@ -266,7 +266,7 @@ fn build_http_client_with_fallback( pub fn create_cache(config: &Config) -> Option>> { let lru_cache = config.reverse_proxy.as_ref().and_then(|r| r.cache.as_ref()).and_then(|c| { if c.enabled { - Some(LRUResourceCache::new(c.size, c.dir.as_str())) + Some(LRUResourceCache::new(c.size, c.directory.as_str())) } else { None } @@ -341,20 +341,23 @@ pub struct AppState { impl AppState { pub(in crate::api::model) async fn set_config(&self, config: Config) -> Result { + let old_storage_dir = self.app_config.config.load().storage_dir.clone(); let changes = self.detect_changes_for_config(&config); config.update_runtime(); let use_geoip = config.is_geoip_enabled(); - let working_dir = config.working_dir.clone(); + let storage_dir = config.storage_dir.clone(); self.active_users.update_config(&config); self.app_config.set_config(config)?; self.active_provider.update_config(&self.app_config).await; self.update_config().await?; - if changes.flags.contains(UpdateChangesFlags::Geoip) { + let geoip_reload_needed = + changes.flags.contains(UpdateChangesFlags::Geoip) || (use_geoip && old_storage_dir != storage_dir); + if geoip_reload_needed { let new_geoip = if use_geoip { - let path = get_geoip_path(&working_dir); + let path = get_geoip_path(&storage_dir); let _file_lock = self.app_config.file_locks.read_lock(&path).await; GeoIp::load(&path).ok().map(Arc::new) } else { @@ -381,7 +384,7 @@ impl AppState { .reverse_proxy .as_ref() .and_then(|r| r.cache.as_ref()) - .map_or((false, 0, ""), |c| (c.enabled, c.size, c.dir.as_str())); + .map_or((false, 0, ""), |c| (c.enabled, c.size, c.directory.as_str())); if let Some(cache) = self.cache.load().as_ref() { if enabled { @@ -401,6 +404,22 @@ impl AppState { sources: SourcesConfig, ) -> Result { let changes = self.detect_changes_for_sources(&sources); + // Carry over DNS caches from old providers so resolved IPs survive hot-reloads + // without waiting for the background resolver or the persisted-file seed. + { + let old_sources = self.app_config.sources.load(); + for new_provider in &sources.provider { + if let Some(old_provider) = old_sources.get_provider_by_name(&new_provider.name) { + if new_provider.get_dns_config().is_some_and(|cfg| cfg.enabled) { + for (host, ips) in old_provider.snapshot_resolved() { + if !ips.is_empty() && new_provider.dns_cache.ip_count(&host) == 0 { + new_provider.dns_cache.store_resolved(&host, ips); + } + } + } + } + } + } self.app_config.set_sources(sources)?; self.active_provider.update_config(&self.app_config).await; @@ -420,6 +439,9 @@ impl AppState { let old_config = self.app_config.config.load(); let changed_schedules = change_detect!(schedules_changed, old_config.schedules.as_ref(), config.schedules.as_ref()); + let library_enabled = config.library.as_ref().is_some_and(|library| library.enabled); + let old_library_enabled = old_config.library.as_ref().is_some_and(|library| library.enabled); + let changed_library_enabled = library_enabled != old_library_enabled; let changed_hdhomerun = change_detect!(hdhomerun_changed, old_config.hdhomerun.as_ref(), config.hdhomerun.as_ref()); let changed_file_watch = @@ -430,7 +452,7 @@ impl AppState { let geoip_enabled_old = old_config.is_geoip_enabled(); let mut changes = UpdateChanges { flags: UpdateChangesFlagsSet::new(), targets: None }; - changes.set_flag_if(changed_schedules, UpdateChangesFlags::Scheduler); + changes.set_flag_if(changed_schedules || changed_library_enabled || geoip_enabled != geoip_enabled_old, UpdateChangesFlags::Scheduler); changes.set_flag_if(changed_hdhomerun, UpdateChangesFlags::Hdhomerun); changes.set_flag_if(changed_file_watch, UpdateChangesFlags::FileWatch); changes.set_flag_if(geoip_enabled != geoip_enabled_old, UpdateChangesFlags::Geoip); @@ -582,19 +604,30 @@ fn schedules_changed(a: &[ScheduleConfig], b: &[ScheduleConfig]) -> bool { if a.len() != b.len() { return true; } + let mut used = vec![false; b.len()]; + for schedule in a { - let Some(found) = b.iter().find(|&s| s.schedule == schedule.schedule) else { + let Some(found_idx) = b.iter().enumerate().find_map(|(idx, candidate)| { + if used[idx] + || candidate.schedule != schedule.schedule + || candidate.task_type != schedule.task_type + { + return None; + } + let targets_match = match (schedule.targets.as_ref(), candidate.targets.as_ref()) { + (None, None) => true, + (Some(_), None) | (None, Some(_)) => false, + (Some(a_targets), Some(b_targets)) => small_vecs_equal_unordered(a_targets, b_targets), + }; + if targets_match { + Some(idx) + } else { + None + } + }) else { return true; }; - match (schedule.targets.as_ref(), found.targets.as_ref()) { - (None, None) => {} - (Some(_), None) | (None, Some(_)) => return true, - (Some(a_targets), Some(b_targets)) => { - if !small_vecs_equal_unordered(a_targets, b_targets) { - return true; - } - } - } + used[found_idx] = true; } false } @@ -635,7 +668,9 @@ pub struct HdHomerunAppState { #[cfg(test)] mod tests { - use super::{should_use_manual_redirect_for_proxy, should_use_manual_redirects_for_env_vars}; + use super::{schedules_changed, should_use_manual_redirect_for_proxy, should_use_manual_redirects_for_env_vars}; + use crate::model::ScheduleConfig; + use shared::model::ScheduleTaskType; #[test] fn should_use_manual_redirect_for_proxy_only_http_or_https() { @@ -672,4 +707,48 @@ mod tests { "http://localhost".to_string(), )])); } + + #[test] + fn schedules_changed_detects_task_type_changes() { + let a = vec![ScheduleConfig { + schedule: "0 0 3 * * * *".to_string(), + task_type: ScheduleTaskType::PlaylistUpdate, + targets: None, + }]; + let b = vec![ScheduleConfig { + schedule: "0 0 3 * * * *".to_string(), + task_type: ScheduleTaskType::GeoIpUpdate, + targets: None, + }]; + assert!(schedules_changed(&a, &b)); + } + + #[test] + fn schedules_changed_treats_same_entries_as_unchanged() { + let a = vec![ + ScheduleConfig { + schedule: "0 0 3 * * * *".to_string(), + task_type: ScheduleTaskType::GeoIpUpdate, + targets: None, + }, + ScheduleConfig { + schedule: "0 0 8 * * * *".to_string(), + task_type: ScheduleTaskType::PlaylistUpdate, + targets: Some(vec!["a".to_string(), "b".to_string()]), + }, + ]; + let b = vec![ + ScheduleConfig { + schedule: "0 0 8 * * * *".to_string(), + task_type: ScheduleTaskType::PlaylistUpdate, + targets: Some(vec!["b".to_string(), "a".to_string()]), + }, + ScheduleConfig { + schedule: "0 0 3 * * * *".to_string(), + task_type: ScheduleTaskType::GeoIpUpdate, + targets: None, + }, + ]; + assert!(!schedules_changed(&a, &b)); + } } diff --git a/backend/src/api/model/metadata_update_manager.rs b/backend/src/api/model/metadata_update_manager.rs index ad7183703..45ebf6749 100644 --- a/backend/src/api/model/metadata_update_manager.rs +++ b/backend/src/api/model/metadata_update_manager.rs @@ -596,6 +596,7 @@ impl MetadataUpdateManager { /// * `task` - The task to process #[allow(clippy::too_many_lines)] pub async fn queue_task(&self, input_name: Arc, task: UpdateTask) { + debug!("[Task] Queuing task for input {input_name}: {task}"); // Read app state once and reuse for worker creation when needed. let app_state_weak = { let guard = self.app_state.lock().await; @@ -747,8 +748,12 @@ impl MetadataUpdateManager { } } else { let mut existing = entry.task.lock(); + let before = format!("{existing}"); if Self::merge_task_payload(&mut existing, task_to_submit) { entry.generation.fetch_add(1, Ordering::Relaxed); + debug!("[Task] Merged task for input {input_name}: before={before}, after={existing}"); + } else { + debug!("[Task] Task already pending for input {input_name} (no merge needed): {existing}"); } return SubmitTaskResult::QueuedOrMerged; } @@ -987,8 +992,8 @@ impl InputWorker { let mut task_for_execution = current_task.clone(); if Self::is_resolve_task(&task_for_execution) && self.resolve_exhausted.contains_key(¤t_key) { - debug_if_enabled!( - "Skipping exhausted resolve task for input {}: {} (reset window: {}s)", + debug!( + "[Metadata-Task] Skipping task (resolve exhausted) for input {}: {} (reset window: {}s)", input_name, task_for_execution, runtime_settings.resolve_exhaustion_reset_gap_secs @@ -1003,10 +1008,8 @@ impl InputWorker { if !skip_execution && self.should_skip_recent_no_change_task(¤t_key, &task_for_execution, &runtime_settings) { - debug_if_enabled!( - "Skipping recently-resolved no-change task for input {}: {}", - input_name, - task_for_execution + debug!( + "[Metadata-Task] Skipping task (recently completed with no changes) for input {input_name}: {task_for_execution}", ); remove_current_task = true; skip_execution = true; @@ -1019,19 +1022,21 @@ impl InputWorker { if let Some(cooldown_until_ts) = tmdb_state.cooldown_until_ts { if now_ts < cooldown_until_ts { if let Some(stripped_task) = Self::strip_tmdb_reasons(&task_for_execution) { - debug_if_enabled!( - "TMDB cooldown active for input {}: {} (cooldown_until={}), continuing with non-TMDB reasons", + debug!( + "[Task] TMDB cooldown active for input {}: {}, continuing with non-TMDB reasons (cooldown_until={}, remaining={}s)", input_name, task_for_execution, - cooldown_until_ts + cooldown_until_ts, + cooldown_until_ts.saturating_sub(now_ts) ); task_for_execution = stripped_task; } else { - debug_if_enabled!( - "Skipping TMDB-only task in cooldown for input {}: {} (cooldown_until={})", + debug!( + "[Metadata-Task] Skipping task (TMDB-only in cooldown) for input {}: {} (cooldown_until={}, remaining={}s)", input_name, task_for_execution, - cooldown_until_ts + cooldown_until_ts, + cooldown_until_ts.saturating_sub(now_ts) ); self.scheduled_requeues.remove(¤t_key); remove_current_task = true; @@ -1053,11 +1058,12 @@ impl InputWorker { if active_retry_domain == RetryDomain::Probe { if let Some(cooldown_until_ts) = active_state.cooldown_until_ts { if now_ts < cooldown_until_ts { - debug_if_enabled!( - "Skipping probe task in cooldown for input {}: {} (cooldown_until={})", + debug!( + "[Metadata-Task] Skipping task (probe cooldown) for input {}: {} (cooldown_until={}, remaining={}s)", input_name, task_for_execution, - cooldown_until_ts + cooldown_until_ts, + cooldown_until_ts.saturating_sub(now_ts) ); self.scheduled_requeues.remove(¤t_key); remove_current_task = true; @@ -1069,6 +1075,14 @@ impl InputWorker { } if !skip_execution && active_state.next_allowed_at_ts > now_ts { + debug!( + "[Task] Deferring task (retry backoff) for input {}: {} (next_allowed_at={}, wait={}s, attempts={})", + input_name, + task_for_execution, + active_state.next_allowed_at_ts, + active_state.next_allowed_at_ts.saturating_sub(now_ts), + active_state.attempts + ); schedule_requeue_at_ts = Some(active_state.next_allowed_at_ts); skip_execution = true; } @@ -1103,6 +1117,12 @@ impl InputWorker { } if !skip_execution { + debug!( + "[Task] Executing task for input {}: {} (retry_domain={:?})", + input_name, + task_for_execution, + Self::retry_domain_for_task(&task_for_execution) + ); let task_result = { let Some(_pause_guard) = self.wait_for_update_pause_window().await else { break; @@ -1129,7 +1149,7 @@ impl InputWorker { Self::should_trigger_playlist_update_for_task(&task_for_execution, task_outcome.task_changed); cycle_had_changes |= trigger_playlist_update; debug!( - "Processed metadata task for input {input_name}: {task_for_execution} (changed={}, trigger_playlist_update={}, tmdb_pending={})", + "[Metadata-Task] Task succeeded for input {input_name}: {task_for_execution} (changed={}, trigger_playlist_update={}, tmdb_pending={})", task_outcome.task_changed, trigger_playlist_update, task_outcome.tmdb_pending @@ -1151,11 +1171,12 @@ impl InputWorker { tmdb_state.cooldown_until_ts = Some(cooldown_until_ts); tmdb_state.last_error = Some("TMDB lookup completed without matching result".to_string()); - debug_if_enabled!( - "TMDB resolve produced no match for input {}: {} (cooldown_until={})", + debug!( + "[Metadata-Task] TMDB resolve produced no match (existing retry state), entering cooldown for input {}: {} (cooldown_until={}, cooldown_duration={}s)", input_name, task_for_execution, - cooldown_until_ts + cooldown_until_ts, + runtime_settings.tmdb_cooldown_secs ); } else { state_bundle.clear_domain(RetryDomain::Tmdb); @@ -1183,11 +1204,12 @@ impl InputWorker { }; self.retry_states.insert(current_key.clone(), state_bundle.clone()); state_after_success = Some(state_bundle); - debug_if_enabled!( - "TMDB resolve produced no match for input {}: {} (cooldown_until={})", + debug!( + "[Metadata-Task] TMDB resolve produced no match (new retry state), entering cooldown for input {}: {} (cooldown_until={}, cooldown_duration={}s)", input_name, task_for_execution, - cooldown_until_ts + cooldown_until_ts, + runtime_settings.tmdb_cooldown_secs ); } @@ -1207,6 +1229,10 @@ impl InputWorker { && !task_outcome.tmdb_pending { let reasons = Self::task_reason(&task_for_execution); + debug!( + "[Task] Caching no-change result for input {}: {} (reasons={}, ttl={}s)", + input_name, task_for_execution, reasons, runtime_settings.no_change_cache_ttl_secs + ); self.recently_completed_no_change.insert(current_key.clone(), (Instant::now(), reasons)); } else { self.recently_completed_no_change.remove(¤t_key); @@ -1237,6 +1263,10 @@ impl InputWorker { } Err(e) => { if Self::is_permanent_not_found_error(&e.message) { + debug!( + "[Task] Task failed with permanent not-found for input {}: {} (error={})", + input_name, task_for_execution, e.message + ); let retry_domain = Self::retry_domain_for_task(&task_for_execution); self.scheduled_requeues.remove(¤t_key); if retry_domain == RetryDomain::Probe { @@ -1253,11 +1283,12 @@ impl InputWorker { }; metadata_persist_state = Some(Some(state_bundle_after_update)); remove_current_task = true; - debug_if_enabled!( - "Probe task entered cooldown after permanent not-found for input {}: {} (cooldown_until={})", + debug!( + "[Metadata-Task] Probe task entering cooldown after permanent not-found for input {}: {} (cooldown_until={}, cooldown_duration={}s)", input_name, task_for_execution, - cooldown_until_ts + cooldown_until_ts, + runtime_settings.probe_cooldown_secs ); } else { self.resolve_exhausted.insert(current_key.clone(), now_ts); @@ -1277,10 +1308,11 @@ impl InputWorker { } metadata_persist_state = Some(state_after_clear); remove_current_task = true; - debug_if_enabled!( - "Resolve task marked exhausted after permanent not-found for input {}: {}", + debug!( + "[Metadata-Task] Resolve task marked exhausted after permanent not-found for input {}: {} (error={})", input_name, - task_for_execution + task_for_execution, + e.message ); } } else if Self::is_transient_worker_error(&e.message) { @@ -1293,8 +1325,8 @@ impl InputWorker { Self::compute_retry_delay_secs(current_task.delay(), &runtime_settings); let retry_delay_i64 = i64::try_from(retry_delay_secs).unwrap_or(i64::MAX); schedule_requeue_at_ts = Some(now_ts.saturating_add(retry_delay_i64)); - debug_if_enabled!( - "Transient task deferral for input {}: {} (retry_in={}s, err={})", + debug!( + "[Metadata-Task] Task deferred (transient error) for input {}: {} (retry_in={}s, error={})", input_name, task_for_execution, retry_delay_secs, @@ -1350,12 +1382,14 @@ impl InputWorker { let cooldown_until = state_after_update .cooldown_until_ts .map_or_else(|| "none".to_string(), |ts| ts.to_string()); - debug_if_enabled!( - "Probe task exhausted for input {}: {} (attempts={}, cooldown_until={})", + debug!( + "Metadata-[Task] Probe task exhausted (max attempts reached) for input {}: {} (attempts={}/{}, cooldown_until={}, error={})", input_name, task_for_execution, state_after_update.attempts, - cooldown_until + max_attempts, + cooldown_until, + e.message ); } else { self.resolve_exhausted.insert(current_key.clone(), now_ts); @@ -1375,22 +1409,26 @@ impl InputWorker { } metadata_persist_state = Some(state_after_clear); remove_current_task = true; - debug_if_enabled!( - "Resolve task exhausted for input {}: {} (attempts={})", + debug!( + "[Metadata-Task] Resolve task exhausted (max attempts reached) for input {}: {} (attempts={}/{}, error={})", input_name, task_for_execution, - attempts + attempts, + max_attempts, + e.message ); } } else { schedule_requeue_at_ts = Some(state_after_update.next_allowed_at_ts); metadata_persist_state = Some(Some(state_bundle_after_update)); - debug_if_enabled!( - "Task failed for input {}, scheduling retry: {} (attempt={}, next_allowed_at={}, err={})", + debug!( + "[Metadata-Task] Task failed, scheduling retry for input {}: {} (attempt={}/{}, next_allowed_at={}, backoff={}s, error={})", input_name, task_for_execution, attempts, + max_attempts, state_after_update.next_allowed_at_ts, + state_after_update.next_allowed_at_ts.saturating_sub(now_ts), e.message ); } @@ -1413,6 +1451,13 @@ impl InputWorker { } if let Some(retry_at_ts) = schedule_requeue_at_ts { + debug!( + "[Task] Scheduling requeue for input {}: {:?} (retry_at_ts={}, in={}s)", + input_name, + current_key, + retry_at_ts, + retry_at_ts.saturating_sub(chrono::Utc::now().timestamp()) + ); self.schedule_requeue_at(current_key.clone(), retry_at_ts); } @@ -1510,8 +1555,8 @@ impl InputWorker { return; }; - let working_dir = app_state.app_config.config.load().working_dir.clone(); - let Ok(storage_path) = get_input_storage_path(input_name, &working_dir).await else { + let storage_dir = app_state.app_config.config.load().storage_dir.clone(); + let Ok(storage_path) = get_input_storage_path(input_name, &storage_dir).await else { warn!("Could not resolve storage path for metadata retry state on input {input_name}"); self.metadata_retry_load_retry_at_ts = Some(now_ts.saturating_add(runtime_settings.metadata_retry_load_retry_delay_secs)); @@ -2026,7 +2071,7 @@ impl InputWorker { let Some(app_state) = app_state_weak.and_then(Weak::upgrade) else { return }; let app_config = &app_state.app_config; - let working_dir = &app_config.config.load().working_dir; + let cfg = app_config.config.load(); let vod_updates = batch_buffer.take_vod_updates(); let series_updates = batch_buffer.take_series_updates(); let live_updates = batch_buffer.take_live_updates(); @@ -2035,7 +2080,7 @@ impl InputWorker { return; } - if let Ok(storage_path) = get_input_storage_path(input_name, working_dir).await { + if let Ok(storage_path) = get_input_storage_path(input_name, &cfg.storage_dir).await { if !vod_updates.is_empty() { let mut updates: Vec<(u32, VideoStreamProperties)> = Vec::with_capacity(vod_updates.len()); for (id, props) in &vod_updates { @@ -2531,8 +2576,8 @@ impl InputWorker { } if let std::collections::hash_map::Entry::Vacant(entry) = db_handles.entry(cluster) { - let working_dir = &app_state.app_config.config.load().working_dir; - if let Ok(storage_path) = get_input_storage_path(input_name, working_dir).await { + let cfg = app_state.app_config.config.load(); + if let Ok(storage_path) = get_input_storage_path(input_name, &cfg.storage_dir).await { let file_path = xtream_get_file_path(&storage_path, cluster); if file_path.exists() { let lock = app_state.app_config.file_locks.read_lock(&file_path).await; diff --git a/backend/src/api/model/provider_dns_manager.rs b/backend/src/api/model/provider_dns_manager.rs index 603e7409b..c988d8e65 100644 --- a/backend/src/api/model/provider_dns_manager.rs +++ b/backend/src/api/model/provider_dns_manager.rs @@ -1,15 +1,15 @@ use crate::api::model::AppState; use crate::model::ConfigProvider; -use crate::utils::read_sources_file_from_path; +use crate::repository::{ + load_persisted_dns_resolved, next_dns_writer_generation, prune_persisted_dns_resolved_to_runtime, + queue_provider_resolved_snapshot, spawn_dns_resolved_writer, DnsResolvedWriteTx, +}; use log::{debug, warn}; -use shared::model::{DnsPrefer, OnResolveErrorPolicy, SourcesConfigDto}; +use shared::model::{DnsPrefer, OnResolveErrorPolicy}; use std::collections::HashSet; -use std::io; use std::net::IpAddr; -use std::path::{Path, PathBuf}; use std::sync::Arc; use std::time::{Duration, Instant}; -use tokio::fs; use tokio::net::lookup_host; use tokio_util::sync::CancellationToken; @@ -105,129 +105,12 @@ async fn resolve_provider(provider: &Arc) -> ProviderResolveStat stats } -fn serialize_sources_for_persist(sources: &SourcesConfigDto) -> io::Result { - let mut serialized = String::new(); - let options = serde_saphyr::SerializerOptions { - prefer_block_scalars: false, - ..Default::default() - }; - serde_saphyr::to_fmt_writer_with_options(&mut serialized, sources, options) - .map_err(|err| io::Error::other(format!("Could not serialize source.yml: {err}")))?; - Ok(serialized) -} - -async fn write_sources_file_force(path: &Path, sources: &SourcesConfigDto) -> io::Result<()> { - let serialized = serialize_sources_for_persist(sources)?; - let parent_dir = path.parent().ok_or_else(|| { - io::Error::other(format!( - "Could not write source.yml '{}': missing parent directory", - path.display() - )) - })?; - let dest_file_name = path.file_name().and_then(|s| s.to_str()).unwrap_or("source.yml"); - let mut tmp_path = parent_dir.to_path_buf(); - tmp_path.push(format!( - ".{dest_file_name}.tmp-{}-{}", - std::process::id(), - chrono::Local::now().timestamp_nanos_opt().unwrap_or_default() - )); - - fs::write(&tmp_path, serialized).await?; - match fs::rename(&tmp_path, path).await { - Ok(()) => Ok(()), - Err(err) => { - #[cfg(windows)] - { - // Try to rename again after removing destination (Windows often needs this) - if let Ok(()) = fs::remove_file(path).await { - if fs::rename(&tmp_path, path).await.is_ok() { - return Ok(()); - } - // Rename still failed after removing dest - fall through to clean up - } - } - let _ = fs::remove_file(&tmp_path).await; - Err(io::Error::other(format!( - "Could not replace '{}' with '{}': {err}", - path.display(), - tmp_path.display() - ))) - } - } -} - -async fn persist_provider_resolved_to_source_file(app_state: &Arc, provider: &Arc) { - let source_file = { - let paths = app_state.app_config.paths.load(); - paths.sources_file_path.clone() - }; - let source_path = PathBuf::from(&source_file); - let _lock = app_state.app_config.file_locks.write_lock(&source_path).await; - - let mut sources_dto = match read_sources_file_from_path(&source_path, false, false, None).await { - Ok(dto) => dto, - Err(err) => { - warn!( - "Provider dns '{}' failed to read source.yml '{}': {err}", - provider.name, - source_path.display() - ); - return; - } - }; - - let Some(provider_dtos) = sources_dto.provider.as_mut() else { - debug!( - "Provider dns '{}' source.yml '{}' has no provider section to persist resolved values", - provider.name, - source_path.display() - ); - return; - }; - - let Some(provider_dto) = provider_dtos.iter_mut().find(|dto| dto.name.as_ref() == provider.name.as_ref()) else { - warn!( - "Provider dns '{}' not found in source.yml '{}', cannot persist resolved values", - provider.name, - source_path.display() - ); - return; - }; - - let Some(dns_dto) = provider_dto.dns.as_mut() else { - debug!( - "Provider dns '{}' has no dns section in source.yml '{}', skipping resolved persist", - provider.name, - source_path.display() - ); - return; - }; - - let resolved_hosts = { - let snapshot = provider.snapshot_resolved_ordered(); - dns_dto.resolved = (!snapshot.is_empty()).then_some(snapshot); - dns_dto.resolved.as_ref().map_or(0, indexmap::IndexMap::len) - }; - - match write_sources_file_force(&source_path, &sources_dto).await { - Ok(()) => { - debug!( - "Provider dns '{}' persisted dns.resolved to '{}' (hosts={resolved_hosts})", - provider.name, - source_path.display() - ); - } - Err(err) => { - warn!( - "Provider dns '{}' failed to persist dns.resolved to '{}': {err}", - provider.name, - source_path.display() - ); - } - } -} - -fn spawn_provider_dns_task(app_state: Arc, provider_name: Arc, cancel: CancellationToken) { +fn spawn_provider_dns_task( + app_state: Arc, + provider_name: Arc, + cancel: CancellationToken, + writer_tx: DnsResolvedWriteTx, +) { tokio::spawn(async move { let mut refresh_secs = 300_u64; { @@ -260,7 +143,7 @@ fn spawn_provider_dns_task(app_state: Arc, provider_name: Arc, ca }; refresh_secs = provider.get_dns_config().map_or(300, |dns| dns.refresh_secs.max(10)); let stats = resolve_provider(&provider).await; - persist_provider_resolved_to_source_file(&app_state, &provider).await; + queue_provider_resolved_snapshot(&writer_tx, &provider).await; let cache_hosts = provider.snapshot_resolved().len(); debug!( "Provider dns tick '{}' finished: total_hosts={} resolved={} overridden={} empty={} failed={} cache_hosts={} elapsed_ms={}", @@ -282,6 +165,7 @@ fn spawn_provider_dns_task(app_state: Arc, provider_name: Arc, ca } pub fn exec_provider_dns(app_state: &Arc, cancel: &CancellationToken) { + let generation = next_dns_writer_generation(); let sources = app_state.app_config.sources.load(); let provider_names: Vec<_> = sources .provider @@ -292,13 +176,33 @@ pub fn exec_provider_dns(app_state: &Arc, cancel: &CancellationToken) drop(sources); if provider_names.is_empty() { + let app_for_prune = Arc::clone(app_state); + tokio::spawn(async move { + prune_persisted_dns_resolved_to_runtime(&app_for_prune).await; + }); debug!("Provider dns manager: no enabled providers found"); return; } - debug!("Provider dns manager: starting {} provider task(s)", provider_names.len()); + let app_for_start = Arc::clone(app_state); + let cancel_for_start = cancel.clone(); + tokio::spawn(async move { + // Seed runtime caches first, then start writer/tasks to avoid startup races. + load_persisted_dns_resolved(&app_for_start).await; + if cancel_for_start.is_cancelled() { + return; + } - for provider_name in provider_names { - spawn_provider_dns_task(Arc::clone(app_state), provider_name, cancel.clone()); - } + let writer_tx = spawn_dns_resolved_writer(Arc::clone(&app_for_start), cancel_for_start.clone(), generation); + debug!("Provider dns manager: starting {} provider task(s)", provider_names.len()); + + for provider_name in provider_names { + spawn_provider_dns_task( + Arc::clone(&app_for_start), + provider_name, + cancel_for_start.clone(), + writer_tx.clone(), + ); + } + }); } diff --git a/backend/src/api/model/streams/provider_stream_factory.rs b/backend/src/api/model/streams/provider_stream_factory.rs index 10904db78..945a23db2 100644 --- a/backend/src/api/model/streams/provider_stream_factory.rs +++ b/backend/src/api/model/streams/provider_stream_factory.rs @@ -25,7 +25,6 @@ use reqwest::{ }; use shared::{ create_bitset, - model::{PlaylistItemType, DEFAULT_USER_AGENT}, utils::{filter_request_header, sanitize_sensitive_info}, }; use std::{ @@ -38,6 +37,8 @@ use std::{ time::{Duration, Instant}, }; use url::Url; +use shared::model::PlaylistItemType; +use shared::utils::DEFAULT_USER_AGENT; const RETRY_SECONDS: u64 = 5; const ERR_MAX_RETRY_COUNT: u32 = 5; diff --git a/backend/src/api/scheduler.rs b/backend/src/api/scheduler.rs index 1f07f4746..b2e600016 100644 --- a/backend/src/api/scheduler.rs +++ b/backend/src/api/scheduler.rs @@ -1,6 +1,10 @@ use crate::{ - api::{library_scan::{spawn_library_scan, LibraryScanTaskOptions}, model::AppState}, + api::{ + library_scan::{spawn_library_scan, LibraryScanTaskOptions}, + model::AppState, + }, model::{AppConfig, ProcessTargets, ScheduleConfig}, + processing::geoip::{update_geoip_db, GeoIpUpdateError}, processing::processor::exec_processing, utils::exit, }; @@ -39,6 +43,24 @@ pub fn exec_scheduler( let schedules: Vec = if let Some(schedules) = &config.schedules { schedules.clone() } else { vec![] }; for schedule in schedules { + let task_enabled = match schedule.task_type { + ScheduleTaskType::PlaylistUpdate => true, + ScheduleTaskType::LibraryScan => config.library.as_ref().is_some_and(|library| library.enabled), + ScheduleTaskType::GeoIpUpdate => config + .reverse_proxy + .as_ref() + .and_then(|reverse_proxy| reverse_proxy.geoip.as_ref()) + .is_some_and(|geoip| geoip.enabled), + }; + if !task_enabled { + log::info!( + "Skipping disabled scheduled task {:?} ({})", + schedule.task_type, + schedule.schedule + ); + continue; + } + let expression = schedule.schedule.clone(); let task_type = schedule.task_type; let exec_targets = get_process_targets(cfg, targets, schedule.targets.as_ref()); @@ -75,6 +97,9 @@ async fn start_scheduler( ScheduleTaskType::LibraryScan => { run_library_scan(&client, &app_state); } + ScheduleTaskType::GeoIpUpdate => { + run_geoip_update(&app_state); + } } } () = cancel.cancelled() => { @@ -131,7 +156,7 @@ fn run_library_scan(client: &reqwest::Client, app_state: &Arc) { LibraryScanTaskOptions { force_rescan: false, message_prefix: "Scheduled ", - working_dir: config.working_dir.clone(), + storage_dir: config.storage_dir.clone(), }, permit, ); @@ -140,6 +165,17 @@ fn run_library_scan(client: &reqwest::Client, app_state: &Arc) { } } +fn run_geoip_update(app_state: &Arc) { + let app_state = Arc::clone(app_state); + tokio::spawn(async move { + if let Err(err) = update_geoip_db(&app_state).await { + if !matches!(err, GeoIpUpdateError::Disabled) { + log::error!("Scheduled GeoIp update failed: {err}"); + } + } + }); +} + pub fn get_process_targets( cfg: &Arc, process_targets: &Arc, diff --git a/backend/src/api/setup_api.rs b/backend/src/api/setup_api.rs index 51015c945..5902c90a1 100644 --- a/backend/src/api/setup_api.rs +++ b/backend/src/api/setup_api.rs @@ -3,8 +3,8 @@ use crate::{ auth::generate_password_from_input, model::validate_library_paths_from_dto, utils::{ - file_exists, get_default_path, get_default_web_root_path, read_api_proxy_file, read_config_file, - read_sources_file, read_templates_file, resolve_template_persist_file_path, sanitize_sources_for_persist, + file_exists, get_default_web_root_path_for_home, read_api_proxy_file, read_config_file, read_sources_file, + read_templates_file, resolve_template_persist_file_path, sanitize_sources_for_persist, }, }; use axum::{ @@ -26,10 +26,10 @@ use shared::{ ApiProxyConfigDto, ApiProxyServerInfoDto, AppConfigDto, ConfigApiDto, ConfigDto, ConfigPaths, PatternTemplate, SourcesConfigDto, TemplateDefinitionDto, TokenResponse, WebAuthConfigDto, WebUiConfigDto, TOKEN_NO_AUTH, }, - utils::{default_kick_secs, hex_encode, DEFAULT_PORT, DEFAULT_WORKING_DIR, USER_FILE}, + utils::{default_kick_secs, hex_encode, DEFAULT_PORT, DEFAULT_STORAGE_DIR, USER_FILE}, }; use std::{ - collections::{HashMap, HashSet}, + collections::HashSet, io::ErrorKind, net::{SocketAddr, UdpSocket}, path::{Component, Path as FsPath, PathBuf}, @@ -40,10 +40,9 @@ use std::{ }; use tokio::sync::{oneshot, Mutex, RwLock}; use tower_http::services::ServeDir; +use shared::utils::DEFAULT_CUSTOM_STREAM_RESPONSE_PATH; const DEFAULT_SETUP_HOST: &str = "0.0.0.0"; -const DEFAULT_SETUP_CUSTOM_STREAM_RESPONSE_PATH: &str = "./resources"; -const SETUP_REDACTED_SECRET_VALUE: &str = "__TULIPROX_SETUP_REDACTED__"; #[derive(Clone, Serialize, Deserialize)] pub struct SetupWebUserCredentialDto { @@ -69,6 +68,7 @@ pub struct SetupCompleteRequestDto { struct SetupModeState { draft: Arc>, + home_path: String, output_dir: PathBuf, config_file_path: PathBuf, source_file_path: PathBuf, @@ -166,7 +166,7 @@ fn create_default_api_proxy_server() -> ApiProxyServerInfoDto { } } -fn create_default_config_dto() -> ConfigDto { +fn create_default_config_dto(home_path: &str) -> ConfigDto { let auth = WebAuthConfigDto { issuer: "tuliprox".to_string(), secret: generate_web_auth_secret(), @@ -180,18 +180,20 @@ fn create_default_config_dto() -> ConfigDto { api: ConfigApiDto { host: DEFAULT_SETUP_HOST.to_string(), port: DEFAULT_PORT, - web_root: get_default_web_root_path().display().to_string(), + web_root: get_default_web_root_path_for_home(FsPath::new(home_path)) + .display() + .to_string(), }, - working_dir: get_default_path(DEFAULT_WORKING_DIR).display().to_string(), - custom_stream_response_path: Some(DEFAULT_SETUP_CUSTOM_STREAM_RESPONSE_PATH.to_string()), + storage_dir: Some(DEFAULT_STORAGE_DIR.to_string()), + custom_stream_response_path: Some(DEFAULT_CUSTOM_STREAM_RESPONSE_PATH.to_string()), web_ui: Some(web_ui), ..ConfigDto::default() } } -fn create_default_draft() -> AppConfigDto { +fn create_default_draft(home_path: &str) -> AppConfigDto { AppConfigDto { - config: create_default_config_dto(), + config: create_default_config_dto(home_path), sources: SourcesConfigDto::default(), mappings: None, templates: None, @@ -204,7 +206,7 @@ fn create_default_draft() -> AppConfigDto { } async fn build_initial_draft(paths: &ConfigPaths) -> AppConfigDto { - let mut draft = create_default_draft(); + let mut draft = create_default_draft(paths.home_path.as_str()); if file_exists(&paths.config_file_path) { match read_config_file(paths.config_file_path.as_str(), true, false) { @@ -251,7 +253,7 @@ fn generate_web_auth_secret() -> String { hex_encode(&secret).to_lowercase() } -fn ensure_setup_defaults(config: &mut ConfigDto) { +fn ensure_setup_defaults(config: &mut ConfigDto, home_path: &str) { if config.api.host.trim().is_empty() { config.api.host = DEFAULT_SETUP_HOST.to_string(); } @@ -259,13 +261,15 @@ fn ensure_setup_defaults(config: &mut ConfigDto) { config.api.port = DEFAULT_PORT; } if config.api.web_root.trim().is_empty() { - config.api.web_root = get_default_web_root_path().display().to_string(); + config.api.web_root = get_default_web_root_path_for_home(FsPath::new(home_path)) + .display() + .to_string(); } - if config.working_dir.trim().is_empty() { - config.working_dir = get_default_path(DEFAULT_WORKING_DIR).display().to_string(); + if config.storage_dir.as_ref().is_none_or(|dir| dir.trim().is_empty()) { + config.storage_dir = Some(DEFAULT_STORAGE_DIR.to_string()); } if config.custom_stream_response_path.as_ref().is_none_or(|path| path.trim().is_empty()) { - config.custom_stream_response_path = Some(DEFAULT_SETUP_CUSTOM_STREAM_RESPONSE_PATH.to_string()); + config.custom_stream_response_path = Some(DEFAULT_CUSTOM_STREAM_RESPONSE_PATH.to_string()); } if config.web_ui.is_none() { @@ -288,7 +292,7 @@ fn ensure_setup_defaults(config: &mut ConfigDto) { } } -fn setup_bind_values(draft: &AppConfigDto) -> (String, u16, PathBuf) { +fn setup_bind_values(draft: &AppConfigDto, home_path: &str) -> (String, u16, PathBuf) { let host = if draft.config.api.host.trim().is_empty() { DEFAULT_SETUP_HOST.to_string() } else { @@ -296,18 +300,18 @@ fn setup_bind_values(draft: &AppConfigDto) -> (String, u16, PathBuf) { }; let port = if draft.config.api.port == 0 { DEFAULT_PORT } else { draft.config.api.port }; let web_root = if draft.config.api.web_root.trim().is_empty() { - get_default_web_root_path() + get_default_web_root_path_for_home(FsPath::new(home_path)) } else { PathBuf::from(&draft.config.api.web_root) }; (host, port, web_root) } -fn resolve_setup_web_dir(web_root: &FsPath) -> Option { +fn resolve_setup_web_dir(web_root: &FsPath, home_path: &str) -> Option { if web_root.exists() && web_root.is_dir() { return Some(web_root.to_path_buf()); } - let fallback = get_default_web_root_path(); + let fallback = get_default_web_root_path_for_home(FsPath::new(home_path)); if fallback.exists() && fallback.is_dir() { return Some(fallback); } @@ -322,126 +326,6 @@ fn api_proxy_or_default(draft: &AppConfigDto) -> ApiProxyConfigDto { }) } -fn is_setup_redacted_value(value: &str) -> bool { value == SETUP_REDACTED_SECRET_VALUE } - -fn redact_non_empty_secret(value: &mut String) { - if !value.trim().is_empty() { - *value = SETUP_REDACTED_SECRET_VALUE.to_string(); - } -} - -fn redact_optional_secret(value: &mut Option) { - if value.as_ref().is_some_and(|entry| !entry.trim().is_empty()) { - *value = Some(SETUP_REDACTED_SECRET_VALUE.to_string()); - } -} - -fn redact_api_proxy_user_credentials(api_proxy: &mut ApiProxyConfigDto) { - for target in &mut api_proxy.user { - for user in &mut target.credentials { - redact_non_empty_secret(&mut user.password); - redact_optional_secret(&mut user.token); - } - } -} - -fn redact_app_config_for_setup(mut app_config: AppConfigDto) -> AppConfigDto { - if let Some(web_ui) = app_config.config.web_ui.as_mut() { - if let Some(auth) = web_ui.auth.as_mut() { - redact_non_empty_secret(&mut auth.secret); - } - } - if let Some(api_proxy) = app_config.api_proxy.as_mut() { - redact_api_proxy_user_credentials(api_proxy); - } - app_config -} - -fn redact_api_proxy_for_setup(mut api_proxy: ApiProxyConfigDto) -> ApiProxyConfigDto { - redact_api_proxy_user_credentials(&mut api_proxy); - api_proxy -} - -fn restore_redacted_web_auth_secret(app_config: &mut AppConfigDto, draft: &AppConfigDto) { - let Some(web_ui) = app_config.config.web_ui.as_mut() else { - return; - }; - let Some(auth) = web_ui.auth.as_mut() else { - return; - }; - if !is_setup_redacted_value(&auth.secret) { - return; - } - - let draft_secret = draft - .config - .web_ui - .as_ref() - .and_then(|draft_web_ui| draft_web_ui.auth.as_ref()) - .map(|draft_auth| draft_auth.secret.trim().to_string()) - .filter(|secret| !secret.is_empty()); - if let Some(draft_secret) = draft_secret { - auth.secret = draft_secret; - } -} - -fn restore_redacted_api_proxy_credentials( - api_proxy: &mut ApiProxyConfigDto, - draft_api_proxy: Option<&ApiProxyConfigDto>, -) { - let mut credentials_by_username: HashMap)> = HashMap::new(); - if let Some(draft) = draft_api_proxy { - for target in &draft.user { - for user in &target.credentials { - credentials_by_username.insert(user.username.clone(), (user.password.clone(), user.token.clone())); - } - } - } - - for target in &mut api_proxy.user { - for user in &mut target.credentials { - let draft_credentials = credentials_by_username.get(&user.username); - if is_setup_redacted_value(&user.password) { - if let Some((password, _)) = draft_credentials { - user.password = password.clone(); - } - } - if user.token.as_deref().is_some_and(is_setup_redacted_value) { - if let Some((_, token)) = draft_credentials { - user.token = token.clone(); - } - } - } - } -} - -fn restore_redacted_setup_values(app_config: &mut AppConfigDto, draft: &AppConfigDto) { - restore_redacted_web_auth_secret(app_config, draft); - if let Some(api_proxy) = app_config.api_proxy.as_mut() { - restore_redacted_api_proxy_credentials(api_proxy, draft.api_proxy.as_ref()); - } -} - -fn has_unresolved_redacted_setup_values(app_config: &AppConfigDto) -> bool { - if app_config - .config - .web_ui - .as_ref() - .and_then(|web_ui| web_ui.auth.as_ref()) - .is_some_and(|auth| is_setup_redacted_value(&auth.secret)) - { - return true; - } - - app_config.api_proxy.as_ref().is_some_and(|api_proxy| { - api_proxy.user.iter().any(|target| { - target.credentials.iter().any(|user| { - is_setup_redacted_value(&user.password) || user.token.as_deref().is_some_and(is_setup_redacted_value) - }) - }) - }) -} - fn setup_templates_from_request(app_config: &AppConfigDto) -> Option> { app_config .templates @@ -518,12 +402,12 @@ async fn setup_token_refresh() -> impl IntoResponse + Send { async fn setup_get_config(State(state): State>) -> impl IntoResponse + Send { let draft = state.draft.read().await.clone(); - axum::Json(redact_app_config_for_setup(draft)).into_response() + axum::Json(draft).into_response() } async fn setup_get_api_proxy(State(state): State>) -> impl IntoResponse + Send { let draft = state.draft.read().await.clone(); - axum::Json(redact_api_proxy_for_setup(api_proxy_or_default(&draft))).into_response() + axum::Json(api_proxy_or_default(&draft)).into_response() } async fn setup_config_json(State(state): State>) -> impl IntoResponse + Send { @@ -751,19 +635,7 @@ async fn setup_complete_inner( state: Arc, mut req: SetupCompleteRequestDto, ) -> axum::response::Response { - let draft_snapshot = state.draft.read().await.clone(); - restore_redacted_setup_values(&mut req.app_config, &draft_snapshot); - if has_unresolved_redacted_setup_values(&req.app_config) { - return ( - StatusCode::BAD_REQUEST, - axum::Json(json!({ - "error": "One or more redacted secret fields could not be restored. Please re-enter secret values before saving setup." - })), - ) - .into_response(); - } - - ensure_setup_defaults(&mut req.app_config.config); + ensure_setup_defaults(&mut req.app_config.config, state.home_path.as_str()); if let Err(err) = req.app_config.config.prepare(false) { return (StatusCode::BAD_REQUEST, axum::Json(json!({ "error": err.to_string() }))).into_response(); @@ -983,13 +855,14 @@ fn create_compression_layer() -> tower_http::compression::CompressionLayer { pub async fn start_setup_server(paths: &ConfigPaths, missing_files: &[String]) -> Result<(), TuliproxError> { let draft = build_initial_draft(paths).await; - let (host, port, web_root) = setup_bind_values(&draft); - let web_dir = resolve_setup_web_dir(&web_root) + let (host, port, web_root) = setup_bind_values(&draft, paths.home_path.as_str()); + let web_dir = resolve_setup_web_dir(&web_root, paths.home_path.as_str()) .ok_or_else(|| info_err!("Setup mode requires a web directory. Tried '{}'", web_root.display(),))?; let (shutdown_tx, shutdown_rx) = oneshot::channel::<()>(); let state = Arc::new(SetupModeState { draft: Arc::new(RwLock::new(draft)), + home_path: paths.home_path.clone(), output_dir: PathBuf::from(&paths.config_path), config_file_path: PathBuf::from(&paths.config_file_path), source_file_path: PathBuf::from(&paths.sources_file_path), @@ -1001,7 +874,7 @@ pub async fn start_setup_server(paths: &ConfigPaths, missing_files: &[String]) - shutdown_tx: Mutex::new(Some(shutdown_tx)), }); - info!("Setup mode enabled. Missing required config files: {}", missing_files.join(", ")); + info!("Setup mode enabled."); info!("Setup output directory: {}", state.output_dir.display()); info!("Setup web root: {}", state.web_dir.display()); info!("Setup server running: http://{host}:{port}"); @@ -1048,25 +921,12 @@ pub async fn start_setup_server(paths: &ConfigPaths, missing_files: &[String]) - #[cfg(test)] mod tests { - use super::{ - has_unresolved_redacted_setup_values, redact_app_config_for_setup, restore_redacted_setup_values, - SETUP_REDACTED_SECRET_VALUE, - }; - use shared::model::{ApiProxyConfigDto, AppConfigDto, TargetUserDto, WebAuthConfigDto, WebUiConfigDto}; + use super::api_proxy_or_default; + use shared::model::{ApiProxyConfigDto, AppConfigDto, TargetUserDto}; - fn sample_app_config() -> AppConfigDto { - AppConfigDto { - config: shared::model::ConfigDto { - web_ui: Some(WebUiConfigDto { - auth: Some(WebAuthConfigDto { - issuer: "tuliprox".to_string(), - secret: "very-secret-value".to_string(), - ..Default::default() - }), - ..Default::default() - }), - ..Default::default() - }, + #[test] + fn setup_api_proxy_or_default_preserves_existing_credentials() { + let app_config = AppConfigDto { api_proxy: Some(ApiProxyConfigDto { user: vec![TargetUserDto { target: "target-a".to_string(), @@ -1080,49 +940,17 @@ mod tests { ..Default::default() }), ..Default::default() - } - } - - #[test] - fn setup_redaction_masks_web_auth_and_api_proxy_credentials() { - let redacted = redact_app_config_for_setup(sample_app_config()); - - let auth_secret = - redacted.config.web_ui.as_ref().and_then(|web_ui| web_ui.auth.as_ref()).map(|auth| auth.secret.as_str()); - assert_eq!(auth_secret, Some(SETUP_REDACTED_SECRET_VALUE)); - - let creds = &redacted.api_proxy.expect("api_proxy should be present").user[0].credentials[0]; - assert_eq!(creds.password, SETUP_REDACTED_SECRET_VALUE); - assert_eq!(creds.token.as_deref(), Some(SETUP_REDACTED_SECRET_VALUE)); - } - - #[test] - fn setup_restore_replaces_redacted_values_from_existing_draft() { - let draft = sample_app_config(); - let mut submitted = redact_app_config_for_setup(draft.clone()); - - restore_redacted_setup_values(&mut submitted, &draft); - - let auth_secret = - submitted.config.web_ui.as_ref().and_then(|web_ui| web_ui.auth.as_ref()).map(|auth| auth.secret.as_str()); - assert_eq!(auth_secret, Some("very-secret-value")); - - let creds = &submitted.api_proxy.as_ref().expect("api_proxy should be present").user[0].credentials[0]; + }; + let api_proxy = api_proxy_or_default(&app_config); + let creds = &api_proxy.user[0].credentials[0]; assert_eq!(creds.password, "alice-password"); assert_eq!(creds.token.as_deref(), Some("alice-token")); - assert!(!has_unresolved_redacted_setup_values(&submitted)); } #[test] - fn setup_restore_keeps_unmatched_redacted_credentials_as_unresolved() { - let draft = sample_app_config(); - let mut submitted = redact_app_config_for_setup(draft); - if let Some(api_proxy) = submitted.api_proxy.as_mut() { - api_proxy.user[0].credentials[0].username = "bob".to_string(); - } - - restore_redacted_setup_values(&mut submitted, &AppConfigDto::default()); - - assert!(has_unresolved_redacted_setup_values(&submitted)); + fn setup_api_proxy_or_default_creates_server_when_missing() { + let api_proxy = api_proxy_or_default(&AppConfigDto::default()); + assert_eq!(api_proxy.server.len(), 1); + assert!(!api_proxy.server[0].host.trim().is_empty()); } } diff --git a/backend/src/library/processor.rs b/backend/src/library/processor.rs index 1039a0062..24d9b7aa5 100644 --- a/backend/src/library/processor.rs +++ b/backend/src/library/processor.rs @@ -31,7 +31,7 @@ pub struct LibraryProcessor { pub fn resolve_metadata_storage_path( metadata_update_config: Option<&MetadataUpdateConfig>, - working_dir: &str, + storage_dir: &str, ) -> PathBuf { let configured_path = metadata_update_config.map_or_else( || PathBuf::from(shared::utils::default_metadata_path()), @@ -46,7 +46,7 @@ pub fn resolve_metadata_storage_path( if configured_path.is_absolute() { configured_path.clean() } else { - PathBuf::from(working_dir).join(configured_path).clean() + PathBuf::from(storage_dir).join(configured_path).clean() } } @@ -61,7 +61,7 @@ impl LibraryProcessor { config .library .as_ref() - .map(|lib_cfg| Self::new(lib_cfg.clone(), config.metadata_update.as_ref(), client, &config.working_dir)) + .map(|lib_cfg| Self::new(lib_cfg.clone(), config.metadata_update.as_ref(), client, &config.storage_dir)) } // Creates a new Library processor with the given configuration @@ -69,9 +69,9 @@ impl LibraryProcessor { config: LibraryConfig, metadata_update_config: Option<&MetadataUpdateConfig>, client: reqwest::Client, - working_dir: &str, + storage_dir: &str, ) -> Self { - let storage_path = resolve_metadata_storage_path(metadata_update_config, working_dir); + let storage_path = resolve_metadata_storage_path(metadata_update_config, storage_dir); let scanner = LibraryScanner::new(config.clone()); let storage = MetadataStorage::new(storage_path); let resolver = MetadataResolver::from_config(Some(&config), metadata_update_config, client, Some(storage.clone())); diff --git a/backend/src/library/tmdb.rs b/backend/src/library/tmdb.rs index b4c71edf4..fc624b32b 100644 --- a/backend/src/library/tmdb.rs +++ b/backend/src/library/tmdb.rs @@ -473,7 +473,7 @@ impl TmdbSeason { #[cfg(test)] mod tests { - use super::{resolve_tmdb_image_url, TmdbMovieDetails, TMDB_IMAGE_BASE_URL}; + use super::{resolve_tmdb_image_url, TMDB_IMAGE_BASE_URL}; #[test] fn test_resolve_tmdb_url_with_absolute_http() { diff --git a/backend/src/main.rs b/backend/src/main.rs index cb247634a..c5608e9b2 100644 --- a/backend/src/main.rs +++ b/backend/src/main.rs @@ -30,7 +30,7 @@ use clap::Parser; use log::{error, info, warn}; use shared::{ model::ConfigPaths, - utils::{CONFIG_FILE, SOURCE_FILE}, + utils::{CONFIG_FILE, CONFIG_PATH, DEFAULT_STORAGE_DIR, SOURCE_FILE}, }; use std::{ fs::File, @@ -45,6 +45,10 @@ use std::{ #[command(version)] #[command(about = "Extended playlist proxy", long_about = None)] struct Args { + /// The home directory (base for config, storage, backup, downloads) + #[arg(short = 'H', long = "home")] + home: Option, + /// The config directory #[arg(short = 'p', long = "config-path")] config_path: Option, @@ -217,21 +221,13 @@ fn run_startup_bplustree_migration(config_paths: &ConfigPaths) { return; } - let config_dto = match utils::read_config_file(config_paths.config_file_path.as_str(), true, false) { - Ok(cfg) => cfg, - Err(err) => { - warn!("Skipping B+Tree startup migration because config could not be read: {err}"); - return; - } - }; - let mut roots: Vec = vec![PathBuf::from(&config_paths.config_path)]; - let working_dir = PathBuf::from(config_dto.working_dir); - if !roots.iter().any(|root| root == &working_dir) { - roots.push(working_dir.clone()); + let storage_dir = PathBuf::from(&config_paths.storage_path); + if !roots.iter().any(|root| root == &storage_dir) { + roots.push(storage_dir.clone()); } - match migrate_bplustree_databases_with_marker(&roots, &working_dir) { + match migrate_bplustree_databases_with_marker(&roots, &storage_dir) { Ok(stats) => { if stats.skipped_by_marker { info!("B+Tree startup migration skipped (marker already present)"); @@ -260,7 +256,7 @@ fn print_info(app_config: &AppConfig) { let paths = > as Access>::load(&app_config.paths); info!("Current time: {}", chrono::offset::Local::now().format("%Y-%m-%d %H:%M:%S")); info!("Temp dir: {}", tempfile::env::temp_dir().display()); - info!("Working dir: {:?}", &config.working_dir); + info!("Storage dir: {:?}", &config.storage_dir); info!("Config dir: {:?}", &paths.config_path); info!("Config file: {:?}", &paths.config_file_path); info!("Source file: {:?}", &paths.sources_file_path); @@ -281,7 +277,7 @@ fn print_info(app_config: &AppConfig) { if let Some(cache) = config.reverse_proxy.as_ref().and_then(|r| r.cache.as_ref()) { if cache.enabled { - info!("Cache dir: {}", cache.dir); + info!("Cache dir: {}", cache.directory); } } if let Some(metadata_update) = config.metadata_update.as_ref() { @@ -293,11 +289,31 @@ fn print_info(app_config: &AppConfig) { } fn get_file_paths(args: &Args) -> ConfigPaths { - let raw_path = args - .config_path + let resolve_storage_path = |home_path: &str, storage_dir: Option<&str>| { + let path = storage_dir.map(str::trim).filter(|value| !value.is_empty()).map(PathBuf::from).map_or_else( + || utils::get_default_path_for_home(Path::new(home_path), DEFAULT_STORAGE_DIR), + |configured_path| { + if configured_path.is_relative() { + PathBuf::from(home_path).join(configured_path) + } else { + configured_path + } + }, + ); + utils::resolve_directory_path(path.to_string_lossy().as_ref()) + }; + + let home_path = args + .home .as_ref() .filter(|p| !p.trim().is_empty()) - .map_or_else(utils::get_default_config_path, |p| resolve_env_var(p)); + .map_or_else(utils::get_home_path, |p| PathBuf::from(resolve_env_var(p))); + let home_path = utils::resolve_directory_path(home_path.to_string_lossy().as_ref()); + + let raw_path = args.config_path.as_ref().filter(|p| !p.trim().is_empty()).map_or_else( + || utils::get_default_path_for_home(Path::new(&home_path), CONFIG_PATH).to_string_lossy().to_string(), + |p| resolve_env_var(p), + ); let config_path: String = utils::resolve_directory_path(&raw_path); let config_file: String = resolve_env_var( @@ -318,11 +334,27 @@ fn get_file_paths(args: &Args) -> ConfigPaths { .as_ref() .map_or_else(|| utils::get_default_sources_file_path(&config_path), ToString::to_string), ); - let mappings_file = args.mapping_file.as_ref().map(|p| resolve_env_var(p)); - let template_file = args.template_file.as_ref().map(|p| resolve_env_var(p)); + let mappings_file = args.mapping_file.as_ref().map(|p| { + let path = resolve_env_var(p); + utils::resolve_mapping_file_path(config_path.as_str(), Some(path.as_str())) + }); + let template_file = args.template_file.as_ref().map(|p| { + let path = resolve_env_var(p); + utils::resolve_template_file_path(config_path.as_str(), Some(path.as_str())) + }); + let storage_path = if Path::new(&config_file).exists() { + match utils::read_config_file(&config_file, true, false) { + Ok(cfg) => resolve_storage_path(&home_path, cfg.storage_dir.as_deref()), + Err(err) => exit!("Can't read config file {} while resolving storage path: {err}", config_file), + } + } else { + resolve_storage_path(&home_path, None) + }; ConfigPaths { + home_path, config_path, + storage_path, config_file_path: config_file, sources_file_path: sources_file, mapping_file_path: mappings_file, // need to be set after config read diff --git a/backend/src/messaging.rs b/backend/src/messaging.rs index e5285d0c6..81fb876a2 100644 --- a/backend/src/messaging.rs +++ b/backend/src/messaging.rs @@ -309,7 +309,9 @@ mod tests { api_proxy: Arc::new(ArcSwapOption::default()), file_locks: Arc::new(FileLockManager::default()), paths: Arc::new(ArcSwap::from_pointee(ConfigPaths { + home_path: String::new(), config_path: String::new(), + storage_path: String::new(), config_file_path: String::new(), sources_file_path: String::new(), mapping_file_path: None, diff --git a/backend/src/model/config/app.rs b/backend/src/model/config/app.rs index 1a0f54fbf..89745aebe 100644 --- a/backend/src/model/config/app.rs +++ b/backend/src/model/config/app.rs @@ -329,9 +329,10 @@ impl AppConfig { fn set_mapping_path(&self, mapping_path: Option<&str>) { let paths_guard = self.paths.load(); let old_path = paths_guard.mapping_file_path.as_deref(); - let new_path = mapping_path - .map(ToString::to_string) - .or_else(|| Some(utils::get_default_mappings_path(&paths_guard.config_path))); + let new_path = Some(utils::resolve_mapping_file_path( + paths_guard.config_path.as_str(), + mapping_path, + )); if old_path != new_path.as_deref() { let mut new_paths = (**paths_guard).clone(); @@ -343,9 +344,10 @@ impl AppConfig { fn set_template_path(&self, template_path: Option<&str>) { let paths_guard = self.paths.load(); let old_path = paths_guard.template_file_path.as_deref(); - let new_path = template_path - .map(ToString::to_string) - .or_else(|| Some(utils::get_default_templates_path(&paths_guard.config_path))); + let new_path = Some(utils::resolve_template_file_path( + paths_guard.config_path.as_str(), + template_path, + )); if old_path != new_path.as_deref() { let mut new_paths = (**paths_guard).clone(); @@ -406,7 +408,7 @@ impl AppConfig { } let path = PathBuf::from(custom_stream_response_path); - let path = utils::make_path_absolute(&path, &config.working_dir); + let path = utils::make_path_absolute(&path, &config.storage_dir); let paths = self.paths.load_full(); let mut new_paths = paths.as_ref().clone(); diff --git a/backend/src/model/config/base.rs b/backend/src/model/config/base.rs index 606d37ba8..46eda40c7 100644 --- a/backend/src/model/config/base.rs +++ b/backend/src/model/config/base.rs @@ -8,21 +8,19 @@ use log::{error, info}; use path_clean::PathClean; use shared::error::TuliproxError; use shared::model::{ConfigDto, HdHomeRunDeviceOverview}; -use shared::utils::{default_grace_period_millis, default_grace_period_timeout_secs, set_sanitize_sensitive_info, DEFAULT_WEB_DIR}; +use shared::utils::{default_grace_period_millis, default_grace_period_timeout_secs, set_sanitize_sensitive_info, DEFAULT_BACKUP_DIR, DEFAULT_CACHE_DIR, DEFAULT_DOWNLOAD_DIR, DEFAULT_STORAGE_DIR, DEFAULT_STORAGE_TEMP_DIR, DEFAULT_USER_CONFIG_DIR}; use std::borrow::Cow; use std::path::{Path, PathBuf}; -use crate::utils::get_default_path; - -const DEFAULT_BACKUP_DIR: &str = "backup"; +use crate::utils::get_default_path_for_home; fn create_directories(cfg: &Config, temp_path: &Path) { // Collect the paths into a vector. let paths_strings = [ - Some(cfg.working_dir.clone()), + Some(cfg.storage_dir.clone()), cfg.backup_dir.clone(), cfg.user_config_dir.clone(), cfg.video.as_ref().and_then(|v| v.download.as_ref()).map(|d| d.directory.clone()), - cfg.reverse_proxy.as_ref().and_then(|r| r.cache.as_ref().and_then(|c| if c.enabled { Some(c.dir.clone()) } else { None })), + cfg.reverse_proxy.as_ref().and_then(|r| r.cache.as_ref().and_then(|c| if c.enabled { Some(c.directory.clone()) } else { None })), cfg.metadata_update.as_ref().map(|m| m.cache_path.clone()), ]; @@ -67,7 +65,7 @@ impl Default for GracePeriodOptions { pub struct Config { pub process_parallel: bool, pub api: ConfigApi, - pub working_dir: String, + pub storage_dir: String, pub default_user_agent: Option, pub backup_dir: Option, pub user_config_dir: Option, @@ -95,24 +93,58 @@ pub struct Config { } impl Config { - pub fn prepare(&mut self, config_path: &str) -> Result<(), TuliproxError> { - let work_dir = &self.working_dir; - self.working_dir = utils::resolve_directory_path(work_dir); + fn lexical_normalize(path: &Path) -> PathBuf { + path.components().collect::() + } - self.prepare_directories(); - self.prepare_api_web_root(); + fn normalize_storage_path(raw_storage_dir: &str, storage_dir: &str, input: &str, default_dir: &str) -> Option { + let trimmed_input = input.trim(); + if trimmed_input.is_empty() { + return Some(PathBuf::from(storage_dir).join(default_dir)); + } + + let input_path = PathBuf::from(trimmed_input); + if input_path.is_absolute() { + return None; + } + + let mut normalized_relative = Self::lexical_normalize(&input_path); + let trimmed_raw_storage = raw_storage_dir.trim(); + if !trimmed_raw_storage.is_empty() { + let raw_path = PathBuf::from(trimmed_raw_storage); + if raw_path.is_relative() { + let normalized_raw = Self::lexical_normalize(&raw_path); + if normalized_raw.components().next().is_some() { + while normalized_relative.starts_with(&normalized_raw) { + match normalized_relative.strip_prefix(&normalized_raw) { + Ok(stripped) => { + normalized_relative = Self::lexical_normalize(stripped); + } + Err(_) => break, + } + } + } + } + } + + Some(PathBuf::from(storage_dir).join(normalized_relative)) + } + + pub fn prepare(&mut self, config_path: &str, home_path: &str) -> Result<(), TuliproxError> { + + self.prepare_directories(home_path); if let Some(ref mut webui) = &mut self.web_ui { webui.prepare(config_path)?; } if let Some(library) = self.library.as_mut() { - library.prepare(&self.working_dir)?; + library.prepare(&self.storage_dir)?; } if let Some(metadata_update) = self.metadata_update.as_mut() { let meta_path = PathBuf::from(&metadata_update.cache_path); let meta_path = if meta_path.is_relative() { - PathBuf::from(&self.working_dir).join(meta_path) + PathBuf::from(&self.storage_dir).join(meta_path) } else { meta_path }; @@ -125,38 +157,87 @@ impl Config { if let Some(video) = self.video.as_mut() { video.prepare(); + if let Some(download) = video.download.as_mut() { + let download_path = PathBuf::from(&download.directory); + if download.directory.trim().is_empty() { + download.directory = get_default_path_for_home(Path::new(home_path), DEFAULT_DOWNLOAD_DIR) + .clean() + .to_string_lossy() + .to_string(); + } else if download_path.is_relative() { + download.directory = PathBuf::from(home_path) + .join(download_path) + .clean() + .to_string_lossy() + .to_string(); + } + } } Ok(()) } - fn prepare_directories(&mut self) { - fn set_directory(path: &mut Option, default_subdir: &str, working_dir: &str) { - *path = Some(match path.as_ref() { - Some(existing) => existing.to_owned(), - None => PathBuf::from(working_dir).join(default_subdir).clean().to_string_lossy().to_string(), - }); + fn prepare_reverse_proxy_cache_dir(&mut self, raw_storage_dir: &str) { + let Some(reverse_proxy) = self.reverse_proxy.as_mut() else { + return; + }; + let Some(cache) = reverse_proxy.cache.as_mut() else { + return; + }; + if !cache.enabled { + return; } - set_directory(&mut self.backup_dir, DEFAULT_BACKUP_DIR, &self.working_dir); - set_directory(&mut self.user_config_dir, "user_config", &self.working_dir); + let normalized = Self::normalize_storage_path(raw_storage_dir, &self.storage_dir, &cache.directory, DEFAULT_CACHE_DIR) + .unwrap_or_else(|| PathBuf::from(cache.directory.trim())); + cache.directory = normalized.clean().to_string_lossy().to_string(); + } + + fn prepare_directories(&mut self, home_path: &str) { + let raw_storage_dir = self.storage_dir.trim().to_string(); + let storage_dir_path = if raw_storage_dir.is_empty() { + get_default_path_for_home(Path::new(home_path), DEFAULT_STORAGE_DIR) + } else { + let configured_storage_path = PathBuf::from(&raw_storage_dir); + if configured_storage_path.is_relative() { + PathBuf::from(home_path).join(configured_storage_path) + } else { + configured_storage_path + } + }; + self.storage_dir = utils::resolve_directory_path(storage_dir_path.to_string_lossy().as_ref()); + self.prepare_reverse_proxy_cache_dir(&raw_storage_dir); + + let storage_dir = self.storage_dir.clone(); + let normalize_optional_path = |value: Option<&str>, default_dir: &str| -> String { + let configured = value.unwrap_or_default(); + let normalized = Self::normalize_storage_path(&raw_storage_dir, &storage_dir, configured, default_dir) + .unwrap_or_else(|| PathBuf::from(configured.trim())); + normalized.clean().to_string_lossy().to_string() + }; + + self.backup_dir = Some(normalize_optional_path(self.backup_dir.as_deref(), DEFAULT_BACKUP_DIR)); + self.user_config_dir = Some(normalize_optional_path(self.user_config_dir.as_deref(), DEFAULT_USER_CONFIG_DIR)); + self.prepare_api_web_root(home_path); } pub fn get_backup_dir(&self) -> Cow<'_, str> { self.backup_dir.as_ref().map_or_else(|| Cow::Borrowed(DEFAULT_BACKUP_DIR), |v| Cow::Borrowed(v)) } - fn prepare_api_web_root(&mut self) { + fn prepare_api_web_root(&mut self, home_path: &str) { if self.api.web_root.is_empty() { - self.api.web_root = get_default_path(DEFAULT_WEB_DIR).display().to_string(); + self.api.web_root = utils::get_default_web_root_path_for_home(Path::new(home_path)) + .display() + .to_string(); } else { - self.api.web_root = utils::make_absolute_path(&self.api.web_root, &self.working_dir); + self.api.web_root = utils::make_absolute_path(&self.api.web_root, &self.storage_dir); } } pub fn update_runtime(&self) { set_sanitize_sensitive_info(self.log.as_ref().is_none_or(|l| l.sanitize_sensitive_info)); - let temp_path = PathBuf::from(&self.working_dir).join("tmp"); + let temp_path = PathBuf::from(&self.storage_dir).join(DEFAULT_STORAGE_TEMP_DIR); create_directories(self, &temp_path); let _ = tempfile::env::override_temp_dir(&temp_path); } @@ -200,7 +281,7 @@ impl From<&ConfigDto> for Config { process_parallel: dto.process_parallel, disk_based_processing: dto.disk_based_processing, api: ConfigApi::from(&dto.api), - working_dir: dto.working_dir.clone(), + storage_dir: dto.storage_dir.clone().unwrap_or_default(), default_user_agent: dto.default_user_agent.clone(), backup_dir: dto.backup_dir.clone(), user_config_dir: dto.user_config_dir.clone(), diff --git a/backend/src/model/config/cache.rs b/backend/src/model/config/cache.rs index 5690ce4b0..c80741d0f 100644 --- a/backend/src/model/config/cache.rs +++ b/backend/src/model/config/cache.rs @@ -5,7 +5,7 @@ use crate::model::macros; #[derive(Debug, Clone)] pub struct CacheConfig { pub enabled: bool, - pub dir: String, + pub directory: String, pub size: usize, pub size_str: Option, } @@ -16,7 +16,7 @@ impl From<&CacheConfigDto> for CacheConfig { Self { enabled: dto.enabled, // Dto prepare should have set the right path - dir: dto.dir.as_ref().map_or_else(Default::default, std::string::ToString::to_string), + directory: dto.directory.as_ref().map_or_else(Default::default, std::string::ToString::to_string), size_str: dto.size.clone(), size: get_size(dto) } @@ -28,7 +28,7 @@ impl From<&CacheConfig> for CacheConfigDto { Self { enabled: instance.enabled, // Dto prepare should have set the right path - dir: Some(instance.dir.clone()), + directory: Some(instance.directory.clone()), size: instance.size_str.clone(), } } diff --git a/backend/src/model/config/input.rs b/backend/src/model/config/input.rs index 4ec25072a..76a9eb736 100644 --- a/backend/src/model/config/input.rs +++ b/backend/src/model/config/input.rs @@ -242,6 +242,14 @@ pub struct ConfigInput { } impl ConfigInput { + #[inline] + pub fn get_download_input_type(&self) -> InputType { + self.staged + .as_ref() + .filter(|staged| staged.enabled) + .map_or(self.input_type, |staged| staged.input_type) + } + #[inline] pub fn has_flag(&self, flag: ConfigInputFlags) -> bool { self.has_flag_or(flag, false) @@ -734,4 +742,34 @@ mod tests { let err = input.prepare(&[]).unwrap_err(); assert!(err.to_string().contains("Malformed provider URL")); } + + #[test] + fn test_get_download_input_type_uses_staged_when_enabled() { + let input = ConfigInput { + input_type: InputType::Xtream, + staged: Some(StagedInput { + enabled: true, + input_type: InputType::M3u, + ..Default::default() + }), + ..Default::default() + }; + + assert_eq!(input.get_download_input_type(), InputType::M3u); + } + + #[test] + fn test_get_download_input_type_uses_primary_when_staged_disabled() { + let input = ConfigInput { + input_type: InputType::Xtream, + staged: Some(StagedInput { + enabled: false, + input_type: InputType::M3u, + ..Default::default() + }), + ..Default::default() + }; + + assert_eq!(input.get_download_input_type(), InputType::Xtream); + } } diff --git a/backend/src/model/config/library.rs b/backend/src/model/config/library.rs index 24109556c..fe9cdffc8 100644 --- a/backend/src/model/config/library.rs +++ b/backend/src/model/config/library.rs @@ -1,6 +1,7 @@ use crate::model::macros; use shared::error::{info_err_res, TuliproxError}; use shared::model::{ConfigDto, LibraryConfigDto, LibraryContentType, LibraryMetadataFormat}; +use shared::utils::DEFAULT_STORAGE_DIR; use shared::utils::Internable; use std::path::PathBuf; use std::sync::Arc; @@ -43,14 +44,14 @@ pub struct LibraryConfig { } impl LibraryConfig { - fn canonicalize_scan_directory_path(path: &str, working_dir: &str) -> Result { + fn canonicalize_scan_directory_path(path: &str, storage_dir: &str) -> Result { let path = path.trim(); if path.is_empty() { return info_err_res!("Library scan directory path cannot be empty"); } let scan_path = PathBuf::from(path); - let scan_path = if scan_path.is_relative() { PathBuf::from(working_dir).join(scan_path) } else { scan_path }; + let scan_path = if scan_path.is_relative() { PathBuf::from(storage_dir).join(scan_path) } else { scan_path }; match scan_path.canonicalize() { Ok(path_buf) => Ok(path_buf.to_string_lossy().to_string()), @@ -58,16 +59,16 @@ impl LibraryConfig { } } - fn canonicalize_scan_directories(&mut self, working_dir: &str) -> Result<(), TuliproxError> { + fn canonicalize_scan_directories(&mut self, storage_dir: &str) -> Result<(), TuliproxError> { for dir in &mut self.scan_directories { - dir.path = Self::canonicalize_scan_directory_path(&dir.path, working_dir)?; + dir.path = Self::canonicalize_scan_directory_path(&dir.path, storage_dir)?; } Ok(()) } - pub fn prepare(&mut self, working_dir: &str) -> Result<(), TuliproxError> { + pub fn prepare(&mut self, storage_dir: &str) -> Result<(), TuliproxError> { if self.enabled { - self.canonicalize_scan_directories(working_dir)?; + self.canonicalize_scan_directories(storage_dir)?; } Ok(()) } @@ -119,11 +120,18 @@ pub fn validate_library_paths_from_dto(cfg: &ConfigDto) -> Result<(), TuliproxEr }; let mut library_cfg = LibraryConfig::from(library_dto); + let storage_dir = cfg + .storage_dir + .as_deref() + .map(str::trim) + .filter(|dir| !dir.is_empty()) + .unwrap_or(DEFAULT_STORAGE_DIR); // Always validate configured scan directories, even when library is disabled. // This prevents persisting invalid paths that would later break startup when enabled. - library_cfg.canonicalize_scan_directories(&cfg.working_dir)?; if library_cfg.enabled { - library_cfg.prepare(&cfg.working_dir)?; + library_cfg.prepare(storage_dir)?; + } else { + library_cfg.canonicalize_scan_directories(storage_dir)?; } Ok(()) } @@ -143,10 +151,12 @@ mod tests { #[test] fn validate_library_paths_rejects_missing_scan_directory() { let cfg = ConfigDto { - working_dir: std::env::current_dir() - .expect("current_dir should be available") - .to_string_lossy() - .to_string(), + storage_dir: Some( + std::env::current_dir() + .expect("current_dir should be available") + .to_string_lossy() + .to_string(), + ), library: Some(LibraryConfigDto { enabled: true, scan_directories: vec![LibraryScanDirectoryDto { path: unique_missing_path(), ..Default::default() }], @@ -162,10 +172,12 @@ mod tests { #[test] fn validate_library_paths_rejects_missing_scan_directory_even_when_disabled() { let cfg = ConfigDto { - working_dir: std::env::current_dir() - .expect("current_dir should be available") - .to_string_lossy() - .to_string(), + storage_dir: Some( + std::env::current_dir() + .expect("current_dir should be available") + .to_string_lossy() + .to_string(), + ), library: Some(LibraryConfigDto { enabled: false, scan_directories: vec![LibraryScanDirectoryDto { path: unique_missing_path(), ..Default::default() }], @@ -182,10 +194,12 @@ mod tests { #[test] fn validate_library_paths_accepts_existing_scan_directory() { let cfg = ConfigDto { - working_dir: std::env::current_dir() - .expect("current_dir should be available") - .to_string_lossy() - .to_string(), + storage_dir: Some( + std::env::current_dir() + .expect("current_dir should be available") + .to_string_lossy() + .to_string(), + ), library: Some(LibraryConfigDto { enabled: true, scan_directories: vec![LibraryScanDirectoryDto { diff --git a/backend/src/model/config/source.rs b/backend/src/model/config/source.rs index 194af8fe8..bf48ab4ff 100644 --- a/backend/src/model/config/source.rs +++ b/backend/src/model/config/source.rs @@ -182,13 +182,6 @@ impl From<&ConfigProviderDto> for ConfigProvider { fn from(dto: &ConfigProviderDto) -> Self { let dns_cfg = dto.dns.as_ref().map(ProviderDnsConfig::from); let dns_cache = Arc::new(ProviderDnsCache::default()); - if let Some(dns_dto) = dto.dns.as_ref() { - if let Some(resolved) = dns_dto.resolved.as_ref() { - for (host, ips) in resolved { - dns_cache.store_resolved(host, ips.clone()); - } - } - } Self { name: dto.name.clone(), urls: dto.urls.clone(), diff --git a/backend/src/model/config/video_download.rs b/backend/src/model/config/video_download.rs index 8b8c2315e..09a57fdcc 100644 --- a/backend/src/model/config/video_download.rs +++ b/backend/src/model/config/video_download.rs @@ -1,6 +1,7 @@ use crate::model::macros; use regex::Regex; use shared::model::{VideoConfigDto, VideoDownloadConfigDto}; +use shared::utils::DEFAULT_DOWNLOAD_DIR; use std::collections::HashMap; use std::sync::Arc; @@ -17,7 +18,7 @@ impl From<&VideoDownloadConfigDto> for VideoDownloadConfig { fn from(dto: &VideoDownloadConfigDto) -> Self { Self { headers: dto.headers.clone(), - directory: dto.directory.as_ref().map_or_else(|| "downloads".to_string(), ToString::to_string), + directory: dto.directory.as_ref().map_or_else(|| DEFAULT_DOWNLOAD_DIR.to_string(), ToString::to_string), organize_into_directories: dto.organize_into_directories, episode_pattern: dto.episode_pattern.as_ref().and_then(|s| shared::model::REGEX_CACHE.get_or_compile(s) .map_err(|e| log::warn!("Invalid episode_pattern regex '{s}': {e}")) diff --git a/backend/src/processing/geoip.rs b/backend/src/processing/geoip.rs new file mode 100644 index 000000000..4911791ce --- /dev/null +++ b/backend/src/processing/geoip.rs @@ -0,0 +1,95 @@ +use crate::{ + api::model::AppState, + model::InputSource, + repository::get_geoip_path, + utils::{request::download_text_content, GeoIp}, +}; +use log::{error, info}; +use shared::{ + model::{default_geoip_url, InputFetchMethod}, + utils::Internable, +}; +use std::{collections::HashMap, io::Cursor, sync::Arc}; + +#[derive(Debug)] +pub(crate) enum GeoIpUpdateError { + Disabled, + DownloadFailed(String), + ProcessFailed(String), + UnknownProcessing, +} + +impl std::fmt::Display for GeoIpUpdateError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Disabled => write!(f, "GeoIp update is disabled"), + Self::DownloadFailed(err) => write!(f, "Failed to download geoip db: {err}"), + Self::ProcessFailed(err) => write!(f, "Failed to process geoip db: {err}"), + Self::UnknownProcessing => write!(f, "Unknown GeoIp processing error"), + } + } +} + +impl std::error::Error for GeoIpUpdateError {} + +pub(crate) async fn update_geoip_db(app_state: &Arc) -> Result<(), GeoIpUpdateError> { + let config = app_state.app_config.config.load(); + if let Some(geoip) = config.reverse_proxy.as_ref().and_then(|r| r.geoip.as_ref()) { + if geoip.enabled { + let geoip_db_path = &*get_geoip_path(&config.storage_dir); + let _file_lock = app_state.app_config.file_locks.write_lock(geoip_db_path).await; + + let url = if geoip.url.trim().is_empty() { default_geoip_url() } else { geoip.url.clone() }; + let input_source = InputSource { + name: "GeoIP".intern(), + url, + provider: None, + username: None, + password: None, + method: InputFetchMethod::GET, + headers: HashMap::default(), + }; + return match download_text_content( + &app_state.app_config, + &app_state.http_client.load(), + &input_source, + None, + None, + false, + ) + .await + { + Ok((content, _)) => { + let reader = Cursor::new(content); + let mut geoip = GeoIp::new(); + let result = { + match geoip.import_ipv4_from_csv(reader, geoip_db_path) { + Ok(size) => (Some(size), None), + Err(err) => (None, Some(err)), + } + }; + + return match result { + (Some(_), None) => { + info!("GeoIp db updated"); + app_state.geoip.store(Some(Arc::new(geoip))); + Ok(()) + } + (None, Some(err)) => { + let error = GeoIpUpdateError::ProcessFailed(err.to_string()); + error!("{error}"); + Err(error) + } + _ => Err(GeoIpUpdateError::UnknownProcessing), + }; + } + Err(err) => { + let error = GeoIpUpdateError::DownloadFailed(err.to_string()); + error!("{error}"); + Err(error) + } + }; + } + } + Err(GeoIpUpdateError::Disabled) +} diff --git a/backend/src/processing/input_cache.rs b/backend/src/processing/input_cache.rs index 5c354bcc7..3ea033729 100644 --- a/backend/src/processing/input_cache.rs +++ b/backend/src/processing/input_cache.rs @@ -27,9 +27,9 @@ pub struct InputStatus { pub clusters: HashMap, } -pub async fn resolve_input_storage_path(working_dir: &str, input_name: &str) -> PathBuf { - if let Ok(path) = get_input_storage_path(input_name, working_dir).await { path } else { - build_input_storage_path(input_name, working_dir) +pub async fn resolve_input_storage_path(storage_dir: &str, input_name: &str) -> PathBuf { + if let Ok(path) = get_input_storage_path(input_name, storage_dir).await { path } else { + build_input_storage_path(input_name, storage_dir) } } diff --git a/backend/src/processing/mod.rs b/backend/src/processing/mod.rs index 83cbd1f46..3bb1e3368 100644 --- a/backend/src/processing/mod.rs +++ b/backend/src/processing/mod.rs @@ -1,5 +1,5 @@ +pub(crate) mod geoip; pub(crate) mod input_cache; pub(crate) mod playlist_watch; pub(crate) mod parser; pub(crate) mod processor; - diff --git a/backend/src/processing/parser/xmltv.rs b/backend/src/processing/parser/xmltv.rs index 5a48fb3de..b3f0f80b5 100644 --- a/backend/src/processing/parser/xmltv.rs +++ b/backend/src/processing/parser/xmltv.rs @@ -123,14 +123,12 @@ impl TVGuide { let src = "src".intern(); for child in children { match child.name.as_ref() { - EPG_TAG_DISPLAY_NAME => { - if smart_match { + EPG_TAG_DISPLAY_NAME if smart_match => { if let Some(name) = &child.value { tag.normalized_epg_ids .get_or_insert_with(Vec::new) .push(normalize_channel_name(name, &id_cache.smart_match_config).intern()); } - } } EPG_TAG_ICON => { if let Some(src) = child.get_attribute_value(&src) { diff --git a/backend/src/processing/parser/xtream.rs b/backend/src/processing/parser/xtream.rs index 6dcebf1a5..ef994d4a7 100644 --- a/backend/src/processing/parser/xtream.rs +++ b/backend/src/processing/parser/xtream.rs @@ -174,9 +174,7 @@ pub async fn parse_xtream(input: &ConfigInput, input.has_flag(ConfigInputFlags::XtreamLiveStreamWithoutExtension), ); - // Re-implement the loop to add ordinal - let mut ord_counter: u32 = 1; - for stream in xtream_streams { + for (ord_counter, stream) in (1_u32..).zip(xtream_streams) { let group = group_map.get_mut(&stream.get_category_id()).unwrap_or(&mut unknown_grp); let category_name = &group.category_name; let stream_url = create_xtream_url(xtream_cluster, url, username, password, &stream, live_stream_use_prefix, live_stream_without_extension); @@ -200,7 +198,6 @@ pub async fn parse_xtream(input: &ConfigInput, }, }; item.header.source_ordinal = ord_counter; - ord_counter += 1; group.add(item); } diff --git a/backend/src/processing/playlist_watch.rs b/backend/src/processing/playlist_watch.rs index 8f8269dc6..85d921866 100644 --- a/backend/src/processing/playlist_watch.rs +++ b/backend/src/processing/playlist_watch.rs @@ -21,7 +21,7 @@ pub async fn process_group_watch(app_config: &Arc, client: &reqwest:: let watch_filename = format!("{}/{}.bin", utils::sanitize_filename(target_name), utils::sanitize_filename(&pl.title)); let cfg = app_config.config.load(); - match utils::get_file_path(&cfg.working_dir, Some(std::path::PathBuf::from(&watch_filename))) { + match utils::get_file_path(&cfg.storage_dir, Some(std::path::PathBuf::from(&watch_filename))) { Some(path) => { let save_path = path.as_path(); let mut changed = false; diff --git a/backend/src/processing/processor/playlist.rs b/backend/src/processing/processor/playlist.rs index f8f861cef..5dcd0248d 100644 --- a/backend/src/processing/processor/playlist.rs +++ b/backend/src/processing/processor/playlist.rs @@ -50,13 +50,15 @@ use std::{ collections::{HashMap, HashSet}, path::PathBuf, sync::{Arc, Weak}, - time::Instant, + time::{Duration, Instant}, }; use tokio::{ sync::{Mutex, OwnedRwLockWriteGuard, RwLock}, task::JoinSet, }; +const PLAYLIST_UPDATE_MAX_DURATION_SECS: u64 = 3600; + fn is_valid(pli: &PlaylistItem, filter: &Filter, match_as_ascii: bool) -> bool { let provider = ValueProvider { pli, match_as_ascii }; filter.filter(&provider) @@ -324,10 +326,11 @@ async fn playlist_download_from_input( input: &ConfigInput, ) -> PlaylistDownloadResult { let config = &*app_config.config.load(); - let working_dir = &config.working_dir; + let storage_dir = &config.storage_dir; + let download_input_type = input.get_download_input_type(); // Check Status - let storage_path = input_cache::resolve_input_storage_path(working_dir, &input.name).await; + let storage_path = input_cache::resolve_input_storage_path(storage_dir, &input.name).await; let mut status = input_cache::load_input_status(&storage_path); let cache_duration = input.cache_duration_seconds; @@ -336,7 +339,7 @@ async fn playlist_download_from_input( let _ = std::fs::create_dir_all(&storage_path); } - let (clusters_to_download, fully_cached) = match input.input_type { + let (clusters_to_download, fully_cached) = match download_input_type { InputType::Xtream => { let mut to_download = vec![]; for c in XTREAM_CLUSTER { @@ -364,7 +367,7 @@ async fn playlist_download_from_input( return PlaylistDownloadResult::new(vec![], vec![], true, false); } - let (playlist, errors, persisted) = match input.input_type { + let (playlist, errors, persisted) = match download_input_type { InputType::M3u => { let (p, e) = m3u::download_m3u_playlist(app_config, client, config, input).await; (p, e, false) @@ -381,7 +384,7 @@ async fn playlist_download_from_input( // Update Status if errors.is_empty() { - if let InputType::Xtream = input.input_type { + if let InputType::Xtream = download_input_type { if let Some(clusters) = clusters_to_download { for c in clusters { input_cache::update_cluster_status(&mut status, &c.to_string(), ClusterState::Ok); @@ -401,7 +404,7 @@ async fn playlist_download_from_input( // We could mark specific clusters as failed if we knew which one failed. // For simplicity, if error, we don't update the timestamp (so it stays expired/invalid). // Or we mark as Failed. - if let InputType::Xtream = input.input_type { + if let InputType::Xtream = download_input_type { if let Some(clusters) = clusters_to_download { for c in clusters { // Optimistic: Only mark failed if we are sure? @@ -437,6 +440,7 @@ async fn process_source( continue; }; if is_input_enabled(input, &ctx.user_targets) { + let effective_input_type = input.get_download_input_type(); source_downloaded = true; log_memory_snapshot(format!("source[{source_idx}] input '{}' before_download", input.name).as_str()); @@ -459,7 +463,7 @@ async fn process_source( }; log_memory_snapshot(format!("source[{source_idx}] input '{}' after_download", input.name).as_str()); - let tvguide = if input.input_type == InputType::Library { + let tvguide = if effective_input_type == InputType::Library { None } else { download_input_epg(ctx, input, &mut error_list).await @@ -483,7 +487,7 @@ async fn process_source( let elapsed = start_time.elapsed().as_secs(); input_stats.insert( input_name.clone(), - create_input_stat(group_count, channel_count, errors.len(), input.input_type, input_name, elapsed), + create_input_stat(group_count, channel_count, errors.len(), effective_input_type, input_name, elapsed), ); } } @@ -547,8 +551,8 @@ async fn download_input_epg( ) -> Option { // Download epg for input let (tvguide, mut tvguide_errors) = if error_list.is_empty() { - let working_dir = &ctx.config.config.load().working_dir; - epg::get_xmltv(ctx, input, None, working_dir).await + let storage_dir = &ctx.config.config.load().storage_dir; + epg::get_xmltv(ctx, input, None, storage_dir).await } else { (None, vec![]) }; @@ -1018,10 +1022,10 @@ async fn playlist_probe(ctx: &PlaylistProcessingContext, target: &ConfigTarget, } let input_name = fpl.input.name.clone(); - let input_type = fpl.input.input_type; - let xtream_probe_handled = input_type.is_xtream() && target.get_xtream_output().is_some(); + let effective_input_type = fpl.input.get_download_input_type(); + let xtream_probe_handled = effective_input_type.is_xtream() && target.get_xtream_output().is_some(); let live_probe_settings = if probe_live_enabled { - get_live_probe_interval_settings(target, input_type, Some(opts)).map(|(delay, interval_secs)| { + get_live_probe_interval_settings(target, effective_input_type, Some(opts)).map(|(delay, interval_secs)| { let interval_signed = i64::try_from(interval_secs).unwrap_or(i64::MAX); let cutoff_ts = chrono::Utc::now().timestamp().saturating_sub(interval_signed); (delay, interval_secs, cutoff_ts) @@ -1050,6 +1054,16 @@ async fn playlist_probe(ctx: &PlaylistProcessingContext, target: &ConfigTarget, if needs_live_probe(&item, cutoff_ts) { if let Some(provider_id) = provider_id_from_item(&item) { if queued_live_keys.insert(provider_id.clone()) { + if log_enabled!(Level::Debug) { + let last_probed = match item.header.additional_properties.as_ref() { + Some(StreamProperties::Live(props)) => props.last_probed_timestamp, + _ => None, + }; + debug!( + "[Task] Creating ProbeLive task for input {}: id={}, last_probed_ts={:?}, cutoff_ts={}, interval={}s, title=\"{}\"", + input_name, provider_id, last_probed, cutoff_ts, interval_secs, item.header.title + ); + } let task = UpdateTask::ProbeLive { id: provider_id, reason: ResolveReasonSet::from_variants(&[ResolveReason::Probe]), @@ -1092,7 +1106,7 @@ async fn playlist_probe(ctx: &PlaylistProcessingContext, target: &ConfigTarget, } // For M3U, ID is a provider id; for Library, ID is UUID. - let unique_id = if input_type == InputType::Library { + let unique_id = if effective_input_type == InputType::Library { item.header.uuid.to_valid_uuid() } else { item.header.id.to_string() @@ -1104,6 +1118,10 @@ async fn playlist_probe(ctx: &PlaylistProcessingContext, target: &ConfigTarget, continue; } + debug!( + "[Task] Creating ProbeStream task for input {}: scope={}, unique_id={}, item_type={:?}, title=\"{}\"", + input_name, probe_scope, unique_id, item.header.item_type, item.header.title + ); let task = UpdateTask::ProbeStream { probe_scope, unique_id, @@ -1204,7 +1222,7 @@ async fn process_watch( } } -#[allow(clippy::too_many_arguments)] +#[allow(clippy::too_many_arguments, clippy::too_many_lines)] pub async fn exec_processing( client: &reqwest::Client, app_config: Arc, @@ -1219,6 +1237,7 @@ pub async fn exec_processing( pre_processed_inputs: Option>>, acquired_permit: Option, ) { + let max_update_duration = Duration::from_secs(PLAYLIST_UPDATE_MAX_DURATION_SECS); let playlist_guard = if let Some(permit) = acquired_permit { Some(permit) } else if let Some(guard) = &update_guard { @@ -1237,7 +1256,15 @@ pub async fn exec_processing( if playlist_guard.is_some() { if let Some(state) = app_state.as_ref() { - sync_panel_api_exp_dates(state).await; + if tokio::time::timeout(max_update_duration, sync_panel_api_exp_dates(state)).await.is_err() { + error!( + "Playlist update bootstrap timed out after {PLAYLIST_UPDATE_MAX_DURATION_SECS} secs while holding playlist lock", + ); + if let Some(events) = event_manager.as_deref() { + events.send_event(EventMessage::PlaylistUpdate(shared::model::PlaylistUpdateState::Failure)); + } + return; + } } } @@ -1266,13 +1293,29 @@ pub async fn exec_processing( }; let start_time = Instant::now(); - let process_result = std::panic::AssertUnwindSafe(process_sources(&ctx)).catch_unwind().await; - let Ok((stats, errors)) = process_result else { - error!("Playlist processing panicked"); - if let Some(events) = event_manager.as_deref() { - events.send_event(EventMessage::PlaylistUpdate(shared::model::PlaylistUpdateState::Failure)); + let process_result = tokio::time::timeout( + max_update_duration, + std::panic::AssertUnwindSafe(process_sources(&ctx)).catch_unwind(), + ) + .await; + let (stats, errors) = match process_result { + Ok(Ok((stats, errors))) => (stats, errors), + Ok(Err(_)) => { + error!("Playlist processing panicked"); + if let Some(events) = event_manager.as_deref() { + events.send_event(EventMessage::PlaylistUpdate(shared::model::PlaylistUpdateState::Failure)); + } + return; + } + Err(_) => { + error!( + "Playlist processing timed out after {PLAYLIST_UPDATE_MAX_DURATION_SECS} secs while holding playlist lock", + ); + if let Some(events) = event_manager.as_deref() { + events.send_event(EventMessage::PlaylistUpdate(shared::model::PlaylistUpdateState::Failure)); + } + return; } - return; }; log_memory_snapshot("exec_processing after_process_sources"); diff --git a/backend/src/processing/processor/stream_probe.rs b/backend/src/processing/processor/stream_probe.rs index 36506c15e..7ddd4b447 100644 --- a/backend/src/processing/processor/stream_probe.rs +++ b/backend/src/processing/processor/stream_probe.rs @@ -41,7 +41,7 @@ pub async fn update_generic_stream_metadata( active_provider: &Arc, active_handle: Option<&crate::api::model::ProviderHandle>, ) -> Result { - let working_dir = &app_config.config.load().working_dir; + let storage_dir = &app_config.config.load().storage_dir; // Check if probing is enabled globally let ffprobe_enabled = app_config.is_ffprobe_enabled().await; @@ -50,7 +50,7 @@ pub async fn update_generic_stream_metadata( } // Determine storage file path based on input type - let storage_path = get_input_storage_path(&input.name, working_dir).await + let storage_path = get_input_storage_path(&input.name, storage_dir).await .map_err(|e| shared::error::info_err!("Storage path error: {e}"))?; let (db_path, storage_kind) = match input.input_type { diff --git a/backend/src/processing/processor/xtream.rs b/backend/src/processing/processor/xtream.rs index 8e6862697..5779d8949 100644 --- a/backend/src/processing/processor/xtream.rs +++ b/backend/src/processing/processor/xtream.rs @@ -21,8 +21,8 @@ pub async fn update_live_stream_metadata( _active_handle: Option<&ProviderHandle>, _active_provider: &Arc, ) -> Result, TuliproxError> { - let working_dir = &app_config.config.load().working_dir; - let storage_path = get_input_storage_path(&input.name, working_dir).await + let storage_dir = &app_config.config.load().storage_dir; + let storage_path = get_input_storage_path(&input.name, storage_dir).await .map_err(|e| shared::error::info_err!("Storage path error: {e}"))?; // Try to load existing info first to preserve data diff --git a/backend/src/processing/processor/xtream_series.rs b/backend/src/processing/processor/xtream_series.rs index 0d1903289..512b1b74e 100644 --- a/backend/src/processing/processor/xtream_series.rs +++ b/backend/src/processing/processor/xtream_series.rs @@ -207,6 +207,19 @@ fn queue_background_series_info( let reasons = check_resolve_reasons(resolve_options, resolve_tmdb_enabled, pli); if !reasons.is_empty() { + if log_enabled!(Level::Debug) { + let has_details = pli.has_details(); + let (has_tmdb, has_date) = match pli.header.additional_properties.as_ref() { + Some(StreamProperties::Series(props)) => { + (props.tmdb.is_some(), props.release_date.is_some()) + } + _ => (false, false), + }; + debug!( + "[Task] Creating ResolveSeries task for input {}: id={}, reasons={}, has_details={}, has_tmdb={}, has_date={}, title=\"{}\"", + input_name_arc, provider_id, reasons, has_details, has_tmdb, has_date, pli.header.title + ); + } let task = UpdateTask::ResolveSeries { id: provider_id, reason: reasons, @@ -233,8 +246,8 @@ async fn process_immediate_series_info( resolve_tmdb_enabled: bool, ) -> Vec { let input = fpl.input; - let working_dir = &ctx.config.config.load().working_dir; - let storage_path = match get_input_storage_path(&input.name, working_dir).await { + let storage_dir = &ctx.config.config.load().storage_dir; + let storage_path = match get_input_storage_path(&input.name, storage_dir).await { Ok(path) => path, Err(err) => { error!("Can't resolve series, input storage directory for input '{}' failed: {err}", input.name); @@ -624,8 +637,8 @@ pub async fn update_series_metadata( db_query: Option>>>, tmdb_and_date_present_out: Option<&AtomicBool>, ) -> Result, TuliproxError> { - let working_dir = &app_config.config.load().working_dir; - let storage_path = get_input_storage_path(&input.name, working_dir) + let storage_dir = &app_config.config.load().storage_dir; + let storage_path = get_input_storage_path(&input.name, storage_dir) .await .map_err(|e| shared::error::info_err!("Storage path error: {e}"))?; diff --git a/backend/src/processing/processor/xtream_vod.rs b/backend/src/processing/processor/xtream_vod.rs index 638b5d6a2..e1bbdf13c 100644 --- a/backend/src/processing/processor/xtream_vod.rs +++ b/backend/src/processing/processor/xtream_vod.rs @@ -5,6 +5,7 @@ use crate::model::FetchedPlaylist; use crate::model::InputSource; use crate::model::{AppConfig, ConfigTarget}; use crate::model::{ConfigInput, ConfigInputFlags}; +use crate::processing::input_cache::resolve_input_storage_path; use crate::processing::processor::playlist::PlaylistProcessingContext; use crate::processing::processor::{ create_resolve_options_function_for_xtream_target, process_foreground_retry_once, select_cancel_token, @@ -12,7 +13,6 @@ use crate::processing::processor::{ FOREGROUND_RETRY_BATCH_MAX_SIZE as RETRY_BATCH_MAX_SIZE, }; use crate::ptt::ptt_parse_title; -use crate::repository::get_input_storage_path; use crate::repository::persist_input_vod_info; use crate::repository::persist_input_vod_info_batch; use crate::repository::{xtream_get_file_path, BPlusTreeQuery}; @@ -148,15 +148,9 @@ async fn process_immediate_vod_info( resolve_tmdb_enabled: bool, ) { let input = fpl.input; - let working_dir = &ctx.config.config.load().working_dir; + let storage_dir = &ctx.config.config.load().storage_dir; - let storage_path = match get_input_storage_path(&input.name, working_dir).await { - Ok(path) => path, - Err(err) => { - error!("Can't resolve vod, input storage directory for input '{}' failed: {err}", input.name); - return; - } - }; + let storage_path = resolve_input_storage_path(storage_dir, &input.name).await; // Keep an optional read query open and reopen lazily only when needed. let xtream_path = xtream_get_file_path(&storage_path, XtreamCluster::Video); @@ -447,6 +441,26 @@ fn queue_background_vod_info( let reasons = check_resolve_reasons(resolve_options, do_probe, resolve_tmdb_enabled, pli); if !reasons.is_empty() { + if log_enabled!(Level::Debug) { + let has_details = pli.has_details(); + let (has_tmdb, has_date, has_video, has_audio) = + match pli.header.additional_properties.as_ref() { + Some(StreamProperties::Video(props)) => { + let details = props.details.as_ref(); + ( + props.tmdb.is_some(), + details.and_then(|d| d.release_date.as_ref()).is_some(), + MediaQuality::is_valid_media_info(details.and_then(|d| d.video.as_deref())), + MediaQuality::is_valid_media_info(details.and_then(|d| d.audio.as_deref())), + ) + } + _ => (false, false, false, false), + }; + debug!( + "[Task] Creating ResolveVod task for input {}: id={}, reasons={}, has_details={}, has_tmdb={}, has_date={}, has_video_info={}, has_audio_info={}, title=\"{}\"", + input.name, provider_id, reasons, has_details, has_tmdb, has_date, has_video, has_audio, pli.header.title + ); + } let task = UpdateTask::ResolveVod { id: provider_id, reason: reasons, delay: resolve_options.resolve_delay }; mgr.queue_task_background(input.name.clone(), task); @@ -509,10 +523,8 @@ pub async fn update_vod_metadata( db_query: Option>>>, tmdb_resolved_out: Option<&AtomicBool>, ) -> Result, TuliproxError> { - let working_dir = &app_config.config.load().working_dir; - let storage_path = get_input_storage_path(&input.name, working_dir) - .await - .map_err(|e| shared::error::info_err!("Storage path error: {e}"))?; + let storage_dir = &app_config.config.load().storage_dir; + let storage_path = resolve_input_storage_path(storage_dir, &input.name).await; // Check if we should skip based on input options if input.has_flag(ConfigInputFlags::XtreamSkipVod) { diff --git a/backend/src/repository/alias_repository.rs b/backend/src/repository/alias_repository.rs index 02f39519b..d093b5cf1 100644 --- a/backend/src/repository/alias_repository.rs +++ b/backend/src/repository/alias_repository.rs @@ -7,7 +7,7 @@ use log::{error, warn}; use shared::error::{string_to_io_error, to_io_error, TuliproxError}; use shared::info_err; use shared::model::{ConfigInputAliasDto, InputType}; -use shared::utils::{get_credentials_from_url, get_credentials_from_url_str, parse_timestamp, sanitize_sensitive_info, Internable, PROVIDER_SCHEME_PREFIX}; +use shared::utils::{get_credentials_from_url, get_credentials_from_url_str, parse_timestamp, sanitize_sensitive_info, Internable, BATCH_SCHEME_PREFIX, PROVIDER_SCHEME_PREFIX}; use std::io; use std::io::{BufRead, Cursor, Error}; use std::path::{Path, PathBuf}; @@ -267,21 +267,23 @@ pub async fn csv_read_inputs( } pub fn get_csv_file_path(file_uri: &str) -> Result { + // Handle batch:// scheme: strip prefix and treat remainder as file path. + if let Some(path_str) = file_uri.strip_prefix(BATCH_SCHEME_PREFIX) { + let path = Path::new(path_str); + return if path.is_absolute() { + Ok(path.to_path_buf()) + } else { + resolve_relative_path(path_str) + }; + } let raw_path = Path::new(file_uri); if raw_path.is_absolute() { return Ok(raw_path.to_path_buf()); } - if let Ok(url) = file_uri.parse::() { - if url.scheme() == "file" { - match url.to_file_path() { - Ok(path) => Ok(path), - Err(()) => Err(string_to_io_error(format!("Could not open {file_uri}"))), - } - } else { - Err(string_to_io_error(format!( - "Only file:// is supported {file_uri}" - ))) - } + if let Ok(_url) = file_uri.parse::() { + Err(string_to_io_error(format!( + "Unsupported URL scheme for batch CSV, use batch:// instead: {file_uri}" + ))) } else { resolve_relative_path(file_uri) } diff --git a/backend/src/repository/bplustree.rs b/backend/src/repository/bplustree.rs index 5171fcd47..76d8cbef3 100644 --- a/backend/src/repository/bplustree.rs +++ b/backend/src/repository/bplustree.rs @@ -1747,7 +1747,7 @@ where let mut v = Vec::with_capacity(info.len()); let original_keys = std::mem::take(&mut keys); - for (entry_key, entry_info) in original_keys.into_iter().zip(info.into_iter()) { + for (entry_key, entry_info) in original_keys.into_iter().zip(info) { if entry_info.is_tombstone() { continue; } @@ -1854,7 +1854,7 @@ where let mut filtered_info: Vec = Vec::with_capacity(info.len()); let mut last_packed_block: Option<(u64, Vec)> = None; let original_keys = std::mem::take(&mut keys); - for (entry_key, entry_info) in original_keys.into_iter().zip(info.into_iter()) { + for (entry_key, entry_info) in original_keys.into_iter().zip(info) { if entry_info.is_tombstone() { continue; } diff --git a/backend/src/repository/m3u_repository.rs b/backend/src/repository/m3u_repository.rs index 9eebbca2b..3edce6933 100644 --- a/backend/src/repository/m3u_repository.rs +++ b/backend/src/repository/m3u_repository.rs @@ -181,8 +181,8 @@ async fn persist_m3u_playlist_as_text( let Some(filename) = target_output.filename.as_ref() else { return Ok(()); }; - let working_dir = &app_config.config.load().working_dir; - let Some(m3u_filename) = utils::get_file_path(working_dir, Some(PathBuf::from(filename))) else { + let cfg = app_config.config.load(); + let Some(m3u_filename) = utils::get_file_path(&cfg.storage_dir, Some(PathBuf::from(filename))) else { return Ok(()); }; @@ -375,8 +375,8 @@ pub async fn iter_raw_m3u_input_playlist( input: &ConfigInput, cluster: Option, ) -> Option> + Send + Unpin>> { - let working_dir = &app_config.config.load().working_dir; - let storage_path = get_input_storage_path(&input.name, working_dir).await.ok()?; + let cfg = app_config.config.load(); + let storage_path = get_input_storage_path(&input.name, &cfg.storage_dir).await.ok()?; let m3u_path = get_input_m3u_playlist_file_path(&storage_path, &input.name); iter_raw_m3u_playlist::>(app_config, &m3u_path, cluster).await diff --git a/backend/src/repository/mod.rs b/backend/src/repository/mod.rs index aa9bbbcf6..7b6fa695b 100644 --- a/backend/src/repository/mod.rs +++ b/backend/src/repository/mod.rs @@ -17,6 +17,7 @@ mod library_repository; mod sorted_index; mod alias_repository; mod playlist_stream; +mod provider_dns_repository; pub use storage::*; pub use target_id_mapping::*; @@ -36,6 +37,7 @@ pub use alias_repository::*; pub use playlist_source::*; pub use library_repository::*; pub use playlist_stream::*; +pub use provider_dns_repository::*; #[cfg(test)] mod bplustree_stress; diff --git a/backend/src/repository/playlist_repository.rs b/backend/src/repository/playlist_repository.rs index 90b769d91..b99e60174 100644 --- a/backend/src/repository/playlist_repository.rs +++ b/backend/src/repository/playlist_repository.rs @@ -371,28 +371,21 @@ pub async fn persist_input_playlist(app_config: &Arc, input: &ConfigI return (playlist, None); } playlist.iter_mut().for_each(PlaylistGroup::on_load); + let cfg = app_config.config.load(); + let storage_path = match get_input_storage_path(&input.name, &cfg.storage_dir).await { + Ok(storage_path) => storage_path, + Err(err) => { + return (playlist, Some(info_err!("Error creating input storage directory for input '{}' failed: {err}", input.name))); + } + }; - match input.input_type { + match input.get_download_input_type() { InputType::Xtream | InputType::XtreamBatch => { - let working_dir = &app_config.config.load().working_dir; - let storage_path = match get_input_storage_path(&input.name, working_dir).await { - Ok(storage_path) => storage_path, - Err(err) => { - return (playlist, Some(info_err!("Error creating input storage directory for input '{}' failed: {err}", input.name))); - } - }; persist_input_xtream_playlist(app_config, &storage_path, playlist).await } InputType::M3u | InputType::M3uBatch => { // Persist M3U - let working_dir = &app_config.config.load().working_dir; - let storage_path = match get_input_storage_path(&input.name, working_dir).await { - Ok(storage_path) => storage_path, - Err(err) => { - return (playlist, Some(info_err!("Error creating input storage directory for input '{}' failed: {err}", input.name))); - } - }; let file_path = get_input_m3u_playlist_file_path(&storage_path, &input.name); if let Err(err) = persist_input_m3u_playlist(app_config, &file_path, &playlist).await { return (playlist, Some(err)); @@ -401,13 +394,6 @@ pub async fn persist_input_playlist(app_config: &Arc, input: &ConfigI } InputType::Library => { // Persist local library playlist - let working_dir = &app_config.config.load().working_dir; - let storage_path = match get_input_storage_path(&input.name, working_dir).await { - Ok(storage_path) => storage_path, - Err(err) => { - return (playlist, Some(info_err!("Error creating input storage directory for input '{}' failed: {err}", input.name))); - } - }; let file_path = get_input_local_library_playlist_file_path(&storage_path, &input.name); let (playlist, result) = persist_input_library_playlist(app_config, &file_path, playlist).await; if let Err(err) = result { @@ -420,13 +406,12 @@ pub async fn persist_input_playlist(app_config: &Arc, input: &ConfigI pub async fn load_input_playlist(ctx: &PlaylistProcessingContext, input: &ConfigInput, clusters: Option<&[XtreamCluster]>) -> Result, TuliproxError> { let app_config = &ctx.config; - let working_dir = &app_config.config.load().working_dir; - let storage_path = get_input_storage_path(&input.name, working_dir).await + let cfg = app_config.config.load(); + let storage_path = get_input_storage_path(&input.name, &cfg.storage_dir).await .map_err(|e| info_err!("Error getting input path: {e}"))?; + let disk_based_processing = cfg.disk_based_processing; - let disk_based_processing = app_config.config.load().disk_based_processing; - - match input.input_type { + match input.get_download_input_type() { InputType::Xtream | InputType::XtreamBatch => { if disk_based_processing { Ok(Box::new(XtreamDiskPlaylistSource::new(app_config, &storage_path).await)) diff --git a/backend/src/repository/provider_dns_repository.rs b/backend/src/repository/provider_dns_repository.rs new file mode 100644 index 000000000..e8402b0e2 --- /dev/null +++ b/backend/src/repository/provider_dns_repository.rs @@ -0,0 +1,434 @@ +use indexmap::IndexMap; +use log::{debug, info, warn}; +use serde::{Deserialize, Serialize}; +use shared::utils::DNS_RESOLVED_FILE; +use std::collections::{HashMap, HashSet}; +use std::io::ErrorKind; +use std::net::IpAddr; +use std::path::{Path, PathBuf}; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::Arc; +use std::time::Duration; +use tokio::fs; +use tokio::sync::mpsc; +use tokio_util::sync::CancellationToken; +use crate::api::model::AppState; +use crate::model::ConfigProvider; + +/// All providers' resolved DNS data, keyed by provider name → hostname → IPs. +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct DnsResolvedStore { + #[serde(flatten)] + pub providers: HashMap>>, +} + +static DNS_WRITER_GENERATION: AtomicU64 = AtomicU64::new(0); +const DNS_WRITER_FLUSH_INTERVAL_SECS: u64 = 2; +const DNS_WRITER_FLUSH_BATCH_THRESHOLD: usize = 32; + +pub fn next_dns_writer_generation() -> u64 { DNS_WRITER_GENERATION.fetch_add(1, Ordering::SeqCst) + 1 } + +fn is_dns_writer_generation_current(generation: u64) -> bool { + DNS_WRITER_GENERATION.load(Ordering::SeqCst) == generation +} + +pub fn dns_resolved_file_path(storage_dir: &str) -> PathBuf { + PathBuf::from(storage_dir).join(DNS_RESOLVED_FILE) +} + +#[derive(Debug)] +pub enum DnsResolvedStoreLoadError { + Read(std::io::Error), + Parse(serde_json::Error), +} + +impl std::fmt::Display for DnsResolvedStoreLoadError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Read(err) => write!(f, "read failed: {err}"), + Self::Parse(err) => write!(f, "parse failed: {err}"), + } + } +} + +pub async fn load_dns_resolved_store_from_path(path: &Path) -> Result, DnsResolvedStoreLoadError> { + match fs::read_to_string(path).await { + Ok(data) => serde_json::from_str(&data).map(Some).map_err(DnsResolvedStoreLoadError::Parse), + Err(err) if err.kind() == ErrorKind::NotFound => Ok(None), + Err(err) => Err(DnsResolvedStoreLoadError::Read(err)), + } +} + +pub async fn persist_dns_resolved_store(path: &Path, store: &DnsResolvedStore) -> Result<(), String> { + let json = match serde_json::to_string_pretty(store) { + Ok(value) => value, + Err(err) => return Err(format!("serialize failed: {err}")), + }; + + if let Some(parent) = path.parent() { + fs::create_dir_all(parent) + .await + .map_err(|err| format!("create parent dir '{}' failed: {err}", parent.display()))?; + } + + let tmp_path = path.with_extension("json.tmp"); + fs::write(&tmp_path, json) + .await + .map_err(|err| format!("write temp file '{}' failed: {err}", tmp_path.display()))?; + + if let Err(err) = fs::rename(&tmp_path, path).await { + #[cfg(windows)] + { + if fs::remove_file(path).await.is_ok() && fs::rename(&tmp_path, path).await.is_ok() { + debug!( + "Persisted DNS resolved store to '{}' (providers={})", + path.display(), + store.providers.len() + ); + return Ok(()); + } + } + let _ = fs::remove_file(&tmp_path).await; + return Err(format!( + "rename temp file '{}' → '{}' failed: {err}", + tmp_path.display(), + path.display() + )); + } + + debug!( + "Persisted DNS resolved store to '{}' (providers={})", + path.display(), + store.providers.len() + ); + Ok(()) +} + +/// Load persisted DNS resolved data and seed the in-memory caches for all providers. +/// Called at startup and after config reloads, before DNS background tasks run. +pub async fn load_persisted_dns_resolved(app_state: &std::sync::Arc) { + let storage_dir = app_state.app_config.config.load().storage_dir.clone(); + let path = dns_resolved_file_path(&storage_dir); + + let store = match load_dns_resolved_store_from_path(&path).await { + Ok(Some(store)) => store, + Ok(None) => { + debug!("No persisted DNS resolved file found at '{}'", path.display()); + return; + } + Err(err) => { + warn!("Failed to load persisted DNS resolved file '{}': {err}", path.display()); + return; + } + }; + + let sources = app_state.app_config.sources.load(); + let mut seeded_count = 0usize; + for (provider_name, hosts) in &store.providers { + let Some(provider) = sources.get_provider_by_name(provider_name) else { + debug!("Persisted DNS resolved data for unknown provider '{provider_name}', skipping"); + continue; + }; + if !provider.get_dns_config().is_some_and(|cfg| cfg.enabled) { + debug!("Persisted DNS resolved data for disabled-dns provider '{provider_name}', skipping"); + continue; + } + for (host, ips) in hosts { + if !ips.is_empty() { + // Only seed if the cache is currently empty for this host (don't overwrite runtime data). + if provider.dns_cache.ip_count(host) == 0 { + provider.dns_cache.store_resolved(host, ips.clone()); + seeded_count += 1; + } + } + } + } + + if seeded_count > 0 { + info!("Seeded {seeded_count} host(s) from persisted DNS resolved file '{}'", path.display()); + } +} + +fn prune_store_to_runtime_enabled_providers(app_state: &Arc, store: &mut DnsResolvedStore) -> usize { + let sources = app_state.app_config.sources.load(); + let enabled_provider_names = sources + .provider + .iter() + .filter(|provider| provider.get_dns_config().is_some_and(|cfg| cfg.enabled)) + .map(|provider| provider.name.to_string()) + .collect::>(); + prune_store_to_enabled_provider_names(store, &enabled_provider_names) +} + +fn prune_store_to_enabled_provider_names( + store: &mut DnsResolvedStore, + enabled_provider_names: &HashSet, +) -> usize { + let before = store.providers.len(); + store + .providers + .retain(|provider_name, _| enabled_provider_names.contains(provider_name)); + before.saturating_sub(store.providers.len()) +} + +fn is_provider_dns_enabled_in_runtime(app_state: &Arc, provider_name: &str) -> bool { + let sources = app_state.app_config.sources.load(); + sources + .get_provider_by_name(provider_name) + .is_some_and(|provider| provider.get_dns_config().is_some_and(|cfg| cfg.enabled)) +} + +pub async fn prune_persisted_dns_resolved_to_runtime(app_state: &Arc) { + let storage_dir = app_state.app_config.config.load().storage_dir.clone(); + let path = dns_resolved_file_path(&storage_dir); + let mut store = match load_dns_resolved_store_from_path(&path).await { + Ok(Some(existing)) => existing, + Ok(None) => return, + Err(err) => { + warn!( + "Failed to load DNS resolved store for runtime-prune '{}': {err}", + path.display() + ); + return; + } + }; + + let removed = prune_store_to_runtime_enabled_providers(app_state, &mut store); + if removed == 0 { + return; + } + + if let Err(err) = persist_dns_resolved_store(&path, &store).await { + warn!( + "Failed to persist DNS resolved store after runtime-prune '{}': {err}", + path.display() + ); + } else { + info!( + "Pruned {removed} stale provider(s) from DNS resolved store '{}'", + path.display() + ); + } +} + +#[derive(Debug)] +pub struct DnsResolvedWriteUpdate { + pub provider_name: String, + pub resolved: IndexMap>, +} + +pub type DnsResolvedWriteTx = mpsc::Sender; + +#[allow(clippy::too_many_lines)] +pub fn spawn_dns_resolved_writer( + app_state: Arc, + cancel: CancellationToken, + generation: u64, +) -> DnsResolvedWriteTx { + let storage_dir = app_state.app_config.config.load().storage_dir.clone(); + let path = dns_resolved_file_path(&storage_dir); + let (tx, mut rx) = mpsc::channel::(256); + + tokio::spawn(async move { + let mut store = match load_dns_resolved_store_from_path(&path).await { + Ok(Some(existing)) => existing, + Ok(None) => DnsResolvedStore::default(), + Err(err) => { + warn!("Failed to load existing DNS resolved store '{}': {err}", path.display()); + DnsResolvedStore::default() + } + }; + let removed_on_start = prune_store_to_runtime_enabled_providers(&app_state, &mut store); + if removed_on_start > 0 { + info!( + "Pruned {removed_on_start} stale provider(s) from DNS resolved store on writer start '{}'", + path.display() + ); + if let Err(err) = persist_dns_resolved_store(&path, &store).await { + warn!( + "Failed to persist DNS resolved store after startup prune '{}': {err}", + path.display() + ); + } + } + let mut dirty = false; + let mut dirty_updates = 0usize; + let mut flush_before_exit = false; + let mut flush_timer = std::pin::pin!(tokio::time::sleep(Duration::from_secs( + DNS_WRITER_FLUSH_INTERVAL_SECS, + ))); + let mut flush_timer_active = false; + + loop { + if !is_dns_writer_generation_current(generation) { + debug!( + "Stopping stale DNS resolved writer generation {} for '{}'", + generation, + path.display() + ); + break; + } + tokio::select! { + () = cancel.cancelled() => { + debug!("Stopping DNS resolved writer task for '{}'", path.display()); + flush_before_exit = true; + break; + } + maybe_update = rx.recv() => { + let Some(update) = maybe_update else { + debug!("DNS resolved writer channel closed for '{}'", path.display()); + flush_before_exit = true; + break; + }; + if !is_dns_writer_generation_current(generation) { + debug!( + "Dropping update for stale DNS resolved writer generation {} on '{}'", + generation, + path.display() + ); + break; + } + + if update.resolved.is_empty() || !is_provider_dns_enabled_in_runtime(&app_state, update.provider_name.as_str()) { + store.providers.remove(update.provider_name.as_str()); + } else { + store.providers.insert(update.provider_name, update.resolved); + } + + dirty = true; + dirty_updates = dirty_updates.saturating_add(1); + + if dirty_updates >= DNS_WRITER_FLUSH_BATCH_THRESHOLD { + let _ = prune_store_to_runtime_enabled_providers(&app_state, &mut store); + if let Err(err) = persist_dns_resolved_store(&path, &store).await { + warn!("Failed to persist DNS resolved store '{}': {err}", path.display()); + } else { + dirty = false; + dirty_updates = 0; + flush_timer_active = false; + } + } else if !flush_timer_active { + flush_timer.as_mut().reset( + tokio::time::Instant::now() + Duration::from_secs(DNS_WRITER_FLUSH_INTERVAL_SECS), + ); + flush_timer_active = true; + } + } + () = &mut flush_timer, if flush_timer_active => { + flush_timer_active = false; + if dirty { + let _ = prune_store_to_runtime_enabled_providers(&app_state, &mut store); + if let Err(err) = persist_dns_resolved_store(&path, &store).await { + warn!("Failed to persist DNS resolved store '{}': {err}", path.display()); + } else { + dirty = false; + dirty_updates = 0; + } + } + } + } + } + + if flush_before_exit && dirty { + let _ = prune_store_to_runtime_enabled_providers(&app_state, &mut store); + if let Err(err) = persist_dns_resolved_store(&path, &store).await { + warn!( + "Failed to persist DNS resolved store during writer shutdown '{}': {err}", + path.display() + ); + } + } + }); + + tx +} + +pub async fn queue_provider_resolved_snapshot(writer_tx: &DnsResolvedWriteTx, provider: &Arc) { + let update = DnsResolvedWriteUpdate { + provider_name: provider.name.to_string(), + resolved: provider.snapshot_resolved_ordered(), + }; + if let Err(err) = writer_tx.send(update).await { + warn!("Failed to queue DNS resolved update for provider '{}': {err}", provider.name); + } +} + +#[cfg(test)] +mod tests { + use super::{ + is_dns_writer_generation_current, load_dns_resolved_store_from_path, next_dns_writer_generation, + persist_dns_resolved_store, prune_store_to_enabled_provider_names, DnsResolvedStore, DnsResolvedStoreLoadError, + }; + use indexmap::IndexMap; + use std::collections::{HashMap, HashSet}; + use std::net::IpAddr; + + #[tokio::test] + async fn load_dns_resolved_store_reports_parse_error() { + let dir = tempfile::tempdir().expect("temp dir should be created"); + let path = dir.path().join("provider_dns_resolved.json"); + tokio::fs::write(&path, "{ not-json").await.expect("test file should be written"); + + let result = load_dns_resolved_store_from_path(&path).await; + assert!(matches!(result, Err(DnsResolvedStoreLoadError::Parse(_)))); + } + + #[tokio::test] + async fn persist_and_load_dns_resolved_store_roundtrip() { + let dir = tempfile::tempdir().expect("temp dir should be created"); + let path = dir.path().join("provider_dns_resolved.json"); + let mut providers = HashMap::new(); + providers.insert( + "provider-a".to_string(), + IndexMap::from([( + "example.com".to_string(), + vec!["203.0.113.10".parse::().expect("valid ip")], + )]), + ); + let store = DnsResolvedStore { providers }; + + persist_dns_resolved_store(&path, &store).await.expect("persist should succeed"); + let loaded = load_dns_resolved_store_from_path(&path) + .await + .expect("load should succeed") + .expect("store should exist"); + + assert_eq!(loaded.providers.len(), 1); + assert_eq!( + loaded + .providers + .get("provider-a") + .and_then(|hosts| hosts.get("example.com")), + Some(&vec!["203.0.113.10".parse::().expect("valid ip")]) + ); + } + + #[test] + fn generation_switch_marks_previous_writer_as_stale() { + let old_generation = next_dns_writer_generation(); + let new_generation = next_dns_writer_generation(); + assert_ne!(old_generation, new_generation); + assert!(!is_dns_writer_generation_current(old_generation)); + assert!(is_dns_writer_generation_current(new_generation)); + } + + #[test] + fn prune_store_to_enabled_provider_names_removes_stale_entries() { + let mut providers = HashMap::new(); + providers.insert( + "provider-a".to_string(), + IndexMap::from([("example.com".to_string(), vec!["203.0.113.10".parse::().expect("valid ip")])]), + ); + providers.insert( + "provider-b".to_string(), + IndexMap::from([("example.org".to_string(), vec!["203.0.113.20".parse::().expect("valid ip")])]), + ); + let mut store = DnsResolvedStore { providers }; + let enabled = HashSet::from(["provider-a".to_string()]); + + let removed = prune_store_to_enabled_provider_names(&mut store, &enabled); + assert_eq!(removed, 1); + assert!(store.providers.contains_key("provider-a")); + assert!(!store.providers.contains_key("provider-b")); + } +} diff --git a/backend/src/repository/storage.rs b/backend/src/repository/storage.rs index 9cda3ac57..1297d89be 100644 --- a/backend/src/repository/storage.rs +++ b/backend/src/repository/storage.rs @@ -25,7 +25,7 @@ pub async fn ensure_target_storage_path(cfg: &Config, target_name: &str) -> Resu } pub fn get_target_storage_path(cfg: &Config, target_name: &str) -> Option { - utils::get_file_path(&cfg.working_dir, Some(std::path::PathBuf::from(target_name.replace(' ', "_")))) + utils::get_file_path(&cfg.storage_dir, Some(std::path::PathBuf::from(target_name.replace(' ', "_")))) } pub fn sanitize_name(name: &str) -> String { @@ -34,27 +34,27 @@ pub fn sanitize_name(name: &str) -> String { .collect() } -pub fn build_input_storage_path(input_name: &str, working_dir: &str) -> PathBuf { +pub fn build_input_storage_path(input_name: &str, storage_dir: &str) -> PathBuf { let sanitized_name: String = sanitize_name(input_name); let name = concat_string!(cap = 6 + sanitized_name.len(); "input_", &sanitized_name); - Path::new(working_dir).join(name) + Path::new(storage_dir).join(name) } -pub async fn get_input_storage_path(input_name: &str, working_dir: &str) -> std::io::Result { - let path = build_input_storage_path(input_name, working_dir); +pub async fn get_input_storage_path(input_name: &str, storage_dir: &str) -> std::io::Result { + let path = build_input_storage_path(input_name, storage_dir); // Create the directory and return the path or propagate the error tokio::fs::create_dir_all(&path).await.map(|()| path) } pub async fn ensure_input_storage_path(cfg: &Config, input_name: &str) -> Result { - get_input_storage_path(input_name, &cfg.working_dir).await + get_input_storage_path(input_name, &cfg.storage_dir).await .map_err(|err| { notify_err!("Failed to save input data, can't create directory for input {input_name}: {err}") }) } -pub fn get_geoip_path(working_dir: &str) -> PathBuf { - Path::new(working_dir).join("geoip.db") +pub fn get_geoip_path(storage_dir: &str) -> PathBuf { + Path::new(storage_dir).join("geoip.db") } pub fn get_file_path_for_db_index(db_path: &Path) -> PathBuf { diff --git a/backend/src/repository/strm_repository.rs b/backend/src/repository/strm_repository.rs index aee5772f1..4ce4174c0 100644 --- a/backend/src/repository/strm_repository.rs +++ b/backend/src/repository/strm_repository.rs @@ -850,7 +850,7 @@ pub async fn write_strm_playlist( let config = app_config.config.load(); let Some(root_path) = crate::utils::get_file_path( - &config.working_dir, + &config.storage_dir, Some(std::path::PathBuf::from(&target_output.directory)), ) else { return info_err_res!("Failed to get file path for {}",target_output.directory); diff --git a/backend/src/repository/user_repository.rs b/backend/src/repository/user_repository.rs index ff29f42ff..d00d828cf 100644 --- a/backend/src/repository/user_repository.rs +++ b/backend/src/repository/user_repository.rs @@ -517,7 +517,9 @@ mod tests { hdhomerun: Arc::new(ArcSwapAny::default()), api_proxy: Arc::new(ArcSwapAny::default()), paths: Arc::new(ArcSwap::from(Arc::new(ConfigPaths { + home_path: String::new(), config_path: temp_dir().to_string_lossy().to_string(), + storage_path: temp_dir().to_string_lossy().to_string(), config_file_path: String::new(), sources_file_path: String::new(), mapping_file_path: None, diff --git a/backend/src/repository/xtream_repository.rs b/backend/src/repository/xtream_repository.rs index b899d5da9..05111e823 100644 --- a/backend/src/repository/xtream_repository.rs +++ b/backend/src/repository/xtream_repository.rs @@ -654,8 +654,8 @@ pub async fn iter_raw_xtream_target_playlist(app_config: &AppConfig, target: &Co pub async fn iter_raw_xtream_input_playlist(app_config: &AppConfig, input: &ConfigInput, cluster: XtreamCluster) -> Option + Send + Unpin>> { let config = app_config.config.load(); - let working_dir = &config.working_dir; - let storage_path = get_input_storage_path(&input.name, working_dir).await.ok()?; + let storage_dir = &config.storage_dir; + let storage_path = get_input_storage_path(&input.name, storage_dir).await.ok()?; let xtream_path = xtream_get_file_path(&storage_path, cluster); iter_raw_xtream_playlist(app_config, &xtream_path).await @@ -1211,15 +1211,14 @@ fn needs_preserved_stream_property_merge( let preserve_info_details = old_stream_props.has_details() && !needs_update_info_details(new_stream_props, old_stream_props); match (new_stream_props, old_stream_props) { - (StreamProperties::Video(v_new), StreamProperties::Video(v_old)) - if preserve_info_details && v_old.details.is_some() => - { - v_new.details != v_old.details + (StreamProperties::Video(v_new), StreamProperties::Video(v_old)) => { + (preserve_info_details && v_old.details.is_some() && v_new.details != v_old.details) + || (v_new.tmdb.is_none() && v_old.tmdb.is_some()) } - (StreamProperties::Series(s_new), StreamProperties::Series(s_old)) - if preserve_info_details && s_old.details.is_some() => - { - s_new.details != s_old.details + (StreamProperties::Series(s_new), StreamProperties::Series(s_old)) => { + (preserve_info_details && s_old.details.is_some() && s_new.details != s_old.details) + || (s_new.tmdb.is_none() && s_old.tmdb.is_some()) + || (s_new.release_date.is_none() && s_old.release_date.is_some()) } (StreamProperties::Live(l_new), StreamProperties::Live(l_old)) => { (l_new.video.is_none() && l_old.video.is_some()) @@ -1245,26 +1244,41 @@ pub(crate) fn merge_preserved_stream_properties( old_stream_props.has_details() && !needs_update_info_details(new_stream_props, old_stream_props); match (new_stream_props, old_stream_props) { - (StreamProperties::Video(v_new), StreamProperties::Video(v_old)) - if preserve_info_details && v_old.details.is_some() => - { - if v_new.details == v_old.details { - false - } else { - v_new.details.clone_from(&v_old.details); - true - } + (StreamProperties::Video(v_new), StreamProperties::Video(v_old)) => { + let mut changed = false; + + if preserve_info_details && v_old.details.is_some() && v_new.details != v_old.details { + v_new.details.clone_from(&v_old.details); + changed = true; } - (StreamProperties::Series(s_new), StreamProperties::Series(s_old)) - if preserve_info_details && s_old.details.is_some() => - { - if s_new.details == s_old.details { - false - } else { - s_new.details.clone_from(&s_old.details); - true - } + + if v_new.tmdb.is_none() && v_old.tmdb.is_some() { + v_new.tmdb = v_old.tmdb; + changed = true; } + + changed + } + (StreamProperties::Series(s_new), StreamProperties::Series(s_old)) => { + let mut changed = false; + + if preserve_info_details && s_old.details.is_some() && s_new.details != s_old.details { + s_new.details.clone_from(&s_old.details); + changed = true; + } + + if s_new.tmdb.is_none() && s_old.tmdb.is_some() { + s_new.tmdb = s_old.tmdb; + changed = true; + } + + if s_new.release_date.is_none() && s_old.release_date.is_some() { + s_new.release_date.clone_from(&s_old.release_date); + changed = true; + } + + changed + } (StreamProperties::Live(l_new), StreamProperties::Live(l_old)) => { let mut changed = false; @@ -1586,6 +1600,49 @@ mod tests { } } + #[test] + fn merge_preserves_missing_video_tmdb() { + let mut new_props = StreamProperties::Video(Box::new(VideoStreamProperties { + tmdb: None, + ..VideoStreamProperties::default() + })); + let old_props = StreamProperties::Video(Box::new(VideoStreamProperties { + tmdb: Some(317981), + ..VideoStreamProperties::default() + })); + + let changed = merge_preserved_stream_properties(&mut new_props, &old_props); + assert!(changed); + match new_props { + StreamProperties::Video(video) => assert_eq!(video.tmdb, Some(317981)), + _ => panic!("expected video properties"), + } + } + + #[test] + fn merge_preserves_missing_series_tmdb_and_release_date() { + let mut new_props = StreamProperties::Series(Box::new(SeriesStreamProperties { + tmdb: None, + release_date: None, + ..SeriesStreamProperties::default() + })); + let old_props = StreamProperties::Series(Box::new(SeriesStreamProperties { + tmdb: Some(12345), + release_date: Some("2015-01-01".into()), + ..SeriesStreamProperties::default() + })); + + let changed = merge_preserved_stream_properties(&mut new_props, &old_props); + assert!(changed); + match new_props { + StreamProperties::Series(series) => { + assert_eq!(series.tmdb, Some(12345)); + assert_eq!(series.release_date.as_deref(), Some("2015-01-01")); + } + _ => panic!("expected series properties"), + } + } + fn make_live_item( provider_id: u32, video: Option<&str>, diff --git a/backend/src/utils/file/config_reader.rs b/backend/src/utils/file/config_reader.rs index f3feb8a3a..48879ecb8 100644 --- a/backend/src/utils/file/config_reader.rs +++ b/backend/src/utils/file/config_reader.rs @@ -22,7 +22,6 @@ use shared::model::{ MsgKind, PatternTemplate, SourcesConfigDto, TargetUserDto, TemplateDefinitionDto, }; use shared::utils::{generate_default_access_secret, generate_default_encrypt_secret, CONSTANTS, PROVIDER_SCHEME_PREFIX, TEMPLATE_FILE}; -use std::borrow::Cow; use std::collections::{HashMap, HashSet}; use std::env; use std::fs::File; @@ -112,14 +111,19 @@ async fn parse_sources_file_from_path( .map_err(|join_err| info_err!("Failed to read sources-config file: {join_err}"))? } -pub fn resolve_template_and_mapping_paths<'a>(paths: &'a ConfigPaths, template_path: Option<&'a str>, mapping_path: Option<&'a str>) -> (Cow<'a, str>, Cow<'a, str>) { - // Resolve effective paths for templates and mappings (with robust fallbacks) - let effective_template_path = paths.template_file_path.as_deref() - .or(template_path).map_or_else(|| Cow::Owned(utils::get_default_templates_path(&paths.config_path)), Cow::Borrowed); - - let effective_mapping_path = paths.mapping_file_path.as_deref() - .or(mapping_path).map_or_else(|| Cow::Owned(utils::get_default_mappings_path(&paths.config_path)), Cow::Borrowed); - +pub fn resolve_template_and_mapping_paths( + paths: &ConfigPaths, + template_path: Option<&str>, + mapping_path: Option<&str>, +) -> (String, String) { + let effective_template_path = utils::resolve_template_file_path( + paths.config_path.as_str(), + paths.template_file_path.as_deref().or(template_path), + ); + let effective_mapping_path = utils::resolve_mapping_file_path( + paths.config_path.as_str(), + paths.mapping_file_path.as_deref().or(mapping_path), + ); (effective_template_path, effective_mapping_path) } @@ -322,14 +326,15 @@ pub async fn read_app_config_dto( let config = read_config_file(config_file, resolve_env, include_computed)?; // Resolve effective paths for templates and mappings (with robust fallbacks) - let (effective_template_path,effective_mapping_path) = resolve_template_and_mapping_paths(paths, config.template_path.as_deref(), config.mapping_path.as_deref()); + let (effective_template_path, effective_mapping_path) = + resolve_template_and_mapping_paths(paths, config.template_path.as_deref(), config.mapping_path.as_deref()); let mut sources = parse_sources_file_from_path(&PathBuf::from(sources_file), resolve_env).await?; - let mut mappings = read_mappings_file_unprepared(effective_mapping_path.as_ref(), resolve_env)? + let mut mappings = read_mappings_file_unprepared(&effective_mapping_path, resolve_env)? .map(|(_, mapping)| mapping); let template_bundle = read_templates( - Some(effective_template_path.as_ref()), + Some(&effective_template_path), resolve_env, sources.templates.as_deref(), mappings @@ -447,7 +452,7 @@ pub async fn get_batch_aliases( if url.starts_with(PROVIDER_SCHEME_PREFIX) { return info_err_res!( "Batch input type '{input_type}' does not support provider:// URLs. \ -Use a local CSV path (absolute/relative) or file:// URL." +Use a batch:// URL or a local CSV path (absolute/relative)." ); } @@ -505,26 +510,25 @@ pub async fn read_initial_app_config( let config_dto = read_config_file(config_file, resolve_env, include_computed)?; - if paths.mapping_file_path.is_none() { - let mut path = config_dto.mapping_path.as_ref().map_or_else( - || utils::get_default_mappings_path(config_path), - ToString::to_string, - ); - if resolve_env { - path = resolve_env_var(&path); - } - paths.mapping_file_path.replace(path); - } - if paths.template_file_path.is_none() { - let mut path = config_dto.template_path.as_ref().map_or_else( - || utils::get_default_templates_path(config_path), - ToString::to_string, - ); - if resolve_env { - path = resolve_env_var(&path); - } - paths.template_file_path.replace(path); - } + let configured_mapping_path = paths + .mapping_file_path + .as_deref() + .or(config_dto.mapping_path.as_deref()) + .map(|path| if resolve_env { resolve_env_var(path) } else { path.to_string() }); + let configured_template_path = paths + .template_file_path + .as_deref() + .or(config_dto.template_path.as_deref()) + .map(|path| if resolve_env { resolve_env_var(path) } else { path.to_string() }); + + paths.mapping_file_path = Some(utils::resolve_mapping_file_path( + config_path, + configured_mapping_path.as_deref(), + )); + paths.template_file_path = Some(utils::resolve_template_file_path( + config_path, + configured_template_path.as_deref(), + )); let mut sources_dto = parse_sources_file_from_path(&PathBuf::from(sources_file), resolve_env).await?; @@ -539,7 +543,7 @@ pub async fn read_initial_app_config( }; let template_bundle = read_templates( - paths.template_file_path.as_deref().or(config_dto.template_path.as_deref()), + paths.template_file_path.as_deref(), resolve_env, sources_dto.templates.as_deref(), mappings_dto @@ -562,7 +566,8 @@ pub async fn read_initial_app_config( let sources: SourcesConfig = SourcesConfig::try_from(sources_dto)?; let mut config: Config = Config::from(config_dto); - config.prepare(config_path)?; + config.prepare(config_path, paths.home_path.as_str())?; + paths.storage_path.clone_from(&config.storage_dir); config.update_runtime(); let mut app_config = AppConfig { diff --git a/backend/src/utils/file/file_utils.rs b/backend/src/utils/file/file_utils.rs index 3883dfc0e..894eac6d8 100644 --- a/backend/src/utils/file/file_utils.rs +++ b/backend/src/utils/file/file_utils.rs @@ -67,17 +67,27 @@ pub fn get_home_path() -> PathBuf { } pub fn get_default_web_root_path() -> PathBuf { - if let Some(web_root) = env::var(DEFAULT_WEB_ROOT_ENV_VAR) - .ok() - .filter(|p| !p.trim().is_empty()) { - return PathBuf::from(web_root) + get_default_web_root_path_for_home(get_home_path().as_path()) +} + +pub fn get_default_web_root_path_for_home(home_path: &Path) -> PathBuf { + if let Some(web_root) = env::var(DEFAULT_WEB_ROOT_ENV_VAR).ok().filter(|p| !p.trim().is_empty()) { + let configured = PathBuf::from(web_root); + return if configured.is_absolute() { + configured + } else { + home_path.join(configured) + }; } - get_exe_path().join(DEFAULT_WEB_DIR) + get_default_path_for_home(home_path, DEFAULT_WEB_DIR) } pub fn get_default_path(file: &str) -> PathBuf { - let home_path = get_home_path(); + get_default_path_for_home(get_home_path().as_path(), file) +} + +pub fn get_default_path_for_home(home_path: &Path, file: &str) -> PathBuf { home_path.join(file) } @@ -116,11 +126,52 @@ pub fn get_default_templates_path(config_path: &str) -> String { get_default_file_path(config_path, TEMPLATE_FILE) } +fn resolve_config_scoped_path(config_path: &str, candidate: &str) -> String { + let candidate_path = PathBuf::from(candidate.trim()).clean(); + if candidate_path.is_absolute() { + return candidate_path.to_string_lossy().to_string(); + } + + let config_dir = PathBuf::from(config_path).clean(); + let relative = if let Some(config_dir_name) = config_dir.file_name() { + candidate_path + .strip_prefix(Path::new(config_dir_name)) + .map_or_else(|_| candidate_path.clone(), Path::to_path_buf) + } else { + candidate_path.clone() + }; + + config_dir.join(relative).clean().to_string_lossy().to_string() +} + +#[inline] +pub fn resolve_mapping_file_path(config_path: &str, mapping_path: Option<&str>) -> String { + let configured = mapping_path.map(str::trim).filter(|path| !path.is_empty()).map(ToString::to_string); + let candidate = if shared::utils::is_blank_or_default_mapping_path(&configured) { + get_default_mappings_path(config_path) + } else { + configured.unwrap_or_else(|| get_default_mappings_path(config_path)) + }; + resolve_config_scoped_path(config_path, &candidate) +} + +#[inline] +pub fn resolve_template_file_path(config_path: &str, template_path: Option<&str>) -> String { + let configured = template_path.map(str::trim).filter(|path| !path.is_empty()).map(ToString::to_string); + let candidate = if shared::utils::is_blank_or_default_template_path(&configured) { + get_default_templates_path(config_path) + } else { + configured.unwrap_or_else(|| get_default_templates_path(config_path)) + }; + resolve_config_scoped_path(config_path, &candidate) +} + #[inline] pub fn resolve_template_persist_file_path(template_path: Option<&str>, config_path: &str) -> String { - let candidate = template_path + let raw_candidate = template_path .filter(|path| !path.trim().is_empty()) .map_or_else(|| get_default_templates_path(config_path), ToString::to_string); + let candidate = resolve_template_file_path(config_path, Some(raw_candidate.as_str())); let path = PathBuf::from(&candidate); if path.exists() { @@ -130,8 +181,8 @@ pub fn resolve_template_persist_file_path(template_path: Option<&str>, config_pa return path.to_string_lossy().to_string(); } - let looks_like_directory = candidate.ends_with('/') - || candidate.ends_with('\\') + let looks_like_directory = raw_candidate.ends_with('/') + || raw_candidate.ends_with('\\') || path.extension().is_none() || path.extension().is_some_and(|ext| ext == "d"); @@ -329,11 +380,11 @@ where Ok(()) } -pub fn prepare_file_path(persist: Option<&str>, working_dir: &str, action: &str) -> Option { +pub fn prepare_file_path(persist: Option<&str>, storage_dir: &str, action: &str) -> Option { let persist_file: Option = persist.map(|persist_path| prepare_persist_path(persist_path, action)); if persist_file.is_some() { - let file_path = get_file_path(working_dir, persist_file); + let file_path = get_file_path(storage_dir, persist_file); debug_if_enabled!("persist to file: {}", file_path.as_ref().map_or(Cow::from("?"), |p| p.to_string_lossy())); file_path } else { @@ -348,27 +399,28 @@ pub fn read_file_as_bytes(path: &Path) -> std::io::Result> { Ok(buffer) } -pub fn make_absolute_path(path: &str, working_dir: &str) -> String { +pub fn make_absolute_path(path: &str, storage_dir: &str) -> String { let rpb = std::path::PathBuf::from(path); - let pathbuf = make_path_absolute(&rpb, working_dir); + let pathbuf = make_path_absolute(&rpb, storage_dir); pathbuf.to_str().unwrap_or_default().to_string() } -pub fn make_path_absolute(rpb: &Path, working_dir: &str) -> PathBuf { +pub fn make_path_absolute(rpb: &Path, storage_dir: &str) -> PathBuf { if rpb.is_relative() { - let mut rpb2 = std::path::PathBuf::from(working_dir).join(rpb); - if !rpb2.exists() { - rpb2 = get_exe_path().join(rpb); + if !storage_dir.trim().is_empty() { + return std::path::PathBuf::from(storage_dir).join(rpb).clean(); } - if !rpb2.exists() { - let cwd = std::env::current_dir(); - if let Ok(cwd_path) = cwd { - rpb2 = cwd_path.join(rpb); - } + + let exe_based = get_exe_path().join(rpb).clean(); + if exe_based.is_absolute() { + return exe_based; } - if rpb2.exists() { - return rpb2.clean(); + + if let Ok(cwd_path) = std::env::current_dir() { + return cwd_path.join(rpb).clean(); } + + return exe_based; } rpb.to_path_buf() } @@ -473,7 +525,11 @@ pub fn get_file_extension(path: &str) -> Option { #[cfg(test)] mod tests { - use super::normalize_string_path; + use super::{ + normalize_string_path, resolve_mapping_file_path, resolve_template_file_path, + resolve_template_persist_file_path, + }; + use std::path::PathBuf; #[test] fn test_simple_relative_path() { @@ -509,4 +565,40 @@ mod tests { let normalized = normalize_string_path(input); assert_eq!(normalized, ""); } + + #[test] + fn test_resolve_template_default_path_uses_config_dir() { + let config_path = std::env::temp_dir().join("tuliprox").join("settings").join("config"); + let resolved = resolve_template_file_path( + config_path.to_string_lossy().as_ref(), + Some("./config/template.yml"), + ); + let expected = config_path.join("template.yml"); + assert_eq!(PathBuf::from(resolved), expected); + } + + #[test] + fn test_resolve_template_directory_path_uses_config_dir() { + let config_path = std::env::temp_dir().join("tuliprox").join("settings").join("config"); + let resolved = resolve_template_file_path(config_path.to_string_lossy().as_ref(), Some("template.d")); + let expected = config_path.join("template.d"); + assert_eq!(PathBuf::from(resolved), expected); + } + + #[test] + fn test_resolve_mapping_directory_path_uses_config_dir() { + let config_path = std::env::temp_dir().join("tuliprox").join("settings").join("config"); + let resolved = resolve_mapping_file_path(config_path.to_string_lossy().as_ref(), Some("mapping.d")); + let expected = config_path.join("mapping.d"); + assert_eq!(PathBuf::from(resolved), expected); + } + + #[test] + fn test_resolve_template_persist_file_for_directory_path() { + let config_path = std::env::temp_dir().join("tuliprox").join("settings").join("config"); + let resolved = + resolve_template_persist_file_path(Some("template.d"), config_path.to_string_lossy().as_ref()); + let expected = config_path.join("template.d").join("template.yml"); + assert_eq!(PathBuf::from(resolved), expected); + } } diff --git a/backend/src/utils/file/template_reader.rs b/backend/src/utils/file/template_reader.rs index c02780319..5384eb55c 100644 --- a/backend/src/utils/file/template_reader.rs +++ b/backend/src/utils/file/template_reader.rs @@ -1,28 +1,47 @@ use crate::utils::traverse_dir; use crate::utils::{config_file_reader, open_file}; -use log::warn; +use log::{debug, warn}; use shared::error::{info_err_res, TuliproxError}; use shared::info_err; use shared::model::TemplateDefinitionDto; +use std::io::ErrorKind; use std::path::{Path, PathBuf}; fn read_template_definition( template_file: &Path, resolve_var: bool, ) -> Result, TuliproxError> { - if let Ok(file) = open_file(template_file) { - let maybe_definition: Result = - serde_saphyr::from_reader(config_file_reader(file, resolve_var)); - return match maybe_definition { - Ok(definition) => Ok(Some(definition)), - Err(err) => info_err_res!("{err}"), - }; + match open_file(template_file) { + Ok(file) => { + let maybe_definition: Result = + serde_saphyr::from_reader(config_file_reader(file, resolve_var)); + match maybe_definition { + Ok(definition) => Ok(Some(definition)), + Err(err) => info_err_res!( + "Failed to parse template file {}: {err}", + template_file.to_string_lossy() + ), + } + } + Err(err) => { + if err.kind() == ErrorKind::NotFound { + debug!( + "Optional template file not found: {}", + template_file.to_str().unwrap_or("?") + ); + Ok(None) + } else { + warn!( + "Can't read template file {}: {err}", + template_file.to_str().unwrap_or("?") + ); + info_err_res!( + "Can't read template file {}: {err}", + template_file.to_string_lossy() + ) + } + } } - warn!( - "Can't read template file: {}", - template_file.to_str().unwrap_or("?") - ); - Ok(None) } fn merge_template_definitions(definitions: Vec) -> Option { @@ -104,11 +123,19 @@ pub fn read_templates_file( } } Err(err) => { - warn!( - "Can't read template path metadata for {}: {err}", - path.to_string_lossy() - ); - Ok(None) + if err.kind() == ErrorKind::NotFound { + debug!("Optional template path not found: {}", path.to_string_lossy()); + Ok(None) + } else { + warn!( + "Can't read template path metadata for {}: {err}", + path.to_string_lossy() + ); + info_err_res!( + "Can't read template path metadata for {}: {err}", + path.to_string_lossy() + ) + } } } } diff --git a/backend/src/utils/network/epg.rs b/backend/src/utils/network/epg.rs index ecea36de7..ce0f811f3 100644 --- a/backend/src/utils/network/epg.rs +++ b/backend/src/utils/network/epg.rs @@ -11,28 +11,28 @@ use shared::error::{info_err, TuliproxError}; use shared::utils::{sanitize_sensitive_info, short_hash}; use std::path::PathBuf; -pub async fn get_input_raw_epg_file_path(url: &str, input: &ConfigInput, working_dir: &str) -> std::io::Result { +pub async fn get_input_raw_epg_file_path(url: &str, input: &ConfigInput, storage_dir: &str) -> std::io::Result { let file_prefix = short_hash(url); if let Some(persist_path) = input.persist.as_deref() { if !persist_path.is_empty() { - if let Some(path) = prepare_file_path(input.persist.as_deref(), working_dir, "") + if let Some(path) = prepare_file_path(input.persist.as_deref(), storage_dir, "") .map(|path| add_prefix_to_filename(&path, concat_string!(&file_prefix, "_epg_").as_str(), Some("xml"))) { return Ok(path); } } } - let download_path = get_input_storage_path(&input.name, working_dir).await?; + let download_path = get_input_storage_path(&input.name, storage_dir).await?; Ok(download_path.join(format!("{}_{}", file_prefix, storage_const::FILE_EPG))) } async fn download_epg_file(url: &str, ctx: &PlaylistProcessingContext, input: &ConfigInput, headers: Option<&reqwest::header::HeaderMap>, - working_dir: &str) -> Result { + storage_dir: &str) -> Result { debug!("Getting epg file path for url: {}", sanitize_sensitive_info(url)); - let persist_file_path = get_input_raw_epg_file_path(url, input, working_dir).await.map_err(|e| info_err!("Could not access epg file download directory: {}", e))?; + let persist_file_path = get_input_raw_epg_file_path(url, input, storage_dir).await.map_err(|e| info_err!("Could not access epg file download directory: {}", e))?; if input.cache_duration_seconds > 0 { if let Ok(metadata) = tokio::fs::metadata(&persist_file_path).await { @@ -55,7 +55,7 @@ async fn download_epg_file(url: &str, ctx: &PlaylistProcessingContext, return Ok(persist_file_path); } debug!("Downloading epg for input '{}'", input.name); - match request::get_input_epg_content_as_file(&ctx.config, &ctx.client, input, headers, working_dir, url, &persist_file_path).await { + match request::get_input_epg_content_as_file(&ctx.config, &ctx.client, input, headers, storage_dir, url, &persist_file_path).await { Ok(path) => { ctx.mark_input_downloaded(lock_key.clone()).await; Ok(path) @@ -66,7 +66,7 @@ async fn download_epg_file(url: &str, ctx: &PlaylistProcessingContext, pub async fn get_xmltv(ctx: &PlaylistProcessingContext, input: &ConfigInput, headers: Option<&reqwest::header::HeaderMap>, - working_dir: &str) -> (Option, Vec) { + storage_dir: &str) -> (Option, Vec) { match &input.epg { None => (None, vec![]), Some(epg_config) => { @@ -75,7 +75,7 @@ pub async fn get_xmltv(ctx: &PlaylistProcessingContext, input: &ConfigInput, let mut stored_file_paths = vec![]; for epg_source in &epg_config.sources { - match download_epg_file(&epg_source.url, ctx, input, headers, working_dir).await { + match download_epg_file(&epg_source.url, ctx, input, headers, storage_dir).await { Ok(file_path) => { stored_file_paths.push(file_path.clone()); file_paths.push(PersistedEpgSource { file_path, priority: epg_source.priority, logo_override: epg_source.logo_override }); diff --git a/backend/src/utils/network/m3u.rs b/backend/src/utils/network/m3u.rs index cb359f97f..2eba66ac6 100644 --- a/backend/src/utils/network/m3u.rs +++ b/backend/src/utils/network/m3u.rs @@ -12,19 +12,19 @@ pub async fn download_m3u_playlist( cfg: &Arc, input: &ConfigInput, ) -> (Vec, Vec) { - let working_dir = &cfg.working_dir; + let storage_dir = &cfg.storage_dir; let input_source: InputSource = { match input.staged.as_ref() { None => input.into(), Some(staged) => if staged.enabled { staged.into() } else { input.into() }, } }; - let persist_file_path = prepare_file_path(input.persist.as_deref(), working_dir, ""); + let persist_file_path = prepare_file_path(input.persist.as_deref(), storage_dir, ""); match request::get_input_text_content_as_stream( app_config, client, &input_source, - working_dir, + storage_dir, persist_file_path, ) .await diff --git a/backend/src/utils/network/request.rs b/backend/src/utils/network/request.rs index 25a6dd876..b01401179 100644 --- a/backend/src/utils/network/request.rs +++ b/backend/src/utils/network/request.rs @@ -20,7 +20,7 @@ use reqwest::{ }; use shared::{ error::{notify_err_res, string_to_io_error, TuliproxError}, - model::{format_elapsed_time, InputFetchMethod, OnConnectErrorPolicy, DEFAULT_USER_AGENT}, + model::{format_elapsed_time, InputFetchMethod, OnConnectErrorPolicy}, utils::{ filter_request_header, human_readable_byte_size, sanitize_sensitive_info, CONTENT_TYPE_JSON, ENCODING_DEFLATE, ENCODING_GZIP, @@ -42,6 +42,7 @@ use tokio::{ }; use tokio_util::io::StreamReader; use url::Url; +use shared::utils::DEFAULT_USER_AGENT; static PROXY_DIAGNOSTICS_ONCE: Once = Once::new(); @@ -533,13 +534,13 @@ pub async fn get_input_epg_content_as_file( client: &reqwest::Client, input: &ConfigInput, headers: Option<&HeaderMap>, - working_dir: &str, + storage_dir: &str, url_str: &str, persist_filepath: &Path, ) -> Result { debug_if_enabled!( - "getting input epg content working_dir: {}, url: {}", - working_dir, + "getting input epg content storage_dir: {}, url: {}", + storage_dir, sanitize_sensitive_info(url_str) ); if url_str.parse::().is_ok() { @@ -556,7 +557,7 @@ pub async fn get_input_epg_content_as_file( } } } else { - let result = match get_file_path(working_dir, Some(PathBuf::from(url_str))) { + let result = match get_file_path(storage_dir, Some(PathBuf::from(url_str))) { Some(filepath) => { if filepath.exists() { if let Err(e) = tokio::fs::copy(&filepath, persist_filepath).await { @@ -590,12 +591,12 @@ pub async fn get_input_text_content( app_state: &Arc, client: &reqwest::Client, input: &InputSource, - working_dir: &str, + storage_dir: &str, persist_filepath: Option, ) -> Result { debug_if_enabled!( - "getting input text content working_dir: {}, url: {}", - working_dir, + "getting input text content storage_dir: {}, url: {}", + storage_dir, sanitize_sensitive_info(&input.url) ); @@ -612,7 +613,7 @@ pub async fn get_input_text_content( } } } else { - let result = match get_file_path(working_dir, Some(PathBuf::from(&input.url))) { + let result = match get_file_path(storage_dir, Some(PathBuf::from(&input.url))) { Some(filepath) => { if filepath.exists() { if let Some(persist_file_value) = persist_filepath { @@ -650,12 +651,12 @@ pub async fn get_input_text_content_as_stream( app_config: &Arc, client: &reqwest::Client, input: &InputSource, - working_dir: &str, + storage_dir: &str, persist_filepath: Option, ) -> Result { debug_if_enabled!( - "getting input text content working_dir: {}, url: {}", - working_dir, + "getting input text content storage_dir: {}, url: {}", + storage_dir, sanitize_sensitive_info(&input.url) ); @@ -672,7 +673,7 @@ pub async fn get_input_text_content_as_stream( } } } else { - let result = match get_file_path(working_dir, Some(PathBuf::from(&input.url))) { + let result = match get_file_path(storage_dir, Some(PathBuf::from(&input.url))) { Some(filepath) => { if filepath.exists() { match get_local_file_content_as_stream(&filepath).await { @@ -1506,7 +1507,7 @@ mod tests { }; use crate::{ model::{AppConfig, Config, ConfigProvider, ResourceRetryConfig, ReverseProxyConfig, SourcesConfig}, - utils::FileLockManager, + utils::{FileLockManager, DEFAULT_USER_AGENT} }; use arc_swap::{ArcSwap, ArcSwapOption}; use shared::model::{ @@ -1536,7 +1537,9 @@ mod tests { api_proxy: Arc::new(ArcSwapOption::default()), file_locks: Arc::new(FileLockManager::default()), paths: Arc::new(ArcSwap::from_pointee(ConfigPaths { + home_path: String::new(), config_path: String::new(), + storage_path: String::new(), config_file_path: String::new(), sources_file_path: String::new(), mapping_file_path: None, @@ -1631,7 +1634,7 @@ mod tests { #[test] fn test_get_request_headers_prioritization() { - use super::{get_request_headers, DEFAULT_USER_AGENT}; + use super::{get_request_headers}; use axum::http::header::USER_AGENT; // Case 1: No headers provided -> Default UA diff --git a/backend/src/utils/network/xtream.rs b/backend/src/utils/network/xtream.rs index d7c42c494..819a33b36 100644 --- a/backend/src/utils/network/xtream.rs +++ b/backend/src/utils/network/xtream.rs @@ -14,7 +14,7 @@ use chrono::{DateTime, Utc}; use log::{error, info, warn}; use shared::error::TuliproxError; use shared::model::{PlaylistEntry, PlaylistGroup, ProxyUserStatus, SeriesStreamProperties, - StreamProperties, VideoStreamProperties, XtreamCluster, XtreamPlaylistItem, + StreamProperties, VideoStreamProperties, InputType, XtreamCluster, XtreamPlaylistItem, XtreamSeriesInfo, XtreamVideoInfo, XtreamVideoInfoDoc}; use shared::utils::{extract_extension_from_url, get_i64_from_serde_value, get_string_from_serde_value, sanitize_sensitive_info, Internable, PROVIDER_SCHEME_PREFIX}; use std::collections::HashMap; @@ -93,8 +93,8 @@ pub async fn get_xtream_stream_info(client: &reqwest::Client, match cluster { XtreamCluster::Live => {} XtreamCluster::Video => { - let working_dir = &app_config.config.load().working_dir; - if let Ok(storage_path) = get_input_storage_path(&input.name, working_dir).await { + let storage_dir = &app_config.config.load().storage_dir; + if let Ok(storage_path) = get_input_storage_path(&input.name, storage_dir).await { match serde_json::from_str::(&content) { Ok(info) => { // parse downloaded info into StreamProperties @@ -126,7 +126,7 @@ pub async fn get_xtream_stream_info(client: &reqwest::Client, } } XtreamCluster::Series => { - let working_dir = &app_config.config.load().working_dir; + let storage_dir = &app_config.config.load().storage_dir; let group = pli.get_group(); let series_name = pli.get_name(); @@ -135,7 +135,7 @@ pub async fn get_xtream_stream_info(client: &reqwest::Client, // parse series info let series_stream_props = SeriesStreamProperties::from_info(&info, pli); - if let Ok(storage_path) = get_input_storage_path(&input.name, working_dir).await { + if let Ok(storage_path) = get_input_storage_path(&input.name, storage_dir).await { // update input db if let Err(err) = persists_input_series_info(app_config, &storage_path, cluster, &input.name, provider_id, &series_stream_props).await { error!("Failed to persist series info for input {}: {err}", &input.name); @@ -387,7 +387,11 @@ pub async fn download_xtream_playlist(app_config: &Arc, client: &reqw let skip_cluster = get_skip_cluster(input); let cfg = app_config.config.load(); - let working_dir = &cfg.working_dir; + let storage_dir = &cfg.storage_dir; + // Staged input type is the effective type during download. + let effective_type = input.get_download_input_type(); + let use_disk_based_processing = + cfg.disk_based_processing && matches!(effective_type, InputType::Xtream | InputType::XtreamBatch); let mut errors = vec![]; for (xtream_cluster, category, stream) in &ACTIONS { @@ -396,16 +400,16 @@ pub async fn download_xtream_playlist(app_config: &Arc, client: &reqw let input_source_category = input_source.with_url(concat_string!(&base_url, "&action=", category)); let input_source_stream = input_source.with_url(concat_string!(&base_url, "&action=", stream)); let category_file_path = crate::utils::prepare_file_path(input.persist.as_deref(), - working_dir, concat_string!(category, "_").as_str()); + storage_dir, concat_string!(category, "_").as_str()); let stream_file_path = crate::utils::prepare_file_path(input.persist.as_deref(), - working_dir, concat_string!(stream, "_").as_str()); + storage_dir, concat_string!(stream, "_").as_str()); match futures::join!( request::get_input_json_content_as_stream(app_config, client, &input_source_category, category_file_path), request::get_input_json_content_as_stream(app_config, client, &input_source_stream, stream_file_path) ) { (Ok(category_content), Ok(stream_content)) => { - if cfg.disk_based_processing { + if use_disk_based_processing { // trace!("Using disk input playlist optimization for cluster {}", xtream_cluster); if let Err(err) = persist_input_xtream_playlist_cluster_to_disk(app_config, input, *xtream_cluster, category_content, stream_content).await { error!("persist_input_xtream_playlist_cluster_to_disk failed: {err}"); @@ -443,7 +447,7 @@ pub async fn download_xtream_playlist(app_config: &Arc, client: &reqw plg.id = grp_id; } - (playlist_groups, errors, cfg.disk_based_processing) + (playlist_groups, errors, use_disk_based_processing) } async fn check_alias_user_state(app_config: &Arc, client: &reqwest::Client, input: &ConfigInput) { diff --git a/backend/src/utils/trakt/client.rs b/backend/src/utils/trakt/client.rs index 86fa0f746..4c806305e 100644 --- a/backend/src/utils/trakt/client.rs +++ b/backend/src/utils/trakt/client.rs @@ -3,8 +3,7 @@ use shared::error::TuliproxError; use reqwest::header::{HeaderMap, HeaderValue}; use log::{debug, info}; use shared::error::{info_err}; -use shared::model::DEFAULT_USER_AGENT; -use shared::utils::{trim_last_slash, TRAKT_API_KEY}; +use shared::utils::{trim_last_slash, DEFAULT_USER_AGENT, TRAKT_API_KEY}; use super::errors::{handle_trakt_api_error}; pub struct TraktClient { diff --git a/frontend/public/assets/i18n/en.json b/frontend/public/assets/i18n/en.json index e4c9b2577..8dc0267f6 100644 --- a/frontend/public/assets/i18n/en.json +++ b/frontend/public/assets/i18n/en.json @@ -286,7 +286,7 @@ "NAME": "Human readable name for this entry. Must be unique.", "PASSWORD": "Password or credential used for this account or service.", "PRIORITY": "The order of processing. Lower numbers mean higher priority (e.g. -1 is higher than 0).", - "URL": "Target URL for this input. Supports provider:// scheme for failover.", + "URL": "Input source URL. Supported schemes:\n\n```http(s)://\nfile://\nprovider:///...\nbatch://...(auto-switches m3u/xtream to batch types).```\n\nFor m3u_batch/xtream_batch only local CSV sources are allowed (batch://...).", "USERNAME": "Login username for this account or service." }, "CONFIG_INPUT_ALIAS": { @@ -425,7 +425,7 @@ "UPDATE_ON_BOOT": "If enabled, Tuliprox triggers a full playlist update immediately upon startup. Helpful during initial setup, but usually set to false for stable operations.", "USER_ACCESS_CONTROL": "If enabled, the attributes (if set)\n\n- expiration date,\n- status and\n- max_connections\n\nare checked to permit or deny access for api users.", "USER_CONFIG_DIR": "Directory path where user-specific overrides and configuration files are stored.", - "WORKING_DIR": "The primary directory where Tuliprox stores its data files and databases." + "STORAGE_DIR": "The primary directory where Tuliprox stores its data files and databases." }, "MESSAGING_CONFIG": { "NOTIFY_ON": "Select events (e.g. 'info', 'stats', 'error') that trigger notifications for this specific type.", @@ -544,7 +544,7 @@ "STAGED_INPUT": { "HEADERS": "Custom HTTP headers sent when retrieving this staged configuration.", "PASSWORD": "Password or credential used for this account or service.", - "URL": "Target URL for this setting.", + "URL": "Source URL for the staged input. Use http(s)://, file:// or provider:///... . Batch URLs are not supported for staged inputs.", "USERNAME": "Login username for this account or service." }, "STREAM_BUFFER_CONFIG": { @@ -1138,7 +1138,7 @@ "WEB_UI_CONFIG": "Web-Ui", "WEB_UI_PATH": "Path prefix", "WELCOME": "Welcome", - "WORKING_DIR": "Working dir", + "STORAGE_DIR": "Storage dir", "XC": "xc", "XTREAM": "xtream", "XTREAM_BATCH": "xc batch", diff --git a/frontend/scss/app/components/_form.scss b/frontend/scss/app/components/_form.scss index 3f29999b8..06580b0f7 100644 --- a/frontend/scss/app/components/_form.scss +++ b/frontend/scss/app/components/_form.scss @@ -21,7 +21,9 @@ &__list, &__text { + display: flex; flex-flow: column; + gap: var(--gap-default); } &__value { @@ -40,6 +42,8 @@ border-radius: var(--border-radius); box-sizing: border-box; white-space: nowrap; + overflow-x: auto; + scrollbar-width: none; } &__bool { diff --git a/frontend/scss/app/components/source_editor/_source_editor.scss b/frontend/scss/app/components/source_editor/_source_editor.scss index 33d4f1728..341ea949a 100644 --- a/frontend/scss/app/components/source_editor/_source_editor.scss +++ b/frontend/scss/app/components/source_editor/_source_editor.scss @@ -137,6 +137,14 @@ cursor: pointer; } +.tp__form-list { + display: flex; + flex-flow: column; + gap: var(--gap-default); + box-sizing: border-box; + overflow: hidden; +} + .tp__form-list__item-arrow-btn { svg { transform: scale(2); diff --git a/frontend/scss/app/components/source_editor/_source_editor_form.scss b/frontend/scss/app/components/source_editor/_source_editor_form.scss index 1e2763c28..b284bd930 100644 --- a/frontend/scss/app/components/source_editor/_source_editor_form.scss +++ b/frontend/scss/app/components/source_editor/_source_editor_form.scss @@ -92,4 +92,8 @@ $form-width: 520px; .tp__source-editor-form.disabled { pointer-events: none; opacity: 0.3; +} + +.tp__input-form__body { + width: 100%; } \ No newline at end of file diff --git a/frontend/src/app/components/config/api_config_view.rs b/frontend/src/app/components/config/api_config_view.rs index 0e077ac5b..d2e5b93b5 100644 --- a/frontend/src/app/components/config/api_config_view.rs +++ b/frontend/src/app/components/config/api_config_view.rs @@ -184,9 +184,13 @@ pub fn ApiConfigView() -> Html { let deps = (api_config, *config_view_ctx.edit_mode); use_effect_with(deps, move |(cfg, _mode)| { if let Some(api) = cfg { - form_state_api_config.dispatch(ApiConfigFormAction::SetAll(api.clone())); + let mut next_api = api.clone(); + next_api.prepare(); + form_state_api_config.dispatch(ApiConfigFormAction::SetAll(next_api)); } else { - form_state_api_config.dispatch(ApiConfigFormAction::SetAll(ConfigApiDto::default())); + let mut defaults = ConfigApiDto::default(); + defaults.prepare(); + form_state_api_config.dispatch(ApiConfigFormAction::SetAll(defaults)); } || () }); @@ -421,11 +425,13 @@ pub fn ApiConfigView() -> Html { let render_view_mode_api_config = || { if let Some(config) = &config_ctx.config { + let mut api_config = config.config.api.clone(); + api_config.prepare(); html! { - { config_field!(config.config.api, translate.t(LABEL_HOST), host) } - { config_field!(config.config.api, translate.t(LABEL_PORT), port) } - { config_field!(config.config.api, translate.t(LABEL_WEB_ROOT), web_root) } + { config_field!(api_config, translate.t(LABEL_HOST), host) } + { config_field!(api_config, translate.t(LABEL_PORT), port) } + { config_field!(api_config, translate.t(LABEL_WEB_ROOT), web_root) } } } else { diff --git a/frontend/src/app/components/config/config_view.rs b/frontend/src/app/components/config/config_view.rs index 7c3a2aeb9..6047a9cbc 100644 --- a/frontend/src/app/components/config/config_view.rs +++ b/frontend/src/app/components/config/config_view.rs @@ -474,7 +474,7 @@ pub fn ConfigView() -> Html { { html_if!(*edit_mode || setup_mode, {
- diff --git a/frontend/src/app/components/config/main_config_view.rs b/frontend/src/app/components/config/main_config_view.rs index 39e778c5a..faeb97f2a 100644 --- a/frontend/src/app/components/config/main_config_view.rs +++ b/frontend/src/app/components/config/main_config_view.rs @@ -7,10 +7,10 @@ use crate::{ context::ConfigContext, }, config_field, config_field_bool, config_field_optional, edit_field_bool, edit_field_number, - edit_field_number_option_u32, edit_field_text, edit_field_text_option, generate_form_reducer, + edit_field_number_option_u32, edit_field_text_option, generate_form_reducer, i18n::use_translation, }; -use shared::model::MainConfigDto; +use shared::model::{ConfigDto, MainConfigDto}; use yew::prelude::*; const LABEL_UPDATE_ON_BOOT: &str = "LABEL.UPDATE_ON_BOOT"; @@ -18,7 +18,7 @@ const LABEL_CONFIG_HOT_RELOAD: &str = "LABEL.CONFIG_HOT_RELOAD"; const LABEL_USER_ACCESS_CONTROL: &str = "LABEL.USER_ACCESS_CONTROL"; const LABEL_PROCESS_PARALLEL: &str = "LABEL.PROCESS_PARALLEL"; const LABEL_DISK_BASED_PROCESSING: &str = "LABEL.DISK_BASED_PROCESSING"; -const LABEL_WORKING_DIR: &str = "LABEL.WORKING_DIR"; +const LABEL_STORAGE_DIR: &str = "LABEL.STORAGE_DIR"; const LABEL_DEFAULT_USER_AGENT: &str = "LABEL.DEFAULT_USER_AGENT"; const LABEL_MAPPING_PATH: &str = "LABEL.MAPPING_PATH"; const LABEL_TEMPLATE_PATH: &str = "LABEL.TEMPLATE_PATH"; @@ -39,7 +39,7 @@ generate_form_reducer!( AcceptInsecureSslCertificates => accept_insecure_ssl_certificates: bool, ProcessParallel => process_parallel: bool, DiskBasedProcessing => disk_based_processing: bool, - WorkingDir => working_dir: String, + StorageDir => storage_dir: Option, DefaultUserAgent => default_user_agent: Option, MappingPath => mapping_path: Option, TemplatePath => template_path: Option, @@ -73,8 +73,16 @@ pub fn MainConfigView() -> Html { let config = config_ctx.config.as_ref().map(|c| c.config.clone()); use_effect_with((config, config_view_ctx.edit_mode.clone()), move |(cfg, _mode)| { if let Some(main) = cfg { - let main_config = MainConfigDto::from(main); - form_state.dispatch(MainConfigFormAction::SetAll(main_config.clone())); + let mut prepared_main: ConfigDto = main.clone(); + match prepared_main.prepare(false) { + Ok(()) => { + let main_config = MainConfigDto::from(&prepared_main); + form_state.dispatch(MainConfigFormAction::SetAll(main_config.clone())); + } + Err(err) => { + log::error!("Failed to prepare main config for view state: {err}"); + } + } } else { form_state.dispatch(MainConfigFormAction::SetAll(MainConfigDto::default())); } @@ -91,7 +99,7 @@ pub fn MainConfigView() -> Html { { config_field_bool!(form_state.form, translate.t(LABEL_ACCEPT_INSECURE_SSL_CERTIFICATES), accept_insecure_ssl_certificates) } { config_field_bool!(form_state.form, translate.t(LABEL_PROCESS_PARALLEL), process_parallel) } { config_field_bool!(form_state.form, translate.t(LABEL_DISK_BASED_PROCESSING), disk_based_processing) } - { config_field!(form_state.form, translate.t(LABEL_WORKING_DIR), working_dir) } + { config_field_optional!(form_state.form, translate.t(LABEL_STORAGE_DIR), storage_dir) } { config_field_optional!(form_state.form, translate.t(LABEL_DEFAULT_USER_AGENT), default_user_agent) } { config_field_optional!(form_state.form, translate.t(LABEL_MAPPING_PATH), mapping_path) } { config_field_optional!(form_state.form, translate.t(LABEL_TEMPLATE_PATH), template_path) } @@ -113,7 +121,7 @@ pub fn MainConfigView() -> Html { { edit_field_bool!(form_state, translate.t(LABEL_ACCEPT_INSECURE_SSL_CERTIFICATES), accept_insecure_ssl_certificates, MainConfigFormAction::AcceptInsecureSslCertificates) } { edit_field_bool!(form_state, translate.t(LABEL_PROCESS_PARALLEL), process_parallel, MainConfigFormAction::ProcessParallel) } { edit_field_bool!(form_state, translate.t(LABEL_DISK_BASED_PROCESSING), disk_based_processing, MainConfigFormAction::DiskBasedProcessing) } - { edit_field_text!(form_state, translate.t(LABEL_WORKING_DIR), working_dir, MainConfigFormAction::WorkingDir) } + { edit_field_text_option!(form_state, translate.t(LABEL_STORAGE_DIR), storage_dir, MainConfigFormAction::StorageDir) } { edit_field_text_option!(form_state, translate.t(LABEL_DEFAULT_USER_AGENT), default_user_agent, MainConfigFormAction::DefaultUserAgent) } { edit_field_text_option!(form_state, translate.t(LABEL_MAPPING_PATH), mapping_path, MainConfigFormAction::MappingPath) } { edit_field_text_option!(form_state, translate.t(LABEL_TEMPLATE_PATH), template_path, MainConfigFormAction::TemplatePath) } diff --git a/frontend/src/app/components/config/reverse_proxy_config_view.rs b/frontend/src/app/components/config/reverse_proxy_config_view.rs index 51108a78a..367e94681 100644 --- a/frontend/src/app/components/config/reverse_proxy_config_view.rs +++ b/frontend/src/app/components/config/reverse_proxy_config_view.rs @@ -70,7 +70,7 @@ generate_form_reducer!( fields { Enabled => enabled: bool, Size => size: Option, - Dir => dir: Option, + Dir => directory: Option, } ); @@ -389,7 +389,7 @@ pub fn ReverseProxyConfigView() -> Html {

{translate.t(LABEL_CACHE)}

{ config_field_bool!(cache_state.form, translate.t(LABEL_ENABLED), enabled) } { config_field_optional!(cache_state.form, translate.t(LABEL_SIZE), size) } - { config_field_optional!(cache_state.form, translate.t(LABEL_DIRECTORY), dir) } + { config_field_optional!(cache_state.form, translate.t(LABEL_DIRECTORY), directory) } } }; @@ -543,7 +543,7 @@ pub fn ReverseProxyConfigView() -> Html {

{translate.t(LABEL_CACHE)}

{ edit_field_bool!(cache_state, translate.t(LABEL_ENABLED), enabled, CacheConfigFormAction::Enabled) } { edit_field_text_option!(cache_state, translate.t(LABEL_SIZE), size, CacheConfigFormAction::Size) } - { edit_field_text_option!(cache_state, translate.t(LABEL_DIRECTORY), dir, CacheConfigFormAction::Dir) } + { edit_field_text_option!(cache_state, translate.t(LABEL_DIRECTORY), directory, CacheConfigFormAction::Dir) } } }; diff --git a/frontend/src/app/components/config/schedules_config_view.rs b/frontend/src/app/components/config/schedules_config_view.rs index fc57e261b..a603d8d95 100644 --- a/frontend/src/app/components/config/schedules_config_view.rs +++ b/frontend/src/app/components/config/schedules_config_view.rs @@ -188,6 +188,7 @@ pub fn SchedulesConfigView() -> Html { { match entry.task_type { ScheduleTaskType::PlaylistUpdate => translate.t("LABEL.PLAYLIST_UPDATE"), ScheduleTaskType::LibraryScan => translate.t("LABEL.LIBRARY"), + ScheduleTaskType::GeoIpUpdate => translate.t("LABEL.GEOIP"), }}
@@ -246,6 +247,11 @@ pub fn SchedulesConfigView() -> Html { label: html! { translate.t("LABEL.LIBRARY") }, selected: *selected_type == ScheduleTaskType::LibraryScan, }, + DropDownOption { + id: "GeoIpUpdate".to_string(), + label: html! { translate.t("LABEL.GEOIP") }, + selected: *selected_type == ScheduleTaskType::GeoIpUpdate, + }, ]); html! { @@ -260,6 +266,8 @@ pub fn SchedulesConfigView() -> Html { set_selected_type.set(ScheduleTaskType::PlaylistUpdate); } else if option == "LibraryScan" { set_selected_type.set(ScheduleTaskType::LibraryScan); + } else if option == "GeoIpUpdate" { + set_selected_type.set(ScheduleTaskType::GeoIpUpdate); } })} options={types} diff --git a/frontend/src/app/components/source_editor/input_form.rs b/frontend/src/app/components/source_editor/input_form.rs index 38c4be5a5..c6e126567 100644 --- a/frontend/src/app/components/source_editor/input_form.rs +++ b/frontend/src/app/components/source_editor/input_form.rs @@ -179,6 +179,14 @@ generate_form_reducer!( } ); +fn apply_parsed_input_type(staged_input_state: &UseReducerHandle, selected: Option<&str>) { + let input_type = + selected.and_then(|value| value.parse::().ok()).unwrap_or(staged_input_state.form.input_type); + if staged_input_state.form.input_type != input_type { + staged_input_state.dispatch(StagedInputFormAction::InputType(input_type)); + } +} + #[derive(Properties, PartialEq, Clone)] pub struct ConfigInputViewProps { #[prop_or_default] @@ -631,16 +639,18 @@ pub fn ConfigInputView(props: &ConfigInputViewProps) -> Html { multi_select={false} on_select={Callback::from(move |(_, selections):(String, DropDownSelection)| { match selections { - DropDownSelection::Empty => { - staged_input_state_2.dispatch(StagedInputFormAction::InputType(InputType::Xtream)); - } + DropDownSelection::Empty => {} DropDownSelection::Single(option) => { - staged_input_state_2.dispatch(StagedInputFormAction::InputType(option.parse::().unwrap_or(InputType::Xtream))); + apply_parsed_input_type( + &staged_input_state_2, + Some(option.as_str()), + ); } DropDownSelection::Multi(options) => { - if let Some(first) = options.first() { - staged_input_state_2.dispatch(StagedInputFormAction::InputType(first.parse::().unwrap_or(InputType::Xtream))); - } + apply_parsed_input_type( + &staged_input_state_2, + options.first().map(String::as_str), + ); } } })} @@ -896,7 +906,7 @@ pub fn ConfigInputView(props: &ConfigInputViewProps) -> Html { }) }
- bool { || dns.schemes.as_ref().is_some_and(|schemes| !schemes.is_empty()) || dns.keep_vhost || dns.overrides.as_ref().is_some_and(|overrides| !overrides.is_empty()) - || dns.resolved.as_ref().is_some_and(|resolved| !resolved.is_empty()) || dns.on_resolve_error != OnResolveErrorPolicy::default() || dns.on_connect_error != OnConnectErrorPolicy::default() } diff --git a/shared/src/foundation/mapper.rs b/shared/src/foundation/mapper.rs index 1ae6500f2..1f9d5f13e 100644 --- a/shared/src/foundation/mapper.rs +++ b/shared/src/foundation/mapper.rs @@ -802,33 +802,23 @@ impl<'a> MapperContext<'a> { BuiltInFunction::ToNumber | BuiltInFunction::Template | BuiltInFunction::First - | BuiltInFunction::AddFavourite => { - if args.len() > 1 { - return info_err_res!( - "Function accepts only one argument {:?}, {} given", - name, - args.len() - ); - } + | BuiltInFunction::AddFavourite + if args.len() > 1 => + { + return info_err_res!("Function accepts only one argument {:?}, {} given", name, args.len()); } - BuiltInFunction::Split => { - if args.len() != 2 { - return info_err_res!("Function accepts two arguments {:?}, {} given", name, args.len()); - } + BuiltInFunction::Split if args.len() != 2 => { + return info_err_res!("Function accepts two arguments {:?}, {} given", name, args.len()); } - BuiltInFunction::Replace => { - if args.len() != 3 { - return info_err_res!("Function accepts three arguments {:?}, {} given", name, args.len()); - } + BuiltInFunction::Replace if args.len() != 3 => { + return info_err_res!("Function accepts three arguments {:?}, {} given", name, args.len()); } - BuiltInFunction::Pad => { - if !(args.len() == 3 || args.len() == 4) { - return info_err_res!( - "Function accepts three or four arguments {:?}, {} given", - name, - args.len() - ); - } + BuiltInFunction::Pad if !(args.len() == 3 || args.len() == 4) => { + return info_err_res!( + "Function accepts three or four arguments {:?}, {} given", + name, + args.len() + ); } _ => {} } diff --git a/shared/src/model/config/api.rs b/shared/src/model/config/api.rs index 6cc445ecc..9e591d26f 100644 --- a/shared/src/model/config/api.rs +++ b/shared/src/model/config/api.rs @@ -1,8 +1,52 @@ +use crate::utils::{get_default_web_root, is_blank_or_default_web_root}; + #[derive(Debug, Clone, serde::Serialize, serde::Deserialize, Default, PartialEq)] #[serde(deny_unknown_fields)] pub struct ConfigApiDto { pub host: String, pub port: u16, - #[serde(default, skip_serializing_if = "String::is_empty")] + #[serde(default, skip_serializing_if = "is_blank_or_default_web_root")] pub web_root: String, } + +impl ConfigApiDto { + pub fn default_web_root() -> String { get_default_web_root() } + + pub fn prepare(&mut self) { + if self.web_root.trim().is_empty() { + self.web_root = Self::default_web_root(); + } + } +} + +#[cfg(test)] +mod tests { + use super::ConfigApiDto; + + #[test] + fn serialization_skips_default_web_root() { + let dto = ConfigApiDto { host: "0.0.0.0".to_string(), port: 8901, web_root: "./web".to_string() }; + + let serialized = serde_json::to_string(&dto).expect("api dto serialization should succeed"); + assert!(!serialized.contains("\"web_root\""), "expected default web_root to be skipped, got: {serialized}"); + } + + #[test] + fn serialization_keeps_non_default_web_root() { + let dto = ConfigApiDto { host: "0.0.0.0".to_string(), port: 8901, web_root: "/srv/tuliprox/web".to_string() }; + + let serialized = serde_json::to_string(&dto).expect("api dto serialization should succeed"); + assert!( + serialized.contains("\"web_root\""), + "expected non-default web_root to be persisted, got: {serialized}" + ); + } + + #[test] + fn prepare_sets_default_web_root_when_empty() { + let mut dto = ConfigApiDto { host: "0.0.0.0".to_string(), port: 8901, web_root: String::new() }; + let expected_web_root = ConfigApiDto::default_web_root(); + dto.prepare(); + assert_eq!(dto.web_root, expected_web_root); + } +} diff --git a/shared/src/model/config/base.rs b/shared/src/model/config/base.rs index bdd1de2c6..2faa42d3c 100644 --- a/shared/src/model/config/base.rs +++ b/shared/src/model/config/base.rs @@ -6,55 +6,18 @@ use crate::{ WebUiConfigDto, }, utils::{ - default_connect_timeout_secs, default_supported_video_extensions, is_blank_optional_string, - is_default_connect_timeout_secs, is_false, CONFIG_PATH, MAPPING_FILE, TEMPLATE_FILE, + default_connect_timeout_secs, default_custom_stream_response_path, default_default_user_agent, + default_main_backup_dir, default_main_mapping_path, default_main_storage_dir, default_main_template_path, + default_main_user_config_dir, default_supported_video_extensions, is_blank_optional_string, + is_blank_or_default_backup_dir, is_blank_or_default_custom_stream_response_path, + is_blank_or_default_mapping_path, is_blank_or_default_storage_dir, is_blank_or_default_template_path, + is_blank_or_default_user_config_dir, is_default_connect_timeout_secs, is_false, + is_none_or_empty_metadata_update, is_none_or_empty_video, normalize_optional_config_file_path, + normalize_optional_dir, DEFAULT_BACKUP_DIR, DEFAULT_CUSTOM_STREAM_RESPONSE_PATH, DEFAULT_STORAGE_DIR, + DEFAULT_USER_CONFIG_DIR, MAPPING_FILE, TEMPLATE_FILE, }, }; -pub const DEFAULT_USER_AGENT: &str = "VLC/3.0.16 LibVLC/3.0.16"; - -fn default_default_user_agent() -> Option { Some(DEFAULT_USER_AGENT.to_string()) } -fn default_main_mapping_path() -> Option { Some(format!("./{CONFIG_PATH}/{MAPPING_FILE}")) } -fn default_main_template_path() -> Option { Some(format!("./{CONFIG_PATH}/{TEMPLATE_FILE}")) } - -fn is_default_config_file_path(value: &str, file_name: &str) -> bool { - let normalized = value.trim().replace('\\', "/"); - let normalized = normalized.trim_start_matches("./"); - normalized == file_name - || normalized.rsplit_once('/').is_some_and(|(dir, file)| dir == CONFIG_PATH && file == file_name) -} - -fn is_blank_or_default_mapping_path(path: &Option) -> bool { - path.as_ref().is_none_or(|value| value.trim().is_empty() || is_default_config_file_path(value, MAPPING_FILE)) -} - -fn is_blank_or_default_template_path(path: &Option) -> bool { - path.as_ref().is_none_or(|value| value.trim().is_empty() || is_default_config_file_path(value, TEMPLATE_FILE)) -} - -fn effective_main_mapping_path(path: Option) -> Option { - if is_blank_or_default_mapping_path(&path) { - default_main_mapping_path() - } else { - path - } -} - -fn effective_main_template_path(path: Option) -> Option { - if is_blank_or_default_template_path(&path) { - default_main_template_path() - } else { - path - } -} - -fn is_none_or_empty_video(video: &Option) -> bool { - video.as_ref().is_none_or(VideoConfigDto::is_empty) -} -fn is_none_or_empty_metadata_update(metadata_update: &Option) -> bool { - metadata_update.as_ref().is_none_or(MetadataUpdateConfigDto::is_empty) -} - #[allow(clippy::struct_excessive_bools)] #[derive(Debug, Clone, serde::Serialize, serde::Deserialize, PartialEq)] #[serde(deny_unknown_fields)] @@ -62,18 +25,22 @@ pub struct ConfigDto { #[serde(default, skip_serializing_if = "is_false")] pub process_parallel: bool, pub api: ConfigApiDto, - pub working_dir: String, + #[serde(default, alias = "working_dir", skip_serializing_if = "is_blank_or_default_storage_dir")] + pub storage_dir: Option, #[serde(default = "default_default_user_agent", skip_serializing_if = "is_blank_optional_string")] pub default_user_agent: Option, - #[serde(default, skip_serializing_if = "is_blank_optional_string")] + #[serde(default, skip_serializing_if = "is_blank_or_default_backup_dir")] pub backup_dir: Option, - #[serde(default, skip_serializing_if = "is_blank_optional_string")] + #[serde(default, skip_serializing_if = "is_blank_or_default_user_config_dir")] pub user_config_dir: Option, #[serde(default, skip_serializing_if = "is_blank_or_default_mapping_path")] pub mapping_path: Option, #[serde(default, skip_serializing_if = "is_blank_or_default_template_path")] pub template_path: Option, - #[serde(default, skip_serializing_if = "is_blank_optional_string")] + #[serde( + default = "default_custom_stream_response_path", + skip_serializing_if = "is_blank_or_default_custom_stream_response_path" + )] pub custom_stream_response_path: Option, #[serde(default, skip_serializing_if = "is_none_or_empty_video")] pub video: Option, @@ -118,7 +85,7 @@ impl Default for ConfigDto { Self { process_parallel: false, api: ConfigApiDto::default(), - working_dir: String::new(), + storage_dir: None, default_user_agent: default_default_user_agent(), backup_dir: None, user_config_dir: None, @@ -153,12 +120,13 @@ impl Default for ConfigDto { pub struct MainConfigDto { #[serde(default, skip_serializing_if = "is_false")] pub process_parallel: bool, - pub working_dir: String, + #[serde(default = "default_main_storage_dir", skip_serializing_if = "is_blank_or_default_storage_dir")] + pub storage_dir: Option, #[serde(default = "default_default_user_agent", skip_serializing_if = "is_blank_optional_string")] pub default_user_agent: Option, - #[serde(default, skip_serializing_if = "is_blank_optional_string")] + #[serde(default = "default_main_backup_dir", skip_serializing_if = "is_blank_or_default_backup_dir")] pub backup_dir: Option, - #[serde(default, skip_serializing_if = "is_blank_optional_string")] + #[serde(default = "default_main_user_config_dir", skip_serializing_if = "is_blank_or_default_user_config_dir")] pub user_config_dir: Option, #[serde(default = "default_main_mapping_path", skip_serializing_if = "is_blank_or_default_mapping_path")] pub mapping_path: Option, @@ -187,10 +155,10 @@ impl Default for MainConfigDto { MainConfigDto { process_parallel: false, disk_based_processing: false, - working_dir: String::new(), + storage_dir: default_main_storage_dir(), default_user_agent: default_default_user_agent(), - backup_dir: None, - user_config_dir: None, + backup_dir: default_main_backup_dir(), + user_config_dir: default_main_user_config_dir(), mapping_path: default_main_mapping_path(), template_path: default_main_template_path(), custom_stream_response_path: None, @@ -209,12 +177,12 @@ impl From<&ConfigDto> for MainConfigDto { Self { process_parallel: config.process_parallel, disk_based_processing: config.disk_based_processing, - working_dir: config.working_dir.clone(), + storage_dir: config.storage_dir.clone(), default_user_agent: config.default_user_agent.clone(), backup_dir: config.backup_dir.clone(), user_config_dir: config.user_config_dir.clone(), - mapping_path: effective_main_mapping_path(config.mapping_path.clone()), - template_path: effective_main_template_path(config.template_path.clone()), + mapping_path: config.mapping_path.clone(), + template_path: config.template_path.clone(), custom_stream_response_path: config.custom_stream_response_path.clone(), user_access_control: config.user_access_control, connect_timeout_secs: config.connect_timeout_secs, @@ -249,9 +217,26 @@ pub struct HdHomeRunDeviceOverview { impl ConfigDto { pub fn prepare(&mut self, include_computed: bool) -> Result<(), TuliproxError> { + self.api.prepare(); + if is_blank_optional_string(&self.default_user_agent) { self.default_user_agent = default_default_user_agent(); } + if is_blank_or_default_storage_dir(&self.storage_dir) { + self.storage_dir = default_main_storage_dir(); + } + if is_blank_or_default_backup_dir(&self.backup_dir) { + self.backup_dir = default_main_backup_dir(); + } + if is_blank_or_default_user_config_dir(&self.user_config_dir) { + self.user_config_dir = default_main_user_config_dir(); + } + if is_blank_or_default_mapping_path(&self.mapping_path) { + self.mapping_path = default_main_mapping_path(); + } + if is_blank_or_default_template_path(&self.template_path) { + self.template_path = default_main_template_path(); + } if let Some(mins) = self.sleep_timer_mins { if mins == 0 { @@ -268,7 +253,7 @@ impl ConfigDto { self.prepare_metadata_update_config()?; if let Some(reverse_proxy) = self.reverse_proxy.as_mut() { - reverse_proxy.prepare(&self.working_dir)?; + reverse_proxy.prepare(self.storage_dir.as_deref().unwrap_or_default())?; } if let Some(proxy) = &mut self.proxy { proxy.prepare()?; @@ -280,6 +265,9 @@ impl ConfigDto { if let Some(messaging) = &mut self.messaging { messaging.prepare(include_computed)?; } + if let Some(library) = &mut self.library { + library.playlist.prepare(); + } Ok(()) } @@ -361,13 +349,14 @@ impl ConfigDto { pub fn update_from_main_config(&mut self, main_config: &MainConfigDto) { self.process_parallel = main_config.process_parallel; self.disk_based_processing = main_config.disk_based_processing; - self.working_dir = main_config.working_dir.clone(); + self.storage_dir = normalize_optional_dir(&main_config.storage_dir, DEFAULT_STORAGE_DIR); self.default_user_agent = main_config.default_user_agent.clone(); - self.backup_dir = main_config.backup_dir.clone(); - self.user_config_dir = main_config.user_config_dir.clone(); - self.mapping_path = main_config.mapping_path.clone(); - self.template_path = main_config.template_path.clone(); - self.custom_stream_response_path = main_config.custom_stream_response_path.clone(); + self.backup_dir = normalize_optional_dir(&main_config.backup_dir, DEFAULT_BACKUP_DIR); + self.user_config_dir = normalize_optional_dir(&main_config.user_config_dir, DEFAULT_USER_CONFIG_DIR); + self.mapping_path = normalize_optional_config_file_path(&main_config.mapping_path, MAPPING_FILE); + self.template_path = normalize_optional_config_file_path(&main_config.template_path, TEMPLATE_FILE); + self.custom_stream_response_path = + normalize_optional_dir(&main_config.custom_stream_response_path, DEFAULT_CUSTOM_STREAM_RESPONSE_PATH); self.user_access_control = main_config.user_access_control; self.connect_timeout_secs = main_config.connect_timeout_secs; self.sleep_timer_mins = main_config.sleep_timer_mins; @@ -386,7 +375,7 @@ impl ConfigDto { #[cfg(test)] mod tests { use super::*; - use crate::utils::default_supported_video_extensions; + use crate::utils::{default_supported_video_extensions, CONFIG_PATH}; use serde_json::json; #[test] @@ -425,6 +414,104 @@ mod tests { assert!(serialized.contains("\"video\""), "expected video field, got: {serialized}"); } + #[test] + fn serializing_skips_default_storage_backup_and_user_config_dirs() { + let cfg = ConfigDto { + storage_dir: Some(DEFAULT_STORAGE_DIR.to_string()), + backup_dir: Some(DEFAULT_BACKUP_DIR.to_string()), + user_config_dir: Some(DEFAULT_USER_CONFIG_DIR.to_string()), + ..ConfigDto::default() + }; + + let serialized = serde_json::to_string(&cfg).expect("config serialization should succeed"); + assert!( + !serialized.contains("\"storage_dir\""), + "expected no storage_dir field for default value, got: {serialized}" + ); + assert!( + !serialized.contains("\"backup_dir\""), + "expected no backup_dir field for default value, got: {serialized}" + ); + assert!( + !serialized.contains("\"user_config_dir\""), + "expected no user_config_dir field for default value, got: {serialized}" + ); + } + + #[test] + fn serializing_keeps_non_default_storage_and_backup_dirs() { + let cfg = ConfigDto { + storage_dir: Some("custom-storage".to_string()), + backup_dir: Some("custom-backup".to_string()), + user_config_dir: Some("custom-user-config".to_string()), + ..ConfigDto::default() + }; + + let serialized = serde_json::to_string(&cfg).expect("config serialization should succeed"); + assert!( + serialized.contains("\"storage_dir\""), + "expected storage_dir field for non-default value, got: {serialized}" + ); + assert!( + serialized.contains("\"backup_dir\""), + "expected backup_dir field for non-default value, got: {serialized}" + ); + assert!( + serialized.contains("\"user_config_dir\""), + "expected user_config_dir field for non-default value, got: {serialized}" + ); + } + + #[test] + fn main_config_from_applies_default_storage_backup_and_user_config_dirs() { + let mut cfg = ConfigDto::default(); + cfg.prepare(false).expect("prepare should succeed"); + let main = MainConfigDto::from(&cfg); + assert_eq!(main.storage_dir.as_deref(), Some(DEFAULT_STORAGE_DIR)); + assert_eq!(main.backup_dir.as_deref(), Some(DEFAULT_BACKUP_DIR)); + assert_eq!(main.user_config_dir.as_deref(), Some(DEFAULT_USER_CONFIG_DIR)); + assert_eq!(main.mapping_path.as_deref(), Some(format!("./{CONFIG_PATH}/{MAPPING_FILE}").as_str())); + assert_eq!(main.template_path.as_deref(), Some(format!("./{CONFIG_PATH}/{TEMPLATE_FILE}").as_str())); + } + + #[test] + fn update_from_main_config_omits_default_optional_paths() { + let mut cfg = ConfigDto::default(); + let main = MainConfigDto { + storage_dir: Some(DEFAULT_STORAGE_DIR.to_string()), + backup_dir: Some(DEFAULT_BACKUP_DIR.to_string()), + user_config_dir: Some(DEFAULT_USER_CONFIG_DIR.to_string()), + mapping_path: Some(format!("./{CONFIG_PATH}/{MAPPING_FILE}")), + template_path: Some(format!("./{CONFIG_PATH}/{TEMPLATE_FILE}")), + ..MainConfigDto::default() + }; + + cfg.update_from_main_config(&main); + assert!(cfg.storage_dir.is_none()); + assert!(cfg.backup_dir.is_none()); + assert!(cfg.user_config_dir.is_none()); + assert!(cfg.mapping_path.is_none()); + assert!(cfg.template_path.is_none()); + } + + #[test] + fn prepare_sets_default_optional_paths() { + let mut cfg = ConfigDto { + storage_dir: None, + backup_dir: None, + user_config_dir: None, + mapping_path: None, + template_path: None, + ..ConfigDto::default() + }; + cfg.prepare(false).expect("prepare should succeed"); + assert_eq!(cfg.storage_dir.as_deref(), Some(DEFAULT_STORAGE_DIR)); + assert_eq!(cfg.backup_dir.as_deref(), Some(DEFAULT_BACKUP_DIR)); + assert_eq!(cfg.user_config_dir.as_deref(), Some(DEFAULT_USER_CONFIG_DIR)); + assert_eq!(cfg.mapping_path.as_deref(), Some(format!("./{CONFIG_PATH}/{MAPPING_FILE}").as_str())); + assert_eq!(cfg.template_path.as_deref(), Some(format!("./{CONFIG_PATH}/{TEMPLATE_FILE}").as_str())); + } + #[test] fn deserializing_rejects_legacy_video_ffprobe_fields() { let raw = json!({ @@ -433,7 +520,7 @@ mod tests { "port": 8901, "web_root": "./web" }, - "working_dir": ".", + "storage_dir": ".", "video": { "extensions": ["mp4"], "ffprobe_enabled": true @@ -445,4 +532,48 @@ mod tests { let err = result.unwrap_err().to_string(); assert!(err.contains("ffprobe_enabled"), "unexpected error text: {err}"); } + + #[test] + fn deserializing_rejects_legacy_data_dir_alias() { + let raw = json!({ + "api": { + "host": "127.0.0.1", + "port": 8901, + "web_root": "./web" + }, + "data_dir": "." + }); + + let result: Result = serde_json::from_value(raw); + assert!(result.is_err(), "data_dir should not deserialize"); + } + + #[test] + fn deserializing_accepts_legacy_working_dir_alias() { + let raw = json!({ + "api": { + "host": "127.0.0.1", + "port": 8901, + "web_root": "./web" + }, + "working_dir": "." + }); + + let cfg: ConfigDto = serde_json::from_value(raw).expect("working_dir should deserialize as legacy alias"); + assert_eq!(cfg.storage_dir.as_deref(), Some(".")); + } + + #[test] + fn deserializing_accepts_missing_storage_dir() { + let raw = json!({ + "api": { + "host": "127.0.0.1", + "port": 8901, + "web_root": "./web" + } + }); + + let cfg: ConfigDto = serde_json::from_value(raw).expect("missing storage_dir should deserialize"); + assert!(cfg.storage_dir.is_none()); + } } diff --git a/shared/src/model/config/cache.rs b/shared/src/model/config/cache.rs index eb64059e5..477052415 100644 --- a/shared/src/model/config/cache.rs +++ b/shared/src/model/config/cache.rs @@ -1,10 +1,11 @@ use crate::{ error::TuliproxError, info_err_res, - utils::{is_blank_optional_str, is_blank_optional_string, parse_size_base_2}, + utils::{ + is_blank_optional_str, is_blank_optional_string, is_blank_or_default_cache_dir, parse_size_base_2, + DEFAULT_CACHE_DIR, + }, }; -use path_clean::PathClean; -use std::path::PathBuf; #[derive(Debug, Clone, serde::Serialize, serde::Deserialize, Default, PartialEq)] #[serde(deny_unknown_fields)] @@ -13,27 +14,21 @@ pub struct CacheConfigDto { pub enabled: bool, #[serde(default, skip_serializing_if = "is_blank_optional_string")] pub size: Option, - #[serde(default, skip_serializing_if = "is_blank_optional_string")] - pub dir: Option, + #[serde(default, alias = "dir", skip_serializing_if = "is_blank_or_default_cache_dir")] + pub directory: Option, } impl CacheConfigDto { pub fn is_empty(&self) -> bool { - !self.enabled && is_blank_optional_str(self.size.as_deref()) && is_blank_optional_str(self.dir.as_deref()) + !self.enabled && is_blank_optional_str(self.size.as_deref()) && is_blank_optional_str(self.directory.as_deref()) } - pub(crate) fn prepare(&mut self, working_dir: &str) -> Result<(), TuliproxError> { + pub(crate) fn prepare(&mut self, _storage_dir: &str) -> Result<(), TuliproxError> { if self.enabled { - let work_path = PathBuf::from(working_dir); - match self.dir.as_ref() { - None => self.dir = Some(work_path.join("cache").to_string_lossy().to_string()), - Some(work_dir) => { - let mut cache_dir = work_dir.to_string(); - if PathBuf::from(&cache_dir).is_relative() { - cache_dir = work_path.join(&cache_dir).clean().to_string_lossy().to_string(); - } - self.dir = Some(cache_dir.to_string()); - } + if is_blank_or_default_cache_dir(&self.directory) { + self.directory = Some(DEFAULT_CACHE_DIR.to_string()); + } else if let Some(dir) = self.directory.as_ref() { + self.directory = Some(dir.trim().to_string()); } if let Some(val) = self.size.as_ref() { @@ -50,3 +45,36 @@ impl CacheConfigDto { Ok(()) } } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn prepare_sets_default_cache_dir_when_enabled_and_missing() { + let mut cache = CacheConfigDto { enabled: true, directory: None, size: None }; + cache.prepare("storage").expect("prepare should succeed"); + assert_eq!(cache.directory.as_deref(), Some(DEFAULT_CACHE_DIR)); + } + + #[test] + fn prepare_keeps_custom_cache_dir_when_enabled() { + let mut cache = CacheConfigDto { enabled: true, directory: Some("custom-cache".to_string()), size: None }; + cache.prepare("storage").expect("prepare should succeed"); + assert_eq!(cache.directory.as_deref(), Some("custom-cache")); + } + + #[test] + fn serializing_skips_default_cache_dir() { + let cache = CacheConfigDto { enabled: true, directory: Some(DEFAULT_CACHE_DIR.to_string()), size: None }; + let serialized = serde_json::to_string(&cache).expect("cache serialization should succeed"); + assert!(!serialized.contains("\"directory\""), "expected no dir field for default value, got: {serialized}"); + } + + #[test] + fn serializing_keeps_non_default_cache_dir() { + let cache = CacheConfigDto { enabled: true, directory: Some("custom-cache".to_string()), size: None }; + let serialized = serde_json::to_string(&cache).expect("cache serialization should succeed"); + assert!(serialized.contains("\"directory\""), "expected dir field for custom value, got: {serialized}"); + } +} diff --git a/shared/src/model/config/input.rs b/shared/src/model/config/input.rs index f4dd0c7a6..2e345733d 100644 --- a/shared/src/model/config/input.rs +++ b/shared/src/model/config/input.rs @@ -10,11 +10,10 @@ use crate::{ deserialize_timestamp, get_credentials_from_url_str, get_trimmed_string, is_blank_optional_string, is_default_probe_delay_secs, is_default_probe_live_interval, is_default_resolve_delay_secs, is_false, is_true, is_zero_i16, is_zero_u16, parse_duration_seconds, parse_provider_scheme_url_parts, sanitize_sensitive_info, - serialize_option_vec_flow_map_items, trim_last_slash, Internable, PROVIDER_SCHEME_PREFIX, + serialize_option_vec_flow_map_items, trim_last_slash, Internable, BATCH_SCHEME_PREFIX, PROVIDER_SCHEME_PREFIX, }, }; use enum_iterator::Sequence; -use indexmap::IndexMap; use log::warn; use std::{ collections::{HashMap, HashSet}, @@ -511,7 +510,7 @@ impl ConfigInputDto { self.persist = get_trimmed_string(self.persist.as_deref()); - if self.url.starts_with("file://") { + if self.url.starts_with(BATCH_SCHEME_PREFIX) { match self.input_type { InputType::M3u => { self.input_type = InputType::M3uBatch; @@ -524,7 +523,10 @@ impl ConfigInputDto { } else if self.url.starts_with(PROVIDER_SCHEME_PREFIX) && matches!(self.input_type, InputType::M3uBatch | InputType::XtreamBatch) { - return info_err_res!("input type {} does not support provider:// URLs for batch definitions; use a local CSV path or file:// URL", self.input_type); + return info_err_res!( + "input type {} does not support provider:// URLs for batch definitions; use batch:// URL", + self.input_type + ); } check_provider_scheme_url!(self.url, provider_names); @@ -654,7 +656,10 @@ impl ConfigInputDto { if self.url.starts_with(PROVIDER_SCHEME_PREFIX) && matches!(self.input_type, InputType::M3uBatch | InputType::XtreamBatch) { - return info_err_res!("input type {} does not support provider:// URLs for batch definitions; use a local CSV path or file:// URL", self.input_type); + return info_err_res!( + "input type {} does not support provider:// URLs for batch definitions; use batch:// URL", + self.input_type + ); } Ok(()) } @@ -791,8 +796,6 @@ pub struct ProviderDnsDto { pub keep_vhost: bool, #[serde(default, skip_serializing_if = "Option::is_none")] pub overrides: Option>>, - #[serde(default, skip_serializing_if = "Option::is_none")] - pub resolved: Option>>, #[serde(default, skip_serializing_if = "is_default_on_resolve_error")] pub on_resolve_error: OnResolveErrorPolicy, #[serde(default, skip_serializing_if = "is_default_on_connect_error")] @@ -809,7 +812,6 @@ impl Default for ProviderDnsDto { schemes: None, keep_vhost: false, overrides: None, - resolved: None, on_resolve_error: OnResolveErrorPolicy::default(), on_connect_error: OnConnectErrorPolicy::default(), } @@ -1010,29 +1012,12 @@ mod tests { assert_eq!(overrides["example.com"].len(), 1); } - #[test] - fn test_provider_dns_resolved_deserializes() { - let json = r#"{ - "name":"p1", - "urls":["http://example.com"], - "dns":{ - "enabled":true, - "resolved":{"example.com":["203.0.113.10"]} - } - }"#; - - let dto: ConfigProviderDto = serde_json::from_str(json).expect("provider json should parse"); - let dns = dto.dns.expect("dns should be present"); - let resolved = dns.resolved.expect("resolved must be deserialized"); - assert_eq!(resolved.get("example.com"), Some(&vec!["203.0.113.10".parse::().expect("valid ip")])); - } - #[test] fn prepare_switches_xtream_to_xtream_batch_when_alias_exists() { let mut dto = ConfigInputDto { name: "input_alias".intern(), input_type: InputType::Xtream, - url: "file:///tmp/input_alias.csv".to_string(), + url: "batch:///tmp/input_alias.csv".to_string(), aliases: Some(vec![ConfigInputAliasDto { id: 1, name: "alias_1".intern(), @@ -1046,7 +1031,7 @@ mod tests { }; dto.prepare_type().expect("prepare type should succeed"); - dto.prepare(0, true, &HashSet::new()).expect("prepare should succeed and infer batch type"); + dto.prepare(0, true, &HashSet::new()).expect("prepare should succeed and infer batch type from batch:// URL"); assert_eq!(dto.input_type, InputType::XtreamBatch); } @@ -1069,6 +1054,6 @@ mod tests { }; let err = dto.prepare(0, true, &HashSet::new()).expect_err("prepare must reject provider:// for batch input"); - assert!(err.to_string().contains("does not support provider:// URLs")); + assert!(err.to_string().contains("does not support provider:// URLs"), "Error: {err}"); } } diff --git a/shared/src/model/config/library.rs b/shared/src/model/config/library.rs index abeb68b08..be715cfcd 100644 --- a/shared/src/model/config/library.rs +++ b/shared/src/model/config/library.rs @@ -106,7 +106,7 @@ pub enum LibraryMetadataFormat { Nfo, } -#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Default)] +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)] #[serde(deny_unknown_fields)] pub struct LibraryPlaylistConfigDto { #[serde(default = "default_movie_category")] @@ -119,7 +119,7 @@ impl LibraryPlaylistConfigDto { pub fn is_empty(&self) -> bool { self.movie_category == default_movie_category() && self.series_category == default_series_category() } - pub fn clean(&mut self) { + pub fn prepare(&mut self) { if self.movie_category.trim().is_empty() { self.movie_category = default_movie_category(); } @@ -127,10 +127,19 @@ impl LibraryPlaylistConfigDto { self.series_category = default_series_category(); } } + pub fn clean(&mut self) { self.prepare(); } +} + +impl Default for LibraryPlaylistConfigDto { + fn default() -> Self { + Self { movie_category: default_movie_category(), series_category: default_series_category() } + } } impl LibraryConfigDto { pub fn prepare(&mut self) -> Result<(), TuliproxError> { + self.playlist.prepare(); + // Validate enabled state if self.enabled && self.scan_directories.is_empty() { return info_err_res!("Library enabled but no scan_directories configured"); @@ -146,3 +155,24 @@ impl LibraryConfigDto { Ok(()) } } + +#[cfg(test)] +mod tests { + use super::{default_movie_category, default_series_category, LibraryPlaylistConfigDto}; + + #[test] + fn playlist_default_uses_default_categories() { + let playlist = LibraryPlaylistConfigDto::default(); + assert_eq!(playlist.movie_category, default_movie_category()); + assert_eq!(playlist.series_category, default_series_category()); + } + + #[test] + fn playlist_prepare_sets_default_categories_when_empty() { + let mut playlist = + LibraryPlaylistConfigDto { movie_category: String::new(), series_category: " ".to_string() }; + playlist.prepare(); + assert_eq!(playlist.movie_category, default_movie_category()); + assert_eq!(playlist.series_category, default_series_category()); + } +} diff --git a/shared/src/model/config/paths.rs b/shared/src/model/config/paths.rs index 2d5903e09..e4264a550 100644 --- a/shared/src/model/config/paths.rs +++ b/shared/src/model/config/paths.rs @@ -2,7 +2,11 @@ use crate::utils::{is_blank_optional_string, is_empty_optional_vec}; #[derive(Debug, Clone, serde::Serialize, serde::Deserialize)] pub struct ConfigPaths { + #[serde(default, skip_serializing_if = "String::is_empty")] + pub home_path: String, pub config_path: String, + #[serde(default, skip_serializing_if = "String::is_empty")] + pub storage_path: String, pub config_file_path: String, pub sources_file_path: String, #[serde(default, skip_serializing_if = "is_blank_optional_string")] diff --git a/shared/src/model/config/reverse_proxy.rs b/shared/src/model/config/reverse_proxy.rs index 68307a205..84d3147e1 100644 --- a/shared/src/model/config/reverse_proxy.rs +++ b/shared/src/model/config/reverse_proxy.rs @@ -89,7 +89,7 @@ impl ReverseProxyConfigDto { } } - pub(crate) fn prepare(&mut self, working_dir: &str) -> Result<(), TuliproxError> { + pub(crate) fn prepare(&mut self, storage_dir: &str) -> Result<(), TuliproxError> { self.rewrite_secret = self.rewrite_secret.trim().to_string(); if !self.resource_rewrite_disabled { if self.rewrite_secret.is_empty() { @@ -106,7 +106,7 @@ impl ReverseProxyConfigDto { warn!("The cache is disabled because resource rewrite is disabled"); cache.enabled = false; } - cache.prepare(working_dir)?; + cache.prepare(storage_dir)?; } if let Some(rate_limit) = self.rate_limit.as_mut() { diff --git a/shared/src/model/config/schedule.rs b/shared/src/model/config/schedule.rs index 4bbe38e75..04add9ee8 100644 --- a/shared/src/model/config/schedule.rs +++ b/shared/src/model/config/schedule.rs @@ -3,6 +3,51 @@ pub enum ScheduleTaskType { #[default] PlaylistUpdate, LibraryScan, + GeoIpUpdate, +} + +const fn default_schedule_task_type() -> ScheduleTaskType { ScheduleTaskType::PlaylistUpdate } + +fn parse_schedule_task_type(value: &str) -> Option { + match value.trim().to_ascii_lowercase().as_str() { + "playlistupdate" | "playlist_update" | "playlist-update" | "update" => Some(ScheduleTaskType::PlaylistUpdate), + "libraryscan" | "library_scan" | "library-scan" | "scan" => Some(ScheduleTaskType::LibraryScan), + "geoipupdate" | "geo_ip_update" | "geoip_update" | "geoip-update" | "geoip" => { + Some(ScheduleTaskType::GeoIpUpdate) + } + _ => None, + } +} + +fn deserialize_schedule_task_type<'de, D>(deserializer: D) -> Result +where + D: serde::Deserializer<'de>, +{ + use serde::Deserialize as _; + + #[derive(serde::Deserialize)] + #[serde(untagged)] + enum CompatTaskType { + Enum(ScheduleTaskType), + String(String), + } + + let value = Option::::deserialize(deserializer)?; + match value { + None => Ok(default_schedule_task_type()), + Some(CompatTaskType::Enum(task_type)) => Ok(task_type), + Some(CompatTaskType::String(raw)) => { + if raw.trim().is_empty() { + Ok(default_schedule_task_type()) + } else if let Some(task_type) = parse_schedule_task_type(&raw) { + Ok(task_type) + } else { + Err(serde::de::Error::custom(format!( + "invalid schedule type '{raw}', expected one of: PlaylistUpdate, LibraryScan, GeoIpUpdate" + ))) + } + } + } } #[derive(Debug, Clone, serde::Serialize, serde::Deserialize, Default, PartialEq, Eq)] @@ -10,8 +55,50 @@ pub enum ScheduleTaskType { pub struct ScheduleConfigDto { #[serde(default)] pub schedule: String, - #[serde(default, rename = "type")] + #[serde( + default = "default_schedule_task_type", + rename = "type", + alias = "task_type", + deserialize_with = "deserialize_schedule_task_type" + )] pub task_type: ScheduleTaskType, #[serde(default, skip_serializing_if = "Option::is_none")] pub targets: Option>, } + +#[cfg(test)] +mod tests { + use super::{ScheduleConfigDto, ScheduleTaskType}; + + #[test] + fn defaults_to_playlist_update_when_type_is_missing() { + let dto: ScheduleConfigDto = serde_saphyr::from_str("schedule: \"0 0 * * * * *\"\ntargets: [\"main\"]") + .expect("schedule dto should parse"); + assert_eq!(dto.task_type, ScheduleTaskType::PlaylistUpdate); + } + + #[test] + fn defaults_to_playlist_update_when_type_is_null_or_empty() { + let null_type: ScheduleConfigDto = + serde_saphyr::from_str("schedule: \"0 0 * * * * *\"\ntype: ~").expect("null type should parse"); + let empty_type: ScheduleConfigDto = + serde_saphyr::from_str("schedule: \"0 0 * * * * *\"\ntype: \"\"").expect("empty type should parse"); + + assert_eq!(null_type.task_type, ScheduleTaskType::PlaylistUpdate); + assert_eq!(empty_type.task_type, ScheduleTaskType::PlaylistUpdate); + } + + #[test] + fn accepts_legacy_task_type_aliases() { + let legacy: ScheduleConfigDto = serde_saphyr::from_str("schedule: \"0 0 * * * * *\"\ntask_type: scan") + .expect("legacy task_type alias should parse"); + assert_eq!(legacy.task_type, ScheduleTaskType::LibraryScan); + } + + #[test] + fn accepts_geoip_update_aliases() { + let legacy: ScheduleConfigDto = + serde_saphyr::from_str("schedule: \"0 0 * * * * *\"\ntype: geoip").expect("geoip alias should parse"); + assert_eq!(legacy.task_type, ScheduleTaskType::GeoIpUpdate); + } +} diff --git a/shared/src/model/config/trakt.rs b/shared/src/model/config/trakt.rs index d49cf967f..3e5f6995f 100644 --- a/shared/src/model/config/trakt.rs +++ b/shared/src/model/config/trakt.rs @@ -1,6 +1,6 @@ -use crate::{ - model::DEFAULT_USER_AGENT, - utils::{default_as_true, default_trakt_fuzzy_threshold, is_true, TRAKT_API_KEY, TRAKT_API_URL, TRAKT_API_VERSION}, +use crate::utils::{ + default_as_true, default_trakt_fuzzy_threshold, is_true, DEFAULT_USER_AGENT, TRAKT_API_KEY, TRAKT_API_URL, + TRAKT_API_VERSION, }; use serde::{Deserialize, Serialize}; use std::{fmt, str::FromStr}; diff --git a/shared/src/model/config/video_download.rs b/shared/src/model/config/video_download.rs index f09a40180..7a80b86b3 100644 --- a/shared/src/model/config/video_download.rs +++ b/shared/src/model/config/video_download.rs @@ -1,10 +1,10 @@ use crate::{ error::TuliproxError, info_err_res, - model::DEFAULT_USER_AGENT, utils::{ - default_supported_video_extensions, is_blank_optional_str, is_blank_optional_string, - is_default_supported_video_extensions, is_false, + default_download_dir, default_episode_pattern, default_supported_video_extensions, is_blank_optional_str, + is_blank_optional_string, is_blank_or_default_download_dir, is_blank_or_default_episode_pattern, + is_default_supported_video_extensions, is_false, DEFAULT_USER_AGENT, }, }; use std::{borrow::BorrowMut, collections::HashMap}; @@ -14,12 +14,12 @@ use std::{borrow::BorrowMut, collections::HashMap}; pub struct VideoDownloadConfigDto { #[serde(default, skip_serializing_if = "HashMap::is_empty")] pub headers: HashMap, - #[serde(default, skip_serializing_if = "is_blank_optional_string")] + #[serde(default = "default_download_dir", skip_serializing_if = "is_blank_or_default_download_dir")] pub directory: Option, #[serde(default, skip_serializing_if = "is_false")] pub organize_into_directories: bool, // TODO use ptt - #[serde(default, skip_serializing_if = "is_blank_optional_string")] + #[serde(default = "default_episode_pattern", skip_serializing_if = "is_blank_or_default_episode_pattern")] pub episode_pattern: Option, } @@ -27,8 +27,8 @@ impl VideoDownloadConfigDto { pub fn is_empty(&self) -> bool { !self.organize_into_directories && self.headers.is_empty() - && is_blank_optional_str(self.directory.as_deref()) - && is_blank_optional_str(self.episode_pattern.as_deref()) + && is_blank_or_default_download_dir(&self.directory) + && is_blank_or_default_episode_pattern(&self.episode_pattern) } } @@ -69,11 +69,23 @@ impl VideoConfigDto { match &mut self.download { None => {} Some(downl) => { + if is_blank_or_default_download_dir(&downl.directory) { + downl.directory = default_download_dir(); + } else if let Some(directory) = downl.directory.as_ref() { + downl.directory = Some(directory.trim().to_string()); + } + if downl.headers.is_empty() { downl.headers.borrow_mut().insert("Accept".to_string(), "video/*".to_string()); downl.headers.borrow_mut().insert("User-Agent".to_string(), DEFAULT_USER_AGENT.to_string()); } + if is_blank_or_default_episode_pattern(&downl.episode_pattern) { + downl.episode_pattern = default_episode_pattern(); + } else if let Some(episode_pattern) = downl.episode_pattern.as_ref() { + downl.episode_pattern = Some(episode_pattern.trim().to_string()); + } + if let Some(episode_pattern) = &downl.episode_pattern { if let Err(err) = crate::model::REGEX_CACHE.get_or_compile(episode_pattern) { return info_err_res!("can't parse regex: {episode_pattern} {err}"); @@ -84,3 +96,102 @@ impl VideoConfigDto { Ok(()) } } + +#[cfg(test)] +mod tests { + use super::*; + use crate::utils::DEFAULT_DOWNLOAD_DIR; + + #[test] + fn prepare_sets_default_download_dir_when_missing() { + let mut video = VideoConfigDto { + extensions: Vec::new(), + download: Some(VideoDownloadConfigDto { + headers: HashMap::new(), + directory: None, + organize_into_directories: false, + episode_pattern: None, + }), + web_search: None, + }; + video.prepare().expect("prepare should succeed"); + let download = video.download.expect("download should exist"); + assert_eq!(download.directory.as_deref(), Some(DEFAULT_DOWNLOAD_DIR)); + } + + #[test] + fn prepare_sets_default_episode_pattern_when_missing() { + let mut video = VideoConfigDto { + extensions: Vec::new(), + download: Some(VideoDownloadConfigDto { + headers: HashMap::new(), + directory: None, + organize_into_directories: false, + episode_pattern: None, + }), + web_search: None, + }; + video.prepare().expect("prepare should succeed"); + let download = video.download.expect("download should exist"); + assert!(download.episode_pattern.is_some(), "expected default episode pattern to be set"); + } + + #[test] + fn prepare_keeps_custom_download_dir() { + let mut video = VideoConfigDto { + extensions: Vec::new(), + download: Some(VideoDownloadConfigDto { + headers: HashMap::new(), + directory: Some("custom-downloads".to_string()), + organize_into_directories: false, + episode_pattern: None, + }), + web_search: None, + }; + video.prepare().expect("prepare should succeed"); + let download = video.download.expect("download should exist"); + assert_eq!(download.directory.as_deref(), Some("custom-downloads")); + } + + #[test] + fn serializing_skips_default_download_dir() { + let download = VideoDownloadConfigDto { + headers: HashMap::new(), + directory: Some(DEFAULT_DOWNLOAD_DIR.to_string()), + organize_into_directories: false, + episode_pattern: None, + }; + let serialized = serde_json::to_string(&download).expect("download serialization should succeed"); + assert!( + !serialized.contains("\"directory\""), + "expected no directory field for default value, got: {serialized}" + ); + } + + #[test] + fn serializing_keeps_custom_download_dir() { + let download = VideoDownloadConfigDto { + headers: HashMap::new(), + directory: Some("custom-downloads".to_string()), + organize_into_directories: false, + episode_pattern: None, + }; + let serialized = serde_json::to_string(&download).expect("download serialization should succeed"); + assert!(serialized.contains("\"directory\""), "expected directory field for custom value, got: {serialized}"); + } + + #[test] + fn serializing_skips_default_episode_pattern() { + let download = VideoDownloadConfigDto { + headers: HashMap::new(), + directory: None, + organize_into_directories: false, + episode_pattern: default_episode_pattern(), + }; + let serialized = serde_json::to_string(&download).expect("download serialization should succeed"); + assert!( + !serialized.contains("\"episode_pattern\""), + "expected no episode_pattern field for default value, got: {serialized}" + ); + } +} diff --git a/shared/src/model/config/web_auth.rs b/shared/src/model/config/web_auth.rs index dbcd0e14c..d98301af2 100644 --- a/shared/src/model/config/web_auth.rs +++ b/shared/src/model/config/web_auth.rs @@ -1,5 +1,5 @@ use crate::utils::{ - default_as_true, default_token_ttl_mins, is_blank_optional_str, is_blank_optional_string, + default_as_true, default_token_ttl_mins, default_user_file_path, is_blank_or_default_user_file_path, is_default_token_ttl_mins, is_true, }; @@ -12,7 +12,7 @@ pub struct WebAuthConfigDto { pub secret: String, #[serde(default = "default_token_ttl_mins", skip_serializing_if = "is_default_token_ttl_mins")] pub token_ttl_mins: u32, - #[serde(default, skip_serializing_if = "is_blank_optional_string")] + #[serde(default = "default_user_file_path", skip_serializing_if = "is_blank_or_default_user_file_path")] pub userfile: Option, } @@ -35,6 +35,6 @@ impl WebAuthConfigDto { && self.token_ttl_mins == empty.token_ttl_mins && self.issuer.trim().is_empty() && self.secret.trim().is_empty() - && is_blank_optional_str(self.userfile.as_deref()) + && is_blank_or_default_user_file_path(&self.userfile) } } diff --git a/shared/src/utils/constants.rs b/shared/src/utils/constants.rs index a29d7a1f8..937025c2a 100644 --- a/shared/src/utils/constants.rs +++ b/shared/src/utils/constants.rs @@ -7,6 +7,14 @@ use std::{ pub const DEFAULT_HOME_ENV_VAR: &str = "TULIPROX_HOME"; pub const DEFAULT_WEB_ROOT_ENV_VAR: &str = "TULIPROX_WEB_ROOT"; +pub const DEFAULT_USER_AGENT: &str = "VLC/3.0.16 LibVLC/3.0.16"; +pub const DEFAULT_BACKUP_DIR: &str = "backup"; +pub const DEFAULT_CACHE_DIR: &str = "cache"; +pub const DEFAULT_STORAGE_TEMP_DIR: &str = "tmp"; +pub const DEFAULT_USER_CONFIG_DIR: &str = "user_config"; +pub const DEFAULT_DOWNLOAD_DIR: &str = "downloads"; +pub const DEFAULT_CUSTOM_STREAM_RESPONSE_PATH: &str = "resources"; +pub const DNS_RESOLVED_FILE: &str = "provider_dns_resolved.json"; pub const USER_FILE: &str = "user.txt"; pub const CONFIG_PATH: &str = "config"; pub const CONFIG_FILE: &str = "config.yml"; @@ -15,7 +23,7 @@ pub const MAPPING_FILE: &str = "mapping.yml"; pub const TEMPLATE_FILE: &str = "template.yml"; pub const API_PROXY_FILE: &str = "api-proxy.yml"; pub const DEFAULT_WEB_DIR: &str = "web"; -pub const DEFAULT_WORKING_DIR: &str = "data"; +pub const DEFAULT_STORAGE_DIR: &str = "data"; // TODO rename to storage and use data for config, storage, ... pub const DEFAULT_PORT: u16 = 8901; pub const ENCODING_GZIP: &str = "gzip"; pub const ENCODING_DEFLATE: &str = "deflate"; @@ -31,6 +39,8 @@ pub const HLS_EXT_FRAGMENT: &str = ".m3u8#"; pub const DASH_EXT_QUERY: &str = ".mpd?"; pub const DASH_EXT_FRAGMENT: &str = ".mpd#"; +pub const DEFAULT_EPISODE_PATTERN: &str = r".*(?P[Ss]\d{1,2}(.*?)[Ee]\d{1,2}).*"; + pub const FILENAME_TRIM_PATTERNS: &[char] = &['.', '-', '_']; const SUPPORTED_RESPONSE_HEADERS: &[&str] = &[ diff --git a/shared/src/utils/default_utils.rs b/shared/src/utils/default_utils.rs index f029e31e6..a2bb0b122 100644 --- a/shared/src/utils/default_utils.rs +++ b/shared/src/utils/default_utils.rs @@ -1,4 +1,11 @@ -use crate::model::{ConfigTargetOptions, LibraryMetadataFormat, ProcessingOrder}; +use crate::{ + model::{ConfigTargetOptions, LibraryMetadataFormat, MetadataUpdateConfigDto, ProcessingOrder, VideoConfigDto}, + utils::{ + CONFIG_PATH, DEFAULT_BACKUP_DIR, DEFAULT_CACHE_DIR, DEFAULT_CUSTOM_STREAM_RESPONSE_PATH, DEFAULT_DOWNLOAD_DIR, + DEFAULT_EPISODE_PATTERN, DEFAULT_STORAGE_DIR, DEFAULT_USER_AGENT, DEFAULT_USER_CONFIG_DIR, DEFAULT_WEB_DIR, + MAPPING_FILE, TEMPLATE_FILE, USER_FILE, + }, +}; use std::sync::Arc; pub const fn is_zero_u16(v: &u16) -> bool { *v == 0 } @@ -218,6 +225,112 @@ pub fn is_default_metadata_ffprobe_live_probe_size(v: &String) -> bool { *v == default_metadata_ffprobe_live_probe_size() } +pub fn get_default_web_root() -> String { DEFAULT_WEB_DIR.to_string() } +pub fn is_blank_or_default_web_root(value: &str) -> bool { + let normalized = value.trim().replace('\\', "/"); + if normalized.is_empty() { + return true; + } + + let normalized = normalized.trim_end_matches('/'); + normalized.trim_start_matches("./") == DEFAULT_WEB_DIR +} + +pub fn is_default_dir_path(value: &str, default_dir: &str) -> bool { + let normalized = value.trim().replace('\\', "/"); + let normalized = normalized.trim_end_matches('/'); + let normalized = normalized.trim_start_matches("./"); + let default_dir = default_dir.trim().replace('\\', "/"); + let default_dir = default_dir.trim_end_matches('/'); + normalized == default_dir +} + +pub fn is_blank_or_default_download_dir(path: &Option) -> bool { + path.as_ref().is_none_or(|value| value.trim().is_empty() || is_default_dir_path(value, DEFAULT_DOWNLOAD_DIR)) +} +pub fn default_download_dir() -> Option { Some(DEFAULT_DOWNLOAD_DIR.to_string()) } + +pub fn default_episode_pattern() -> Option { Some(DEFAULT_EPISODE_PATTERN.to_string()) } + +pub fn is_blank_or_default_episode_pattern(path: &Option) -> bool { + path.as_ref().is_none_or(|value| value.trim().is_empty() || value.trim() == DEFAULT_EPISODE_PATTERN) +} + +pub fn is_blank_or_default_cache_dir(path: &Option) -> bool { + path.as_ref().is_none_or(|value| value.trim().is_empty() || is_default_dir_path(value, DEFAULT_CACHE_DIR)) +} + +pub fn default_default_user_agent() -> Option { Some(DEFAULT_USER_AGENT.to_string()) } +pub fn default_main_storage_dir() -> Option { Some(DEFAULT_STORAGE_DIR.to_string()) } +pub fn default_main_backup_dir() -> Option { Some(DEFAULT_BACKUP_DIR.to_string()) } +pub fn default_main_user_config_dir() -> Option { Some(DEFAULT_USER_CONFIG_DIR.to_string()) } +pub fn default_main_mapping_path() -> Option { Some(format!("./{CONFIG_PATH}/{MAPPING_FILE}")) } +pub fn default_main_template_path() -> Option { Some(format!("./{CONFIG_PATH}/{TEMPLATE_FILE}")) } +pub fn default_custom_stream_response_path() -> Option { Some(DEFAULT_CUSTOM_STREAM_RESPONSE_PATH.to_string()) } +pub fn default_user_file_path() -> Option { Some(format!("./{CONFIG_PATH}/{USER_FILE}")) } +fn is_default_config_file_path(value: &str, file_name: &str) -> bool { + let normalized = value.trim().replace('\\', "/"); + let normalized = normalized.trim_start_matches("./"); + normalized == file_name + || normalized.rsplit_once('/').is_some_and(|(dir, file)| dir == CONFIG_PATH && file == file_name) +} + +pub fn is_blank_or_default_custom_stream_response_path(path: &Option) -> bool { + path.as_ref() + .is_none_or(|value| value.trim().is_empty() || is_default_dir_path(value, DEFAULT_CUSTOM_STREAM_RESPONSE_PATH)) +} + +pub fn is_blank_or_default_mapping_path(path: &Option) -> bool { + path.as_ref().is_none_or(|value| value.trim().is_empty() || is_default_config_file_path(value, MAPPING_FILE)) +} + +pub fn is_blank_or_default_template_path(path: &Option) -> bool { + path.as_ref().is_none_or(|value| value.trim().is_empty() || is_default_config_file_path(value, TEMPLATE_FILE)) +} + +pub fn is_blank_or_default_storage_dir(path: &Option) -> bool { + path.as_ref().is_none_or(|value| value.trim().is_empty() || is_default_dir_path(value, DEFAULT_STORAGE_DIR)) +} + +pub fn is_blank_or_default_backup_dir(path: &Option) -> bool { + path.as_ref().is_none_or(|value| value.trim().is_empty() || is_default_dir_path(value, DEFAULT_BACKUP_DIR)) +} + +pub fn is_blank_or_default_user_config_dir(path: &Option) -> bool { + path.as_ref().is_none_or(|value| value.trim().is_empty() || is_default_dir_path(value, DEFAULT_USER_CONFIG_DIR)) +} + +pub fn is_blank_or_default_user_file_path(path: &Option) -> bool { + path.as_ref().is_none_or(|value| value.trim().is_empty() || is_default_config_file_path(value, USER_FILE)) +} + +pub fn normalize_optional_dir(path: &Option, default_dir: &str) -> Option { + path.as_ref().and_then(|value| { + if value.trim().is_empty() || is_default_dir_path(value, default_dir) { + None + } else { + Some(value.clone()) + } + }) +} + +pub fn normalize_optional_config_file_path(path: &Option, default_file_name: &str) -> Option { + path.as_ref().and_then(|value| { + if value.trim().is_empty() || is_default_config_file_path(value, default_file_name) { + None + } else { + Some(value.clone()) + } + }) +} + +pub fn is_none_or_empty_video(video: &Option) -> bool { + video.as_ref().is_none_or(VideoConfigDto::is_empty) +} +pub fn is_none_or_empty_metadata_update(metadata_update: &Option) -> bool { + metadata_update.as_ref().is_none_or(MetadataUpdateConfigDto::is_empty) +} + ////////////////////////////// // HDHomerun Device Defaults ////////////////////////////// diff --git a/shared/src/utils/request.rs b/shared/src/utils/request.rs index 93a7b8d13..3160cfc78 100644 --- a/shared/src/utils/request.rs +++ b/shared/src/utils/request.rs @@ -8,6 +8,7 @@ use std::{borrow::Cow, sync::atomic::Ordering}; use url::Url; pub const PROVIDER_SCHEME_PREFIX: &str = "provider://"; +pub const BATCH_SCHEME_PREFIX: &str = "batch://"; pub const CONTENT_TYPE_JSON: &str = "application/json"; pub const CONTENT_TYPE_CBOR: &str = "application/cbor";