Feature/staged input and path improvements (#632)

* No template file not found error message
* Setup wizard improvmenets
* path improvements
* staged input fix
* path improvements
* Provider dns resolved ips now writing to a file instead of source.yml
* Playlist update hard kill after 1h
* matadata update tmdb fix
* Web UI fixes
* GeoIp update schedule
* batch:// prefix  for aliases
This commit is contained in:
euzu
2026-03-06 17:29:42 +01:00
committed by GitHub
parent c5632d0c98
commit 737d04d08b
82 changed files with 2506 additions and 1433 deletions
+23
View File
@@ -4,6 +4,7 @@
## ⚠️ Breaking Changes
- `working_dir` in `config.yml` renamed to `storage_dir`.
- **Global Input Definitions**: To align input definitions with the SourceEditor, inputs are now defined globally in the `inputs` section of the
config file. Each source can reference one or more inputs by their name in the `inputs` attribute.
- **Data Format Migration**: Due to heavy refactoring, the old data format is invalid. You need to clean your `data` folder and update the playlists.
@@ -38,6 +39,27 @@
fallback_to_filename: true
```
- **Input Batch URL Scheme**: Batch input URLs now use the `batch://` scheme instead of `file://`.
`file://` is no longer accepted for batch CSV definitions. Update your `source.yml`:
```yaml
# Before
inputs:
- type: xtream_batch
url: 'file:///home/tuliprox/config/batch.csv'
# After
inputs:
- type: xtream_batch
url: 'batch:///home/tuliprox/config/batch.csv'
```
Local paths without a scheme (`/path/file.csv`, `./file.csv`) continue to work.
The `batch://` scheme clearly distinguishes batch alias files from provider `file://` URLs.
- **DNS Resolved Persistence**: `dns.resolved` has been removed from `source.yml` and the `ProviderDnsDto`.
Resolved IPs are now persisted separately in `{storage_dir}/provider_dns_resolved.json`.
This eliminates hot-reload interference caused by DNS refresh cycles writing to `source.yml`.
DNS caches are automatically carried over during config hot-reloads.
- **Input Batch Changes**: `name` attribute is now mandatory for input type batch to ensure stable playlist UUIDs.
- **Favorites Redesign**: Replaced implicit `create_alias` with explicit `add_favourite(group_name)` script function.
- **EpgSmartMatch**: Field `name_prefix` syntax needs to be changed from `name_prefix: !suffix "."` to `name_prefix: { suffix: "." }`.
@@ -104,6 +126,7 @@
- **Playlist Caching**: Added `cache_duration` to inputs, allowing configurable provider playlist cache times during subsequent updates (e.g., `60s`,
`5m` `12h`, `1d`).
- **Database Viewer**: New CLI flags `--dbx` and `--dbm` to inspect internal database content.
- **Home Directory Override**: Added `--home` (`-H`) CLI argument to set the base directory for config, data, backup, and downloads.
- **Added `disk_based_processing`**: (boolean, default `false`) to `config.yml`. When enabled, input playlists are processed from disk instead of
memory.
- **User-Agent `default_user_agent`**: Ensures that outgoing requests always pass a default user agent.
+86 -30
View File
@@ -61,6 +61,7 @@ Define complex filters using expressive logic, e.g.:
Usage: tuliprox [OPTIONS]
Options:
-H, --home <HOME> The home directory (base for config, storage, backup, downloads, web)
-p, --config-path <CONFIG_PATH> The config directory
-c, --config <CONFIG_FILE> The config file
-i, --source <SOURCE_FILE> The source config file
@@ -83,10 +84,27 @@ Options:
```
Home directory resolution order:
1. `--home`
2. `TULIPROX_HOME` environment variable
3. Directory of the `tuliprox` binary
All relative paths that are resolved against `home` (including `storage_dir` when configured as relative) use this order.
Default layout under `{home}`:
- `{home}/config`
- `{home}/data` (`storage_dir` default)
- `{home}/data/backup` (`backup_dir` default = `{storage_dir}/backup`)
- `{home}/downloads`
- `{home}/web`
## 1. `config.yml`
For running in cli mode, you need to define a `config.yml` file which can be inside config directory next to the executable or provided with the
`-c` cli argument.
For CLI mode, `config.yml` is loaded from `{home}/config/config.yml` by default.
`{home}` is resolved in this order: `--home` -> `TULIPROX_HOME` -> directory of the `tuliprox` binary.
You can override the config file with `-c` (and config directory with `-p`).
For running specific targets use the `-t` argument like `tuliprox -t <target_name> -t <other_target_name>`.
Target names should be provided in the config. The -t option overrides `enabled` attributes of `input` and `target` elements.
@@ -95,7 +113,7 @@ This means, even disabled inputs and targets are processed when the given target
Top level entries in the config files are:
- `api`
- `working_dir`
- `storage_dir`
- `default_user_agent` _optional_, used as fallback for upstream requests when no `User-Agent` is provided by input headers or the client request
(client request overrides it).
- `process_parallel` _optional_
@@ -132,12 +150,16 @@ If you process the same provider multiple times each thread uses a connection. K
`api` contains the `server-mode` settings. To run `tuliprox` in `server-mode` you need to start it with the `-s`cli argument.
-`api: {host: localhost, port: 8901, web_root: ./web}`
### 1.3. `working_dir`
`web_root` follows the same home-based resolution rules:
with `web_root: ./web`, the effective path is `{home}/web` (`--home` -> `TULIPROX_HOME` -> binary directory).
`working_dir` is the directory where files are written which are given with relative paths.
-`working_dir: ./data`
### 1.3. `storage_dir`
With this configuration, you should create a `data` directory where you execute the binary.
`storage_dir` is the directory where files are written which are given with relative paths.
-`storage_dir: ./data`
`storage_dir` is resolved relative to `home` (resolved via `--home`, then `TULIPROX_HOME`, then binary directory).
With `storage_dir: ./data`, the effective directory is `{home}/data`.
Be aware that different configurations (e.g. user bouquets) along the playlists are stored in this directory.
@@ -208,7 +230,6 @@ Each provider can optionally enable a `dns` block:
- `overrides`: static host -> IP list (used before DNS lookup)
- `on_resolve_error`: `keep_last_good` | `fallback_to_hostname` (default: `keep_last_good`)
- `on_connect_error`: `try_next_ip` | `rotate_provider_url` (default: `try_next_ip`)
- `resolved`: runtime-managed resolved snapshot per host
Behavior:
@@ -219,12 +240,12 @@ Behavior:
- `keep_vhost=true`: `Host` header keeps `hostname[:port]`.
- On connect/timeout errors and `on_connect_error=try_next_ip`, Tuliprox tries the next IP for the same host before rotating provider URL.
### 1.4.4 `dns.resolved` persistence and visibility
### 1.4.4 DNS resolved IP persistence
- `dns.resolved` is runtime-managed.
- It is exposed in `GET /api/v1/config`.
- It is also written back to `source.yml` and overwritten on each DNS refresh cycle.
- Save endpoints treat `dns.resolved` as managed data and do not accept it as user-controlled input.
Resolved IPs are persisted to `{storage_dir}/provider_dns_resolved.json` (not to `source.yml`).
This file is written atomically after each DNS refresh cycle and read at startup to seed DNS caches
before the background resolver completes its first cycle. On config hot-reloads, DNS caches are
carried over from the previous provider instances so that resolved IPs are available immediately.
### 1.4.5 Provider Failover + DNS Configuration Example
@@ -253,10 +274,6 @@ provider:
overrides:
stable.golden-bridge.con:
- 203.0.113.10
# runtime-managed, written by tuliprox:
resolved:
hello.provider.me:
- 203.0.113.20
inputs:
- name: my_input
type: xtream
@@ -477,7 +494,7 @@ metadata_update:
**Group overview:**
- `cache_path` (default `metadata`): Directory where TMDB cache files and metadata are stored. Relative paths are resolved against `working_dir`.
- `cache_path` (default `metadata`): Directory where TMDB cache files and metadata are stored. Relative paths are resolved against `storage_dir`.
Used by all metadata resolution paths (Xtream VOD/Series, local library).
- `resolve`: Controls retries for metadata lookup tasks (for example title/date/TMDB resolution).
- `probe`: Controls retries and cooldown for technical stream probing tasks.
@@ -653,12 +670,25 @@ schedules:
- xtream
- schedule: "0 0 20 * * * *"
type: LibraryScan
- schedule: "0 0 4 * * 1 *"
type: GeoIpUpdate
- schedule: "0 0 4 1 * * *"
type: GeoIpUpdate
```
The `type` attribute defines the task to be executed and defaults to `PlaylistUpdate` if omitted. Possible values:
- `PlaylistUpdate`: Updates the target playlists (optionally filtered by `targets`).
- `LibraryScan`: Triggers a scan of the local media library (requires `library` configuration to be enabled).
- `GeoIpUpdate`: Downloads and rebuilds the Geo-IP database file (requires `reverse_proxy.geoip.enabled: true`).
Cron fields for Tuliprox schedules always start with **seconds**:
```yaml
# sec min hour day-of-month month day-of-week year
schedule: "0 0 4 * * 1 *" # every Monday at 04:00:00
schedule: "0 0 4 1 * * *" # every 1st of the month at 04:00:00
```
At the given times the update is started. Do not start it every second or minute.
You could be banned from your server. Twice a day should be enough.
@@ -741,7 +771,7 @@ This is useful for players that might time out or error if they receive data and
#### 1.7.2 `cache`
LRU-Cache is for resources. If it is `enabled`, the resources/images are persisted in the given `dir`. If the cache size exceeds `size`,
LRU-Cache is for resources. If it is `enabled`, the resources/images are persisted in the given `directory`. If the cache size exceeds `size`,
In an LRU cache, the least recently used items are evicted to make room for new items if the cache `size`is exceeded.
#### 1.7.3 `resource_rewrite_disabled`
@@ -762,7 +792,7 @@ reverse_proxy:
cache:
enabled: true
size: 1GB
dir: ./cache
directory: ./cache
```
#### 1.7.4 `rate_limit`
@@ -983,13 +1013,17 @@ The encrypted pasword needs to be added manually into the users file.
```yaml
threads: 4
working_dir: ./data
storage_dir: ./data
api:
host: localhost
port: 8901
web_root: ./web
```
Relative paths in this example are resolved under `{home}`.
So `storage_dir: ./data` -> `{home}/data` and `web_root: ./web` -> `{home}/web`
(`--home` -> `TULIPROX_HOME` -> binary directory).
### 1.12 `user_access_control`
The default is `false`.
@@ -1463,6 +1497,23 @@ inputs:
probe_stream: true
```
#### Input URL Schemes (`inputs[].url`)
Tuliprox supports the following URL schemes for input sources:
- `http://...` / `https://...`: Download from remote provider endpoints.
- `file://...`: Read from a local file.
- `provider://<provider_name>/...`: Resolve the URL via `provider` definitions (supports failover/rotation).
- `batch://...`: CSV source for batch aliases (e.g. `batch:///path/file.csv` or `batch://./file.csv`).
Notes:
- If `type` is `m3u` or `xtream` and the URL starts with `batch://`, Tuliprox automatically treats it as `m3u_batch` / `xtream_batch`.
- For `m3u_batch` and `xtream_batch`, only local CSV sources are supported:
- `batch://...` or
- plain local filesystem path (absolute/relative).
- For batch inputs, `provider://...` and other URI schemes such as `http(s)://...` or `file://...` are rejected.
Input alias definition for same provider with same content but different credentials.
`max_connections` default is unlimited
@@ -1488,7 +1539,7 @@ sources:
Input aliases can be defined as batches in csv files with `;` separator.
There are 2 batch input types `xtream_batch` and `m3u_batch`.
Batch inputs are file-based only: use a local path (`/path/file.csv`, `./file.csv`) or `file://...`.
Batch inputs use the `batch://` URL scheme: `batch:///path/file.csv` (absolute) or `batch://./file.csv` (relative).
#### `XtreamBatch`
@@ -1496,7 +1547,7 @@ Batch inputs are file-based only: use a local path (`/path/file.csv`, `./file.cs
inputs:
- type: xtream_batch
name: my_provider # Mandatory: used for playlist UUID generation
url: 'file:///home/tuliprox/config/my_provider_batch.csv'
url: 'batch:///home/tuliprox/config/my_provider_batch.csv'
sources:
- inputs:
- my_provider
@@ -1520,7 +1571,7 @@ This is necessary because of playlist uuid generation and assigning same channel
```yaml
inputs:
- type: m3u_batch
url: 'file:///home/tuliprox/config/my_provider_batch.csv'
url: 'batch:///home/tuliprox/config/my_provider_batch.csv'
sources:
- inputs:
- m3u_batch
@@ -2993,25 +3044,30 @@ The provider gives you:
- epg_url: `http://fantastic.provider.xyz:8080/xmltv.php?username=tvjunkie&password=junkie.secret`
To use `tuliprox` you need to create the configuration.
The configuration consist of 4 files.
The configuration consists of 3 required files and up to 2 optional files.
Required files:
- config.yml
- source.yml
- mapping.yml
- api-proxy.yml
The file `mapping.yml`is optional and only needed if you want to do something linke renaming titles or changing attributes.
Optional files:
- mapping.yml (for advanced transformations like renaming titles or changing attributes)
- template.yml (for centralized pattern reuse)
Lets start with `config.yml`. An example basic configuration is:
```yaml
api: {host: 0.0.0.0, port: 8901, web_root: ./web}
working_dir: ./data
storage_dir: ./data
update_on_boot: true
```
This configuration starts `tuliprox`and listens on the 8901 port. The downloaded playlists are stored inside the `data`-folder in the current working
directory.
This configuration starts `tuliprox`and listens on the 8901 port. The downloaded playlists are stored inside the `data` folder under the resolved
home directory (`--home`, then `TULIPROX_HOME`, then binary directory), i.e. `{home}/data`.
`web_root: ./web` resolves to `{home}/web` with the same order.
The property `update_on_boot` is optional and can be helpful in the beginning until you have found a working configuration. I prefer to set it to
false.
+13 -9
View File
@@ -223,20 +223,24 @@ impl ConfigFile {
// ── Parse ────────────────────────────────────────────────────
let config_dto = read_config_file(config_file.as_str(), true, true)?;
let default_mapping_path = utils::get_default_mappings_path(paths.config_path.as_str());
let current_mapping_path = paths.mapping_file_path.clone().unwrap_or_else(|| default_mapping_path.clone());
let next_mapping_path =
config_dto.mapping_path.clone().filter(|path| !path.trim().is_empty()).unwrap_or(default_mapping_path);
let current_mapping_path = paths.mapping_file_path.clone().unwrap_or_else(|| {
utils::resolve_mapping_file_path(paths.config_path.as_str(), None)
});
let next_mapping_path = paths.mapping_file_path.clone().unwrap_or_else(|| {
utils::resolve_mapping_file_path(paths.config_path.as_str(), config_dto.mapping_path.as_deref())
});
let mapping_changed = current_mapping_path != next_mapping_path;
let default_template_path = utils::get_default_templates_path(paths.config_path.as_str());
let current_template_path = paths.template_file_path.clone().unwrap_or_else(|| default_template_path.clone());
let next_template_path =
config_dto.template_path.clone().filter(|path| !path.trim().is_empty()).unwrap_or(default_template_path);
let current_template_path = paths.template_file_path.clone().unwrap_or_else(|| {
utils::resolve_template_file_path(paths.config_path.as_str(), None)
});
let next_template_path = paths.template_file_path.clone().unwrap_or_else(|| {
utils::resolve_template_file_path(paths.config_path.as_str(), config_dto.template_path.as_deref())
});
let template_changed = current_template_path != next_template_path;
let mut config: Config = Config::from(config_dto);
config.prepare(paths.config_path.as_str())?;
config.prepare(paths.config_path.as_str(), paths.home_path.as_str())?;
// Compute effective runtime paths for the NEW config before apply.
// This ensures prepare-phase reads/validates against the same files that will be active after apply.
@@ -129,7 +129,7 @@ pub(in crate::api::endpoints) async fn get_playlist_for_custom_provider(
let cfg = app_state.app_config.config.load();
match cfg_input {
Some(input) => {
let (result, errors) = match input.input_type {
let (result, errors) = match input.get_download_input_type() {
InputType::M3u | InputType::M3uBatch => {
m3u::download_m3u_playlist(&app_state.app_config, client, &cfg, input).await
}
+5 -5
View File
@@ -34,10 +34,10 @@ async fn scan_library(
};
// Check if Library is enabled
let (lib_config, metadata_update_config, working_dir) = {
let (lib_config, metadata_update_config, storage_dir) = {
let config = app_state.app_config.config.load();
match config.library.as_ref() {
Some(lib) if lib.enabled => (lib.clone(), config.metadata_update.clone(), config.working_dir.clone()),
Some(lib) if lib.enabled => (lib.clone(), config.metadata_update.clone(), config.storage_dir.clone()),
_ => {
let response = LibraryScanSummary {
status: LibraryScanSummaryStatus::Error,
@@ -61,7 +61,7 @@ async fn scan_library(
lib_config,
metadata_update_config,
client,
LibraryScanTaskOptions { force_rescan: request.force_rescan, message_prefix: "", working_dir },
LibraryScanTaskOptions { force_rescan: request.force_rescan, message_prefix: "", storage_dir },
permit,
);
@@ -78,7 +78,7 @@ async fn get_library_status(
let client = app_state.http_client.load_full().as_ref().clone();
// Get statistics from processor
let processor =
LibraryProcessor::new(config.clone(), config_snapshot.metadata_update.as_ref(), client, &config_snapshot.working_dir);
LibraryProcessor::new(config.clone(), config_snapshot.metadata_update.as_ref(), client, &config_snapshot.storage_dir);
let entries = processor.get_all_entries().await;
let movies = entries.iter().filter(|e| e.metadata.is_movie()).count();
@@ -90,7 +90,7 @@ async fn get_library_status(
movies,
series,
path: Some(
resolve_metadata_storage_path(config_snapshot.metadata_update.as_ref(), &config_snapshot.working_dir)
resolve_metadata_storage_path(config_snapshot.metadata_update.as_ref(), &config_snapshot.storage_dir)
.to_string_lossy()
.to_string(),
),
+13 -66
View File
@@ -10,22 +10,16 @@ use crate::{
model::AppState,
},
auth::validator_admin,
model::InputSource,
repository::get_geoip_path,
utils::{ip_checker::get_ips, request::download_text_content, GeoIp},
processing::geoip::{update_geoip_db, GeoIpUpdateError},
utils::ip_checker::get_ips,
VERSION,
};
use axum::response::IntoResponse;
use log::{error, info};
use shared::{
model::{default_geoip_url, InputFetchMethod, IpCheckDto, StatusCheck},
utils::{concat_path_leading_slash, Internable},
};
use std::{
collections::{BTreeMap, HashMap},
io::Cursor,
sync::Arc,
model::{IpCheckDto, StatusCheck},
utils::concat_path_leading_slash,
};
use std::{collections::BTreeMap, sync::Arc};
async fn create_ipinfo_check(app_state: &Arc<AppState>) -> Option<(Option<String>, Option<String>)> {
let config = app_state.app_config.config.load();
@@ -85,63 +79,16 @@ async fn streams(
async fn geoip_update(
axum::extract::State(app_state): axum::extract::State<Arc<AppState>>,
) -> axum::response::Response {
let config = app_state.app_config.config.load();
if let Some(geoip) = config.reverse_proxy.as_ref().and_then(|r| r.geoip.as_ref()) {
if geoip.enabled {
let geoip_db_path = &*get_geoip_path(&config.working_dir);
let _file_lock = app_state.app_config.file_locks.write_lock(geoip_db_path).await;
let url = if geoip.url.trim().is_empty() { default_geoip_url() } else { geoip.url.clone() };
let input_source = InputSource {
name: "GeoIP".intern(),
url,
provider: None,
username: None,
password: None,
method: InputFetchMethod::GET,
headers: HashMap::default(),
};
return match download_text_content(
&app_state.app_config,
&app_state.http_client.load(),
&input_source,
None,
None,
false,
)
.await
{
Ok((content, _)) => {
let reader = Cursor::new(content);
let mut geoip = GeoIp::new();
let result = {
match geoip.import_ipv4_from_csv(reader, geoip_db_path) {
Ok(size) => (Some(size), None),
Err(err) => (None, Some(err)),
}
};
return match result {
(Some(_), None) => {
info!("GeoIp db updated");
app_state.geoip.store(Some(Arc::new(geoip)));
axum::http::StatusCode::OK.into_response()
}
(None, Some(err)) => {
error!("Failed to process geoip db: {err}");
internal_server_error!()
}
_ => internal_server_error!(),
};
}
Err(err) => {
error!("Failed to download geoip db: {err}");
axum::http::StatusCode::BAD_REQUEST.into_response()
}
};
match update_geoip_db(&app_state).await {
Ok(()) => axum::http::StatusCode::OK.into_response(),
Err(GeoIpUpdateError::Disabled | GeoIpUpdateError::DownloadFailed(_)) => {
axum::http::StatusCode::BAD_REQUEST.into_response()
}
Err(err) => {
log::error!("GeoIp update failed: {err}");
internal_server_error!()
}
}
axum::http::StatusCode::BAD_REQUEST.into_response()
}
async fn ipinfo(axum::extract::State(app_state): axum::extract::State<Arc<AppState>>) -> axum::response::Response {
+4 -370
View File
@@ -16,11 +16,11 @@ use axum::{
response::IntoResponse,
Router,
};
use log::{error, warn};
use log::error;
use serde_json::json;
use shared::{
error::TuliproxError,
model::{ApiProxyConfigDto, ConfigDto, ConfigProviderDto, ProviderDnsDto, SourcesConfigDto},
model::{ApiProxyConfigDto, ConfigDto, SourcesConfigDto},
utils::{
HEADER_CONFIG_API_PROXY_REVISION, HEADER_CONFIG_MAIN_REVISION, HEADER_CONFIG_SOURCES_REVISION, HEADER_IF_MATCH,
},
@@ -28,112 +28,7 @@ use shared::{
use std::path::Path;
use std::sync::Arc;
fn inject_provider_dns_resolved(sources_dto: &mut SourcesConfigDto, runtime_sources: &crate::model::SourcesConfig) {
let Some(provider_dtos) = sources_dto.provider.as_mut() else {
return;
};
for provider_dto in provider_dtos {
let Some(runtime_provider) = runtime_sources.get_provider_by_name(provider_dto.name.as_ref()) else {
continue;
};
let Some(dns_dto) = provider_dto.dns.as_mut() else {
continue;
};
if runtime_provider.get_dns_config().is_none() {
dns_dto.resolved = None;
continue;
}
let snapshot = runtime_provider.snapshot_resolved_ordered();
dns_dto.resolved = (!snapshot.is_empty()).then_some(snapshot);
}
}
fn strip_provider_dns_resolved(sources_dto: &mut SourcesConfigDto) {
let Some(provider_dtos) = sources_dto.provider.as_mut() else {
return;
};
for provider_dto in provider_dtos {
if let Some(dns_dto) = provider_dto.dns.as_mut() {
dns_dto.resolved = None;
}
}
}
fn merge_provider_dns_resolved_from_existing(
sources_dto: &mut SourcesConfigDto,
existing_sources_dto: &SourcesConfigDto,
) {
let Some(provider_dtos) = sources_dto.provider.as_mut() else {
return;
};
let Some(existing_provider_dtos) = existing_sources_dto.provider.as_ref() else {
return;
};
let mut resolved_by_provider_name = std::collections::HashMap::new();
for provider in existing_provider_dtos {
let Some(dns) = provider.dns.as_ref() else {
continue;
};
let Some(resolved) = dns.resolved.as_ref() else {
continue;
};
if !resolved.is_empty() {
resolved_by_provider_name.insert(provider.name.to_string(), resolved.clone());
}
}
if resolved_by_provider_name.is_empty() {
return;
}
for provider in provider_dtos {
let Some(dns) = provider.dns.as_mut() else {
continue;
};
if !dns.enabled {
continue;
}
if dns.resolved.is_some() {
continue;
}
if let Some(resolved) = resolved_by_provider_name.get(provider.name.as_ref()) {
dns.resolved = Some(resolved.clone());
}
}
}
fn build_existing_sources_from_runtime(runtime_sources: &crate::model::SourcesConfig) -> Option<SourcesConfigDto> {
let providers: Vec<ConfigProviderDto> = runtime_sources
.provider
.iter()
.filter_map(|runtime_provider| {
let resolved = runtime_provider.snapshot_resolved_ordered();
if resolved.is_empty() {
return None;
}
Some(ConfigProviderDto {
name: runtime_provider.name.clone(),
urls: runtime_provider.urls.clone(),
dns: Some(ProviderDnsDto {
enabled: runtime_provider.get_dns_config().is_some_and(|cfg| cfg.enabled),
resolved: Some(resolved),
..ProviderDnsDto::default()
}),
})
})
.collect();
if providers.is_empty() {
None
} else {
Some(SourcesConfigDto {
provider: Some(providers),
..SourcesConfigDto::default()
})
}
}
fn file_revision_from_bytes(bytes: &[u8]) -> String { blake3::hash(bytes).to_hex().to_string() }
@@ -273,7 +168,7 @@ async fn save_config_main(
async fn save_config_sources(
axum::extract::State(app_state): axum::extract::State<Arc<AppState>>,
headers: HeaderMap,
axum::extract::Json(mut sources): axum::extract::Json<SourcesConfigDto>,
axum::extract::Json(sources): axum::extract::Json<SourcesConfigDto>,
) -> impl axum::response::IntoResponse + Send {
let sources_file_path = {
let paths = app_state.app_config.paths.load();
@@ -294,27 +189,6 @@ async fn save_config_sources(
return response;
}
let existing_sources_dto =
match utils::read_sources_file_from_path(Path::new(&sources_file_path), false, false, None).await {
Ok(existing) => Some(existing),
Err(err) => {
warn!("Failed to preload existing source.yml from '{sources_file_path}' before save: {err}");
let runtime_sources = app_state.app_config.sources.load();
let runtime_fallback = build_existing_sources_from_runtime(runtime_sources.as_ref());
if runtime_fallback.is_some() {
warn!("Using runtime provider dns snapshot as fallback for preserving dns.resolved");
}
runtime_fallback
}
};
// `dns.resolved` is runtime-managed and must not be accepted from API input,
// but existing runtime values should survive UI saves until the next DNS refresh updates them.
strip_provider_dns_resolved(&mut sources);
if let Some(existing_sources_dto) = existing_sources_dto.as_ref() {
merge_provider_dns_resolved_from_existing(&mut sources, existing_sources_dto);
}
let templates_to_persist = match utils::validate_source_config_for_persist(&app_state, &sources).await {
Ok(value) => value,
Err(err) => {
@@ -499,8 +373,6 @@ async fn config(axum::extract::State(app_state): axum::extract::State<Arc<AppSta
error!("Failed to prepare users: {err}");
internal_server_error!()
} else {
let runtime_sources = app_state.app_config.sources.load();
inject_provider_dns_resolved(&mut app_config.sources, &runtime_sources);
let response = axum::response::Json(app_config).into_response();
let response = response_with_revision_header(response, HEADER_CONFIG_MAIN_REVISION, &main_revision);
let response = response_with_revision_header(response, HEADER_CONFIG_SOURCES_REVISION, &sources_revision);
@@ -562,247 +434,9 @@ pub fn v1_api_config_register(router: Router<Arc<AppState>>) -> axum::Router<Arc
#[cfg(test)]
mod tests {
use super::{
build_existing_sources_from_runtime, inject_provider_dns_resolved, merge_provider_dns_resolved_from_existing,
require_matching_revision, strip_provider_dns_resolved,
};
use super::require_matching_revision;
use axum::http::{HeaderMap, HeaderValue, StatusCode};
use crate::model::{ConfigProvider, SourcesConfig};
use indexmap::IndexMap;
use shared::model::{ConfigProviderDto, DnsScheme, ProviderDnsDto, SourcesConfigDto};
use shared::utils::{HEADER_CONFIG_SOURCES_REVISION, HEADER_IF_MATCH};
use std::{net::IpAddr, sync::Arc};
#[test]
fn inject_provider_dns_resolved_populates_runtime_snapshot() {
let mut dto = SourcesConfigDto {
provider: Some(vec![ConfigProviderDto {
name: "p1".into(),
urls: vec!["http://example.com".into()],
dns: Some(ProviderDnsDto {
enabled: true,
schemes: Some(vec![DnsScheme::Http]),
..ProviderDnsDto::default()
}),
}]),
..SourcesConfigDto::default()
};
let runtime_provider = Arc::new(ConfigProvider::from(&ConfigProviderDto {
name: "p1".into(),
urls: vec!["http://example.com".into()],
dns: Some(ProviderDnsDto {
enabled: true,
schemes: Some(vec![DnsScheme::Http]),
..ProviderDnsDto::default()
}),
}));
runtime_provider.store_resolved(
"example.com",
vec!["203.0.113.10".parse::<IpAddr>().expect("ip parse should work")],
);
let runtime_sources = SourcesConfig {
provider: vec![runtime_provider],
..SourcesConfig::default()
};
inject_provider_dns_resolved(&mut dto, &runtime_sources);
let resolved = dto.provider.as_ref()
.and_then(|providers| providers.first())
.and_then(|provider| provider.dns.as_ref())
.and_then(|dns| dns.resolved.as_ref())
.expect("resolved dns snapshot should be present");
assert_eq!(
resolved.get("example.com"),
Some(&vec!["203.0.113.10".parse::<IpAddr>().expect("ip parse should work")])
);
}
#[test]
fn inject_provider_dns_resolved_clears_value_when_runtime_dns_disabled() {
let mut dto = SourcesConfigDto {
provider: Some(vec![ConfigProviderDto {
name: "p1".into(),
urls: vec!["http://example.com".into()],
dns: Some(ProviderDnsDto {
enabled: true,
resolved: Some(IndexMap::from([(
"example.com".to_string(),
vec!["203.0.113.10".parse::<IpAddr>().expect("ip parse should work")],
)])),
..ProviderDnsDto::default()
}),
}]),
..SourcesConfigDto::default()
};
let runtime_provider = Arc::new(ConfigProvider::from(&ConfigProviderDto {
name: "p1".into(),
urls: vec!["http://example.com".into()],
dns: None,
}));
let runtime_sources = SourcesConfig {
provider: vec![runtime_provider],
..SourcesConfig::default()
};
inject_provider_dns_resolved(&mut dto, &runtime_sources);
let resolved = dto.provider.as_ref()
.and_then(|providers| providers.first())
.and_then(|provider| provider.dns.as_ref())
.and_then(|dns| dns.resolved.as_ref());
assert!(resolved.is_none(), "resolved output must be empty when runtime dns is disabled");
}
#[test]
fn strip_provider_dns_resolved_removes_payload_values() {
let mut dto = SourcesConfigDto {
provider: Some(vec![ConfigProviderDto {
name: "p1".into(),
urls: vec!["http://example.com".into()],
dns: Some(ProviderDnsDto {
enabled: true,
resolved: Some(IndexMap::from([(
"example.com".to_string(),
vec!["203.0.113.10".parse::<IpAddr>().expect("ip parse should work")],
)])),
..ProviderDnsDto::default()
}),
}]),
..SourcesConfigDto::default()
};
strip_provider_dns_resolved(&mut dto);
let resolved = dto
.provider
.as_ref()
.and_then(|providers| providers.first())
.and_then(|provider| provider.dns.as_ref())
.and_then(|dns| dns.resolved.as_ref());
assert!(resolved.is_none(), "resolved must be stripped from incoming payload");
}
#[test]
fn merge_provider_dns_resolved_from_existing_preserves_runtime_snapshot() {
let mut incoming = SourcesConfigDto {
provider: Some(vec![ConfigProviderDto {
name: "p1".into(),
urls: vec!["http://example.com".into()],
dns: Some(ProviderDnsDto {
enabled: true,
..ProviderDnsDto::default()
}),
}]),
..SourcesConfigDto::default()
};
let existing = SourcesConfigDto {
provider: Some(vec![ConfigProviderDto {
name: "p1".into(),
urls: vec!["http://example.com".into()],
dns: Some(ProviderDnsDto {
enabled: true,
resolved: Some(IndexMap::from([(
"example.com".to_string(),
vec!["203.0.113.10".parse::<IpAddr>().expect("ip parse should work")],
)])),
..ProviderDnsDto::default()
}),
}]),
..SourcesConfigDto::default()
};
merge_provider_dns_resolved_from_existing(&mut incoming, &existing);
let resolved = incoming
.provider
.as_ref()
.and_then(|providers| providers.first())
.and_then(|provider| provider.dns.as_ref())
.and_then(|dns| dns.resolved.as_ref())
.expect("resolved should be merged from existing source dto");
assert_eq!(
resolved.get("example.com"),
Some(&vec!["203.0.113.10".parse::<IpAddr>().expect("ip parse should work")])
);
}
#[test]
fn merge_provider_dns_resolved_from_existing_skips_disabled_dns() {
let mut incoming = SourcesConfigDto {
provider: Some(vec![ConfigProviderDto {
name: "p1".into(),
urls: vec!["http://example.com".into()],
dns: Some(ProviderDnsDto {
enabled: false,
..ProviderDnsDto::default()
}),
}]),
..SourcesConfigDto::default()
};
let existing = SourcesConfigDto {
provider: Some(vec![ConfigProviderDto {
name: "p1".into(),
urls: vec!["http://example.com".into()],
dns: Some(ProviderDnsDto {
enabled: true,
resolved: Some(IndexMap::from([(
"example.com".to_string(),
vec!["203.0.113.10".parse::<IpAddr>().expect("ip parse should work")],
)])),
..ProviderDnsDto::default()
}),
}]),
..SourcesConfigDto::default()
};
merge_provider_dns_resolved_from_existing(&mut incoming, &existing);
let resolved = incoming
.provider
.as_ref()
.and_then(|providers| providers.first())
.and_then(|provider| provider.dns.as_ref())
.and_then(|dns| dns.resolved.as_ref());
assert!(resolved.is_none(), "resolved should not be merged for disabled dns entries");
}
#[test]
fn build_existing_sources_from_runtime_exports_provider_dns_resolved() {
let runtime_provider = Arc::new(ConfigProvider::from(&ConfigProviderDto {
name: "p1".into(),
urls: vec!["http://example.com".into()],
dns: Some(ProviderDnsDto {
enabled: true,
schemes: Some(vec![DnsScheme::Http]),
..ProviderDnsDto::default()
}),
}));
runtime_provider.store_resolved(
"example.com",
vec!["203.0.113.10".parse::<IpAddr>().expect("ip parse should work")],
);
let runtime_sources = SourcesConfig {
provider: vec![runtime_provider],
..SourcesConfig::default()
};
let exported =
build_existing_sources_from_runtime(&runtime_sources).expect("runtime fallback dto should exist");
let resolved = exported
.provider
.as_ref()
.and_then(|providers| providers.first())
.and_then(|provider| provider.dns.as_ref())
.and_then(|dns| dns.resolved.as_ref())
.expect("resolved should be present in runtime fallback dto");
assert_eq!(
resolved.get("example.com"),
Some(&vec!["203.0.113.10".parse::<IpAddr>().expect("ip parse should work")])
);
}
#[test]
fn require_matching_revision_rejects_missing_if_match_header() {
+3 -3
View File
@@ -10,7 +10,7 @@ use std::sync::Arc;
pub(crate) struct LibraryScanTaskOptions {
pub force_rescan: bool,
pub message_prefix: &'static str,
pub working_dir: String,
pub storage_dir: String,
}
pub(crate) fn spawn_library_scan(
@@ -21,11 +21,11 @@ pub(crate) fn spawn_library_scan(
options: LibraryScanTaskOptions,
permit: UpdateGuardPermit,
) {
let LibraryScanTaskOptions { force_rescan, message_prefix, working_dir } = options;
let LibraryScanTaskOptions { force_rescan, message_prefix, storage_dir } = options;
let prefix = message_prefix.to_string();
tokio::spawn(async move {
let _permit = permit;
let processor = LibraryProcessor::new(lib_config, metadata_update_config.as_ref(), client, &working_dir);
let processor = LibraryProcessor::new(lib_config, metadata_update_config.as_ref(), client, &storage_dir);
match processor.scan(force_rescan).await {
Ok(result) => {
info!("{prefix}Library scan completed successfully");
+1 -1
View File
@@ -204,7 +204,7 @@ async fn create_shared_data(
let use_geoip = config.is_geoip_enabled();
let geoip = if use_geoip {
let path = get_geoip_path(&config.working_dir);
let path = get_geoip_path(&config.storage_dir);
let _file_lock = app_config.file_locks.read_lock(&path).await;
match GeoIp::load(&path) {
Ok(db) => {
@@ -708,7 +708,9 @@ mod tests {
api_proxy: Arc::new(ArcSwapOption::default()),
file_locks: Arc::new(FileLockManager::default()),
paths: Arc::new(ArcSwap::from_pointee(ConfigPaths {
home_path: String::new(),
config_path: String::new(),
storage_path: String::new(),
config_file_path: String::new(),
sources_file_path: String::new(),
mapping_file_path: None,
+96 -17
View File
@@ -266,7 +266,7 @@ fn build_http_client_with_fallback(
pub fn create_cache(config: &Config) -> Option<Arc<Mutex<LRUResourceCache>>> {
let lru_cache = config.reverse_proxy.as_ref().and_then(|r| r.cache.as_ref()).and_then(|c| {
if c.enabled {
Some(LRUResourceCache::new(c.size, c.dir.as_str()))
Some(LRUResourceCache::new(c.size, c.directory.as_str()))
} else {
None
}
@@ -341,20 +341,23 @@ pub struct AppState {
impl AppState {
pub(in crate::api::model) async fn set_config(&self, config: Config) -> Result<UpdateChanges, TuliproxError> {
let old_storage_dir = self.app_config.config.load().storage_dir.clone();
let changes = self.detect_changes_for_config(&config);
config.update_runtime();
let use_geoip = config.is_geoip_enabled();
let working_dir = config.working_dir.clone();
let storage_dir = config.storage_dir.clone();
self.active_users.update_config(&config);
self.app_config.set_config(config)?;
self.active_provider.update_config(&self.app_config).await;
self.update_config().await?;
if changes.flags.contains(UpdateChangesFlags::Geoip) {
let geoip_reload_needed =
changes.flags.contains(UpdateChangesFlags::Geoip) || (use_geoip && old_storage_dir != storage_dir);
if geoip_reload_needed {
let new_geoip = if use_geoip {
let path = get_geoip_path(&working_dir);
let path = get_geoip_path(&storage_dir);
let _file_lock = self.app_config.file_locks.read_lock(&path).await;
GeoIp::load(&path).ok().map(Arc::new)
} else {
@@ -381,7 +384,7 @@ impl AppState {
.reverse_proxy
.as_ref()
.and_then(|r| r.cache.as_ref())
.map_or((false, 0, ""), |c| (c.enabled, c.size, c.dir.as_str()));
.map_or((false, 0, ""), |c| (c.enabled, c.size, c.directory.as_str()));
if let Some(cache) = self.cache.load().as_ref() {
if enabled {
@@ -401,6 +404,22 @@ impl AppState {
sources: SourcesConfig,
) -> Result<UpdateChanges, TuliproxError> {
let changes = self.detect_changes_for_sources(&sources);
// Carry over DNS caches from old providers so resolved IPs survive hot-reloads
// without waiting for the background resolver or the persisted-file seed.
{
let old_sources = self.app_config.sources.load();
for new_provider in &sources.provider {
if let Some(old_provider) = old_sources.get_provider_by_name(&new_provider.name) {
if new_provider.get_dns_config().is_some_and(|cfg| cfg.enabled) {
for (host, ips) in old_provider.snapshot_resolved() {
if !ips.is_empty() && new_provider.dns_cache.ip_count(&host) == 0 {
new_provider.dns_cache.store_resolved(&host, ips);
}
}
}
}
}
}
self.app_config.set_sources(sources)?;
self.active_provider.update_config(&self.app_config).await;
@@ -420,6 +439,9 @@ impl AppState {
let old_config = self.app_config.config.load();
let changed_schedules =
change_detect!(schedules_changed, old_config.schedules.as_ref(), config.schedules.as_ref());
let library_enabled = config.library.as_ref().is_some_and(|library| library.enabled);
let old_library_enabled = old_config.library.as_ref().is_some_and(|library| library.enabled);
let changed_library_enabled = library_enabled != old_library_enabled;
let changed_hdhomerun =
change_detect!(hdhomerun_changed, old_config.hdhomerun.as_ref(), config.hdhomerun.as_ref());
let changed_file_watch =
@@ -430,7 +452,7 @@ impl AppState {
let geoip_enabled_old = old_config.is_geoip_enabled();
let mut changes = UpdateChanges { flags: UpdateChangesFlagsSet::new(), targets: None };
changes.set_flag_if(changed_schedules, UpdateChangesFlags::Scheduler);
changes.set_flag_if(changed_schedules || changed_library_enabled || geoip_enabled != geoip_enabled_old, UpdateChangesFlags::Scheduler);
changes.set_flag_if(changed_hdhomerun, UpdateChangesFlags::Hdhomerun);
changes.set_flag_if(changed_file_watch, UpdateChangesFlags::FileWatch);
changes.set_flag_if(geoip_enabled != geoip_enabled_old, UpdateChangesFlags::Geoip);
@@ -582,19 +604,30 @@ fn schedules_changed(a: &[ScheduleConfig], b: &[ScheduleConfig]) -> bool {
if a.len() != b.len() {
return true;
}
let mut used = vec![false; b.len()];
for schedule in a {
let Some(found) = b.iter().find(|&s| s.schedule == schedule.schedule) else {
let Some(found_idx) = b.iter().enumerate().find_map(|(idx, candidate)| {
if used[idx]
|| candidate.schedule != schedule.schedule
|| candidate.task_type != schedule.task_type
{
return None;
}
let targets_match = match (schedule.targets.as_ref(), candidate.targets.as_ref()) {
(None, None) => true,
(Some(_), None) | (None, Some(_)) => false,
(Some(a_targets), Some(b_targets)) => small_vecs_equal_unordered(a_targets, b_targets),
};
if targets_match {
Some(idx)
} else {
None
}
}) else {
return true;
};
match (schedule.targets.as_ref(), found.targets.as_ref()) {
(None, None) => {}
(Some(_), None) | (None, Some(_)) => return true,
(Some(a_targets), Some(b_targets)) => {
if !small_vecs_equal_unordered(a_targets, b_targets) {
return true;
}
}
}
used[found_idx] = true;
}
false
}
@@ -635,7 +668,9 @@ pub struct HdHomerunAppState {
#[cfg(test)]
mod tests {
use super::{should_use_manual_redirect_for_proxy, should_use_manual_redirects_for_env_vars};
use super::{schedules_changed, should_use_manual_redirect_for_proxy, should_use_manual_redirects_for_env_vars};
use crate::model::ScheduleConfig;
use shared::model::ScheduleTaskType;
#[test]
fn should_use_manual_redirect_for_proxy_only_http_or_https() {
@@ -672,4 +707,48 @@ mod tests {
"http://localhost".to_string(),
)]));
}
#[test]
fn schedules_changed_detects_task_type_changes() {
let a = vec![ScheduleConfig {
schedule: "0 0 3 * * * *".to_string(),
task_type: ScheduleTaskType::PlaylistUpdate,
targets: None,
}];
let b = vec![ScheduleConfig {
schedule: "0 0 3 * * * *".to_string(),
task_type: ScheduleTaskType::GeoIpUpdate,
targets: None,
}];
assert!(schedules_changed(&a, &b));
}
#[test]
fn schedules_changed_treats_same_entries_as_unchanged() {
let a = vec![
ScheduleConfig {
schedule: "0 0 3 * * * *".to_string(),
task_type: ScheduleTaskType::GeoIpUpdate,
targets: None,
},
ScheduleConfig {
schedule: "0 0 8 * * * *".to_string(),
task_type: ScheduleTaskType::PlaylistUpdate,
targets: Some(vec!["a".to_string(), "b".to_string()]),
},
];
let b = vec![
ScheduleConfig {
schedule: "0 0 8 * * * *".to_string(),
task_type: ScheduleTaskType::PlaylistUpdate,
targets: Some(vec!["b".to_string(), "a".to_string()]),
},
ScheduleConfig {
schedule: "0 0 3 * * * *".to_string(),
task_type: ScheduleTaskType::GeoIpUpdate,
targets: None,
},
];
assert!(!schedules_changed(&a, &b));
}
}
@@ -596,6 +596,7 @@ impl MetadataUpdateManager {
/// * `task` - The task to process
#[allow(clippy::too_many_lines)]
pub async fn queue_task(&self, input_name: Arc<str>, task: UpdateTask) {
debug!("[Task] Queuing task for input {input_name}: {task}");
// Read app state once and reuse for worker creation when needed.
let app_state_weak = {
let guard = self.app_state.lock().await;
@@ -747,8 +748,12 @@ impl MetadataUpdateManager {
}
} else {
let mut existing = entry.task.lock();
let before = format!("{existing}");
if Self::merge_task_payload(&mut existing, task_to_submit) {
entry.generation.fetch_add(1, Ordering::Relaxed);
debug!("[Task] Merged task for input {input_name}: before={before}, after={existing}");
} else {
debug!("[Task] Task already pending for input {input_name} (no merge needed): {existing}");
}
return SubmitTaskResult::QueuedOrMerged;
}
@@ -987,8 +992,8 @@ impl InputWorker {
let mut task_for_execution = current_task.clone();
if Self::is_resolve_task(&task_for_execution) && self.resolve_exhausted.contains_key(&current_key) {
debug_if_enabled!(
"Skipping exhausted resolve task for input {}: {} (reset window: {}s)",
debug!(
"[Metadata-Task] Skipping task (resolve exhausted) for input {}: {} (reset window: {}s)",
input_name,
task_for_execution,
runtime_settings.resolve_exhaustion_reset_gap_secs
@@ -1003,10 +1008,8 @@ impl InputWorker {
if !skip_execution
&& self.should_skip_recent_no_change_task(&current_key, &task_for_execution, &runtime_settings)
{
debug_if_enabled!(
"Skipping recently-resolved no-change task for input {}: {}",
input_name,
task_for_execution
debug!(
"[Metadata-Task] Skipping task (recently completed with no changes) for input {input_name}: {task_for_execution}",
);
remove_current_task = true;
skip_execution = true;
@@ -1019,19 +1022,21 @@ impl InputWorker {
if let Some(cooldown_until_ts) = tmdb_state.cooldown_until_ts {
if now_ts < cooldown_until_ts {
if let Some(stripped_task) = Self::strip_tmdb_reasons(&task_for_execution) {
debug_if_enabled!(
"TMDB cooldown active for input {}: {} (cooldown_until={}), continuing with non-TMDB reasons",
debug!(
"[Task] TMDB cooldown active for input {}: {}, continuing with non-TMDB reasons (cooldown_until={}, remaining={}s)",
input_name,
task_for_execution,
cooldown_until_ts
cooldown_until_ts,
cooldown_until_ts.saturating_sub(now_ts)
);
task_for_execution = stripped_task;
} else {
debug_if_enabled!(
"Skipping TMDB-only task in cooldown for input {}: {} (cooldown_until={})",
debug!(
"[Metadata-Task] Skipping task (TMDB-only in cooldown) for input {}: {} (cooldown_until={}, remaining={}s)",
input_name,
task_for_execution,
cooldown_until_ts
cooldown_until_ts,
cooldown_until_ts.saturating_sub(now_ts)
);
self.scheduled_requeues.remove(&current_key);
remove_current_task = true;
@@ -1053,11 +1058,12 @@ impl InputWorker {
if active_retry_domain == RetryDomain::Probe {
if let Some(cooldown_until_ts) = active_state.cooldown_until_ts {
if now_ts < cooldown_until_ts {
debug_if_enabled!(
"Skipping probe task in cooldown for input {}: {} (cooldown_until={})",
debug!(
"[Metadata-Task] Skipping task (probe cooldown) for input {}: {} (cooldown_until={}, remaining={}s)",
input_name,
task_for_execution,
cooldown_until_ts
cooldown_until_ts,
cooldown_until_ts.saturating_sub(now_ts)
);
self.scheduled_requeues.remove(&current_key);
remove_current_task = true;
@@ -1069,6 +1075,14 @@ impl InputWorker {
}
if !skip_execution && active_state.next_allowed_at_ts > now_ts {
debug!(
"[Task] Deferring task (retry backoff) for input {}: {} (next_allowed_at={}, wait={}s, attempts={})",
input_name,
task_for_execution,
active_state.next_allowed_at_ts,
active_state.next_allowed_at_ts.saturating_sub(now_ts),
active_state.attempts
);
schedule_requeue_at_ts = Some(active_state.next_allowed_at_ts);
skip_execution = true;
}
@@ -1103,6 +1117,12 @@ impl InputWorker {
}
if !skip_execution {
debug!(
"[Task] Executing task for input {}: {} (retry_domain={:?})",
input_name,
task_for_execution,
Self::retry_domain_for_task(&task_for_execution)
);
let task_result = {
let Some(_pause_guard) = self.wait_for_update_pause_window().await else {
break;
@@ -1129,7 +1149,7 @@ impl InputWorker {
Self::should_trigger_playlist_update_for_task(&task_for_execution, task_outcome.task_changed);
cycle_had_changes |= trigger_playlist_update;
debug!(
"Processed metadata task for input {input_name}: {task_for_execution} (changed={}, trigger_playlist_update={}, tmdb_pending={})",
"[Metadata-Task] Task succeeded for input {input_name}: {task_for_execution} (changed={}, trigger_playlist_update={}, tmdb_pending={})",
task_outcome.task_changed,
trigger_playlist_update,
task_outcome.tmdb_pending
@@ -1151,11 +1171,12 @@ impl InputWorker {
tmdb_state.cooldown_until_ts = Some(cooldown_until_ts);
tmdb_state.last_error =
Some("TMDB lookup completed without matching result".to_string());
debug_if_enabled!(
"TMDB resolve produced no match for input {}: {} (cooldown_until={})",
debug!(
"[Metadata-Task] TMDB resolve produced no match (existing retry state), entering cooldown for input {}: {} (cooldown_until={}, cooldown_duration={}s)",
input_name,
task_for_execution,
cooldown_until_ts
cooldown_until_ts,
runtime_settings.tmdb_cooldown_secs
);
} else {
state_bundle.clear_domain(RetryDomain::Tmdb);
@@ -1183,11 +1204,12 @@ impl InputWorker {
};
self.retry_states.insert(current_key.clone(), state_bundle.clone());
state_after_success = Some(state_bundle);
debug_if_enabled!(
"TMDB resolve produced no match for input {}: {} (cooldown_until={})",
debug!(
"[Metadata-Task] TMDB resolve produced no match (new retry state), entering cooldown for input {}: {} (cooldown_until={}, cooldown_duration={}s)",
input_name,
task_for_execution,
cooldown_until_ts
cooldown_until_ts,
runtime_settings.tmdb_cooldown_secs
);
}
@@ -1207,6 +1229,10 @@ impl InputWorker {
&& !task_outcome.tmdb_pending
{
let reasons = Self::task_reason(&task_for_execution);
debug!(
"[Task] Caching no-change result for input {}: {} (reasons={}, ttl={}s)",
input_name, task_for_execution, reasons, runtime_settings.no_change_cache_ttl_secs
);
self.recently_completed_no_change.insert(current_key.clone(), (Instant::now(), reasons));
} else {
self.recently_completed_no_change.remove(&current_key);
@@ -1237,6 +1263,10 @@ impl InputWorker {
}
Err(e) => {
if Self::is_permanent_not_found_error(&e.message) {
debug!(
"[Task] Task failed with permanent not-found for input {}: {} (error={})",
input_name, task_for_execution, e.message
);
let retry_domain = Self::retry_domain_for_task(&task_for_execution);
self.scheduled_requeues.remove(&current_key);
if retry_domain == RetryDomain::Probe {
@@ -1253,11 +1283,12 @@ impl InputWorker {
};
metadata_persist_state = Some(Some(state_bundle_after_update));
remove_current_task = true;
debug_if_enabled!(
"Probe task entered cooldown after permanent not-found for input {}: {} (cooldown_until={})",
debug!(
"[Metadata-Task] Probe task entering cooldown after permanent not-found for input {}: {} (cooldown_until={}, cooldown_duration={}s)",
input_name,
task_for_execution,
cooldown_until_ts
cooldown_until_ts,
runtime_settings.probe_cooldown_secs
);
} else {
self.resolve_exhausted.insert(current_key.clone(), now_ts);
@@ -1277,10 +1308,11 @@ impl InputWorker {
}
metadata_persist_state = Some(state_after_clear);
remove_current_task = true;
debug_if_enabled!(
"Resolve task marked exhausted after permanent not-found for input {}: {}",
debug!(
"[Metadata-Task] Resolve task marked exhausted after permanent not-found for input {}: {} (error={})",
input_name,
task_for_execution
task_for_execution,
e.message
);
}
} else if Self::is_transient_worker_error(&e.message) {
@@ -1293,8 +1325,8 @@ impl InputWorker {
Self::compute_retry_delay_secs(current_task.delay(), &runtime_settings);
let retry_delay_i64 = i64::try_from(retry_delay_secs).unwrap_or(i64::MAX);
schedule_requeue_at_ts = Some(now_ts.saturating_add(retry_delay_i64));
debug_if_enabled!(
"Transient task deferral for input {}: {} (retry_in={}s, err={})",
debug!(
"[Metadata-Task] Task deferred (transient error) for input {}: {} (retry_in={}s, error={})",
input_name,
task_for_execution,
retry_delay_secs,
@@ -1350,12 +1382,14 @@ impl InputWorker {
let cooldown_until = state_after_update
.cooldown_until_ts
.map_or_else(|| "none".to_string(), |ts| ts.to_string());
debug_if_enabled!(
"Probe task exhausted for input {}: {} (attempts={}, cooldown_until={})",
debug!(
"Metadata-[Task] Probe task exhausted (max attempts reached) for input {}: {} (attempts={}/{}, cooldown_until={}, error={})",
input_name,
task_for_execution,
state_after_update.attempts,
cooldown_until
max_attempts,
cooldown_until,
e.message
);
} else {
self.resolve_exhausted.insert(current_key.clone(), now_ts);
@@ -1375,22 +1409,26 @@ impl InputWorker {
}
metadata_persist_state = Some(state_after_clear);
remove_current_task = true;
debug_if_enabled!(
"Resolve task exhausted for input {}: {} (attempts={})",
debug!(
"[Metadata-Task] Resolve task exhausted (max attempts reached) for input {}: {} (attempts={}/{}, error={})",
input_name,
task_for_execution,
attempts
attempts,
max_attempts,
e.message
);
}
} else {
schedule_requeue_at_ts = Some(state_after_update.next_allowed_at_ts);
metadata_persist_state = Some(Some(state_bundle_after_update));
debug_if_enabled!(
"Task failed for input {}, scheduling retry: {} (attempt={}, next_allowed_at={}, err={})",
debug!(
"[Metadata-Task] Task failed, scheduling retry for input {}: {} (attempt={}/{}, next_allowed_at={}, backoff={}s, error={})",
input_name,
task_for_execution,
attempts,
max_attempts,
state_after_update.next_allowed_at_ts,
state_after_update.next_allowed_at_ts.saturating_sub(now_ts),
e.message
);
}
@@ -1413,6 +1451,13 @@ impl InputWorker {
}
if let Some(retry_at_ts) = schedule_requeue_at_ts {
debug!(
"[Task] Scheduling requeue for input {}: {:?} (retry_at_ts={}, in={}s)",
input_name,
current_key,
retry_at_ts,
retry_at_ts.saturating_sub(chrono::Utc::now().timestamp())
);
self.schedule_requeue_at(current_key.clone(), retry_at_ts);
}
@@ -1510,8 +1555,8 @@ impl InputWorker {
return;
};
let working_dir = app_state.app_config.config.load().working_dir.clone();
let Ok(storage_path) = get_input_storage_path(input_name, &working_dir).await else {
let storage_dir = app_state.app_config.config.load().storage_dir.clone();
let Ok(storage_path) = get_input_storage_path(input_name, &storage_dir).await else {
warn!("Could not resolve storage path for metadata retry state on input {input_name}");
self.metadata_retry_load_retry_at_ts =
Some(now_ts.saturating_add(runtime_settings.metadata_retry_load_retry_delay_secs));
@@ -2026,7 +2071,7 @@ impl InputWorker {
let Some(app_state) = app_state_weak.and_then(Weak::upgrade) else { return };
let app_config = &app_state.app_config;
let working_dir = &app_config.config.load().working_dir;
let cfg = app_config.config.load();
let vod_updates = batch_buffer.take_vod_updates();
let series_updates = batch_buffer.take_series_updates();
let live_updates = batch_buffer.take_live_updates();
@@ -2035,7 +2080,7 @@ impl InputWorker {
return;
}
if let Ok(storage_path) = get_input_storage_path(input_name, working_dir).await {
if let Ok(storage_path) = get_input_storage_path(input_name, &cfg.storage_dir).await {
if !vod_updates.is_empty() {
let mut updates: Vec<(u32, VideoStreamProperties)> = Vec::with_capacity(vod_updates.len());
for (id, props) in &vod_updates {
@@ -2531,8 +2576,8 @@ impl InputWorker {
}
if let std::collections::hash_map::Entry::Vacant(entry) = db_handles.entry(cluster) {
let working_dir = &app_state.app_config.config.load().working_dir;
if let Ok(storage_path) = get_input_storage_path(input_name, working_dir).await {
let cfg = app_state.app_config.config.load();
if let Ok(storage_path) = get_input_storage_path(input_name, &cfg.storage_dir).await {
let file_path = xtream_get_file_path(&storage_path, cluster);
if file_path.exists() {
let lock = app_state.app_config.file_locks.read_lock(&file_path).await;
+37 -133
View File
@@ -1,15 +1,15 @@
use crate::api::model::AppState;
use crate::model::ConfigProvider;
use crate::utils::read_sources_file_from_path;
use crate::repository::{
load_persisted_dns_resolved, next_dns_writer_generation, prune_persisted_dns_resolved_to_runtime,
queue_provider_resolved_snapshot, spawn_dns_resolved_writer, DnsResolvedWriteTx,
};
use log::{debug, warn};
use shared::model::{DnsPrefer, OnResolveErrorPolicy, SourcesConfigDto};
use shared::model::{DnsPrefer, OnResolveErrorPolicy};
use std::collections::HashSet;
use std::io;
use std::net::IpAddr;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::time::{Duration, Instant};
use tokio::fs;
use tokio::net::lookup_host;
use tokio_util::sync::CancellationToken;
@@ -105,129 +105,12 @@ async fn resolve_provider(provider: &Arc<ConfigProvider>) -> ProviderResolveStat
stats
}
fn serialize_sources_for_persist(sources: &SourcesConfigDto) -> io::Result<String> {
let mut serialized = String::new();
let options = serde_saphyr::SerializerOptions {
prefer_block_scalars: false,
..Default::default()
};
serde_saphyr::to_fmt_writer_with_options(&mut serialized, sources, options)
.map_err(|err| io::Error::other(format!("Could not serialize source.yml: {err}")))?;
Ok(serialized)
}
async fn write_sources_file_force(path: &Path, sources: &SourcesConfigDto) -> io::Result<()> {
let serialized = serialize_sources_for_persist(sources)?;
let parent_dir = path.parent().ok_or_else(|| {
io::Error::other(format!(
"Could not write source.yml '{}': missing parent directory",
path.display()
))
})?;
let dest_file_name = path.file_name().and_then(|s| s.to_str()).unwrap_or("source.yml");
let mut tmp_path = parent_dir.to_path_buf();
tmp_path.push(format!(
".{dest_file_name}.tmp-{}-{}",
std::process::id(),
chrono::Local::now().timestamp_nanos_opt().unwrap_or_default()
));
fs::write(&tmp_path, serialized).await?;
match fs::rename(&tmp_path, path).await {
Ok(()) => Ok(()),
Err(err) => {
#[cfg(windows)]
{
// Try to rename again after removing destination (Windows often needs this)
if let Ok(()) = fs::remove_file(path).await {
if fs::rename(&tmp_path, path).await.is_ok() {
return Ok(());
}
// Rename still failed after removing dest - fall through to clean up
}
}
let _ = fs::remove_file(&tmp_path).await;
Err(io::Error::other(format!(
"Could not replace '{}' with '{}': {err}",
path.display(),
tmp_path.display()
)))
}
}
}
async fn persist_provider_resolved_to_source_file(app_state: &Arc<AppState>, provider: &Arc<ConfigProvider>) {
let source_file = {
let paths = app_state.app_config.paths.load();
paths.sources_file_path.clone()
};
let source_path = PathBuf::from(&source_file);
let _lock = app_state.app_config.file_locks.write_lock(&source_path).await;
let mut sources_dto = match read_sources_file_from_path(&source_path, false, false, None).await {
Ok(dto) => dto,
Err(err) => {
warn!(
"Provider dns '{}' failed to read source.yml '{}': {err}",
provider.name,
source_path.display()
);
return;
}
};
let Some(provider_dtos) = sources_dto.provider.as_mut() else {
debug!(
"Provider dns '{}' source.yml '{}' has no provider section to persist resolved values",
provider.name,
source_path.display()
);
return;
};
let Some(provider_dto) = provider_dtos.iter_mut().find(|dto| dto.name.as_ref() == provider.name.as_ref()) else {
warn!(
"Provider dns '{}' not found in source.yml '{}', cannot persist resolved values",
provider.name,
source_path.display()
);
return;
};
let Some(dns_dto) = provider_dto.dns.as_mut() else {
debug!(
"Provider dns '{}' has no dns section in source.yml '{}', skipping resolved persist",
provider.name,
source_path.display()
);
return;
};
let resolved_hosts = {
let snapshot = provider.snapshot_resolved_ordered();
dns_dto.resolved = (!snapshot.is_empty()).then_some(snapshot);
dns_dto.resolved.as_ref().map_or(0, indexmap::IndexMap::len)
};
match write_sources_file_force(&source_path, &sources_dto).await {
Ok(()) => {
debug!(
"Provider dns '{}' persisted dns.resolved to '{}' (hosts={resolved_hosts})",
provider.name,
source_path.display()
);
}
Err(err) => {
warn!(
"Provider dns '{}' failed to persist dns.resolved to '{}': {err}",
provider.name,
source_path.display()
);
}
}
}
fn spawn_provider_dns_task(app_state: Arc<AppState>, provider_name: Arc<str>, cancel: CancellationToken) {
fn spawn_provider_dns_task(
app_state: Arc<AppState>,
provider_name: Arc<str>,
cancel: CancellationToken,
writer_tx: DnsResolvedWriteTx,
) {
tokio::spawn(async move {
let mut refresh_secs = 300_u64;
{
@@ -260,7 +143,7 @@ fn spawn_provider_dns_task(app_state: Arc<AppState>, provider_name: Arc<str>, ca
};
refresh_secs = provider.get_dns_config().map_or(300, |dns| dns.refresh_secs.max(10));
let stats = resolve_provider(&provider).await;
persist_provider_resolved_to_source_file(&app_state, &provider).await;
queue_provider_resolved_snapshot(&writer_tx, &provider).await;
let cache_hosts = provider.snapshot_resolved().len();
debug!(
"Provider dns tick '{}' finished: total_hosts={} resolved={} overridden={} empty={} failed={} cache_hosts={} elapsed_ms={}",
@@ -282,6 +165,7 @@ fn spawn_provider_dns_task(app_state: Arc<AppState>, provider_name: Arc<str>, ca
}
pub fn exec_provider_dns(app_state: &Arc<AppState>, cancel: &CancellationToken) {
let generation = next_dns_writer_generation();
let sources = app_state.app_config.sources.load();
let provider_names: Vec<_> = sources
.provider
@@ -292,13 +176,33 @@ pub fn exec_provider_dns(app_state: &Arc<AppState>, cancel: &CancellationToken)
drop(sources);
if provider_names.is_empty() {
let app_for_prune = Arc::clone(app_state);
tokio::spawn(async move {
prune_persisted_dns_resolved_to_runtime(&app_for_prune).await;
});
debug!("Provider dns manager: no enabled providers found");
return;
}
debug!("Provider dns manager: starting {} provider task(s)", provider_names.len());
let app_for_start = Arc::clone(app_state);
let cancel_for_start = cancel.clone();
tokio::spawn(async move {
// Seed runtime caches first, then start writer/tasks to avoid startup races.
load_persisted_dns_resolved(&app_for_start).await;
if cancel_for_start.is_cancelled() {
return;
}
for provider_name in provider_names {
spawn_provider_dns_task(Arc::clone(app_state), provider_name, cancel.clone());
}
let writer_tx = spawn_dns_resolved_writer(Arc::clone(&app_for_start), cancel_for_start.clone(), generation);
debug!("Provider dns manager: starting {} provider task(s)", provider_names.len());
for provider_name in provider_names {
spawn_provider_dns_task(
Arc::clone(&app_for_start),
provider_name,
cancel_for_start.clone(),
writer_tx.clone(),
);
}
});
}
@@ -25,7 +25,6 @@ use reqwest::{
};
use shared::{
create_bitset,
model::{PlaylistItemType, DEFAULT_USER_AGENT},
utils::{filter_request_header, sanitize_sensitive_info},
};
use std::{
@@ -38,6 +37,8 @@ use std::{
time::{Duration, Instant},
};
use url::Url;
use shared::model::PlaylistItemType;
use shared::utils::DEFAULT_USER_AGENT;
const RETRY_SECONDS: u64 = 5;
const ERR_MAX_RETRY_COUNT: u32 = 5;
+38 -2
View File
@@ -1,6 +1,10 @@
use crate::{
api::{library_scan::{spawn_library_scan, LibraryScanTaskOptions}, model::AppState},
api::{
library_scan::{spawn_library_scan, LibraryScanTaskOptions},
model::AppState,
},
model::{AppConfig, ProcessTargets, ScheduleConfig},
processing::geoip::{update_geoip_db, GeoIpUpdateError},
processing::processor::exec_processing,
utils::exit,
};
@@ -39,6 +43,24 @@ pub fn exec_scheduler(
let schedules: Vec<ScheduleConfig> =
if let Some(schedules) = &config.schedules { schedules.clone() } else { vec![] };
for schedule in schedules {
let task_enabled = match schedule.task_type {
ScheduleTaskType::PlaylistUpdate => true,
ScheduleTaskType::LibraryScan => config.library.as_ref().is_some_and(|library| library.enabled),
ScheduleTaskType::GeoIpUpdate => config
.reverse_proxy
.as_ref()
.and_then(|reverse_proxy| reverse_proxy.geoip.as_ref())
.is_some_and(|geoip| geoip.enabled),
};
if !task_enabled {
log::info!(
"Skipping disabled scheduled task {:?} ({})",
schedule.task_type,
schedule.schedule
);
continue;
}
let expression = schedule.schedule.clone();
let task_type = schedule.task_type;
let exec_targets = get_process_targets(cfg, targets, schedule.targets.as_ref());
@@ -75,6 +97,9 @@ async fn start_scheduler(
ScheduleTaskType::LibraryScan => {
run_library_scan(&client, &app_state);
}
ScheduleTaskType::GeoIpUpdate => {
run_geoip_update(&app_state);
}
}
}
() = cancel.cancelled() => {
@@ -131,7 +156,7 @@ fn run_library_scan(client: &reqwest::Client, app_state: &Arc<AppState>) {
LibraryScanTaskOptions {
force_rescan: false,
message_prefix: "Scheduled ",
working_dir: config.working_dir.clone(),
storage_dir: config.storage_dir.clone(),
},
permit,
);
@@ -140,6 +165,17 @@ fn run_library_scan(client: &reqwest::Client, app_state: &Arc<AppState>) {
}
}
fn run_geoip_update(app_state: &Arc<AppState>) {
let app_state = Arc::clone(app_state);
tokio::spawn(async move {
if let Err(err) = update_geoip_db(&app_state).await {
if !matches!(err, GeoIpUpdateError::Disabled) {
log::error!("Scheduled GeoIp update failed: {err}");
}
}
});
}
pub fn get_process_targets(
cfg: &Arc<AppConfig>,
process_targets: &Arc<ProcessTargets>,
+45 -217
View File
@@ -3,8 +3,8 @@ use crate::{
auth::generate_password_from_input,
model::validate_library_paths_from_dto,
utils::{
file_exists, get_default_path, get_default_web_root_path, read_api_proxy_file, read_config_file,
read_sources_file, read_templates_file, resolve_template_persist_file_path, sanitize_sources_for_persist,
file_exists, get_default_web_root_path_for_home, read_api_proxy_file, read_config_file, read_sources_file,
read_templates_file, resolve_template_persist_file_path, sanitize_sources_for_persist,
},
};
use axum::{
@@ -26,10 +26,10 @@ use shared::{
ApiProxyConfigDto, ApiProxyServerInfoDto, AppConfigDto, ConfigApiDto, ConfigDto, ConfigPaths, PatternTemplate,
SourcesConfigDto, TemplateDefinitionDto, TokenResponse, WebAuthConfigDto, WebUiConfigDto, TOKEN_NO_AUTH,
},
utils::{default_kick_secs, hex_encode, DEFAULT_PORT, DEFAULT_WORKING_DIR, USER_FILE},
utils::{default_kick_secs, hex_encode, DEFAULT_PORT, DEFAULT_STORAGE_DIR, USER_FILE},
};
use std::{
collections::{HashMap, HashSet},
collections::HashSet,
io::ErrorKind,
net::{SocketAddr, UdpSocket},
path::{Component, Path as FsPath, PathBuf},
@@ -40,10 +40,9 @@ use std::{
};
use tokio::sync::{oneshot, Mutex, RwLock};
use tower_http::services::ServeDir;
use shared::utils::DEFAULT_CUSTOM_STREAM_RESPONSE_PATH;
const DEFAULT_SETUP_HOST: &str = "0.0.0.0";
const DEFAULT_SETUP_CUSTOM_STREAM_RESPONSE_PATH: &str = "./resources";
const SETUP_REDACTED_SECRET_VALUE: &str = "__TULIPROX_SETUP_REDACTED__";
#[derive(Clone, Serialize, Deserialize)]
pub struct SetupWebUserCredentialDto {
@@ -69,6 +68,7 @@ pub struct SetupCompleteRequestDto {
struct SetupModeState {
draft: Arc<RwLock<AppConfigDto>>,
home_path: String,
output_dir: PathBuf,
config_file_path: PathBuf,
source_file_path: PathBuf,
@@ -166,7 +166,7 @@ fn create_default_api_proxy_server() -> ApiProxyServerInfoDto {
}
}
fn create_default_config_dto() -> ConfigDto {
fn create_default_config_dto(home_path: &str) -> ConfigDto {
let auth = WebAuthConfigDto {
issuer: "tuliprox".to_string(),
secret: generate_web_auth_secret(),
@@ -180,18 +180,20 @@ fn create_default_config_dto() -> ConfigDto {
api: ConfigApiDto {
host: DEFAULT_SETUP_HOST.to_string(),
port: DEFAULT_PORT,
web_root: get_default_web_root_path().display().to_string(),
web_root: get_default_web_root_path_for_home(FsPath::new(home_path))
.display()
.to_string(),
},
working_dir: get_default_path(DEFAULT_WORKING_DIR).display().to_string(),
custom_stream_response_path: Some(DEFAULT_SETUP_CUSTOM_STREAM_RESPONSE_PATH.to_string()),
storage_dir: Some(DEFAULT_STORAGE_DIR.to_string()),
custom_stream_response_path: Some(DEFAULT_CUSTOM_STREAM_RESPONSE_PATH.to_string()),
web_ui: Some(web_ui),
..ConfigDto::default()
}
}
fn create_default_draft() -> AppConfigDto {
fn create_default_draft(home_path: &str) -> AppConfigDto {
AppConfigDto {
config: create_default_config_dto(),
config: create_default_config_dto(home_path),
sources: SourcesConfigDto::default(),
mappings: None,
templates: None,
@@ -204,7 +206,7 @@ fn create_default_draft() -> AppConfigDto {
}
async fn build_initial_draft(paths: &ConfigPaths) -> AppConfigDto {
let mut draft = create_default_draft();
let mut draft = create_default_draft(paths.home_path.as_str());
if file_exists(&paths.config_file_path) {
match read_config_file(paths.config_file_path.as_str(), true, false) {
@@ -251,7 +253,7 @@ fn generate_web_auth_secret() -> String {
hex_encode(&secret).to_lowercase()
}
fn ensure_setup_defaults(config: &mut ConfigDto) {
fn ensure_setup_defaults(config: &mut ConfigDto, home_path: &str) {
if config.api.host.trim().is_empty() {
config.api.host = DEFAULT_SETUP_HOST.to_string();
}
@@ -259,13 +261,15 @@ fn ensure_setup_defaults(config: &mut ConfigDto) {
config.api.port = DEFAULT_PORT;
}
if config.api.web_root.trim().is_empty() {
config.api.web_root = get_default_web_root_path().display().to_string();
config.api.web_root = get_default_web_root_path_for_home(FsPath::new(home_path))
.display()
.to_string();
}
if config.working_dir.trim().is_empty() {
config.working_dir = get_default_path(DEFAULT_WORKING_DIR).display().to_string();
if config.storage_dir.as_ref().is_none_or(|dir| dir.trim().is_empty()) {
config.storage_dir = Some(DEFAULT_STORAGE_DIR.to_string());
}
if config.custom_stream_response_path.as_ref().is_none_or(|path| path.trim().is_empty()) {
config.custom_stream_response_path = Some(DEFAULT_SETUP_CUSTOM_STREAM_RESPONSE_PATH.to_string());
config.custom_stream_response_path = Some(DEFAULT_CUSTOM_STREAM_RESPONSE_PATH.to_string());
}
if config.web_ui.is_none() {
@@ -288,7 +292,7 @@ fn ensure_setup_defaults(config: &mut ConfigDto) {
}
}
fn setup_bind_values(draft: &AppConfigDto) -> (String, u16, PathBuf) {
fn setup_bind_values(draft: &AppConfigDto, home_path: &str) -> (String, u16, PathBuf) {
let host = if draft.config.api.host.trim().is_empty() {
DEFAULT_SETUP_HOST.to_string()
} else {
@@ -296,18 +300,18 @@ fn setup_bind_values(draft: &AppConfigDto) -> (String, u16, PathBuf) {
};
let port = if draft.config.api.port == 0 { DEFAULT_PORT } else { draft.config.api.port };
let web_root = if draft.config.api.web_root.trim().is_empty() {
get_default_web_root_path()
get_default_web_root_path_for_home(FsPath::new(home_path))
} else {
PathBuf::from(&draft.config.api.web_root)
};
(host, port, web_root)
}
fn resolve_setup_web_dir(web_root: &FsPath) -> Option<PathBuf> {
fn resolve_setup_web_dir(web_root: &FsPath, home_path: &str) -> Option<PathBuf> {
if web_root.exists() && web_root.is_dir() {
return Some(web_root.to_path_buf());
}
let fallback = get_default_web_root_path();
let fallback = get_default_web_root_path_for_home(FsPath::new(home_path));
if fallback.exists() && fallback.is_dir() {
return Some(fallback);
}
@@ -322,126 +326,6 @@ fn api_proxy_or_default(draft: &AppConfigDto) -> ApiProxyConfigDto {
})
}
fn is_setup_redacted_value(value: &str) -> bool { value == SETUP_REDACTED_SECRET_VALUE }
fn redact_non_empty_secret(value: &mut String) {
if !value.trim().is_empty() {
*value = SETUP_REDACTED_SECRET_VALUE.to_string();
}
}
fn redact_optional_secret(value: &mut Option<String>) {
if value.as_ref().is_some_and(|entry| !entry.trim().is_empty()) {
*value = Some(SETUP_REDACTED_SECRET_VALUE.to_string());
}
}
fn redact_api_proxy_user_credentials(api_proxy: &mut ApiProxyConfigDto) {
for target in &mut api_proxy.user {
for user in &mut target.credentials {
redact_non_empty_secret(&mut user.password);
redact_optional_secret(&mut user.token);
}
}
}
fn redact_app_config_for_setup(mut app_config: AppConfigDto) -> AppConfigDto {
if let Some(web_ui) = app_config.config.web_ui.as_mut() {
if let Some(auth) = web_ui.auth.as_mut() {
redact_non_empty_secret(&mut auth.secret);
}
}
if let Some(api_proxy) = app_config.api_proxy.as_mut() {
redact_api_proxy_user_credentials(api_proxy);
}
app_config
}
fn redact_api_proxy_for_setup(mut api_proxy: ApiProxyConfigDto) -> ApiProxyConfigDto {
redact_api_proxy_user_credentials(&mut api_proxy);
api_proxy
}
fn restore_redacted_web_auth_secret(app_config: &mut AppConfigDto, draft: &AppConfigDto) {
let Some(web_ui) = app_config.config.web_ui.as_mut() else {
return;
};
let Some(auth) = web_ui.auth.as_mut() else {
return;
};
if !is_setup_redacted_value(&auth.secret) {
return;
}
let draft_secret = draft
.config
.web_ui
.as_ref()
.and_then(|draft_web_ui| draft_web_ui.auth.as_ref())
.map(|draft_auth| draft_auth.secret.trim().to_string())
.filter(|secret| !secret.is_empty());
if let Some(draft_secret) = draft_secret {
auth.secret = draft_secret;
}
}
fn restore_redacted_api_proxy_credentials(
api_proxy: &mut ApiProxyConfigDto,
draft_api_proxy: Option<&ApiProxyConfigDto>,
) {
let mut credentials_by_username: HashMap<String, (String, Option<String>)> = HashMap::new();
if let Some(draft) = draft_api_proxy {
for target in &draft.user {
for user in &target.credentials {
credentials_by_username.insert(user.username.clone(), (user.password.clone(), user.token.clone()));
}
}
}
for target in &mut api_proxy.user {
for user in &mut target.credentials {
let draft_credentials = credentials_by_username.get(&user.username);
if is_setup_redacted_value(&user.password) {
if let Some((password, _)) = draft_credentials {
user.password = password.clone();
}
}
if user.token.as_deref().is_some_and(is_setup_redacted_value) {
if let Some((_, token)) = draft_credentials {
user.token = token.clone();
}
}
}
}
}
fn restore_redacted_setup_values(app_config: &mut AppConfigDto, draft: &AppConfigDto) {
restore_redacted_web_auth_secret(app_config, draft);
if let Some(api_proxy) = app_config.api_proxy.as_mut() {
restore_redacted_api_proxy_credentials(api_proxy, draft.api_proxy.as_ref());
}
}
fn has_unresolved_redacted_setup_values(app_config: &AppConfigDto) -> bool {
if app_config
.config
.web_ui
.as_ref()
.and_then(|web_ui| web_ui.auth.as_ref())
.is_some_and(|auth| is_setup_redacted_value(&auth.secret))
{
return true;
}
app_config.api_proxy.as_ref().is_some_and(|api_proxy| {
api_proxy.user.iter().any(|target| {
target.credentials.iter().any(|user| {
is_setup_redacted_value(&user.password) || user.token.as_deref().is_some_and(is_setup_redacted_value)
})
})
})
}
fn setup_templates_from_request(app_config: &AppConfigDto) -> Option<Vec<PatternTemplate>> {
app_config
.templates
@@ -518,12 +402,12 @@ async fn setup_token_refresh() -> impl IntoResponse + Send {
async fn setup_get_config(State(state): State<Arc<SetupModeState>>) -> impl IntoResponse + Send {
let draft = state.draft.read().await.clone();
axum::Json(redact_app_config_for_setup(draft)).into_response()
axum::Json(draft).into_response()
}
async fn setup_get_api_proxy(State(state): State<Arc<SetupModeState>>) -> impl IntoResponse + Send {
let draft = state.draft.read().await.clone();
axum::Json(redact_api_proxy_for_setup(api_proxy_or_default(&draft))).into_response()
axum::Json(api_proxy_or_default(&draft)).into_response()
}
async fn setup_config_json(State(state): State<Arc<SetupModeState>>) -> impl IntoResponse + Send {
@@ -751,19 +635,7 @@ async fn setup_complete_inner(
state: Arc<SetupModeState>,
mut req: SetupCompleteRequestDto,
) -> axum::response::Response {
let draft_snapshot = state.draft.read().await.clone();
restore_redacted_setup_values(&mut req.app_config, &draft_snapshot);
if has_unresolved_redacted_setup_values(&req.app_config) {
return (
StatusCode::BAD_REQUEST,
axum::Json(json!({
"error": "One or more redacted secret fields could not be restored. Please re-enter secret values before saving setup."
})),
)
.into_response();
}
ensure_setup_defaults(&mut req.app_config.config);
ensure_setup_defaults(&mut req.app_config.config, state.home_path.as_str());
if let Err(err) = req.app_config.config.prepare(false) {
return (StatusCode::BAD_REQUEST, axum::Json(json!({ "error": err.to_string() }))).into_response();
@@ -983,13 +855,14 @@ fn create_compression_layer() -> tower_http::compression::CompressionLayer {
pub async fn start_setup_server(paths: &ConfigPaths, missing_files: &[String]) -> Result<(), TuliproxError> {
let draft = build_initial_draft(paths).await;
let (host, port, web_root) = setup_bind_values(&draft);
let web_dir = resolve_setup_web_dir(&web_root)
let (host, port, web_root) = setup_bind_values(&draft, paths.home_path.as_str());
let web_dir = resolve_setup_web_dir(&web_root, paths.home_path.as_str())
.ok_or_else(|| info_err!("Setup mode requires a web directory. Tried '{}'", web_root.display(),))?;
let (shutdown_tx, shutdown_rx) = oneshot::channel::<()>();
let state = Arc::new(SetupModeState {
draft: Arc::new(RwLock::new(draft)),
home_path: paths.home_path.clone(),
output_dir: PathBuf::from(&paths.config_path),
config_file_path: PathBuf::from(&paths.config_file_path),
source_file_path: PathBuf::from(&paths.sources_file_path),
@@ -1001,7 +874,7 @@ pub async fn start_setup_server(paths: &ConfigPaths, missing_files: &[String]) -
shutdown_tx: Mutex::new(Some(shutdown_tx)),
});
info!("Setup mode enabled. Missing required config files: {}", missing_files.join(", "));
info!("Setup mode enabled.");
info!("Setup output directory: {}", state.output_dir.display());
info!("Setup web root: {}", state.web_dir.display());
info!("Setup server running: http://{host}:{port}");
@@ -1048,25 +921,12 @@ pub async fn start_setup_server(paths: &ConfigPaths, missing_files: &[String]) -
#[cfg(test)]
mod tests {
use super::{
has_unresolved_redacted_setup_values, redact_app_config_for_setup, restore_redacted_setup_values,
SETUP_REDACTED_SECRET_VALUE,
};
use shared::model::{ApiProxyConfigDto, AppConfigDto, TargetUserDto, WebAuthConfigDto, WebUiConfigDto};
use super::api_proxy_or_default;
use shared::model::{ApiProxyConfigDto, AppConfigDto, TargetUserDto};
fn sample_app_config() -> AppConfigDto {
AppConfigDto {
config: shared::model::ConfigDto {
web_ui: Some(WebUiConfigDto {
auth: Some(WebAuthConfigDto {
issuer: "tuliprox".to_string(),
secret: "very-secret-value".to_string(),
..Default::default()
}),
..Default::default()
}),
..Default::default()
},
#[test]
fn setup_api_proxy_or_default_preserves_existing_credentials() {
let app_config = AppConfigDto {
api_proxy: Some(ApiProxyConfigDto {
user: vec![TargetUserDto {
target: "target-a".to_string(),
@@ -1080,49 +940,17 @@ mod tests {
..Default::default()
}),
..Default::default()
}
}
#[test]
fn setup_redaction_masks_web_auth_and_api_proxy_credentials() {
let redacted = redact_app_config_for_setup(sample_app_config());
let auth_secret =
redacted.config.web_ui.as_ref().and_then(|web_ui| web_ui.auth.as_ref()).map(|auth| auth.secret.as_str());
assert_eq!(auth_secret, Some(SETUP_REDACTED_SECRET_VALUE));
let creds = &redacted.api_proxy.expect("api_proxy should be present").user[0].credentials[0];
assert_eq!(creds.password, SETUP_REDACTED_SECRET_VALUE);
assert_eq!(creds.token.as_deref(), Some(SETUP_REDACTED_SECRET_VALUE));
}
#[test]
fn setup_restore_replaces_redacted_values_from_existing_draft() {
let draft = sample_app_config();
let mut submitted = redact_app_config_for_setup(draft.clone());
restore_redacted_setup_values(&mut submitted, &draft);
let auth_secret =
submitted.config.web_ui.as_ref().and_then(|web_ui| web_ui.auth.as_ref()).map(|auth| auth.secret.as_str());
assert_eq!(auth_secret, Some("very-secret-value"));
let creds = &submitted.api_proxy.as_ref().expect("api_proxy should be present").user[0].credentials[0];
};
let api_proxy = api_proxy_or_default(&app_config);
let creds = &api_proxy.user[0].credentials[0];
assert_eq!(creds.password, "alice-password");
assert_eq!(creds.token.as_deref(), Some("alice-token"));
assert!(!has_unresolved_redacted_setup_values(&submitted));
}
#[test]
fn setup_restore_keeps_unmatched_redacted_credentials_as_unresolved() {
let draft = sample_app_config();
let mut submitted = redact_app_config_for_setup(draft);
if let Some(api_proxy) = submitted.api_proxy.as_mut() {
api_proxy.user[0].credentials[0].username = "bob".to_string();
}
restore_redacted_setup_values(&mut submitted, &AppConfigDto::default());
assert!(has_unresolved_redacted_setup_values(&submitted));
fn setup_api_proxy_or_default_creates_server_when_missing() {
let api_proxy = api_proxy_or_default(&AppConfigDto::default());
assert_eq!(api_proxy.server.len(), 1);
assert!(!api_proxy.server[0].host.trim().is_empty());
}
}
+5 -5
View File
@@ -31,7 +31,7 @@ pub struct LibraryProcessor {
pub fn resolve_metadata_storage_path(
metadata_update_config: Option<&MetadataUpdateConfig>,
working_dir: &str,
storage_dir: &str,
) -> PathBuf {
let configured_path = metadata_update_config.map_or_else(
|| PathBuf::from(shared::utils::default_metadata_path()),
@@ -46,7 +46,7 @@ pub fn resolve_metadata_storage_path(
if configured_path.is_absolute() {
configured_path.clean()
} else {
PathBuf::from(working_dir).join(configured_path).clean()
PathBuf::from(storage_dir).join(configured_path).clean()
}
}
@@ -61,7 +61,7 @@ impl LibraryProcessor {
config
.library
.as_ref()
.map(|lib_cfg| Self::new(lib_cfg.clone(), config.metadata_update.as_ref(), client, &config.working_dir))
.map(|lib_cfg| Self::new(lib_cfg.clone(), config.metadata_update.as_ref(), client, &config.storage_dir))
}
// Creates a new Library processor with the given configuration
@@ -69,9 +69,9 @@ impl LibraryProcessor {
config: LibraryConfig,
metadata_update_config: Option<&MetadataUpdateConfig>,
client: reqwest::Client,
working_dir: &str,
storage_dir: &str,
) -> Self {
let storage_path = resolve_metadata_storage_path(metadata_update_config, working_dir);
let storage_path = resolve_metadata_storage_path(metadata_update_config, storage_dir);
let scanner = LibraryScanner::new(config.clone());
let storage = MetadataStorage::new(storage_path);
let resolver = MetadataResolver::from_config(Some(&config), metadata_update_config, client, Some(storage.clone()));
+1 -1
View File
@@ -473,7 +473,7 @@ impl TmdbSeason {
#[cfg(test)]
mod tests {
use super::{resolve_tmdb_image_url, TmdbMovieDetails, TMDB_IMAGE_BASE_URL};
use super::{resolve_tmdb_image_url, TMDB_IMAGE_BASE_URL};
#[test]
fn test_resolve_tmdb_url_with_absolute_http() {
+52 -20
View File
@@ -30,7 +30,7 @@ use clap::Parser;
use log::{error, info, warn};
use shared::{
model::ConfigPaths,
utils::{CONFIG_FILE, SOURCE_FILE},
utils::{CONFIG_FILE, CONFIG_PATH, DEFAULT_STORAGE_DIR, SOURCE_FILE},
};
use std::{
fs::File,
@@ -45,6 +45,10 @@ use std::{
#[command(version)]
#[command(about = "Extended playlist proxy", long_about = None)]
struct Args {
/// The home directory (base for config, storage, backup, downloads)
#[arg(short = 'H', long = "home")]
home: Option<String>,
/// The config directory
#[arg(short = 'p', long = "config-path")]
config_path: Option<String>,
@@ -217,21 +221,13 @@ fn run_startup_bplustree_migration(config_paths: &ConfigPaths) {
return;
}
let config_dto = match utils::read_config_file(config_paths.config_file_path.as_str(), true, false) {
Ok(cfg) => cfg,
Err(err) => {
warn!("Skipping B+Tree startup migration because config could not be read: {err}");
return;
}
};
let mut roots: Vec<PathBuf> = vec![PathBuf::from(&config_paths.config_path)];
let working_dir = PathBuf::from(config_dto.working_dir);
if !roots.iter().any(|root| root == &working_dir) {
roots.push(working_dir.clone());
let storage_dir = PathBuf::from(&config_paths.storage_path);
if !roots.iter().any(|root| root == &storage_dir) {
roots.push(storage_dir.clone());
}
match migrate_bplustree_databases_with_marker(&roots, &working_dir) {
match migrate_bplustree_databases_with_marker(&roots, &storage_dir) {
Ok(stats) => {
if stats.skipped_by_marker {
info!("B+Tree startup migration skipped (marker already present)");
@@ -260,7 +256,7 @@ fn print_info(app_config: &AppConfig) {
let paths = <Arc<ArcSwap<ConfigPaths>> as Access<ConfigPaths>>::load(&app_config.paths);
info!("Current time: {}", chrono::offset::Local::now().format("%Y-%m-%d %H:%M:%S"));
info!("Temp dir: {}", tempfile::env::temp_dir().display());
info!("Working dir: {:?}", &config.working_dir);
info!("Storage dir: {:?}", &config.storage_dir);
info!("Config dir: {:?}", &paths.config_path);
info!("Config file: {:?}", &paths.config_file_path);
info!("Source file: {:?}", &paths.sources_file_path);
@@ -281,7 +277,7 @@ fn print_info(app_config: &AppConfig) {
if let Some(cache) = config.reverse_proxy.as_ref().and_then(|r| r.cache.as_ref()) {
if cache.enabled {
info!("Cache dir: {}", cache.dir);
info!("Cache dir: {}", cache.directory);
}
}
if let Some(metadata_update) = config.metadata_update.as_ref() {
@@ -293,11 +289,31 @@ fn print_info(app_config: &AppConfig) {
}
fn get_file_paths(args: &Args) -> ConfigPaths {
let raw_path = args
.config_path
let resolve_storage_path = |home_path: &str, storage_dir: Option<&str>| {
let path = storage_dir.map(str::trim).filter(|value| !value.is_empty()).map(PathBuf::from).map_or_else(
|| utils::get_default_path_for_home(Path::new(home_path), DEFAULT_STORAGE_DIR),
|configured_path| {
if configured_path.is_relative() {
PathBuf::from(home_path).join(configured_path)
} else {
configured_path
}
},
);
utils::resolve_directory_path(path.to_string_lossy().as_ref())
};
let home_path = args
.home
.as_ref()
.filter(|p| !p.trim().is_empty())
.map_or_else(utils::get_default_config_path, |p| resolve_env_var(p));
.map_or_else(utils::get_home_path, |p| PathBuf::from(resolve_env_var(p)));
let home_path = utils::resolve_directory_path(home_path.to_string_lossy().as_ref());
let raw_path = args.config_path.as_ref().filter(|p| !p.trim().is_empty()).map_or_else(
|| utils::get_default_path_for_home(Path::new(&home_path), CONFIG_PATH).to_string_lossy().to_string(),
|p| resolve_env_var(p),
);
let config_path: String = utils::resolve_directory_path(&raw_path);
let config_file: String = resolve_env_var(
@@ -318,11 +334,27 @@ fn get_file_paths(args: &Args) -> ConfigPaths {
.as_ref()
.map_or_else(|| utils::get_default_sources_file_path(&config_path), ToString::to_string),
);
let mappings_file = args.mapping_file.as_ref().map(|p| resolve_env_var(p));
let template_file = args.template_file.as_ref().map(|p| resolve_env_var(p));
let mappings_file = args.mapping_file.as_ref().map(|p| {
let path = resolve_env_var(p);
utils::resolve_mapping_file_path(config_path.as_str(), Some(path.as_str()))
});
let template_file = args.template_file.as_ref().map(|p| {
let path = resolve_env_var(p);
utils::resolve_template_file_path(config_path.as_str(), Some(path.as_str()))
});
let storage_path = if Path::new(&config_file).exists() {
match utils::read_config_file(&config_file, true, false) {
Ok(cfg) => resolve_storage_path(&home_path, cfg.storage_dir.as_deref()),
Err(err) => exit!("Can't read config file {} while resolving storage path: {err}", config_file),
}
} else {
resolve_storage_path(&home_path, None)
};
ConfigPaths {
home_path,
config_path,
storage_path,
config_file_path: config_file,
sources_file_path: sources_file,
mapping_file_path: mappings_file, // need to be set after config read
+2
View File
@@ -309,7 +309,9 @@ mod tests {
api_proxy: Arc::new(ArcSwapOption::default()),
file_locks: Arc::new(FileLockManager::default()),
paths: Arc::new(ArcSwap::from_pointee(ConfigPaths {
home_path: String::new(),
config_path: String::new(),
storage_path: String::new(),
config_file_path: String::new(),
sources_file_path: String::new(),
mapping_file_path: None,
+9 -7
View File
@@ -329,9 +329,10 @@ impl AppConfig {
fn set_mapping_path(&self, mapping_path: Option<&str>) {
let paths_guard = self.paths.load();
let old_path = paths_guard.mapping_file_path.as_deref();
let new_path = mapping_path
.map(ToString::to_string)
.or_else(|| Some(utils::get_default_mappings_path(&paths_guard.config_path)));
let new_path = Some(utils::resolve_mapping_file_path(
paths_guard.config_path.as_str(),
mapping_path,
));
if old_path != new_path.as_deref() {
let mut new_paths = (**paths_guard).clone();
@@ -343,9 +344,10 @@ impl AppConfig {
fn set_template_path(&self, template_path: Option<&str>) {
let paths_guard = self.paths.load();
let old_path = paths_guard.template_file_path.as_deref();
let new_path = template_path
.map(ToString::to_string)
.or_else(|| Some(utils::get_default_templates_path(&paths_guard.config_path)));
let new_path = Some(utils::resolve_template_file_path(
paths_guard.config_path.as_str(),
template_path,
));
if old_path != new_path.as_deref() {
let mut new_paths = (**paths_guard).clone();
@@ -406,7 +408,7 @@ impl AppConfig {
}
let path = PathBuf::from(custom_stream_response_path);
let path = utils::make_path_absolute(&path, &config.working_dir);
let path = utils::make_path_absolute(&path, &config.storage_dir);
let paths = self.paths.load_full();
let mut new_paths = paths.as_ref().clone();
+108 -27
View File
@@ -8,21 +8,19 @@ use log::{error, info};
use path_clean::PathClean;
use shared::error::TuliproxError;
use shared::model::{ConfigDto, HdHomeRunDeviceOverview};
use shared::utils::{default_grace_period_millis, default_grace_period_timeout_secs, set_sanitize_sensitive_info, DEFAULT_WEB_DIR};
use shared::utils::{default_grace_period_millis, default_grace_period_timeout_secs, set_sanitize_sensitive_info, DEFAULT_BACKUP_DIR, DEFAULT_CACHE_DIR, DEFAULT_DOWNLOAD_DIR, DEFAULT_STORAGE_DIR, DEFAULT_STORAGE_TEMP_DIR, DEFAULT_USER_CONFIG_DIR};
use std::borrow::Cow;
use std::path::{Path, PathBuf};
use crate::utils::get_default_path;
const DEFAULT_BACKUP_DIR: &str = "backup";
use crate::utils::get_default_path_for_home;
fn create_directories(cfg: &Config, temp_path: &Path) {
// Collect the paths into a vector.
let paths_strings = [
Some(cfg.working_dir.clone()),
Some(cfg.storage_dir.clone()),
cfg.backup_dir.clone(),
cfg.user_config_dir.clone(),
cfg.video.as_ref().and_then(|v| v.download.as_ref()).map(|d| d.directory.clone()),
cfg.reverse_proxy.as_ref().and_then(|r| r.cache.as_ref().and_then(|c| if c.enabled { Some(c.dir.clone()) } else { None })),
cfg.reverse_proxy.as_ref().and_then(|r| r.cache.as_ref().and_then(|c| if c.enabled { Some(c.directory.clone()) } else { None })),
cfg.metadata_update.as_ref().map(|m| m.cache_path.clone()),
];
@@ -67,7 +65,7 @@ impl Default for GracePeriodOptions {
pub struct Config {
pub process_parallel: bool,
pub api: ConfigApi,
pub working_dir: String,
pub storage_dir: String,
pub default_user_agent: Option<String>,
pub backup_dir: Option<String>,
pub user_config_dir: Option<String>,
@@ -95,24 +93,58 @@ pub struct Config {
}
impl Config {
pub fn prepare(&mut self, config_path: &str) -> Result<(), TuliproxError> {
let work_dir = &self.working_dir;
self.working_dir = utils::resolve_directory_path(work_dir);
fn lexical_normalize(path: &Path) -> PathBuf {
path.components().collect::<PathBuf>()
}
self.prepare_directories();
self.prepare_api_web_root();
fn normalize_storage_path(raw_storage_dir: &str, storage_dir: &str, input: &str, default_dir: &str) -> Option<PathBuf> {
let trimmed_input = input.trim();
if trimmed_input.is_empty() {
return Some(PathBuf::from(storage_dir).join(default_dir));
}
let input_path = PathBuf::from(trimmed_input);
if input_path.is_absolute() {
return None;
}
let mut normalized_relative = Self::lexical_normalize(&input_path);
let trimmed_raw_storage = raw_storage_dir.trim();
if !trimmed_raw_storage.is_empty() {
let raw_path = PathBuf::from(trimmed_raw_storage);
if raw_path.is_relative() {
let normalized_raw = Self::lexical_normalize(&raw_path);
if normalized_raw.components().next().is_some() {
while normalized_relative.starts_with(&normalized_raw) {
match normalized_relative.strip_prefix(&normalized_raw) {
Ok(stripped) => {
normalized_relative = Self::lexical_normalize(stripped);
}
Err(_) => break,
}
}
}
}
}
Some(PathBuf::from(storage_dir).join(normalized_relative))
}
pub fn prepare(&mut self, config_path: &str, home_path: &str) -> Result<(), TuliproxError> {
self.prepare_directories(home_path);
if let Some(ref mut webui) = &mut self.web_ui {
webui.prepare(config_path)?;
}
if let Some(library) = self.library.as_mut() {
library.prepare(&self.working_dir)?;
library.prepare(&self.storage_dir)?;
}
if let Some(metadata_update) = self.metadata_update.as_mut() {
let meta_path = PathBuf::from(&metadata_update.cache_path);
let meta_path = if meta_path.is_relative() {
PathBuf::from(&self.working_dir).join(meta_path)
PathBuf::from(&self.storage_dir).join(meta_path)
} else {
meta_path
};
@@ -125,38 +157,87 @@ impl Config {
if let Some(video) = self.video.as_mut() {
video.prepare();
if let Some(download) = video.download.as_mut() {
let download_path = PathBuf::from(&download.directory);
if download.directory.trim().is_empty() {
download.directory = get_default_path_for_home(Path::new(home_path), DEFAULT_DOWNLOAD_DIR)
.clean()
.to_string_lossy()
.to_string();
} else if download_path.is_relative() {
download.directory = PathBuf::from(home_path)
.join(download_path)
.clean()
.to_string_lossy()
.to_string();
}
}
}
Ok(())
}
fn prepare_directories(&mut self) {
fn set_directory(path: &mut Option<String>, default_subdir: &str, working_dir: &str) {
*path = Some(match path.as_ref() {
Some(existing) => existing.to_owned(),
None => PathBuf::from(working_dir).join(default_subdir).clean().to_string_lossy().to_string(),
});
fn prepare_reverse_proxy_cache_dir(&mut self, raw_storage_dir: &str) {
let Some(reverse_proxy) = self.reverse_proxy.as_mut() else {
return;
};
let Some(cache) = reverse_proxy.cache.as_mut() else {
return;
};
if !cache.enabled {
return;
}
set_directory(&mut self.backup_dir, DEFAULT_BACKUP_DIR, &self.working_dir);
set_directory(&mut self.user_config_dir, "user_config", &self.working_dir);
let normalized = Self::normalize_storage_path(raw_storage_dir, &self.storage_dir, &cache.directory, DEFAULT_CACHE_DIR)
.unwrap_or_else(|| PathBuf::from(cache.directory.trim()));
cache.directory = normalized.clean().to_string_lossy().to_string();
}
fn prepare_directories(&mut self, home_path: &str) {
let raw_storage_dir = self.storage_dir.trim().to_string();
let storage_dir_path = if raw_storage_dir.is_empty() {
get_default_path_for_home(Path::new(home_path), DEFAULT_STORAGE_DIR)
} else {
let configured_storage_path = PathBuf::from(&raw_storage_dir);
if configured_storage_path.is_relative() {
PathBuf::from(home_path).join(configured_storage_path)
} else {
configured_storage_path
}
};
self.storage_dir = utils::resolve_directory_path(storage_dir_path.to_string_lossy().as_ref());
self.prepare_reverse_proxy_cache_dir(&raw_storage_dir);
let storage_dir = self.storage_dir.clone();
let normalize_optional_path = |value: Option<&str>, default_dir: &str| -> String {
let configured = value.unwrap_or_default();
let normalized = Self::normalize_storage_path(&raw_storage_dir, &storage_dir, configured, default_dir)
.unwrap_or_else(|| PathBuf::from(configured.trim()));
normalized.clean().to_string_lossy().to_string()
};
self.backup_dir = Some(normalize_optional_path(self.backup_dir.as_deref(), DEFAULT_BACKUP_DIR));
self.user_config_dir = Some(normalize_optional_path(self.user_config_dir.as_deref(), DEFAULT_USER_CONFIG_DIR));
self.prepare_api_web_root(home_path);
}
pub fn get_backup_dir(&self) -> Cow<'_, str> {
self.backup_dir.as_ref().map_or_else(|| Cow::Borrowed(DEFAULT_BACKUP_DIR), |v| Cow::Borrowed(v))
}
fn prepare_api_web_root(&mut self) {
fn prepare_api_web_root(&mut self, home_path: &str) {
if self.api.web_root.is_empty() {
self.api.web_root = get_default_path(DEFAULT_WEB_DIR).display().to_string();
self.api.web_root = utils::get_default_web_root_path_for_home(Path::new(home_path))
.display()
.to_string();
} else {
self.api.web_root = utils::make_absolute_path(&self.api.web_root, &self.working_dir);
self.api.web_root = utils::make_absolute_path(&self.api.web_root, &self.storage_dir);
}
}
pub fn update_runtime(&self) {
set_sanitize_sensitive_info(self.log.as_ref().is_none_or(|l| l.sanitize_sensitive_info));
let temp_path = PathBuf::from(&self.working_dir).join("tmp");
let temp_path = PathBuf::from(&self.storage_dir).join(DEFAULT_STORAGE_TEMP_DIR);
create_directories(self, &temp_path);
let _ = tempfile::env::override_temp_dir(&temp_path);
}
@@ -200,7 +281,7 @@ impl From<&ConfigDto> for Config {
process_parallel: dto.process_parallel,
disk_based_processing: dto.disk_based_processing,
api: ConfigApi::from(&dto.api),
working_dir: dto.working_dir.clone(),
storage_dir: dto.storage_dir.clone().unwrap_or_default(),
default_user_agent: dto.default_user_agent.clone(),
backup_dir: dto.backup_dir.clone(),
user_config_dir: dto.user_config_dir.clone(),
+3 -3
View File
@@ -5,7 +5,7 @@ use crate::model::macros;
#[derive(Debug, Clone)]
pub struct CacheConfig {
pub enabled: bool,
pub dir: String,
pub directory: String,
pub size: usize,
pub size_str: Option<String>,
}
@@ -16,7 +16,7 @@ impl From<&CacheConfigDto> for CacheConfig {
Self {
enabled: dto.enabled,
// Dto prepare should have set the right path
dir: dto.dir.as_ref().map_or_else(Default::default, std::string::ToString::to_string),
directory: dto.directory.as_ref().map_or_else(Default::default, std::string::ToString::to_string),
size_str: dto.size.clone(),
size: get_size(dto)
}
@@ -28,7 +28,7 @@ impl From<&CacheConfig> for CacheConfigDto {
Self {
enabled: instance.enabled,
// Dto prepare should have set the right path
dir: Some(instance.dir.clone()),
directory: Some(instance.directory.clone()),
size: instance.size_str.clone(),
}
}
+38
View File
@@ -242,6 +242,14 @@ pub struct ConfigInput {
}
impl ConfigInput {
#[inline]
pub fn get_download_input_type(&self) -> InputType {
self.staged
.as_ref()
.filter(|staged| staged.enabled)
.map_or(self.input_type, |staged| staged.input_type)
}
#[inline]
pub fn has_flag(&self, flag: ConfigInputFlags) -> bool {
self.has_flag_or(flag, false)
@@ -734,4 +742,34 @@ mod tests {
let err = input.prepare(&[]).unwrap_err();
assert!(err.to_string().contains("Malformed provider URL"));
}
#[test]
fn test_get_download_input_type_uses_staged_when_enabled() {
let input = ConfigInput {
input_type: InputType::Xtream,
staged: Some(StagedInput {
enabled: true,
input_type: InputType::M3u,
..Default::default()
}),
..Default::default()
};
assert_eq!(input.get_download_input_type(), InputType::M3u);
}
#[test]
fn test_get_download_input_type_uses_primary_when_staged_disabled() {
let input = ConfigInput {
input_type: InputType::Xtream,
staged: Some(StagedInput {
enabled: false,
input_type: InputType::M3u,
..Default::default()
}),
..Default::default()
};
assert_eq!(input.get_download_input_type(), InputType::Xtream);
}
}
+34 -20
View File
@@ -1,6 +1,7 @@
use crate::model::macros;
use shared::error::{info_err_res, TuliproxError};
use shared::model::{ConfigDto, LibraryConfigDto, LibraryContentType, LibraryMetadataFormat};
use shared::utils::DEFAULT_STORAGE_DIR;
use shared::utils::Internable;
use std::path::PathBuf;
use std::sync::Arc;
@@ -43,14 +44,14 @@ pub struct LibraryConfig {
}
impl LibraryConfig {
fn canonicalize_scan_directory_path(path: &str, working_dir: &str) -> Result<String, TuliproxError> {
fn canonicalize_scan_directory_path(path: &str, storage_dir: &str) -> Result<String, TuliproxError> {
let path = path.trim();
if path.is_empty() {
return info_err_res!("Library scan directory path cannot be empty");
}
let scan_path = PathBuf::from(path);
let scan_path = if scan_path.is_relative() { PathBuf::from(working_dir).join(scan_path) } else { scan_path };
let scan_path = if scan_path.is_relative() { PathBuf::from(storage_dir).join(scan_path) } else { scan_path };
match scan_path.canonicalize() {
Ok(path_buf) => Ok(path_buf.to_string_lossy().to_string()),
@@ -58,16 +59,16 @@ impl LibraryConfig {
}
}
fn canonicalize_scan_directories(&mut self, working_dir: &str) -> Result<(), TuliproxError> {
fn canonicalize_scan_directories(&mut self, storage_dir: &str) -> Result<(), TuliproxError> {
for dir in &mut self.scan_directories {
dir.path = Self::canonicalize_scan_directory_path(&dir.path, working_dir)?;
dir.path = Self::canonicalize_scan_directory_path(&dir.path, storage_dir)?;
}
Ok(())
}
pub fn prepare(&mut self, working_dir: &str) -> Result<(), TuliproxError> {
pub fn prepare(&mut self, storage_dir: &str) -> Result<(), TuliproxError> {
if self.enabled {
self.canonicalize_scan_directories(working_dir)?;
self.canonicalize_scan_directories(storage_dir)?;
}
Ok(())
}
@@ -119,11 +120,18 @@ pub fn validate_library_paths_from_dto(cfg: &ConfigDto) -> Result<(), TuliproxEr
};
let mut library_cfg = LibraryConfig::from(library_dto);
let storage_dir = cfg
.storage_dir
.as_deref()
.map(str::trim)
.filter(|dir| !dir.is_empty())
.unwrap_or(DEFAULT_STORAGE_DIR);
// Always validate configured scan directories, even when library is disabled.
// This prevents persisting invalid paths that would later break startup when enabled.
library_cfg.canonicalize_scan_directories(&cfg.working_dir)?;
if library_cfg.enabled {
library_cfg.prepare(&cfg.working_dir)?;
library_cfg.prepare(storage_dir)?;
} else {
library_cfg.canonicalize_scan_directories(storage_dir)?;
}
Ok(())
}
@@ -143,10 +151,12 @@ mod tests {
#[test]
fn validate_library_paths_rejects_missing_scan_directory() {
let cfg = ConfigDto {
working_dir: std::env::current_dir()
.expect("current_dir should be available")
.to_string_lossy()
.to_string(),
storage_dir: Some(
std::env::current_dir()
.expect("current_dir should be available")
.to_string_lossy()
.to_string(),
),
library: Some(LibraryConfigDto {
enabled: true,
scan_directories: vec![LibraryScanDirectoryDto { path: unique_missing_path(), ..Default::default() }],
@@ -162,10 +172,12 @@ mod tests {
#[test]
fn validate_library_paths_rejects_missing_scan_directory_even_when_disabled() {
let cfg = ConfigDto {
working_dir: std::env::current_dir()
.expect("current_dir should be available")
.to_string_lossy()
.to_string(),
storage_dir: Some(
std::env::current_dir()
.expect("current_dir should be available")
.to_string_lossy()
.to_string(),
),
library: Some(LibraryConfigDto {
enabled: false,
scan_directories: vec![LibraryScanDirectoryDto { path: unique_missing_path(), ..Default::default() }],
@@ -182,10 +194,12 @@ mod tests {
#[test]
fn validate_library_paths_accepts_existing_scan_directory() {
let cfg = ConfigDto {
working_dir: std::env::current_dir()
.expect("current_dir should be available")
.to_string_lossy()
.to_string(),
storage_dir: Some(
std::env::current_dir()
.expect("current_dir should be available")
.to_string_lossy()
.to_string(),
),
library: Some(LibraryConfigDto {
enabled: true,
scan_directories: vec![LibraryScanDirectoryDto {
-7
View File
@@ -182,13 +182,6 @@ impl From<&ConfigProviderDto> for ConfigProvider {
fn from(dto: &ConfigProviderDto) -> Self {
let dns_cfg = dto.dns.as_ref().map(ProviderDnsConfig::from);
let dns_cache = Arc::new(ProviderDnsCache::default());
if let Some(dns_dto) = dto.dns.as_ref() {
if let Some(resolved) = dns_dto.resolved.as_ref() {
for (host, ips) in resolved {
dns_cache.store_resolved(host, ips.clone());
}
}
}
Self {
name: dto.name.clone(),
urls: dto.urls.clone(),
+2 -1
View File
@@ -1,6 +1,7 @@
use crate::model::macros;
use regex::Regex;
use shared::model::{VideoConfigDto, VideoDownloadConfigDto};
use shared::utils::DEFAULT_DOWNLOAD_DIR;
use std::collections::HashMap;
use std::sync::Arc;
@@ -17,7 +18,7 @@ impl From<&VideoDownloadConfigDto> for VideoDownloadConfig {
fn from(dto: &VideoDownloadConfigDto) -> Self {
Self {
headers: dto.headers.clone(),
directory: dto.directory.as_ref().map_or_else(|| "downloads".to_string(), ToString::to_string),
directory: dto.directory.as_ref().map_or_else(|| DEFAULT_DOWNLOAD_DIR.to_string(), ToString::to_string),
organize_into_directories: dto.organize_into_directories,
episode_pattern: dto.episode_pattern.as_ref().and_then(|s| shared::model::REGEX_CACHE.get_or_compile(s)
.map_err(|e| log::warn!("Invalid episode_pattern regex '{s}': {e}"))
+95
View File
@@ -0,0 +1,95 @@
use crate::{
api::model::AppState,
model::InputSource,
repository::get_geoip_path,
utils::{request::download_text_content, GeoIp},
};
use log::{error, info};
use shared::{
model::{default_geoip_url, InputFetchMethod},
utils::Internable,
};
use std::{collections::HashMap, io::Cursor, sync::Arc};
#[derive(Debug)]
pub(crate) enum GeoIpUpdateError {
Disabled,
DownloadFailed(String),
ProcessFailed(String),
UnknownProcessing,
}
impl std::fmt::Display for GeoIpUpdateError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Self::Disabled => write!(f, "GeoIp update is disabled"),
Self::DownloadFailed(err) => write!(f, "Failed to download geoip db: {err}"),
Self::ProcessFailed(err) => write!(f, "Failed to process geoip db: {err}"),
Self::UnknownProcessing => write!(f, "Unknown GeoIp processing error"),
}
}
}
impl std::error::Error for GeoIpUpdateError {}
pub(crate) async fn update_geoip_db(app_state: &Arc<AppState>) -> Result<(), GeoIpUpdateError> {
let config = app_state.app_config.config.load();
if let Some(geoip) = config.reverse_proxy.as_ref().and_then(|r| r.geoip.as_ref()) {
if geoip.enabled {
let geoip_db_path = &*get_geoip_path(&config.storage_dir);
let _file_lock = app_state.app_config.file_locks.write_lock(geoip_db_path).await;
let url = if geoip.url.trim().is_empty() { default_geoip_url() } else { geoip.url.clone() };
let input_source = InputSource {
name: "GeoIP".intern(),
url,
provider: None,
username: None,
password: None,
method: InputFetchMethod::GET,
headers: HashMap::default(),
};
return match download_text_content(
&app_state.app_config,
&app_state.http_client.load(),
&input_source,
None,
None,
false,
)
.await
{
Ok((content, _)) => {
let reader = Cursor::new(content);
let mut geoip = GeoIp::new();
let result = {
match geoip.import_ipv4_from_csv(reader, geoip_db_path) {
Ok(size) => (Some(size), None),
Err(err) => (None, Some(err)),
}
};
return match result {
(Some(_), None) => {
info!("GeoIp db updated");
app_state.geoip.store(Some(Arc::new(geoip)));
Ok(())
}
(None, Some(err)) => {
let error = GeoIpUpdateError::ProcessFailed(err.to_string());
error!("{error}");
Err(error)
}
_ => Err(GeoIpUpdateError::UnknownProcessing),
};
}
Err(err) => {
let error = GeoIpUpdateError::DownloadFailed(err.to_string());
error!("{error}");
Err(error)
}
};
}
}
Err(GeoIpUpdateError::Disabled)
}
+3 -3
View File
@@ -27,9 +27,9 @@ pub struct InputStatus {
pub clusters: HashMap<String, ClusterStatus>,
}
pub async fn resolve_input_storage_path(working_dir: &str, input_name: &str) -> PathBuf {
if let Ok(path) = get_input_storage_path(input_name, working_dir).await { path } else {
build_input_storage_path(input_name, working_dir)
pub async fn resolve_input_storage_path(storage_dir: &str, input_name: &str) -> PathBuf {
if let Ok(path) = get_input_storage_path(input_name, storage_dir).await { path } else {
build_input_storage_path(input_name, storage_dir)
}
}
+1 -1
View File
@@ -1,5 +1,5 @@
pub(crate) mod geoip;
pub(crate) mod input_cache;
pub(crate) mod playlist_watch;
pub(crate) mod parser;
pub(crate) mod processor;
+1 -3
View File
@@ -123,14 +123,12 @@ impl TVGuide {
let src = "src".intern();
for child in children {
match child.name.as_ref() {
EPG_TAG_DISPLAY_NAME => {
if smart_match {
EPG_TAG_DISPLAY_NAME if smart_match => {
if let Some(name) = &child.value {
tag.normalized_epg_ids
.get_or_insert_with(Vec::new)
.push(normalize_channel_name(name, &id_cache.smart_match_config).intern());
}
}
}
EPG_TAG_ICON => {
if let Some(src) = child.get_attribute_value(&src) {
+1 -4
View File
@@ -174,9 +174,7 @@ pub async fn parse_xtream(input: &ConfigInput,
input.has_flag(ConfigInputFlags::XtreamLiveStreamWithoutExtension),
);
// Re-implement the loop to add ordinal
let mut ord_counter: u32 = 1;
for stream in xtream_streams {
for (ord_counter, stream) in (1_u32..).zip(xtream_streams) {
let group = group_map.get_mut(&stream.get_category_id()).unwrap_or(&mut unknown_grp);
let category_name = &group.category_name;
let stream_url = create_xtream_url(xtream_cluster, url, username, password, &stream, live_stream_use_prefix, live_stream_without_extension);
@@ -200,7 +198,6 @@ pub async fn parse_xtream(input: &ConfigInput,
},
};
item.header.source_ordinal = ord_counter;
ord_counter += 1;
group.add(item);
}
+1 -1
View File
@@ -21,7 +21,7 @@ pub async fn process_group_watch(app_config: &Arc<AppConfig>, client: &reqwest::
let watch_filename = format!("{}/{}.bin", utils::sanitize_filename(target_name), utils::sanitize_filename(&pl.title));
let cfg = app_config.config.load();
match utils::get_file_path(&cfg.working_dir, Some(std::path::PathBuf::from(&watch_filename))) {
match utils::get_file_path(&cfg.storage_dir, Some(std::path::PathBuf::from(&watch_filename))) {
Some(path) => {
let save_path = path.as_path();
let mut changed = false;
+66 -23
View File
@@ -50,13 +50,15 @@ use std::{
collections::{HashMap, HashSet},
path::PathBuf,
sync::{Arc, Weak},
time::Instant,
time::{Duration, Instant},
};
use tokio::{
sync::{Mutex, OwnedRwLockWriteGuard, RwLock},
task::JoinSet,
};
const PLAYLIST_UPDATE_MAX_DURATION_SECS: u64 = 3600;
fn is_valid(pli: &PlaylistItem, filter: &Filter, match_as_ascii: bool) -> bool {
let provider = ValueProvider { pli, match_as_ascii };
filter.filter(&provider)
@@ -324,10 +326,11 @@ async fn playlist_download_from_input(
input: &ConfigInput,
) -> PlaylistDownloadResult {
let config = &*app_config.config.load();
let working_dir = &config.working_dir;
let storage_dir = &config.storage_dir;
let download_input_type = input.get_download_input_type();
// Check Status
let storage_path = input_cache::resolve_input_storage_path(working_dir, &input.name).await;
let storage_path = input_cache::resolve_input_storage_path(storage_dir, &input.name).await;
let mut status = input_cache::load_input_status(&storage_path);
let cache_duration = input.cache_duration_seconds;
@@ -336,7 +339,7 @@ async fn playlist_download_from_input(
let _ = std::fs::create_dir_all(&storage_path);
}
let (clusters_to_download, fully_cached) = match input.input_type {
let (clusters_to_download, fully_cached) = match download_input_type {
InputType::Xtream => {
let mut to_download = vec![];
for c in XTREAM_CLUSTER {
@@ -364,7 +367,7 @@ async fn playlist_download_from_input(
return PlaylistDownloadResult::new(vec![], vec![], true, false);
}
let (playlist, errors, persisted) = match input.input_type {
let (playlist, errors, persisted) = match download_input_type {
InputType::M3u => {
let (p, e) = m3u::download_m3u_playlist(app_config, client, config, input).await;
(p, e, false)
@@ -381,7 +384,7 @@ async fn playlist_download_from_input(
// Update Status
if errors.is_empty() {
if let InputType::Xtream = input.input_type {
if let InputType::Xtream = download_input_type {
if let Some(clusters) = clusters_to_download {
for c in clusters {
input_cache::update_cluster_status(&mut status, &c.to_string(), ClusterState::Ok);
@@ -401,7 +404,7 @@ async fn playlist_download_from_input(
// We could mark specific clusters as failed if we knew which one failed.
// For simplicity, if error, we don't update the timestamp (so it stays expired/invalid).
// Or we mark as Failed.
if let InputType::Xtream = input.input_type {
if let InputType::Xtream = download_input_type {
if let Some(clusters) = clusters_to_download {
for c in clusters {
// Optimistic: Only mark failed if we are sure?
@@ -437,6 +440,7 @@ async fn process_source(
continue;
};
if is_input_enabled(input, &ctx.user_targets) {
let effective_input_type = input.get_download_input_type();
source_downloaded = true;
log_memory_snapshot(format!("source[{source_idx}] input '{}' before_download", input.name).as_str());
@@ -459,7 +463,7 @@ async fn process_source(
};
log_memory_snapshot(format!("source[{source_idx}] input '{}' after_download", input.name).as_str());
let tvguide = if input.input_type == InputType::Library {
let tvguide = if effective_input_type == InputType::Library {
None
} else {
download_input_epg(ctx, input, &mut error_list).await
@@ -483,7 +487,7 @@ async fn process_source(
let elapsed = start_time.elapsed().as_secs();
input_stats.insert(
input_name.clone(),
create_input_stat(group_count, channel_count, errors.len(), input.input_type, input_name, elapsed),
create_input_stat(group_count, channel_count, errors.len(), effective_input_type, input_name, elapsed),
);
}
}
@@ -547,8 +551,8 @@ async fn download_input_epg(
) -> Option<TVGuide> {
// Download epg for input
let (tvguide, mut tvguide_errors) = if error_list.is_empty() {
let working_dir = &ctx.config.config.load().working_dir;
epg::get_xmltv(ctx, input, None, working_dir).await
let storage_dir = &ctx.config.config.load().storage_dir;
epg::get_xmltv(ctx, input, None, storage_dir).await
} else {
(None, vec![])
};
@@ -1018,10 +1022,10 @@ async fn playlist_probe(ctx: &PlaylistProcessingContext, target: &ConfigTarget,
}
let input_name = fpl.input.name.clone();
let input_type = fpl.input.input_type;
let xtream_probe_handled = input_type.is_xtream() && target.get_xtream_output().is_some();
let effective_input_type = fpl.input.get_download_input_type();
let xtream_probe_handled = effective_input_type.is_xtream() && target.get_xtream_output().is_some();
let live_probe_settings = if probe_live_enabled {
get_live_probe_interval_settings(target, input_type, Some(opts)).map(|(delay, interval_secs)| {
get_live_probe_interval_settings(target, effective_input_type, Some(opts)).map(|(delay, interval_secs)| {
let interval_signed = i64::try_from(interval_secs).unwrap_or(i64::MAX);
let cutoff_ts = chrono::Utc::now().timestamp().saturating_sub(interval_signed);
(delay, interval_secs, cutoff_ts)
@@ -1050,6 +1054,16 @@ async fn playlist_probe(ctx: &PlaylistProcessingContext, target: &ConfigTarget,
if needs_live_probe(&item, cutoff_ts) {
if let Some(provider_id) = provider_id_from_item(&item) {
if queued_live_keys.insert(provider_id.clone()) {
if log_enabled!(Level::Debug) {
let last_probed = match item.header.additional_properties.as_ref() {
Some(StreamProperties::Live(props)) => props.last_probed_timestamp,
_ => None,
};
debug!(
"[Task] Creating ProbeLive task for input {}: id={}, last_probed_ts={:?}, cutoff_ts={}, interval={}s, title=\"{}\"",
input_name, provider_id, last_probed, cutoff_ts, interval_secs, item.header.title
);
}
let task = UpdateTask::ProbeLive {
id: provider_id,
reason: ResolveReasonSet::from_variants(&[ResolveReason::Probe]),
@@ -1092,7 +1106,7 @@ async fn playlist_probe(ctx: &PlaylistProcessingContext, target: &ConfigTarget,
}
// For M3U, ID is a provider id; for Library, ID is UUID.
let unique_id = if input_type == InputType::Library {
let unique_id = if effective_input_type == InputType::Library {
item.header.uuid.to_valid_uuid()
} else {
item.header.id.to_string()
@@ -1104,6 +1118,10 @@ async fn playlist_probe(ctx: &PlaylistProcessingContext, target: &ConfigTarget,
continue;
}
debug!(
"[Task] Creating ProbeStream task for input {}: scope={}, unique_id={}, item_type={:?}, title=\"{}\"",
input_name, probe_scope, unique_id, item.header.item_type, item.header.title
);
let task = UpdateTask::ProbeStream {
probe_scope,
unique_id,
@@ -1204,7 +1222,7 @@ async fn process_watch(
}
}
#[allow(clippy::too_many_arguments)]
#[allow(clippy::too_many_arguments, clippy::too_many_lines)]
pub async fn exec_processing(
client: &reqwest::Client,
app_config: Arc<AppConfig>,
@@ -1219,6 +1237,7 @@ pub async fn exec_processing(
pre_processed_inputs: Option<HashSet<Arc<str>>>,
acquired_permit: Option<crate::api::model::UpdateGuardPermit>,
) {
let max_update_duration = Duration::from_secs(PLAYLIST_UPDATE_MAX_DURATION_SECS);
let playlist_guard = if let Some(permit) = acquired_permit {
Some(permit)
} else if let Some(guard) = &update_guard {
@@ -1237,7 +1256,15 @@ pub async fn exec_processing(
if playlist_guard.is_some() {
if let Some(state) = app_state.as_ref() {
sync_panel_api_exp_dates(state).await;
if tokio::time::timeout(max_update_duration, sync_panel_api_exp_dates(state)).await.is_err() {
error!(
"Playlist update bootstrap timed out after {PLAYLIST_UPDATE_MAX_DURATION_SECS} secs while holding playlist lock",
);
if let Some(events) = event_manager.as_deref() {
events.send_event(EventMessage::PlaylistUpdate(shared::model::PlaylistUpdateState::Failure));
}
return;
}
}
}
@@ -1266,13 +1293,29 @@ pub async fn exec_processing(
};
let start_time = Instant::now();
let process_result = std::panic::AssertUnwindSafe(process_sources(&ctx)).catch_unwind().await;
let Ok((stats, errors)) = process_result else {
error!("Playlist processing panicked");
if let Some(events) = event_manager.as_deref() {
events.send_event(EventMessage::PlaylistUpdate(shared::model::PlaylistUpdateState::Failure));
let process_result = tokio::time::timeout(
max_update_duration,
std::panic::AssertUnwindSafe(process_sources(&ctx)).catch_unwind(),
)
.await;
let (stats, errors) = match process_result {
Ok(Ok((stats, errors))) => (stats, errors),
Ok(Err(_)) => {
error!("Playlist processing panicked");
if let Some(events) = event_manager.as_deref() {
events.send_event(EventMessage::PlaylistUpdate(shared::model::PlaylistUpdateState::Failure));
}
return;
}
Err(_) => {
error!(
"Playlist processing timed out after {PLAYLIST_UPDATE_MAX_DURATION_SECS} secs while holding playlist lock",
);
if let Some(events) = event_manager.as_deref() {
events.send_event(EventMessage::PlaylistUpdate(shared::model::PlaylistUpdateState::Failure));
}
return;
}
return;
};
log_memory_snapshot("exec_processing after_process_sources");
@@ -41,7 +41,7 @@ pub async fn update_generic_stream_metadata(
active_provider: &Arc<ActiveProviderManager>,
active_handle: Option<&crate::api::model::ProviderHandle>,
) -> Result<GenericProbeOutcome, TuliproxError> {
let working_dir = &app_config.config.load().working_dir;
let storage_dir = &app_config.config.load().storage_dir;
// Check if probing is enabled globally
let ffprobe_enabled = app_config.is_ffprobe_enabled().await;
@@ -50,7 +50,7 @@ pub async fn update_generic_stream_metadata(
}
// Determine storage file path based on input type
let storage_path = get_input_storage_path(&input.name, working_dir).await
let storage_path = get_input_storage_path(&input.name, storage_dir).await
.map_err(|e| shared::error::info_err!("Storage path error: {e}"))?;
let (db_path, storage_kind) = match input.input_type {
+2 -2
View File
@@ -21,8 +21,8 @@ pub async fn update_live_stream_metadata(
_active_handle: Option<&ProviderHandle>,
_active_provider: &Arc<ActiveProviderManager>,
) -> Result<Option<LiveStreamProperties>, TuliproxError> {
let working_dir = &app_config.config.load().working_dir;
let storage_path = get_input_storage_path(&input.name, working_dir).await
let storage_dir = &app_config.config.load().storage_dir;
let storage_path = get_input_storage_path(&input.name, storage_dir).await
.map_err(|e| shared::error::info_err!("Storage path error: {e}"))?;
// Try to load existing info first to preserve data
@@ -207,6 +207,19 @@ fn queue_background_series_info(
let reasons = check_resolve_reasons(resolve_options, resolve_tmdb_enabled, pli);
if !reasons.is_empty() {
if log_enabled!(Level::Debug) {
let has_details = pli.has_details();
let (has_tmdb, has_date) = match pli.header.additional_properties.as_ref() {
Some(StreamProperties::Series(props)) => {
(props.tmdb.is_some(), props.release_date.is_some())
}
_ => (false, false),
};
debug!(
"[Task] Creating ResolveSeries task for input {}: id={}, reasons={}, has_details={}, has_tmdb={}, has_date={}, title=\"{}\"",
input_name_arc, provider_id, reasons, has_details, has_tmdb, has_date, pli.header.title
);
}
let task = UpdateTask::ResolveSeries {
id: provider_id,
reason: reasons,
@@ -233,8 +246,8 @@ async fn process_immediate_series_info(
resolve_tmdb_enabled: bool,
) -> Vec<PlaylistGroup> {
let input = fpl.input;
let working_dir = &ctx.config.config.load().working_dir;
let storage_path = match get_input_storage_path(&input.name, working_dir).await {
let storage_dir = &ctx.config.config.load().storage_dir;
let storage_path = match get_input_storage_path(&input.name, storage_dir).await {
Ok(path) => path,
Err(err) => {
error!("Can't resolve series, input storage directory for input '{}' failed: {err}", input.name);
@@ -624,8 +637,8 @@ pub async fn update_series_metadata(
db_query: Option<Arc<Mutex<BPlusTreeQuery<u32, XtreamPlaylistItem>>>>,
tmdb_and_date_present_out: Option<&AtomicBool>,
) -> Result<Option<SeriesStreamProperties>, TuliproxError> {
let working_dir = &app_config.config.load().working_dir;
let storage_path = get_input_storage_path(&input.name, working_dir)
let storage_dir = &app_config.config.load().storage_dir;
let storage_path = get_input_storage_path(&input.name, storage_dir)
.await
.map_err(|e| shared::error::info_err!("Storage path error: {e}"))?;
+25 -13
View File
@@ -5,6 +5,7 @@ use crate::model::FetchedPlaylist;
use crate::model::InputSource;
use crate::model::{AppConfig, ConfigTarget};
use crate::model::{ConfigInput, ConfigInputFlags};
use crate::processing::input_cache::resolve_input_storage_path;
use crate::processing::processor::playlist::PlaylistProcessingContext;
use crate::processing::processor::{
create_resolve_options_function_for_xtream_target, process_foreground_retry_once, select_cancel_token,
@@ -12,7 +13,6 @@ use crate::processing::processor::{
FOREGROUND_RETRY_BATCH_MAX_SIZE as RETRY_BATCH_MAX_SIZE,
};
use crate::ptt::ptt_parse_title;
use crate::repository::get_input_storage_path;
use crate::repository::persist_input_vod_info;
use crate::repository::persist_input_vod_info_batch;
use crate::repository::{xtream_get_file_path, BPlusTreeQuery};
@@ -148,15 +148,9 @@ async fn process_immediate_vod_info(
resolve_tmdb_enabled: bool,
) {
let input = fpl.input;
let working_dir = &ctx.config.config.load().working_dir;
let storage_dir = &ctx.config.config.load().storage_dir;
let storage_path = match get_input_storage_path(&input.name, working_dir).await {
Ok(path) => path,
Err(err) => {
error!("Can't resolve vod, input storage directory for input '{}' failed: {err}", input.name);
return;
}
};
let storage_path = resolve_input_storage_path(storage_dir, &input.name).await;
// Keep an optional read query open and reopen lazily only when needed.
let xtream_path = xtream_get_file_path(&storage_path, XtreamCluster::Video);
@@ -447,6 +441,26 @@ fn queue_background_vod_info(
let reasons = check_resolve_reasons(resolve_options, do_probe, resolve_tmdb_enabled, pli);
if !reasons.is_empty() {
if log_enabled!(Level::Debug) {
let has_details = pli.has_details();
let (has_tmdb, has_date, has_video, has_audio) =
match pli.header.additional_properties.as_ref() {
Some(StreamProperties::Video(props)) => {
let details = props.details.as_ref();
(
props.tmdb.is_some(),
details.and_then(|d| d.release_date.as_ref()).is_some(),
MediaQuality::is_valid_media_info(details.and_then(|d| d.video.as_deref())),
MediaQuality::is_valid_media_info(details.and_then(|d| d.audio.as_deref())),
)
}
_ => (false, false, false, false),
};
debug!(
"[Task] Creating ResolveVod task for input {}: id={}, reasons={}, has_details={}, has_tmdb={}, has_date={}, has_video_info={}, has_audio_info={}, title=\"{}\"",
input.name, provider_id, reasons, has_details, has_tmdb, has_date, has_video, has_audio, pli.header.title
);
}
let task =
UpdateTask::ResolveVod { id: provider_id, reason: reasons, delay: resolve_options.resolve_delay };
mgr.queue_task_background(input.name.clone(), task);
@@ -509,10 +523,8 @@ pub async fn update_vod_metadata(
db_query: Option<Arc<Mutex<BPlusTreeQuery<u32, XtreamPlaylistItem>>>>,
tmdb_resolved_out: Option<&AtomicBool>,
) -> Result<Option<VideoStreamProperties>, TuliproxError> {
let working_dir = &app_config.config.load().working_dir;
let storage_path = get_input_storage_path(&input.name, working_dir)
.await
.map_err(|e| shared::error::info_err!("Storage path error: {e}"))?;
let storage_dir = &app_config.config.load().storage_dir;
let storage_path = resolve_input_storage_path(storage_dir, &input.name).await;
// Check if we should skip based on input options
if input.has_flag(ConfigInputFlags::XtreamSkipVod) {
+14 -12
View File
@@ -7,7 +7,7 @@ use log::{error, warn};
use shared::error::{string_to_io_error, to_io_error, TuliproxError};
use shared::info_err;
use shared::model::{ConfigInputAliasDto, InputType};
use shared::utils::{get_credentials_from_url, get_credentials_from_url_str, parse_timestamp, sanitize_sensitive_info, Internable, PROVIDER_SCHEME_PREFIX};
use shared::utils::{get_credentials_from_url, get_credentials_from_url_str, parse_timestamp, sanitize_sensitive_info, Internable, BATCH_SCHEME_PREFIX, PROVIDER_SCHEME_PREFIX};
use std::io;
use std::io::{BufRead, Cursor, Error};
use std::path::{Path, PathBuf};
@@ -267,21 +267,23 @@ pub async fn csv_read_inputs(
}
pub fn get_csv_file_path(file_uri: &str) -> Result<PathBuf, Error> {
// Handle batch:// scheme: strip prefix and treat remainder as file path.
if let Some(path_str) = file_uri.strip_prefix(BATCH_SCHEME_PREFIX) {
let path = Path::new(path_str);
return if path.is_absolute() {
Ok(path.to_path_buf())
} else {
resolve_relative_path(path_str)
};
}
let raw_path = Path::new(file_uri);
if raw_path.is_absolute() {
return Ok(raw_path.to_path_buf());
}
if let Ok(url) = file_uri.parse::<Url>() {
if url.scheme() == "file" {
match url.to_file_path() {
Ok(path) => Ok(path),
Err(()) => Err(string_to_io_error(format!("Could not open {file_uri}"))),
}
} else {
Err(string_to_io_error(format!(
"Only file:// is supported {file_uri}"
)))
}
if let Ok(_url) = file_uri.parse::<Url>() {
Err(string_to_io_error(format!(
"Unsupported URL scheme for batch CSV, use batch:// instead: {file_uri}"
)))
} else {
resolve_relative_path(file_uri)
}
+2 -2
View File
@@ -1747,7 +1747,7 @@ where
let mut v = Vec::with_capacity(info.len());
let original_keys = std::mem::take(&mut keys);
for (entry_key, entry_info) in original_keys.into_iter().zip(info.into_iter()) {
for (entry_key, entry_info) in original_keys.into_iter().zip(info) {
if entry_info.is_tombstone() {
continue;
}
@@ -1854,7 +1854,7 @@ where
let mut filtered_info: Vec<ValueInfo> = Vec::with_capacity(info.len());
let mut last_packed_block: Option<(u64, Vec<u8>)> = None;
let original_keys = std::mem::take(&mut keys);
for (entry_key, entry_info) in original_keys.into_iter().zip(info.into_iter()) {
for (entry_key, entry_info) in original_keys.into_iter().zip(info) {
if entry_info.is_tombstone() {
continue;
}
+4 -4
View File
@@ -181,8 +181,8 @@ async fn persist_m3u_playlist_as_text(
let Some(filename) = target_output.filename.as_ref() else {
return Ok(());
};
let working_dir = &app_config.config.load().working_dir;
let Some(m3u_filename) = utils::get_file_path(working_dir, Some(PathBuf::from(filename))) else {
let cfg = app_config.config.load();
let Some(m3u_filename) = utils::get_file_path(&cfg.storage_dir, Some(PathBuf::from(filename))) else {
return Ok(());
};
@@ -375,8 +375,8 @@ pub async fn iter_raw_m3u_input_playlist(
input: &ConfigInput,
cluster: Option<XtreamCluster>,
) -> Option<Box<dyn Stream<Item=Result<M3uPlaylistItem, TuliproxError>> + Send + Unpin>> {
let working_dir = &app_config.config.load().working_dir;
let storage_path = get_input_storage_path(&input.name, working_dir).await.ok()?;
let cfg = app_config.config.load();
let storage_path = get_input_storage_path(&input.name, &cfg.storage_dir).await.ok()?;
let m3u_path = get_input_m3u_playlist_file_path(&storage_path, &input.name);
iter_raw_m3u_playlist::<u32, Arc<str>>(app_config, &m3u_path, cluster).await
+2
View File
@@ -17,6 +17,7 @@ mod library_repository;
mod sorted_index;
mod alias_repository;
mod playlist_stream;
mod provider_dns_repository;
pub use storage::*;
pub use target_id_mapping::*;
@@ -36,6 +37,7 @@ pub use alias_repository::*;
pub use playlist_source::*;
pub use library_repository::*;
pub use playlist_stream::*;
pub use provider_dns_repository::*;
#[cfg(test)]
mod bplustree_stress;
+12 -27
View File
@@ -371,28 +371,21 @@ pub async fn persist_input_playlist(app_config: &Arc<AppConfig>, input: &ConfigI
return (playlist, None);
}
playlist.iter_mut().for_each(PlaylistGroup::on_load);
let cfg = app_config.config.load();
let storage_path = match get_input_storage_path(&input.name, &cfg.storage_dir).await {
Ok(storage_path) => storage_path,
Err(err) => {
return (playlist, Some(info_err!("Error creating input storage directory for input '{}' failed: {err}", input.name)));
}
};
match input.input_type {
match input.get_download_input_type() {
InputType::Xtream | InputType::XtreamBatch => {
let working_dir = &app_config.config.load().working_dir;
let storage_path = match get_input_storage_path(&input.name, working_dir).await {
Ok(storage_path) => storage_path,
Err(err) => {
return (playlist, Some(info_err!("Error creating input storage directory for input '{}' failed: {err}", input.name)));
}
};
persist_input_xtream_playlist(app_config, &storage_path, playlist).await
}
InputType::M3u | InputType::M3uBatch => {
// Persist M3U
let working_dir = &app_config.config.load().working_dir;
let storage_path = match get_input_storage_path(&input.name, working_dir).await {
Ok(storage_path) => storage_path,
Err(err) => {
return (playlist, Some(info_err!("Error creating input storage directory for input '{}' failed: {err}", input.name)));
}
};
let file_path = get_input_m3u_playlist_file_path(&storage_path, &input.name);
if let Err(err) = persist_input_m3u_playlist(app_config, &file_path, &playlist).await {
return (playlist, Some(err));
@@ -401,13 +394,6 @@ pub async fn persist_input_playlist(app_config: &Arc<AppConfig>, input: &ConfigI
}
InputType::Library => {
// Persist local library playlist
let working_dir = &app_config.config.load().working_dir;
let storage_path = match get_input_storage_path(&input.name, working_dir).await {
Ok(storage_path) => storage_path,
Err(err) => {
return (playlist, Some(info_err!("Error creating input storage directory for input '{}' failed: {err}", input.name)));
}
};
let file_path = get_input_local_library_playlist_file_path(&storage_path, &input.name);
let (playlist, result) = persist_input_library_playlist(app_config, &file_path, playlist).await;
if let Err(err) = result {
@@ -420,13 +406,12 @@ pub async fn persist_input_playlist(app_config: &Arc<AppConfig>, input: &ConfigI
pub async fn load_input_playlist(ctx: &PlaylistProcessingContext, input: &ConfigInput, clusters: Option<&[XtreamCluster]>) -> Result<Box<dyn PlaylistSource>, TuliproxError> {
let app_config = &ctx.config;
let working_dir = &app_config.config.load().working_dir;
let storage_path = get_input_storage_path(&input.name, working_dir).await
let cfg = app_config.config.load();
let storage_path = get_input_storage_path(&input.name, &cfg.storage_dir).await
.map_err(|e| info_err!("Error getting input path: {e}"))?;
let disk_based_processing = cfg.disk_based_processing;
let disk_based_processing = app_config.config.load().disk_based_processing;
match input.input_type {
match input.get_download_input_type() {
InputType::Xtream | InputType::XtreamBatch => {
if disk_based_processing {
Ok(Box::new(XtreamDiskPlaylistSource::new(app_config, &storage_path).await))
@@ -0,0 +1,434 @@
use indexmap::IndexMap;
use log::{debug, info, warn};
use serde::{Deserialize, Serialize};
use shared::utils::DNS_RESOLVED_FILE;
use std::collections::{HashMap, HashSet};
use std::io::ErrorKind;
use std::net::IpAddr;
use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::Arc;
use std::time::Duration;
use tokio::fs;
use tokio::sync::mpsc;
use tokio_util::sync::CancellationToken;
use crate::api::model::AppState;
use crate::model::ConfigProvider;
/// All providers' resolved DNS data, keyed by provider name → hostname → IPs.
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct DnsResolvedStore {
#[serde(flatten)]
pub providers: HashMap<String, IndexMap<String, Vec<IpAddr>>>,
}
static DNS_WRITER_GENERATION: AtomicU64 = AtomicU64::new(0);
const DNS_WRITER_FLUSH_INTERVAL_SECS: u64 = 2;
const DNS_WRITER_FLUSH_BATCH_THRESHOLD: usize = 32;
pub fn next_dns_writer_generation() -> u64 { DNS_WRITER_GENERATION.fetch_add(1, Ordering::SeqCst) + 1 }
fn is_dns_writer_generation_current(generation: u64) -> bool {
DNS_WRITER_GENERATION.load(Ordering::SeqCst) == generation
}
pub fn dns_resolved_file_path(storage_dir: &str) -> PathBuf {
PathBuf::from(storage_dir).join(DNS_RESOLVED_FILE)
}
#[derive(Debug)]
pub enum DnsResolvedStoreLoadError {
Read(std::io::Error),
Parse(serde_json::Error),
}
impl std::fmt::Display for DnsResolvedStoreLoadError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Self::Read(err) => write!(f, "read failed: {err}"),
Self::Parse(err) => write!(f, "parse failed: {err}"),
}
}
}
pub async fn load_dns_resolved_store_from_path(path: &Path) -> Result<Option<DnsResolvedStore>, DnsResolvedStoreLoadError> {
match fs::read_to_string(path).await {
Ok(data) => serde_json::from_str(&data).map(Some).map_err(DnsResolvedStoreLoadError::Parse),
Err(err) if err.kind() == ErrorKind::NotFound => Ok(None),
Err(err) => Err(DnsResolvedStoreLoadError::Read(err)),
}
}
pub async fn persist_dns_resolved_store(path: &Path, store: &DnsResolvedStore) -> Result<(), String> {
let json = match serde_json::to_string_pretty(store) {
Ok(value) => value,
Err(err) => return Err(format!("serialize failed: {err}")),
};
if let Some(parent) = path.parent() {
fs::create_dir_all(parent)
.await
.map_err(|err| format!("create parent dir '{}' failed: {err}", parent.display()))?;
}
let tmp_path = path.with_extension("json.tmp");
fs::write(&tmp_path, json)
.await
.map_err(|err| format!("write temp file '{}' failed: {err}", tmp_path.display()))?;
if let Err(err) = fs::rename(&tmp_path, path).await {
#[cfg(windows)]
{
if fs::remove_file(path).await.is_ok() && fs::rename(&tmp_path, path).await.is_ok() {
debug!(
"Persisted DNS resolved store to '{}' (providers={})",
path.display(),
store.providers.len()
);
return Ok(());
}
}
let _ = fs::remove_file(&tmp_path).await;
return Err(format!(
"rename temp file '{}' → '{}' failed: {err}",
tmp_path.display(),
path.display()
));
}
debug!(
"Persisted DNS resolved store to '{}' (providers={})",
path.display(),
store.providers.len()
);
Ok(())
}
/// Load persisted DNS resolved data and seed the in-memory caches for all providers.
/// Called at startup and after config reloads, before DNS background tasks run.
pub async fn load_persisted_dns_resolved(app_state: &std::sync::Arc<crate::api::model::AppState>) {
let storage_dir = app_state.app_config.config.load().storage_dir.clone();
let path = dns_resolved_file_path(&storage_dir);
let store = match load_dns_resolved_store_from_path(&path).await {
Ok(Some(store)) => store,
Ok(None) => {
debug!("No persisted DNS resolved file found at '{}'", path.display());
return;
}
Err(err) => {
warn!("Failed to load persisted DNS resolved file '{}': {err}", path.display());
return;
}
};
let sources = app_state.app_config.sources.load();
let mut seeded_count = 0usize;
for (provider_name, hosts) in &store.providers {
let Some(provider) = sources.get_provider_by_name(provider_name) else {
debug!("Persisted DNS resolved data for unknown provider '{provider_name}', skipping");
continue;
};
if !provider.get_dns_config().is_some_and(|cfg| cfg.enabled) {
debug!("Persisted DNS resolved data for disabled-dns provider '{provider_name}', skipping");
continue;
}
for (host, ips) in hosts {
if !ips.is_empty() {
// Only seed if the cache is currently empty for this host (don't overwrite runtime data).
if provider.dns_cache.ip_count(host) == 0 {
provider.dns_cache.store_resolved(host, ips.clone());
seeded_count += 1;
}
}
}
}
if seeded_count > 0 {
info!("Seeded {seeded_count} host(s) from persisted DNS resolved file '{}'", path.display());
}
}
fn prune_store_to_runtime_enabled_providers(app_state: &Arc<AppState>, store: &mut DnsResolvedStore) -> usize {
let sources = app_state.app_config.sources.load();
let enabled_provider_names = sources
.provider
.iter()
.filter(|provider| provider.get_dns_config().is_some_and(|cfg| cfg.enabled))
.map(|provider| provider.name.to_string())
.collect::<HashSet<String>>();
prune_store_to_enabled_provider_names(store, &enabled_provider_names)
}
fn prune_store_to_enabled_provider_names(
store: &mut DnsResolvedStore,
enabled_provider_names: &HashSet<String>,
) -> usize {
let before = store.providers.len();
store
.providers
.retain(|provider_name, _| enabled_provider_names.contains(provider_name));
before.saturating_sub(store.providers.len())
}
fn is_provider_dns_enabled_in_runtime(app_state: &Arc<AppState>, provider_name: &str) -> bool {
let sources = app_state.app_config.sources.load();
sources
.get_provider_by_name(provider_name)
.is_some_and(|provider| provider.get_dns_config().is_some_and(|cfg| cfg.enabled))
}
pub async fn prune_persisted_dns_resolved_to_runtime(app_state: &Arc<AppState>) {
let storage_dir = app_state.app_config.config.load().storage_dir.clone();
let path = dns_resolved_file_path(&storage_dir);
let mut store = match load_dns_resolved_store_from_path(&path).await {
Ok(Some(existing)) => existing,
Ok(None) => return,
Err(err) => {
warn!(
"Failed to load DNS resolved store for runtime-prune '{}': {err}",
path.display()
);
return;
}
};
let removed = prune_store_to_runtime_enabled_providers(app_state, &mut store);
if removed == 0 {
return;
}
if let Err(err) = persist_dns_resolved_store(&path, &store).await {
warn!(
"Failed to persist DNS resolved store after runtime-prune '{}': {err}",
path.display()
);
} else {
info!(
"Pruned {removed} stale provider(s) from DNS resolved store '{}'",
path.display()
);
}
}
#[derive(Debug)]
pub struct DnsResolvedWriteUpdate {
pub provider_name: String,
pub resolved: IndexMap<String, Vec<IpAddr>>,
}
pub type DnsResolvedWriteTx = mpsc::Sender<DnsResolvedWriteUpdate>;
#[allow(clippy::too_many_lines)]
pub fn spawn_dns_resolved_writer(
app_state: Arc<AppState>,
cancel: CancellationToken,
generation: u64,
) -> DnsResolvedWriteTx {
let storage_dir = app_state.app_config.config.load().storage_dir.clone();
let path = dns_resolved_file_path(&storage_dir);
let (tx, mut rx) = mpsc::channel::<DnsResolvedWriteUpdate>(256);
tokio::spawn(async move {
let mut store = match load_dns_resolved_store_from_path(&path).await {
Ok(Some(existing)) => existing,
Ok(None) => DnsResolvedStore::default(),
Err(err) => {
warn!("Failed to load existing DNS resolved store '{}': {err}", path.display());
DnsResolvedStore::default()
}
};
let removed_on_start = prune_store_to_runtime_enabled_providers(&app_state, &mut store);
if removed_on_start > 0 {
info!(
"Pruned {removed_on_start} stale provider(s) from DNS resolved store on writer start '{}'",
path.display()
);
if let Err(err) = persist_dns_resolved_store(&path, &store).await {
warn!(
"Failed to persist DNS resolved store after startup prune '{}': {err}",
path.display()
);
}
}
let mut dirty = false;
let mut dirty_updates = 0usize;
let mut flush_before_exit = false;
let mut flush_timer = std::pin::pin!(tokio::time::sleep(Duration::from_secs(
DNS_WRITER_FLUSH_INTERVAL_SECS,
)));
let mut flush_timer_active = false;
loop {
if !is_dns_writer_generation_current(generation) {
debug!(
"Stopping stale DNS resolved writer generation {} for '{}'",
generation,
path.display()
);
break;
}
tokio::select! {
() = cancel.cancelled() => {
debug!("Stopping DNS resolved writer task for '{}'", path.display());
flush_before_exit = true;
break;
}
maybe_update = rx.recv() => {
let Some(update) = maybe_update else {
debug!("DNS resolved writer channel closed for '{}'", path.display());
flush_before_exit = true;
break;
};
if !is_dns_writer_generation_current(generation) {
debug!(
"Dropping update for stale DNS resolved writer generation {} on '{}'",
generation,
path.display()
);
break;
}
if update.resolved.is_empty() || !is_provider_dns_enabled_in_runtime(&app_state, update.provider_name.as_str()) {
store.providers.remove(update.provider_name.as_str());
} else {
store.providers.insert(update.provider_name, update.resolved);
}
dirty = true;
dirty_updates = dirty_updates.saturating_add(1);
if dirty_updates >= DNS_WRITER_FLUSH_BATCH_THRESHOLD {
let _ = prune_store_to_runtime_enabled_providers(&app_state, &mut store);
if let Err(err) = persist_dns_resolved_store(&path, &store).await {
warn!("Failed to persist DNS resolved store '{}': {err}", path.display());
} else {
dirty = false;
dirty_updates = 0;
flush_timer_active = false;
}
} else if !flush_timer_active {
flush_timer.as_mut().reset(
tokio::time::Instant::now() + Duration::from_secs(DNS_WRITER_FLUSH_INTERVAL_SECS),
);
flush_timer_active = true;
}
}
() = &mut flush_timer, if flush_timer_active => {
flush_timer_active = false;
if dirty {
let _ = prune_store_to_runtime_enabled_providers(&app_state, &mut store);
if let Err(err) = persist_dns_resolved_store(&path, &store).await {
warn!("Failed to persist DNS resolved store '{}': {err}", path.display());
} else {
dirty = false;
dirty_updates = 0;
}
}
}
}
}
if flush_before_exit && dirty {
let _ = prune_store_to_runtime_enabled_providers(&app_state, &mut store);
if let Err(err) = persist_dns_resolved_store(&path, &store).await {
warn!(
"Failed to persist DNS resolved store during writer shutdown '{}': {err}",
path.display()
);
}
}
});
tx
}
pub async fn queue_provider_resolved_snapshot(writer_tx: &DnsResolvedWriteTx, provider: &Arc<ConfigProvider>) {
let update = DnsResolvedWriteUpdate {
provider_name: provider.name.to_string(),
resolved: provider.snapshot_resolved_ordered(),
};
if let Err(err) = writer_tx.send(update).await {
warn!("Failed to queue DNS resolved update for provider '{}': {err}", provider.name);
}
}
#[cfg(test)]
mod tests {
use super::{
is_dns_writer_generation_current, load_dns_resolved_store_from_path, next_dns_writer_generation,
persist_dns_resolved_store, prune_store_to_enabled_provider_names, DnsResolvedStore, DnsResolvedStoreLoadError,
};
use indexmap::IndexMap;
use std::collections::{HashMap, HashSet};
use std::net::IpAddr;
#[tokio::test]
async fn load_dns_resolved_store_reports_parse_error() {
let dir = tempfile::tempdir().expect("temp dir should be created");
let path = dir.path().join("provider_dns_resolved.json");
tokio::fs::write(&path, "{ not-json").await.expect("test file should be written");
let result = load_dns_resolved_store_from_path(&path).await;
assert!(matches!(result, Err(DnsResolvedStoreLoadError::Parse(_))));
}
#[tokio::test]
async fn persist_and_load_dns_resolved_store_roundtrip() {
let dir = tempfile::tempdir().expect("temp dir should be created");
let path = dir.path().join("provider_dns_resolved.json");
let mut providers = HashMap::new();
providers.insert(
"provider-a".to_string(),
IndexMap::from([(
"example.com".to_string(),
vec!["203.0.113.10".parse::<IpAddr>().expect("valid ip")],
)]),
);
let store = DnsResolvedStore { providers };
persist_dns_resolved_store(&path, &store).await.expect("persist should succeed");
let loaded = load_dns_resolved_store_from_path(&path)
.await
.expect("load should succeed")
.expect("store should exist");
assert_eq!(loaded.providers.len(), 1);
assert_eq!(
loaded
.providers
.get("provider-a")
.and_then(|hosts| hosts.get("example.com")),
Some(&vec!["203.0.113.10".parse::<IpAddr>().expect("valid ip")])
);
}
#[test]
fn generation_switch_marks_previous_writer_as_stale() {
let old_generation = next_dns_writer_generation();
let new_generation = next_dns_writer_generation();
assert_ne!(old_generation, new_generation);
assert!(!is_dns_writer_generation_current(old_generation));
assert!(is_dns_writer_generation_current(new_generation));
}
#[test]
fn prune_store_to_enabled_provider_names_removes_stale_entries() {
let mut providers = HashMap::new();
providers.insert(
"provider-a".to_string(),
IndexMap::from([("example.com".to_string(), vec!["203.0.113.10".parse::<IpAddr>().expect("valid ip")])]),
);
providers.insert(
"provider-b".to_string(),
IndexMap::from([("example.org".to_string(), vec!["203.0.113.20".parse::<IpAddr>().expect("valid ip")])]),
);
let mut store = DnsResolvedStore { providers };
let enabled = HashSet::from(["provider-a".to_string()]);
let removed = prune_store_to_enabled_provider_names(&mut store, &enabled);
assert_eq!(removed, 1);
assert!(store.providers.contains_key("provider-a"));
assert!(!store.providers.contains_key("provider-b"));
}
}
+8 -8
View File
@@ -25,7 +25,7 @@ pub async fn ensure_target_storage_path(cfg: &Config, target_name: &str) -> Resu
}
pub fn get_target_storage_path(cfg: &Config, target_name: &str) -> Option<PathBuf> {
utils::get_file_path(&cfg.working_dir, Some(std::path::PathBuf::from(target_name.replace(' ', "_"))))
utils::get_file_path(&cfg.storage_dir, Some(std::path::PathBuf::from(target_name.replace(' ', "_"))))
}
pub fn sanitize_name(name: &str) -> String {
@@ -34,27 +34,27 @@ pub fn sanitize_name(name: &str) -> String {
.collect()
}
pub fn build_input_storage_path(input_name: &str, working_dir: &str) -> PathBuf {
pub fn build_input_storage_path(input_name: &str, storage_dir: &str) -> PathBuf {
let sanitized_name: String = sanitize_name(input_name);
let name = concat_string!(cap = 6 + sanitized_name.len(); "input_", &sanitized_name);
Path::new(working_dir).join(name)
Path::new(storage_dir).join(name)
}
pub async fn get_input_storage_path(input_name: &str, working_dir: &str) -> std::io::Result<PathBuf> {
let path = build_input_storage_path(input_name, working_dir);
pub async fn get_input_storage_path(input_name: &str, storage_dir: &str) -> std::io::Result<PathBuf> {
let path = build_input_storage_path(input_name, storage_dir);
// Create the directory and return the path or propagate the error
tokio::fs::create_dir_all(&path).await.map(|()| path)
}
pub async fn ensure_input_storage_path(cfg: &Config, input_name: &str) -> Result<PathBuf, TuliproxError> {
get_input_storage_path(input_name, &cfg.working_dir).await
get_input_storage_path(input_name, &cfg.storage_dir).await
.map_err(|err| {
notify_err!("Failed to save input data, can't create directory for input {input_name}: {err}")
})
}
pub fn get_geoip_path(working_dir: &str) -> PathBuf {
Path::new(working_dir).join("geoip.db")
pub fn get_geoip_path(storage_dir: &str) -> PathBuf {
Path::new(storage_dir).join("geoip.db")
}
pub fn get_file_path_for_db_index(db_path: &Path) -> PathBuf {
+1 -1
View File
@@ -850,7 +850,7 @@ pub async fn write_strm_playlist(
let config = app_config.config.load();
let Some(root_path) = crate::utils::get_file_path(
&config.working_dir,
&config.storage_dir,
Some(std::path::PathBuf::from(&target_output.directory)),
) else {
return info_err_res!("Failed to get file path for {}",target_output.directory);
@@ -517,7 +517,9 @@ mod tests {
hdhomerun: Arc::new(ArcSwapAny::default()),
api_proxy: Arc::new(ArcSwapAny::default()),
paths: Arc::new(ArcSwap::from(Arc::new(ConfigPaths {
home_path: String::new(),
config_path: temp_dir().to_string_lossy().to_string(),
storage_path: temp_dir().to_string_lossy().to_string(),
config_file_path: String::new(),
sources_file_path: String::new(),
mapping_file_path: None,
+85 -28
View File
@@ -654,8 +654,8 @@ pub async fn iter_raw_xtream_target_playlist(app_config: &AppConfig, target: &Co
pub async fn iter_raw_xtream_input_playlist(app_config: &AppConfig, input: &ConfigInput, cluster: XtreamCluster) -> Option<Box<dyn Stream<Item = XtreamPlaylistItem> + Send + Unpin>> {
let config = app_config.config.load();
let working_dir = &config.working_dir;
let storage_path = get_input_storage_path(&input.name, working_dir).await.ok()?;
let storage_dir = &config.storage_dir;
let storage_path = get_input_storage_path(&input.name, storage_dir).await.ok()?;
let xtream_path = xtream_get_file_path(&storage_path, cluster);
iter_raw_xtream_playlist(app_config, &xtream_path).await
@@ -1211,15 +1211,14 @@ fn needs_preserved_stream_property_merge(
let preserve_info_details = old_stream_props.has_details() && !needs_update_info_details(new_stream_props, old_stream_props);
match (new_stream_props, old_stream_props) {
(StreamProperties::Video(v_new), StreamProperties::Video(v_old))
if preserve_info_details && v_old.details.is_some() =>
{
v_new.details != v_old.details
(StreamProperties::Video(v_new), StreamProperties::Video(v_old)) => {
(preserve_info_details && v_old.details.is_some() && v_new.details != v_old.details)
|| (v_new.tmdb.is_none() && v_old.tmdb.is_some())
}
(StreamProperties::Series(s_new), StreamProperties::Series(s_old))
if preserve_info_details && s_old.details.is_some() =>
{
s_new.details != s_old.details
(StreamProperties::Series(s_new), StreamProperties::Series(s_old)) => {
(preserve_info_details && s_old.details.is_some() && s_new.details != s_old.details)
|| (s_new.tmdb.is_none() && s_old.tmdb.is_some())
|| (s_new.release_date.is_none() && s_old.release_date.is_some())
}
(StreamProperties::Live(l_new), StreamProperties::Live(l_old)) => {
(l_new.video.is_none() && l_old.video.is_some())
@@ -1245,26 +1244,41 @@ pub(crate) fn merge_preserved_stream_properties(
old_stream_props.has_details() && !needs_update_info_details(new_stream_props, old_stream_props);
match (new_stream_props, old_stream_props) {
(StreamProperties::Video(v_new), StreamProperties::Video(v_old))
if preserve_info_details && v_old.details.is_some() =>
{
if v_new.details == v_old.details {
false
} else {
v_new.details.clone_from(&v_old.details);
true
}
(StreamProperties::Video(v_new), StreamProperties::Video(v_old)) => {
let mut changed = false;
if preserve_info_details && v_old.details.is_some() && v_new.details != v_old.details {
v_new.details.clone_from(&v_old.details);
changed = true;
}
(StreamProperties::Series(s_new), StreamProperties::Series(s_old))
if preserve_info_details && s_old.details.is_some() =>
{
if s_new.details == s_old.details {
false
} else {
s_new.details.clone_from(&s_old.details);
true
}
if v_new.tmdb.is_none() && v_old.tmdb.is_some() {
v_new.tmdb = v_old.tmdb;
changed = true;
}
changed
}
(StreamProperties::Series(s_new), StreamProperties::Series(s_old)) => {
let mut changed = false;
if preserve_info_details && s_old.details.is_some() && s_new.details != s_old.details {
s_new.details.clone_from(&s_old.details);
changed = true;
}
if s_new.tmdb.is_none() && s_old.tmdb.is_some() {
s_new.tmdb = s_old.tmdb;
changed = true;
}
if s_new.release_date.is_none() && s_old.release_date.is_some() {
s_new.release_date.clone_from(&s_old.release_date);
changed = true;
}
changed
}
(StreamProperties::Live(l_new), StreamProperties::Live(l_old)) => {
let mut changed = false;
@@ -1586,6 +1600,49 @@ mod tests {
}
}
#[test]
fn merge_preserves_missing_video_tmdb() {
let mut new_props = StreamProperties::Video(Box::new(VideoStreamProperties {
tmdb: None,
..VideoStreamProperties::default()
}));
let old_props = StreamProperties::Video(Box::new(VideoStreamProperties {
tmdb: Some(317981),
..VideoStreamProperties::default()
}));
let changed = merge_preserved_stream_properties(&mut new_props, &old_props);
assert!(changed);
match new_props {
StreamProperties::Video(video) => assert_eq!(video.tmdb, Some(317981)),
_ => panic!("expected video properties"),
}
}
#[test]
fn merge_preserves_missing_series_tmdb_and_release_date() {
let mut new_props = StreamProperties::Series(Box::new(SeriesStreamProperties {
tmdb: None,
release_date: None,
..SeriesStreamProperties::default()
}));
let old_props = StreamProperties::Series(Box::new(SeriesStreamProperties {
tmdb: Some(12345),
release_date: Some("2015-01-01".into()),
..SeriesStreamProperties::default()
}));
let changed = merge_preserved_stream_properties(&mut new_props, &old_props);
assert!(changed);
match new_props {
StreamProperties::Series(series) => {
assert_eq!(series.tmdb, Some(12345));
assert_eq!(series.release_date.as_deref(), Some("2015-01-01"));
}
_ => panic!("expected series properties"),
}
}
fn make_live_item(
provider_id: u32,
video: Option<&str>,
+40 -35
View File
@@ -22,7 +22,6 @@ use shared::model::{
MsgKind, PatternTemplate, SourcesConfigDto, TargetUserDto, TemplateDefinitionDto,
};
use shared::utils::{generate_default_access_secret, generate_default_encrypt_secret, CONSTANTS, PROVIDER_SCHEME_PREFIX, TEMPLATE_FILE};
use std::borrow::Cow;
use std::collections::{HashMap, HashSet};
use std::env;
use std::fs::File;
@@ -112,14 +111,19 @@ async fn parse_sources_file_from_path(
.map_err(|join_err| info_err!("Failed to read sources-config file: {join_err}"))?
}
pub fn resolve_template_and_mapping_paths<'a>(paths: &'a ConfigPaths, template_path: Option<&'a str>, mapping_path: Option<&'a str>) -> (Cow<'a, str>, Cow<'a, str>) {
// Resolve effective paths for templates and mappings (with robust fallbacks)
let effective_template_path = paths.template_file_path.as_deref()
.or(template_path).map_or_else(|| Cow::Owned(utils::get_default_templates_path(&paths.config_path)), Cow::Borrowed);
let effective_mapping_path = paths.mapping_file_path.as_deref()
.or(mapping_path).map_or_else(|| Cow::Owned(utils::get_default_mappings_path(&paths.config_path)), Cow::Borrowed);
pub fn resolve_template_and_mapping_paths(
paths: &ConfigPaths,
template_path: Option<&str>,
mapping_path: Option<&str>,
) -> (String, String) {
let effective_template_path = utils::resolve_template_file_path(
paths.config_path.as_str(),
paths.template_file_path.as_deref().or(template_path),
);
let effective_mapping_path = utils::resolve_mapping_file_path(
paths.config_path.as_str(),
paths.mapping_file_path.as_deref().or(mapping_path),
);
(effective_template_path, effective_mapping_path)
}
@@ -322,14 +326,15 @@ pub async fn read_app_config_dto(
let config = read_config_file(config_file, resolve_env, include_computed)?;
// Resolve effective paths for templates and mappings (with robust fallbacks)
let (effective_template_path,effective_mapping_path) = resolve_template_and_mapping_paths(paths, config.template_path.as_deref(), config.mapping_path.as_deref());
let (effective_template_path, effective_mapping_path) =
resolve_template_and_mapping_paths(paths, config.template_path.as_deref(), config.mapping_path.as_deref());
let mut sources = parse_sources_file_from_path(&PathBuf::from(sources_file), resolve_env).await?;
let mut mappings = read_mappings_file_unprepared(effective_mapping_path.as_ref(), resolve_env)?
let mut mappings = read_mappings_file_unprepared(&effective_mapping_path, resolve_env)?
.map(|(_, mapping)| mapping);
let template_bundle = read_templates(
Some(effective_template_path.as_ref()),
Some(&effective_template_path),
resolve_env,
sources.templates.as_deref(),
mappings
@@ -447,7 +452,7 @@ pub async fn get_batch_aliases(
if url.starts_with(PROVIDER_SCHEME_PREFIX) {
return info_err_res!(
"Batch input type '{input_type}' does not support provider:// URLs. \
Use a local CSV path (absolute/relative) or file:// URL."
Use a batch:// URL or a local CSV path (absolute/relative)."
);
}
@@ -505,26 +510,25 @@ pub async fn read_initial_app_config(
let config_dto = read_config_file(config_file, resolve_env, include_computed)?;
if paths.mapping_file_path.is_none() {
let mut path = config_dto.mapping_path.as_ref().map_or_else(
|| utils::get_default_mappings_path(config_path),
ToString::to_string,
);
if resolve_env {
path = resolve_env_var(&path);
}
paths.mapping_file_path.replace(path);
}
if paths.template_file_path.is_none() {
let mut path = config_dto.template_path.as_ref().map_or_else(
|| utils::get_default_templates_path(config_path),
ToString::to_string,
);
if resolve_env {
path = resolve_env_var(&path);
}
paths.template_file_path.replace(path);
}
let configured_mapping_path = paths
.mapping_file_path
.as_deref()
.or(config_dto.mapping_path.as_deref())
.map(|path| if resolve_env { resolve_env_var(path) } else { path.to_string() });
let configured_template_path = paths
.template_file_path
.as_deref()
.or(config_dto.template_path.as_deref())
.map(|path| if resolve_env { resolve_env_var(path) } else { path.to_string() });
paths.mapping_file_path = Some(utils::resolve_mapping_file_path(
config_path,
configured_mapping_path.as_deref(),
));
paths.template_file_path = Some(utils::resolve_template_file_path(
config_path,
configured_template_path.as_deref(),
));
let mut sources_dto = parse_sources_file_from_path(&PathBuf::from(sources_file), resolve_env).await?;
@@ -539,7 +543,7 @@ pub async fn read_initial_app_config(
};
let template_bundle = read_templates(
paths.template_file_path.as_deref().or(config_dto.template_path.as_deref()),
paths.template_file_path.as_deref(),
resolve_env,
sources_dto.templates.as_deref(),
mappings_dto
@@ -562,7 +566,8 @@ pub async fn read_initial_app_config(
let sources: SourcesConfig = SourcesConfig::try_from(sources_dto)?;
let mut config: Config = Config::from(config_dto);
config.prepare(config_path)?;
config.prepare(config_path, paths.home_path.as_str())?;
paths.storage_path.clone_from(&config.storage_dir);
config.update_runtime();
let mut app_config = AppConfig {
+117 -25
View File
@@ -67,17 +67,27 @@ pub fn get_home_path() -> PathBuf {
}
pub fn get_default_web_root_path() -> PathBuf {
if let Some(web_root) = env::var(DEFAULT_WEB_ROOT_ENV_VAR)
.ok()
.filter(|p| !p.trim().is_empty()) {
return PathBuf::from(web_root)
get_default_web_root_path_for_home(get_home_path().as_path())
}
pub fn get_default_web_root_path_for_home(home_path: &Path) -> PathBuf {
if let Some(web_root) = env::var(DEFAULT_WEB_ROOT_ENV_VAR).ok().filter(|p| !p.trim().is_empty()) {
let configured = PathBuf::from(web_root);
return if configured.is_absolute() {
configured
} else {
home_path.join(configured)
};
}
get_exe_path().join(DEFAULT_WEB_DIR)
get_default_path_for_home(home_path, DEFAULT_WEB_DIR)
}
pub fn get_default_path(file: &str) -> PathBuf {
let home_path = get_home_path();
get_default_path_for_home(get_home_path().as_path(), file)
}
pub fn get_default_path_for_home(home_path: &Path, file: &str) -> PathBuf {
home_path.join(file)
}
@@ -116,11 +126,52 @@ pub fn get_default_templates_path(config_path: &str) -> String {
get_default_file_path(config_path, TEMPLATE_FILE)
}
fn resolve_config_scoped_path(config_path: &str, candidate: &str) -> String {
let candidate_path = PathBuf::from(candidate.trim()).clean();
if candidate_path.is_absolute() {
return candidate_path.to_string_lossy().to_string();
}
let config_dir = PathBuf::from(config_path).clean();
let relative = if let Some(config_dir_name) = config_dir.file_name() {
candidate_path
.strip_prefix(Path::new(config_dir_name))
.map_or_else(|_| candidate_path.clone(), Path::to_path_buf)
} else {
candidate_path.clone()
};
config_dir.join(relative).clean().to_string_lossy().to_string()
}
#[inline]
pub fn resolve_mapping_file_path(config_path: &str, mapping_path: Option<&str>) -> String {
let configured = mapping_path.map(str::trim).filter(|path| !path.is_empty()).map(ToString::to_string);
let candidate = if shared::utils::is_blank_or_default_mapping_path(&configured) {
get_default_mappings_path(config_path)
} else {
configured.unwrap_or_else(|| get_default_mappings_path(config_path))
};
resolve_config_scoped_path(config_path, &candidate)
}
#[inline]
pub fn resolve_template_file_path(config_path: &str, template_path: Option<&str>) -> String {
let configured = template_path.map(str::trim).filter(|path| !path.is_empty()).map(ToString::to_string);
let candidate = if shared::utils::is_blank_or_default_template_path(&configured) {
get_default_templates_path(config_path)
} else {
configured.unwrap_or_else(|| get_default_templates_path(config_path))
};
resolve_config_scoped_path(config_path, &candidate)
}
#[inline]
pub fn resolve_template_persist_file_path(template_path: Option<&str>, config_path: &str) -> String {
let candidate = template_path
let raw_candidate = template_path
.filter(|path| !path.trim().is_empty())
.map_or_else(|| get_default_templates_path(config_path), ToString::to_string);
let candidate = resolve_template_file_path(config_path, Some(raw_candidate.as_str()));
let path = PathBuf::from(&candidate);
if path.exists() {
@@ -130,8 +181,8 @@ pub fn resolve_template_persist_file_path(template_path: Option<&str>, config_pa
return path.to_string_lossy().to_string();
}
let looks_like_directory = candidate.ends_with('/')
|| candidate.ends_with('\\')
let looks_like_directory = raw_candidate.ends_with('/')
|| raw_candidate.ends_with('\\')
|| path.extension().is_none()
|| path.extension().is_some_and(|ext| ext == "d");
@@ -329,11 +380,11 @@ where
Ok(())
}
pub fn prepare_file_path(persist: Option<&str>, working_dir: &str, action: &str) -> Option<PathBuf> {
pub fn prepare_file_path(persist: Option<&str>, storage_dir: &str, action: &str) -> Option<PathBuf> {
let persist_file: Option<PathBuf> =
persist.map(|persist_path| prepare_persist_path(persist_path, action));
if persist_file.is_some() {
let file_path = get_file_path(working_dir, persist_file);
let file_path = get_file_path(storage_dir, persist_file);
debug_if_enabled!("persist to file: {}", file_path.as_ref().map_or(Cow::from("?"), |p| p.to_string_lossy()));
file_path
} else {
@@ -348,27 +399,28 @@ pub fn read_file_as_bytes(path: &Path) -> std::io::Result<Vec<u8>> {
Ok(buffer)
}
pub fn make_absolute_path(path: &str, working_dir: &str) -> String {
pub fn make_absolute_path(path: &str, storage_dir: &str) -> String {
let rpb = std::path::PathBuf::from(path);
let pathbuf = make_path_absolute(&rpb, working_dir);
let pathbuf = make_path_absolute(&rpb, storage_dir);
pathbuf.to_str().unwrap_or_default().to_string()
}
pub fn make_path_absolute(rpb: &Path, working_dir: &str) -> PathBuf {
pub fn make_path_absolute(rpb: &Path, storage_dir: &str) -> PathBuf {
if rpb.is_relative() {
let mut rpb2 = std::path::PathBuf::from(working_dir).join(rpb);
if !rpb2.exists() {
rpb2 = get_exe_path().join(rpb);
if !storage_dir.trim().is_empty() {
return std::path::PathBuf::from(storage_dir).join(rpb).clean();
}
if !rpb2.exists() {
let cwd = std::env::current_dir();
if let Ok(cwd_path) = cwd {
rpb2 = cwd_path.join(rpb);
}
let exe_based = get_exe_path().join(rpb).clean();
if exe_based.is_absolute() {
return exe_based;
}
if rpb2.exists() {
return rpb2.clean();
if let Ok(cwd_path) = std::env::current_dir() {
return cwd_path.join(rpb).clean();
}
return exe_based;
}
rpb.to_path_buf()
}
@@ -473,7 +525,11 @@ pub fn get_file_extension(path: &str) -> Option<String> {
#[cfg(test)]
mod tests {
use super::normalize_string_path;
use super::{
normalize_string_path, resolve_mapping_file_path, resolve_template_file_path,
resolve_template_persist_file_path,
};
use std::path::PathBuf;
#[test]
fn test_simple_relative_path() {
@@ -509,4 +565,40 @@ mod tests {
let normalized = normalize_string_path(input);
assert_eq!(normalized, "");
}
#[test]
fn test_resolve_template_default_path_uses_config_dir() {
let config_path = std::env::temp_dir().join("tuliprox").join("settings").join("config");
let resolved = resolve_template_file_path(
config_path.to_string_lossy().as_ref(),
Some("./config/template.yml"),
);
let expected = config_path.join("template.yml");
assert_eq!(PathBuf::from(resolved), expected);
}
#[test]
fn test_resolve_template_directory_path_uses_config_dir() {
let config_path = std::env::temp_dir().join("tuliprox").join("settings").join("config");
let resolved = resolve_template_file_path(config_path.to_string_lossy().as_ref(), Some("template.d"));
let expected = config_path.join("template.d");
assert_eq!(PathBuf::from(resolved), expected);
}
#[test]
fn test_resolve_mapping_directory_path_uses_config_dir() {
let config_path = std::env::temp_dir().join("tuliprox").join("settings").join("config");
let resolved = resolve_mapping_file_path(config_path.to_string_lossy().as_ref(), Some("mapping.d"));
let expected = config_path.join("mapping.d");
assert_eq!(PathBuf::from(resolved), expected);
}
#[test]
fn test_resolve_template_persist_file_for_directory_path() {
let config_path = std::env::temp_dir().join("tuliprox").join("settings").join("config");
let resolved =
resolve_template_persist_file_path(Some("template.d"), config_path.to_string_lossy().as_ref());
let expected = config_path.join("template.d").join("template.yml");
assert_eq!(PathBuf::from(resolved), expected);
}
}
+45 -18
View File
@@ -1,28 +1,47 @@
use crate::utils::traverse_dir;
use crate::utils::{config_file_reader, open_file};
use log::warn;
use log::{debug, warn};
use shared::error::{info_err_res, TuliproxError};
use shared::info_err;
use shared::model::TemplateDefinitionDto;
use std::io::ErrorKind;
use std::path::{Path, PathBuf};
fn read_template_definition(
template_file: &Path,
resolve_var: bool,
) -> Result<Option<TemplateDefinitionDto>, TuliproxError> {
if let Ok(file) = open_file(template_file) {
let maybe_definition: Result<TemplateDefinitionDto, _> =
serde_saphyr::from_reader(config_file_reader(file, resolve_var));
return match maybe_definition {
Ok(definition) => Ok(Some(definition)),
Err(err) => info_err_res!("{err}"),
};
match open_file(template_file) {
Ok(file) => {
let maybe_definition: Result<TemplateDefinitionDto, _> =
serde_saphyr::from_reader(config_file_reader(file, resolve_var));
match maybe_definition {
Ok(definition) => Ok(Some(definition)),
Err(err) => info_err_res!(
"Failed to parse template file {}: {err}",
template_file.to_string_lossy()
),
}
}
Err(err) => {
if err.kind() == ErrorKind::NotFound {
debug!(
"Optional template file not found: {}",
template_file.to_str().unwrap_or("?")
);
Ok(None)
} else {
warn!(
"Can't read template file {}: {err}",
template_file.to_str().unwrap_or("?")
);
info_err_res!(
"Can't read template file {}: {err}",
template_file.to_string_lossy()
)
}
}
}
warn!(
"Can't read template file: {}",
template_file.to_str().unwrap_or("?")
);
Ok(None)
}
fn merge_template_definitions(definitions: Vec<TemplateDefinitionDto>) -> Option<TemplateDefinitionDto> {
@@ -104,11 +123,19 @@ pub fn read_templates_file(
}
}
Err(err) => {
warn!(
"Can't read template path metadata for {}: {err}",
path.to_string_lossy()
);
Ok(None)
if err.kind() == ErrorKind::NotFound {
debug!("Optional template path not found: {}", path.to_string_lossy());
Ok(None)
} else {
warn!(
"Can't read template path metadata for {}: {err}",
path.to_string_lossy()
);
info_err_res!(
"Can't read template path metadata for {}: {err}",
path.to_string_lossy()
)
}
}
}
}
+8 -8
View File
@@ -11,28 +11,28 @@ use shared::error::{info_err, TuliproxError};
use shared::utils::{sanitize_sensitive_info, short_hash};
use std::path::PathBuf;
pub async fn get_input_raw_epg_file_path(url: &str, input: &ConfigInput, working_dir: &str) -> std::io::Result<PathBuf> {
pub async fn get_input_raw_epg_file_path(url: &str, input: &ConfigInput, storage_dir: &str) -> std::io::Result<PathBuf> {
let file_prefix = short_hash(url);
if let Some(persist_path) = input.persist.as_deref() {
if !persist_path.is_empty() {
if let Some(path) = prepare_file_path(input.persist.as_deref(), working_dir, "")
if let Some(path) = prepare_file_path(input.persist.as_deref(), storage_dir, "")
.map(|path| add_prefix_to_filename(&path, concat_string!(&file_prefix, "_epg_").as_str(), Some("xml"))) {
return Ok(path);
}
}
}
let download_path = get_input_storage_path(&input.name, working_dir).await?;
let download_path = get_input_storage_path(&input.name, storage_dir).await?;
Ok(download_path.join(format!("{}_{}", file_prefix, storage_const::FILE_EPG)))
}
async fn download_epg_file(url: &str, ctx: &PlaylistProcessingContext,
input: &ConfigInput,
headers: Option<&reqwest::header::HeaderMap>,
working_dir: &str) -> Result<PathBuf, TuliproxError> {
storage_dir: &str) -> Result<PathBuf, TuliproxError> {
debug!("Getting epg file path for url: {}", sanitize_sensitive_info(url));
let persist_file_path = get_input_raw_epg_file_path(url, input, working_dir).await.map_err(|e| info_err!("Could not access epg file download directory: {}", e))?;
let persist_file_path = get_input_raw_epg_file_path(url, input, storage_dir).await.map_err(|e| info_err!("Could not access epg file download directory: {}", e))?;
if input.cache_duration_seconds > 0 {
if let Ok(metadata) = tokio::fs::metadata(&persist_file_path).await {
@@ -55,7 +55,7 @@ async fn download_epg_file(url: &str, ctx: &PlaylistProcessingContext,
return Ok(persist_file_path);
}
debug!("Downloading epg for input '{}'", input.name);
match request::get_input_epg_content_as_file(&ctx.config, &ctx.client, input, headers, working_dir, url, &persist_file_path).await {
match request::get_input_epg_content_as_file(&ctx.config, &ctx.client, input, headers, storage_dir, url, &persist_file_path).await {
Ok(path) => {
ctx.mark_input_downloaded(lock_key.clone()).await;
Ok(path)
@@ -66,7 +66,7 @@ async fn download_epg_file(url: &str, ctx: &PlaylistProcessingContext,
pub async fn get_xmltv(ctx: &PlaylistProcessingContext, input: &ConfigInput,
headers: Option<&reqwest::header::HeaderMap>,
working_dir: &str) -> (Option<TVGuide>, Vec<TuliproxError>) {
storage_dir: &str) -> (Option<TVGuide>, Vec<TuliproxError>) {
match &input.epg {
None => (None, vec![]),
Some(epg_config) => {
@@ -75,7 +75,7 @@ pub async fn get_xmltv(ctx: &PlaylistProcessingContext, input: &ConfigInput,
let mut stored_file_paths = vec![];
for epg_source in &epg_config.sources {
match download_epg_file(&epg_source.url, ctx, input, headers, working_dir).await {
match download_epg_file(&epg_source.url, ctx, input, headers, storage_dir).await {
Ok(file_path) => {
stored_file_paths.push(file_path.clone());
file_paths.push(PersistedEpgSource { file_path, priority: epg_source.priority, logo_override: epg_source.logo_override });
+3 -3
View File
@@ -12,19 +12,19 @@ pub async fn download_m3u_playlist(
cfg: &Arc<Config>,
input: &ConfigInput,
) -> (Vec<PlaylistGroup>, Vec<TuliproxError>) {
let working_dir = &cfg.working_dir;
let storage_dir = &cfg.storage_dir;
let input_source: InputSource = {
match input.staged.as_ref() {
None => input.into(),
Some(staged) => if staged.enabled { staged.into() } else { input.into() },
}
};
let persist_file_path = prepare_file_path(input.persist.as_deref(), working_dir, "");
let persist_file_path = prepare_file_path(input.persist.as_deref(), storage_dir, "");
match request::get_input_text_content_as_stream(
app_config,
client,
&input_source,
working_dir,
storage_dir,
persist_file_path,
)
.await
+18 -15
View File
@@ -20,7 +20,7 @@ use reqwest::{
};
use shared::{
error::{notify_err_res, string_to_io_error, TuliproxError},
model::{format_elapsed_time, InputFetchMethod, OnConnectErrorPolicy, DEFAULT_USER_AGENT},
model::{format_elapsed_time, InputFetchMethod, OnConnectErrorPolicy},
utils::{
filter_request_header, human_readable_byte_size, sanitize_sensitive_info, CONTENT_TYPE_JSON, ENCODING_DEFLATE,
ENCODING_GZIP,
@@ -42,6 +42,7 @@ use tokio::{
};
use tokio_util::io::StreamReader;
use url::Url;
use shared::utils::DEFAULT_USER_AGENT;
static PROXY_DIAGNOSTICS_ONCE: Once = Once::new();
@@ -533,13 +534,13 @@ pub async fn get_input_epg_content_as_file(
client: &reqwest::Client,
input: &ConfigInput,
headers: Option<&HeaderMap>,
working_dir: &str,
storage_dir: &str,
url_str: &str,
persist_filepath: &Path,
) -> Result<PathBuf, TuliproxError> {
debug_if_enabled!(
"getting input epg content working_dir: {}, url: {}",
working_dir,
"getting input epg content storage_dir: {}, url: {}",
storage_dir,
sanitize_sensitive_info(url_str)
);
if url_str.parse::<url::Url>().is_ok() {
@@ -556,7 +557,7 @@ pub async fn get_input_epg_content_as_file(
}
}
} else {
let result = match get_file_path(working_dir, Some(PathBuf::from(url_str))) {
let result = match get_file_path(storage_dir, Some(PathBuf::from(url_str))) {
Some(filepath) => {
if filepath.exists() {
if let Err(e) = tokio::fs::copy(&filepath, persist_filepath).await {
@@ -590,12 +591,12 @@ pub async fn get_input_text_content(
app_state: &Arc<AppState>,
client: &reqwest::Client,
input: &InputSource,
working_dir: &str,
storage_dir: &str,
persist_filepath: Option<PathBuf>,
) -> Result<String, TuliproxError> {
debug_if_enabled!(
"getting input text content working_dir: {}, url: {}",
working_dir,
"getting input text content storage_dir: {}, url: {}",
storage_dir,
sanitize_sensitive_info(&input.url)
);
@@ -612,7 +613,7 @@ pub async fn get_input_text_content(
}
}
} else {
let result = match get_file_path(working_dir, Some(PathBuf::from(&input.url))) {
let result = match get_file_path(storage_dir, Some(PathBuf::from(&input.url))) {
Some(filepath) => {
if filepath.exists() {
if let Some(persist_file_value) = persist_filepath {
@@ -650,12 +651,12 @@ pub async fn get_input_text_content_as_stream(
app_config: &Arc<AppConfig>,
client: &reqwest::Client,
input: &InputSource,
working_dir: &str,
storage_dir: &str,
persist_filepath: Option<PathBuf>,
) -> Result<DynReader, TuliproxError> {
debug_if_enabled!(
"getting input text content working_dir: {}, url: {}",
working_dir,
"getting input text content storage_dir: {}, url: {}",
storage_dir,
sanitize_sensitive_info(&input.url)
);
@@ -672,7 +673,7 @@ pub async fn get_input_text_content_as_stream(
}
}
} else {
let result = match get_file_path(working_dir, Some(PathBuf::from(&input.url))) {
let result = match get_file_path(storage_dir, Some(PathBuf::from(&input.url))) {
Some(filepath) => {
if filepath.exists() {
match get_local_file_content_as_stream(&filepath).await {
@@ -1506,7 +1507,7 @@ mod tests {
};
use crate::{
model::{AppConfig, Config, ConfigProvider, ResourceRetryConfig, ReverseProxyConfig, SourcesConfig},
utils::FileLockManager,
utils::{FileLockManager, DEFAULT_USER_AGENT}
};
use arc_swap::{ArcSwap, ArcSwapOption};
use shared::model::{
@@ -1536,7 +1537,9 @@ mod tests {
api_proxy: Arc::new(ArcSwapOption::default()),
file_locks: Arc::new(FileLockManager::default()),
paths: Arc::new(ArcSwap::from_pointee(ConfigPaths {
home_path: String::new(),
config_path: String::new(),
storage_path: String::new(),
config_file_path: String::new(),
sources_file_path: String::new(),
mapping_file_path: None,
@@ -1631,7 +1634,7 @@ mod tests {
#[test]
fn test_get_request_headers_prioritization() {
use super::{get_request_headers, DEFAULT_USER_AGENT};
use super::{get_request_headers};
use axum::http::header::USER_AGENT;
// Case 1: No headers provided -> Default UA
+14 -10
View File
@@ -14,7 +14,7 @@ use chrono::{DateTime, Utc};
use log::{error, info, warn};
use shared::error::TuliproxError;
use shared::model::{PlaylistEntry, PlaylistGroup, ProxyUserStatus, SeriesStreamProperties,
StreamProperties, VideoStreamProperties, XtreamCluster, XtreamPlaylistItem,
StreamProperties, VideoStreamProperties, InputType, XtreamCluster, XtreamPlaylistItem,
XtreamSeriesInfo, XtreamVideoInfo, XtreamVideoInfoDoc};
use shared::utils::{extract_extension_from_url, get_i64_from_serde_value, get_string_from_serde_value, sanitize_sensitive_info, Internable, PROVIDER_SCHEME_PREFIX};
use std::collections::HashMap;
@@ -93,8 +93,8 @@ pub async fn get_xtream_stream_info(client: &reqwest::Client,
match cluster {
XtreamCluster::Live => {}
XtreamCluster::Video => {
let working_dir = &app_config.config.load().working_dir;
if let Ok(storage_path) = get_input_storage_path(&input.name, working_dir).await {
let storage_dir = &app_config.config.load().storage_dir;
if let Ok(storage_path) = get_input_storage_path(&input.name, storage_dir).await {
match serde_json::from_str::<XtreamVideoInfo>(&content) {
Ok(info) => {
// parse downloaded info into StreamProperties
@@ -126,7 +126,7 @@ pub async fn get_xtream_stream_info(client: &reqwest::Client,
}
}
XtreamCluster::Series => {
let working_dir = &app_config.config.load().working_dir;
let storage_dir = &app_config.config.load().storage_dir;
let group = pli.get_group();
let series_name = pli.get_name();
@@ -135,7 +135,7 @@ pub async fn get_xtream_stream_info(client: &reqwest::Client,
// parse series info
let series_stream_props = SeriesStreamProperties::from_info(&info, pli);
if let Ok(storage_path) = get_input_storage_path(&input.name, working_dir).await {
if let Ok(storage_path) = get_input_storage_path(&input.name, storage_dir).await {
// update input db
if let Err(err) = persists_input_series_info(app_config, &storage_path, cluster, &input.name, provider_id, &series_stream_props).await {
error!("Failed to persist series info for input {}: {err}", &input.name);
@@ -387,7 +387,11 @@ pub async fn download_xtream_playlist(app_config: &Arc<AppConfig>, client: &reqw
let skip_cluster = get_skip_cluster(input);
let cfg = app_config.config.load();
let working_dir = &cfg.working_dir;
let storage_dir = &cfg.storage_dir;
// Staged input type is the effective type during download.
let effective_type = input.get_download_input_type();
let use_disk_based_processing =
cfg.disk_based_processing && matches!(effective_type, InputType::Xtream | InputType::XtreamBatch);
let mut errors = vec![];
for (xtream_cluster, category, stream) in &ACTIONS {
@@ -396,16 +400,16 @@ pub async fn download_xtream_playlist(app_config: &Arc<AppConfig>, client: &reqw
let input_source_category = input_source.with_url(concat_string!(&base_url, "&action=", category));
let input_source_stream = input_source.with_url(concat_string!(&base_url, "&action=", stream));
let category_file_path = crate::utils::prepare_file_path(input.persist.as_deref(),
working_dir, concat_string!(category, "_").as_str());
storage_dir, concat_string!(category, "_").as_str());
let stream_file_path = crate::utils::prepare_file_path(input.persist.as_deref(),
working_dir, concat_string!(stream, "_").as_str());
storage_dir, concat_string!(stream, "_").as_str());
match futures::join!(
request::get_input_json_content_as_stream(app_config, client, &input_source_category, category_file_path),
request::get_input_json_content_as_stream(app_config, client, &input_source_stream, stream_file_path)
) {
(Ok(category_content), Ok(stream_content)) => {
if cfg.disk_based_processing {
if use_disk_based_processing {
// trace!("Using disk input playlist optimization for cluster {}", xtream_cluster);
if let Err(err) = persist_input_xtream_playlist_cluster_to_disk(app_config, input, *xtream_cluster, category_content, stream_content).await {
error!("persist_input_xtream_playlist_cluster_to_disk failed: {err}");
@@ -443,7 +447,7 @@ pub async fn download_xtream_playlist(app_config: &Arc<AppConfig>, client: &reqw
plg.id = grp_id;
}
(playlist_groups, errors, cfg.disk_based_processing)
(playlist_groups, errors, use_disk_based_processing)
}
async fn check_alias_user_state(app_config: &Arc<AppConfig>, client: &reqwest::Client, input: &ConfigInput) {
+1 -2
View File
@@ -3,8 +3,7 @@ use shared::error::TuliproxError;
use reqwest::header::{HeaderMap, HeaderValue};
use log::{debug, info};
use shared::error::{info_err};
use shared::model::DEFAULT_USER_AGENT;
use shared::utils::{trim_last_slash, TRAKT_API_KEY};
use shared::utils::{trim_last_slash, DEFAULT_USER_AGENT, TRAKT_API_KEY};
use super::errors::{handle_trakt_api_error};
pub struct TraktClient {
+4 -4
View File
@@ -286,7 +286,7 @@
"NAME": "Human readable name for this entry. Must be unique.",
"PASSWORD": "Password or credential used for this account or service.",
"PRIORITY": "The order of processing. Lower numbers mean higher priority (e.g. -1 is higher than 0).",
"URL": "Target URL for this input. Supports provider:// scheme for failover.",
"URL": "Input source URL. Supported schemes:\n\n```http(s)://\nfile://\nprovider://<provider>/...\nbatch://...(auto-switches m3u/xtream to batch types).```\n\nFor m3u_batch/xtream_batch only local CSV sources are allowed (batch://...).",
"USERNAME": "Login username for this account or service."
},
"CONFIG_INPUT_ALIAS": {
@@ -425,7 +425,7 @@
"UPDATE_ON_BOOT": "If enabled, Tuliprox triggers a full playlist update immediately upon startup. Helpful during initial setup, but usually set to false for stable operations.",
"USER_ACCESS_CONTROL": "If enabled, the attributes (if set)\n\n- expiration date,\n- status and\n- max_connections\n\nare checked to permit or deny access for api users.",
"USER_CONFIG_DIR": "Directory path where user-specific overrides and configuration files are stored.",
"WORKING_DIR": "The primary directory where Tuliprox stores its data files and databases."
"STORAGE_DIR": "The primary directory where Tuliprox stores its data files and databases."
},
"MESSAGING_CONFIG": {
"NOTIFY_ON": "Select events (e.g. 'info', 'stats', 'error') that trigger notifications for this specific type.",
@@ -544,7 +544,7 @@
"STAGED_INPUT": {
"HEADERS": "Custom HTTP headers sent when retrieving this staged configuration.",
"PASSWORD": "Password or credential used for this account or service.",
"URL": "Target URL for this setting.",
"URL": "Source URL for the staged input. Use http(s)://, file:// or provider://<provider>/... . Batch URLs are not supported for staged inputs.",
"USERNAME": "Login username for this account or service."
},
"STREAM_BUFFER_CONFIG": {
@@ -1138,7 +1138,7 @@
"WEB_UI_CONFIG": "Web-Ui",
"WEB_UI_PATH": "Path prefix",
"WELCOME": "Welcome",
"WORKING_DIR": "Working dir",
"STORAGE_DIR": "Storage dir",
"XC": "xc",
"XTREAM": "xtream",
"XTREAM_BATCH": "xc batch",
+4
View File
@@ -21,7 +21,9 @@
&__list,
&__text {
display: flex;
flex-flow: column;
gap: var(--gap-default);
}
&__value {
@@ -40,6 +42,8 @@
border-radius: var(--border-radius);
box-sizing: border-box;
white-space: nowrap;
overflow-x: auto;
scrollbar-width: none;
}
&__bool {
@@ -137,6 +137,14 @@
cursor: pointer;
}
.tp__form-list {
display: flex;
flex-flow: column;
gap: var(--gap-default);
box-sizing: border-box;
overflow: hidden;
}
.tp__form-list__item-arrow-btn {
svg {
transform: scale(2);
@@ -92,4 +92,8 @@ $form-width: 520px;
.tp__source-editor-form.disabled {
pointer-events: none;
opacity: 0.3;
}
.tp__input-form__body {
width: 100%;
}
@@ -184,9 +184,13 @@ pub fn ApiConfigView() -> Html {
let deps = (api_config, *config_view_ctx.edit_mode);
use_effect_with(deps, move |(cfg, _mode)| {
if let Some(api) = cfg {
form_state_api_config.dispatch(ApiConfigFormAction::SetAll(api.clone()));
let mut next_api = api.clone();
next_api.prepare();
form_state_api_config.dispatch(ApiConfigFormAction::SetAll(next_api));
} else {
form_state_api_config.dispatch(ApiConfigFormAction::SetAll(ConfigApiDto::default()));
let mut defaults = ConfigApiDto::default();
defaults.prepare();
form_state_api_config.dispatch(ApiConfigFormAction::SetAll(defaults));
}
|| ()
});
@@ -421,11 +425,13 @@ pub fn ApiConfigView() -> Html {
let render_view_mode_api_config = || {
if let Some(config) = &config_ctx.config {
let mut api_config = config.config.api.clone();
api_config.prepare();
html! {
<Card class="tp__api-config-card">
{ config_field!(config.config.api, translate.t(LABEL_HOST), host) }
{ config_field!(config.config.api, translate.t(LABEL_PORT), port) }
{ config_field!(config.config.api, translate.t(LABEL_WEB_ROOT), web_root) }
{ config_field!(api_config, translate.t(LABEL_HOST), host) }
{ config_field!(api_config, translate.t(LABEL_PORT), port) }
{ config_field!(api_config, translate.t(LABEL_WEB_ROOT), web_root) }
</Card>
}
} else {
@@ -474,7 +474,7 @@ pub fn ConfigView() -> Html {
{ html_if!(*edit_mode || setup_mode, {
<div class="tp__config-view__toolbar tp__form-page__toolbar">
<TextButton class="primary" name="save_config"
<TextButton class="secondary" name="save_config"
icon="Save"
title={ if setup_mode { translate.t(LABEL_SETUP_FINISH) } else { translate.t(LABEL_SAVE) }}
onclick={handle_save_config}></TextButton>
@@ -7,10 +7,10 @@ use crate::{
context::ConfigContext,
},
config_field, config_field_bool, config_field_optional, edit_field_bool, edit_field_number,
edit_field_number_option_u32, edit_field_text, edit_field_text_option, generate_form_reducer,
edit_field_number_option_u32, edit_field_text_option, generate_form_reducer,
i18n::use_translation,
};
use shared::model::MainConfigDto;
use shared::model::{ConfigDto, MainConfigDto};
use yew::prelude::*;
const LABEL_UPDATE_ON_BOOT: &str = "LABEL.UPDATE_ON_BOOT";
@@ -18,7 +18,7 @@ const LABEL_CONFIG_HOT_RELOAD: &str = "LABEL.CONFIG_HOT_RELOAD";
const LABEL_USER_ACCESS_CONTROL: &str = "LABEL.USER_ACCESS_CONTROL";
const LABEL_PROCESS_PARALLEL: &str = "LABEL.PROCESS_PARALLEL";
const LABEL_DISK_BASED_PROCESSING: &str = "LABEL.DISK_BASED_PROCESSING";
const LABEL_WORKING_DIR: &str = "LABEL.WORKING_DIR";
const LABEL_STORAGE_DIR: &str = "LABEL.STORAGE_DIR";
const LABEL_DEFAULT_USER_AGENT: &str = "LABEL.DEFAULT_USER_AGENT";
const LABEL_MAPPING_PATH: &str = "LABEL.MAPPING_PATH";
const LABEL_TEMPLATE_PATH: &str = "LABEL.TEMPLATE_PATH";
@@ -39,7 +39,7 @@ generate_form_reducer!(
AcceptInsecureSslCertificates => accept_insecure_ssl_certificates: bool,
ProcessParallel => process_parallel: bool,
DiskBasedProcessing => disk_based_processing: bool,
WorkingDir => working_dir: String,
StorageDir => storage_dir: Option<String>,
DefaultUserAgent => default_user_agent: Option<String>,
MappingPath => mapping_path: Option<String>,
TemplatePath => template_path: Option<String>,
@@ -73,8 +73,16 @@ pub fn MainConfigView() -> Html {
let config = config_ctx.config.as_ref().map(|c| c.config.clone());
use_effect_with((config, config_view_ctx.edit_mode.clone()), move |(cfg, _mode)| {
if let Some(main) = cfg {
let main_config = MainConfigDto::from(main);
form_state.dispatch(MainConfigFormAction::SetAll(main_config.clone()));
let mut prepared_main: ConfigDto = main.clone();
match prepared_main.prepare(false) {
Ok(()) => {
let main_config = MainConfigDto::from(&prepared_main);
form_state.dispatch(MainConfigFormAction::SetAll(main_config.clone()));
}
Err(err) => {
log::error!("Failed to prepare main config for view state: {err}");
}
}
} else {
form_state.dispatch(MainConfigFormAction::SetAll(MainConfigDto::default()));
}
@@ -91,7 +99,7 @@ pub fn MainConfigView() -> Html {
{ config_field_bool!(form_state.form, translate.t(LABEL_ACCEPT_INSECURE_SSL_CERTIFICATES), accept_insecure_ssl_certificates) }
{ config_field_bool!(form_state.form, translate.t(LABEL_PROCESS_PARALLEL), process_parallel) }
{ config_field_bool!(form_state.form, translate.t(LABEL_DISK_BASED_PROCESSING), disk_based_processing) }
{ config_field!(form_state.form, translate.t(LABEL_WORKING_DIR), working_dir) }
{ config_field_optional!(form_state.form, translate.t(LABEL_STORAGE_DIR), storage_dir) }
{ config_field_optional!(form_state.form, translate.t(LABEL_DEFAULT_USER_AGENT), default_user_agent) }
{ config_field_optional!(form_state.form, translate.t(LABEL_MAPPING_PATH), mapping_path) }
{ config_field_optional!(form_state.form, translate.t(LABEL_TEMPLATE_PATH), template_path) }
@@ -113,7 +121,7 @@ pub fn MainConfigView() -> Html {
{ edit_field_bool!(form_state, translate.t(LABEL_ACCEPT_INSECURE_SSL_CERTIFICATES), accept_insecure_ssl_certificates, MainConfigFormAction::AcceptInsecureSslCertificates) }
{ edit_field_bool!(form_state, translate.t(LABEL_PROCESS_PARALLEL), process_parallel, MainConfigFormAction::ProcessParallel) }
{ edit_field_bool!(form_state, translate.t(LABEL_DISK_BASED_PROCESSING), disk_based_processing, MainConfigFormAction::DiskBasedProcessing) }
{ edit_field_text!(form_state, translate.t(LABEL_WORKING_DIR), working_dir, MainConfigFormAction::WorkingDir) }
{ edit_field_text_option!(form_state, translate.t(LABEL_STORAGE_DIR), storage_dir, MainConfigFormAction::StorageDir) }
{ edit_field_text_option!(form_state, translate.t(LABEL_DEFAULT_USER_AGENT), default_user_agent, MainConfigFormAction::DefaultUserAgent) }
{ edit_field_text_option!(form_state, translate.t(LABEL_MAPPING_PATH), mapping_path, MainConfigFormAction::MappingPath) }
{ edit_field_text_option!(form_state, translate.t(LABEL_TEMPLATE_PATH), template_path, MainConfigFormAction::TemplatePath) }
@@ -70,7 +70,7 @@ generate_form_reducer!(
fields {
Enabled => enabled: bool,
Size => size: Option<String>,
Dir => dir: Option<String>,
Dir => directory: Option<String>,
}
);
@@ -389,7 +389,7 @@ pub fn ReverseProxyConfigView() -> Html {
<h1>{translate.t(LABEL_CACHE)}</h1>
{ config_field_bool!(cache_state.form, translate.t(LABEL_ENABLED), enabled) }
{ config_field_optional!(cache_state.form, translate.t(LABEL_SIZE), size) }
{ config_field_optional!(cache_state.form, translate.t(LABEL_DIRECTORY), dir) }
{ config_field_optional!(cache_state.form, translate.t(LABEL_DIRECTORY), directory) }
</Card>
}
};
@@ -543,7 +543,7 @@ pub fn ReverseProxyConfigView() -> Html {
<h1>{translate.t(LABEL_CACHE)}</h1>
{ edit_field_bool!(cache_state, translate.t(LABEL_ENABLED), enabled, CacheConfigFormAction::Enabled) }
{ edit_field_text_option!(cache_state, translate.t(LABEL_SIZE), size, CacheConfigFormAction::Size) }
{ edit_field_text_option!(cache_state, translate.t(LABEL_DIRECTORY), dir, CacheConfigFormAction::Dir) }
{ edit_field_text_option!(cache_state, translate.t(LABEL_DIRECTORY), directory, CacheConfigFormAction::Dir) }
</Card>
}
};
@@ -188,6 +188,7 @@ pub fn SchedulesConfigView() -> Html {
<td>{ match entry.task_type {
ScheduleTaskType::PlaylistUpdate => translate.t("LABEL.PLAYLIST_UPDATE"),
ScheduleTaskType::LibraryScan => translate.t("LABEL.LIBRARY"),
ScheduleTaskType::GeoIpUpdate => translate.t("LABEL.GEOIP"),
}}</td>
<td>
<div class="tp__config-view__tags">
@@ -246,6 +247,11 @@ pub fn SchedulesConfigView() -> Html {
label: html! { translate.t("LABEL.LIBRARY") },
selected: *selected_type == ScheduleTaskType::LibraryScan,
},
DropDownOption {
id: "GeoIpUpdate".to_string(),
label: html! { translate.t("LABEL.GEOIP") },
selected: *selected_type == ScheduleTaskType::GeoIpUpdate,
},
]);
html! {
@@ -260,6 +266,8 @@ pub fn SchedulesConfigView() -> Html {
set_selected_type.set(ScheduleTaskType::PlaylistUpdate);
} else if option == "LibraryScan" {
set_selected_type.set(ScheduleTaskType::LibraryScan);
} else if option == "GeoIpUpdate" {
set_selected_type.set(ScheduleTaskType::GeoIpUpdate);
}
})}
options={types}
@@ -179,6 +179,14 @@ generate_form_reducer!(
}
);
fn apply_parsed_input_type(staged_input_state: &UseReducerHandle<StagedInputDtoFormState>, selected: Option<&str>) {
let input_type =
selected.and_then(|value| value.parse::<InputType>().ok()).unwrap_or(staged_input_state.form.input_type);
if staged_input_state.form.input_type != input_type {
staged_input_state.dispatch(StagedInputFormAction::InputType(input_type));
}
}
#[derive(Properties, PartialEq, Clone)]
pub struct ConfigInputViewProps {
#[prop_or_default]
@@ -631,16 +639,18 @@ pub fn ConfigInputView(props: &ConfigInputViewProps) -> Html {
multi_select={false}
on_select={Callback::from(move |(_, selections):(String, DropDownSelection)| {
match selections {
DropDownSelection::Empty => {
staged_input_state_2.dispatch(StagedInputFormAction::InputType(InputType::Xtream));
}
DropDownSelection::Empty => {}
DropDownSelection::Single(option) => {
staged_input_state_2.dispatch(StagedInputFormAction::InputType(option.parse::<InputType>().unwrap_or(InputType::Xtream)));
apply_parsed_input_type(
&staged_input_state_2,
Some(option.as_str()),
);
}
DropDownSelection::Multi(options) => {
if let Some(first) = options.first() {
staged_input_state_2.dispatch(StagedInputFormAction::InputType(first.parse::<InputType>().unwrap_or(InputType::Xtream)));
}
apply_parsed_input_type(
&staged_input_state_2,
options.first().map(String::as_str),
);
}
}
})}
@@ -896,7 +906,7 @@ pub fn ConfigInputView(props: &ConfigInputViewProps) -> Html {
})
}
</div>
<TextButton
<TextButton
class="primary"
name="add_epg_source"
icon="Add"
@@ -141,7 +141,6 @@ fn has_meaningful_dns_config(dns: &ProviderDnsDto) -> bool {
|| dns.schemes.as_ref().is_some_and(|schemes| !schemes.is_empty())
|| dns.keep_vhost
|| dns.overrides.as_ref().is_some_and(|overrides| !overrides.is_empty())
|| dns.resolved.as_ref().is_some_and(|resolved| !resolved.is_empty())
|| dns.on_resolve_error != OnResolveErrorPolicy::default()
|| dns.on_connect_error != OnConnectErrorPolicy::default()
}
+14 -24
View File
@@ -802,33 +802,23 @@ impl<'a> MapperContext<'a> {
BuiltInFunction::ToNumber
| BuiltInFunction::Template
| BuiltInFunction::First
| BuiltInFunction::AddFavourite => {
if args.len() > 1 {
return info_err_res!(
"Function accepts only one argument {:?}, {} given",
name,
args.len()
);
}
| BuiltInFunction::AddFavourite
if args.len() > 1 =>
{
return info_err_res!("Function accepts only one argument {:?}, {} given", name, args.len());
}
BuiltInFunction::Split => {
if args.len() != 2 {
return info_err_res!("Function accepts two arguments {:?}, {} given", name, args.len());
}
BuiltInFunction::Split if args.len() != 2 => {
return info_err_res!("Function accepts two arguments {:?}, {} given", name, args.len());
}
BuiltInFunction::Replace => {
if args.len() != 3 {
return info_err_res!("Function accepts three arguments {:?}, {} given", name, args.len());
}
BuiltInFunction::Replace if args.len() != 3 => {
return info_err_res!("Function accepts three arguments {:?}, {} given", name, args.len());
}
BuiltInFunction::Pad => {
if !(args.len() == 3 || args.len() == 4) {
return info_err_res!(
"Function accepts three or four arguments {:?}, {} given",
name,
args.len()
);
}
BuiltInFunction::Pad if !(args.len() == 3 || args.len() == 4) => {
return info_err_res!(
"Function accepts three or four arguments {:?}, {} given",
name,
args.len()
);
}
_ => {}
}
+45 -1
View File
@@ -1,8 +1,52 @@
use crate::utils::{get_default_web_root, is_blank_or_default_web_root};
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize, Default, PartialEq)]
#[serde(deny_unknown_fields)]
pub struct ConfigApiDto {
pub host: String,
pub port: u16,
#[serde(default, skip_serializing_if = "String::is_empty")]
#[serde(default, skip_serializing_if = "is_blank_or_default_web_root")]
pub web_root: String,
}
impl ConfigApiDto {
pub fn default_web_root() -> String { get_default_web_root() }
pub fn prepare(&mut self) {
if self.web_root.trim().is_empty() {
self.web_root = Self::default_web_root();
}
}
}
#[cfg(test)]
mod tests {
use super::ConfigApiDto;
#[test]
fn serialization_skips_default_web_root() {
let dto = ConfigApiDto { host: "0.0.0.0".to_string(), port: 8901, web_root: "./web".to_string() };
let serialized = serde_json::to_string(&dto).expect("api dto serialization should succeed");
assert!(!serialized.contains("\"web_root\""), "expected default web_root to be skipped, got: {serialized}");
}
#[test]
fn serialization_keeps_non_default_web_root() {
let dto = ConfigApiDto { host: "0.0.0.0".to_string(), port: 8901, web_root: "/srv/tuliprox/web".to_string() };
let serialized = serde_json::to_string(&dto).expect("api dto serialization should succeed");
assert!(
serialized.contains("\"web_root\""),
"expected non-default web_root to be persisted, got: {serialized}"
);
}
#[test]
fn prepare_sets_default_web_root_when_empty() {
let mut dto = ConfigApiDto { host: "0.0.0.0".to_string(), port: 8901, web_root: String::new() };
let expected_web_root = ConfigApiDto::default_web_root();
dto.prepare();
assert_eq!(dto.web_root, expected_web_root);
}
}
+200 -69
View File
@@ -6,55 +6,18 @@ use crate::{
WebUiConfigDto,
},
utils::{
default_connect_timeout_secs, default_supported_video_extensions, is_blank_optional_string,
is_default_connect_timeout_secs, is_false, CONFIG_PATH, MAPPING_FILE, TEMPLATE_FILE,
default_connect_timeout_secs, default_custom_stream_response_path, default_default_user_agent,
default_main_backup_dir, default_main_mapping_path, default_main_storage_dir, default_main_template_path,
default_main_user_config_dir, default_supported_video_extensions, is_blank_optional_string,
is_blank_or_default_backup_dir, is_blank_or_default_custom_stream_response_path,
is_blank_or_default_mapping_path, is_blank_or_default_storage_dir, is_blank_or_default_template_path,
is_blank_or_default_user_config_dir, is_default_connect_timeout_secs, is_false,
is_none_or_empty_metadata_update, is_none_or_empty_video, normalize_optional_config_file_path,
normalize_optional_dir, DEFAULT_BACKUP_DIR, DEFAULT_CUSTOM_STREAM_RESPONSE_PATH, DEFAULT_STORAGE_DIR,
DEFAULT_USER_CONFIG_DIR, MAPPING_FILE, TEMPLATE_FILE,
},
};
pub const DEFAULT_USER_AGENT: &str = "VLC/3.0.16 LibVLC/3.0.16";
fn default_default_user_agent() -> Option<String> { Some(DEFAULT_USER_AGENT.to_string()) }
fn default_main_mapping_path() -> Option<String> { Some(format!("./{CONFIG_PATH}/{MAPPING_FILE}")) }
fn default_main_template_path() -> Option<String> { Some(format!("./{CONFIG_PATH}/{TEMPLATE_FILE}")) }
fn is_default_config_file_path(value: &str, file_name: &str) -> bool {
let normalized = value.trim().replace('\\', "/");
let normalized = normalized.trim_start_matches("./");
normalized == file_name
|| normalized.rsplit_once('/').is_some_and(|(dir, file)| dir == CONFIG_PATH && file == file_name)
}
fn is_blank_or_default_mapping_path(path: &Option<String>) -> bool {
path.as_ref().is_none_or(|value| value.trim().is_empty() || is_default_config_file_path(value, MAPPING_FILE))
}
fn is_blank_or_default_template_path(path: &Option<String>) -> bool {
path.as_ref().is_none_or(|value| value.trim().is_empty() || is_default_config_file_path(value, TEMPLATE_FILE))
}
fn effective_main_mapping_path(path: Option<String>) -> Option<String> {
if is_blank_or_default_mapping_path(&path) {
default_main_mapping_path()
} else {
path
}
}
fn effective_main_template_path(path: Option<String>) -> Option<String> {
if is_blank_or_default_template_path(&path) {
default_main_template_path()
} else {
path
}
}
fn is_none_or_empty_video(video: &Option<VideoConfigDto>) -> bool {
video.as_ref().is_none_or(VideoConfigDto::is_empty)
}
fn is_none_or_empty_metadata_update(metadata_update: &Option<MetadataUpdateConfigDto>) -> bool {
metadata_update.as_ref().is_none_or(MetadataUpdateConfigDto::is_empty)
}
#[allow(clippy::struct_excessive_bools)]
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize, PartialEq)]
#[serde(deny_unknown_fields)]
@@ -62,18 +25,22 @@ pub struct ConfigDto {
#[serde(default, skip_serializing_if = "is_false")]
pub process_parallel: bool,
pub api: ConfigApiDto,
pub working_dir: String,
#[serde(default, alias = "working_dir", skip_serializing_if = "is_blank_or_default_storage_dir")]
pub storage_dir: Option<String>,
#[serde(default = "default_default_user_agent", skip_serializing_if = "is_blank_optional_string")]
pub default_user_agent: Option<String>,
#[serde(default, skip_serializing_if = "is_blank_optional_string")]
#[serde(default, skip_serializing_if = "is_blank_or_default_backup_dir")]
pub backup_dir: Option<String>,
#[serde(default, skip_serializing_if = "is_blank_optional_string")]
#[serde(default, skip_serializing_if = "is_blank_or_default_user_config_dir")]
pub user_config_dir: Option<String>,
#[serde(default, skip_serializing_if = "is_blank_or_default_mapping_path")]
pub mapping_path: Option<String>,
#[serde(default, skip_serializing_if = "is_blank_or_default_template_path")]
pub template_path: Option<String>,
#[serde(default, skip_serializing_if = "is_blank_optional_string")]
#[serde(
default = "default_custom_stream_response_path",
skip_serializing_if = "is_blank_or_default_custom_stream_response_path"
)]
pub custom_stream_response_path: Option<String>,
#[serde(default, skip_serializing_if = "is_none_or_empty_video")]
pub video: Option<VideoConfigDto>,
@@ -118,7 +85,7 @@ impl Default for ConfigDto {
Self {
process_parallel: false,
api: ConfigApiDto::default(),
working_dir: String::new(),
storage_dir: None,
default_user_agent: default_default_user_agent(),
backup_dir: None,
user_config_dir: None,
@@ -153,12 +120,13 @@ impl Default for ConfigDto {
pub struct MainConfigDto {
#[serde(default, skip_serializing_if = "is_false")]
pub process_parallel: bool,
pub working_dir: String,
#[serde(default = "default_main_storage_dir", skip_serializing_if = "is_blank_or_default_storage_dir")]
pub storage_dir: Option<String>,
#[serde(default = "default_default_user_agent", skip_serializing_if = "is_blank_optional_string")]
pub default_user_agent: Option<String>,
#[serde(default, skip_serializing_if = "is_blank_optional_string")]
#[serde(default = "default_main_backup_dir", skip_serializing_if = "is_blank_or_default_backup_dir")]
pub backup_dir: Option<String>,
#[serde(default, skip_serializing_if = "is_blank_optional_string")]
#[serde(default = "default_main_user_config_dir", skip_serializing_if = "is_blank_or_default_user_config_dir")]
pub user_config_dir: Option<String>,
#[serde(default = "default_main_mapping_path", skip_serializing_if = "is_blank_or_default_mapping_path")]
pub mapping_path: Option<String>,
@@ -187,10 +155,10 @@ impl Default for MainConfigDto {
MainConfigDto {
process_parallel: false,
disk_based_processing: false,
working_dir: String::new(),
storage_dir: default_main_storage_dir(),
default_user_agent: default_default_user_agent(),
backup_dir: None,
user_config_dir: None,
backup_dir: default_main_backup_dir(),
user_config_dir: default_main_user_config_dir(),
mapping_path: default_main_mapping_path(),
template_path: default_main_template_path(),
custom_stream_response_path: None,
@@ -209,12 +177,12 @@ impl From<&ConfigDto> for MainConfigDto {
Self {
process_parallel: config.process_parallel,
disk_based_processing: config.disk_based_processing,
working_dir: config.working_dir.clone(),
storage_dir: config.storage_dir.clone(),
default_user_agent: config.default_user_agent.clone(),
backup_dir: config.backup_dir.clone(),
user_config_dir: config.user_config_dir.clone(),
mapping_path: effective_main_mapping_path(config.mapping_path.clone()),
template_path: effective_main_template_path(config.template_path.clone()),
mapping_path: config.mapping_path.clone(),
template_path: config.template_path.clone(),
custom_stream_response_path: config.custom_stream_response_path.clone(),
user_access_control: config.user_access_control,
connect_timeout_secs: config.connect_timeout_secs,
@@ -249,9 +217,26 @@ pub struct HdHomeRunDeviceOverview {
impl ConfigDto {
pub fn prepare(&mut self, include_computed: bool) -> Result<(), TuliproxError> {
self.api.prepare();
if is_blank_optional_string(&self.default_user_agent) {
self.default_user_agent = default_default_user_agent();
}
if is_blank_or_default_storage_dir(&self.storage_dir) {
self.storage_dir = default_main_storage_dir();
}
if is_blank_or_default_backup_dir(&self.backup_dir) {
self.backup_dir = default_main_backup_dir();
}
if is_blank_or_default_user_config_dir(&self.user_config_dir) {
self.user_config_dir = default_main_user_config_dir();
}
if is_blank_or_default_mapping_path(&self.mapping_path) {
self.mapping_path = default_main_mapping_path();
}
if is_blank_or_default_template_path(&self.template_path) {
self.template_path = default_main_template_path();
}
if let Some(mins) = self.sleep_timer_mins {
if mins == 0 {
@@ -268,7 +253,7 @@ impl ConfigDto {
self.prepare_metadata_update_config()?;
if let Some(reverse_proxy) = self.reverse_proxy.as_mut() {
reverse_proxy.prepare(&self.working_dir)?;
reverse_proxy.prepare(self.storage_dir.as_deref().unwrap_or_default())?;
}
if let Some(proxy) = &mut self.proxy {
proxy.prepare()?;
@@ -280,6 +265,9 @@ impl ConfigDto {
if let Some(messaging) = &mut self.messaging {
messaging.prepare(include_computed)?;
}
if let Some(library) = &mut self.library {
library.playlist.prepare();
}
Ok(())
}
@@ -361,13 +349,14 @@ impl ConfigDto {
pub fn update_from_main_config(&mut self, main_config: &MainConfigDto) {
self.process_parallel = main_config.process_parallel;
self.disk_based_processing = main_config.disk_based_processing;
self.working_dir = main_config.working_dir.clone();
self.storage_dir = normalize_optional_dir(&main_config.storage_dir, DEFAULT_STORAGE_DIR);
self.default_user_agent = main_config.default_user_agent.clone();
self.backup_dir = main_config.backup_dir.clone();
self.user_config_dir = main_config.user_config_dir.clone();
self.mapping_path = main_config.mapping_path.clone();
self.template_path = main_config.template_path.clone();
self.custom_stream_response_path = main_config.custom_stream_response_path.clone();
self.backup_dir = normalize_optional_dir(&main_config.backup_dir, DEFAULT_BACKUP_DIR);
self.user_config_dir = normalize_optional_dir(&main_config.user_config_dir, DEFAULT_USER_CONFIG_DIR);
self.mapping_path = normalize_optional_config_file_path(&main_config.mapping_path, MAPPING_FILE);
self.template_path = normalize_optional_config_file_path(&main_config.template_path, TEMPLATE_FILE);
self.custom_stream_response_path =
normalize_optional_dir(&main_config.custom_stream_response_path, DEFAULT_CUSTOM_STREAM_RESPONSE_PATH);
self.user_access_control = main_config.user_access_control;
self.connect_timeout_secs = main_config.connect_timeout_secs;
self.sleep_timer_mins = main_config.sleep_timer_mins;
@@ -386,7 +375,7 @@ impl ConfigDto {
#[cfg(test)]
mod tests {
use super::*;
use crate::utils::default_supported_video_extensions;
use crate::utils::{default_supported_video_extensions, CONFIG_PATH};
use serde_json::json;
#[test]
@@ -425,6 +414,104 @@ mod tests {
assert!(serialized.contains("\"video\""), "expected video field, got: {serialized}");
}
#[test]
fn serializing_skips_default_storage_backup_and_user_config_dirs() {
let cfg = ConfigDto {
storage_dir: Some(DEFAULT_STORAGE_DIR.to_string()),
backup_dir: Some(DEFAULT_BACKUP_DIR.to_string()),
user_config_dir: Some(DEFAULT_USER_CONFIG_DIR.to_string()),
..ConfigDto::default()
};
let serialized = serde_json::to_string(&cfg).expect("config serialization should succeed");
assert!(
!serialized.contains("\"storage_dir\""),
"expected no storage_dir field for default value, got: {serialized}"
);
assert!(
!serialized.contains("\"backup_dir\""),
"expected no backup_dir field for default value, got: {serialized}"
);
assert!(
!serialized.contains("\"user_config_dir\""),
"expected no user_config_dir field for default value, got: {serialized}"
);
}
#[test]
fn serializing_keeps_non_default_storage_and_backup_dirs() {
let cfg = ConfigDto {
storage_dir: Some("custom-storage".to_string()),
backup_dir: Some("custom-backup".to_string()),
user_config_dir: Some("custom-user-config".to_string()),
..ConfigDto::default()
};
let serialized = serde_json::to_string(&cfg).expect("config serialization should succeed");
assert!(
serialized.contains("\"storage_dir\""),
"expected storage_dir field for non-default value, got: {serialized}"
);
assert!(
serialized.contains("\"backup_dir\""),
"expected backup_dir field for non-default value, got: {serialized}"
);
assert!(
serialized.contains("\"user_config_dir\""),
"expected user_config_dir field for non-default value, got: {serialized}"
);
}
#[test]
fn main_config_from_applies_default_storage_backup_and_user_config_dirs() {
let mut cfg = ConfigDto::default();
cfg.prepare(false).expect("prepare should succeed");
let main = MainConfigDto::from(&cfg);
assert_eq!(main.storage_dir.as_deref(), Some(DEFAULT_STORAGE_DIR));
assert_eq!(main.backup_dir.as_deref(), Some(DEFAULT_BACKUP_DIR));
assert_eq!(main.user_config_dir.as_deref(), Some(DEFAULT_USER_CONFIG_DIR));
assert_eq!(main.mapping_path.as_deref(), Some(format!("./{CONFIG_PATH}/{MAPPING_FILE}").as_str()));
assert_eq!(main.template_path.as_deref(), Some(format!("./{CONFIG_PATH}/{TEMPLATE_FILE}").as_str()));
}
#[test]
fn update_from_main_config_omits_default_optional_paths() {
let mut cfg = ConfigDto::default();
let main = MainConfigDto {
storage_dir: Some(DEFAULT_STORAGE_DIR.to_string()),
backup_dir: Some(DEFAULT_BACKUP_DIR.to_string()),
user_config_dir: Some(DEFAULT_USER_CONFIG_DIR.to_string()),
mapping_path: Some(format!("./{CONFIG_PATH}/{MAPPING_FILE}")),
template_path: Some(format!("./{CONFIG_PATH}/{TEMPLATE_FILE}")),
..MainConfigDto::default()
};
cfg.update_from_main_config(&main);
assert!(cfg.storage_dir.is_none());
assert!(cfg.backup_dir.is_none());
assert!(cfg.user_config_dir.is_none());
assert!(cfg.mapping_path.is_none());
assert!(cfg.template_path.is_none());
}
#[test]
fn prepare_sets_default_optional_paths() {
let mut cfg = ConfigDto {
storage_dir: None,
backup_dir: None,
user_config_dir: None,
mapping_path: None,
template_path: None,
..ConfigDto::default()
};
cfg.prepare(false).expect("prepare should succeed");
assert_eq!(cfg.storage_dir.as_deref(), Some(DEFAULT_STORAGE_DIR));
assert_eq!(cfg.backup_dir.as_deref(), Some(DEFAULT_BACKUP_DIR));
assert_eq!(cfg.user_config_dir.as_deref(), Some(DEFAULT_USER_CONFIG_DIR));
assert_eq!(cfg.mapping_path.as_deref(), Some(format!("./{CONFIG_PATH}/{MAPPING_FILE}").as_str()));
assert_eq!(cfg.template_path.as_deref(), Some(format!("./{CONFIG_PATH}/{TEMPLATE_FILE}").as_str()));
}
#[test]
fn deserializing_rejects_legacy_video_ffprobe_fields() {
let raw = json!({
@@ -433,7 +520,7 @@ mod tests {
"port": 8901,
"web_root": "./web"
},
"working_dir": ".",
"storage_dir": ".",
"video": {
"extensions": ["mp4"],
"ffprobe_enabled": true
@@ -445,4 +532,48 @@ mod tests {
let err = result.unwrap_err().to_string();
assert!(err.contains("ffprobe_enabled"), "unexpected error text: {err}");
}
#[test]
fn deserializing_rejects_legacy_data_dir_alias() {
let raw = json!({
"api": {
"host": "127.0.0.1",
"port": 8901,
"web_root": "./web"
},
"data_dir": "."
});
let result: Result<ConfigDto, _> = serde_json::from_value(raw);
assert!(result.is_err(), "data_dir should not deserialize");
}
#[test]
fn deserializing_accepts_legacy_working_dir_alias() {
let raw = json!({
"api": {
"host": "127.0.0.1",
"port": 8901,
"web_root": "./web"
},
"working_dir": "."
});
let cfg: ConfigDto = serde_json::from_value(raw).expect("working_dir should deserialize as legacy alias");
assert_eq!(cfg.storage_dir.as_deref(), Some("."));
}
#[test]
fn deserializing_accepts_missing_storage_dir() {
let raw = json!({
"api": {
"host": "127.0.0.1",
"port": 8901,
"web_root": "./web"
}
});
let cfg: ConfigDto = serde_json::from_value(raw).expect("missing storage_dir should deserialize");
assert!(cfg.storage_dir.is_none());
}
}
+45 -17
View File
@@ -1,10 +1,11 @@
use crate::{
error::TuliproxError,
info_err_res,
utils::{is_blank_optional_str, is_blank_optional_string, parse_size_base_2},
utils::{
is_blank_optional_str, is_blank_optional_string, is_blank_or_default_cache_dir, parse_size_base_2,
DEFAULT_CACHE_DIR,
},
};
use path_clean::PathClean;
use std::path::PathBuf;
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize, Default, PartialEq)]
#[serde(deny_unknown_fields)]
@@ -13,27 +14,21 @@ pub struct CacheConfigDto {
pub enabled: bool,
#[serde(default, skip_serializing_if = "is_blank_optional_string")]
pub size: Option<String>,
#[serde(default, skip_serializing_if = "is_blank_optional_string")]
pub dir: Option<String>,
#[serde(default, alias = "dir", skip_serializing_if = "is_blank_or_default_cache_dir")]
pub directory: Option<String>,
}
impl CacheConfigDto {
pub fn is_empty(&self) -> bool {
!self.enabled && is_blank_optional_str(self.size.as_deref()) && is_blank_optional_str(self.dir.as_deref())
!self.enabled && is_blank_optional_str(self.size.as_deref()) && is_blank_optional_str(self.directory.as_deref())
}
pub(crate) fn prepare(&mut self, working_dir: &str) -> Result<(), TuliproxError> {
pub(crate) fn prepare(&mut self, _storage_dir: &str) -> Result<(), TuliproxError> {
if self.enabled {
let work_path = PathBuf::from(working_dir);
match self.dir.as_ref() {
None => self.dir = Some(work_path.join("cache").to_string_lossy().to_string()),
Some(work_dir) => {
let mut cache_dir = work_dir.to_string();
if PathBuf::from(&cache_dir).is_relative() {
cache_dir = work_path.join(&cache_dir).clean().to_string_lossy().to_string();
}
self.dir = Some(cache_dir.to_string());
}
if is_blank_or_default_cache_dir(&self.directory) {
self.directory = Some(DEFAULT_CACHE_DIR.to_string());
} else if let Some(dir) = self.directory.as_ref() {
self.directory = Some(dir.trim().to_string());
}
if let Some(val) = self.size.as_ref() {
@@ -50,3 +45,36 @@ impl CacheConfigDto {
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn prepare_sets_default_cache_dir_when_enabled_and_missing() {
let mut cache = CacheConfigDto { enabled: true, directory: None, size: None };
cache.prepare("storage").expect("prepare should succeed");
assert_eq!(cache.directory.as_deref(), Some(DEFAULT_CACHE_DIR));
}
#[test]
fn prepare_keeps_custom_cache_dir_when_enabled() {
let mut cache = CacheConfigDto { enabled: true, directory: Some("custom-cache".to_string()), size: None };
cache.prepare("storage").expect("prepare should succeed");
assert_eq!(cache.directory.as_deref(), Some("custom-cache"));
}
#[test]
fn serializing_skips_default_cache_dir() {
let cache = CacheConfigDto { enabled: true, directory: Some(DEFAULT_CACHE_DIR.to_string()), size: None };
let serialized = serde_json::to_string(&cache).expect("cache serialization should succeed");
assert!(!serialized.contains("\"directory\""), "expected no dir field for default value, got: {serialized}");
}
#[test]
fn serializing_keeps_non_default_cache_dir() {
let cache = CacheConfigDto { enabled: true, directory: Some("custom-cache".to_string()), size: None };
let serialized = serde_json::to_string(&cache).expect("cache serialization should succeed");
assert!(serialized.contains("\"directory\""), "expected dir field for custom value, got: {serialized}");
}
}
+13 -28
View File
@@ -10,11 +10,10 @@ use crate::{
deserialize_timestamp, get_credentials_from_url_str, get_trimmed_string, is_blank_optional_string,
is_default_probe_delay_secs, is_default_probe_live_interval, is_default_resolve_delay_secs, is_false, is_true,
is_zero_i16, is_zero_u16, parse_duration_seconds, parse_provider_scheme_url_parts, sanitize_sensitive_info,
serialize_option_vec_flow_map_items, trim_last_slash, Internable, PROVIDER_SCHEME_PREFIX,
serialize_option_vec_flow_map_items, trim_last_slash, Internable, BATCH_SCHEME_PREFIX, PROVIDER_SCHEME_PREFIX,
},
};
use enum_iterator::Sequence;
use indexmap::IndexMap;
use log::warn;
use std::{
collections::{HashMap, HashSet},
@@ -511,7 +510,7 @@ impl ConfigInputDto {
self.persist = get_trimmed_string(self.persist.as_deref());
if self.url.starts_with("file://") {
if self.url.starts_with(BATCH_SCHEME_PREFIX) {
match self.input_type {
InputType::M3u => {
self.input_type = InputType::M3uBatch;
@@ -524,7 +523,10 @@ impl ConfigInputDto {
} else if self.url.starts_with(PROVIDER_SCHEME_PREFIX)
&& matches!(self.input_type, InputType::M3uBatch | InputType::XtreamBatch)
{
return info_err_res!("input type {} does not support provider:// URLs for batch definitions; use a local CSV path or file:// URL", self.input_type);
return info_err_res!(
"input type {} does not support provider:// URLs for batch definitions; use batch:// URL",
self.input_type
);
}
check_provider_scheme_url!(self.url, provider_names);
@@ -654,7 +656,10 @@ impl ConfigInputDto {
if self.url.starts_with(PROVIDER_SCHEME_PREFIX)
&& matches!(self.input_type, InputType::M3uBatch | InputType::XtreamBatch)
{
return info_err_res!("input type {} does not support provider:// URLs for batch definitions; use a local CSV path or file:// URL", self.input_type);
return info_err_res!(
"input type {} does not support provider:// URLs for batch definitions; use batch:// URL",
self.input_type
);
}
Ok(())
}
@@ -791,8 +796,6 @@ pub struct ProviderDnsDto {
pub keep_vhost: bool,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub overrides: Option<HashMap<String, Vec<IpAddr>>>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub resolved: Option<IndexMap<String, Vec<IpAddr>>>,
#[serde(default, skip_serializing_if = "is_default_on_resolve_error")]
pub on_resolve_error: OnResolveErrorPolicy,
#[serde(default, skip_serializing_if = "is_default_on_connect_error")]
@@ -809,7 +812,6 @@ impl Default for ProviderDnsDto {
schemes: None,
keep_vhost: false,
overrides: None,
resolved: None,
on_resolve_error: OnResolveErrorPolicy::default(),
on_connect_error: OnConnectErrorPolicy::default(),
}
@@ -1010,29 +1012,12 @@ mod tests {
assert_eq!(overrides["example.com"].len(), 1);
}
#[test]
fn test_provider_dns_resolved_deserializes() {
let json = r#"{
"name":"p1",
"urls":["http://example.com"],
"dns":{
"enabled":true,
"resolved":{"example.com":["203.0.113.10"]}
}
}"#;
let dto: ConfigProviderDto = serde_json::from_str(json).expect("provider json should parse");
let dns = dto.dns.expect("dns should be present");
let resolved = dns.resolved.expect("resolved must be deserialized");
assert_eq!(resolved.get("example.com"), Some(&vec!["203.0.113.10".parse::<IpAddr>().expect("valid ip")]));
}
#[test]
fn prepare_switches_xtream_to_xtream_batch_when_alias_exists() {
let mut dto = ConfigInputDto {
name: "input_alias".intern(),
input_type: InputType::Xtream,
url: "file:///tmp/input_alias.csv".to_string(),
url: "batch:///tmp/input_alias.csv".to_string(),
aliases: Some(vec![ConfigInputAliasDto {
id: 1,
name: "alias_1".intern(),
@@ -1046,7 +1031,7 @@ mod tests {
};
dto.prepare_type().expect("prepare type should succeed");
dto.prepare(0, true, &HashSet::new()).expect("prepare should succeed and infer batch type");
dto.prepare(0, true, &HashSet::new()).expect("prepare should succeed and infer batch type from batch:// URL");
assert_eq!(dto.input_type, InputType::XtreamBatch);
}
@@ -1069,6 +1054,6 @@ mod tests {
};
let err = dto.prepare(0, true, &HashSet::new()).expect_err("prepare must reject provider:// for batch input");
assert!(err.to_string().contains("does not support provider:// URLs"));
assert!(err.to_string().contains("does not support provider:// URLs"), "Error: {err}");
}
}
+32 -2
View File
@@ -106,7 +106,7 @@ pub enum LibraryMetadataFormat {
Nfo,
}
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Default)]
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
#[serde(deny_unknown_fields)]
pub struct LibraryPlaylistConfigDto {
#[serde(default = "default_movie_category")]
@@ -119,7 +119,7 @@ impl LibraryPlaylistConfigDto {
pub fn is_empty(&self) -> bool {
self.movie_category == default_movie_category() && self.series_category == default_series_category()
}
pub fn clean(&mut self) {
pub fn prepare(&mut self) {
if self.movie_category.trim().is_empty() {
self.movie_category = default_movie_category();
}
@@ -127,10 +127,19 @@ impl LibraryPlaylistConfigDto {
self.series_category = default_series_category();
}
}
pub fn clean(&mut self) { self.prepare(); }
}
impl Default for LibraryPlaylistConfigDto {
fn default() -> Self {
Self { movie_category: default_movie_category(), series_category: default_series_category() }
}
}
impl LibraryConfigDto {
pub fn prepare(&mut self) -> Result<(), TuliproxError> {
self.playlist.prepare();
// Validate enabled state
if self.enabled && self.scan_directories.is_empty() {
return info_err_res!("Library enabled but no scan_directories configured");
@@ -146,3 +155,24 @@ impl LibraryConfigDto {
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::{default_movie_category, default_series_category, LibraryPlaylistConfigDto};
#[test]
fn playlist_default_uses_default_categories() {
let playlist = LibraryPlaylistConfigDto::default();
assert_eq!(playlist.movie_category, default_movie_category());
assert_eq!(playlist.series_category, default_series_category());
}
#[test]
fn playlist_prepare_sets_default_categories_when_empty() {
let mut playlist =
LibraryPlaylistConfigDto { movie_category: String::new(), series_category: " ".to_string() };
playlist.prepare();
assert_eq!(playlist.movie_category, default_movie_category());
assert_eq!(playlist.series_category, default_series_category());
}
}
+4
View File
@@ -2,7 +2,11 @@ use crate::utils::{is_blank_optional_string, is_empty_optional_vec};
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
pub struct ConfigPaths {
#[serde(default, skip_serializing_if = "String::is_empty")]
pub home_path: String,
pub config_path: String,
#[serde(default, skip_serializing_if = "String::is_empty")]
pub storage_path: String,
pub config_file_path: String,
pub sources_file_path: String,
#[serde(default, skip_serializing_if = "is_blank_optional_string")]
+2 -2
View File
@@ -89,7 +89,7 @@ impl ReverseProxyConfigDto {
}
}
pub(crate) fn prepare(&mut self, working_dir: &str) -> Result<(), TuliproxError> {
pub(crate) fn prepare(&mut self, storage_dir: &str) -> Result<(), TuliproxError> {
self.rewrite_secret = self.rewrite_secret.trim().to_string();
if !self.resource_rewrite_disabled {
if self.rewrite_secret.is_empty() {
@@ -106,7 +106,7 @@ impl ReverseProxyConfigDto {
warn!("The cache is disabled because resource rewrite is disabled");
cache.enabled = false;
}
cache.prepare(working_dir)?;
cache.prepare(storage_dir)?;
}
if let Some(rate_limit) = self.rate_limit.as_mut() {
+88 -1
View File
@@ -3,6 +3,51 @@ pub enum ScheduleTaskType {
#[default]
PlaylistUpdate,
LibraryScan,
GeoIpUpdate,
}
const fn default_schedule_task_type() -> ScheduleTaskType { ScheduleTaskType::PlaylistUpdate }
fn parse_schedule_task_type(value: &str) -> Option<ScheduleTaskType> {
match value.trim().to_ascii_lowercase().as_str() {
"playlistupdate" | "playlist_update" | "playlist-update" | "update" => Some(ScheduleTaskType::PlaylistUpdate),
"libraryscan" | "library_scan" | "library-scan" | "scan" => Some(ScheduleTaskType::LibraryScan),
"geoipupdate" | "geo_ip_update" | "geoip_update" | "geoip-update" | "geoip" => {
Some(ScheduleTaskType::GeoIpUpdate)
}
_ => None,
}
}
fn deserialize_schedule_task_type<'de, D>(deserializer: D) -> Result<ScheduleTaskType, D::Error>
where
D: serde::Deserializer<'de>,
{
use serde::Deserialize as _;
#[derive(serde::Deserialize)]
#[serde(untagged)]
enum CompatTaskType {
Enum(ScheduleTaskType),
String(String),
}
let value = Option::<CompatTaskType>::deserialize(deserializer)?;
match value {
None => Ok(default_schedule_task_type()),
Some(CompatTaskType::Enum(task_type)) => Ok(task_type),
Some(CompatTaskType::String(raw)) => {
if raw.trim().is_empty() {
Ok(default_schedule_task_type())
} else if let Some(task_type) = parse_schedule_task_type(&raw) {
Ok(task_type)
} else {
Err(serde::de::Error::custom(format!(
"invalid schedule type '{raw}', expected one of: PlaylistUpdate, LibraryScan, GeoIpUpdate"
)))
}
}
}
}
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize, Default, PartialEq, Eq)]
@@ -10,8 +55,50 @@ pub enum ScheduleTaskType {
pub struct ScheduleConfigDto {
#[serde(default)]
pub schedule: String,
#[serde(default, rename = "type")]
#[serde(
default = "default_schedule_task_type",
rename = "type",
alias = "task_type",
deserialize_with = "deserialize_schedule_task_type"
)]
pub task_type: ScheduleTaskType,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub targets: Option<Vec<String>>,
}
#[cfg(test)]
mod tests {
use super::{ScheduleConfigDto, ScheduleTaskType};
#[test]
fn defaults_to_playlist_update_when_type_is_missing() {
let dto: ScheduleConfigDto = serde_saphyr::from_str("schedule: \"0 0 * * * * *\"\ntargets: [\"main\"]")
.expect("schedule dto should parse");
assert_eq!(dto.task_type, ScheduleTaskType::PlaylistUpdate);
}
#[test]
fn defaults_to_playlist_update_when_type_is_null_or_empty() {
let null_type: ScheduleConfigDto =
serde_saphyr::from_str("schedule: \"0 0 * * * * *\"\ntype: ~").expect("null type should parse");
let empty_type: ScheduleConfigDto =
serde_saphyr::from_str("schedule: \"0 0 * * * * *\"\ntype: \"\"").expect("empty type should parse");
assert_eq!(null_type.task_type, ScheduleTaskType::PlaylistUpdate);
assert_eq!(empty_type.task_type, ScheduleTaskType::PlaylistUpdate);
}
#[test]
fn accepts_legacy_task_type_aliases() {
let legacy: ScheduleConfigDto = serde_saphyr::from_str("schedule: \"0 0 * * * * *\"\ntask_type: scan")
.expect("legacy task_type alias should parse");
assert_eq!(legacy.task_type, ScheduleTaskType::LibraryScan);
}
#[test]
fn accepts_geoip_update_aliases() {
let legacy: ScheduleConfigDto =
serde_saphyr::from_str("schedule: \"0 0 * * * * *\"\ntype: geoip").expect("geoip alias should parse");
assert_eq!(legacy.task_type, ScheduleTaskType::GeoIpUpdate);
}
}
+3 -3
View File
@@ -1,6 +1,6 @@
use crate::{
model::DEFAULT_USER_AGENT,
utils::{default_as_true, default_trakt_fuzzy_threshold, is_true, TRAKT_API_KEY, TRAKT_API_URL, TRAKT_API_VERSION},
use crate::utils::{
default_as_true, default_trakt_fuzzy_threshold, is_true, DEFAULT_USER_AGENT, TRAKT_API_KEY, TRAKT_API_URL,
TRAKT_API_VERSION,
};
use serde::{Deserialize, Serialize};
use std::{fmt, str::FromStr};
+118 -7
View File
@@ -1,10 +1,10 @@
use crate::{
error::TuliproxError,
info_err_res,
model::DEFAULT_USER_AGENT,
utils::{
default_supported_video_extensions, is_blank_optional_str, is_blank_optional_string,
is_default_supported_video_extensions, is_false,
default_download_dir, default_episode_pattern, default_supported_video_extensions, is_blank_optional_str,
is_blank_optional_string, is_blank_or_default_download_dir, is_blank_or_default_episode_pattern,
is_default_supported_video_extensions, is_false, DEFAULT_USER_AGENT,
},
};
use std::{borrow::BorrowMut, collections::HashMap};
@@ -14,12 +14,12 @@ use std::{borrow::BorrowMut, collections::HashMap};
pub struct VideoDownloadConfigDto {
#[serde(default, skip_serializing_if = "HashMap::is_empty")]
pub headers: HashMap<String, String>,
#[serde(default, skip_serializing_if = "is_blank_optional_string")]
#[serde(default = "default_download_dir", skip_serializing_if = "is_blank_or_default_download_dir")]
pub directory: Option<String>,
#[serde(default, skip_serializing_if = "is_false")]
pub organize_into_directories: bool,
// TODO use ptt
#[serde(default, skip_serializing_if = "is_blank_optional_string")]
#[serde(default = "default_episode_pattern", skip_serializing_if = "is_blank_or_default_episode_pattern")]
pub episode_pattern: Option<String>,
}
@@ -27,8 +27,8 @@ impl VideoDownloadConfigDto {
pub fn is_empty(&self) -> bool {
!self.organize_into_directories
&& self.headers.is_empty()
&& is_blank_optional_str(self.directory.as_deref())
&& is_blank_optional_str(self.episode_pattern.as_deref())
&& is_blank_or_default_download_dir(&self.directory)
&& is_blank_or_default_episode_pattern(&self.episode_pattern)
}
}
@@ -69,11 +69,23 @@ impl VideoConfigDto {
match &mut self.download {
None => {}
Some(downl) => {
if is_blank_or_default_download_dir(&downl.directory) {
downl.directory = default_download_dir();
} else if let Some(directory) = downl.directory.as_ref() {
downl.directory = Some(directory.trim().to_string());
}
if downl.headers.is_empty() {
downl.headers.borrow_mut().insert("Accept".to_string(), "video/*".to_string());
downl.headers.borrow_mut().insert("User-Agent".to_string(), DEFAULT_USER_AGENT.to_string());
}
if is_blank_or_default_episode_pattern(&downl.episode_pattern) {
downl.episode_pattern = default_episode_pattern();
} else if let Some(episode_pattern) = downl.episode_pattern.as_ref() {
downl.episode_pattern = Some(episode_pattern.trim().to_string());
}
if let Some(episode_pattern) = &downl.episode_pattern {
if let Err(err) = crate::model::REGEX_CACHE.get_or_compile(episode_pattern) {
return info_err_res!("can't parse regex: {episode_pattern} {err}");
@@ -84,3 +96,102 @@ impl VideoConfigDto {
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::utils::DEFAULT_DOWNLOAD_DIR;
#[test]
fn prepare_sets_default_download_dir_when_missing() {
let mut video = VideoConfigDto {
extensions: Vec::new(),
download: Some(VideoDownloadConfigDto {
headers: HashMap::new(),
directory: None,
organize_into_directories: false,
episode_pattern: None,
}),
web_search: None,
};
video.prepare().expect("prepare should succeed");
let download = video.download.expect("download should exist");
assert_eq!(download.directory.as_deref(), Some(DEFAULT_DOWNLOAD_DIR));
}
#[test]
fn prepare_sets_default_episode_pattern_when_missing() {
let mut video = VideoConfigDto {
extensions: Vec::new(),
download: Some(VideoDownloadConfigDto {
headers: HashMap::new(),
directory: None,
organize_into_directories: false,
episode_pattern: None,
}),
web_search: None,
};
video.prepare().expect("prepare should succeed");
let download = video.download.expect("download should exist");
assert!(download.episode_pattern.is_some(), "expected default episode pattern to be set");
}
#[test]
fn prepare_keeps_custom_download_dir() {
let mut video = VideoConfigDto {
extensions: Vec::new(),
download: Some(VideoDownloadConfigDto {
headers: HashMap::new(),
directory: Some("custom-downloads".to_string()),
organize_into_directories: false,
episode_pattern: None,
}),
web_search: None,
};
video.prepare().expect("prepare should succeed");
let download = video.download.expect("download should exist");
assert_eq!(download.directory.as_deref(), Some("custom-downloads"));
}
#[test]
fn serializing_skips_default_download_dir() {
let download = VideoDownloadConfigDto {
headers: HashMap::new(),
directory: Some(DEFAULT_DOWNLOAD_DIR.to_string()),
organize_into_directories: false,
episode_pattern: None,
};
let serialized = serde_json::to_string(&download).expect("download serialization should succeed");
assert!(
!serialized.contains("\"directory\""),
"expected no directory field for default value, got: {serialized}"
);
}
#[test]
fn serializing_keeps_custom_download_dir() {
let download = VideoDownloadConfigDto {
headers: HashMap::new(),
directory: Some("custom-downloads".to_string()),
organize_into_directories: false,
episode_pattern: None,
};
let serialized = serde_json::to_string(&download).expect("download serialization should succeed");
assert!(serialized.contains("\"directory\""), "expected directory field for custom value, got: {serialized}");
}
#[test]
fn serializing_skips_default_episode_pattern() {
let download = VideoDownloadConfigDto {
headers: HashMap::new(),
directory: None,
organize_into_directories: false,
episode_pattern: default_episode_pattern(),
};
let serialized = serde_json::to_string(&download).expect("download serialization should succeed");
assert!(
!serialized.contains("\"episode_pattern\""),
"expected no episode_pattern field for default value, got: {serialized}"
);
}
}
+3 -3
View File
@@ -1,5 +1,5 @@
use crate::utils::{
default_as_true, default_token_ttl_mins, is_blank_optional_str, is_blank_optional_string,
default_as_true, default_token_ttl_mins, default_user_file_path, is_blank_or_default_user_file_path,
is_default_token_ttl_mins, is_true,
};
@@ -12,7 +12,7 @@ pub struct WebAuthConfigDto {
pub secret: String,
#[serde(default = "default_token_ttl_mins", skip_serializing_if = "is_default_token_ttl_mins")]
pub token_ttl_mins: u32,
#[serde(default, skip_serializing_if = "is_blank_optional_string")]
#[serde(default = "default_user_file_path", skip_serializing_if = "is_blank_or_default_user_file_path")]
pub userfile: Option<String>,
}
@@ -35,6 +35,6 @@ impl WebAuthConfigDto {
&& self.token_ttl_mins == empty.token_ttl_mins
&& self.issuer.trim().is_empty()
&& self.secret.trim().is_empty()
&& is_blank_optional_str(self.userfile.as_deref())
&& is_blank_or_default_user_file_path(&self.userfile)
}
}
+11 -1
View File
@@ -7,6 +7,14 @@ use std::{
pub const DEFAULT_HOME_ENV_VAR: &str = "TULIPROX_HOME";
pub const DEFAULT_WEB_ROOT_ENV_VAR: &str = "TULIPROX_WEB_ROOT";
pub const DEFAULT_USER_AGENT: &str = "VLC/3.0.16 LibVLC/3.0.16";
pub const DEFAULT_BACKUP_DIR: &str = "backup";
pub const DEFAULT_CACHE_DIR: &str = "cache";
pub const DEFAULT_STORAGE_TEMP_DIR: &str = "tmp";
pub const DEFAULT_USER_CONFIG_DIR: &str = "user_config";
pub const DEFAULT_DOWNLOAD_DIR: &str = "downloads";
pub const DEFAULT_CUSTOM_STREAM_RESPONSE_PATH: &str = "resources";
pub const DNS_RESOLVED_FILE: &str = "provider_dns_resolved.json";
pub const USER_FILE: &str = "user.txt";
pub const CONFIG_PATH: &str = "config";
pub const CONFIG_FILE: &str = "config.yml";
@@ -15,7 +23,7 @@ pub const MAPPING_FILE: &str = "mapping.yml";
pub const TEMPLATE_FILE: &str = "template.yml";
pub const API_PROXY_FILE: &str = "api-proxy.yml";
pub const DEFAULT_WEB_DIR: &str = "web";
pub const DEFAULT_WORKING_DIR: &str = "data";
pub const DEFAULT_STORAGE_DIR: &str = "data"; // TODO rename to storage and use data for config, storage, ...
pub const DEFAULT_PORT: u16 = 8901;
pub const ENCODING_GZIP: &str = "gzip";
pub const ENCODING_DEFLATE: &str = "deflate";
@@ -31,6 +39,8 @@ pub const HLS_EXT_FRAGMENT: &str = ".m3u8#";
pub const DASH_EXT_QUERY: &str = ".mpd?";
pub const DASH_EXT_FRAGMENT: &str = ".mpd#";
pub const DEFAULT_EPISODE_PATTERN: &str = r".*(?P<episode>[Ss]\d{1,2}(.*?)[Ee]\d{1,2}).*";
pub const FILENAME_TRIM_PATTERNS: &[char] = &['.', '-', '_'];
const SUPPORTED_RESPONSE_HEADERS: &[&str] = &[
+114 -1
View File
@@ -1,4 +1,11 @@
use crate::model::{ConfigTargetOptions, LibraryMetadataFormat, ProcessingOrder};
use crate::{
model::{ConfigTargetOptions, LibraryMetadataFormat, MetadataUpdateConfigDto, ProcessingOrder, VideoConfigDto},
utils::{
CONFIG_PATH, DEFAULT_BACKUP_DIR, DEFAULT_CACHE_DIR, DEFAULT_CUSTOM_STREAM_RESPONSE_PATH, DEFAULT_DOWNLOAD_DIR,
DEFAULT_EPISODE_PATTERN, DEFAULT_STORAGE_DIR, DEFAULT_USER_AGENT, DEFAULT_USER_CONFIG_DIR, DEFAULT_WEB_DIR,
MAPPING_FILE, TEMPLATE_FILE, USER_FILE,
},
};
use std::sync::Arc;
pub const fn is_zero_u16(v: &u16) -> bool { *v == 0 }
@@ -218,6 +225,112 @@ pub fn is_default_metadata_ffprobe_live_probe_size(v: &String) -> bool {
*v == default_metadata_ffprobe_live_probe_size()
}
pub fn get_default_web_root() -> String { DEFAULT_WEB_DIR.to_string() }
pub fn is_blank_or_default_web_root(value: &str) -> bool {
let normalized = value.trim().replace('\\', "/");
if normalized.is_empty() {
return true;
}
let normalized = normalized.trim_end_matches('/');
normalized.trim_start_matches("./") == DEFAULT_WEB_DIR
}
pub fn is_default_dir_path(value: &str, default_dir: &str) -> bool {
let normalized = value.trim().replace('\\', "/");
let normalized = normalized.trim_end_matches('/');
let normalized = normalized.trim_start_matches("./");
let default_dir = default_dir.trim().replace('\\', "/");
let default_dir = default_dir.trim_end_matches('/');
normalized == default_dir
}
pub fn is_blank_or_default_download_dir(path: &Option<String>) -> bool {
path.as_ref().is_none_or(|value| value.trim().is_empty() || is_default_dir_path(value, DEFAULT_DOWNLOAD_DIR))
}
pub fn default_download_dir() -> Option<String> { Some(DEFAULT_DOWNLOAD_DIR.to_string()) }
pub fn default_episode_pattern() -> Option<String> { Some(DEFAULT_EPISODE_PATTERN.to_string()) }
pub fn is_blank_or_default_episode_pattern(path: &Option<String>) -> bool {
path.as_ref().is_none_or(|value| value.trim().is_empty() || value.trim() == DEFAULT_EPISODE_PATTERN)
}
pub fn is_blank_or_default_cache_dir(path: &Option<String>) -> bool {
path.as_ref().is_none_or(|value| value.trim().is_empty() || is_default_dir_path(value, DEFAULT_CACHE_DIR))
}
pub fn default_default_user_agent() -> Option<String> { Some(DEFAULT_USER_AGENT.to_string()) }
pub fn default_main_storage_dir() -> Option<String> { Some(DEFAULT_STORAGE_DIR.to_string()) }
pub fn default_main_backup_dir() -> Option<String> { Some(DEFAULT_BACKUP_DIR.to_string()) }
pub fn default_main_user_config_dir() -> Option<String> { Some(DEFAULT_USER_CONFIG_DIR.to_string()) }
pub fn default_main_mapping_path() -> Option<String> { Some(format!("./{CONFIG_PATH}/{MAPPING_FILE}")) }
pub fn default_main_template_path() -> Option<String> { Some(format!("./{CONFIG_PATH}/{TEMPLATE_FILE}")) }
pub fn default_custom_stream_response_path() -> Option<String> { Some(DEFAULT_CUSTOM_STREAM_RESPONSE_PATH.to_string()) }
pub fn default_user_file_path() -> Option<String> { Some(format!("./{CONFIG_PATH}/{USER_FILE}")) }
fn is_default_config_file_path(value: &str, file_name: &str) -> bool {
let normalized = value.trim().replace('\\', "/");
let normalized = normalized.trim_start_matches("./");
normalized == file_name
|| normalized.rsplit_once('/').is_some_and(|(dir, file)| dir == CONFIG_PATH && file == file_name)
}
pub fn is_blank_or_default_custom_stream_response_path(path: &Option<String>) -> bool {
path.as_ref()
.is_none_or(|value| value.trim().is_empty() || is_default_dir_path(value, DEFAULT_CUSTOM_STREAM_RESPONSE_PATH))
}
pub fn is_blank_or_default_mapping_path(path: &Option<String>) -> bool {
path.as_ref().is_none_or(|value| value.trim().is_empty() || is_default_config_file_path(value, MAPPING_FILE))
}
pub fn is_blank_or_default_template_path(path: &Option<String>) -> bool {
path.as_ref().is_none_or(|value| value.trim().is_empty() || is_default_config_file_path(value, TEMPLATE_FILE))
}
pub fn is_blank_or_default_storage_dir(path: &Option<String>) -> bool {
path.as_ref().is_none_or(|value| value.trim().is_empty() || is_default_dir_path(value, DEFAULT_STORAGE_DIR))
}
pub fn is_blank_or_default_backup_dir(path: &Option<String>) -> bool {
path.as_ref().is_none_or(|value| value.trim().is_empty() || is_default_dir_path(value, DEFAULT_BACKUP_DIR))
}
pub fn is_blank_or_default_user_config_dir(path: &Option<String>) -> bool {
path.as_ref().is_none_or(|value| value.trim().is_empty() || is_default_dir_path(value, DEFAULT_USER_CONFIG_DIR))
}
pub fn is_blank_or_default_user_file_path(path: &Option<String>) -> bool {
path.as_ref().is_none_or(|value| value.trim().is_empty() || is_default_config_file_path(value, USER_FILE))
}
pub fn normalize_optional_dir(path: &Option<String>, default_dir: &str) -> Option<String> {
path.as_ref().and_then(|value| {
if value.trim().is_empty() || is_default_dir_path(value, default_dir) {
None
} else {
Some(value.clone())
}
})
}
pub fn normalize_optional_config_file_path(path: &Option<String>, default_file_name: &str) -> Option<String> {
path.as_ref().and_then(|value| {
if value.trim().is_empty() || is_default_config_file_path(value, default_file_name) {
None
} else {
Some(value.clone())
}
})
}
pub fn is_none_or_empty_video(video: &Option<VideoConfigDto>) -> bool {
video.as_ref().is_none_or(VideoConfigDto::is_empty)
}
pub fn is_none_or_empty_metadata_update(metadata_update: &Option<MetadataUpdateConfigDto>) -> bool {
metadata_update.as_ref().is_none_or(MetadataUpdateConfigDto::is_empty)
}
//////////////////////////////
// HDHomerun Device Defaults
//////////////////////////////
+1
View File
@@ -8,6 +8,7 @@ use std::{borrow::Cow, sync::atomic::Ordering};
use url::Url;
pub const PROVIDER_SCHEME_PREFIX: &str = "provider://";
pub const BATCH_SCHEME_PREFIX: &str = "batch://";
pub const CONTENT_TYPE_JSON: &str = "application/json";
pub const CONTENT_TYPE_CBOR: &str = "application/cbor";