euzu/tuliprox
1
0
Fork 0
mirror of https://github.com/euzu/tuliprox.git synced 2026-09-28 20:12:12 +02:00
Code Issues Packages Projects Releases 5 Wiki Activity

New options for downloading into folders over web-ui.

Browse Source
This commit is contained in:
euzu
2023-11-06 12:21:16 +01:00
parent 67703bae8d
commit 075ec974e8
5 changed files with 124 additions and 67 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+3
View File
@@ -4,6 +4,9 @@
- `xtream_skip_live_direct_source`
- `xtream_skip_video_direct_source`
* internal optimization/refactoring to avoid string cloning.
* new options for downloading media files from web-ui
- `organize_into_directories`
- `episode_pattern`
# v1.1.2(2023-11-03)
* Fixed epg for xtream
README.md
+52 -47
View File
@@ -68,11 +68,11 @@ Default is `0`.
### 1.2. `api`
`api` contains the `server-mode` settings. To run `m3u-filter` in `server-mode` you need to start it with the `-s`cli argument.
* `api: {host: localhost, port: 8901, web_root: ./web}`
-`api: {host: localhost, port: 8901, web_root: ./web}`
### 1.3. `working_dir`
`working_dir` is the directory where files are written which are given with relative paths.
* `working_dir: ./data`
-`working_dir: ./data`
With this configuration, you should create a `data` directory where you execute the binary.
@@ -92,8 +92,8 @@ This will replace all occurrences of `!delimiter!` and `!quality!` in the regexp
### 1.5. `sources`
`sources` is a sequence of source definitions, which have two top level entries:
* `inputs`
* `targets`
-`inputs`
-`targets`
### 1.5.1 `inputs`
`inputs` is a list of sources.
@@ -155,30 +155,30 @@ sources:
### 1.5.2 `targets`
Has the following top level entries:
* `enabled` _optional_ default is `true`, if you disable the processing is skipped
* `name` _optional_ default is `default`, if not default it has to be unique, for running selective targets
* `sort` _optional_
* `output` _mandatory_ list of output formats
* `processing_order` _optional_ default is `frm`
* `options` _optional_
* `filter` _mandatory_,
* `rename` _optional_
* `mapping` _optional_
* `watch` _optional_
- `enabled` _optional_ default is `true`, if you disable the processing is skipped
- `name` _optional_ default is `default`, if not default it has to be unique, for running selective targets
- `sort` _optional_
- `output` _mandatory_ list of output formats
- `processing_order` _optional_ default is `frm`
- `options` _optional_
- `filter` _mandatory_,
- `rename` _optional_
- `mapping` _optional_
- `watch` _optional_
### 1.5.2.1 `sort`
Has three top level attributes
* `match_as_ascii` _optional_ default is `false`
* `groups`
* `channels`
- `match_as_ascii` _optional_ default is `false`
- `groups`
- `channels`
#### `groups`
has one top level attribute `order` which can be set to `asc`or `desc`.
#### `channels`
is a list of sort configurations for groups. Each configuration has 3 top level entries.
* `field` can be `group`, `title`, `name` or `url`.
* `group_pattern` is a regular expression like `'^TR.:\s?(.*)'` which is matched against group title.
* `order` can be `asc` or `desc`
- `field` can be `group`, `title`, `name` or `url`.
- `group_pattern` is a regular expression like `'^TR.:\s?(.*)'` which is matched against group title.
- `order` can be `asc` or `desc`
The pattern should be selected taking into account the processing sequence.
@@ -220,10 +220,10 @@ The processing order (Filter, Rename and Map) can be configured for each target
`processing_order: frm` (valid values are: frm, fmr, rfm, rmf, mfr, mrf. default is frm)
### 1.5.2.4 `options`
* ignore_logo `true` or `false`
* underscore_whitespace `true` or `false`
* cleanup `true` or `false`
* kodi_style `true` or `false`
- ignore_logo `true` or `false`
- underscore_whitespace `true` or `false`
- cleanup `true` or `false`
- kodi_style `true` or `false`
`underscore_whitespace`, `cleanup` and `kodi_style` are only valid for `strm` output.
@@ -245,9 +245,9 @@ but lacks a few features like look around and backreferences.
### 1.5.2.6 `rename`
Is a List of rename configurations. Each configuration has 3 top level entries.
* `field` can be `group`, `title`, `name` or `url`.
* `pattern` is a regular expression like `'^TR.:\s?(.*)'`
* `new_name` can contain capture groups variables addressed with `$1`,`$2`,...
- `field` can be `group`, `title`, `name` or `url`.
- `pattern` is a regular expression like `'^TR.:\s?(.*)'`
- `new_name` can contain capture groups variables addressed with `$1`,`$2`,...
`rename` supports capture groups. Each group can be addressed with `$1`, `$2` .. in the `new_name` attribute.
@@ -375,13 +375,16 @@ For more information: [Telegram bots](https://core.telegram.org/bots/tutorial)
It has 2 entries `extensions` and `download`.
The `extensions` are a list of video file extensions like `mp4`, `avi`, `mkv`.
It used for 2 purposes:
a. When you have input `m3u` and output `xtream` the url's with the matching endings will be categorized as `video`.
b. The download button from the `ui` is available.
- `extensions` are a list of video file extensions like `mp4`, `avi`, `mkv`.
When you have input `m3u` and output `xtream` the url's with the matching endings will be categorized as `video`.
`download` is optional and is only necessary if you want to download the video files from the ui
to a specific directory.
- `download` is optional and is only necessary if you want to download the video files from the ui
to a specific directory. if defined, the download button from the `ui` is available.
- `headers` _optional_, download headers
- `organize_into_directories` _optional_, orgainize downloads into directories
- `episode_pattern` _optional_ if you download episodes, the suffix like `S01.E01` should be removed to place all
files into one folder. The named capture group `episode` is mandatory.
Example: `.*(?P<episode>[Ss]\\d{1,2}(.*?)[Ee]\\d{1,2}).*`
```yaml
video:
@@ -394,6 +397,8 @@ video:
User-Agent: "AppleTV/tvOS/9.1.1."
Accept: "video/*"
directory: /tmp/
organize_into_directories: true
episode_pattern: '.*(?P<episode>[Ss]\\d{1,2}(.*?)[Ee]\\d{1,2}).*'
```
### 1.7 `schedule`
@@ -401,17 +406,17 @@ Schedule is optional.
Format is
```yaml
# sec min hour day of month month day of week year
schedule: "0 0 8,12,16,20,22,1 * * * *"
schedule: "0 0 8,20 * * * *"
```
At the given times the complete processing is started. Do not start it every second or minute.
You could be banned from your server.
You could be banned from your server. Twice a day should be enough.
## 2. `mapping.yml`
Has the root item `mappings` which has the following top level entries:
* `templates` _optional_
* `tags` _optional_
* `mapping` _mandatory_
-`templates` _optional_
-`tags` _optional_
-`mapping` _mandatory_
### 2.1 `templates`
If you have a lot of repeats in you regexps, you can use `templates` to make your regexps cleaner.
@@ -437,9 +442,9 @@ Has the following top level entries:
### 2.3 `mapping`
Has the following top level entries:
* `id` _mandatory_
* `match_as_ascii` _optional_ default is `false`
* `mapper` _mandatory_
-`id` _mandatory_
-`match_as_ascii` _optional_ default is `false`
-`mapper` _mandatory_
### 2.3.1 `id`
Is referenced in the `config.yml`, should be a unique identifier
@@ -452,12 +457,12 @@ write regexp without considering chars like `é` and use `e` instead, set this o
### 2.3.3 `mapper`
Has the following top level entries:
* `filter` _optional_
* `pattern`
* `attributes`
* `suffix`
* `prefix`
* `assignments`
- `filter` _optional_
- `pattern`
- `attributes`
- `suffix`
- `prefix`
- `assignments`
#### 2.3.4.1 `filter`
The filter is a string with a statement (@see filter statements).
config.yml
+2
View File
@@ -22,6 +22,8 @@ video:
User-Agent: "AppleTV/tvOS/9.1.1."
Accept: "video/*"
directory: /tmp/
organize_into_directories: true
episode_pattern: '.*(?P<episode>[Ss]\d{1,2}(.*?)[Ee]\d{1,2}).*'
templates:
- name: DE_CHAN
value: >-
src/api/v1_api.rs
+24 -5
View File
@@ -10,11 +10,10 @@ use serde_json::json;
use uuid::Uuid;
use crate::api::api_model::{AppState, FileDownloadRequest, PlaylistRequest, ServerConfig, ServerInputConfig, ServerSourceConfig, ServerTargetConfig};
use crate::download::{get_m3u_playlist, get_xtream_playlist};
use crate::model::config::{ConfigInput, InputType, validate_targets};
use crate::model::config::{ConfigInput, InputType, validate_targets, VideoDownloadConfig};
use crate::utils::{bytes_to_megabytes};
use futures::stream::TryStreamExt;
use log::{error, info};
use regex::Regex;
use reqwest::{header, Response};
use reqwest::header::{HeaderName, HeaderValue};
use unidecode::unidecode;
@@ -239,10 +238,9 @@ pub(crate) async fn download_file(
}
match client.get(_url).headers(headers).send().await {
Ok(response) => {
let filename_re = Regex::new(r"[^A-Za-z0-9_.-]").unwrap();
let filename_re = download._re_filename.as_ref().unwrap();
let filename = filename_re.replace_all(&unidecode(&req.filename).replace(' ', "_"), "").to_string();
let file_stem = Path::new(&filename).file_stem().and_then(OsStr::to_str).unwrap_or("");
let file_dir: PathBuf = [download.directory.as_ref().unwrap(), file_stem].iter().collect();
let file_dir = get_download_directory(download, &filename);
match fs::create_dir_all(&file_dir) {
Ok(_) => {
let path = file_dir.join(filename.as_str());
@@ -281,6 +279,27 @@ pub(crate) async fn download_file(
}
}
fn get_download_directory(download: &VideoDownloadConfig, filename: &String) -> PathBuf {
if download.organize_into_directories {
let mut file_stem = Path::new(&filename).file_stem().and_then(OsStr::to_str).unwrap_or("");
if let Some(re) = &download._re_episode_pattern {
if let Some(captures) = re.captures(file_stem) {
if let Some(episode) = captures.name("episode") {
if !episode.as_str().is_empty() {
file_stem = &file_stem[..episode.start()];
}
}
}
}
let re_ending = download._re_remove_filename_ending.as_ref().unwrap();
let dir_name = re_ending.replace(file_stem, "");
let file_dir: PathBuf = [download.directory.as_ref().unwrap(), dir_name.as_ref()].iter().collect();
file_dir
} else {
PathBuf::from(download.directory.as_ref().unwrap())
}
}
pub(crate) fn v1_api_register() -> Scope {
web::scope("/api/v1")
.route("/config", web::get().to(config))
src/model/config.rs
+43 -15
View File
@@ -427,7 +427,15 @@ pub(crate) struct VideoDownloadConfig {
#[serde(default = "default_as_empty_map")]
pub headers: HashMap<String, String>,
pub directory: Option<String>,
#[serde(default = "default_as_false")]
pub organize_into_directories: bool,
pub episode_pattern: Option<String>,
#[serde(skip_serializing, skip_deserializing)]
pub _re_episode_pattern: Option<regex::Regex>,
#[serde(skip_serializing, skip_deserializing)]
pub _re_filename: Option<regex::Regex>,
#[serde(skip_serializing, skip_deserializing)]
pub _re_remove_filename_ending: Option<regex::Regex>,
}
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
@@ -437,6 +445,33 @@ pub(crate) struct VideoConfig {
pub download: Option<VideoDownloadConfig>,
}
impl VideoConfig {
pub fn prepare(&mut self) -> Result<(), M3uFilterError> {
self.extensions = vec!["mkv".to_string(), "avi".to_string(), "mp4".to_string()];
match &mut self.download {
None => {}
Some(downl) => {
if downl.headers.is_empty() {
downl.headers.borrow_mut().insert("Accept".to_string(), "video/*".to_string());
downl.headers.borrow_mut().insert("User-Agent".to_string(), "AppleTV/tvOS/9.1.1.".to_string());
}
if let Some(episode_pattern) = &downl.episode_pattern {
let re = regex::Regex::new(episode_pattern);
if re.is_err() {
return create_m3u_filter_error_result!(M3uFilterErrorKind::Info, "cant parse regex: {}", episode_pattern);
}
downl._re_episode_pattern = Some(re.unwrap());
}
downl._re_filename = Some(regex::Regex::new(r"[^A-Za-z0-9_.-]").unwrap());
downl._re_remove_filename_ending = Some(regex::Regex::new(r"[_.\s-]$").unwrap());
}
}
Ok(())
}
}
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
pub(crate) struct Config {
#[serde(default = "default_as_zero")]
@@ -568,22 +603,15 @@ impl Config {
self.video = Some(VideoConfig {
extensions: vec!["mkv".to_string(), "avi".to_string(), "mp4".to_string()],
download: None,
})
});
}
Some(video) =>
{
video.extensions = vec!["mkv".to_string(), "avi".to_string(), "mp4".to_string()];
match &mut video.download {
None => {}
Some(downl) => {
if downl.headers.is_empty() {
downl.headers.borrow_mut().insert("Accept".to_string(), "video/*".to_string());
downl.headers.borrow_mut().insert("User-Agent".to_string(), "AppleTV/tvOS/9.1.1.".to_string());
}
}
}
Some(video) => {
match video.prepare() {
Ok(_) => {}
Err(err) => return Err(err)
}
}
},
};
Ok(())
}