thegame_1980/TwinVine
1
0
Fork 0
mirror of https://github.com/vinefeeder/TwinVine.git synced 2026-07-15 18:10:04 +02:00
Code Issues Packages Projects Releases Wiki Activity

initial commit

Browse Source
This commit is contained in:
VineFeeder
2025-08-24 13:43:11 +01:00
parent 6333d06038
commit c50af78cce
209 changed files with 37390 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
packages/vinefeeder/README.md
+419
View File
@@ -0,0 +1,419 @@
# VineFeeder
VineFeeder is a dynamic Python-based application and **framework** that presents a graphical user interface (GUI) to act as a front-end for Devine (https://github.com/devine-dl/devine) or envied (https://github.com/envied-dl/envied) - each a video-downloader, envied being a fork of devine.
Vinefeeder enables users to more easily interact with Devine's streaming services. The project allows users to browse and select media content from multiple services, each of which is dynamically loaded as a separate module. Each service has a configuration which will work out of the box, but Devine download options may be set on a service by service basis.
Features:
*Dynamic Service Loading*: Services are loaded dynamically from a predefined folder.
Each service has its own configuration and logic, allowing flexible media handling.
*Service-Specific Parsing*: Each service can define its own method of parsing media
content, making the platform adaptable to different service architectures.
Additionally each service may store a string of download options for devine such as
video resolution or subtitle format.
*Alphabetical Service Listing*: Services are displayed as clickable buttons in
alphabetical order, enhancing the user experience.
*GUI Interface*: The interface is built using PyQt6, offering a user-friendly experience
for selecting media content.
*Category Browsing*: Services support browsing by category, allowing users to explore
content within specific genres or collections.
*Video Download*: Once media content is selected, videos can be downloaded directly
from the service using predefined tools (e.g., devine).
*Ultra-High-definition*: Videos in UHD are automatically requested from sites that provide such quality.
hlg_status in config.yaml may be set to False to not do this.
*Precision Episode Selection*: Vinefeeder handles the selection of episodes and 'feeds'
Devine one-by-one, making Devine's -w switch defunct.
**Installation**
Requirements
**python 3.12**
Python 3.12 or higher Note: PyQT6 needs PyQt6.QtWidgets
Required Python modules (install via pip):
beaupy
devine
httpx
jmespath
PyQt6
PyQt6_sip
PyYAML
rich
Scrapy
You can install the necessary VineFeeder packages with bash or a Window's Terminal. Make sure to install python modules to the same python environment (env) as Devine
First be sure to follow Devine or envied's install and set-up procedure and ensure it all works correctly to your liking.
**If you have installed Devine using Poetry and you run Devine by 'poetry run devine' then install devine again using pip, keeping the poetry install for when you use the command line directly. The pip installed version is the developer's recommended way. If you go off-piste you may break something! Both versions sit happily together**
**If you have installed Devine by any other way than via poetry or by 'pip install devine' then remove it and re-install using the correct method before running VineFeeder! The re-install should pick-up the last configuration. if not re-configure Devine. Make absolutely sure devine can be called from any folder on your system.**
**If you are using envied make sure you have installed the tool version that runs from the command envied. See https://github.com/envied-dl/envied for installation details**
**Setup**
if you've installed Devine or envied , via pip or uv, in a specific python environment (virtual environment - venv) , first start that environment before installing VineFeeder to the same environment.
For all users, after devine/envied is configured
Clone the repository:
With bash or a Window's Terminal
git clone https://github.com/vinefeeder/VineFeeder.git
cd VineFeeder
pip install -r requirements.txt
will install a VineFeeder folder with all files. Change directory, move into VineFeeder
**Usage**
From the VineFeeder folder run the application by executing the main script:
With bash or a Window's Terminal
python vinefeeder.py
or
python vinefeeder.py -help
As set-up on start it will run with zero modification if Devine is correctly installed. There are six UK services and TVNZ active.
Regard TVNZ as experimental for the time being. For one thing Devine's TVNZ service fails on some
Sports titles and the VineFeeder service is still in testing. If Devine fails use the URL with Ozivine -
there are reports it works. This Ozivine fork has a downloader https://github.com/liam8888999/ozivine
It is recommended to use PowerShell or Terminal in Windows and a Unix-like terminal in Linux - but select, via
preferences, a dark background for better contrast with the colours used in Vinefeeder.
Note: envied has an issue releasing screen control back to VineFeeder when a download is finished. There is an option in the top-level config.yaml to reset the terminal after a download. As delivered the tetrminal-reset option is off. I don't think the issue appears with Windows; it has not been tested.
The reset clears screen history - which you may not want. So use batch mode. Select as many downloads as you need and the run the batch download. Download history will remain on the screen until all dowloads are finished.
**Interacting with Services**
Once the GUI is launched, you may interact with various streaming services by clicking on their corresponding buttons.
The 'URL or search' box MAY be used for an immediate search entry or direct-download URL or it may be left empty.
If left empty a menu is offered.
A search for a series-name will usually end with a list of episodes to select from.
If a series-url is pasted in the text-box then Vinefeeder will immediatly pass the url to the downloader. And being a series-url the downloader will download the complete series. So if you wish to select episodes use a search term.
**Help**
Starting VineFeeder with python vinefeeder.py --help will show options to set a service congfiguration - if required.
In the top level folder is config.yaml. Open the file in a text editor and set your preferred DOWNLOAD_ORCHESTRATOR to devine or envied.
**Service Configuration**
Most users will not need alter any configuration to download videos!
However, if you have, in your past use of Devine resorted to setting Devine-download options other than -w etc.
You may continue to do with VineFeeder only you do so just once, in the config.yaml for each service.
Each service's config.yaml has an 'options:' entry just add the string you would use with devine e.g -q 720 to
ensure only 720p resolution videos are selected.
To open a config.yaml for a service:-
python vinefeeder.py --service-folder <SERVICE-NAME>
python vinefeeder.py --service-folder ALL4, for example
Edit the line starting 'options'. Use exactly the same syntax as Devine would require on its command line
Image
![Vinefeeder GUI](https://github.com/vinefeeder/VineFeeder/blob/main/images/vinefeeder8.png)
Services are displayed in alphabetical order for easy access. From there, you can:
Browse Categories: Select a media category to view available content.
Search by URL: Input a direct URL to download media content.
Download Media: Select a media item to start the download process.
**Services**
Currently seven services are working - All4, BBC, ITVX, My5 STV, U and TPTV , all UK services plus TVNZ.
Other services: awaiting contributors!!
**Text Entry Box**
This will be the predominant method of using VineFeeder. Search on a keyword to locate a
programme series and click on its title when search finishes, to select any combination of
episodes for download.
Additionlly the search box will take a URL for direct download. The URL may be for a single
episode and it will download directly. If you feed a URL for a *series of episode* i.e a
series-link then Devine will download the series.
The text-entry-box is cleared automatically after use. Once a download successfully finishes
the screen displays 'Ready!' and waits for further entry.
**Series numbers**
Normally services will provide enough data to collect a series number.
When series numbers are not available, Vinfeeder sets a default number of 100.
This is for housekeeping reasons. Devine labelling defaults to 00 for any saved
videos without a specified series number.
**Backing out**
If you have followed a path to a selection and do not wish to continue then,
in most circumstances, depending on operating system, ctrl-C will return you
to a 'Ready!' status display on the screen within a second or two, and await
further operations.
**Batch Mode**
To select Batch Mode, move the GUI's batch-mode slider to the right.
When batch-mode is selected all devine download commands will be saved to
a file, batch.txt, created in the top level folder of VineFeeder.
*BE SURE TO SELECT BATCH MODE BEFORE SELECTING A SERVICE!*
The batch file will only be created once a service's video download is initiated.
A banner in the GUI will then display the existence of batch.txt.
Any service may be used to add to batch.txt for later batch download.
Clicking the Run Batch button will then download all the selected videos.
After devine's downloads are complete, the option to delete the batch file will
appear. The file may also be manually deleted.
Note: While batch-run is downloading you may still add videos to batch.txt if you paste
the video link for a service into VineFeeder's text box. You may not interact with a service
if the service needs to use Terminal to display.
Batch mode preserves state; you may close VineFeeder down, if you wish,
and resume later.
**UHD**
Currently only the BBC service carries HLG output. Vinefeeder will attempt to automatically
detect if HLG videos are available. Software compares the search term to a fetched list of
HLG titles. For this reason the search term *must* accurately match the programme title.
The BBC programme "The Gold" will only be found with search terms 'the gold'; 'The Gold'
or upper/lower-case variations.
The current BBC H.265 output:
1. A Good Girl's Guide to Murder
2. A Perfect Planet
3. Asia
4. Attenborough and the Giant Sea Monster
5. Attenborough's Wonder of Song
6. Blue Lights
7. Blue Planet II
8. Boarders
9. Boat Story
10. Boiling Point
11. Champion
12. Cunk on Life
13. Doctor Who
14. Doctor Who: 60th Anniversary Special
15. Doctor Who: Revolution of the Daleks
16. Doctor Who: The Power of the Doctor
17. Domino Day
18. Dynasties
19. Everything I know about Love
20. Families Like Ours
21. Frozen Planet II
22. Fungi: The Web of Life
23. Glastonbury
24. His Dark Materials
25. Industry, Series 2 and 3
26. Life After Life
27. Mammals
28. Men Up
29. Mood
30. Mr Loverman
31. Murder is Easy
32. Nightsleeper
33. Peaky Blinders, Series 6
34. Planet Earth III
35. Red Rose
36. Reunion
37. SAS Rogue Heroes
38. Serengeti, Series 3
39. Seven Worlds, One Planet
40. Sherwood
41. Showtrial
42. Squad Goals: Dorking 'Til I Die
43. Storyville: Your Fat Friend
44. Strike: The Ink Black Heart
45. Swan Lake from English National Ballet
46. Ten Pound Poms
47. The Americas
48. The Bombing of Pan Am 103
49. The Boy, the Mole, the Fox and the Horse
50. The English
51. The Gold
52. The Green Planet
53. The Jetty
54. The Mating Game
55. The Reckoning
56. The Responder
57. The Sixth Commandment
58. The Tourist
59. The Way
60. The Woman in the Wall
61. This City is Ours
62. This Town
63. Towards Zero
64. Vigil
65. What It Feels Like for a Girl
66. Wild Isles
67. Wreck
| |
#### NOTE
In the series above not every episode is available in HLG. Glastonbury is particularly
tricky. When Devine is told to expect HLG and then finds no output, devine will throw an error.
E.g this fails; devine dl -r HLG --list iP https://www.bbc.co.uk/iplayer/episodes/b007r6vx/glastonbury
as some output is SDR
A way around this is to start the service editor with the command: " python vinefeeder.py --service-folder BBC".
Then edit the line hlg_status: True; change True to False, and save the file. This turns off HLG from being downloaded.
**Closing Down**
The GUI should first be shutdown by mouse-clicking the top right X. The GUI then should
release control of the Terminal. ctrl-C in the Terminal may be required if shutting down
following an error.
**Custom Services**
Vinefeeder is a *framework* too. It has been written in such a way that most features
needed to scape and parse a website are already written and available.
If you feel moderately confident with python, then writing a new service to allow Devine
to be run more interactively, is relatively straightforward. Most of the processes for
downloading, parsing and displaying for selection form part of the base functions and
are already written in this framework.
Any new service will need an __init__.py to be written and to implement just four
methods.
The ultimate aim of a services' __init__.py is to extract a list of video-data from a content
provide and store it a numerous dict items in a list container. The video-data is collected in
dict objects labelled 'episode'.
Each episode has
episode = {
'series_no': ser_number,
'title': ep_number,
'url': url,
'synopsis': synopsis
}
Note 'title' may be an episode number or descriptive text. It varies with provider
The dict is added to the Vinefeeder storage list by:-
self.add_episode(brand_slug, episode)
Note brand_slug is a series-name. self.add_episode creates a lists of episode dicts
This happens in the __init__.py super-class of BaseLoader
To add a new service:
Create a new folder for the service inside the services directory.
Add the following:
A config.yaml file with the media_dict. see examples in the existing services folder.
An __init__.py file defining the loader class for the service.
The name of the loader class is constrained. It must take the service tag name, e.g. TVNZ
and use that to create the class-name TvnzLoader. This allows vinefeeder.py discover and
to call its use later.
Note: the loader class file MUST inherit (or sub-class) BaseLoader see ALL4/__init__.py
as an example.
Note: Most web-sites that provide on-demand streaming have a 'browse' or 'category'
page where video categories may be selected for view. Use some/all of these links to
produce a media_dict of ( catergory: link, }
There are 4 methods to implement
receive
fetch_videos
second_fetch
fetch_videos_by_category
Follow any channel as a model except BBC and U. Most sites provide json to describe their video content.
Usually it is within a script.
There are two methods to extract the json in parsing_utils depending on whether a script id
is used or not.
Parsing utils uses XPATH as a locator syntax.
ChatGPT will helpt to find the XPATH syntax if you give it the web page and tell it
which javascript you need.XPATH is considerably faster than alternates such as bs4.
The BaseLoader class which your service must inherit, has methods to GET, POST or
return OPTIONS from the web.
DO NOT USE OTHER METHODS THAN THESE PROVIDED.
RECEIVE
I find it easier to start off implementing a keyword search from the GUI text box.
Very few changes to an existing service receive method will be needed.
Copy one and adjust. Note the use of inx (index) to help specify action.
Vinefeeder's GUI calls the service, passing parameters in the process such as a
service's Devine download options.
By definiton some receive() parameters may be empty.
FETCH_VIDEOS
Fetch_videos is very service specific and will need to send data to a web-site and processs the response to provide
an 'episode' list for display.
Here 'episode' is a container for a web site's response of suggestion of a single video or series that match the
search-word.
In the example below the parsed_data contains json from which we extract to add to BaseLoader's episode list via
self.add_episode(series_name, episode)
if parsed_data and 'results' in parsed_data:
for item in parsed_data['results']:
series_name = item.get('brand', {}).get('websafeTitle', 'Unknown Series')
episode = {
'title': item.get('brand', {}).get('title', 'Unknown Title'),
'url': f"{item.get('brand', {}).get('href', '')}",
'synopsis': item.get('brand', {}).get('description', 'No synopsis available.')
}
self.add_episode(series_name, episode)
SECOND_FETCH
Again the process is much the same as in fetch_videos() a web site has its html harvested,
a script is extracted from which json is pulled using the facility methods in BaseLoader and parsing _utils.
FETCH_VIDEO_BY_CATEGORY
Displays the media_dict from congig.yaml - so put any category heading and links in the new service config.
Once a category is selected, again parse the json and follow an existing service for a model answer, adjusting to suit the syntax required by our new service.
Potentially, you may not implement this if the site does not provide a worthwhile browse by category list,
or the data cannot be parsed readily. The service U falls into this category.
VineFeeder will dynamically detect and load the new services on the next start.
**Special Devine Commands**
If your use requirements are such that each video you download using Devine needs many different parameters to be passed
to the downloader, and set different each time, this tool may not meet your needs:
see https://github.com/billybanana80/DevineGUI
If however you are happy to set your download parameters once, variable for each service, the service's config.yaml,
then this project may work well for you, and provide the front-end Devine has always lacked.
**Contributing**
Contributors are welcomed to add services to front-end those services provided by Devine - see above.
Presently six UK services and New Zealand's TVNZ are fully configured; contributions for other services welcome.
.
Images
![VineFeeder GUI](https://github.com/vinefeeder/VineFeeder/blob/main/images/vinefeeder1.png)
![VineFeeder GUI](https://github.com/vinefeeder/VineFeeder/blob/main/images/vinefeeder2.png)
![Vinefeeder GUI](https://github.com/vinefeeder/VineFeeder/blob/main/images/vinefeeder3.png)
![Vinefeeder GUI](https://github.com/vinefeeder/VineFeeder/blob/main/images/vinefeeder4.png)
![Vinefeeder GUI](https://github.com/vinefeeder/VineFeeder/blob/main/images/vinefeeder5.png)
![Vinefeeder GUI](https://github.com/vinefeeder/VineFeeder/blob/main/images/vinefeeder6.png)
![Vinefeeder GUI](https://github.com/vinefeeder/VineFeeder/blob/main/images/vinefeeder7.png)
![Vinefeeder GUI](https://github.com/vinefeeder/VineFeeder/blob/main/images/vinefeeder9.png)
**License**
This project is licensed under the MIT License. See the LICENSE file for more details.
packages/vinefeeder/pyproject.toml
+21
View File
@@ -0,0 +1,21 @@
[project]
name = "vinefeeder"
version = "0.1.0"
description = "GUI front-end for envied"
readme = "README.md"
requires-python = ">=3.11"
dependencies = [
"envied", # resolved via [tool.uv.sources] at the workspace root
"PyQt6", # or GUI toolkit(s)
"rich",
]
[project.scripts]
vinefeeder = "vinefeeder.__main__:main" # GUI entry point
[build-system]
requires = ["uv_build>=0.8.12,<0.9.0"]
build-backend = "uv_build"
[tool.uv.package-data]
vinefeeder = ["config.yaml", "services/**", "images/**"]
packages/vinefeeder/src/vinefeeder/LICENSE
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2024 A_n_g_e_l_a
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
packages/vinefeeder/src/vinefeeder/README.md
+419
View File
@@ -0,0 +1,419 @@
# VineFeeder
VineFeeder is a dynamic Python-based application and **framework** that presents a graphical user interface (GUI) to act as a front-end for Devine (https://github.com/devine-dl/devine) or envied (https://github.com/envied-dl/envied) - each a video-downloader, envied being a fork of devine.
Vinefeeder enables users to more easily interact with Devine's streaming services. The project allows users to browse and select media content from multiple services, each of which is dynamically loaded as a separate module. Each service has a configuration which will work out of the box, but Devine download options may be set on a service by service basis.
Features:
*Dynamic Service Loading*: Services are loaded dynamically from a predefined folder.
Each service has its own configuration and logic, allowing flexible media handling.
*Service-Specific Parsing*: Each service can define its own method of parsing media
content, making the platform adaptable to different service architectures.
Additionally each service may store a string of download options for devine such as
video resolution or subtitle format.
*Alphabetical Service Listing*: Services are displayed as clickable buttons in
alphabetical order, enhancing the user experience.
*GUI Interface*: The interface is built using PyQt6, offering a user-friendly experience
for selecting media content.
*Category Browsing*: Services support browsing by category, allowing users to explore
content within specific genres or collections.
*Video Download*: Once media content is selected, videos can be downloaded directly
from the service using predefined tools (e.g., devine).
*Ultra-High-definition*: Videos in UHD are automatically requested from sites that provide such quality.
hlg_status in config.yaml may be set to False to not do this.
*Precision Episode Selection*: Vinefeeder handles the selection of episodes and 'feeds'
Devine one-by-one, making Devine's -w switch defunct.
**Installation**
Requirements
**python 3.12**
Python 3.12 or higher Note: PyQT6 needs PyQt6.QtWidgets
Required Python modules (install via pip):
beaupy
devine
httpx
jmespath
PyQt6
PyQt6_sip
PyYAML
rich
Scrapy
You can install the necessary VineFeeder packages with bash or a Window's Terminal. Make sure to install python modules to the same python environment (env) as Devine
First be sure to follow Devine or envied's install and set-up procedure and ensure it all works correctly to your liking.
**If you have installed Devine using Poetry and you run Devine by 'poetry run devine' then install devine again using pip, keeping the poetry install for when you use the command line directly. The pip installed version is the developer's recommended way. If you go off-piste you may break something! Both versions sit happily together**
**If you have installed Devine by any other way than via poetry or by 'pip install devine' then remove it and re-install using the correct method before running VineFeeder! The re-install should pick-up the last configuration. if not re-configure Devine. Make absolutely sure devine can be called from any folder on your system.**
**If you are using envied make sure you have installed the tool version that runs from the command envied. See https://github.com/envied-dl/envied for installation details**
**Setup**
if you've installed Devine or envied , via pip or uv, in a specific python environment (virtual environment - venv) , first start that environment before installing VineFeeder to the same environment.
For all users, after devine/envied is configured
Clone the repository:
With bash or a Window's Terminal
git clone https://github.com/vinefeeder/VineFeeder.git
cd VineFeeder
pip install -r requirements.txt
will install a VineFeeder folder with all files. Change directory, move into VineFeeder
**Usage**
From the VineFeeder folder run the application by executing the main script:
With bash or a Window's Terminal
python vinefeeder.py
or
python vinefeeder.py -help
As set-up on start it will run with zero modification if Devine is correctly installed. There are six UK services and TVNZ active.
Regard TVNZ as experimental for the time being. For one thing Devine's TVNZ service fails on some
Sports titles and the VineFeeder service is still in testing. If Devine fails use the URL with Ozivine -
there are reports it works. This Ozivine fork has a downloader https://github.com/liam8888999/ozivine
It is recommended to use PowerShell or Terminal in Windows and a Unix-like terminal in Linux - but select, via
preferences, a dark background for better contrast with the colours used in Vinefeeder.
Note: envied has an issue releasing screen control back to VineFeeder when a download is finished. There is an option in the top-level config.yaml to reset the terminal after a download. As delivered the tetrminal-reset option is off. I don't think the issue appears with Windows; it has not been tested.
The reset clears screen history - which you may not want. So use batch mode. Select as many downloads as you need and the run the batch download. Download history will remain on the screen until all dowloads are finished.
**Interacting with Services**
Once the GUI is launched, you may interact with various streaming services by clicking on their corresponding buttons.
The 'URL or search' box MAY be used for an immediate search entry or direct-download URL or it may be left empty.
If left empty a menu is offered.
A search for a series-name will usually end with a list of episodes to select from.
If a series-url is pasted in the text-box then Vinefeeder will immediatly pass the url to the downloader. And being a series-url the downloader will download the complete series. So if you wish to select episodes use a search term.
**Help**
Starting VineFeeder with python vinefeeder.py --help will show options to set a service congfiguration - if required.
In the top level folder is config.yaml. Open the file in a text editor and set your preferred DOWNLOAD_ORCHESTRATOR to devine or envied.
**Service Configuration**
Most users will not need alter any configuration to download videos!
However, if you have, in your past use of Devine resorted to setting Devine-download options other than -w etc.
You may continue to do with VineFeeder only you do so just once, in the config.yaml for each service.
Each service's config.yaml has an 'options:' entry just add the string you would use with devine e.g -q 720 to
ensure only 720p resolution videos are selected.
To open a config.yaml for a service:-
python vinefeeder.py --service-folder <SERVICE-NAME>
python vinefeeder.py --service-folder ALL4, for example
Edit the line starting 'options'. Use exactly the same syntax as Devine would require on its command line
Image
![Vinefeeder GUI](https://github.com/vinefeeder/VineFeeder/blob/main/images/vinefeeder8.png)
Services are displayed in alphabetical order for easy access. From there, you can:
Browse Categories: Select a media category to view available content.
Search by URL: Input a direct URL to download media content.
Download Media: Select a media item to start the download process.
**Services**
Currently seven services are working - All4, BBC, ITVX, My5 STV, U and TPTV , all UK services plus TVNZ.
Other services: awaiting contributors!!
**Text Entry Box**
This will be the predominant method of using VineFeeder. Search on a keyword to locate a
programme series and click on its title when search finishes, to select any combination of
episodes for download.
Additionlly the search box will take a URL for direct download. The URL may be for a single
episode and it will download directly. If you feed a URL for a *series of episode* i.e a
series-link then Devine will download the series.
The text-entry-box is cleared automatically after use. Once a download successfully finishes
the screen displays 'Ready!' and waits for further entry.
**Series numbers**
Normally services will provide enough data to collect a series number.
When series numbers are not available, Vinfeeder sets a default number of 100.
This is for housekeeping reasons. Devine labelling defaults to 00 for any saved
videos without a specified series number.
**Backing out**
If you have followed a path to a selection and do not wish to continue then,
in most circumstances, depending on operating system, ctrl-C will return you
to a 'Ready!' status display on the screen within a second or two, and await
further operations.
**Batch Mode**
To select Batch Mode, move the GUI's batch-mode slider to the right.
When batch-mode is selected all devine download commands will be saved to
a file, batch.txt, created in the top level folder of VineFeeder.
*BE SURE TO SELECT BATCH MODE BEFORE SELECTING A SERVICE!*
The batch file will only be created once a service's video download is initiated.
A banner in the GUI will then display the existence of batch.txt.
Any service may be used to add to batch.txt for later batch download.
Clicking the Run Batch button will then download all the selected videos.
After devine's downloads are complete, the option to delete the batch file will
appear. The file may also be manually deleted.
Note: While batch-run is downloading you may still add videos to batch.txt if you paste
the video link for a service into VineFeeder's text box. You may not interact with a service
if the service needs to use Terminal to display.
Batch mode preserves state; you may close VineFeeder down, if you wish,
and resume later.
**UHD**
Currently only the BBC service carries HLG output. Vinefeeder will attempt to automatically
detect if HLG videos are available. Software compares the search term to a fetched list of
HLG titles. For this reason the search term *must* accurately match the programme title.
The BBC programme "The Gold" will only be found with search terms 'the gold'; 'The Gold'
or upper/lower-case variations.
The current BBC H.265 output:
1. A Good Girl's Guide to Murder
2. A Perfect Planet
3. Asia
4. Attenborough and the Giant Sea Monster
5. Attenborough's Wonder of Song
6. Blue Lights
7. Blue Planet II
8. Boarders
9. Boat Story
10. Boiling Point
11. Champion
12. Cunk on Life
13. Doctor Who
14. Doctor Who: 60th Anniversary Special
15. Doctor Who: Revolution of the Daleks
16. Doctor Who: The Power of the Doctor
17. Domino Day
18. Dynasties
19. Everything I know about Love
20. Families Like Ours
21. Frozen Planet II
22. Fungi: The Web of Life
23. Glastonbury
24. His Dark Materials
25. Industry, Series 2 and 3
26. Life After Life
27. Mammals
28. Men Up
29. Mood
30. Mr Loverman
31. Murder is Easy
32. Nightsleeper
33. Peaky Blinders, Series 6
34. Planet Earth III
35. Red Rose
36. Reunion
37. SAS Rogue Heroes
38. Serengeti, Series 3
39. Seven Worlds, One Planet
40. Sherwood
41. Showtrial
42. Squad Goals: Dorking 'Til I Die
43. Storyville: Your Fat Friend
44. Strike: The Ink Black Heart
45. Swan Lake from English National Ballet
46. Ten Pound Poms
47. The Americas
48. The Bombing of Pan Am 103
49. The Boy, the Mole, the Fox and the Horse
50. The English
51. The Gold
52. The Green Planet
53. The Jetty
54. The Mating Game
55. The Reckoning
56. The Responder
57. The Sixth Commandment
58. The Tourist
59. The Way
60. The Woman in the Wall
61. This City is Ours
62. This Town
63. Towards Zero
64. Vigil
65. What It Feels Like for a Girl
66. Wild Isles
67. Wreck
| |
#### NOTE
In the series above not every episode is available in HLG. Glastonbury is particularly
tricky. When Devine is told to expect HLG and then finds no output, devine will throw an error.
E.g this fails; devine dl -r HLG --list iP https://www.bbc.co.uk/iplayer/episodes/b007r6vx/glastonbury
as some output is SDR
A way around this is to start the service editor with the command: " python vinefeeder.py --service-folder BBC".
Then edit the line hlg_status: True; change True to False, and save the file. This turns off HLG from being downloaded.
**Closing Down**
The GUI should first be shutdown by mouse-clicking the top right X. The GUI then should
release control of the Terminal. ctrl-C in the Terminal may be required if shutting down
following an error.
**Custom Services**
Vinefeeder is a *framework* too. It has been written in such a way that most features
needed to scape and parse a website are already written and available.
If you feel moderately confident with python, then writing a new service to allow Devine
to be run more interactively, is relatively straightforward. Most of the processes for
downloading, parsing and displaying for selection form part of the base functions and
are already written in this framework.
Any new service will need an __init__.py to be written and to implement just four
methods.
The ultimate aim of a services' __init__.py is to extract a list of video-data from a content
provide and store it a numerous dict items in a list container. The video-data is collected in
dict objects labelled 'episode'.
Each episode has
episode = {
'series_no': ser_number,
'title': ep_number,
'url': url,
'synopsis': synopsis
}
Note 'title' may be an episode number or descriptive text. It varies with provider
The dict is added to the Vinefeeder storage list by:-
self.add_episode(brand_slug, episode)
Note brand_slug is a series-name. self.add_episode creates a lists of episode dicts
This happens in the __init__.py super-class of BaseLoader
To add a new service:
Create a new folder for the service inside the services directory.
Add the following:
A config.yaml file with the media_dict. see examples in the existing services folder.
An __init__.py file defining the loader class for the service.
The name of the loader class is constrained. It must take the service tag name, e.g. TVNZ
and use that to create the class-name TvnzLoader. This allows vinefeeder.py discover and
to call its use later.
Note: the loader class file MUST inherit (or sub-class) BaseLoader see ALL4/__init__.py
as an example.
Note: Most web-sites that provide on-demand streaming have a 'browse' or 'category'
page where video categories may be selected for view. Use some/all of these links to
produce a media_dict of ( catergory: link, }
There are 4 methods to implement
receive
fetch_videos
second_fetch
fetch_videos_by_category
Follow any channel as a model except BBC and U. Most sites provide json to describe their video content.
Usually it is within a script.
There are two methods to extract the json in parsing_utils depending on whether a script id
is used or not.
Parsing utils uses XPATH as a locator syntax.
ChatGPT will helpt to find the XPATH syntax if you give it the web page and tell it
which javascript you need.XPATH is considerably faster than alternates such as bs4.
The BaseLoader class which your service must inherit, has methods to GET, POST or
return OPTIONS from the web.
DO NOT USE OTHER METHODS THAN THESE PROVIDED.
RECEIVE
I find it easier to start off implementing a keyword search from the GUI text box.
Very few changes to an existing service receive method will be needed.
Copy one and adjust. Note the use of inx (index) to help specify action.
Vinefeeder's GUI calls the service, passing parameters in the process such as a
service's Devine download options.
By definiton some receive() parameters may be empty.
FETCH_VIDEOS
Fetch_videos is very service specific and will need to send data to a web-site and processs the response to provide
an 'episode' list for display.
Here 'episode' is a container for a web site's response of suggestion of a single video or series that match the
search-word.
In the example below the parsed_data contains json from which we extract to add to BaseLoader's episode list via
self.add_episode(series_name, episode)
if parsed_data and 'results' in parsed_data:
for item in parsed_data['results']:
series_name = item.get('brand', {}).get('websafeTitle', 'Unknown Series')
episode = {
'title': item.get('brand', {}).get('title', 'Unknown Title'),
'url': f"{item.get('brand', {}).get('href', '')}",
'synopsis': item.get('brand', {}).get('description', 'No synopsis available.')
}
self.add_episode(series_name, episode)
SECOND_FETCH
Again the process is much the same as in fetch_videos() a web site has its html harvested,
a script is extracted from which json is pulled using the facility methods in BaseLoader and parsing _utils.
FETCH_VIDEO_BY_CATEGORY
Displays the media_dict from congig.yaml - so put any category heading and links in the new service config.
Once a category is selected, again parse the json and follow an existing service for a model answer, adjusting to suit the syntax required by our new service.
Potentially, you may not implement this if the site does not provide a worthwhile browse by category list,
or the data cannot be parsed readily. The service U falls into this category.
VineFeeder will dynamically detect and load the new services on the next start.
**Special Devine Commands**
If your use requirements are such that each video you download using Devine needs many different parameters to be passed
to the downloader, and set different each time, this tool may not meet your needs:
see https://github.com/billybanana80/DevineGUI
If however you are happy to set your download parameters once, variable for each service, the service's config.yaml,
then this project may work well for you, and provide the front-end Devine has always lacked.
**Contributing**
Contributors are welcomed to add services to front-end those services provided by Devine - see above.
Presently six UK services and New Zealand's TVNZ are fully configured; contributions for other services welcome.
.
Images
![VineFeeder GUI](https://github.com/vinefeeder/VineFeeder/blob/main/images/vinefeeder1.png)
![VineFeeder GUI](https://github.com/vinefeeder/VineFeeder/blob/main/images/vinefeeder2.png)
![Vinefeeder GUI](https://github.com/vinefeeder/VineFeeder/blob/main/images/vinefeeder3.png)
![Vinefeeder GUI](https://github.com/vinefeeder/VineFeeder/blob/main/images/vinefeeder4.png)
![Vinefeeder GUI](https://github.com/vinefeeder/VineFeeder/blob/main/images/vinefeeder5.png)
![Vinefeeder GUI](https://github.com/vinefeeder/VineFeeder/blob/main/images/vinefeeder6.png)
![Vinefeeder GUI](https://github.com/vinefeeder/VineFeeder/blob/main/images/vinefeeder7.png)
![Vinefeeder GUI](https://github.com/vinefeeder/VineFeeder/blob/main/images/vinefeeder9.png)
**License**
This project is licensed under the MIT License. See the LICENSE file for more details.
packages/vinefeeder/src/vinefeeder/__init__.py
View File
packages/vinefeeder/src/vinefeeder/__main__.py
+688
View File
@@ -0,0 +1,688 @@
from PyQt6.QtWidgets import (QApplication,QWidget,QVBoxLayout,QLabel,QLineEdit,QPushButton,QCheckBox,QFrame,)
from PyQt6.QtCore import Qt
from PyQt6.QtGui import QPalette, QColor
from PyQt6.QtCore import QTimer
from PyQt6.QtWidgets import QHBoxLayout, QSlider
import sys
import os
import re
from importlib import resources
import importlib
from pathlib import Path
import yaml
from beaupy import select
import threading
from .pretty import pretty_print
from rich.console import Console
from .parsing_utils import prettify
import click
import subprocess
from .batchloader import batchload
import pkgutil
#import inspect
PAGE_SIZE = 8 # size of beaupy pagination
console = Console()
catppuccin_mocha = {
# Colors based on "CatppuccinMocha" from Gogh themes
"bg": "rgb(30,30,46)",
"text": "rgb(205,214,244)",
"text2": "rgb(162,169,193)", # slightly darker
"black": "rgb(69,71,90)",
"bright_black": "rgb(88,91,112)",
"red": "rgb(243,139,168)",
"green": "rgb(166,227,161)",
"yellow": "rgb(249,226,175)",
"blue": "rgb(137,180,250)",
"pink": "rgb(245,194,231)",
"cyan": "rgb(148,226,213)",
"gray": "rgb(166,173,200)",
"bright_gray": "rgb(186,194,222)",
"dark_gray": "rgb(54,54,84)",
}
"""
Example usage:
uv run vinefeeder --help # Show help text
uv run vinefeeder # Launch VineFeeder GUI
To run envied: uv run envied
In the GUI:-
enter search text and select a service
or
enter a video URL for direct download
and select a service
or
leave the seach box blank and select a service
a further menu will appear in the terminal
After a download has finished and 'Ready!' appears
another service may be started.
In the terminal:-
enter the number(s) of the service to download
see
"""
# --- config helpers ----------------------------------------------------------
_PKG_NAME = "vinefeeder"
_CFG_NAME = "config.yaml"
def _user_config_dir() -> Path:
# XDG on Linux; AppData\Roaming on Windows; ~/.config on others
if os.name == "nt":
base = os.environ.get("APPDATA") or (Path.home() / "AppData" / "Roaming")
return Path(base) / "VineFeeder"
xdg = os.environ.get("XDG_CONFIG_HOME")
return (Path(xdg) if xdg else Path.home() / ".config") / "vinefeeder"
def _user_config_path() -> Path:
return _user_config_dir() / _CFG_NAME
def load_config_with_fallback() -> tuple[dict, Path | None]:
"""
Returns (config_dict, user_path_if_used_or_None).
Prefers user config; falls back to package-bundled default.
"""
user_path = _user_config_path()
if user_path.exists():
data = yaml.safe_load(user_path.read_text(encoding="utf-8")) or {}
return data, user_path
# packaged default
with resources.files(_PKG_NAME).joinpath(_CFG_NAME).open("rb") as f:
data = yaml.safe_load(f) or {}
return data, None
def save_user_config(cfg: dict) -> Path:
"""Writes config to the user path, creating the directory as needed."""
p = _user_config_path()
p.parent.mkdir(parents=True, exist_ok=True)
with p.open("w", encoding="utf-8") as f:
yaml.safe_dump(cfg, f, default_flow_style=False)
return p
def derive_loader_class_name(service_modname: str) -> str:
"""
Derive a class name from a service module name.
Examples:
'ALL4' -> 'All4Loader'
'BBC' -> 'BbcLoader'
'ITVX' -> 'ItvxLoader'
'TVNZ' -> 'TvnzLoader'
'MY5' -> 'My5Loader'
'STV' -> 'StvLoader'
'TPTV' -> 'TptvLoader'
'U' -> 'ULoader'
"""
# Keep only alphanumerics, split on boundaries, then TitleCase & join
cleaned = re.sub(r'[^A-Za-z0-9]+', ' ', service_modname).strip()
# Title-case the whole thing (digits stay as-is)
titled = ''.join(part.capitalize() for part in cleaned.split())
if not titled: # safety
titled = service_modname.capitalize()
return f"{titled}Loader"
class VineFeeder(QWidget):
def __init__(self):
"""
Initialize the VineFeeder object.
This method sets up the VineFeeder object by calling necessary functions to
initialize the UI, store available services dynamically, load services,
and create buttons dynamically.
"""
super().__init__()
self.init_ui()
self.available_services = {} # Store available services dynamically
self.available_service_media_dict = {}
self.available_services_hlg_status = {}
self.available_services_options = {}
self.load_services() # Discover and load services
self.create_service_buttons() # Create buttons dynamically
def init_ui(self):
"""
Initialize the UI components and layout.
This method creates the necessary UI components and sets up the layout.
"""
self.setWindowTitle("VineFeeder")
layout = QVBoxLayout()
self.search_url_label = QLabel("URL or Search")
layout.addWidget(self.search_url_label)
self.search_url_entry = QLineEdit()
self.search_url_entry.setStyleSheet("""
QLineEdit {
border: 2px solid pink;
}
QLineEdit:focus {
border: 2px solid hotpink;
outline: none;
}
""")
layout.addWidget(self.search_url_entry)
highlighted_frame = QFrame()
self.highlighted_layout = QVBoxLayout()
highlighted_frame.setLayout(self.highlighted_layout)
highlighted_frame.setStyleSheet(
"border: 1px solid pink;"
)
layout.addWidget(highlighted_frame)
#
sechighlighted_frame = QFrame()
self.sechighlighted_layout = QVBoxLayout()
sechighlighted_frame.setLayout(self.sechighlighted_layout)
sechighlighted_frame.setStyleSheet(
"border: 1px solid pink;"
)
layout.addWidget(sechighlighted_frame)
# Batch Mode Layout
batch_mode_layout = QHBoxLayout()
self.batch_slider = QSlider(Qt.Orientation.Horizontal)
self.batch_slider.setMinimum(0)
self.batch_slider.setMaximum(1)
self.batch_slider.setTickPosition(QSlider.TickPosition.NoTicks)
self.batch_slider.setSingleStep(1)
self.batch_slider.setFixedWidth(80)
self.batch_slider.valueChanged.connect(self.toggle_batch_mode)
self.batch_slider.setStyleSheet("""QSlider::groove:horizontal {
border: 1px solid #999999;
height: 5px;
background: qlineargradient(x1:0, y1:0, x2:0, y2:1, stop:0 hotpink, stop:1 #c4c4c4);
margin: 2px 0;
}
QSlider::handle:horizontal {
background: qlineargradient(x1:0, y1:0, x2:1, y2:1, stop:0 #b4b4b4, stop:1 #8f8f8f);
border: 1px solid #5c5c5c;
width: 18px;
height: 18px;
margin: -12px 0;
border-radius: 3px;
}
""")
# Batch Mode Label
self.batch_label = QLabel("Batch Mode")
self.batch_label.setStyleSheet("color: white; padding-left: 5px; border: none;")
batch_mode_layout.addWidget(self.batch_label)
batch_mode_layout.addWidget(self.batch_slider)
batch_mode_frame = QFrame()
batch_mode_frame.setLayout(batch_mode_layout)
batch_mode_frame.setStyleSheet("border: none;")
self.sechighlighted_layout.addWidget(batch_mode_frame)
# Batch File Existence Indicator
self.batch_file_status = QLabel("")
self.sechighlighted_layout.addWidget(self.batch_file_status)
# Run Batch Button
self.run_batch_button = QPushButton("Run Batch")
self.run_batch_button.clicked.connect(batchload)
self.run_batch_button.setEnabled(False) # Initially disabled
self.style_batch_button(self.run_batch_button)
self.sechighlighted_layout.addWidget(self.run_batch_button)
# Load batch mode from config
self.load_batch_mode()
self.dark_mode_checkbox = QCheckBox("Dark Mode")
self.dark_mode_checkbox.setChecked(True) # Set dark mode by default
self.dark_mode_checkbox.stateChanged.connect(self.toggle_dark_mode)
layout.addWidget(self.dark_mode_checkbox, alignment=Qt.AlignmentFlag.AlignLeft)
self.setLayout(layout)
# Poll for batch.txt changes every 2 seconds
self.batch_poll_timer = QTimer(self)
self.batch_poll_timer.timeout.connect(self.update_batch_file_indicator)
self.batch_poll_timer.start(2000) # 2000 ms = 2 seconds
# Use a timer to delay the dark mode application slightly
QTimer.singleShot(100, self.toggle_dark_mode) # 100ms delay to ensure rendering
def load_batch_mode(self):
try:
cfg, _ = load_config_with_fallback()
batch_on = bool(cfg.get("BATCH_DOWNLOAD", False))
self.batch_slider.setValue(1 if batch_on else 0)
self.run_batch_button.setEnabled(batch_on)
except Exception as e:
console.print(f"[{catppuccin_mocha['text2']}][warning] Could not load batch mode config: {e}[/]")
self.batch_slider.setValue(0)
self.run_batch_button.setEnabled(False)
self.update_batch_file_indicator()
def toggle_batch_mode(self):
state = self.batch_slider.value() == 1
if state:
self.batch_label.setStyleSheet("color: lightgreen; padding-left: 5px; border: none;")
else:
self.toggle_dark_mode()
self.run_batch_button.setEnabled(state)
self.update_batch_file_indicator()
# Update config (write to the *user* config location)
try:
cfg, _ = load_config_with_fallback()
cfg["BATCH_DOWNLOAD"] = state
save_user_config(cfg)
except Exception as e:
console.print(f"[{catppuccin_mocha['text2']}][warning] Could not update config.yaml: {e}[/]")
def style_batch_button(self, button):
button.setStyleSheet("""
color: white;
background-color:#1E1E2E;
border: none;
padding: 5px;
""")
def update_batch_file_indicator(self):
exists = os.path.exists("./batch.txt")
if hasattr(self, "_batch_file_last_state") and self._batch_file_last_state == exists:
return # no change
self._batch_file_last_state = exists
if exists:
self.batch_file_status.setText("✅ batch file exists")
self.batch_file_status.setStyleSheet("color: lightgreen; padding-left: 25px; border: none;")
else:
self.batch_file_status.setText("❌ batch file missing")
self.batch_file_status.setStyleSheet("color: hotpink; padding-left: 25px; border: none;")
def toggle_dark_mode(self):
"""
Toggle the application's dark mode on or off.
This method is connected to the dark mode checkbox's stateChanged signal.
When the checkbox is checked, the application is set to dark mode. When unchecked,
the application is set to light mode.
In dark mode, the window and text colors are changed to dark colors,
and the search label and dark mode checkbox text are changed to white.
Buttons are also set to have a white text color and a dark grey background color.
In light mode, the window and text colors are set back to their default values,
and the search label and dark mode checkbox text are changed back to black.
Buttons are also reset to their default appearance.
NOTE: This method uses a QTimer to delay the application of the dark mode style slightly.
This is to ensure that the rendering of the UI components is complete
before applying the style changes.
"""
if self.dark_mode_checkbox.isChecked():
palette = QPalette()
palette.setColor(QPalette.ColorRole.Window, QColor(53, 53, 53))
palette.setColor(QPalette.ColorRole.WindowText, Qt.GlobalColor.white)
palette.setColor(
QPalette.ColorRole.Base, QColor(30, 30, 54)
) # QColor(35, 35, 35))
palette.setColor(QPalette.ColorRole.Text, Qt.GlobalColor.white)
self.setPalette(palette)
self.search_url_label.setStyleSheet("color: white;")
self.dark_mode_checkbox.setStyleSheet("color: white;")
# Set button text to white in dark mode, remove red border
for i in range(self.highlighted_layout.count()):
button = self.highlighted_layout.itemAt(i).widget()
if isinstance(button, QPushButton):
button.setStyleSheet("""
color: white;
background-color:#1E1E2E;
border: none;
padding: 5px;
""")
button.repaint() # Force update of the button's appearance
else:
self.setPalette(QApplication.palette())
self.search_url_label.setStyleSheet("color: black;")
self.dark_mode_checkbox.setStyleSheet("color: black;")
self.batch_label.setStyleSheet("color: black;")
for i in range(self.highlighted_layout.count()):
button = self.highlighted_layout.itemAt(i).widget()
if isinstance(button, QPushButton):
button.setStyleSheet("""
color: black;
background-color: #aeaeae;
border: none;
padding: 5px;
""")
button.repaint() # Force update of the button's appearance
# Update batch_label based on dark mode and batch mode state
if self.batch_slider.value() == 1:
self.batch_label.setStyleSheet("color: lightgreen; padding-left: 5px; border: none;")
else:
if self.dark_mode_checkbox.isChecked():
self.batch_label.setStyleSheet("color: white; padding-left: 5px; border: none;")
else:
self.batch_label.setStyleSheet("color: black; padding-left: 5px; border: none;")
def load_services(self):
"""
Discover subpackages under vinefeeder.services, import them (executes __init__.py),
read config.yaml, and store both module + loader class for later use.
"""
import vinefeeder.services as services_pkg
self.available_services.clear()
self.available_services_hlg_status.clear()
self.available_services_options.clear()
self.available_service_media_dict.clear()
for _finder, modname, ispkg in pkgutil.iter_modules(services_pkg.__path__):
if not ispkg:
continue
fqname = f"{services_pkg.__name__}.{modname}"
# Import executes the service's __init__.py
try:
module = importlib.import_module(fqname)
except Exception as e:
console.print(f"[{catppuccin_mocha['text2']}][warning] Could not import service {fqname}: {e}[/]")
continue
# Try reading config.yaml alongside the service package
cfg = {}
try:
cfg_res = resources.files(fqname).joinpath("config.yaml")
if cfg_res.exists():
cfg = yaml.safe_load(cfg_res.read_text(encoding="utf-8")) or {}
except Exception as e:
console.print(f"[{catppuccin_mocha['text2']}][warning] Could not read config for {fqname}: {e}[/]")
service_name = cfg.get("service_name", modname)
service_media_dict = cfg.get("media_dict", {})
service_hlg_status = cfg.get("hlg_status", False)
service_options = cfg.get("options", {})
# Either explicit loader class from config, or derived
loader_class = cfg.get("loader_class") or derive_loader_class_name(modname)
self.available_services_hlg_status[service_name] = service_hlg_status
self.available_services_options[service_name] = service_options
self.available_service_media_dict[service_name] = service_media_dict
# Store metadata for later use (module name + class name)
self.available_services[service_name] = {
"module": fqname,
"loader_class": loader_class,
}
def load_service(self, service_name: str):
self.update_batch_file_indicator()
meta = self.available_services.get(service_name)
if not meta:
console.print(f"[{catppuccin_mocha['text2']}]Service {service_name} not found![/]")
console.print(f"[{catppuccin_mocha['text2']}]Try again[/]")
sys.exit(0)
fqname = meta["module"]
cls_name = meta["loader_class"]
try:
module = importlib.import_module(fqname)
if not hasattr(module, cls_name):
raise AttributeError(f"{fqname} has no class {cls_name}")
loader_class = getattr(module, cls_name)
loader_instance = loader_class()
hlg_status = self.available_services_hlg_status[service_name]
options = self.available_services_options[service_name]
text = self.search_url_entry.text().strip()
text_to_pass = text if text else None
if hasattr(loader_instance, "receive"):
if text_to_pass:
if "http" in text_to_pass:
loader_instance.receive(1, text_to_pass, None, hlg_status, options)
self.clear_search_box()
loader_instance.clean_terminal()
sys.exit(0)
else:
loader_instance.receive(3, text_to_pass, None, hlg_status, options)
self.clear_search_box()
loader_instance.clean_terminal()
sys.exit(0)
else:
inx, text_to_pass, found = self.do_action_select(service_name)
loader_instance.receive(inx, text_to_pass, found, hlg_status, options)
loader_instance.clean_terminal()
sys.exit(0)
else:
console.print(f"[{catppuccin_mocha['pink']}]Service class {cls_name} has no 'receive' method[/]")
except Exception as e:
console.print(f"[{catppuccin_mocha['pink']}]Error loading service: {service_name} {e}[/]")
console.print(f"[{catppuccin_mocha['pink']}]Try again[/]")
sys.exit(0)
def create_service_buttons(self):
"""Create buttons for each dynamically loaded service in alphabetical order."""
# Sort the services alphabetically by their names
for service_name in sorted(self.available_services.keys()):
button = QPushButton(service_name)
button.clicked.connect(
self.run_load_service_thread(service_name)
) # Bind to threaded service loading
self.highlighted_layout.addWidget(button)
#
def do_action_select(self, service_name):
"""
Top level choice for action required. Called if search_box is empty.
Uses beaupy to display a list of 4 options:
- Search by keyword
- Greedy Search by URL
- Browse by Category
- Download by URL
Uses the selected option to call the appropriate function:
- 0 for greedy search with url
- 1 for direct url download
- 2 for browse
- 3 for search with keyword
Returns a tuple of the function selector and the url or None if no valid data is entered.
"""
fn = [
"Greedy Search by URL",
"Download by URL",
"Browse by Category",
"Search by keyword(s)",
]
# check for batch.txt
self.update_batch_file_indicator()
action = select(
fn, preprocessor=lambda val: prettify(val), cursor="🢧", cursor_style="pink1"
)
if "Greedy" in action:
url = input("URL for greedy search ")
return 0, url, None
elif "Download" in action:
url = input("URL for direct download ")
return 1, url, None
elif "Browse" in action:
media_dict = self.available_service_media_dict[service_name]
beaupylist = []
for item in media_dict:
beaupylist.append(item)
found = select(
beaupylist,
preprocessor=lambda val: prettify(val),
cursor="🢧",
cursor_style="pink1",
page_size=PAGE_SIZE,
pagination=True,
)
url = media_dict[found]
return 2, url, found # found is category
elif "Search" in action:
keyword = input("Keyword(s) for search ")
return 3, keyword, None
else:
console.print(f"[{catppuccin_mocha['text2']}]No valid data entered![/]")
sys.exit(0)
def run_load_service_thread(self, service_name):
"""Start a new thread to load the service."""
return lambda: threading.Thread(
target=self.load_service, args=(service_name,)
).start()
def clear_search_box(self):
self.search_url_entry.clear()
@click.command()
@click.option(
"--service-folder",
type=str,
default="services",
help="Specify a service folder for adding **Devine** download options.",
)
@click.option(
"--list-services",
is_flag=True,
help="List available services in the specified service folder.",
)
@click.option(
"--select-series",
is_flag=True,
help="How to select which series you need from those available",
)
def cli(service_folder, list_services, select_series):
"""
uv run vinefeeder --help to show help\n
uv run vinefeeder --list-services to list available services\n
uv run vinefeeder --service-folder <folder_name> to edit config.yaml
uv run vinefeeder --select-series list, range or 'all'\n\n
In the GUI:-
The text box will take keyword(s) or a URL for download from a button selected service.
Or leave the text box blank for further options when the service button is clicked.\n
Batch Mode: slide the slider to the right to engage Batch Mode.
All devine commands will be saved to a batch.txt file in the VineFeeder folder.
When a green 'batch file exists' notice is present the 'Run Batch' button will process
the batch.txt file.
At the end of download the option to delete the batch.txt file will appear.
It may be deleted manually at any time.
"""
# Ensure service-folder paths are handled correctly
if os.path.isabs(service_folder):
base_path = os.path.abspath(service_folder)
else:
base_path = (
os.path.abspath(os.path.join("packages/vinefeeder/src/vinefeeder/services", service_folder))
if service_folder != "services"
else os.path.abspath("services")
)
# Handle --list-services option
if list_services:
if not os.path.exists(base_path):
console.print(f"[{catppuccin_mocha['pink']}]Error: The service folder '{base_path}' does not exist![/]")
return
console.print(f"[{catppuccin_mocha['pink']}]Available services in '{base_path}':")
for service in os.listdir(base_path):
service_dir = os.path.join(base_path, service)
config_path = os.path.join(service_dir, "config.yaml")
if os.path.isdir(service_dir) and os.path.exists(config_path):
console.print(f"[{catppuccin_mocha['text2']}] - {service}[/]")
return
# Handle --select-series option
if select_series:
console.print(f"[{catppuccin_mocha['text2']}]Series Selection:[/]")
console.print(
f"[{catppuccin_mocha['text2']}]Check the available series.\nUse, for example,\n1,3,7 or a range 3..8,\nor 'all' or 0 to show all series.[/]"
)
return
# Default behavior: Open config.yaml
config_path = os.path.join(base_path, "config.yaml")
# Check if the services folder exists
if not os.path.exists(base_path):
console.print(f"[{catppuccin_mocha['pink']}]Error: The service folder '{base_path}' does not exist![/]")
return
# Check if config.yaml exists
if not os.path.exists(config_path):
console.print(
f"[{catppuccin_mocha['text2']}]Error: The file '{config_path}' does not exist! Please create it or specify a valid service folder.[/]"
)
return
# Open the file in the system's default text editor
try:
if os.name == "nt": # For Windows
os.startfile(config_path)
elif os.name == "posix": # For Linux/Mac
subprocess.run(["xdg-open", config_path], check=True)
else:
console.print(f"[{catppuccin_mocha['text2']}]Unsupported operating system.[/]")
except Exception as e:
console.print(f"[{catppuccin_mocha['pink']}]Failed to open the file: {e}[/]")
def main():
"""
Entry point for the script. Decides between GUI launch and CLI behavior.
"""
if len(sys.argv) == 1:
# say hello nicely
pretty_print()
app = QApplication(sys.argv)
window = VineFeeder()
window.show()
sys.exit(app.exec())
else:
# CLI arguments passed, handle them with click
cli()
if __name__ == "__main__":
main()
packages/vinefeeder/src/vinefeeder/base_loader.py
+486
View File
@@ -0,0 +1,486 @@
from httpx import Client
from .parsing_utils import parse_json, prettify, list_prettify
from beaupy import select, select_multiple
from rich.console import Console
from abc import abstractmethod
import os
import platform
import sys
import time
from .pretty import create_clean_panel
from .pretty import catppuccin_mocha
import yaml
import subprocess
from .__main__ import load_config_with_fallback, save_user_config
console = Console()
class BaseLoader:
def __init__(self, headers):
"""Initialize the BaseLoader class with the provided headers.
Parameters:
headers (dict): The headers to be used for making HTTP requests.
Attributes:
client: An instance of the Client class for making HTTP requests.
headers (dict): The headers used for making HTTP requests.
series_data (dict): In-memory store for initial series selection.
final_episode_data (list): List to store final episode data.
console: An instance of the Console class for displaying output.
"""
self.client = Client(timeout=20)
self.headers = headers
self.series_data = {}
self.final_episode_data = []
self.browse_video_list = []
self.category = None
myconfig, _ = load_config_with_fallback()
self.BATCH_DOWNLOAD = myconfig['BATCH_DOWNLOAD']
#self.DOWNLOAD_ORCHESTRATOR = myconfig['DOWNLOAD_ORCHESTRATOR']
self.TERMINAL_RESET = myconfig['TERMINAL_RESET']
def reset_terminal(self):
if self.DOWNLOAD_ORCHESTRATOR == "envied" and not self.BATCH_DOWNLOAD:
if os.name == 'nt': # Windows
os.system('cls')
# Optionally, reinitialize ANSI or console buffer here if needed
else: # Unix/Linux/macOS
try:
subprocess.run(['reset'], check=True)
except Exception:
os.system('clear') # fallback if 'reset' is not available
def clear_series_data(self):
self.series_data = {}
def get_series_data(self):
return self.series_data
def get_data(self, url, headers=None, params=None):
"""Fetch data from a given URL."""
if not headers:
headers = self.headers
response = self.client.get(
url, headers=headers, params=params, follow_redirects=True
)
if response.status_code != 200:
raise Exception(f"Request failed with status {response.status_code}")
return response.text
def get_options(self, url, headers=None):
if not headers:
headers = self.headers
response = self.client.options(url, headers=headers, follow_redirects=True)
if response.status_code != 200:
raise Exception("Failed to retrieve options-data.")
return response.headers
def post_data(self, url, data=None, json=None, headers=None):
if not headers:
headers = self.headers
response = self.client.post(
url, data=data, json=json, headers=headers, follow_redirects=True
)
if response.status_code != 200:
raise Exception("Failed to retrieve data.")
return response
def parse_data(self, html):
"""Parse HTML data into JSON format."""
return parse_json(html)
def normalize_episode(self, episode):
"""
Normalize the episode dictionary for comparison.
Focus on series_no, title, and synopsis.
"""
return (
str(episode.get("series_no", "")).strip().lower(),
episode.get("title", "").strip().lower(),
episode.get("synopsis", "").strip().lower(),
)
def add_episode_remove_duplicates(self, series_name, episode):
"""Add an episode to the series in memory.
Remove duplicates in episode stream
Expects episode to be a dict with series_no, title, and synopsis keys."""
if series_name not in self.series_data:
self.series_data[series_name] = []
# Normalize the current episode
normalized_episode = self.normalize_episode(episode)
# Check for duplicates using normalized episodes
if all(
self.normalize_episode(existing) != normalized_episode
for existing in self.series_data[series_name]
):
self.series_data[series_name].append(episode)
return
def add_episode(self, series_name, episode):
"""Add an episode to the series in memory.
Episode may be Any"""
if series_name not in self.series_data:
self.series_data[series_name] = []
self.series_data[series_name].append(episode)
return
def get_number_of_episodes(self, series_name):
return len(self.series_data.get(series_name, []))
def add_final_episode(self, episode):
"""build episode list for final selction for download"""
if episode not in self.final_episode_data: # Ensure no duplicates
self.final_episode_data.append(episode)
def sort_episodes(self, data): # sort final episode data
# ONLY for some services
# Sort the list by series_no and episode_no (title: will fail if title is free string)
try:
sorted_data = sorted(
data,
key=lambda x: (int(x["series_no"]), int(x["title"])),
)
return sorted_data
except Exception:
return data
def get_series(self, series_name=None):
"""Return all episodes from a specific series or all series data."""
if series_name:
return self.series_data.get(series_name, [])
return self.series_data
def display_series_list(self):
"""Use beaupy to list all series and allow user to select one."""
series_list = list(self.series_data.keys())
selected_series = select(
series_list,
preprocessor=lambda val: prettify(val),
cursor="🢧",
cursor_style="pink1",
page_size=12,
pagination=True,
)
return selected_series
def display_episode_list(self, series_name):
"""Use beaupy to display episodes for a selected series."""
episodes = self.series_data.get(series_name, [])
episode_list = [
f"{ep['series_no']}, {ep['title']}, {ep['url']}, \n\t {ep['synopsis']}"
for ep in episodes
]
selected_episodes = select_multiple(
episode_list,
preprocessor=lambda val: prettify(val),
minimal_count=1,
cursor_style="pink1",
pagination=True,
page_size=8,
)
return selected_episodes
def get_final_episode_list(self):
return self.final_episode_data
def display_final_episode_list(self, final_episode_data):
"""Use beaupy to display episodes for a selected series."""
# episodes = self.series_data.get(series_name, [])
episode_list = [
f"{ep['series_no']}, {ep['title']}, {ep['url']}, \n\t {ep['synopsis']}"
for ep in final_episode_data
]
selected_episodes = select_multiple(
episode_list,
preprocessor=lambda val: prettify(val),
minimal_count=1,
cursor_style="pink1",
pagination=True,
page_size=8,
)
return selected_episodes
def get_selected_url(self, series_name):
"""Return single url for series"""
episode = self.series_data.get(series_name, [])
# list with one dictionary item
url = episode[0]["url"]
return url
def get_episodes_series_numbers(self, series_name):
"""Return a sorted list of series numbers for a given series name."""
try:
mysorted_list = sorted(
{int(ep["series_no"]) for ep in self.series_data[series_name]}
)
except Exception:
# chars in series number cannot be parsed to an int
unsorted_list = [ep["series_no"] for ep in self.series_data[series_name]]
return unsorted_list
return mysorted_list
def display_non_contiguous_series(self, episode_series_numbers):
"""
Display the non-contiguous series numbers in a grid format with 7 columns.
This is used when the series numbers are not contiguous, so the user can
easily see which series numbers are available.
"""
num_columns = 7
num_rows = (len(episode_series_numbers) + num_columns - 1) // num_columns
# Create a grid-like string to display in the panel
grid_content = ""
for row in range(num_rows):
for col in range(num_columns):
idx = row * num_columns + col
if idx < len(episode_series_numbers):
grid_content += f"{episode_series_numbers[idx]:<3} "
else:
grid_content += " " # Add spacing for alignment
grid_content += "\n" # Add a newline after each row
return grid_content.strip()
def prepare_series_for_episode_selection(self, series_name):
"""
Prepare the series data for episode selection and display the final episode list.
Given a selected series name, fetch its HTML and extract its episodes.
The function will check if the series numbers are contiguous and display
the final episode list.
It will return the URLs for the selected episodes.
"""
# bail out if episode count is limited to 12
number_episodes = self.get_number_of_episodes(series_name)
if number_episodes <= 12:
for episode in self.series_data[series_name]:
"""store in list container"""
self.add_final_episode(episode)
return
try:
self.episode_series_numbers = self.get_episodes_series_numbers(series_name)
except Exception:
console.print(f"[{catppuccin_mocha['text2']}]No data found")
self.clean_terminal()
sys.exit(0)
# Check if the series numbers are contiguous (convert to integers for comparison)
episode_series_numbers_int = sorted(
int(num) for num in self.episode_series_numbers
)
if episode_series_numbers_int == list(
range(1, max(episode_series_numbers_int) + 1)
): # Contiguous series numbers
max_series = max(episode_series_numbers_int)
console.print(
create_clean_panel(
(
f"[{catppuccin_mocha['text2']}]There are {max_series} series for {series_name.replace('-', ' ').title()}.\nChoose from 1 to {max_series}. 0 for all."
),
title="info",
)
)
else: # Non-contiguous series
grid_content = self.display_non_contiguous_series(
self.episode_series_numbers
)
console.print(
create_clean_panel(
(
f"[{catppuccin_mocha['text2']}]Data for {series_name.replace('-', ' ').title()}\nSeries are non-contiguous:\n{grid_content}"
),
title="info",
)
)
user_input = console.input(f"[{catppuccin_mocha['text2']}]Series to download? ")
print("\n")
selected_series = []
if user_input == "all" or user_input == "0":
selected_series = self.episode_series_numbers
else:
# Handle single digits, multiple digits, and ranges
for part in user_input.split(","):
part = part.strip()
if ".." in part: # Range of digits (e.g., 2..5)
start, end = map(int, part.split(".."))
selected_series.extend(range(start, end + 1))
else: # Single digit
selected_series.append(int(part)) # Store as int for comparison
# Filter the episodes based on the selected series numbers
for series_no in selected_series:
for episode in self.series_data[series_name]:
if int(episode["series_no"]) == int(series_no):
"""store in list container"""
self.add_final_episode(episode)
def list_display_beaupylist(self, beaupylist):
"""
List all the episodes in the beaupylist, and allow the user to select an episode.
The function will return the selected episode.
This version takes a list of LISTS which is more in keeping with beaupy's intentions
And is the preferred method for future services.
"""
found = select(
beaupylist,
preprocessor=lambda val: list_prettify(val),
cursor="🢧",
cursor_style="pink1",
page_size=8,
pagination=True,
)
return found
def display_beaupylist(self, beaupylist):
"""
List all the episodes in the beaupylist, and allow the user to select an episode.
The function will return the selected episode.
This function takes a list item containing a single STRING and is now deprecated
"""
found = select(
beaupylist,
preprocessor=lambda val: prettify(val),
cursor="🢧",
cursor_style="pink1",
page_size=8,
pagination=True,
)
return found
def process_received_url_from_category(self, url, category=None):
"""
Process the receive category vidoes based on the response length and URL.
Parameters
----------
url : str
The URL to process.
res : list
The response from the URL.
Returns
-------
None
"""
if not category: # no category
category = self.category
if category.lower() in [
"films",
"featured-category-films",
"movies",
"film",
"movie",
]: # by defintion - single
# direct download
self.receive(1, url)
return
if "https" not in url: # browse entry
self.receive(3, url)
return
else:
# if 'https'
# greedy search
self.receive(0, url)
return
def clean_terminal(self):
# clear for next use
time.sleep(1)
if self.TERMINAL_RESET:
self.reset_terminal()
'''if os.name == "posix":
# os.system('clear')
print("Ready!")
return
else:
reset_terminal()
print("Ready!")
return'''
console.print(f"[{catppuccin_mocha['pink']}]Ready![/]")
def runsubprocess(self, command):
"""
Process the subprocess command for download by devine.
Either save to a batch file or run download directly
inherited by all __init__.py files for services
Parameters
----------
command : list
The command to process.
Returns
-------
None
"""
if self.BATCH_DOWNLOAD:
with open('./batch.txt', 'a') as f:
console.print(f"[{catppuccin_mocha['text2']}]writing {command} to batch.txt")
f.write(' '.join(command) + '\n')
else:
subprocess.run(command)
return
### methods that must be implemented by a service
@abstractmethod
def receive(self, inx, search_term: None):
"""
search_term for video search
or for direct url download
Leave search_entry blank for
further search optiions
greedy-search with url
browse-category search using
media_dict from config.yaml for service.
Uses inx, an int variable to switch:-
0 for greedy search using url
1 for direct url download
2 for category browse
3 for search with keyword
inx is sent automatically from vinefeeder.py
method called from vinefeeder and implemented in service/__init__
"""
raise NotImplementedError(
"This method must be implemented in the service-specific loader."
)
@abstractmethod
def fetch_videos_by_category(self, url):
"""This is a base method to be overridden by each service."""
raise NotImplementedError(
"This method must be implemented in the service-specific loader."
)
@abstractmethod
def second_fetch(self, url):
"""This is a base method to be overridden by each service."""
raise NotImplementedError(
"This method must be implemented in the service-specific loader."
)
@abstractmethod
def fetch_videos(self, search_term: str):
"""This is a base method to be overridden by each service."""
raise NotImplementedError(
"This method must be implemented in the service-specific loader."
)
packages/vinefeeder/src/vinefeeder/batchloader.py
+35
View File
@@ -0,0 +1,35 @@
import subprocess
import os
import threading
def _batchloader():
try:
with open('./batch.txt', 'r') as file:
for line in file:
line = line.strip()
if line:
print(line)
subprocess.run(line, shell=True)
print("Batch processing complete.")
yesno = input("delete batch file? (y/n) ")
if yesno.lower() == 'y':
os.remove("./batch.txt")
print("batch.txt deleted.\n\nReady.")
except FileNotFoundError:
print("batch.txt not found.\nFirst set batch mode to True in the GUI, then choose some videos.")
except Exception as e:
print(f"An error occurred: {e}")
def batchload():
thread = threading.Thread(target=_batchloader, daemon=True)
thread.start()
if __name__ == "__main__":
batchload()
packages/vinefeeder/src/vinefeeder/bbc_uhd.py
+21
View File
@@ -0,0 +1,21 @@
from playwright.sync_api import sync_playwright
def fetch_uhd_links():
url = "https://www.bbc.co.uk/iplayer/help/questions/programme-availability/uhd-content"
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
page.goto(url)
page.wait_for_timeout(5000) # Wait for content to load
links = page.locator("a").all()
uhd_links = []
for link in links:
href = link.get_attribute('href')
if href and '/iplayer/episodes/' in href:
uhd_links.append(href)
browser.close()
return list(set(uhd_links))
print(fetch_uhd_links())
packages/vinefeeder/src/vinefeeder/config.yaml
+6
View File
@@ -0,0 +1,6 @@
BATCH_DOWNLOAD: False
DOWNLOAD_ORCHESTRATOR: envied
# Set to True if up/down cursor arrows stop working in the
# Terminal window after unschackle has downloaded.
TERMINAL_RESET: False
packages/vinefeeder/src/vinefeeder/config_io.py
+13
View File
@@ -0,0 +1,13 @@
from __future__ import annotations
from importlib import resources
import yaml
def read_service_config(service_module) -> dict:
"""
Read config.yaml that sits NEXT TO the given service module's __init__.py.
Works for any 'vinefeeder.services.<Name>' package.
"""
cfg = resources.files(service_module.__package__).joinpath("config.yaml")
# .open() works whether installed from wheel or editable src
with cfg.open("rb") as f:
return yaml.safe_load(f) or {}
packages/vinefeeder/src/vinefeeder/images/vinefeeder1.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 608 KiB

packages/vinefeeder/src/vinefeeder/images/vinefeeder2.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 565 KiB

packages/vinefeeder/src/vinefeeder/images/vinefeeder3.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 775 KiB

packages/vinefeeder/src/vinefeeder/images/vinefeeder4.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 777 KiB

packages/vinefeeder/src/vinefeeder/images/vinefeeder5.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 865 KiB

packages/vinefeeder/src/vinefeeder/images/vinefeeder6.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 871 KiB

packages/vinefeeder/src/vinefeeder/parsing_utils.py
+298
View File
@@ -0,0 +1,298 @@
import json
from scrapy import Selector
catppuccin_mocha = {
# Colors based on "CatppuccinMocha" from Gogh themes
"bg": "rgb(30,30,46)",
"text": "rgb(205,214,244)",
"text2": "rgb(162,169,193)", # slightly darker
"black": "rgb(69,71,90)",
"bright_black": "rgb(88,91,112)",
"red": "rgb(243,139,168)",
"green": "rgb(166,227,161)",
"yellow": "rgb(249,226,175)",
"blue": "rgb(137,180,250)",
"pink": "rgb(245,194,231)",
"cyan": "rgb(148,226,213)",
"gray": "rgb(166,173,200)",
"bright_gray": "rgb(186,194,222)",
"dark_gray": "rgb(54,54,84)",
}
def parse_json(html):
try:
# Replace 'undefined' with 'null' to make the JSON valid
json_data = html.replace("undefined", "null")
# Parse the JSON string
parsed_json = json.loads(json_data)
return parsed_json
except json.JSONDecodeError as e:
print(f"Error parsing JSON: {e}")
return None
except Exception as e:
print(f"Unexpected error: {e}")
return None
def extract_params_json(html, discriminator="__PARAMS__", index=0):
"""
For scripts like <script>window.__PARAMS__ = ...;</script>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Extract the JSON data from the HTML content that is stored in the
__PARAMS__ variable. This is typically used to parse the JSON data
from the HTML content of a page that contains a script tag with a
__PARAMS__ variable.
pass other discriminator if needed.
Args:
html (str): The HTML content from which to extract the JSON data.
Returns:
dict: The parsed JSON data, or None if no __PARAMS__ variable
was found, or if the JSON could not be parsed.
"""
try:
# Use Scrapy's Selector to locate the script tag containing "__PARAMS__"
sel = Selector(text=html)
# Extract the script content that contains 'window.__PARAMS__'
# selected_script = sel.xpath(f'//script[contains(text(), "{discriminator}")]/text()').extract_first()
scripts = sel.xpath(f'//script[contains(text(), "{discriminator}")]/text()')
if 0 <= index < len(scripts):
selected_script = scripts[index].get()
else:
return None
# Remove 'window.__PARAMS__ =' and trailing semicolon
json_data = (
selected_script.strip()
.replace(f"window.{discriminator} = ", "")
.rstrip(";")
.replace("\u200c", "")
.replace("\r\n", "")
)
# Replace 'undefined' with 'null' to make the JSON valid
json_data = json_data.replace("undefined", "null")
# Parse the JSON string
parsed_json = json.loads(json_data)
# Debugging: Print parsed JSON to verify it worked
# print("Parsed JSON:")
# print(json.dumps(parsed_json, indent=4))
return parsed_json
except json.JSONDecodeError as e:
print(f"Error parsing JSON: {e}")
return None
except Exception as e:
print(f"Unexpected error: {e}")
return None
def extract_script_with_id_json(html: str, discriminator: str, index: int = 0):
"""
For scripts like <script id="__NEXT_DATA__" type="application/json">
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Extract the JSON data from the HTML content that is stored in the
__PARAMS__ variable. This is typically used to parse the JSON data
from the HTML content of a page that contains a script tag with a
__PARAMS__ variable.
pass other discriminator if needed.
Args:
html (str): The HTML content from which to extract the JSON data.
Returns:
dict: The parsed JSON data, or None if no __PARAMS__ variable
was found, or if the JSON could not be parsed.
"""
try:
# Use Scrapy's Selector to locate the script tag containing "__PARAMS__"
sel = Selector(text=html)
# Extract the script content that contains 'window.__PARAMS__'
# selected_script = sel.xpath(f'//script[contains(text(), "{discriminator}")]/text()').extract_first()
# xpath = '//script[@id="__NEXT_DATA__" and @type="application/json"]/text()'
scripts = sel.xpath(
f'//script[@id="{discriminator}" and @type="application/json"]/text()'
)
if 0 <= index < len(scripts):
selected_script = scripts[index].get()
else:
return None
# Replace 'undefined' with 'null' to make the JSON valid
json_data = selected_script.replace("undefined", "null")
# Parse the JSON string
parsed_json = json.loads(json_data)
# Debugging: Print parsed JSON to verify it worked
# print("Parsed JSON:")
# print(json.dumps(parsed_json, indent=4))
return parsed_json
except json.JSONDecodeError as e:
print(f"Error parsing JSON: {e}")
return None
except Exception as e:
print(f"Unexpected error: {e}")
return None
def extract_with_xpath(html, pattern, delete_pattern=None, index=0):
# Depreciated. Please use extract_script_with_id_json or extract_params_json
"""
For scripts like <script id="__NEXT_DATA__" type="application/json">
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Extract the JSON data from the HTML content that is stored in the
__PARAMS__ variable. This is typically used to parse the JSON data
from the HTML content of a page that contains a script tag with a
__PARAMS__ variable.
pass other discriminator if needed.
Args:
html (str): The HTML content from which to extract the JSON data.
Returns:
dict: The parsed JSON data, or None if no __PARAMS__ variable
was found, or if the JSON could not be parsed.
"""
try:
# Use Scrapy's Selector to locate the script tag containing "__PARAMS__"
sel = Selector(text=html)
# Extract the script content that contains 'window.__PARAMS__'
# selected_script = sel.xpath(f'//script[contains(text(), "{discriminator}")]/text()').extract_first()
# xpath = '//script[@id="__NEXT_DATA__" and @type="application/json"]/text()'
scripts = sel.xpath(f"{pattern}/text()")
if 0 <= index < len(scripts):
selected_script = scripts[index].get()
else:
return None
# Remove 'window.__PARAMS__ =' and trailing semicolon
json_data = (
selected_script.strip()
.replace(f"{delete_pattern}", "")
.rstrip(";")
.replace("\u200c", "")
.replace("\r\n", "")
)
# Replace 'undefined' with 'null' to make the JSON valid
json_data = json_data.replace("undefined", "null")
# Parse the JSON string
parsed_json = json.loads(json_data)
# Debugging: Print parsed JSON to verify it worked
# print("Parsed JSON:")
# print(json.dumps(parsed_json, indent=4))
return parsed_json
except json.JSONDecodeError as e:
print(f"Error parsing JSON: {e}")
return None
except Exception as e:
print(f"Unexpected error: {e}")
return None
def rinse(string):
"""Remove non-printable characters from a string."""
illegals = "*'%$!(),;"
return "".join(c for c in string if c.isprintable() and c not in illegals)
def prettify(val, splitchar: str = "\t"):
"""
called as a beaupy.preprocessor to colour url and synopsis text.
Format a string value with color codes.
This function takes a string value, splits it by tab characters,
(optionally, alllows a custom split character),
and formats the first part as a title with a specific color code,
and the second part as a synopsis with another color code.
If splitting fails, it returns the value formatted as a title.
Args:
val (str): The string value to be prettified.
Returns:
str: The formatted string with color codes.
"""
try:
parts = val.split(splitchar)
title = f"[{catppuccin_mocha['bright_gray']}]{parts[0]}" + "[/]"
synopsis = f"[{catppuccin_mocha['pink']}]{parts[1]}" + "[/]"
return f"{title}\t{synopsis}"
except Exception:
return f"[{catppuccin_mocha['bright_gray']}]{val}[/]"
def list_prettify(my_list):
"""
Formats a list of three elements into a styled string using rich text formatting.
Parameters:
my_list (list): A list containing three elements to be formatted.
Returns:
str: A formatted string where the first two elements are displayed in green
and the third element is displayed in cyan on a new line with indentation.
"""
my_beaupystring = []
try:
one = my_list[0]
two = my_list[1]
three = my_list[2]
my_beaupystring = f"[#89B4FA]{one}, {two}[/][cyan]\n\t{three}[/cyan]"
return my_beaupystring
except Exception:
return my_list
def split(strng, sep, pos):
"""
Split a string into two parts at a specified position.
This function takes a string and splits it into two parts using a given
separator. It returns two substrings: the first containing the elements
before the specified position, and the second containing the elements
from the specified position onward.
Args:
strng (str): The input string to be split.
sep (str): The separator to use for splitting the string.
pos (int): The position at which to split the string.
Returns:
tuple: A tuple containing two strings. The first string is the
concatenation of elements before the specified position,
and the second string is the concatenation of elements
from the specified position onward.
"""
strng = strng.split(sep)
return sep.join(strng[:pos]), sep.join(strng[pos:])
def split_options(option_string):
if option_string is None:
return []
else:
return option_string.strip().split(" ")
packages/vinefeeder/src/vinefeeder/pretty.py
+74
View File
@@ -0,0 +1,74 @@
from rich.console import Console
from rich.panel import Panel
# thanks rlaphoenix for catppuccin_mocha color scheme
# Define the Catppuccin Mocha color scheme
catppuccin_mocha = {
"bg": "rgb(30,30,46)",
"text": "rgb(205,214,244)",
"text2": "rgb(162,169,193)",
"black": "rgb(69,71,90)",
"bright_black": "rgb(88,91,112)",
"red": "rgb(243,139,168)",
"green": "rgb(166,227,161)",
"yellow": "rgb(249,226,175)",
"blue": "rgb(137,180,250)",
"pink": "rgb(245,194,231)",
"cyan": "rgb(148,226,213)",
"gray": "rgb(166,173,200)",
"bright_gray": "rgb(186,194,222)",
"dark_gray": "rgb(54,54,84)"
}
def pretty_print():
console = Console()
# Combined text block with the color scheme
text_block = (
f"[{catppuccin_mocha['pink']}] _ ___ ____ __" "\n"
f"| | / (_)__ ___ / __/__ ___ ___/ /__ ____" "\n"
f"| |/ / / _ \\/ -_) _// -_) -_) _ / -_) __/" "\n"
f"|___/_/_//_/\\__/__/ \\__/\\__/\\_,_/\\__/_/ [/]" + "\n\n"
f"[{catppuccin_mocha['text2']}]© 2024-25 A_n_g_e_l_a[/]\n\n"
f"[{catppuccin_mocha['pink']}]driving ...[/]\n\n"
f"[{catppuccin_mocha['text2']}]░█▀▀░█▀█░█░█░▀█▀░█▀▀░█▀▄" + "\n"
f"[{catppuccin_mocha['text2']}]░█▀▀░█░█░▀▄▀░░█░░█▀▀░█░█" + "\n"
f"[{catppuccin_mocha['text2']}]░▀▀▀░▀░▀░░▀░░▀▀▀░▀▀▀░▀▀░" + "\n\n"
f"[{catppuccin_mocha['gray']}]© 2019-2025 rlaphoenix [/]" + "\n"
#f"[{catppuccin_mocha['blue']}]https://github.com/vinefeeder[/]"
)
#
instructions = (
f"[{catppuccin_mocha['pink']}]uv run vinefeeder --help for options[/]\n"
f"[{catppuccin_mocha['pink']}]uv run envied -? for options.[/]")
# Display the panel
panel = Panel(
text_block + "\n\n" + instructions,
title=f"[{catppuccin_mocha['pink']}]TwinVine[/]",
border_style=catppuccin_mocha["text2"],
padding=(0, 17, 1, 16),
style=f"on {catppuccin_mocha['bg']}",
expand=False
)
console.print(panel)
def create_clean_panel(content, title=""):
"""Create and return a clean panel with specified content and optional title."""
panel = Panel(
content,
title=title,
border_style=catppuccin_mocha["text2"],
padding=(1, 1, 1, 1),
style=f"on {catppuccin_mocha['bg']}",
expand=False
)
return panel
packages/vinefeeder/src/vinefeeder/requirements.txt
+9
View File
@@ -0,0 +1,9 @@
beaupy
click
httpx
jmespath
PyQt6
PyQt6_sip
PyYAML
rich
Scrapy
packages/vinefeeder/src/vinefeeder/selenium_bbc_uhd.py
+39
View File
@@ -0,0 +1,39 @@
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
import time
def fetch_uhd_links():
url = "https://www.bbc.co.uk/iplayer/help/questions/programme-availability/uhd-content"
chrome_options = Options()
chrome_options.add_argument("--headless") # Run in headless mode
chrome_options.add_argument("--disable-gpu")
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--window-size=1920,1080")
# Optional: Adjust path to chromedriver if not system-wide
driver = webdriver.Chrome(options=chrome_options)
driver.get(url)
time.sleep(5) # Wait for dynamic content to load
links = driver.find_elements(By.XPATH, "//a[contains(@href, '/iplayer/episodes/')]")
uhd_links = set()
for link in links:
href = link.get_attribute("href")
if href:
uhd_links.add(href)
driver.quit()
return sorted(uhd_links)
if __name__ == "__main__":
links = fetch_uhd_links()
for link in links:
print(link)
packages/vinefeeder/src/vinefeeder/services/ALL4/__init__.py
+320
View File
@@ -0,0 +1,320 @@
# channel 4 __init__.py
from vinefeeder.base_loader import BaseLoader
from vinefeeder.parsing_utils import extract_params_json, split_options
from rich.console import Console
import sys
import jmespath
console = Console()
class All4Loader(BaseLoader):
# global options
options = ""
def __init__(self):
"""
Initialize the All4Loader class with the provided headers.
Parameters:
None
Attributes:
options (str): Global options; later taken from service config.yaml
headers (dict): Global headers; may be overridden
"""
# self.options = ''
headers = {
"Accept": "*/*",
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64)",
"Origin": "https://www.channel4.com",
"Referer": "https://www.channel4.com/",
}
super().__init__(headers)
# entry point from Vinefeeder
def receive(
self, inx: None, search_term: None, category=None, hlg_status=False, opts=None
):
"""
First fetch for series titles matching all or part of search_term.
Uses inx, an int variable to switch:-
0 for greedy search using url
1 for direct url download
2 for category browse
3 for search with keyword
If search_url_entry is left blank vinefeeder generates a
a menu with 3 options:
- Greedy Search by URL
- Browse by Category
- Search by Keyword
results are forwarded to receive().
If inx == 1, do direct download using url.
If inx == 3, do keyword search.
If inx == 0, fetch videos using a greedy search or browse-category.
If inx == 2, fetch videos from a category url.
If an unknown error occurs, exit with code 0.
"""
# re-entry here for second time loses options settings
# so reset
if opts:
All4Loader.options = opts
self.options_list = split_options(All4Loader.options)
# direct download
if "http" in search_term and inx == 1:
try:
if self.options_list[0] == "":
command = ['uv', 'run', 'envied', "dl", "ALL4", search_term]
else:
command = ['uv', 'run', 'envied', "dl", *self.options_list, "ALL4", search_term]
self.runsubprocess(command)
return
except Exception as e:
print(
"Error downloading video:",
e,
f"Is {self.DOWNLOAD_ORCHESTRATOR} installed correctly via 'pip install {self.DOWNLOAD_ORCHESTRATOR}?",
)
return
# keyword search
elif inx == 3:
print(f"Searching for {search_term}")
return self.fetch_videos(search_term)
# POP-UP MENU alternates:
elif inx == 0:
# from greedy-search OR selecting Browse-category
# example: https://www.channel4.com/programmes/the-great-british-bake-off/on-demand/75228-001
# need a search keyword(s) from url
# split and select series name
search_term = search_term.split("/")[4]
# fetch_videos_by_category search_term may have other params to remove
if "?" in search_term:
search_term = search_term.split("?")[0].replace("-", " ")
return self.fetch_videos(search_term)
elif "http" in search_term and inx == 2:
self.category = category
self.fetch_videos_by_category(
search_term
) # search_term here holds a URL!!!
else:
print(f"Unknown error when searching for {search_term}")
return
def fetch_videos(self, search_term):
"""Fetch videos from Channel 4 using a search term.
Here the first search for series titles matches all or part of search_term.
The function will prepare the series data, matching the search term for display.
"""
# returns json as type String
url = f"https://all4nav.channel4.com/v1/api/search?q={search_term}&limit=100"
try:
html = self.get_data(url)
if "No Matches" in html:
print("Nothing found for that search; try again.")
sys.exit(0)
else:
parsed_data = self.parse_data(html) # to json
except Exception:
print(f"No valid data returned for {url}")
return
# Assuming that parsed_data has a 'results' key containing video data
if parsed_data and "results" in parsed_data:
for item in parsed_data["results"]:
series_name = item.get("brand", {}).get(
"websafeTitle", "Unknown Series"
)
episode = {
"title": item.get("brand", {}).get("title", "Unknown Title"),
"url": f"{item.get('brand', {}).get('href', '')}",
"synopsis": item.get("brand", {}).get(
"description", "No synopsis available."
),
}
self.add_episode(series_name, episode)
else:
print(f"No valid data returned for {url}")
return None
# List series and episodes using beaupy
selected_series = self.display_series_list()
if selected_series:
return self.second_fetch(
selected_series
) # Return URLs for selected episodes
return None
def second_fetch(self, selected):
"""
Given a selected series name, fetch its HTML and extract its episodes.
Or if given a direct url, fetch it and process for greedy download.
The function will prepare the series data for episode selection and display the final episode list.
It will return the URLs for the selected episodes.
"""
if "https" in selected: # direct url provided skip url preparation
url = selected
else:
url = self.get_selected_url(selected)
try:
myhtml = self.get_data(url=url)
except Exception as e:
print(f"[ALL4] Error occurred: {e}\nIs this title live on All4?\nThey list near-future titles (not yet downloadable). \n Exiting")
return
parsed_data = extract_params_json(myhtml)
self.clear_series_data() # Clear existing series data
# Extract the episodes from the parsed data of the selected series
if parsed_data and "initialData" in parsed_data:
series_name = (
parsed_data["initialData"]["brand"]["websafeTitle"] or "unknownTitle"
)
# Go through each episode in the selected series
episodes = parsed_data["initialData"]["brand"].get("episodes", [])
if not episodes:
print(
f"No episodes found for {series_name}\nThis is quite common with Channel 4. Exiting."
)
return
for item in episodes:
try:
episode = {
"series_no": item.get(
"seriesNumber", "00"
), # number exists or '00'
"title": item.get("title", "Title unknown"),
"url": item.get("hrefLink"),
"synopsis": item["summary"] or None,
}
except KeyError:
continue # Skip any episode that doesn't have the required information
self.add_episode(series_name, episode)
self.options_list = split_options(self.options)
if self.get_number_of_episodes(series_name) == 1:
item = self.get_series(series_name)[0]
try:
url = "https://www.channel4.com" + item["url"]
except Exception as e:
print(f"[ALL4] Error occurred:\nIs this title live on All4?\nThey list near-future titles (not yet downloadable). \n\nExiting!")
exit(1)
try:
if self.options_list[0] == "":
command = ['uv', 'run', 'envied', "dl", "ALL4", url]
else:
command = ['uv', 'run', 'envied', "dl", *self.options_list, "ALL4", url]
self.runsubprocess(command)
return None
except Exception as e:
print(
"Error downloading video:",
e,
f"Is {self.DOWNLOAD_ORCHESTRATOR} installed correctly via 'pip install {self.DOWNLOAD_ORCHESTRATOR}?",
)
return
self.prepare_series_for_episode_selection(
series_name
) # creates list of series; allows user selection of wanted series prepares an episode list over chosen series
selected_final_episodes = self.display_final_episode_list(
self.final_episode_data
)
# specific to ALL4
for item in selected_final_episodes:
url = item.split(",")[2].lstrip()
if url == "None":
print(f"No valid URL for {item.split(',')[1]}")
continue
url = "https://www.channel4.com" + url
try:
if self.options_list[0] == "":
command = ['uv', 'run', 'envied', "dl", "ALL4", url]
else:
command = ['uv', 'run', 'envied', "dl", *self.options_list, "ALL4", url]
self.runsubprocess(command)
except Exception as e:
print(
"Error downloading video:",
e,
"Is the package installed correctly? Try uv clean\nuv lock\n uv sysnc\n?",
)
return
return
def fetch_videos_by_category(self, browse_url):
"""
Fetches videos from a category (Channel 4 specific).
Args:
browse_url (str): URL of the category page.
Returns:
None
"""
beaupylist = []
try:
req = self.get_data(browse_url, headers=self.headers)
# Parse the __PARAMS__ data
init_data = extract_params_json(req, "__PARAMS__")
# Extract brand items
myjson = init_data["initialData"]["brands"]["items"]
# jmespath is an efficient json parser that searches complex json
# and, in this case, produces a simple dict from which
# res(ults) are more easily extracted.
# C4 specific
res = jmespath.search(
"""
[*].{
href: hrefLink,
label: labelText,
overlaytext: overlayText
} """,
myjson,
)
# Build the beaupylist for display
for i, item in enumerate(res):
label = item["label"]
overlaytext = item["overlaytext"]
beaupylist.append(
f"{i} {label}\n\t{overlaytext}"
) # \t used to split text later
except Exception as e:
print(f"Error fetching category data: {e}")
return
# call function in BaseLoader
found = self.display_beaupylist(beaupylist)
if found:
ind = found.split(" ")[0]
url = res[int(ind)]["href"]
# url may be for series or single Film
url = url.encode("utf-8", "ignore").decode().strip() # has spaces!
# process short-cut download or do greedy search on url
return self.process_received_url_from_category(url)
else:
print("No video selected.")
return
packages/vinefeeder/src/vinefeeder/services/ALL4/__init__.py.bak
+325
View File
@@ -0,0 +1,325 @@
# channel 4 __init__.py
from base_loader import BaseLoader
from parsing_utils import extract_params_json, split_options
from rich.console import Console
import sys
import jmespath
from vinefeeder import VineFeeder
console = Console()
class All4Loader(BaseLoader):
# global options
options = ""
def __init__(self):
"""
Initialize the All4Loader class with the provided headers.
Parameters:
None
Attributes:
options (str): Global options; later taken from service config.yaml
headers (dict): Global headers; may be overridden
"""
# self.options = ''
headers = {
"Accept": "*/*",
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64)",
"Origin": "https://www.channel4.com",
"Referer": "https://www.channel4.com/",
}
super().__init__(headers)
# entry point from Vinefeeder
def receive(
self, inx: None, search_term: None, category=None, hlg_status=False, opts=None
):
"""
First fetch for series titles matching all or part of search_term.
Uses inx, an int variable to switch:-
0 for greedy search using url
1 for direct url download
2 for category browse
3 for search with keyword
If search_url_entry is left blank vinefeeder generates a
a menu with 3 options:
- Greedy Search by URL
- Browse by Category
- Search by Keyword
results are forwarded to receive().
If inx == 1, do direct download using url.
If inx == 3, do keyword search.
If inx == 0, fetch videos using a greedy search or browse-category.
If inx == 2, fetch videos from a category url.
If an unknown error occurs, exit with code 0.
"""
# re-entry here for second time loses options settings
# so reset
if opts:
All4Loader.options = opts
self.options_list = split_options(All4Loader.options)
# direct download
if "http" in search_term and inx == 1:
try:
if self.options_list[0] == "":
command = [self.DOWNLOAD_ORCHESTRATOR, "dl", "ALL4", search_term]
else:
command = [self.DOWNLOAD_ORCHESTRATOR, "dl", *self.options_list, "ALL4", search_term]
self.runsubprocess(command)
return
except Exception as e:
print(
"Error downloading video:",
e,
f"Is {self.DOWNLOAD_ORCHESTRATOR} installed correctly via 'pip install {self.DOWNLOAD_ORCHESTRATOR}?",
)
return
# keyword search
elif inx == 3:
print(f"Searching for {search_term}")
return self.fetch_videos(search_term)
# POP-UP MENU alternates:
elif inx == 0:
# from greedy-search OR selecting Browse-category
# example: https://www.channel4.com/programmes/the-great-british-bake-off/on-demand/75228-001
# need a search keyword(s) from url
# split and select series name
search_term = search_term.split("/")[4]
# fetch_videos_by_category search_term may have other params to remove
if "?" in search_term:
search_term = search_term.split("?")[0].replace("-", " ")
return self.fetch_videos(search_term)
elif "http" in search_term and inx == 2:
self.category = category
self.fetch_videos_by_category(
search_term
) # search_term here holds a URL!!!
else:
print(f"Unknown error when searching for {search_term}")
return
def fetch_videos(self, search_term):
"""Fetch videos from Channel 4 using a search term.
Here the first search for series titles matches all or part of search_term.
The function will prepare the series data, matching the search term for display.
"""
# returns json as type String
url = f"https://all4nav.channel4.com/v1/api/search?q={search_term}&limit=100"
try:
html = self.get_data(url)
if "No Matches" in html:
print("Nothing found for that search; try again.")
sys.exit(0)
else:
parsed_data = self.parse_data(html) # to json
except Exception:
print(f"No valid data returned for {url}")
return
# Assuming that parsed_data has a 'results' key containing video data
if parsed_data and "results" in parsed_data:
for item in parsed_data["results"]:
series_name = item.get("brand", {}).get(
"websafeTitle", "Unknown Series"
)
episode = {
"title": item.get("brand", {}).get("title", "Unknown Title"),
"url": f"{item.get('brand', {}).get('href', '')}",
"synopsis": item.get("brand", {}).get(
"description", "No synopsis available."
),
}
self.add_episode(series_name, episode)
else:
print(f"No valid data returned for {url}")
return None
# List series and episodes using beaupy
selected_series = self.display_series_list()
if selected_series:
return self.second_fetch(
selected_series
) # Return URLs for selected episodes
return None
def second_fetch(self, selected):
"""
Given a selected series name, fetch its HTML and extract its episodes.
Or if given a direct url, fetch it and process for greedy download.
The function will prepare the series data for episode selection and display the final episode list.
It will return the URLs for the selected episodes.
"""
if "https" in selected: # direct url provided skip url preparation
url = selected
else:
url = self.get_selected_url(selected)
try:
myhtml = self.get_data(url=url)
except Exception as e:
print(f"[ALL4] Error occurred: {e}\nIs this title live on All4?\nThey list near-future titles (not yet downloadable). \n Exiting")
return
parsed_data = extract_params_json(myhtml)
self.clear_series_data() # Clear existing series data
# Extract the episodes from the parsed data of the selected series
if parsed_data and "initialData" in parsed_data:
series_name = (
parsed_data["initialData"]["brand"]["websafeTitle"] or "unknownTitle"
)
# Go through each episode in the selected series
episodes = parsed_data["initialData"]["brand"].get("episodes", [])
if not episodes:
print(
f"No episodes found for {series_name}\nThis is quite common with Channel 4. Exiting."
)
return
for item in episodes:
try:
episode = {
"series_no": item.get(
"seriesNumber", "00"
), # number exists or '00'
"title": item.get("title", "Title unknown"),
"url": item.get("hrefLink"),
"synopsis": item["summary"] or None,
}
except KeyError:
continue # Skip any episode that doesn't have the required information
self.add_episode(series_name, episode)
self.options_list = split_options(self.options)
if self.get_number_of_episodes(series_name) == 1:
item = self.get_series(series_name)[0]
try:
url = "https://www.channel4.com" + item["url"]
except Exception as e:
print(f"[ALL4] Error occurred:\nIs this title live on All4?\nThey list near-future titles (not yet downloadable). \n\nExiting!")
exit(1)
try:
if self.options_list[0] == "":
command = [self.DOWNLOAD_ORCHESTRATOR, "dl", "ALL4", url]
else:
command = [self.DOWNLOAD_ORCHESTRATOR, "dl", *self.options_list, "ALL4", url]
self.runsubprocess(command)
return None
except Exception as e:
print(
"Error downloading video:",
e,
f"Is {self.DOWNLOAD_ORCHESTRATOR} installed correctly via 'pip install {self.DOWNLOAD_ORCHESTRATOR}?",
)
return
self.prepare_series_for_episode_selection(
series_name
) # creates list of series; allows user selection of wanted series prepares an episode list over chosen series
selected_final_episodes = self.display_final_episode_list(
self.final_episode_data
)
# specific to ALL4
for item in selected_final_episodes:
url = item.split(",")[2].lstrip()
if url == "None":
print(f"No valid URL for {item.split(',')[1]}")
continue
url = "https://www.channel4.com" + url
try:
if self.options_list[0] == "":
command = [self.DOWNLOAD_ORCHESTRATOR, "dl", "ALL4", url]
else:
command = [self.DOWNLOAD_ORCHESTRATOR, "dl", *self.options_list, "ALL4", url]
self.runsubprocess(command)
except Exception as e:
print(
"Error downloading video:",
e,
f"Is {self.DOWNLOAD_ORCHESTRATOR} installed correctly via 'pip install {self.DOWNLOAD_ORCHESTRATOR}?",
)
return
return
def fetch_videos_by_category(self, browse_url):
"""
Fetches videos from a category (Channel 4 specific).
Args:
browse_url (str): URL of the category page.
Returns:
None
"""
beaupylist = []
try:
req = self.get_data(browse_url, headers=self.headers)
# Parse the __PARAMS__ data
init_data = extract_params_json(req, "__PARAMS__")
# Extract brand items
myjson = init_data["initialData"]["brands"]["items"]
# jmespath is an efficient json parser that searches complex json
# and, in this case, produces a simple dict from which
# res(ults) are more easily extracted.
# C4 specific
res = jmespath.search(
"""
[*].{
href: hrefLink,
label: labelText,
overlaytext: overlayText
} """,
myjson,
)
# Build the beaupylist for display
for i, item in enumerate(res):
label = item["label"]
overlaytext = item["overlaytext"]
beaupylist.append(
f"{i} {label}\n\t{overlaytext}"
) # \t used to split text later
except Exception as e:
print(f"Error fetching category data: {e}")
return
# call function in BaseLoader
found = self.display_beaupylist(beaupylist)
if found:
ind = found.split(" ")[0]
url = res[int(ind)]["href"]
# url may be for series or single Film
url = url.encode("utf-8", "ignore").decode().strip() # has spaces!
# process short-cut download or do greedy search on url
return self.process_received_url_from_category(url)
else:
print("No video selected.")
return
packages/vinefeeder/src/vinefeeder/services/ALL4/config.yaml
+17
View File
@@ -0,0 +1,17 @@
service_name: ALL4
options:
media_dict:
Film: 'https://www.channel4.com/categories/film'
Documentary: 'https://www.channel4.com/categories/documentaries'
Comedy: 'https://www.channel4.com/categories/comedy'
Drama: 'https://www.channel4.com/categories/drama'
Entertainment: 'https://www.channel4.com/categories/entertainment'
Lifestyle: 'https://www.channel4.com/categories/lifestyle'
News & Current Affairs: 'https://www.channel4.com/categories/news-current-affairs-and-politics'
Sport: 'https://www.channel4.com/categories/sport'
World Drama: 'https://www.channel4.com/categories/world-drama'
Box Sets: 'https://www.channel4.com/categories/boxsets'
packages/vinefeeder/src/vinefeeder/services/BBC/__init__.py
+486
View File
@@ -0,0 +1,486 @@
# BBC __init__.py
"""The BBC does not follow the same structures as most other srvices
The BBC does not use script = window.__PARAMS__ = ..."""
from vinefeeder.base_loader import BaseLoader
from vinefeeder.parsing_utils import extract_params_json, parse_json, split, split_options
from rich.console import Console
import jmespath
from scrapy.selector import Selector
console = Console()
"""
Note: The BBC is outrageously awkward. Do not use this as a template for other services!
"""
## BBC iPlayer sets a limit of 200 episodes per page
## This is a global variable to keep track of the page number
## if the number of episodes exceeds 200 set page to 2
## and increment by 1 for each subsequent page
PAGE = 1
class BbcLoader(BaseLoader):
HLG = None
options = None
def __init__(self):
# self.HLG = None
self.AVAILABLE_HLG = False
self.uhd_list = []
headers = {
"Accept": "*/*",
"User-Agent": "Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko",
"Origin": "https://www.bbc.com",
"Referer": "https://www.bbc.com/",
}
super().__init__(headers)
# Check if UHD content is available BBC specific
def check_uhd(self):
"""
Fetch the list of UHD content from the BBC iPlayer website.
Returns:
list: A list of URLs with UHD content.
"""
uhd_url = "https://www.bbc.co.uk/iplayer/help/questions/programme-availability/uhd-content"
html = self.get_data(uhd_url)
sel = Selector(text=html)
uhd_list = sel.xpath("(//ul)[9]//a/text()").getall()
#url_list = sel.xpath("(//ul)[9]//a/@href").getall()
return uhd_list
def receive(
self, inx: None, search_term: None, category=None, hlg_status=False, opts=None
):
if hlg_status:
BbcLoader.HLG = hlg_status
if opts:
BbcLoader.options = opts
self.options_list = split_options(BbcLoader.options)
# prepare in UHD list
self.uhd_list = self.check_uhd()
# standardize for comparison later
results = []
for item in self.uhd_list:
item = item.title()
results.append(item)
self.uhd_list = results
# standardize
if "https" in search_term:
pass
else:
search_term = search_term.strip().lower().title().replace(' ',' ')
"""
First fetch for series titles matching all or part of search_term.
Uses inx, an int variable to switch:-
0 for greedy search using url
1 for direct url download
2 for category browse
3 for search with keyword
If search_url_entry is left blank vinefeeder generates a
a menu with 3 options:
- Greedy Search by URL
- Browse by Category
- Search by Keyword
results are forwarded to receive().
If inx == 1, do direct download using url.
If inx == 3, do keyword search.
If inx == 0, fetch videos using a greedy search or browse-category.
If inx == 2, fetch videos from a category url.
If an unknown error occurs, exit with code 0.
"""
# TWO ALTERNATES FROM GUI-TEXT ENTRY; POULATED WITH KEYWORD OR URL
# direct download from url
if "http" in search_term and inx == 1:
# 'https://www.bbc.co.uk/iplayer/episode/m0021yyt'
# https://www.bbc.co.uk/iplayer/episode/m0023h9f/asia-series-1-1-beneath-the-waves
series_name = split(search_term, "/", 6)[1].split("-series")[0]
if series_name:
for hlg_item in self.uhd_list:
hlg_item = hlg_item.lower().replace(" ", "-")
if series_name in hlg_item:
self.AVAILABLE_HLG = True
break
search_term = split(search_term, "?", 1)[0].replace(
"episodes", "episode"
)
else:
self.AVAILABLE_HLG = False
self.options_list = split_options(BbcLoader.options)
if BbcLoader.HLG and self.AVAILABLE_HLG:
try:
command = (
[
'uv', 'run', 'envied',
"dl",
*self.options_list,
"--range",
"HLG",
"iP",
search_term,
]
if self.options_list
else ['uv', 'run', 'envied', "dl", "--range", "HLG", "iP", search_term]
)
self.runsubprocess(command)
except Exception as e:
print(
"Error downloading video:",
e,
"Is self.DOWNLOAD_ORCHESTRATOR installed correctly via 'pip install self.DOWNLOAD_ORCHESTRATOR?",
)
else:
try:
command = (
['uv', 'run', 'envied', "dl", *self.options_list, "iP", search_term]
if self.options_list
else ['uv', 'run', 'envied', "dl", "iP", search_term]
)
self.runsubprocess(command)
except Exception as e:
print(
"Error downloading video:",
e,
"Is self.DOWNLOAD_ORCHESTRATOR installed correctly via 'pip install self.DOWNLOAD_ORCHESTRATOR?",
)
return
# keyword search
elif inx == 3:
print(f"Searching for {search_term}")
for hlg_item in self.uhd_list:
if search_term in hlg_item:
self.AVAILABLE_HLG = True
break
return self.fetch_videos(search_term)
# ALTERNATIVES BELOW FROM POP-UP MENU
# from greedy-search OR selecting Browse-category
elif inx == 0 and "https" in search_term:
if "episode" in search_term:
# https://www.bbc.co.uk/iplayer/episodes/p09twdp8/showtrial?seriesId=m0023h9h
search_term = split(search_term, "/", 6)[1] # search_term
# fetch_videos_by_category search_term may have other params to remove
if "?" in search_term:
search_term = search_term.split("?")[0].replace("-", " ")
return self.fetch_videos(search_term)
elif "series" in search_term:
search_term = (
search_term.split("series")[0].replace("-", " ").strip()
)
return self.fetch_videos(search_term)
else:
print(
f"No search term found in {search_term}\nTry again with a series name in the url."
)
return
elif "http" in search_term and inx == 2:
self.category = category
self.fetch_videos_by_category(
search_term
) # search_term here holds a URL!!!
else:
print(f"Unknown error when searching for {search_term}")
return
def fetch_videos(self, search_term):
"""Fetch videos from BBC using a search term."""
url = "https://ibl.api.bbc.co.uk/ibl/v1/new-search"
# url = f"https://search.api.bbci.co.uk/formula/iplayer-ibl-root?q={search_term}&apikey=D2FgtcTxGqqIgLsfBWTJdrQh2tVdeaAp&seqId=0582e0f0-b911-11ee-806c-11c6c885ab56"
params = {
"q": search_term,
"rights": "web",
#'mixin': 'live'
}
try:
html = self.get_data(url, self.headers, params)
if "new_search" not in html:
print("Nothing found for that search; try again.")
return
else:
parsed_data = parse_json(html) # to json
except Exception:
print(f"No valid data returned for {url}")
return
if parsed_data and "new_search" in parsed_data:
for item in parsed_data["new_search"]["results"]:
series_name = item.get(
"title", "Unknown Title"
) # BBC has a 'title' key for series names
episode = {
"title": item.get("title", "Unknown Title"),
"url": f"{item.get('id', '')}",
"synopsis": item.get("synopsis", "No synopsis available."),
}
self.add_episode(series_name, episode)
else:
print(f"No data found for {search_term}")
# List series and episodes using beaupy
selected_series = self.display_series_list()
if selected_series:
return self.second_fetch(
selected_series
) # Return URLs for selected episodes
return None
def second_fetch(self, selected_series):
# 'selected' is search term
"""
Given a selected series name, fetch its HTML and extract its episodes.
need programme Id for url lookup
Or if given a direct url, fetch it and process for greedy download.
The function will prepare the series data for episode selection and display the final episode list.
It will return the URLs for the selected episodes.
"""
selected = selected_series
# option for direct url
if "https" in selected: # direct url provided skip url preparation
url = selected # 'https://www.bbc.co.uk/iplayer/episodes/m000mfhl'
# url = f"https://www.bbc.co.uk/iplayer/episode/{url.split('/')[-1]}"
# option for search
else:
url = self.get_selected_url(
selected
) # Here url is hhtps://www.bbc.co.uk/iplayer/episodes/m000mfhl
url = url.split("/")[-1]
# thanks to kenyard for help with url preparation
# https://www.bbc.co.uk/iplayer/episodes/b007y6k8/silent-witness
myurl = f"https://ibl.api.bbci.co.uk/ibl/v1/episodes/{url}?rights=mobile&availability=available&api_key=D2FgtcTxGqqIgLsfBWTJdrQh2tVdeaAp"
myhtml = self.get_data(url=myurl, headers=self.headers)
parsed_data = parse_json(myhtml)
SINGLE = True
if parsed_data["episodes"] == []:
SINGLE = False
url = f"https://ibl.api.bbci.co.uk/ibl/v1/programmes/{url}/episodes?rights=mobile&availability=available&page={PAGE}&per_page=200&api_key=D2FgtcTxGqqIgLsfBWTJdrQh2tVdeaAp"
myhtml = self.get_data(url=url, headers=self.headers)
parsed_data = parse_json(myhtml)
# testing
#file = open("init_data.json", "w")
#file.write(json.dumps(parsed_data))
#file.close()
#console.print_json(data=parsed_data)
self.clear_series_data() # Clear existing series data
self.options_list = split_options(self.options)
# Extract the episodes from the parsed data of the selected series
if parsed_data and "programme_episodes" in parsed_data and not SINGLE:
try:
series_name = (
parsed_data["programme_episodes"]["programme"]["title"]
or "unknownTitle"
)
except KeyError:
print(f"No data found for {url}")
return
except Exception:
print(f"No episodes currently available for {url}")
print("Try again but be sure to use the series utmost top level url!")
return
# Go through each episode in the selected series and add it to the series data
episodes = parsed_data.get("programme_episodes").get("elements")
for item in episodes:
try:
series_no = item["subtitle"].split(":")[0].split(" ")[1]
if int(series_no):
pass
else:
continue
episode = {
"series_no": series_no,
# 'title' is episode number here, some services use descriptive text
"title": item["subtitle"].split(":")[
1
], # .split(' ')[-1] or '01',
"url": "https://www.bbc.co.uk/iplayer/episode/" + item["id"],
"synopsis": item["synopses"]["small"] or None,
}
except Exception:
try:
episode = {
"series_no": 100, # special or one-off'
# 'title' is episode number here, some services use descriptive text
"title": item["subtitle"], # could be date
"url": "https://www.bbc.co.uk/iplayer/episode/"
+ item["id"],
"synopsis": item["synopses"]["small"] or None,
}
except KeyError as e:
print(f"Error: {e}")
continue # Skip any episode that doesn't have the required information
self.add_episode(series_name, episode)
# Single episode expedite download
else:
series_name = parsed_data["episodes"][0]["title"]
url = f"https://www.bbc.co.uk/iplayer/episode/{parsed_data['episodes'][0]['id']}/{parsed_data['episodes'][0]['title']}"
# check for UHD content
for hlg_item in self.uhd_list:
if series_name.lower() in hlg_item:
self.AVAILABLE_HLG = True
break
# self.options_list = split_options(self.options)
try:
if BbcLoader.HLG and self.AVAILABLE_HLG:
command = (
[
'uv', 'run', 'envied',
"dl",
*self.options_list,
"--range",
"HLG",
"iP",
url.replace(" ", "-"),
]
)
else:
command = (['uv', 'run', 'envied', "dl", *self.options_list, "iP", url.replace(" ", "-")])
self.runsubprocess(command)
except Exception as e:
print(
"Error downloading video:",
e,
"Is self.DOWNLOAD_ORCHESTRATOR installed correctly via 'pip install self.DOWNLOAD_ORCHESTRATOR?",
)
return
self.prepare_series_for_episode_selection(
series_name
) # creates list of series; allows user selection of wanted series prepares an episode list over chosen series
selected_final_episodes = self.display_final_episode_list(
self.final_episode_data
)
# specific to BBC
# self.options_list = split_options(self.options)
for item in selected_final_episodes:
# check for UHD content
for hlg_item in self.uhd_list:
if series_name.lower() in hlg_item:
self.AVAILABLE_HLG = True
break
for part in item.split(","):
if "https" in part:
url = part.strip()
break
if BbcLoader.HLG and self.AVAILABLE_HLG:
command = (
['uv', 'run', 'envied', "dl", *self.options_list, "--range", "HLG", "iP", url.replace(" ", "-")]
)
else:
command = (['uv', 'run', 'envied', "dl", *self.options_list, "iP", url.replace(" ", "-")])
self.runsubprocess(command)
return
def fetch_videos_by_category(self, browse_url):
"""
Fetches videos from a category (Channel 4 specific).
Args:
browse_url (str): URL of the category page.
Returns:
None
"""
beaupylist = []
req = self.client.get(browse_url, headers=self.headers)
init_data = extract_params_json(req.content.decode(), "__IPLAYER_REDUX_STATE__")
"""
# for bug fixing
console.print_json(data=init_data)
file = open("init_data.json", "w")
file.write(json.dumps(init_data))
file.close()
"""
myjson = init_data
# console.print_json(data=myjson)
category = myjson["bundles"][0]["id"]
res = jmespath.search(
"""
bundles[*].entities[*].episode[].{
href: id,
label: title.default,
overlaytext: synopsis.small
}
""",
myjson,
)
# console.print_json(data=res)
seen_titles = set() # To track seen titles
beaupylist = list()
# Build the beaupylist for display but check for duplicates
for i, item in enumerate(res):
label = item["label"] # Title of the programme
overlaytext = item["overlaytext"] # Synopsis of the programme
formatted_item = f"{i} {label}\n\t{overlaytext}"
if (
label not in seen_titles
): # Check if the title has already been processed
seen_titles.add(label)
beaupylist.append(formatted_item)
found = self.display_beaupylist(beaupylist)
if found:
if category not in [
"films",
"featured-category-films",
"featured-films",
"film",
"movie",
]:
# extract search_term for greedy search
search_term = found.split("\n\t")[0][2:]
return self.process_received_url_from_category(search_term)
else:
ind = found.split(" ")[0]
url = res[int(ind)]["href"]
# url may be for series or single Film
url = f"https://www.bbc.co.uk/iplayer/episode/{url}"
#url = url.encode("utf-8", "ignore").decode().strip() # has spaces!
# process short-cut download or do greedy search on url
return self.process_received_url_from_category(url)
else:
print("No video selected.")
return
packages/vinefeeder/src/vinefeeder/services/BBC/config.yaml
+22
View File
@@ -0,0 +1,22 @@
service: BBC
loader_class: BbcLoader
description: A UK Public Sevice broadcaster
hlg_status: True
options:
media_dict:
Film: 'https://www.bbc.co.uk/iplayer/categories/films/featured'
Documentary: 'https://www.bbc.co.uk/iplayer/categories/documentaries/featured'
Comedy: 'https://www.bbc.co.uk/iplayer/categories/comedy/featured'
Drama: 'https://www.bbc.co.uk/iplayer/categories/drama-and-soaps/featured'
Entertainment: 'https://www.bbc.co.uk/iplayer/categories/entertainment/featured'
Lifestyle: 'https://www.channel4.com/categories/lifestyle'
News: 'https://www.bbc.co.uk/iplayer/categories/news/featured'
Sport: 'https://www.bbc.co.uk/iplayer/categories/sport/featured'
Lifestyle: 'https://www.bbc.co.uk/iplayer/categories/lifestyle/featured'
History: 'https://www.bbc.co.uk/iplayer/categories/history/featured'
Science/Nature: 'https://www.bbc.co.uk/iplayer/categories/science-and-nature/featured'
packages/vinefeeder/src/vinefeeder/services/ITVX/__init__.py
+323
View File
@@ -0,0 +1,323 @@
from vinefeeder.base_loader import BaseLoader
from vinefeeder.parsing_utils import rinse, split_options, extract_script_with_id_json
from rich.console import Console
import jmespath
import re
console = Console()
class ItvxLoader(BaseLoader):
options = ""
def __init__(self):
headers = {
"Accept": "*/*",
"user-agent": "Dalvik/2.9.8 (Linux; U; Android 9.9.2; ALE-L94 Build/NJHGGF)",
"Origin": "https://www.itv.com",
"Referer": "https://www.itv.com/",
}
super().__init__(headers)
def receive(
self, inx: None, search_term: None, category=None, hlg_status=False, opts=None
):
"""
First fetch for series titles matching all or part of search_term.
Uses inx, an int variable to switch:-
0 for greedy search using url
1 for direct url download
2 for category browse
3 for search with keyword
If search_url_entry is left blank vinefeeder generates a
a menu with 3 options:
- Greedy Search by URL
- Browse by Category
- Search by Keyword
results are forwarded to receive().
If inx == 1, do direct download using url.
If inx == 3, do keyword search.
If inx == 0, fetch videos using a greedy search or browse-category.
If inx == 2, fetch videos from a category url.
If an unknown error occurs, exit with code 0.
"""
if opts:
ItvxLoader.options = opts
self.options_list = split_options(ItvxLoader.options)
# direct download
if "http" in search_term and inx == 1:
self.options_list = split_options(self.options)
try:
if self.options_list[0] == "":
command = ['uv', 'run', 'envied', "dl", "ITVX", search_term]
else:
command = ['uv', 'run', 'envied', "dl", *self.options_list, "ITVX", search_term]
self.runsubprocess(command)
except Exception as e:
print(
"Error downloading video:",
e,
"Is the package for vinefeeder/envied installed correctly ?",
)
return
# keyword search
elif inx == 3:
print(f"Searching for {search_term}")
return self.fetch_videos(search_term)
# ALTERNATIVES BELOW FROM POP-UP MENU
elif inx == 0:
# from greedy-search OR selecting Browse-category
# example: https://www.channel4.com/programmes/the-great-british-bake-off/on-demand/75228-001
# need a search keyword(s) from url
# split and select series name
search_term = search_term.split("/")[4].replace("-", " ")
# fetch_videos_by_category search_term may have other params to remove
if "?" in search_term:
search_term = search_term.split("?")[0].replace("-", " ")
return self.fetch_videos(search_term)
elif "http" in search_term and inx == 2:
self.category = category
self.fetch_videos_by_category(
search_term
) # search_term here holds a URL!!!
else:
print(f"Unknown error when searching for {search_term}")
return
def fetch_videos(self, search_term):
"""Fetch videos from ITV using a search term."""
headers = {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate, br",
"Host": "textsearch.prd.oasvc.itv.com",
"Origin": "https://www.itv.com",
"Referer": "https://www.itv.com/",
"user-agent": "Dalvik/2.9.8 (Linux; U; Android 9.9.2; ALE-L94 Build/NJHGGF)",
}
url = f"https://textsearch.prd.oasvc.itv.com/search?broadcaster=itv&channelType=simulcast&featureSet=clearkey,outband-webvtt,hls,aes,playready,widevine,fairplay,bbts,progressive,hd,rtmpe&platform=dotcom&pretx=true&query={search_term}&size=24"
html = self.get_data(url, headers)
parsed_data = self.parse_data(html)
# select only from FREE tier
res = jmespath.search(
"""
results[?data.tier=='FREE'].{
api1: data.legacyId.officialFormat,
title: data.[programmeTitle, filmTitle, specialTitle],
synopsis: data.synopsis
} """,
parsed_data,
)
for item in res:
api1 = item["api1"].replace("/", "a")
title = next((value for value in item["title"] if value is not None), "")
title = title.replace(" ", "-")
url = f"https://www.itv.com/watch/{title}/{api1}"
episode = {
"title": item.get("title", "Unknown Title"),
"url": url,
"synopsis": item.get("synopsis", "No synopsis available."),
}
self.add_episode(title, episode)
# List series with search_term using beaupy
selected_series = self.display_series_list()
if selected_series:
# one series selected
return self.second_fetch(selected_series)
return None
def second_fetch(self, selected):
"""
Given a selected series name, fetch its HTML and extract its episodes.
Or if given a direct url, fetch it and process for greedy download.
The function will prepare the series data for episode selection and display the final episode list.
It will return the URLs for the selected episodes.
"""
if "https" in selected: # direct url provided skip url preparation
url = selected
else:
url = self.get_selected_url(selected)
try:
headers = {
"authority": "www.itv.com",
"user-agent": "Dalvik/2.9.8 (Linux; U; Android 9.9.2; ALE-L94 Build/NJHGGF)",
}
myhtml = self.get_data(url=url, headers=headers)
except Exception:
print(f"No valid data at {url} found.\n Exiting")
return
parsed_data = extract_script_with_id_json(myhtml, "__NEXT_DATA__", 0)
self.clear_series_data() # Clear existing series data
"""
# debug
console.print_json(data=parsed_data)
f = open('itv.json', 'w')
f.write(myhtml)
f.close()"""
url_data = jmespath.search(
"""
query.{
programmeSlug: programmeSlug
programmeId: programmeId
}""",
parsed_data,
)
programmeSlug = url_data["programmeSlug"]
programmeId = url_data["programmeId"]
mytitles = jmespath.search("props.pageProps", parsed_data)
res = jmespath.search(
"""
seriesList[].titles[].{
episode: episode
eptitle: episodeTitle
series: series
magniurl: playlistUrl
description: description
episodeId: episodeId
letterA: encodedEpisodeId.letterA
contentInfo: contentInfo
channel: channel
} """,
mytitles,
)
for item in res:
try:
# ITVX sometimes has a word in place of a series number, correct for this.
series_no = 100 # set default
if item["series"]: # check it is a digir
try:
series_no = int(item["series"]) # Try to cast to int
except ValueError:
pass # Leave series_no as 100 if casting fails
episode = {
"series_no": series_no,
"title": f"{item['episode'] or ''}:{item['eptitle'] or ''}",
"url": f"https://www.itv.com/watch/{programmeSlug}/{programmeId}/{item['letterA']}",
"synopsis": rinse(item["description"]) or None,
}
except KeyError:
continue # Skip any episode that doesn't have the required information
self.add_episode(programmeSlug, episode)
self.prepare_series_for_episode_selection(
programmeSlug
) # creates list of series; allows user selection of wanted series prepares an episode list over chosen series
selected_final_episodes = self.display_final_episode_list(
self.final_episode_data
)
self.options_list = split_options(self.options)
for item in selected_final_episodes:
url = item.split(",")[2].lstrip()
if 'http' not in url:
url_pattern = r'https?://[^\s,]+'
match = re.search(url_pattern, item)
if match:
url = match.group()
try:
if self.options_list[0] == "":
command = ['uv', 'run', 'envied', "dl", "ITVX", url]
else:
command = ['uv','run', 'envied', "dl", *self.options_list, "ITVX", url]
self.runsubprocess(command)
except Exception as e:
print(
"Error downloading video:",
e,
"Is package/envied installed correctly?",
)
return None
def fetch_videos_by_category(self, browse_url):
"""
Fetches videos from a category (ITVX specific).
Args:
browse_url (str): URL of the category page.
Returns:
None
"""
beaupylist = [] # hold beaupy data for display and programme selection
headers = {
"host": "www.itv.com",
"user-agent": "Dalvik/2.9.8 (Linux; U; Android 9.9.2; ALE-L94 Build/NJHGGF)",
}
try:
myhtml = self.get_data(browse_url, headers=headers)
parsed_data = extract_script_with_id_json(myhtml, "__NEXT_DATA__", 0)
# jmespath is an efficient json parser that searches complex json
# and, in this case, produces a simple dict from which
# res(ults) are more easily extracted.
# ITVX specific
mytitles = jmespath.search("props.pageProps.collection.shows", parsed_data)
res = jmespath.search(
"""
[].{
title: titleSlug,
programmeId: encodedProgrammeId.letterA,
episodeId: encodedEpisodeId.letterA,
synopsis: description
}
""",
mytitles,
)
# Add the URL to each entry in res
# ITV does not have a url in json data thus need to build.
for entry in res:
entry["url"] = (
f"https://www.itv.com/watch/{entry['title']}/{entry['programmeId']}/{entry['episodeId']}"
)
# Build the beaupylist for display
for i, item in enumerate(res):
title = (item["title"].replace("-", " ")).title()
url = item["url"]
synopsis = item["synopsis"]
beaupylist.append(
f"{i} {title.replace('_',' ')}\n\t{synopsis}"
) # \t used to split text later
except Exception as e:
print(f"Error fetching category data: {e}")
return
# call function in BaseLoader
found = self.display_beaupylist(beaupylist)
if found:
ind = found.split(" ")[0]
url = res[int(ind)]["url"]
# url may be for series or single Film
url = url.encode("utf-8", "ignore").decode().strip() # has spaces!
# process short-cut download or do greedy search on url
return self.process_received_url_from_category(url)
else:
print("No video selected.")
return
packages/vinefeeder/src/vinefeeder/services/ITVX/__init__.py.bak
+323
View File
@@ -0,0 +1,323 @@
from base_loader import BaseLoader
from parsing_utils import rinse, split_options, extract_script_with_id_json
from rich.console import Console
import jmespath
import re
console = Console()
class ItvxLoader(BaseLoader):
options = ""
def __init__(self):
headers = {
"Accept": "*/*",
"user-agent": "Dalvik/2.9.8 (Linux; U; Android 9.9.2; ALE-L94 Build/NJHGGF)",
"Origin": "https://www.itv.com",
"Referer": "https://www.itv.com/",
}
super().__init__(headers)
def receive(
self, inx: None, search_term: None, category=None, hlg_status=False, opts=None
):
"""
First fetch for series titles matching all or part of search_term.
Uses inx, an int variable to switch:-
0 for greedy search using url
1 for direct url download
2 for category browse
3 for search with keyword
If search_url_entry is left blank vinefeeder generates a
a menu with 3 options:
- Greedy Search by URL
- Browse by Category
- Search by Keyword
results are forwarded to receive().
If inx == 1, do direct download using url.
If inx == 3, do keyword search.
If inx == 0, fetch videos using a greedy search or browse-category.
If inx == 2, fetch videos from a category url.
If an unknown error occurs, exit with code 0.
"""
if opts:
ItvxLoader.options = opts
self.options_list = split_options(ItvxLoader.options)
# direct download
if "http" in search_term and inx == 1:
self.options_list = split_options(self.options)
try:
if self.options_list[0] == "":
command = [self.DOWNLOAD_ORCHESTRATOR, "dl", "ITVX", search_term]
else:
command = [self.DOWNLOAD_ORCHESTRATOR, "dl", *self.options_list, "ITVX", search_term]
self.runsubprocess(command)
except Exception as e:
print(
"Error downloading video:",
e,
"Is self.DOWNLOAD_ORCHESTRATOR installed correctly via 'pip install self.DOWNLOAD_ORCHESTRATOR?",
)
return
# keyword search
elif inx == 3:
print(f"Searching for {search_term}")
return self.fetch_videos(search_term)
# ALTERNATIVES BELOW FROM POP-UP MENU
elif inx == 0:
# from greedy-search OR selecting Browse-category
# example: https://www.channel4.com/programmes/the-great-british-bake-off/on-demand/75228-001
# need a search keyword(s) from url
# split and select series name
search_term = search_term.split("/")[4].replace("-", " ")
# fetch_videos_by_category search_term may have other params to remove
if "?" in search_term:
search_term = search_term.split("?")[0].replace("-", " ")
return self.fetch_videos(search_term)
elif "http" in search_term and inx == 2:
self.category = category
self.fetch_videos_by_category(
search_term
) # search_term here holds a URL!!!
else:
print(f"Unknown error when searching for {search_term}")
return
def fetch_videos(self, search_term):
"""Fetch videos from ITV using a search term."""
headers = {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate, br",
"Host": "textsearch.prd.oasvc.itv.com",
"Origin": "https://www.itv.com",
"Referer": "https://www.itv.com/",
"user-agent": "Dalvik/2.9.8 (Linux; U; Android 9.9.2; ALE-L94 Build/NJHGGF)",
}
url = f"https://textsearch.prd.oasvc.itv.com/search?broadcaster=itv&channelType=simulcast&featureSet=clearkey,outband-webvtt,hls,aes,playready,widevine,fairplay,bbts,progressive,hd,rtmpe&platform=dotcom&pretx=true&query={search_term}&size=24"
html = self.get_data(url, headers)
parsed_data = self.parse_data(html)
# select only from FREE tier
res = jmespath.search(
"""
results[?data.tier=='FREE'].{
api1: data.legacyId.officialFormat,
title: data.[programmeTitle, filmTitle, specialTitle],
synopsis: data.synopsis
} """,
parsed_data,
)
for item in res:
api1 = item["api1"].replace("/", "a")
title = next((value for value in item["title"] if value is not None), "")
title = title.replace(" ", "-")
url = f"https://www.itv.com/watch/{title}/{api1}"
episode = {
"title": item.get("title", "Unknown Title"),
"url": url,
"synopsis": item.get("synopsis", "No synopsis available."),
}
self.add_episode(title, episode)
# List series with search_term using beaupy
selected_series = self.display_series_list()
if selected_series:
# one series selected
return self.second_fetch(selected_series)
return None
def second_fetch(self, selected):
"""
Given a selected series name, fetch its HTML and extract its episodes.
Or if given a direct url, fetch it and process for greedy download.
The function will prepare the series data for episode selection and display the final episode list.
It will return the URLs for the selected episodes.
"""
if "https" in selected: # direct url provided skip url preparation
url = selected
else:
url = self.get_selected_url(selected)
try:
headers = {
"authority": "www.itv.com",
"user-agent": "Dalvik/2.9.8 (Linux; U; Android 9.9.2; ALE-L94 Build/NJHGGF)",
}
myhtml = self.get_data(url=url, headers=headers)
except Exception:
print(f"No valid data at {url} found.\n Exiting")
return
parsed_data = extract_script_with_id_json(myhtml, "__NEXT_DATA__", 0)
self.clear_series_data() # Clear existing series data
"""
# debug
console.print_json(data=parsed_data)
f = open('itv.json', 'w')
f.write(myhtml)
f.close()"""
url_data = jmespath.search(
"""
query.{
programmeSlug: programmeSlug
programmeId: programmeId
}""",
parsed_data,
)
programmeSlug = url_data["programmeSlug"]
programmeId = url_data["programmeId"]
mytitles = jmespath.search("props.pageProps", parsed_data)
res = jmespath.search(
"""
seriesList[].titles[].{
episode: episode
eptitle: episodeTitle
series: series
magniurl: playlistUrl
description: description
episodeId: episodeId
letterA: encodedEpisodeId.letterA
contentInfo: contentInfo
channel: channel
} """,
mytitles,
)
for item in res:
try:
# ITVX sometimes has a word in place of a series number, correct for this.
series_no = 100 # set default
if item["series"]: # check it is a digir
try:
series_no = int(item["series"]) # Try to cast to int
except ValueError:
pass # Leave series_no as 100 if casting fails
episode = {
"series_no": series_no,
"title": f"{item['episode'] or ''}:{item['eptitle'] or ''}",
"url": f"https://www.itv.com/watch/{programmeSlug}/{programmeId}/{item['letterA']}",
"synopsis": rinse(item["description"]) or None,
}
except KeyError:
continue # Skip any episode that doesn't have the required information
self.add_episode(programmeSlug, episode)
self.prepare_series_for_episode_selection(
programmeSlug
) # creates list of series; allows user selection of wanted series prepares an episode list over chosen series
selected_final_episodes = self.display_final_episode_list(
self.final_episode_data
)
self.options_list = split_options(self.options)
for item in selected_final_episodes:
url = item.split(",")[2].lstrip()
if 'http' not in url:
url_pattern = r'https?://[^\s,]+'
match = re.search(url_pattern, item)
if match:
url = match.group()
try:
if self.options_list[0] == "":
command = [self.DOWNLOAD_ORCHESTRATOR, "dl", "ITVX", url]
else:
command = [self.DOWNLOAD_ORCHESTRATOR, "dl", *self.options_list, "ITVX", url]
self.runsubprocess(command)
except Exception as e:
print(
"Error downloading video:",
e,
"Is self.DOWNLOAD_ORCHESTRATOR installed correctly via 'pip install self.DOWNLOAD_ORCHESTRATOR?",
)
return None
def fetch_videos_by_category(self, browse_url):
"""
Fetches videos from a category (ITVX specific).
Args:
browse_url (str): URL of the category page.
Returns:
None
"""
beaupylist = [] # hold beaupy data for display and programme selection
headers = {
"host": "www.itv.com",
"user-agent": "Dalvik/2.9.8 (Linux; U; Android 9.9.2; ALE-L94 Build/NJHGGF)",
}
try:
myhtml = self.get_data(browse_url, headers=headers)
parsed_data = extract_script_with_id_json(myhtml, "__NEXT_DATA__", 0)
# jmespath is an efficient json parser that searches complex json
# and, in this case, produces a simple dict from which
# res(ults) are more easily extracted.
# ITVX specific
mytitles = jmespath.search("props.pageProps.collection.shows", parsed_data)
res = jmespath.search(
"""
[].{
title: titleSlug,
programmeId: encodedProgrammeId.letterA,
episodeId: encodedEpisodeId.letterA,
synopsis: description
}
""",
mytitles,
)
# Add the URL to each entry in res
# ITV does not have a url in json data thus need to build.
for entry in res:
entry["url"] = (
f"https://www.itv.com/watch/{entry['title']}/{entry['programmeId']}/{entry['episodeId']}"
)
# Build the beaupylist for display
for i, item in enumerate(res):
title = (item["title"].replace("-", " ")).title()
url = item["url"]
synopsis = item["synopsis"]
beaupylist.append(
f"{i} {title.replace('_',' ')}\n\t{synopsis}"
) # \t used to split text later
except Exception as e:
print(f"Error fetching category data: {e}")
return
# call function in BaseLoader
found = self.display_beaupylist(beaupylist)
if found:
ind = found.split(" ")[0]
url = res[int(ind)]["url"]
# url may be for series or single Film
url = url.encode("utf-8", "ignore").decode().strip() # has spaces!
# process short-cut download or do greedy search on url
return self.process_received_url_from_category(url)
else:
print("No video selected.")
return
packages/vinefeeder/src/vinefeeder/services/ITVX/config.yaml
+22
View File
@@ -0,0 +1,22 @@
service_name: ITVX
options: -q 720p
media_dict:
Films: https://www.itv.com/watch/collections/make-it-a-movie-night/2CIASIVXkb4A6R1XxJ4s1f
Top Picks: https://www.itv.com/watch/collections/top-picks/51Ry6KaT5pg9HYDJ8AqPwk
Gritty Thrillers: https://www.itv.com/watch/collections/gritty-thrillers/5lTuwNT5hAkUyQJdPabiGT
True Life: https://www.itv.com/watch/collections/true-life-drama/1nYAN4ipGU6L0qmgimh1lE
New: https://www.itv.com/watch/collections/fresh-in/7K0pfBiDFvOdBeHr7SnDzo
Comedy Drama: https://www.itv.com/watch/collections/comedy-drama/5lADfSZ7PP5dNeJ6BCW5Gb
Comedy: https://www.itv.com/watch/collections/show-me-the-funny/33idcDIeV32Cd3V7czYDeP
Entertainment & Reality: https://www.itv.com/watch/collections/top-picks/4qcfuXnvuom6zss67k7e6p
Football: https://www.itv.com/watch/collections/football/6iS0VOrNqLWQNCLL5T0nxL
Sport Top Picks: https://www.itv.com/watch/collections/top-picks/4CwpvlYDmVs0jumu0tA0Iu
Sport - Fresh In: https://www.itv.com/watch/collections/fresh-in/2h7nZMeKipItXfeVcGCGlo
Boxsets: https://www.itv.com/watch/collections/unmissable-boxsets/5vk4Kkk8zRTiG3cWI698fR
Kids - Top Picks: https://www.itv.com/watch/collections/kids-top-picks/4ZSxLvMbULBxNbA9AqY96R
packages/vinefeeder/src/vinefeeder/services/MY5/__init__.py
+281
View File
@@ -0,0 +1,281 @@
from vinefeeder.base_loader import BaseLoader
import jmespath
from rich.console import Console
from vinefeeder.parsing_utils import parse_json, split_options
import re
console = Console()
class My5Loader(BaseLoader):
options = ""
def __init__(self):
headers = {
"Accept": "*/*",
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64)",
"Origin": "https://www.channel5.com",
"Referer": "https://www.channel5.com/",
}
super().__init__(headers)
def receive(
self, inx: None, search_term: None, category=None, hlg_status=False, opts=None
):
"""
First fetch for series titles matching all or part of search_term.
Uses inx, an int variable to switch:-
0 for greedy search using url
1 for direct url download
2 for category browse
3 for search with keyword
If search_url_entry is left blank vinefeeder generates a
a menu with 3 options:
- Greedy Search by URL
- Browse by Category
- Search by Keyword
results are forwarded to receive().
If inx == 1, do direct download using url.
If inx == 3, do keyword search.
If inx == 0, fetch videos using a greedy search or browse-category.
If inx == 2, fetch videos from a category url.
If an unknown error occurs, exit with code 0.
"""
if opts:
My5Loader.options = opts
self.options_list = split_options(My5Loader.options)
# direct download
if "http" in search_term and inx == 1:
# options_list = split_options(self.options)
if self.options_list[0] == "":
command = ['uv', 'run', 'envied', "dl", "MY5", search_term]
else:
command = ['uv', 'run', 'envied', "dl", *self.options_list, "MY5", search_term]
self.runsubprocess(command)
return
# keyword search
elif inx == 3:
print(f"Searching for {search_term}")
return self.fetch_videos(search_term)
# ALTERNATIVES BELOW FROM POP-UP MENU
elif inx == 0:
# from greedy-search OR selecting Browse-category
# example: https://www.channel5.com/the-teacher/season-2/episode-2
# need a search keyword(s) from url
# split and select series name
if "show" in search_term:
match = re.search(r"\/show\/([a-z-]+)", search_term)
if match:
search_term = str(match.group(1))
search_term = search_term.split("-")[:-1]
search_term = str(" ".join(search_term)).replace("-", " ")
else:
search_term = search_term.split("/")[3].replace("-", " ")
# fetch_videos_by_category search_term may have other params to remove
if "/" in search_term:
search_term = search_term.split("/")[0].replace("-", " ")
return self.fetch_videos(search_term)
elif "http" in search_term and inx == 2:
self.category = category
self.fetch_videos_by_category(
search_term
) # search_term here holds a URL!!!
else:
print(f"Unknown error when searching for {search_term}")
return
def fetch_videos(self, search_term):
"""Fetch videos from ITV using a search term."""
url = f"https://corona.channel5.com/shows/search.json?platform=my5desktop&friendly=1&query={search_term}"
response = self.get_data(url, headers=self.headers)
parsed_data = parse_json(response)
if parsed_data and "shows" in parsed_data:
for item in parsed_data["shows"]:
series_name = item.get("title", "Unknown Series")
episode = {
"title": item.get("title", "Unknown Title"),
"slug": f"{item.get('f_name', '')}",
"synopsis": f"{item.get('s_desc', '')}",
"url": f"https://corona.channel5.com/shows/{item.get('f_name', '')}/seasons.json?platform=my5desktop&friendly=1",
}
self.add_episode(series_name, episode)
# List series and episodes using beaupy
selected_series = self.display_series_list()
if selected_series:
return self.second_fetch(
selected_series
) # Return URLs for selected episodes
return None
def second_fetch(self, selected):
"""
Given a selected series name, fetch its HTML and extract its episodes.
Or if given a direct url, fetch it and process for greedy download.
The function will prepare the series data for episode selection and display the final episode list.
It will return the URLs for the selected episodes.
"""
if "https" in selected: # direct url: not sure how this may happen with My5??
url = selected
self.options_list = split_options(self.options)
try:
if self.options_list[0] == "":
command = ['uv', 'run', 'envied', "dl", "MY5", url]
else:
command = ['uv', 'run', 'envied', "dl", *self.options_list, "MY5", url]
self.runsubprocess(command)
except Exception as e:
print(
"Error downloading video:",
e,
"Is self.DOWNLOAD_ORCHESTRATOR installed correctly via 'pip install self.DOWNLOAD_ORCHESTRATOR?",
)
return
else:
url = self.get_selected_url(selected)
try:
myhtml = self.get_data(url=url)
except Exception:
print(f"No valid data at {url} found.\n Exiting")
return
parsed_data = parse_json(myhtml)
self.clear_series_data() # Clear existing series data
# Extract the episodes from the parsed data of the selected series
brndslug = url.split("/")[4]
if parsed_data and "seasons" in parsed_data:
for item in parsed_data["seasons"]:
try:
# no season means single episode
# check and call self.receive for single episode
if item["seasonNumber"] is None:
url = f"https://www.channel5.com/show/{brndslug}"
self.receive(1, url)
return
else:
url = f"https://corona.channel5.com/shows/{brndslug}/seasons/{item['seasonNumber']}/episodes.json?platform=my5desktop&friendly=1&linear=true"
url = url.encode("utf-8", "ignore").decode().strip()
myhtml = self.get_data(url=url)
parsed_data = parse_json(myhtml)
if len(parsed_data["episodes"]) == 0:
continue
series_name = parsed_data["episodes"][0]["sh_f_name"]
for item in parsed_data["episodes"]:
episode = {
"series_no": f"{item['sea_num'] or '100'}",
"title": f"{item['ep_num'] or ''}:{item['title'] or ''}",
"url": f"https://www.channel5.com/{item['sh_f_name']}/{item['sea_f_name']}/{item['f_name']}",
"synopsis": item["s_desc"] or None,
}
self.add_episode(series_name, episode)
except KeyError:
continue # Skip any episode that doesn't have the required information
self.prepare_series_for_episode_selection(
series_name
) # creates list of series; allows user selection of wanted series prepares an episode list over chosen series
selected_final_episodes = self.display_final_episode_list(
self.final_episode_data
)
# download
self.options_list = split_options(self.options)
for item in selected_final_episodes:
url = item.split(",")[2].lstrip()
if url == "None":
print(f"No valid URL for {item.split(',')[1]}")
continue
if 'http' not in url:
url_pattern = r'https?://[^\s,]+'
match = re.search(url_pattern, item)
if match:
url = match.group()
try:
if self.options_list[0] == "":
command = ['uv', 'run', 'envied', "dl", "MY5", url]
else:
command = ['uv', 'run', 'envied', "dl", *self.options_list, "MY5", url]
self.runsubprocess(command)
except Exception as e:
print(
"Error downloading video:",
e,
"Is the package installed correctly?",
)
return
def fetch_videos_by_category(self, browse_url):
"""
Fetches videos from a category (Channel 4 specific).
Args:
browse_url (str): URL of the category page.
Returns:
None
"""
beaupylist = []
try:
req = self.get_data(browse_url, headers=self.headers)
# Parse the json data
init_data = parse_json(req)
# Extract brand items
myjson = init_data["shows"]
# jmespath is an efficient json parser that searches complex json
# and, in this case, produces a simple dict from which
# res(ults) are more easily extracted.
# C4 specific
res = jmespath.search(
"""
[*].{
href: f_name,
label: title,
overlaytext: s_desc
} """,
myjson,
)
# Build the beaupylist for display
for i, item in enumerate(res):
label = item["label"]
overlaytext = item["overlaytext"]
beaupylist.append(
f"{i} {label}\n\t{overlaytext}"
) # \t used to split text later
except Exception as e:
print(f"Error fetching category data: {e}")
return
# call function in BaseLoader
found = self.display_beaupylist(beaupylist)
if found:
ind = found.split(" ")[0]
url = f"https://www.channel5.com/show/{res[int(ind)]['href']}"
# url may be for series or single Film
url = url.encode("utf-8", "ignore").decode().strip() # has spaces!
# process short-cut download or do greedy search on url
return self.process_received_url_from_category(url)
else:
print("No video selected.")
return
packages/vinefeeder/src/vinefeeder/services/MY5/config.yaml
+18
View File
@@ -0,0 +1,18 @@
service: MY5
options:
media_dict:
Films: https://corona.channel5.com/shows/search.json?platform=my5desktop&friendly=1&vod_subgenres%5B%5D=6100117389032&vod_subgenres%5B%5D=6100117390032&vod_subgenres%5B%5D=6100117391032
Documentary: https://corona.channel5.com/shows/search.json?platform=my5desktop&friendly=1&vod_subgenres[]=6100110273032&vod_subgenres[]=6100105092032&vod_subgenres[]=6100105093032&vod_subgenres[]=6100105094032&vod_subgenres[]=6100105095032&vod_subgenres[]=6100105096032&vod_subgenres[]=6100105097032&vod_subgenres[]=6100110268032&vod_subgenres[]=6100110269032&vod_subgenres[]=6100110270032&vod_subgenres[]=6100110271032&vod_subgenres[]=6100110272032
Crime: https://corona.channel5.com/shows/search.json?platform=my5desktop&friendly=1&limit=100&sort_by=popular&offset=0&vod_subgenres%5B%5D=7626766659032&vod_subgenres%5B%5D=7626766660032
Dramas & Soaps: https://corona.channel5.com/shows/search.json?platform=my5desktop&friendly=1&vod_subgenres%5B%5D=6100110274032&vod_subgenres%5B%5D=6100110275032
Entertainment: https://corona.channel5.com/shows/search.json?platform=my5desktop&friendly=1&vod_subgenres%5B%5D=6100112646032&vod_subgenres%5B%5D=6100110276032&vod_subgenres%5B%5D=6100110277032&vod_subgenres%5B%5D=6100112638032&vod_subgenres%5B%5D=6100112639032&vod_subgenres%5B%5D=6100112640032&vod_subgenres%5B%5D=6100112641032&vod_subgenres%5B%5D=6100112642032&vod_subgenres%5B%5D=6100112643032
Science and Nature: https://corona.channel5.com/shows/search.json?platform=my5desktop&friendly=1&vod_subgenres%5B%5D=6100118658032&vod_subgenres%5B%5D=6100117395032&vod_subgenres%5B%5D=6100117396032&vod_subgenres%5B%5D=6100117397032
Sport: https://corona.channel5.com/shows/search.json?platform=my5desktop&friendly=1&vod_subgenres%5B%5D=6100118660032
Travel: https://corona.channel5.com/shows/search.json?platform=my5desktop&friendly=1&vod_subgenres%5B%5D=6100118664032&vod_subgenres%5B%5D=6100118662032&vod_subgenres%5B%5D=6100118663032
Real Lives: https://corona.channel5.com/shows/search.json?platform=my5desktop&friendly=1&vod_subgenres%5B%5D=6100117393032
Lifestyle: https://corona.channel5.com/shows/search.json?platform=my5desktop&friendly=1&vod_subgenres%5B%5D=6100114865032&vod_subgenres%5B%5D=6100114859032&vod_subgenres%5B%5D=6100114860032&vod_subgenres%5B%5D=6100114861032&vod_subgenres%5B%5D=6100114862032&vod_subgenres%5B%5D=6100114863032
News & Current Affairs: https://corona.channel5.com/shows/search.json?platform=my5desktop&friendly=1&vod_subgenres%5B%5D=6726148277032
Smithsonian: https://corona.channel5.com/shows/search.json?platform=my5desktop&friendly=1&ids%5B%5D=7200164903527&ids%5B%5D=4365811271527&ids%5B%5D=4301348409527&ids%5B%5D=4783938314527&ids%5B%5D=4977961314527&ids%5B%5D=5600080444527&ids%5B%5D=4489370797527&ids%5B%5D=4617233242527&ids%5B%5D=6638446529527&ids%5B%5D=6275149260527&ids%5B%5D=7810836259527&ids%5B%5D=7741736837527&ids%5B%5D=8103343367527&ids%5B%5D=8103392205527&ids%5B%5D=8103519133527&ids%5B%5D=8103520030527&ids%5B%5D=5285427795527&ids%5B%5D=4867434237527&ids%5B%5D=4301367810527&ids%5B%5D=7059359470527&ids%5B%5D=8164734673527&ids%5B%5D=8164998660527&ids%5B%5D=8165160044527
Milkshake: https://corona.channel5.com/shows/search.json?platform=my5desktop&friendly=1&vod_subgenres%5B%5D=6100114867032
packages/vinefeeder/src/vinefeeder/services/STV/__init__.py
+387
View File
@@ -0,0 +1,387 @@
from vinefeeder.base_loader import BaseLoader
from vinefeeder.parsing_utils import extract_script_with_id_json, parse_json, split_options
from rich.console import Console
import jmespath
import re
console = Console()
class StvLoader(BaseLoader):
options = ""
def __init__(self):
headers = {
"Accept": "*/*",
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64)",
"Origin": "https://player.stv.tv",
"Referer": "https://player.stv.tv/",
}
super().__init__(headers)
def receive(
self, inx: None, search_term: None, category=None, hlg_status=False, opts=None
):
"""
First fetch for series titles matching all or part of search_term.
Uses inx, an int variable to switch:-
0 for greedy search using url
1 for direct url download
2 for category browse
3 for search with keyword
If search_url_entry is left blank vinefeeder generates a
a menu with 3 options:
- Greedy Search by URL
- Browse by Category
- Search by Keyword
results are forwarded to receive().
If inx == 1, do direct download using url.
If inx == 3, do keyword search.
If inx == 0, fetch videos using a greedy search or browse-category.
If inx == 2, fetch videos from a category url.
If an unknown error occurs, exit with code 0.
"""
if opts:
StvLoader.options = opts
self.options_list = split_options(StvLoader.options)
# direct download
if "http" in search_term and inx == 1:
try:
self.options_list = split_options(StvLoader.options)
if self.options_list[0] == "":
command = ['uv', 'run', 'envied', "dl", "STV", search_term]
else:
command = ['uv', 'run', 'envied', "dl", *self.options_list, "STV", search_term]
self.runsubprocess(command) # url
except Exception as e:
print(
"Error downloading video:",
e,
"Is self.DOWNLOAD_ORCHESTRATOR installed correctly via 'pip install self.DOWNLOAD_ORCHESTRATOR?",
)
return
# keyword search
elif inx == 3:
print(f"Searching for {search_term}")
return self.fetch_videos(search_term)
# ALTERNATIVES BELOW FROM POP-UP MENU
elif inx == 0:
# from greedy-search OR selecting Browse-category
# example:
# https://player.stv.tv/summary/joan
# need a search keyword(s) from url
# split and select series name
search_term = search_term.split("/")[-1]
# fetch_videos_by_category search_term may have other params to remove
if "?" in search_term:
search_term = search_term.split("?")[0].replace("-", " ")
return self.fetch_videos(search_term)
elif "http" in search_term and inx == 2:
self.category = category
self.fetch_videos_by_category(
search_term
) # search_term here holds a URL!!!
else:
print(f"Unknown error when searching for {search_term}")
return
def fetch_videos(self, search_term):
headers = {
"Accept": "*/*",
"Accept-Language": "en-GB,en-US;q=0.9,en;q=0.8",
"Connection": "keep-alive",
"Origin": "https://player.stv.tv",
"Referer": "https://player.stv.tv/",
"Host": "search-api.swiftype.com",
"Access-Control-Request-Method": "POST",
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36",
}
url = "https://search-api.swiftype.com/api/v1/public/engines/search.json"
response = self.get_options(url, headers=headers)
xdata = response["x-request-id"]
headers = {
"Accept": "*/*",
"Accept-Language": "en-GB,en-US;q=0.9,en;q=0.8",
"Connection": "keep-alive",
"Origin": "https://player.stv.tv",
"Referer": "https://player.stv.tv/",
"Host": "search-api.swiftype.com",
"Access-Control-Request-Method": "POST",
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36",
"x-request-id": f"{xdata}",
}
json = {
"engine_key": "S1jgssBHdk8ZtMWngK_y",
"per_page": 100,
"page": 1,
"fetch_fields": {"page": ["title", "body", "resultDescriptionTx", "url"]},
"highlight_fields": {"page": {"title": {"size": 100, "fallback": True}}},
"search_fields": {"page": ["title^3", "body", "category", "sections"]},
"q": search_term,
"spelling": "strict",
}
response = self.post_data(url, headers=headers, json=json)
parsed_data = response.json() #
mydata = parsed_data["records"]["page"]
for item in mydata:
title = item["title"]
episode = {
"title": item["title"],
"url": item["url"],
"synopsis": item["resultDescriptionTx"],
}
self.add_episode(title, episode)
# List series with search_term using beaupy
selected_series = self.display_series_list()
if selected_series:
# one series selected
return self.second_fetch(selected_series)
return None
def second_fetch(self, selected):
"""
Given a selected series name, fetch its HTML and extract its episodes.
Or if given a direct url, fetch it and process for greedy download.
The function will prepare the series data for episode selection and display the final episode list.
It will return the URLs for the selected episodes.
"""
if "https" in selected: # direct url provided so skip url preparation
url = selected
else:
url = self.get_selected_url(selected)
try:
myhtml = self.get_data(url=url)
except Exception:
print(f"No valid data at {url} found.\n Exiting")
return
parsed_data = extract_script_with_id_json(myhtml, "__NEXT_DATA__", 0)
self.clear_series_data() # Clear existing series data
'''console.print_json(data=parsed_data) # for debugging
f = open("1stv.json",'w')
f.write(json.dumps(parsed_data))
f.close()'''
series_data = parsed_data["props"]["pageProps"]["data"]["programmeHeader"][
"name"
]
tabs = len(parsed_data["props"]["pageProps"]["data"]["tabs"])
headers = {
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:138.0) Gecko/20100101 Firefox/138.0',
'Accept': '*/*',
'Accept-Language': 'en-GB,en;q=0.5',
# 'Accept-Encoding': 'gzip, deflate, br, zstd',
'Referer': 'https://player.stv.tv/',
'Stv-Drm': 'true',
'Stv-PremierSportsAware': 'true',
'Origin': 'https://player.stv.tv',
'Connection': 'keep-alive',
'Sec-Fetch-Dest': 'empty',
'Sec-Fetch-Mode': 'cors',
'Sec-Fetch-Site': 'same-site',
'Priority': 'u=4',
'Cache-Control': 'max-age=0',
}
# STV mostly provde one series' worth of episodes in one request
# tabs are populated with series_guid to allow for multiple series calls
# some tabs are not series and only have a programme.guid for example:-
# props►pageProps►data►tabs►0►params►query►programme.guid
# test here
try:
PROGGUID = parsed_data["props"]["pageProps"]["data"]["programmeData"]["guid"]
except Exception:
PROGGUID = None
if PROGGUID:
for item in parsed_data["props"]["pageProps"]["data"]["tabs"][0]["data"]:
try:
series_no = parsed_data["props"]["pageProps"]["data"]["tabs"][0]["title"]
if "Episode" in series_no:
series_no = 100
else:
series_no = int(series_no.split(" ")[1])
title = item["title"]
url = f"https://player.stv.tv{item['link']}" # https://player.stv.tv/episode/4nlk/loose-women
synopsis = item["summary"]
episode = {
"series_no": series_no,
"title": title.replace(
", ", "-"
), # data with comma messes up later when selecting url
"url": url,
"synopsis": synopsis,
}
self.add_episode_remove_duplicates(series_data, episode)
except KeyError as e:
print(f"Error: {e}")
if tabs > 1:
for index in range(1, tabs):
# last few tabs may not contain series so check
if (
"Autoplay"
in parsed_data["props"]["pageProps"]["data"]["tabs"][index]["title"]
or "Trailer"
in parsed_data["props"]["pageProps"]["data"]["tabs"][index]["title"]
):
break
series_guid = parsed_data["props"]["pageProps"]["data"]["tabs"][index][
"params"
]["query"]["series.guid"]
response = self.get_data(
f"https://player.api.stv.tv/v1/episodes?series.guid={series_guid}&limit=100&groupToken=0071",
headers=headers,
)
next_parsed_data = parse_json(response)
'''console.print_json(data=next_parsed_data)
f = open("2stv.json",'w')
f.write(json.dumps(next_parsed_data))
f.close()'''
for item in next_parsed_data["results"]:
try:
series_no = item["playerSeries"]["name"]
title = item["title"]
url = item["_permalink"]
synopsis = item["summary"]
episode = {
"series_no": int(series_no.replace("Series ", "")),
"title": title,
"url": url, #
"synopsis": synopsis,
}
self.add_episode_remove_duplicates(series_data, episode)
except KeyError as e:
print(f"Error: {e}")
self.prepare_series_for_episode_selection(
series_data
) # creates list of series;
selected_final_episodes = self.display_final_episode_list(
self.final_episode_data
)
self.options_list = split_options(self.options)
for item in selected_final_episodes:
url = item.split(",")[2].lstrip()
try:
if self.options_list[0] == "":
command = ['uv', 'run', 'envied', "dl", "STV", url]
else:
command = ['uv', 'run', 'envied', "dl", *self.options_list, "STV", url]
self.runsubprocess(command)
except Exception as e:
print(
"Error downloading video:",
e,
"Is the package installed correctly?",
)
return None
def fetch_videos_by_category(self, browse_url):
"""
Fetches videos from a category (Channel 4 specific).
Args:
browse_url (str): URL of the category page.
Returns:
None
"""
beaupylist = []
try:
req = self.get_data(browse_url, headers=self.headers)
# Parse the __PARAMS__ data
init_data = extract_script_with_id_json(req, "__NEXT_DATA__", 0)
"""console.print_json(data=init_data)
f = open("cat_stv.json",'w')
f.write(json.dumps(init_data))
f.close()"""
# Extract brand items
myjson = jmespath.search("props.pageProps.data.assets", init_data)
# jmespath is an efficient json parser that searches complex json
# and, in this case, produces a simple dict from which
# res(ults) are more easily extracted.
# STV specific
res = jmespath.search(
"""
[].{
label: title,
overlaytext: description,
href: link
}
""",
myjson,
)
# Build the beaupylist for display
for i, item in enumerate(res):
label = item["label"]
overlaytext = item["overlaytext"]
beaupylist.append(
f"{i} {label}\n\t{overlaytext}"
) # \t used to split text later
except Exception as e:
print(f"Error fetching category data: {e}")
return
# call function in BaseLoader
found = self.display_beaupylist(beaupylist)
if found:
ind = found.split(" ")[0]
url = f"https://player.stv.tv{res[int(ind)]['href']}"
# url may be for series or single Film
url = url.encode("utf-8", "ignore").decode().strip() # if has spaces!
# STV is unusual in that some urls from Documentary or Films are not valid
# for download but only for browsing.
# Check if url has an identifier,
# if not, get it by calling receive() with index = 3 (greedy search)
test_id = url.split("/")[4] # look for identifier ~4hgf
if re.match(r"[0-9]+", test_id) and len(test_id) == 4:
pass
else:
return self.receive(3, url)
# if film/doc then process url and download
# process short-cut download or do greedy search on url
return self.process_received_url_from_category(url)
else:
print("No video selected.")
return
packages/vinefeeder/src/vinefeeder/services/STV/config.yaml
+22
View File
@@ -0,0 +1,22 @@
service: STV
options:
media_dict:
Films: https://player.stv.tv/categories/movies
Sport: https://player.stv.tv/categories/the-sport-hub
Crime Dramas: https://player.stv.tv/categories/crime-drama
True Crime: https://player.stv.tv/categories/crime-punishment
Comedy Dramas: https://player.stv.tv/categories/comedy-drama
Documentaries: https://player.stv.tv/categories/documentaries
Dramas: https://player.stv.tv/categories/dramas
Entertainment: https://player.stv.tv/categories/entertainment
Soaps: https://player.stv.tv/categories/soaps
Food: https://player.stv.tv/categories/food-lifestyle
Scenic Scotland: https://player.stv.tv/categories/scenic-scotland
News: https://player.stv.tv/categories/news-current-affairs
Thrillers: https://player.stv.tv/categories/thrillers
History Hit: https://player.stv.tv/categories/history-hit
Real Crime: https://player.stv.tv/categories/real-crime
Real Stories: https://player.stv.tv/categories/real-stories
Real Life: https://player.stv.tv/categories/real-life
packages/vinefeeder/src/vinefeeder/services/TPTV/README.txt
+6
View File
@@ -0,0 +1,6 @@
Place the folder TPTV within your devine-services/services/ folder.
devine dl -? should then list all services and TPTV should be among them.
packages/vinefeeder/src/vinefeeder/services/TPTV/__init__.py
+236
View File
@@ -0,0 +1,236 @@
from vinefeeder.base_loader import BaseLoader
from vinefeeder.parsing_utils import split_options
from rich.console import Console
import json
from beaupy import select_multiple
console = Console()
def prettify(my_list):
"""
Formats a list of three elements into a styled string using rich text formatting.
Parameters:
my_list (list): A list containing three elements to be formatted.
Returns:
str: A formatted string where the first two elements are displayed in green
and the third element is displayed in cyan on a new line with indentation.
"""
my_beaupystring = []
one = my_list[0]
two = my_list[1]
three = my_list[2]
my_beaupystring = f"[green]{one}, {two}[/green][cyan]\n\t{three}[/cyan]"
return my_beaupystring
class TptvLoader(BaseLoader):
options = ""
def __init__(self):
headers = {
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:138.0) Gecko/20100101 Firefox/138.0',
'Accept': '*/*',
'Accept-Language': 'en-GB,en;q=0.5',
'api-key': 'zq5pyPd0RTbNg3Fyj52PrkKL9c2Af38HHh4itgZTKDaCzjAyhd',
'Referer': 'https://tptvencore.co.uk/',
'tenant': 'encore',
'Content-Type': 'application/json',
'Origin': 'https://tptvencore.co.uk',
'DNT': '1',
'Connection': 'keep-alive',
'Sec-Fetch-Dest': 'empty',
'Sec-Fetch-Mode': 'cors',
'Sec-Fetch-Site': 'cross-site',
'Priority': 'u=0',
}
super().__init__(headers)
payload = {}
r = self.post_data("https://prod.suggestedtv.com/api/client/v1/session", json=payload, headers=headers)
if r.status_code != 200:
raise ConnectionError
else:
self.session_id = r.json()['id']
def receive(
self, inx: None, search_term: None, category=None, hlg_status=False, opts=None
):
"""
First fetch for series titles matching all or part of search_term.
Uses inx, an int variable to switch:-
0 for greedy search using url
1 for direct url download
2 for category browse
3 for search with keyword
If search_url_entry is left blank vinefeeder generates a
a menu with 3 options:
- Greedy Search by URL
- Browse by Category
- Search by Keyword
results are forwarded to receive().
If inx == 1, do direct download using url.
If inx == 3, do keyword search.
If inx == 0, fetch videos using a greedy search or browse-category.
If inx == 2, fetch videos from a category url.
If an unknown error occurs, exit with code 0.
"""
if opts:
TptvLoader.options = opts
self.options_list = split_options(TptvLoader.options)
# direct download
if "http" in search_term and inx == 1:
try:
self.options_list = split_options(TptvLoader.options)
if self.options_list[0] == "":
command = ['uv', 'run', 'envied', "dl", "TPTV", search_term]
else:
command = ['uv', 'run', 'envied', "dl", *self.options_list, "TPTV", search_term]
self.runsubprocess(command) # url
except Exception as e:
print(
"Error downloading video:",
e,
f"Is {self.DOWNLOAD_ORCHESTRATOR} installed correctly via 'pip install {self.DOWNLOAD_ORCHESTRATOR})?",
)
return
# keyword search
elif inx == 3:
print(f"Searching for {search_term}")
return self.fetch_videos(search_term)
# ALTERNATIVES BELOW FROM POP-UP MENU
elif inx == 0:
# from greedy-search OR selecting Browse-category
search_term = search_term.split("/")[-1]
# fetch_videos_by_category search_term may have other params to remove
if "?" in search_term:
search_term = search_term.split("?")[0].replace("-", " ")
return self.fetch_videos(search_term)
elif "http" in search_term and inx == 2:
self.category = category
self.fetch_videos_by_category(
search_term
) # search_term here holds a URL!!!
else:
print(f"Unknown error when searching for {search_term}")
return
def fetch_videos(self, search_term):
suggested_url = f"https://prod.suggestedtv.com/api/client/v2/search/{search_term}"
suggested_headers = {
'Accept': '*/*',
'Accept-Encoding': 'gzip, deflate, br, zstd',
'Accept-Language': 'en-GB,en-US;q=0.9,en;q=0.8',
'Connection': 'keep-alive',
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36',
'Origin': 'https://tptvencore.co.uk',
'Referer': 'https://tptvencore.co.uk/',
}
get_headers = suggested_headers.copy()
get_headers['Access-Control-Request-Headers'] = 'session,tenant'
get_headers['Access-Control-Request-Method'] = 'GET'
get_headers['session'] = self.session_id
get_headers['tenant'] = 'encore'
response = self.get_data(suggested_url, headers=get_headers)
myjson = json.loads(response)
myitems = []
for item in myjson['data']:
if item.startswith('collection_'):
id = item.replace('collection_','')
collection_url = f"https://prod.suggestedtv.com/api/client/v1/collection/by-reference/{id}?extend=label"
response = self.get_data(collection_url, headers=get_headers)
if response:
data = json.loads(response)
for item in data['children']:
myitems.append(item['id'].replace('product_',''))
elif item.startswith('product_'):
myitems.append(item.replace('product_',''))
mystring = ",".join(myitems)
continued_search_url = "https://prod.suggestedtv.com/api/client/v1/product?ids=" + mystring + "&extend=label"
response = self.get_data(continued_search_url, headers=get_headers)
data = json.loads(response)
try:
for item in data['data']:
vid_id = item['reference']
title = item['name']
synopsis = item['description'].replace('\n', ' ')
if len(synopsis) > 300:
synopsis = f"{synopsis[:300]} ...snip"
episode = {
"title": title,
"url": vid_id,
"synopsis": synopsis,
}
self.add_episode(title,episode)
except Exception as e:
if len(self.series_data) > 0:
pass
else:
print("Nothing found. Try another search term.")
return
eps = self.get_series()
beaupylist = []
for entry in eps.values():
if entry and isinstance(entry[0], dict): # guard clause for safety
beaupylist.append([entry[0]['title'], entry[0]['synopsis'], entry[0]['url']])
selected = select_multiple(beaupylist, preprocessor=lambda val: prettify(val), page_size=6, pagination=True)
if selected:
return self.second_fetch(selected)
return None
def second_fetch(self, selected):
self.options_list = split_options(self.options)
for item in selected:
url = item[2]
url = f"https://tptvencore.co.uk/product/{url}"
#print(url)
try:
if self.options_list[0] == "":
command = ['uv', 'run', 'envied', "dl", "TPTV", url]
else:
command = ['uv', 'run', 'envied', "dl", *self.options_list, "TPTV", url]
self.runsubprocess(command)
except Exception as e:
print(
"Error downloading video:",
e,
"Is devine/envied installed correctly via 'pip install <program name>?",
)
return None
def fetch_videos_by_category(self, browse_url):
"""
Fetches videos from a category.
Args:
browse_url (str): URL of the category page.
Returns:
None
"""
print("Category fetch not Implemented")
return None
packages/vinefeeder/src/vinefeeder/services/TPTV/config.yaml
+6
View File
@@ -0,0 +1,6 @@
service: TPTV
options:
media_dict:
Not Implemented: None
packages/vinefeeder/src/vinefeeder/services/TVNZ/__init__.py
+428
View File
@@ -0,0 +1,428 @@
# TVNZ __init__.py
from vinefeeder.base_loader import BaseLoader
from vinefeeder.parsing_utils import split_options, list_prettify
from rich.console import Console
from beaupy import select_multiple
console = Console()
# TV Shows https://www.tvnz.co.nz/shows/boiling-point/episodes/s1-e1 or
# Movies https://www.tvnz.co.nz/shows/legally-blonde/movie/s1-e1 or
# Sport https://www.tvnz.co.nz/sport/football/uefa-euro/spain-v-france-semi-finals-highlights
class TvnzLoader(BaseLoader):
# global options
options = ""
def __init__(self):
"""
Initialize the All4Loader class with the provided headers.
Parameters:
None
Attributes:
options (str): Global options; later taken from service config.yaml
headers (dict): Global headers; may be overridden
"""
headers = {
"Accept": "*/*",
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64)",
#'Origin': 'https://www.tvnz.co.nz',
#'Referer': 'https://www.tvnz.co.nz',
}
super().__init__(headers)
self.showType = None
# entry point from Vinefeeder
def receive(
self, inx: None, search_term: None, category=None, hlg_status=False, opts=None
):
"""
First fetch for series titles matching all or part of search_term.
Uses inx, an int variable to switch:-
0 for greedy search using url
1 for direct url download
2 for category browse
3 for search with keyword
If search_url_entry is left blank vinefeeder generates a
a menu with 3 options:
- Greedy Search by URL
- Browse by Category
- Search by Keyword
results are forwarded to receive().
If inx == 1, do direct download using url.
If inx == 3, do keyword search.
If inx == 0, fetch videos using a greedy search or browse-category.
If inx == 2, fetch videos from a category url.
If an unknown error occurs, exit with code 0.
"""
# re-entry here for second time loses options settings
# so reset
if opts:
TvnzLoader.options = opts
self.options_list = split_options(TvnzLoader.options)
# direct download
if "http" in search_term and inx == 1:
try:
if self.options_list[0] == "":
command = ['uv', 'run', 'envied', "dl", "TVNZ", search_term]
else:
command = ['uv', 'run', 'envied', "dl", *self.options_list, "TVNZ", search_term]
self.runsubprocess(command)
return
except Exception as e:
print(
"Error downloading video:",
e,
"Is self.DOWNLOAD_ORCHESTRATOR installed correctly via 'pip install self.DOWNLOAD_ORCHESTRATOR?",
)
return
# keyword search
elif inx == 3:
print(f"Searching for {search_term}")
return self.fetch_videos(search_term)
# POP-UP MENU alternates:
elif inx == 0:
# from greedy-search OR selecting Browse-category
# need a search keyword(s) from url
# split and select series name
search_term = search_term.split("/")[4]
return self.fetch_videos(search_term)
elif "http" in search_term and inx == 2:
self.category = category
self.fetch_videos_by_category(
search_term
) # search_term here holds a URL!!!
else:
print(f"Unknown error when searching for {search_term}")
return
def fetch_videos(self, search_term):
"""Fetch videos from TVNZ using a search term.
Here the first search for series titles matches all or part of search_term.
The function will prepare the series data, matching the search term for display.
"""
# returns json as type String
url = f"https://apis-public-prod.tech.tvnz.co.nz/api/v1/web/play/search?q={search_term}"
try:
html = self.get_data(url)
if "No Matches" in html:
print("Nothing found for that search; try again.")
return
else:
parsed_data = self.parse_data(html) # to json
except Exception:
print(f"No valid data returned for {url}")
return
# console.print_json(data=parsed_data)
"""f = open('tvnz.json', 'w')
f.write(json.dumps(parsed_data)) # parsed_data)
f.close()"""
if parsed_data and "results" in parsed_data:
for item in parsed_data["results"]:
series_name = item.get("title", "Unknown Series")
url = "https://apis-edge-prod.tech.tvnz.co.nz" + item.get(
"page", {}
).get("href", "")
episode = {
"type": item.get("type"), # 'type'
"title": item.get("title", "Unknown Title"),
"url": url,
"synopsis": item.get("synopsis", "No synopsis available."),
}
self.add_episode(series_name, episode)
else:
print(f"No valid data returned for {url}")
return None
selected_series = self.display_series_list()
if selected_series:
return self.second_fetch(
selected_series
) # Return URLs for selected episodes
return None
def second_fetch(self, selected):
# TVNZ video type: sportVideo, showVideo or Movie
# makes this extractor unusual and should not be used
# to model other services
"""
Given a selected series name, fetch its HTML and extract its episodes.
Or if given a direct url, fetch it and process for greedy download.
The function will prepare the series data for episode selection and display the final episode list.
It will return the URLs for the selected episodes.
"""
if "https" in selected: # direct url provided skip url preparation
url = selected
else:
# initial pass to find type
series_name = (
selected.lower().replace(" ", "-").replace(":", "").replace(",", "")
)
episodes = self.get_series_data()
episode_test = episodes[selected][0]
type = episode_test.get("type")
beaupylist = []
# if sport video / news video - assume no episodes
# use data from first fetch directly
if type == "sportVideo" or type == "newsVideo":
for item in episodes[selected]: # existing data
url = "https://www.tvnz.co.nz/" + item.get("url").split("/page/")[1]
beaupylist.append(
[
item.get("title"),
url,
item.get("synopsis", "No synopsis available."),
]
)
selected = select_multiple(
beaupylist,
preprocessor=lambda val: list_prettify(val),
minimal_count=1,
cursor_style="pink1",
pagination=True,
page_size=8,
)
for item in selected:
url = item[1]
if self.options_list[0] == "":
command = ['uv', 'run', 'envied', "dl", "TVNZ", url]
else:
command = ['uv', 'run', 'envied', "dl", *self.options_list, "TVNZ", url]
self.runsubprocess(command)
return None
elif type == "show" or type == "showVideo":
url = f"https://apis-public-prod.tech.tvnz.co.nz/api/v1/web/play/page/shows/{series_name}/episodes"
try:
html = self.get_data(url)
parsed_data = self.parse_data(html)
except Exception:
try:
# direct download as seems only one episode
# https://www.tvnz.co.nz/shows/circle-of-friends/movie/s1-e1
url = f"https://www.tvnz.co.nz/shows/{series_name}/movie/s1-e1"
if self.options_list[0] == "":
command = ['uv', 'run', 'envied', "dl", "TVNZ", url]
else:
command = ['uv', 'run', 'envied', "dl", *self.options_list, "TVNZ", url]
self.runsubprocess(command)
return
except Exception as e:
print(
"Error downloading video:",
e,
"Is self.DOWNLOAD_ORCHESTRATOR installed correctly via 'pip install self.DOWNLOAD_ORCHESTRATOR?",
)
return
try:
href_list = []
# iterate over all seasons and capture url for each
for item in parsed_data["layout"]["slots"]["main"]["modules"][0][
"lists"
]:
href_list.append(item["href"])
except Exception:
# object►layout►slots►main►modules►0►mobiledoc►sections►0►2►0►3
try:
message = parsed_data["layout"]["slots"]["main"]["modules"][0][
"mobiledoc"
]["sections"][0][2][0][3]
print(
"There is no content for this series. TVNZ says ..."
+ message
)
except Exception:
print(f"No valid data returned for {url}")
return
# with season url, iterate over each season and capture episodes
try:
for url in href_list: # for all seasons
url = "https://apis-edge-prod.tech.tvnz.co.nz" + url
myhtml = self.get_data(url=url)
parsed_data = self.parse_data(myhtml)
if parsed_data and "_embedded" in parsed_data:
try:
for item_key, item in parsed_data["_embedded"].items():
series_no = item.get("seasonNumber", "100")
myurl = "https://www.tvnz.co.nz" + item.get("page", {}).get(
"url", ""
)
episode_no = item.get("episodeNumber", None)
synopsis = item.get("synopsis", "No synopsis available")
episode = {
"series_no": series_no,
"title": episode_no,
"url": myurl,
"synopsis": synopsis,
}
self.add_episode(series_name, episode)
except Exception:
pass # disregard any episode that does not fit the scheme
else:
print(f"No valid data at {url} found.\n Exiting")
return
except Exception:
print(f"No valid data returned for {url}")
return
self.options_list = split_options(self.options)
# direct download single items
if self.get_number_of_episodes(series_name) == 1:
item = self.get_series(series_name)[0]
# check if has https://www.tvnz.co.nz
if "https://www.tvnz.co.nz" in item["url"]:
url = item["url"]
else:
url = f'https://www.tvnz.co.nz{item["url"]}'
try:
if self.options_list[0] == "":
command = ['uv', 'run', 'envied', "dl", "TVNZ", url]
else:
command = ['uv', 'run', 'envied', "dl", *self.options_list, "TVNZ", url]
self.runsubprocess(command)
return None
except Exception as e:
print(
"Error downloading video:",
e,
"Is self.DOWNLOAD_ORCHESTRATOR installed correctly via 'pip install self.DOWNLOAD_ORCHESTRATOR?",
)
return
# else present list of series and display for multiple selection
self.prepare_series_for_episode_selection(
series_name
) # creates list of series; allows user selection of wanted series prepares an episode list over chosen series
self.final_episode_data = self.sort_episodes(self.get_final_episode_list())
selected_final_episodes = self.display_final_episode_list(
self.final_episode_data
)
for item in selected_final_episodes:
url = item.split(",")[2].lstrip()
if url == "None":
print(f"No valid URL for {item.split(',')[1]}")
continue
if self.options_list[0] == "":
command = ['uv', 'run', 'envied', "dl", "TVNZ", url]
else:
command = ['uv', 'run', 'envied', "dl", *self.options_list, "TVNZ", url]
self.runsubprocess(command)
return
def fetch_videos_by_category(self, browse_url):
"""
Fetches videos from a category (VERY TVNZ specific).
Args:
browse_url (str): URL of the category page.
Returns:
None
"""
try:
req = self.get_data(browse_url, headers=self.headers)
# Parse the json returned
parsed_data = self.parse_data(req)
# for development only
"""console.print_json(data=parsed_data)
f = open("cat_tvnz.json",'w')
f.write(json.dumps(parsed_data))
f.close()"""
i = 1
beaupylist = []
linkList = []
for item_key, item in parsed_data["_embedded"].items():
try:
if item.get("type") == "category":
continue
elif item.get("type") == "showVideo":
myurl = "https://www.tvnz.co.nz" + item.get("page", {}).get(
"url", ""
)
showType = item.get("showType", "unknown")
elif item.get("type") == "show":
showType = item.get("showType", "unknown")
myurl = "https://www.tvnz.co.nz" + item.get(
"watchAction", {}
).get("link", "")
elif item.get("type") == "sportVideo":
showType = item.get("showType", "unknown")
myurl = "https://www.tvnz.co.nz" + item.get("page", {}).get(
"url", ""
)
title = item.get("title", "Unknown Title")
# we are adding an episode to the series data using title as series name
# There will be some episodes with the same series name.
# in the context of category browsing TVNZ allows duplicates.
# Below we only take the fist item with a series name
# and add that to our beaupylist for display. Thus duplicates are not added.
episode = {
"index": i,
"title": item.get("title", "Unknown Title"),
"synopsis": item.get("synopsis", "No synopsis available"),
}
self.add_episode(title, episode)
# linked list allows keeping the url away from beaupy list
# we pull the url from selected by referencing the index in linkList
linkList.append([myurl, showType])
i += 1
except Exception:
continue
except Exception as e:
print(f"Error fetching category data: {e}")
return
data = self.get_series_data()
# unique to TVNZ
# get first item in each list to create a beaupylist
# without duplicates
for key, items in data.items():
if items: # Check if the list is not empty
first_item = items[
0
] # Get the first dictionary, any more may be duplicates
index = first_item["index"]
title = first_item["title"]
synopsis = first_item["synopsis"]
beaupylist.append([index, title, synopsis])
found = self.list_display_beaupylist(beaupylist)
self.clear_series_data()
if found:
ind = found[0]
url, showType = linkList[int(ind) - 1]
if showType == "Movie" or showType == "NonEpisodic":
# direct download
return self.receive(inx=1, search_term=url)
else:
# greedy search
search_term = url.split("/")[4].replace("-", " ")
return self.receive(inx=3, search_term=search_term)
else:
print("No video selected.")
return
packages/vinefeeder/src/vinefeeder/services/TVNZ/config.yaml
+17
View File
@@ -0,0 +1,17 @@
service: TVNZ
options:
media_dict:
Drama: https://apis-edge-prod.tech.tvnz.co.nz/api/v1/web/play/page/categories/drama
Home and Living: https://apis-edge-prod.tech.tvnz.co.nz/api/v1/web/play/page/categories/home-and-living
Sport Documentaries: https://apis-edge-prod.tech.tvnz.co.nz/api/v1/web/play/page/categories/sport-documentaries
Natural World: https://apis-edge-prod.tech.tvnz.co.nz/api/v1/web/play/page/categories/natural-world
Foreign Language: https://apis-edge-prod.tech.tvnz.co.nz/api/v1/web/play/page/categories/foreign-language
True Crime: https://apis-edge-prod.tech.tvnz.co.nz/api/v1/web/play/page/categories/true-crime
Australian Drama: https://apis-edge-prod.tech.tvnz.co.nz/api/v1/web/play/page/categories/australian-drama
British Drama: https://apis-edge-prod.tech.tvnz.co.nz/api/v1/web/play/page/categories/british-drama
sci-fi-and-fantasy: https://apis-edge-prod.tech.tvnz.co.nz/api/v1/web/play/page/categories/sci-fi-and-fantasy
Factual: https://apis-edge-prod.tech.tvnz.co.nz/api/v1/web/play/page/categories/factual
Comedy: https://apis-edge-prod.tech.tvnz.co.nz/api/v1/web/play/page/categories/comedy
packages/vinefeeder/src/vinefeeder/services/U/__init__.py
+220
View File
@@ -0,0 +1,220 @@
from vinefeeder.base_loader import BaseLoader
from vinefeeder.parsing_utils import split_options
import jmespath
from rich.console import Console
console = Console()
class ULoader(BaseLoader):
options = ""
def __init__(self):
headers = {
"Accept": "*/*",
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64)",
"Origin": "https:///u.co.uk",
"Referer": "https:///u.co.uk/",
}
super().__init__(headers)
def receive(
self, inx: None, search_term: None, category=None, hlg_status=False, opts=None
):
"""
First fetch for series titles matching all or part of search_term.
Uses inx, an int variable to switch:-
0 for greedy search using url
1 for direct url download
2 for category browse
3 for search with keyword
If search_url_entry is left blank vinefeeder generates a
a menu with 3 options:
- Greedy Search by URL
- Browse by Category
- Search by Keyword
results are forwarded to receive().
If inx == 1, do direct download using url.
If inx == 3, do keyword search.
If inx == 0, fetch videos using a greedy search or browse-category.
If inx == 2, fetch videos from a category url.
If an unknown error occurs, exit with code 0.
"""
if opts:
ULoader.options = opts
self.options_list = split_options(ULoader.options)
# direct download
if "http" in search_term and inx == 1:
# options_list = split_options(ULoader.options)
try:
if self.options_list[0] == "":
command = ['uv', 'run', 'envied', "dl", "U", search_term]
else:
command = ['uv', 'run', 'envied', "dl", *self.options_list, "U", search_term]
self.runsubprocess(command) # url
except Exception as e:
print(
"Error downloading video:",
e,
"Is self.DOWNLOAD_ORCHESTRATOR installed correctly via 'pip install self.DOWNLOAD_ORCHESTRATOR?",
)
return
# keyword search
elif inx == 3:
print(f"Searching for {search_term}")
return self.fetch_videos(search_term)
# ALTERNATIVES BELOW FROM POP-UP MENU
elif inx == 0:
# from greedy-search OR selecting Browse-category
# example https://u.co.uk/shows/worlds-most-dangerous-roads/series-6/episode-1/6363704256112
# need a search keyword(s) from url
# split and select series name
search_term = search_term.split("/")[4].replace("-", " ")
return self.fetch_videos(search_term)
elif "http" in search_term and inx == 2:
self.category = category
self.fetch_videos_by_category(
search_term
) # search_term here holds a URL!!!
else:
print(f"Unknown error when searching for {search_term}")
return
def fetch_videos(self, search_term):
"""Fetch videos from ITV using a search term."""
headers = {
"user-agent": "Dalvik/2.9.8 (Linux; U; Android 9.9.2; ALE-L94 Build/NJHGGF)",
"Host": "vschedules.uktv.co.uk",
"Origin": "https://uktv.co.uk",
"Referer": "https://uktv.co.uk/",
}
url = f"https://vschedules.uktv.co.uk/vod/search/?q={search_term}"
html = self.get_data(url, headers)
parsed_data = self.parse_data(html) # to json ()
# select only from FREE tier
res = jmespath.search(
"""
[].{
title: name,
slug: slug,
synopsis: synopsis
type: type
} """,
parsed_data,
)
for item in res:
if item["type"] == "COLLECTION":
continue # Skip to the next iteration if type is COLLECTION
title = item["title"]
slug = item["slug"]
synopsis = item["synopsis"]
url = f"https://vschedules.uktv.co.uk/vod/brand/?slug={slug}"
episode = {
"title": item.get("title", "Unknown Title"),
"url": url,
"synopsis": synopsis,
}
self.add_episode(title, episode)
# List series with search_term using beaupy
selected_series = self.display_series_list()
if selected_series:
return self.second_fetch(selected_series)
return None
def second_fetch(self, selected):
"""
Given a selected series name, fetch its HTML and extract its episodes.
Or if given a direct url, fetch it and process for greedy download.
The function will prepare the series data for episode selection and display the final episode list.
It will return the URLs for the selected episodes.
"""
if "https" in selected: # direct url provided skip url preparation
url = selected
else:
url = self.get_selected_url(selected)
try:
myhtml = self.get_data(url)
except Exception:
print(f"No valid data at {url} found.\n Exiting")
return
parsed_data = self.parse_data(myhtml)
series_ids = [series["id"] for series in parsed_data["series"]]
if not series_ids:
print(f"No series data found at {url}.\n Exiting")
return
# Go through each episode in the selected series
for series_id in series_ids:
html = self.get_data(
f"https://vschedules.uktv.co.uk/vod/series/?id={series_id}"
)
parsed_data = self.parse_data(html)
try:
for item in parsed_data["episodes"]:
# series_id = item['series_id']
ep_number = item["episode_number"]
ser_number = item.get("series_number", "100")
video_id = item["video_id"]
brand_slug = item["brand_slug"]
url = f"https://u.co.uk/shows/{brand_slug}/series-{ser_number}/episode-{ep_number}/{video_id}"
synopsis = item["synopsis"]
episode = {
"series_no": ser_number,
"title": ep_number,
"url": url,
"synopsis": synopsis,
}
self.add_episode(brand_slug, episode)
except KeyError:
continue # Skip any episode that doesn't have the required information
self.prepare_series_for_episode_selection(
brand_slug
) # creates list of series; allows user selection of wanted series prepares an episode list over chosen series
selected_final_episodes = self.display_final_episode_list(
self.final_episode_data
)
self.options_list = split_options(self.options)
for item in selected_final_episodes:
url = item.split(",")[2].lstrip()
try:
if self.options_list[0] == "":
command = ['uv', 'run', 'envied', "dl", "U", url]
else:
command = ['uv', 'run', 'envied', "dl", *self.options_list, "U", url]
self.runsubprocess(command)
except Exception as e:
print(
"Error downloading video:",
e,
"Is the package installed correctly?",
)
return None
def fetch_videos_by_category(self, browse_url):
print(
"Method not implemented for fetching videos by category for this channel."
)
return None
packages/vinefeeder/src/vinefeeder/services/U/config.yaml
+6
View File
@@ -0,0 +1,6 @@
service: U
options:
media_dict:
None Avaiable: https://u.co.uk
packages/vinefeeder/src/vinefeeder/services/word_replace.sh
+7
View File
@@ -0,0 +1,7 @@
#!/bin/bash
# Replace 'automatarr' with 'self.DOWNLOAD_ORCHESTRATOR' in all files located exactly 2 directories deep
find . -mindepth 1 -maxdepth 3 -type f | while read -r file; do
sed -i 's/self.DOWNLOAD_ORCHESTRATOR/self.DOWNLOAD_ORCHESTRATOR/g' "$file"
done