haeaeae/proxy-scraper-checker
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Huge update

Browse Source 
- Drop support for Python 3.7.
- Use offline geolocation database instead of ip-api.com.
- Add support for proxy authentication.
- Add pre-compiled binaries.
- Refactor the code by a lot.
- Simplify config.
- Add automatic text encoding detection.
- Use TOML instead of INI for configuration.
- Add support for using httpbin-compatible services and services which return plain IP address for getting proxy's exit node.
- Add support for reading proxy lists from local files.
- Add support for saving to json.
- Remove support for saving geolocation to txt, use json instead.
- Improve parser.
- Improve config validation.
- Use poetry instead of requirements.txt.
- Use venv in start scripts.
This commit is contained in:
monosans
2024-01-21 17:45:01 +00:00
committed by GitHub
parent 2415621157
commit e4047e16ae
35 changed files with 2537 additions and 835 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/auto-merge.yml
+11
View File
@@ -6,6 +6,17 @@ concurrency:
group: ${{ github.workflow }}-${{ github.head_ref }}
cancel-in-progress: true
jobs:
auto-merge-dependabot:
runs-on: ubuntu-latest
if: ${{ github.actor == 'dependabot[bot]' }}
steps:
- id: dependabot-metadata
uses: dependabot/fetch-metadata@v1
- if: ${{ steps.dependabot-metadata.outputs.update-type != 'version-update:semver-major' }}
run: gh pr merge --auto --delete-branch --squash "${PR_URL}"
env:
PR_URL: ${{ github.event.pull_request.html_url }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
auto-merge-updates:
runs-on: ubuntu-latest
if: ${{ github.actor == 'monosans' && startsWith(github.head_ref, 'update/') }}
.github/workflows/ci.yml
+27
View File
@@ -14,3 +14,30 @@ jobs:
steps:
- uses: actions/checkout@v4
- run: pipx run pre-commit run --all-files --show-diff-on-failure
build:
strategy:
matrix:
os:
- ubuntu
- macos
- windows
fail-fast: false
runs-on: ${{ matrix.os }}-latest
steps:
- uses: actions/checkout@v4
- run: pipx install poetry
- uses: actions/setup-python@v5
with:
python-version: "3.11"
cache: poetry
check-latest: true
- run: poetry install --only main,nuitka --sync --no-root --no-interaction
- run: poetry run --no-interaction python -m nuitka --onefile --python-flag='-m' --prefer-source-code --assume-yes-for-downloads --lto=yes proxy_scraper_checker
- uses: actions/upload-artifact@v4
with:
name: artifact-${{ matrix.os }}
path: |
config.toml
proxy_scraper_checker.bin
proxy_scraper_checker.exe
if-no-files-found: error
.github/workflows/update-dependencies.yml
+3
View File
@@ -12,6 +12,9 @@ jobs:
strategy:
matrix:
include:
- cmd: pipx run poetry lock --no-interaction
commit-msg: Update poetry.lock
branch: update/poetry-lock
- cmd: pipx run pre-commit autoupdate
commit-msg: Update .pre-commit-config.yaml
branch: update/pre-commit-config
.gitignore
+1 -4
View File
@@ -376,7 +376,4 @@ $RECYCLE.BIN/
# End of https://www.toptal.com/developers/gitignore/api/jetbrains+all,linux,macos,python,vim,visualstudiocode,windows
proxies/
proxies_anonymous/
proxies_geolocation/
proxies_geolocation_anonymous/
out/
.pre-commit-config.yaml
+9 -2
View File
@@ -35,6 +35,13 @@ repos:
- --scripts-are-modules
additional_dependencies:
- aiohttp<4
- attrs
- certifi
- charset-normalizer<4
- maxminddb<3
- platformdirs<5
- rich<14
- typing-extensions<5
- uvloop<0.20; implementation_name == "cpython" and (sys_platform == "darwin" or sys_platform == "linux")
- types-aiofiles
- typing-extensions<5; python_version < "3.11"
- tomli<3; python_version < "3.11"
- uvloop<0.20; platform_python_implementation == "CPython" and (sys_platform == "darwin" or sys_platform == "linux")
README.md
+25 -24
View File
@@ -6,27 +6,37 @@
HTTP, SOCKS4, SOCKS5 proxies scraper and checker.
- Asynchronous.
- Uses regex to search for proxies (ip:port format) on a web page, allowing proxies to be extracted even from json without making changes to the code.
- It is possible to specify the URL to which to send a request to check the proxy.
- Can sort proxies by speed.
- Supports determining the geolocation of the proxy exit node.
- Can determine if the proxy is anonymous.
- Supports determining the geolocation of the proxy exit node.
- Can sort proxies by speed.
- Uses regex to find proxies of format `protocol://username:password@ip:port` on a web page or in a local file, allowing proxies to be extracted even from json without code changes.
- Supports proxies with authentication.
- It is possible to specify the URL to which to send a request to check the proxy.
- Supports saving to plain text and json.
- Asynchronous.
You can get proxies obtained using this script in [monosans/proxy-list](https://github.com/monosans/proxy-list).
You can get proxies obtained using this project in [monosans/proxy-list](https://github.com/monosans/proxy-list).
## Installation and usage
### Desktop
### Pre-compiled binary
This is the easiest way, but it is only available for x64 Windows, macOS and Linux. Just download the archive for your OS from <https://nightly.link/monosans/proxy-scraper-checker/workflows/ci/main?preview>, unzip it, edit `config.toml` and run the executable.
If Windows Defender detects an executable file as a virus, please read [this](https://github.com/Nuitka/Nuitka/issues/2495#issuecomment-1762836583).
### Running from source code
#### Desktop
- Install [Python](https://python.org/downloads). The minimum version required is 3.8. The recommended version is 3.11, because 3.12 may not install some libraries in the absence of a C compiler.
- Download and unpack [the archive with the program](https://github.com/monosans/proxy-scraper-checker/archive/refs/heads/main.zip).
- Edit `config.ini` to your preference.
- Install [Python](https://python.org/downloads) (minimum required version is 3.7).
- Edit `config.toml` to your preference.
- Run the script that installs dependencies and starts `proxy-scraper-checker`:
- On Windows run `start.cmd`
- On Unix-like operating systems run `start.sh`
### Termux
#### Termux
To use `proxy-scraper-checker` in Termux, knowledge of the Unix command-line interface is required.
@@ -35,27 +45,18 @@ To use `proxy-scraper-checker` in Termux, knowledge of the Unix command-line int
```bash
bash <(curl -fsSL 'https://raw.githubusercontent.com/monosans/proxy-scraper-checker/main/install-termux.sh')
```
- Edit `~/proxy-scraper-checker/config.ini` to your preference using a text editor (vim/nano).
- Edit `~/proxy-scraper-checker/config.toml` to your preference using a text editor (vim/nano).
- To run `proxy-scraper-checker` use the following command:
```bash
cd ~/proxy-scraper-checker && sh start-termux.sh
```
## Checking local proxy lists
## Something else?
To check the local proxy lists, start the Python HTTP server on your local machine by running the `python -m http.server --bind localhost` command in the folder with the proxy lists. After that, add links to the appropriate files in `config.ini`.
## Folders description
When the script finishes running, the following folders will be created (this behavior can be changed in the config):
- `proxies` - proxies with any anonymity level.
- `proxies_anonymous` - anonymous proxies.
- `proxies_geolocation` - same as `proxies`, but includes exit-node's geolocation.
- `proxies_geolocation_anonymous` - same as `proxies_anonymous`, but includes exit-node's geolocation.
Geolocation format is `ip:port|Country|Region|City`.
All other info is available in `config.toml` file.
## License
[MIT](LICENSE)
This product includes GeoLite2 Data created by MaxMind, available from <https://www.maxmind.com>.
config.ini
-157
View File
@@ -1,157 +0,0 @@
[General]
; The number of seconds to wait for a proxied request.
; The higher the number, the longer the check will take
; and the more proxies you get.
Timeout = 5
; The number of seconds to wait for the proxies to be downloaded from the source.
SourceTimeout = 15
; Maximum concurrent connections.
; Windows supports maximum of 512.
; On *nix operating systems, this restriction is much looser.
; The limit on *nix can be seen with the command `ulimit -Hn`.
; Don't be in a hurry to set high values.
; Make sure you have enough RAM first, gradually increasing the default value.
; If set to 0, the maximum value available for your OS will be used.
MaxConnections = 512
; URL to which to send a request to check the proxy.
; If not equal to 'default', it will not be possible
; to determine the anonymity and geolocation of the proxies.
; Example:
; CheckWebsite = https://google.com
CheckWebsite = default
; Set to no to sort proxies alphabetically.
SortBySpeed = yes
; Path to the folder where the proxy folders will be saved.
; Leave empty to save the proxies to the current directory.
SavePath =
; Enable output of all errors
Debug = no
[Folders]
; This section is ignored if CheckWebsite is not 'default'.
; Proxies with any anonymity level.
proxies = yes
; Anonymous proxies.
proxies_anonymous = yes
; Same as proxies, but includes exit-node's geolocation.
; Geolocation format is ip:port|Country|Region|City
proxies_geolocation = yes
; Same as proxies_geolocation, but includes exit-node's geolocation.
proxies_geolocation_anonymous = yes
[HTTP]
Enabled = yes
Sources =
https://api.proxyscrape.com/v2/?request=getproxies&protocol=http
; https://openproxy.space/list/http
https://openproxylist.xyz/http.txt
https://proxyspace.pro/http.txt
https://proxyspace.pro/https.txt
https://raw.githubusercontent.com/ALIILAPRO/Proxy/main/http.txt
https://raw.githubusercontent.com/andigwandi/free-proxy/main/proxy_list.txt
https://raw.githubusercontent.com/Anonym0usWork1221/Free-Proxies/main/proxy_files/http_proxies.txt
https://raw.githubusercontent.com/Anonym0usWork1221/Free-Proxies/main/proxy_files/https_proxies.txt
https://raw.githubusercontent.com/aslisk/proxyhttps/main/https.txt
https://raw.githubusercontent.com/B4RC0DE-TM/proxy-list/main/HTTP.txt
https://raw.githubusercontent.com/clarketm/proxy-list/master/proxy-list-raw.txt
https://raw.githubusercontent.com/ErcinDedeoglu/proxies/main/proxies/http.txt
https://raw.githubusercontent.com/ErcinDedeoglu/proxies/main/proxies/https.txt
https://raw.githubusercontent.com/hendrikbgr/Free-Proxy-Repo/master/proxy_list.txt
https://raw.githubusercontent.com/jetkai/proxy-list/main/online-proxies/txt/proxies-http.txt
https://raw.githubusercontent.com/jetkai/proxy-list/main/online-proxies/txt/proxies-https.txt
https://raw.githubusercontent.com/mmpx12/proxy-list/master/http.txt
https://raw.githubusercontent.com/mmpx12/proxy-list/master/https.txt
https://raw.githubusercontent.com/MuRongPIG/Proxy-Master/main/http.txt
https://raw.githubusercontent.com/NotUnko/autoproxies/main/http.txt
https://raw.githubusercontent.com/officialputuid/KangProxy/KangProxy/http/http.txt
https://raw.githubusercontent.com/officialputuid/KangProxy/KangProxy/https/https.txt
https://raw.githubusercontent.com/proxy4parsing/proxy-list/main/http.txt
https://raw.githubusercontent.com/proxylist-to/proxy-list/main/http.txt
https://raw.githubusercontent.com/prxchk/proxy-list/main/http.txt
https://raw.githubusercontent.com/roosterkid/openproxylist/main/HTTPS_RAW.txt
https://raw.githubusercontent.com/saisuiu/Lionkings-Http-Proxys-Proxies/main/free.txt
https://raw.githubusercontent.com/ShiftyTR/Proxy-List/master/http.txt
https://raw.githubusercontent.com/ShiftyTR/Proxy-List/master/https.txt
https://raw.githubusercontent.com/sunny9577/proxy-scraper/master/generated/http_proxies.txt
https://raw.githubusercontent.com/TheSpeedX/PROXY-List/master/http.txt
https://raw.githubusercontent.com/TylerAmesIsGay/proxy-list/main/http_proxies%20(2).txt
https://raw.githubusercontent.com/TylerAmesIsGay/proxy-list/main/http_proxies%20(3).txt
https://raw.githubusercontent.com/yemixzy/proxy-list/main/proxies/http.txt
https://raw.githubusercontent.com/Zaeem20/FREE_PROXIES_LIST/master/http.txt
https://raw.githubusercontent.com/Zaeem20/FREE_PROXIES_LIST/master/https.txt
https://raw.githubusercontent.com/zevtyardt/proxy-list/main/http.txt
https://raw.githubusercontent.com/zloi-user/hideip.me/main/http.txt
https://raw.githubusercontent.com/zloi-user/hideip.me/main/https.txt
https://rootjazz.com/proxies/proxies.txt
https://www.proxy-list.download/api/v1/get?type=http
https://www.proxy-list.download/api/v1/get?type=https
[SOCKS4]
Enabled = yes
Sources =
https://api.proxyscrape.com/v2/?request=getproxies&protocol=socks4
; https://openproxy.space/list/socks4
https://openproxylist.xyz/socks4.txt
https://proxyspace.pro/socks4.txt
https://raw.githubusercontent.com/ALIILAPRO/Proxy/main/socks4.txt
https://raw.githubusercontent.com/Anonym0usWork1221/Free-Proxies/main/proxy_files/socks4_proxies.txt
https://raw.githubusercontent.com/B4RC0DE-TM/proxy-list/main/SOCKS4.txt
https://raw.githubusercontent.com/ErcinDedeoglu/proxies/main/proxies/socks4.txt
https://raw.githubusercontent.com/iptotal/free-proxy-list/master/socks4.txt
https://raw.githubusercontent.com/jetkai/proxy-list/main/online-proxies/txt/proxies-socks4.txt
https://raw.githubusercontent.com/mmpx12/proxy-list/master/socks4.txt
https://raw.githubusercontent.com/MuRongPIG/Proxy-Master/main/socks4.txt
https://raw.githubusercontent.com/NotUnko/autoproxies/main/socks4.txt
https://raw.githubusercontent.com/officialputuid/KangProxy/KangProxy/socks4/socks4.txt
https://raw.githubusercontent.com/proxylist-to/proxy-list/main/socks4.txt
https://raw.githubusercontent.com/prxchk/proxy-list/main/socks4.txt
https://raw.githubusercontent.com/roosterkid/openproxylist/main/SOCKS4_RAW.txt
https://raw.githubusercontent.com/ShiftyTR/Proxy-List/master/socks4.txt
https://raw.githubusercontent.com/sunny9577/proxy-scraper/master/generated/socks4_proxies.txt
https://raw.githubusercontent.com/TheSpeedX/PROXY-List/master/socks4.txt
https://raw.githubusercontent.com/TylerAmesIsGay/proxy-list/main/socks4_proxies%20(1).txt
https://raw.githubusercontent.com/yemixzy/proxy-list/main/proxies/socks4.txt
https://raw.githubusercontent.com/Zaeem20/FREE_PROXIES_LIST/master/socks4.txt
https://raw.githubusercontent.com/zevtyardt/proxy-list/main/socks4.txt
https://raw.githubusercontent.com/zloi-user/hideip.me/main/socks4.txt
https://www.proxy-list.download/api/v1/get?type=socks4
[SOCKS5]
Enabled = yes
Sources =
https://api.proxyscrape.com/v2/?request=getproxies&protocol=socks5
; https://openproxy.space/list/socks5
https://openproxylist.xyz/socks5.txt
https://proxyspace.pro/socks5.txt
https://raw.githubusercontent.com/ALIILAPRO/Proxy/main/socks5.txt
https://raw.githubusercontent.com/Anonym0usWork1221/Free-Proxies/main/proxy_files/socks5_proxies.txt
https://raw.githubusercontent.com/B4RC0DE-TM/proxy-list/main/SOCKS5.txt
https://raw.githubusercontent.com/ErcinDedeoglu/proxies/main/proxies/socks5.txt
https://raw.githubusercontent.com/hookzof/socks5_list/master/proxy.txt
https://raw.githubusercontent.com/iptotal/free-proxy-list/master/socks5.txt
https://raw.githubusercontent.com/jetkai/proxy-list/main/online-proxies/txt/proxies-socks5.txt
https://raw.githubusercontent.com/manuGMG/proxy-365/main/SOCKS5.txt
https://raw.githubusercontent.com/mmpx12/proxy-list/master/socks5.txt
https://raw.githubusercontent.com/MuRongPIG/Proxy-Master/main/socks5.txt
https://raw.githubusercontent.com/NotUnko/autoproxies/main/socks5.txt
https://raw.githubusercontent.com/officialputuid/KangProxy/KangProxy/socks5/socks5.txt
https://raw.githubusercontent.com/proxylist-to/proxy-list/main/socks5.txt
https://raw.githubusercontent.com/prxchk/proxy-list/main/socks5.txt
https://raw.githubusercontent.com/roosterkid/openproxylist/main/SOCKS5_RAW.txt
https://raw.githubusercontent.com/ShiftyTR/Proxy-List/master/socks5.txt
https://raw.githubusercontent.com/steamboatid/proxyout/main/proxy-socks5.txt
https://raw.githubusercontent.com/sunny9577/proxy-scraper/master/generated/socks5_proxies.txt
https://raw.githubusercontent.com/TheSpeedX/PROXY-List/master/socks5.txt
https://raw.githubusercontent.com/TylerAmesIsGay/proxy-list/main/socks5_proxies.txt
https://raw.githubusercontent.com/yemixzy/proxy-list/main/proxies/socks5.txt
https://raw.githubusercontent.com/Zaeem20/FREE_PROXIES_LIST/master/socks5.txt
https://raw.githubusercontent.com/zevtyardt/proxy-list/main/socks5.txt
https://raw.githubusercontent.com/zloi-user/hideip.me/main/socks5.txt
https://www.proxy-list.download/api/v1/get?type=socks5
config.toml
+163
View File
@@ -0,0 +1,163 @@
# The number of seconds to wait for a proxied request.
# The higher the number, the longer the check will take
# and the more proxies you get.
timeout = 5
# The number of seconds to wait for the proxies to be downloaded from the source.
source_timeout = 15
# Maximum concurrent connections.
# Windows supports maximum of 512.
# On *nix operating systems, this restriction is much looser.
# The limit on *nix can be seen with the command `ulimit -Hn`.
# Don't be in a hurry to set high values.
# Make sure you have enough RAM first, gradually increasing the default value.
# If set to 0, the maximum value available for your OS will be used.
max_connections = 512
# URL to which to send a request to check the proxy.
# If it is not httpbin or a service that returns an IP address,
# it will be impossible to determine the anonymity and geolocation of the proxy.
# Example:
# check_website = "https://google.com"
# check_website = "https://checkip.amazonaws.com"
check_website = "https://httpbin.smp.io/ip"
# Set to false to sort proxies alphabetically.
sort_by_speed = true
# Save the geolocation of the proxy exit-node.
# Uses offline geolocation database.
# Only works for json output.
enable_geolocation = true
# Enable output of all errors. Significantly slows down proxy checking.
debug = false
[output]
# Path to the folder where the proxies will be saved.
# Leave blank to save the proxies to the current directory.
path = "./results"
json = true
txt = true
# If you have sources with a protocol prefix, such as socks5://127.0.0.1:1050, you can add them to any of these sections.
[http]
enabled = true
sources = [
# Local proxy list example
# "./http_proxies.txt",
"https://api.proxyscrape.com/v2/?request=getproxies&protocol=http",
"https://openproxy.space/list/http",
"https://openproxylist.xyz/http.txt",
"https://proxyspace.pro/http.txt",
"https://proxyspace.pro/https.txt",
"https://raw.githubusercontent.com/ALIILAPRO/Proxy/main/http.txt",
"https://raw.githubusercontent.com/andigwandi/free-proxy/main/proxy_list.txt",
"https://raw.githubusercontent.com/Anonym0usWork1221/Free-Proxies/main/proxy_files/http_proxies.txt",
"https://raw.githubusercontent.com/Anonym0usWork1221/Free-Proxies/main/proxy_files/https_proxies.txt",
"https://raw.githubusercontent.com/aslisk/proxyhttps/main/https.txt",
"https://raw.githubusercontent.com/B4RC0DE-TM/proxy-list/main/HTTP.txt",
"https://raw.githubusercontent.com/clarketm/proxy-list/master/proxy-list-raw.txt",
"https://raw.githubusercontent.com/ErcinDedeoglu/proxies/main/proxies/http.txt",
"https://raw.githubusercontent.com/ErcinDedeoglu/proxies/main/proxies/https.txt",
"https://raw.githubusercontent.com/hendrikbgr/Free-Proxy-Repo/master/proxy_list.txt",
"https://raw.githubusercontent.com/jetkai/proxy-list/main/online-proxies/txt/proxies-http.txt",
"https://raw.githubusercontent.com/jetkai/proxy-list/main/online-proxies/txt/proxies-https.txt",
"https://raw.githubusercontent.com/mmpx12/proxy-list/master/http.txt",
"https://raw.githubusercontent.com/mmpx12/proxy-list/master/https.txt",
"https://raw.githubusercontent.com/MuRongPIG/Proxy-Master/main/http.txt",
"https://raw.githubusercontent.com/NotUnko/autoproxies/main/http.txt",
"https://raw.githubusercontent.com/officialputuid/KangProxy/KangProxy/http/http.txt",
"https://raw.githubusercontent.com/officialputuid/KangProxy/KangProxy/https/https.txt",
"https://raw.githubusercontent.com/proxy4parsing/proxy-list/main/http.txt",
"https://raw.githubusercontent.com/proxylist-to/proxy-list/main/http.txt",
"https://raw.githubusercontent.com/prxchk/proxy-list/main/http.txt",
"https://raw.githubusercontent.com/roosterkid/openproxylist/main/HTTPS_RAW.txt",
"https://raw.githubusercontent.com/saisuiu/Lionkings-Http-Proxys-Proxies/main/free.txt",
"https://raw.githubusercontent.com/ShiftyTR/Proxy-List/master/http.txt",
"https://raw.githubusercontent.com/ShiftyTR/Proxy-List/master/https.txt",
"https://raw.githubusercontent.com/sunny9577/proxy-scraper/master/generated/http_proxies.txt",
"https://raw.githubusercontent.com/TheSpeedX/PROXY-List/master/http.txt",
"https://raw.githubusercontent.com/TylerAmesIsGay/proxy-list/main/http_proxies%20(2).txt",
"https://raw.githubusercontent.com/TylerAmesIsGay/proxy-list/main/http_proxies%20(3).txt",
"https://raw.githubusercontent.com/yemixzy/proxy-list/main/proxies/http.txt",
"https://raw.githubusercontent.com/Zaeem20/FREE_PROXIES_LIST/master/http.txt",
"https://raw.githubusercontent.com/Zaeem20/FREE_PROXIES_LIST/master/https.txt",
"https://raw.githubusercontent.com/zevtyardt/proxy-list/main/http.txt",
"https://raw.githubusercontent.com/zloi-user/hideip.me/main/http.txt",
"https://raw.githubusercontent.com/zloi-user/hideip.me/main/https.txt",
"https://rootjazz.com/proxies/proxies.txt",
"https://www.proxy-list.download/api/v1/get?type=http",
"https://www.proxy-list.download/api/v1/get?type=https",
]
[socks4]
enabled = true
sources = [
# Local proxy list example
# "./socks4_proxies.txt",
"https://api.proxyscrape.com/v2/?request=getproxies&protocol=socks4",
"https://openproxy.space/list/socks4",
"https://openproxylist.xyz/socks4.txt",
"https://proxyspace.pro/socks4.txt",
"https://raw.githubusercontent.com/ALIILAPRO/Proxy/main/socks4.txt",
"https://raw.githubusercontent.com/Anonym0usWork1221/Free-Proxies/main/proxy_files/socks4_proxies.txt",
"https://raw.githubusercontent.com/B4RC0DE-TM/proxy-list/main/SOCKS4.txt",
"https://raw.githubusercontent.com/ErcinDedeoglu/proxies/main/proxies/socks4.txt",
"https://raw.githubusercontent.com/iptotal/free-proxy-list/master/socks4.txt",
"https://raw.githubusercontent.com/jetkai/proxy-list/main/online-proxies/txt/proxies-socks4.txt",
"https://raw.githubusercontent.com/mmpx12/proxy-list/master/socks4.txt",
"https://raw.githubusercontent.com/MuRongPIG/Proxy-Master/main/socks4.txt",
"https://raw.githubusercontent.com/NotUnko/autoproxies/main/socks4.txt",
"https://raw.githubusercontent.com/officialputuid/KangProxy/KangProxy/socks4/socks4.txt",
"https://raw.githubusercontent.com/proxylist-to/proxy-list/main/socks4.txt",
"https://raw.githubusercontent.com/prxchk/proxy-list/main/socks4.txt",
"https://raw.githubusercontent.com/roosterkid/openproxylist/main/SOCKS4_RAW.txt",
"https://raw.githubusercontent.com/ShiftyTR/Proxy-List/master/socks4.txt",
"https://raw.githubusercontent.com/sunny9577/proxy-scraper/master/generated/socks4_proxies.txt",
"https://raw.githubusercontent.com/TheSpeedX/PROXY-List/master/socks4.txt",
"https://raw.githubusercontent.com/TylerAmesIsGay/proxy-list/main/socks4_proxies%20(1).txt",
"https://raw.githubusercontent.com/yemixzy/proxy-list/main/proxies/socks4.txt",
"https://raw.githubusercontent.com/Zaeem20/FREE_PROXIES_LIST/master/socks4.txt",
"https://raw.githubusercontent.com/zevtyardt/proxy-list/main/socks4.txt",
"https://raw.githubusercontent.com/zloi-user/hideip.me/main/socks4.txt",
"https://www.proxy-list.download/api/v1/get?type=socks4",
]
[socks5]
enabled = true
sources = [
# Local proxy list example
# "./socks5_proxies.txt",
"https://api.proxyscrape.com/v2/?request=getproxies&protocol=socks5",
"https://openproxy.space/list/socks5",
"https://openproxylist.xyz/socks5.txt",
"https://proxyspace.pro/socks5.txt",
"https://raw.githubusercontent.com/ALIILAPRO/Proxy/main/socks5.txt",
"https://raw.githubusercontent.com/Anonym0usWork1221/Free-Proxies/main/proxy_files/socks5_proxies.txt",
"https://raw.githubusercontent.com/B4RC0DE-TM/proxy-list/main/SOCKS5.txt",
"https://raw.githubusercontent.com/ErcinDedeoglu/proxies/main/proxies/socks5.txt",
"https://raw.githubusercontent.com/hookzof/socks5_list/master/proxy.txt",
"https://raw.githubusercontent.com/iptotal/free-proxy-list/master/socks5.txt",
"https://raw.githubusercontent.com/jetkai/proxy-list/main/online-proxies/txt/proxies-socks5.txt",
"https://raw.githubusercontent.com/manuGMG/proxy-365/main/SOCKS5.txt",
"https://raw.githubusercontent.com/mmpx12/proxy-list/master/socks5.txt",
"https://raw.githubusercontent.com/MuRongPIG/Proxy-Master/main/socks5.txt",
"https://raw.githubusercontent.com/NotUnko/autoproxies/main/socks5.txt",
"https://raw.githubusercontent.com/officialputuid/KangProxy/KangProxy/socks5/socks5.txt",
"https://raw.githubusercontent.com/proxylist-to/proxy-list/main/socks5.txt",
"https://raw.githubusercontent.com/prxchk/proxy-list/main/socks5.txt",
"https://raw.githubusercontent.com/roosterkid/openproxylist/main/SOCKS5_RAW.txt",
"https://raw.githubusercontent.com/ShiftyTR/Proxy-List/master/socks5.txt",
"https://raw.githubusercontent.com/steamboatid/proxyout/main/proxy-socks5.txt",
"https://raw.githubusercontent.com/sunny9577/proxy-scraper/master/generated/socks5_proxies.txt",
"https://raw.githubusercontent.com/TheSpeedX/PROXY-List/master/socks5.txt",
"https://raw.githubusercontent.com/TylerAmesIsGay/proxy-list/main/socks5_proxies.txt",
"https://raw.githubusercontent.com/yemixzy/proxy-list/main/proxies/socks5.txt",
"https://raw.githubusercontent.com/Zaeem20/FREE_PROXIES_LIST/master/socks5.txt",
"https://raw.githubusercontent.com/zevtyardt/proxy-list/main/socks5.txt",
"https://raw.githubusercontent.com/zloi-user/hideip.me/main/socks5.txt",
"https://www.proxy-list.download/api/v1/get?type=socks5",
]
install-termux.sh
-2
View File
@@ -11,6 +11,4 @@ fi &&
curl -fsSLo "${download_path}" 'https://github.com/monosans/proxy-scraper-checker/archive/refs/heads/main.zip' &&
unzip -d "${base_path}" "${download_path}" &&
mv "${path}-main" "${path}" &&
python3 -m pip install -U --no-cache-dir --disable-pip-version-check setuptools wheel &&
python3 -m pip install -U --no-cache-dir --disable-pip-version-check -r "${path}/requirements-termux.txt" &&
printf "proxy-scraper-checker installed successfully.\nRun 'cd %s && sh start-termux.sh'.\n" "${path}"
poetry.lock
Generated
+1284
View File
File diff suppressed because it is too large Load Diff
proxy_scraper_checker/__main__.py
+101 -12
View File
@@ -2,20 +2,42 @@ from __future__ import annotations
import asyncio
import logging
import platform
import sys
from configparser import ConfigParser
from typing import TYPE_CHECKING, Dict, Mapping
import aiofiles
import aiofiles.os
import rich.traceback
from aiohttp import ClientSession, TCPConnector
from aiohttp_socks import ProxyType
from rich.console import Console
from rich.logging import RichHandler
from rich.progress import BarColumn, MofNCompleteColumn, Progress, TextColumn
from rich.table import Table
from .proxy_scraper_checker import ProxyScraperChecker
from . import cache, checker, geodb, http, output, scraper, sort, utils
from .settings import Settings
from .storage import ProxyStorage
from .typing_compat import Any
if sys.version_info >= (3, 11):
try:
import tomllib
except ImportError:
# Help users on older alphas
if not TYPE_CHECKING:
import tomli as tomllib
else:
import tomli as tomllib
logger = logging.getLogger(__name__)
def set_event_loop_policy() -> None:
if sys.platform == "win32":
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
elif sys.implementation.name == "cpython" and sys.platform in {
elif platform.python_implementation() == "CPython" and sys.platform in {
"darwin",
"linux",
}:
@@ -27,7 +49,13 @@ def set_event_loop_policy() -> None:
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
def configure_logging(console: Console, *, debug: bool) -> None:
async def read_config(file: str, /) -> Dict[str, Any]:
async with aiofiles.open(file, "rb") as f:
content = await f.read()
return tomllib.loads(utils.bytes_decode(content))
def configure_logging(*, console: Console, debug: bool) -> None:
rich.traceback.install(
console=console, width=None, extra_lines=0, word_wrap=True
)
@@ -47,19 +75,80 @@ def configure_logging(console: Console, *, debug: bool) -> None:
)
def get_config(file: str) -> ConfigParser:
cfg = ConfigParser(interpolation=None)
cfg.read(file, encoding="utf-8")
return cfg
def get_summary_table(
*, before: Mapping[ProxyType, int], after: Mapping[ProxyType, int]
) -> Table:
table = Table()
table.add_column("Protocol", style="cyan")
table.add_column("Working", style="magenta")
table.add_column("Total", style="green")
for proto in sort.PROTOCOL_ORDER:
if total := before.get(proto):
working = after.get(proto, 0)
percentage = working / total
table.add_row(
proto.name, f"{working} ({percentage:.1%})", str(total)
)
return table
async def main() -> None:
cfg = get_config("config.ini")
cfg = await read_config("config.toml")
console = Console()
configure_logging(console, debug=cfg["General"].getboolean("Debug", False))
configure_logging(console=console, debug=cfg["debug"])
await ProxyScraperChecker.from_configparser(cfg, console=console).run()
storage = ProxyStorage()
async with ClientSession(
connector=TCPConnector(ssl=http.SSL_CONTEXT),
headers=http.HEADERS,
cookie_jar=http.get_cookie_jar(),
fallback_charset_resolver=http.fallback_charset_resolver,
) as session:
settings = await Settings.from_mapping(cfg, session=session)
with Progress(
TextColumn("[progress.description]{task.description}"),
BarColumn(),
MofNCompleteColumn(),
console=console,
transient=True,
) as progress:
scrape = scraper.scrape_all(
progress=progress,
session=session,
settings=settings,
storage=storage,
)
await (
asyncio.gather(
cache.create(),
geodb.download_geodb(progress=progress, session=session),
scrape,
)
if settings.enable_geolocation
else scrape
)
await session.close()
count_before_checking = storage.get_count()
await checker.check_all(
settings=settings,
storage=storage,
progress=progress,
proxies_count=count_before_checking,
)
count_after_checking = storage.get_count()
console.print(
get_summary_table(
before=count_before_checking, after=count_after_checking
)
)
await output.save_proxies(storage=storage, settings=settings)
logger.info(
"Thank you for using https://github.com/monosans/proxy-scraper-checker"
)
if __name__ == "__main__":
proxy_scraper_checker/cache.py
+14
View File
@@ -0,0 +1,14 @@
from __future__ import annotations
import asyncio
import aiofiles.os
import platformdirs
DIR = platformdirs.user_cache_dir("proxy_scraper_checker")
READY_EVENT = asyncio.Event()
async def create() -> None:
await aiofiles.os.makedirs(DIR, exist_ok=True)
READY_EVENT.set()
proxy_scraper_checker/checker.py
+67
View File
@@ -0,0 +1,67 @@
from __future__ import annotations
import asyncio
import logging
from random import shuffle
from typing import Mapping
from aiohttp_socks import ProxyType
from rich.progress import Progress, TaskID
from . import sort
from .proxy import Proxy
from .settings import Settings
from .storage import ProxyStorage
logger = logging.getLogger(__name__)
async def check_one(
*,
progress: Progress,
proxy: Proxy,
settings: Settings,
storage: ProxyStorage,
task: TaskID,
) -> None:
try:
await proxy.check(settings=settings)
except Exception as e:
# Too many open files
if isinstance(e, OSError) and e.errno == 24: # noqa: PLR2004
logger.error("Please, set max_connections to lower value")
logger.debug(
"%s.%s: %s", e.__class__.__module__, e.__class__.__qualname__, e
)
storage.remove(proxy)
progress.update(task, advance=1)
async def check_all(
*,
settings: Settings,
storage: ProxyStorage,
progress: Progress,
proxies_count: Mapping[ProxyType, int],
) -> None:
tasks = {
proto: progress.add_task(
f"[yellow]Checker [red]:: [green]{proto.name}",
total=proxies_count[proto],
)
for proto in sort.PROTOCOL_ORDER
if proto in storage.enabled_protocols
}
coroutines = [
check_one(
progress=progress,
proxy=proxy,
settings=settings,
storage=storage,
task=tasks[proxy.protocol],
)
for proxy in storage
]
shuffle(coroutines)
await asyncio.gather(*coroutines)
proxy_scraper_checker/constants.py
-12
View File
@@ -1,12 +0,0 @@
from __future__ import annotations
from types import MappingProxyType
from aiohttp import hdrs
DEFAULT_CHECK_WEBSITE = "http://ip-api.com/json/?fields=8217"
HEADERS: MappingProxyType[str, str] = MappingProxyType({
hdrs.USER_AGENT: (
"Mozilla/5.0 (Windows NT 10.0; rv:121.0) Gecko/20100101 Firefox/121.0"
)
})
proxy_scraper_checker/folder.py
-24
View File
@@ -1,24 +0,0 @@
from __future__ import annotations
from dataclasses import dataclass
from pathlib import Path
from shutil import rmtree
@dataclass(repr=False, eq=False)
class Folder:
__slots__ = ("path", "is_enabled", "for_anonymous", "for_geolocation")
path: Path
is_enabled: bool
for_anonymous: bool
for_geolocation: bool
def remove(self) -> None:
try:
rmtree(self.path)
except FileNotFoundError:
pass
def create(self) -> None:
self.path.mkdir(parents=True, exist_ok=True)
proxy_scraper_checker/geodb.py
+76
View File
@@ -0,0 +1,76 @@
from __future__ import annotations
import logging
from pathlib import Path
from typing import Optional
import aiofiles
import aiofiles.ospath
from aiohttp import ClientResponse, ClientSession, hdrs
from rich.progress import Progress, TaskID
from . import cache
from .utils import bytes_decode
logger = logging.getLogger(__name__)
GEODB_URL = "https://raw.githubusercontent.com/P3TERX/GeoLite.mmdb/download/GeoLite2-City.mmdb"
GEODB_PATH = Path(cache.DIR, "geolocation_database.mmdb")
GEODB_ETAG_PATH = GEODB_PATH.with_suffix(".mmdb.etag")
async def _read_etag() -> Optional[str]:
try:
async with aiofiles.open(GEODB_ETAG_PATH, "rb") as etag_file:
content = await etag_file.read()
except FileNotFoundError:
return None
return bytes_decode(content)
async def _save_etag(etag: str, /) -> None:
async with aiofiles.open(
GEODB_ETAG_PATH, "w", encoding="utf-8"
) as etag_file:
await etag_file.write(etag)
async def _save_geodb(
*, progress: Progress, response: ClientResponse, task: TaskID
) -> None:
async with aiofiles.open(GEODB_PATH, "wb") as geodb:
async for chunk in response.content.iter_any():
await geodb.write(chunk)
progress.update(task, advance=len(chunk))
async def download_geodb(*, progress: Progress, session: ClientSession) -> None:
headers = (
{hdrs.IF_NONE_MATCH: current_etag}
if await aiofiles.ospath.exists(GEODB_PATH)
and (current_etag := await _read_etag())
else None
)
async with session.get(
GEODB_URL, headers=headers, raise_for_status=True
) as response:
if response.status == 304: # noqa: PLR2004
logger.info(
"Latest geolocation database is already cached at %s",
GEODB_PATH,
)
return
await cache.READY_EVENT.wait()
await _save_geodb(
progress=progress,
response=response,
task=progress.add_task(
"[yellow]Downloader[red] :: [green]GeoDB",
total=response.content_length,
),
)
logger.info("Downloaded geolocation database to %s", GEODB_PATH)
if etag := response.headers.get(hdrs.ETAG):
await _save_etag(etag)
proxy_scraper_checker/http.py
+25
View File
@@ -0,0 +1,25 @@
from __future__ import annotations
import ssl
from functools import lru_cache
from types import MappingProxyType
import certifi
import charset_normalizer
from aiohttp import ClientResponse, DummyCookieJar, hdrs
SSL_CONTEXT = ssl.create_default_context(cafile=certifi.where())
HEADERS: MappingProxyType[str, str] = MappingProxyType({
hdrs.USER_AGENT: (
"Mozilla/5.0 (Windows NT 10.0; rv:121.0) Gecko/20100101 Firefox/121.0"
)
})
@lru_cache(None)
def get_cookie_jar() -> DummyCookieJar:
return DummyCookieJar()
def fallback_charset_resolver(r: ClientResponse, b: bytes) -> str: # noqa: ARG001
return charset_normalizer.from_bytes(b)[0].encoding
proxy_scraper_checker/null_context.py
+7 -1
View File
@@ -1,9 +1,15 @@
from __future__ import annotations
class AsyncNullContext:
class NullContext:
__slots__ = ()
def __enter__(self) -> None:
pass
def __exit__(self, *_: object) -> None:
pass
async def __aenter__(self) -> None:
pass
proxy_scraper_checker/output.py
+100
View File
@@ -0,0 +1,100 @@
from __future__ import annotations
import json
import logging
from shutil import rmtree
from typing import Sequence, Union
import aiofiles
import aiofiles.os
import aiofiles.ospath
import maxminddb
from . import sort
from .geodb import GEODB_PATH
from .null_context import NullContext
from .proxy import Proxy
from .settings import Settings
from .storage import ProxyStorage
logger = logging.getLogger(__name__)
def _create_proxy_list_str(
*, anonymous_only: bool, include_protocol: bool, proxies: Sequence[Proxy]
) -> str:
return "\n".join(
proxy.as_str(include_protocol=include_protocol)
for proxy in proxies
if not anonymous_only or proxy.host != proxy.exit_ip
)
@aiofiles.ospath.wrap
def save_proxies(*, settings: Settings, storage: ProxyStorage) -> None:
if settings.output_json:
mmdb: Union[maxminddb.Reader, NullContext] = (
maxminddb.open_database(GEODB_PATH)
if settings.enable_geolocation
else NullContext()
)
settings.output_path.mkdir(parents=True, exist_ok=True)
with mmdb as mmdb_reader:
proxy_dicts = [
{
"protocol": proxy.protocol.name.lower(),
"username": proxy.username,
"password": proxy.password,
"host": proxy.host,
"port": proxy.port,
"exit_ip": proxy.exit_ip,
"timeout": round(proxy.timeout, 2),
"geolocation": mmdb_reader.get(proxy.exit_ip)
if mmdb_reader is not None
else None,
}
for proxy in sorted(storage, key=sort.timeout_sort_key)
]
with (settings.output_path / "proxies.json").open(
"w", encoding="utf-8"
) as f:
json.dump(
proxy_dicts, f, ensure_ascii=False, separators=(",", ":")
)
with (settings.output_path / "proxies_pretty.json").open(
"w", encoding="utf-8"
) as f:
json.dump(proxy_dicts, f, ensure_ascii=False, indent="\t")
if settings.output_txt:
sorted_proxies = storage.get_sorted(key=settings.sorting_key)
grouped_proxies = tuple(
(k, sorted(v, key=settings.sorting_key))
for k, v in storage.get_grouped().items()
)
for folder, anonymous_only in (
(settings.output_path / "proxies", False),
(settings.output_path / "proxies_anonymous", True),
):
try:
rmtree(folder)
except FileNotFoundError:
pass
folder.mkdir(parents=True, exist_ok=True)
text = _create_proxy_list_str(
proxies=sorted_proxies,
anonymous_only=anonymous_only,
include_protocol=True,
)
(folder / "all.txt").write_text(text, encoding="utf-8")
for proto, proxies in grouped_proxies:
text = _create_proxy_list_str(
proxies=proxies,
anonymous_only=anonymous_only,
include_protocol=False,
)
(folder / f"{proto.name.lower()}.txt").write_text(
text, encoding="utf-8"
)
logger.info(
"Proxies have been saved to %s", settings.output_path.absolute()
)
proxy_scraper_checker/parsers.py
+19
View File
@@ -0,0 +1,19 @@
from __future__ import annotations
import re
PROXY_REGEX = re.compile(
r"(?:^|[^\.\/\d:@A-Za-z])(?:(?P<protocol>https?|socks[45]):\/\/)?(?:(?P<username>[\dA-Za-z]*):(?P<password>[\dA-Za-z]*)@)?(?P<host>(?:\d|[1-9]\d|1\d{2}|2[0-4]\d|25[0-5])(?:\.(?:\d|[1-9]\d|1\d{2}|2[0-4]\d|25[0-5])){3}):(?P<port>\d|[1-9]\d{1,3}|[1-5]\d{4}|6[0-4]\d{3}|65[0-4]\d{2}|655[0-2]\d|6553[0-5])(?=[^\.\/\d:@A-Za-z]|$)",
flags=re.MULTILINE | re.IGNORECASE,
)
IPV4_REGEX = re.compile(
r"^\s*(?P<host>(?:\d|[1-9]\d|1\d{2}|2[0-4]\d|25[0-5])(?:\.(?:\d|[1-9]\d|1\d{2}|2[0-4]\d|25[0-5])){3})(?::(?:\d|[1-9]\d{1,3}|[1-5]\d{4}|6[0-4]\d{3}|65[0-4]\d{2}|655[0-2]\d|6553[0-5]))?\s*$"
)
def parse_ipv4(s: str, /) -> str:
"""Parse ip from ip:port."""
match = IPV4_REGEX.match(s)
if not match:
raise ValueError
return match.group("host")
proxy_scraper_checker/proxy.py
+52 -37
View File
@@ -1,59 +1,74 @@
from __future__ import annotations
import asyncio
from dataclasses import dataclass
from io import StringIO
from time import perf_counter
from typing import Union
from typing import Optional
from aiohttp import ClientSession, ClientTimeout
from aiohttp.abc import AbstractCookieJar
import attrs
from aiohttp import ClientSession
from aiohttp_socks import ProxyConnector, ProxyType
from .constants import HEADERS
from .null_context import AsyncNullContext
from .http import (
HEADERS,
SSL_CONTEXT,
fallback_charset_resolver,
get_cookie_jar,
)
from .parsers import parse_ipv4
from .settings import CheckWebsiteType, Settings
@dataclass(repr=False, unsafe_hash=True)
@attrs.define(
repr=False,
unsafe_hash=True,
weakref_slot=False,
kw_only=True,
eq=False,
getstate_setstate=False,
match_args=False,
)
class Proxy:
__slots__ = ("geolocation", "host", "is_anonymous", "port", "timeout")
protocol: ProxyType
host: str
port: int
username: Optional[str]
password: Optional[str]
timeout: float = attrs.field(hash=False, init=False)
exit_ip: str = attrs.field(hash=False, init=False)
async def check(
self,
*,
website: str,
sem: Union[asyncio.Semaphore, AsyncNullContext],
cookie_jar: AbstractCookieJar,
proto: ProxyType,
timeout: ClientTimeout,
set_geolocation: bool,
) -> None:
async with sem:
async def check(self, *, settings: Settings) -> None:
async with settings.semaphore:
start = perf_counter()
connector = ProxyConnector(
proxy_type=proto, host=self.host, port=self.port
proxy_type=self.protocol,
host=self.host,
port=self.port,
username=self.username,
password=self.password,
ssl=SSL_CONTEXT,
)
async with ClientSession(
connector=connector,
cookie_jar=cookie_jar,
timeout=timeout,
headers=HEADERS,
cookie_jar=get_cookie_jar(),
timeout=settings.timeout,
fallback_charset_resolver=fallback_charset_resolver,
) as session, session.get(
website, raise_for_status=True
settings.check_website, raise_for_status=True
) as response:
if set_geolocation:
await response.read()
await response.read()
self.timeout = perf_counter() - start
if set_geolocation:
data = await response.json(content_type=None)
self.is_anonymous = self.host != data["query"]
self.geolocation = (
f"|{data['country']}|{data['regionName']}|{data['city']}"
)
if settings.check_website_type == CheckWebsiteType.HTTPBIN_IP:
r = await response.json(content_type=None)
self.exit_ip = r["origin"]
elif settings.check_website_type == CheckWebsiteType.PLAIN_IP:
self.exit_ip = parse_ipv4(await response.text())
def as_str(self, *, include_geolocation: bool) -> str:
if include_geolocation:
return f"{self.host}:{self.port}{self.geolocation}"
return f"{self.host}:{self.port}"
def as_str(self, *, include_protocol: bool) -> str:
with StringIO() as buf:
if include_protocol:
buf.write(f"{self.protocol.name.lower()}://")
if self.username is not None and self.password is not None:
buf.write(f"{self.username}:{self.password}@")
buf.write(f"{self.host}:{self.port}")
return buf.getvalue()
proxy_scraper_checker/proxy_scraper_checker.py
-441
View File
@@ -1,441 +0,0 @@
from __future__ import annotations
import asyncio
import logging
import re
from configparser import ConfigParser
from pathlib import Path
from random import shuffle
from typing import (
Callable,
Dict,
FrozenSet,
Iterable,
List,
Optional,
Set,
Tuple,
Union,
)
from aiohttp import ClientSession, ClientTimeout, DummyCookieJar
from aiohttp_socks import ProxyType
from rich.console import Console
from rich.progress import (
BarColumn,
MofNCompleteColumn,
Progress,
TaskID,
TextColumn,
)
from rich.table import Table
from typing_extensions import Self
from . import sort, validators
from .constants import DEFAULT_CHECK_WEBSITE, HEADERS
from .folder import Folder
from .null_context import AsyncNullContext
from .proxy import Proxy
logger = logging.getLogger(__name__)
class ProxyScraperChecker:
"""HTTP, SOCKS4, SOCKS5 proxies scraper and checker."""
__slots__ = (
"check_website",
"console",
"cookie_jar",
"folders",
"geolocation_enabled",
"path",
"proxies_count",
"proxies",
"regex",
"sem",
"sort_by_speed",
"source_timeout",
"sources",
"timeout",
)
def __init__(
self,
*,
timeout: float,
source_timeout: float,
max_connections: int,
check_website: str,
sort_by_speed: bool,
save_path: Path,
folders: Iterable[Folder],
sources: Dict[ProxyType, Optional[str]],
console: Optional[Console] = None,
) -> None:
"""HTTP, SOCKS4, SOCKS5 proxies scraper and checker.
Args:
timeout: The number of seconds to wait for a proxied request.
The higher the number, the longer the check will take and the
more proxies you get.
source_timeout: The number of seconds to wait for the proxies to be
downloaded from the source.
max_connections: Maximum concurrent connections.
Windows supports maximum of 512.
On *nix operating systems, this restriction is much looser.
The limit on *nix can be seen with the command `ulimit -Hn`.
Don't be in a hurry to set high values.
Make sure you have enough RAM first, gradually increasing the
default value.
If set to 0, the maximum value available for your OS will be
used.
check_website: URL to which to send a request to check the proxy.
If not equal to 'default', it will not be possible to determine
the anonymity and geolocation of the proxies.
sort_by_speed: Set to False to sort proxies alphabetically.
save_path: Path to the folder where the proxy folders will be
saved.
Leave empty to save the proxies to the current directory.
"""
validators.timeout(timeout)
self.timeout = ClientTimeout(total=timeout, sock_connect=float("inf"))
validators.source_timeout(source_timeout)
self.source_timeout = source_timeout
max_conn = validators.max_connections(max_connections)
self.sem: Union[asyncio.Semaphore, AsyncNullContext] = (
asyncio.Semaphore(max_conn) if max_conn else AsyncNullContext()
)
self.check_website = check_website
self.sort_by_speed = sort_by_speed
self.path = save_path
self.folders = folders
if self.check_website == "default":
self.check_website = DEFAULT_CHECK_WEBSITE
if self.check_website == DEFAULT_CHECK_WEBSITE:
validators.folders(self.folders)
else:
validators.check_website(check_website)
logger.info(
"CheckWebsite is not 'default', "
"so it will not be possible to determine "
"the anonymity and geolocation of the proxies"
)
for folder in self.folders:
folder.is_enabled = (
not folder.for_anonymous and not folder.for_geolocation
)
self.geolocation_enabled = any(
self.check_website == DEFAULT_CHECK_WEBSITE
and folder.is_enabled
and folder.for_geolocation
for folder in self.folders
)
self.sources: Dict[ProxyType, FrozenSet[str]] = {
proto: frozenset(filter(None, sources.splitlines()))
for proto, sources in sources.items()
if sources
}
validators.sources(self.sources)
self.proxies: Dict[ProxyType, Set[Proxy]] = {
proto: set() for proto in self.sources
}
self.console = console or Console()
self.cookie_jar = DummyCookieJar()
self.regex = re.compile(
r"(?:^|\D)?("
r"(?:[1-9]|[1-9]\d|1\d{2}|2[0-4]\d|25[0-5])" # 1-255
+ r"\.(?:\d|[1-9]\d|1\d{2}|2[0-4]\d|25[0-5])" * 3 # 0-255
+ r"):"
+ (
r"(\d|[1-9]\d{1,3}|[1-5]\d{4}|6[0-4]\d{3}"
r"|65[0-4]\d{2}|655[0-2]\d|6553[0-5])"
) # 0-65535
+ r"(?:\D|$)"
)
@classmethod
def from_configparser(
cls, cfg: ConfigParser, *, console: Optional[Console] = None
) -> Self:
general = cfg["General"]
folders = cfg["Folders"]
http = cfg["HTTP"]
socks4 = cfg["SOCKS4"]
socks5 = cfg["SOCKS5"]
save_path = Path(general.get("SavePath", ""))
return cls(
timeout=general.getfloat("Timeout", 5),
source_timeout=general.getfloat("SourceTimeout", 15),
max_connections=general.getint("MaxConnections", 512),
check_website=general.get("CheckWebsite", "default"),
sort_by_speed=general.getboolean("SortBySpeed", True),
save_path=save_path,
folders=(
Folder(
path=save_path / "proxies",
is_enabled=folders.getboolean("proxies", True),
for_anonymous=False,
for_geolocation=False,
),
Folder(
path=save_path / "proxies_anonymous",
is_enabled=folders.getboolean("proxies_anonymous", True),
for_anonymous=True,
for_geolocation=False,
),
Folder(
path=save_path / "proxies_geolocation",
is_enabled=folders.getboolean("proxies_geolocation", True),
for_anonymous=False,
for_geolocation=True,
),
Folder(
path=save_path / "proxies_geolocation_anonymous",
is_enabled=folders.getboolean(
"proxies_geolocation_anonymous", True
),
for_anonymous=True,
for_geolocation=True,
),
),
sources={
ProxyType.HTTP: (
http.get("Sources")
if http.getboolean("Enabled", True)
else None
),
ProxyType.SOCKS4: (
socks4.get("Sources")
if socks4.getboolean("Enabled", True)
else None
),
ProxyType.SOCKS5: (
socks5.get("Sources")
if socks5.getboolean("Enabled", True)
else None
),
},
console=console,
)
async def fetch_source(
self,
*,
session: ClientSession,
source: str,
proto: ProxyType,
progress: Progress,
task: TaskID,
) -> None:
"""Get proxies from source.
Args:
source: Proxy list URL.
"""
try:
async with session.get(source) as response:
await response.read()
text = await response.text()
except asyncio.TimeoutError:
logger.warning("%s | Timed out", source)
except Exception as e:
e_str = str(e)
args: Tuple[object, ...] = (
(
"%s | %s.%s (%s)",
source,
e.__class__.__module__,
e.__class__.__qualname__,
e_str,
)
if e_str
else (
"%s | %s.%s",
source,
e.__class__.__module__,
e.__class__.__qualname__,
)
)
logger.error(*args)
else:
proxies = self.regex.finditer(text)
try:
proxy = next(proxies)
except StopIteration:
args = (
("%s | No proxies found", source)
if response.status == 200 # noqa: PLR2004
else ("%s | HTTP status code %d", source, response.status)
)
logger.warning(*args)
else:
proxies_set = self.proxies[proto]
proxies_set.add(
Proxy(host=proxy.group(1), port=int(proxy.group(2)))
)
for proxy in proxies:
proxies_set.add(
Proxy(host=proxy.group(1), port=int(proxy.group(2)))
)
progress.update(task, advance=1)
async def check_proxy(
self,
*,
proxy: Proxy,
proto: ProxyType,
progress: Progress,
task: TaskID,
) -> None:
"""Check if proxy is alive."""
try:
await proxy.check(
website=self.check_website,
sem=self.sem,
cookie_jar=self.cookie_jar,
proto=proto,
timeout=self.timeout,
set_geolocation=self.geolocation_enabled,
)
except Exception as e:
# Too many open files
if isinstance(e, OSError) and e.errno == 24: # noqa: PLR2004
logger.error("Please, set MaxConnections to lower value.")
logger.debug(
"%s.%s | %s",
e.__class__.__module__,
e.__class__.__qualname__,
e,
)
self.proxies[proto].remove(proxy)
progress.update(task, advance=1)
async def fetch_all_sources(self, progress: Progress) -> None:
tasks = {
proto: progress.add_task(
f"[yellow]Scraper [red]:: [green]{proto.name}",
total=len(sources),
)
for proto, sources in self.sources.items()
}
async with ClientSession(
headers=HEADERS,
cookie_jar=self.cookie_jar,
timeout=ClientTimeout(total=self.source_timeout),
) as session:
coroutines = (
self.fetch_source(
session=session,
source=source,
proto=proto,
progress=progress,
task=tasks[proto],
)
for proto, sources in self.sources.items()
for source in sources
)
await asyncio.gather(*coroutines)
self.proxies_count = {
proto: len(proxies) for proto, proxies in self.proxies.items()
}
async def check_all_proxies(self, progress: Progress) -> None:
tasks = {
proto: progress.add_task(
f"[yellow]Checker [red]:: [green]{proto.name}",
total=len(proxies),
)
for proto, proxies in self.proxies.items()
}
coroutines = [
self.check_proxy(
proxy=proxy, proto=proto, progress=progress, task=tasks[proto]
)
for proto, proxies in self.proxies.items()
for proxy in proxies
]
shuffle(coroutines)
await asyncio.gather(*coroutines)
def save_proxies(self) -> None:
"""Delete old proxies and save new ones."""
sorted_proxies = self.get_sorted_proxies().items()
for folder in self.folders:
folder.remove()
for folder in self.folders:
if not folder.is_enabled:
continue
folder.create()
for proto, proxies in sorted_proxies:
text = "\n".join(
proxy.as_str(include_geolocation=folder.for_geolocation)
for proxy in proxies
if (proxy.is_anonymous if folder.for_anonymous else True)
)
file = folder.path / f"{proto.name.lower()}.txt"
file.write_text(text, encoding="utf-8")
logger.info(
"Proxy folders have been created in the %s folder.",
self.path.resolve(),
)
async def run(self) -> None:
with self._get_progress_bar() as progress:
await self.fetch_all_sources(progress)
await self.check_all_proxies(progress)
table = self._get_results_table()
self.console.print(table)
self.save_proxies()
logger.info(
"Thank you for using "
"https://github.com/monosans/proxy-scraper-checker :)"
)
def get_sorted_proxies(self) -> Dict[ProxyType, List[Proxy]]:
key: Union[
Callable[[Proxy], float], Callable[[Proxy], Tuple[int, ...]]
] = (
sort.timeout_sort_key
if self.sort_by_speed
else sort.natural_sort_key
)
return {
proto: sorted(proxies, key=key)
for proto, proxies in self.proxies.items()
}
def _get_results_table(self) -> Table:
table = Table()
table.add_column("Protocol", style="cyan")
table.add_column("Working", style="magenta")
table.add_column("Total", style="green")
for proto, proxies in self.proxies.items():
working = len(proxies)
total = self.proxies_count[proto]
percentage = working / total if total else 0
table.add_row(
proto.name, f"{working} ({percentage:.1%})", str(total)
)
return table
def _get_progress_bar(self) -> Progress:
return Progress(
TextColumn("[progress.description]{task.description}"),
BarColumn(),
MofNCompleteColumn(),
console=self.console,
)
proxy_scraper_checker/scraper.py
+111
View File
@@ -0,0 +1,111 @@
from __future__ import annotations
import asyncio
import itertools
import logging
import aiofiles
import aiofiles.os
import aiofiles.ospath
from aiohttp import ClientSession, ClientTimeout
from aiohttp_socks import ProxyType
from rich.progress import Progress, TaskID
from .parsers import PROXY_REGEX
from .proxy import Proxy
from .settings import Settings
from .storage import ProxyStorage
from .utils import bytes_decode, is_url
logger = logging.getLogger(__name__)
async def scrape_one(
*,
progress: Progress,
proto: ProxyType,
session: ClientSession,
source: str,
storage: ProxyStorage,
task: TaskID,
timeout: ClientTimeout,
) -> None:
try:
if is_url(source):
async with session.get(source, timeout=timeout) as response:
await response.read()
text = await response.text()
else:
response = None
async with aiofiles.open(source, "rb") as f:
content = await f.read()
text = bytes_decode(content)
except Exception as e:
logger.warning(
"%s | %s.%s: %s",
source,
e.__class__.__module__,
e.__class__.__qualname__,
e,
)
else:
proxies = PROXY_REGEX.finditer(text)
try:
proxy = next(proxies)
except StopIteration:
if response and response.status != 200: # noqa: PLR2004
logger.warning(
"%s | HTTP status code %d", source, response.status
)
else:
logger.warning("%s | No proxies found", source)
else:
for proxy in itertools.chain((proxy,), proxies): # noqa: B020
try:
protocol = ProxyType[
"HTTP"
if (p := proxy.group("protocol").upper()) == "HTTPS"
else p
]
except AttributeError:
protocol = proto
storage.add(
Proxy(
protocol=protocol,
host=proxy.group("host"),
port=int(proxy.group("port")),
username=proxy.group("username"),
password=proxy.group("password"),
)
)
progress.update(task, advance=1)
async def scrape_all(
*,
progress: Progress,
session: ClientSession,
settings: Settings,
storage: ProxyStorage,
) -> None:
tasks = {
proto: progress.add_task(
f"[yellow]Scraper [red]:: [green]{proto.name}", total=len(sources)
)
for proto, sources in settings.sources.items()
}
timeout = ClientTimeout(total=settings.source_timeout)
coroutines = (
scrape_one(
progress=progress,
proto=proto,
session=session,
source=source,
storage=storage,
task=tasks[proto],
timeout=timeout,
)
for proto, sources in settings.sources.items()
for source in sources
)
await asyncio.gather(*coroutines)
proxy_scraper_checker/settings.py
+295
View File
@@ -0,0 +1,295 @@
from __future__ import annotations
import asyncio
import enum
import logging
import math
import sys
from json import JSONDecodeError
from pathlib import Path
from typing import (
TYPE_CHECKING,
Callable,
Dict,
FrozenSet,
Iterable,
Mapping,
Optional,
Tuple,
Union,
)
from urllib.parse import urlparse
import attrs
from aiohttp import ClientSession, ClientTimeout
from aiohttp_socks import ProxyType
from . import sort
from .null_context import NullContext
from .parsers import parse_ipv4
from .typing_compat import Any, Literal, Self
if TYPE_CHECKING:
from .proxy import Proxy
logger = logging.getLogger(__name__)
def _get_supported_max_connections() -> Optional[int]:
if sys.platform == "win32":
if isinstance(
asyncio.get_event_loop_policy(),
asyncio.WindowsSelectorEventLoopPolicy,
):
return 512
return None
import resource # noqa: PLC0415
soft_limit, hard_limit = resource.getrlimit(resource.RLIMIT_NOFILE)
logger.debug(
"max_connections: soft limit = %d, hard limit = %d, infinity = %d",
soft_limit,
hard_limit,
resource.RLIM_INFINITY,
)
if soft_limit != hard_limit:
try:
resource.setrlimit(resource.RLIMIT_NOFILE, (hard_limit, hard_limit))
except ValueError as e:
logger.warning("Failed setting max_connections: %s", e)
else:
soft_limit = hard_limit
if soft_limit == resource.RLIM_INFINITY:
return None
return soft_limit
def _get_max_connections(value: int, /) -> Optional[int]:
if value < 0:
msg = "max_connections must be non-negative"
raise ValueError(msg)
max_supported = _get_supported_max_connections()
if not value:
logger.info("Using %d as max_connections value", max_supported or 0)
return max_supported
if not max_supported or value <= max_supported:
return value
logger.warning(
"max_connections value is too high. "
"Your OS supports a maximum of %d. "
"The config value will be ignored and %d will be used.",
max_supported,
max_supported,
)
return max_supported
def _semaphore_converter(
value: int, /
) -> Union[asyncio.Semaphore, NullContext]:
v = _get_max_connections(value)
return asyncio.Semaphore(v) if v else NullContext()
def _timeout_converter(value: float, /) -> ClientTimeout:
return ClientTimeout(total=value, sock_connect=math.inf)
def _sources_converter(
value: Mapping[ProxyType, Optional[Iterable[str]]], /
) -> Dict[ProxyType, FrozenSet[str]]:
return {
proxy_type: frozenset(sources)
for proxy_type, sources in value.items()
if sources is not None
}
class CheckWebsiteType(enum.Enum):
UNKNOWN = enum.auto()
PLAIN_IP = enum.auto()
"""https://checkip.amazonaws.com"""
HTTPBIN_IP = enum.auto()
"""https://httpbin.org/ip"""
@property
def supports_geolocation(self) -> bool:
return self != CheckWebsiteType.UNKNOWN
@property
def supports_anonymity(self) -> bool:
return self != CheckWebsiteType.UNKNOWN
async def _get_check_website_type_and_real_ip(
*, check_website: str, session: ClientSession
) -> Union[
Tuple[Literal[CheckWebsiteType.UNKNOWN], None],
Tuple[Literal[CheckWebsiteType.PLAIN_IP, CheckWebsiteType.HTTPBIN_IP], str],
]:
try:
async with session.get(check_website, raise_for_status=True) as r:
await r.read()
except Exception:
logger.exception(
"Error when opening check_website without proxy, it will be "
"impossible to determine anonymity and geolocation of proxies"
)
return CheckWebsiteType.UNKNOWN, None
try:
content = await r.json(content_type=None)
except JSONDecodeError:
try:
return CheckWebsiteType.PLAIN_IP, parse_ipv4(await r.text())
except ValueError:
pass
else:
try:
return CheckWebsiteType.HTTPBIN_IP, parse_ipv4(content["origin"])
except (KeyError, TypeError, ValueError):
pass
logger.warning(
"Check_website is not httpbin and does not return plain ip, so it will"
" be impossible to determine the anonymity and geolocation of proxies"
)
return CheckWebsiteType.UNKNOWN, None
@attrs.define(
repr=False,
weakref_slot=False,
kw_only=True,
eq=False,
getstate_setstate=False,
match_args=False,
)
class Settings:
check_website: str = attrs.field(
validator=attrs.validators.instance_of(str)
)
check_website_type: CheckWebsiteType = attrs.field(
validator=attrs.validators.instance_of(CheckWebsiteType)
)
enable_geolocation: bool = attrs.field(
validator=attrs.validators.instance_of(bool)
)
output_json: bool = attrs.field(
validator=attrs.validators.instance_of(bool)
)
output_path: Path = attrs.field(converter=Path)
output_txt: bool = attrs.field(validator=attrs.validators.instance_of(bool))
real_ip: Optional[str] = attrs.field(
validator=attrs.validators.optional(attrs.validators.instance_of(str))
)
semaphore: Union[asyncio.Semaphore, NullContext] = attrs.field(
converter=_semaphore_converter
)
sort_by_speed: bool = attrs.field(
validator=attrs.validators.instance_of(bool)
)
source_timeout: float = attrs.field(validator=attrs.validators.gt(0))
sources: Dict[ProxyType, FrozenSet[str]] = attrs.field(
validator=attrs.validators.and_(
attrs.validators.instance_of(dict),
attrs.validators.min_len(1),
attrs.validators.deep_mapping(
attrs.validators.instance_of(ProxyType),
attrs.validators.and_(
attrs.validators.min_len(1),
attrs.validators.deep_iterable(
attrs.validators.and_(
attrs.validators.instance_of(str),
attrs.validators.min_len(1),
)
),
),
),
),
converter=_sources_converter,
)
timeout: ClientTimeout = attrs.field(converter=_timeout_converter)
@property
def sorting_key(
self,
) -> Union[Callable[[Proxy], float], Callable[[Proxy], Tuple[int, ...]]]:
return (
sort.timeout_sort_key
if self.sort_by_speed
else sort.natural_sort_key
)
def __attrs_post_init__(self) -> None:
if not self.output_json and not self.output_txt:
msg = "both json and txt outputs are disabled"
raise ValueError(msg)
if not self.output_json and self.enable_geolocation:
msg = "geolocation can not be enabled if json output is disabled"
raise ValueError(msg)
@check_website.validator
def _validate_check_website( # noqa: PLR6301
self,
attribute: attrs.Attribute[str], # noqa: ARG002
value: str,
/,
) -> None:
parsed_url = urlparse(value)
if not parsed_url.scheme or not parsed_url.netloc:
msg = f"invalid URL: {value}"
raise ValueError(msg)
if parsed_url.scheme == "http":
logger.warning(
"check_website uses the http protocol. "
"It is recommended to use https for correct checking."
)
@timeout.validator
def _validate_timeout(
self,
attribute: attrs.Attribute[str], # noqa: ARG002
value: float, # noqa: ARG002
/,
) -> None:
if self.timeout.total is None or self.timeout.total <= 0:
msg = "timeout must be positive"
raise ValueError(msg)
@classmethod
async def from_mapping(
cls, cfg: Mapping[str, Any], /, *, session: ClientSession
) -> Self:
check_website_type, real_ip = await _get_check_website_type_and_real_ip(
check_website=cfg["check_website"], session=session
)
return cls(
check_website=cfg["check_website"],
check_website_type=check_website_type,
enable_geolocation=cfg["enable_geolocation"]
and check_website_type.supports_geolocation,
output_json=cfg["output"]["json"],
output_path=cfg["output"]["path"],
output_txt=cfg["output"]["txt"],
real_ip=real_ip,
semaphore=cfg["max_connections"],
sort_by_speed=cfg["sort_by_speed"],
source_timeout=cfg["source_timeout"],
sources={
ProxyType.HTTP: (
cfg["http"]["sources"] if cfg["http"]["enabled"] else None
),
ProxyType.SOCKS4: (
cfg["socks4"]["sources"]
if cfg["socks4"]["enabled"]
else None
),
ProxyType.SOCKS5: (
cfg["socks5"]["sources"]
if cfg["socks5"]["enabled"]
else None
),
},
timeout=cfg["timeout"],
)
proxy_scraper_checker/sort.py
+13 -5
View File
@@ -2,12 +2,20 @@ from __future__ import annotations
from typing import Tuple
from aiohttp_socks import ProxyType
from .proxy import Proxy
PROTOCOL_ORDER = (ProxyType.HTTP, ProxyType.SOCKS4, ProxyType.SOCKS5)
def timeout_sort_key(proxy: Proxy) -> float:
def protocol_sort_key(proxy: Proxy, /) -> Tuple[int, ProxyType]:
return (PROTOCOL_ORDER.index(proxy.protocol), proxy.protocol)
def natural_sort_key(proxy: Proxy, /) -> Tuple[int, ...]:
return (proxy.protocol.value, *map(int, proxy.host.split(".")), proxy.port)
def timeout_sort_key(proxy: Proxy, /) -> float:
return proxy.timeout
def natural_sort_key(proxy: Proxy) -> Tuple[int, ...]:
return (*map(int, proxy.host.split(".")), proxy.port)
proxy_scraper_checker/storage.py
+54
View File
@@ -0,0 +1,54 @@
from __future__ import annotations
import itertools
from collections import Counter
from typing import TYPE_CHECKING, Callable, Dict, Iterator, List, Set, Tuple
from aiohttp_socks import ProxyType
from . import sort
from .proxy import Proxy
if TYPE_CHECKING:
from _typeshed import SupportsRichComparison
class ProxyStorage:
__slots__ = ("_proxies", "enabled_protocols")
def __init__(self) -> None:
self._proxies: Set[Proxy] = set()
self.enabled_protocols: Set[ProxyType] = set()
def add(self, proxy: Proxy, /) -> None:
self.enabled_protocols.add(proxy.protocol)
self._proxies.add(proxy)
def remove(self, proxy: Proxy, /) -> None:
self._proxies.remove(proxy)
def get_grouped(self) -> Dict[ProxyType, Tuple[Proxy, ...]]:
key = sort.protocol_sort_key
d: Dict[ProxyType, Tuple[Proxy, ...]] = {
proto: ()
for proto in sort.PROTOCOL_ORDER
if proto in self.enabled_protocols
}
for (_, proto), v in itertools.groupby(
sorted(self._proxies, key=key), key=key
):
d[proto] = tuple(v)
return d
def get_sorted(
self,
*,
key: Callable[[Proxy], SupportsRichComparison] = sort.protocol_sort_key,
) -> List[Proxy]:
return sorted(self._proxies, key=key)
def get_count(self) -> Dict[ProxyType, int]:
return dict(Counter(proxy.protocol for proxy in self._proxies))
def __iter__(self) -> Iterator[Proxy]:
return iter(self._proxies)
proxy_scraper_checker/typing_compat.py
+10
View File
@@ -0,0 +1,10 @@
from __future__ import annotations
import sys
if sys.version_info < (3, 11):
from typing_extensions import Any, Literal, Self
else:
from typing import Any, Literal, Self
__all__ = ("Any", "Literal", "Self")
proxy_scraper_checker/utils.py
+14
View File
@@ -0,0 +1,14 @@
from __future__ import annotations
from urllib.parse import urlparse
import charset_normalizer
def is_url(value: str, /) -> bool:
parsed_url = urlparse(value)
return bool(parsed_url.scheme and parsed_url.netloc)
def bytes_decode(value: bytes, /) -> str:
return str(charset_normalizer.from_bytes(value)[0])
proxy_scraper_checker/validators.py
-93
View File
@@ -1,93 +0,0 @@
from __future__ import annotations
import asyncio
import logging
import sys
from typing import Iterable, Optional
from urllib.parse import urlparse
from typing_extensions import Any
from .folder import Folder
logger = logging.getLogger(__name__)
def timeout(value: float) -> None:
if value <= 0:
msg = "Timeout must be positive"
raise ValueError(msg)
def source_timeout(value: float) -> None:
if value <= 0:
msg = "SourceTimeout must be positive"
raise ValueError(msg)
def max_connections(value: int) -> Optional[int]:
if value < 0:
msg = "MaxConnections must be non-negative"
raise ValueError(msg)
max_supported = _get_supported_max_connections()
if not value:
logger.info("Using %d as MaxConnections value", max_supported or 0)
return max_supported
if not max_supported or value <= max_supported:
return value
logger.warning(
"MaxConnections value is too high. "
"Your OS supports a maximum of %d. "
"The config value will be ignored and %d will be used.",
max_supported,
max_supported,
)
return max_supported
def _get_supported_max_connections() -> Optional[int]:
if sys.platform == "win32":
if isinstance(
asyncio.get_event_loop_policy(),
asyncio.WindowsSelectorEventLoopPolicy,
):
return 512
return None
import resource # noqa: PLC0415
soft_limit, hard_limit = resource.getrlimit(resource.RLIMIT_NOFILE)
logger.debug(
"MaxConnections soft limit = %d, hard limit = %d, infinity = %d",
soft_limit,
hard_limit,
resource.RLIM_INFINITY,
)
if soft_limit != hard_limit:
try:
resource.setrlimit(resource.RLIMIT_NOFILE, (hard_limit, hard_limit))
except ValueError as e:
logger.warning("Failed setting MaxConnections: %s", e)
else:
soft_limit = hard_limit
if soft_limit == resource.RLIM_INFINITY:
return None
return soft_limit
def check_website(value: str) -> None:
parsed_url = urlparse(value)
if not parsed_url.scheme or not parsed_url.netloc:
msg = f"invalid CheckWebsite URL: {value}"
raise ValueError(msg)
def folders(value: Iterable[Folder]) -> None:
if not any(folder for folder in value if folder.is_enabled):
msg = "all folders are disabled in the config"
raise ValueError(msg)
def sources(value: Any) -> None:
if not value:
msg = "proxy sources list is empty"
raise ValueError(msg)
pyproject.toml
+47 -1
View File
@@ -1,3 +1,45 @@
[tool.poetry]
name = "proxy-scraper-checker"
version = "0"
description = ""
license = "MIT"
authors = ["monosans <hsyqixco@protonmail.com>"]
repository = "https://github.com/monosans/proxy-scraper-checker"
classifiers = ["Private :: Do Not Upload"]
[tool.poetry.dependencies]
python = "^3.8"
aiodns = { version = ">=1.1.1,<4", optional = true }
aiofiles = ">=0.8"
aiohttp = "^3.8.6"
aiohttp-socks = ">=0.7,<0.9"
attrs = ">=22.2"
brotli = { version = "^1", markers = "platform_python_implementation == 'CPython'" }
brotlicffi = { version = "<2", markers = "platform_python_implementation != 'CPython'" }
certifi = "*"
charset-normalizer = ">=2,<4"
maxminddb = ">=1.3,<3"
platformdirs = "<5"
rich = ">=12.3,<14"
tomli = { version = "<3", python = "<3.11" }
typing-extensions = { version = "^4.4", python = "<3.11" }
uvloop = { version = ">=0.14,<0.20", optional = true, markers = "platform_python_implementation == 'CPython' and (sys_platform == 'darwin' or sys_platform == 'linux')" }
[tool.poetry.extras]
full = ["aiodns", "uvloop"]
[tool.poetry.group.dev.dependencies]
mypy = "1.8.0"
ruff = "0.1.14"
tomli = "2.0.1"
types-aiofiles = "23.2.0.20240106"
[tool.poetry.group.nuitka]
optional = true
[tool.poetry.group.nuitka.dependencies]
nuitka = "1.9.7"
[tool.mypy]
python_version = "3.8"
platform = "linux"
@@ -75,7 +117,7 @@ ignore-init-module-imports = true
line-length = 80
preview = true
select = ["ALL"]
target-version = "py37"
target-version = "py38"
[tool.ruff.flake8-self]
ignore-names = []
@@ -97,3 +139,7 @@ convention = "google"
[tool.ruff.pyupgrade]
keep-runtime-typing = true
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
requirements-termux.txt
-4
View File
@@ -1,4 +0,0 @@
aiohttp-socks>=0.7,<0.9
aiohttp>=3.8,<4
rich>=12.3,<14
typing-extensions>=4.4,<5
requirements.txt
-6
View File
@@ -1,6 +0,0 @@
aiodns>=3.0,<4
aiohttp-socks>=0.7,<0.9
aiohttp>=3.8,<4
rich>=12.3,<14
typing-extensions>=4.4,<5
uvloop>=0.16,<0.20; implementation_name == "cpython" and (sys_platform == "darwin" or sys_platform == "linux")
start-termux.sh
+3 -3
View File
@@ -1,4 +1,4 @@
#!/bin/sh
python3 -m pip install -U --no-cache-dir --disable-pip-version-check setuptools wheel &&
python3 -m pip install -U --no-cache-dir --disable-pip-version-check -r requirements-termux.txt &&
python3 -m proxy_scraper_checker
python3 -m venv --upgrade-deps .venv &&
.venv/bin/python3 -m pip install -U --disable-pip-version-check --editable . &&
.venv/bin/python3 -m proxy_scraper_checker
start.cmd
+3 -4
View File
@@ -1,4 +1,3 @@
py -m pip install -U --no-cache-dir --disable-pip-version-check pip setuptools wheel
py -m pip install -U --no-cache-dir --disable-pip-version-check -r requirements.txt
py -m proxy_scraper_checker
Pause
py -m venv --upgrade-deps .venv
.venv\Scripts\python.exe -m pip install -U --disable-pip-version-check --editable .[full]
.venv\Scripts\python.exe -m proxy_scraper_checker
start.sh
+3 -3
View File
@@ -1,4 +1,4 @@
#!/bin/sh
python3 -m pip install -U --no-cache-dir --disable-pip-version-check pip setuptools wheel &&
python3 -m pip install -U --no-cache-dir --disable-pip-version-check -r requirements.txt &&
python3 -m proxy_scraper_checker
python3 -m venv --upgrade-deps .venv &&
.venv/bin/python3 -m pip install -U --disable-pip-version-check --editable .[full] &&
.venv/bin/python3 -m proxy_scraper_checker