haeaeae/proxy-scraper-checker
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
323 Commits 1 Branch 0 Tags
4fbb2fbe28d61fc4fd679a45346dc67887bfd2da
Commit Graph
81 Commits
Author SHA1 Message Date
monosans 328834eb03 Improve permissions check 2024-02-01 14:52:29 +03:00
monosansandGitHub 5005e0dc6b Add Docker support 2024-02-01 10:24:26 +03:00
monosans 624ffa2698 Add noqa 2024-01-29 14:04:37 +03:00
monosans 326f143d26 Use Chrome User-Agent 2024-01-29 13:52:46 +03:00
monosans 2d30886e74 Iterate over ProxyStorage instead of ._proxies 2024-01-29 00:33:55 +03:00
monosans 51c847811c Remove get_sorted method from ProxyStorage 2024-01-29 00:32:31 +03:00
monosans e201011ed3 Refactor typing_compat 2024-01-25 18:12:36 +03:00
monosans 330b607c59 Use sys.implementation.name instead of platform.python_implementation() 2024-01-25 15:43:12 +03:00
monosans cabc3a0061 Refactor progress bar 2024-01-25 11:49:22 +03:00
monosans 890c000139 Don't shuffle proxies before checking 2024-01-25 10:30:40 +03:00
monosans 3059940212 Monkeypatch os.link to make aiofiles work on Termux 2024-01-25 09:05:53 +03:00
monosans 82055d3417 Remove unused variable 2024-01-23 09:49:17 +03:00
monosans dca437feae Add raise_for_status=True when scraping sources 2024-01-23 09:47:43 +03:00
monosans 73303cab1b Improve response decoding performance 2024-01-23 09:37:29 +03:00
monosans 31a50964eb Fix duplicates 2024-01-23 08:09:08 +03:00
monosans a42448808e Refactor get_grouped 2024-01-21 23:40:05 +03:00
monosans 0984fd1a82 Fix an edge case 2024-01-21 23:37:46 +03:00
monosans f4af55d37a Fix encoding issues 2024-01-21 21:46:01 +03:00
monosans 24f5eb5f2f Less strict parser 2024-01-21 21:24:09 +03:00
monosansandGitHub e4047e16ae Huge update 
- Drop support for Python 3.7.
- Use offline geolocation database instead of ip-api.com.
- Add support for proxy authentication.
- Add pre-compiled binaries.
- Refactor the code by a lot.
- Simplify config.
- Add automatic text encoding detection.
- Use TOML instead of INI for configuration.
- Add support for using httpbin-compatible services and services which return plain IP address for getting proxy's exit node.
- Add support for reading proxy lists from local files.
- Add support for saving to json.
- Remove support for saving geolocation to txt, use json instead.
- Improve parser.
- Improve config validation.
- Use poetry instead of requirements.txt.
- Use venv in start scripts.
 2024-01-21 17:45:01 +00:00
monosans eeca39b8f5 Update User-Agent 2023-12-26 10:09:22 +03:00
monosans 4211933cff Improve traceback 2023-12-25 15:11:23 +03:00
monosans cbf10a37d3 Don't set geolocation if geolocation folders are disabled 2023-12-09 15:16:44 +03:00
monosansGitHubgithub-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 		d45db44c6b Update .pre-commit-config.yaml (#111) 
* Update .pre-commit-config.yaml

* Ignore PLR0917 in ruff

* Reformat

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-12-07 08:08:12 +00:00
monosans fb59bdaf70 Update User-Agent 2023-11-24 19:54:03 +03:00
monosansGitHubgithub-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 		57696e62e3 Update .pre-commit-config.yaml (#101) 
* Update .pre-commit-config.yaml

* Fix

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-11-04 09:29:21 +00:00
monosans 813d320c64 Use logging.Formatter.default_time_format 2023-10-31 19:00:46 +03:00
monosans 496cef3ae9 Update User-Agent 2023-10-28 14:25:31 +03:00
monosans 14f19c6414 Use typing-extensions 2023-10-25 20:27:48 +03:00
monosans de87fbafcd Bump uvloop to v0.19 2023-10-23 01:12:54 +03:00
monosans 32889f066f Fix uvloop deprecation warning 2023-10-14 14:24:04 +03:00
monosans 6c86f297fd Add type: ignore 2023-10-14 11:53:45 +03:00
monosans 8eb9eee886 Update User-Agent 2023-09-29 17:48:32 +03:00
monosans 6f441a7499 Update User-Agent 2023-09-02 12:03:24 +03:00
monosans 5e1a81a369 Minor refactor 2023-08-22 12:23:45 +03:00
monosans 0ea337663c Add type hint 2023-08-21 14:13:16 +03:00
monosans 5170e6f37f Update User-Agent 2023-08-12 22:30:25 +03:00
monosansGitHubgithub-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 		f71826dbc3 Update .pre-commit-config.yaml (#75) 
* Update .pre-commit-config.yaml

* Fix ruff error

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-08-09 08:07:25 +00:00
monosans ad02c79394 Update User-Agent header 2023-07-12 00:06:42 +03:00
monosansGitHubgithub-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 		1ba3bff5bc Update .pre-commit-config.yaml (#69) 
* Update .pre-commit-config.yaml

* Reformat

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-07-11 07:54:08 +00:00
monosans d32f7f8707 Change max line length from 88 to 80 2023-06-27 12:04:44 +03:00
monosans 3fc02c8362 Update User-Agent 2023-06-10 11:41:32 +03:00
monosans cdf2d05ece Update User-Agent 2023-05-31 14:11:51 +03:00
monosans 917bcb4ef5 Simplify proxy connector code 2023-04-17 16:48:11 +03:00
monosans ad5513d1a3 Don't check response content-type 2023-04-17 16:40:20 +03:00
monosans 4fe4e3b0b6 Change final output 2023-04-16 21:49:53 +03:00
monosans a6411bc501 Minor optimizations 2023-04-16 21:11:46 +03:00
monosans 6d5296534f Update User-Agent 2023-04-15 14:21:12 +03:00
monosans 2bd38fdcd1 Update User-Agent 2023-04-12 15:40:01 +03:00
monosans 4d76fe1870 Change timeout log level from error to warning 2023-03-19 00:30:03 +03:00