haeaeae/proxy-scraper-checker
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
297 Commits 1 Branch 0 Tags
dca437feae0fea2ac3ade3f2b06520cf24c0bccb
Commit Graph
69 Commits
Author SHA1 Message Date
monosans dca437feae Add raise_for_status=True when scraping sources 2024-01-23 09:47:43 +03:00
monosans 73303cab1b Improve response decoding performance 2024-01-23 09:37:29 +03:00
monosans 31a50964eb Fix duplicates 2024-01-23 08:09:08 +03:00
monosans a42448808e Refactor get_grouped 2024-01-21 23:40:05 +03:00
monosans 0984fd1a82 Fix an edge case 2024-01-21 23:37:46 +03:00
monosans f4af55d37a Fix encoding issues 2024-01-21 21:46:01 +03:00
monosans 24f5eb5f2f Less strict parser 2024-01-21 21:24:09 +03:00
monosansandGitHub e4047e16ae Huge update 
- Drop support for Python 3.7.
- Use offline geolocation database instead of ip-api.com.
- Add support for proxy authentication.
- Add pre-compiled binaries.
- Refactor the code by a lot.
- Simplify config.
- Add automatic text encoding detection.
- Use TOML instead of INI for configuration.
- Add support for using httpbin-compatible services and services which return plain IP address for getting proxy's exit node.
- Add support for reading proxy lists from local files.
- Add support for saving to json.
- Remove support for saving geolocation to txt, use json instead.
- Improve parser.
- Improve config validation.
- Use poetry instead of requirements.txt.
- Use venv in start scripts.
 2024-01-21 17:45:01 +00:00
monosans eeca39b8f5 Update User-Agent 2023-12-26 10:09:22 +03:00
monosans 4211933cff Improve traceback 2023-12-25 15:11:23 +03:00
monosans cbf10a37d3 Don't set geolocation if geolocation folders are disabled 2023-12-09 15:16:44 +03:00
monosansGitHubgithub-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 		d45db44c6b Update .pre-commit-config.yaml (#111) 
* Update .pre-commit-config.yaml

* Ignore PLR0917 in ruff

* Reformat

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-12-07 08:08:12 +00:00
monosans fb59bdaf70 Update User-Agent 2023-11-24 19:54:03 +03:00
monosansGitHubgithub-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 		57696e62e3 Update .pre-commit-config.yaml (#101) 
* Update .pre-commit-config.yaml

* Fix

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-11-04 09:29:21 +00:00
monosans 813d320c64 Use logging.Formatter.default_time_format 2023-10-31 19:00:46 +03:00
monosans 496cef3ae9 Update User-Agent 2023-10-28 14:25:31 +03:00
monosans 14f19c6414 Use typing-extensions 2023-10-25 20:27:48 +03:00
monosans de87fbafcd Bump uvloop to v0.19 2023-10-23 01:12:54 +03:00
monosans 32889f066f Fix uvloop deprecation warning 2023-10-14 14:24:04 +03:00
monosans 6c86f297fd Add type: ignore 2023-10-14 11:53:45 +03:00
monosans 8eb9eee886 Update User-Agent 2023-09-29 17:48:32 +03:00
monosans 6f441a7499 Update User-Agent 2023-09-02 12:03:24 +03:00
monosans 5e1a81a369 Minor refactor 2023-08-22 12:23:45 +03:00
monosans 0ea337663c Add type hint 2023-08-21 14:13:16 +03:00
monosans 5170e6f37f Update User-Agent 2023-08-12 22:30:25 +03:00
monosansGitHubgithub-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 		f71826dbc3 Update .pre-commit-config.yaml (#75) 
* Update .pre-commit-config.yaml

* Fix ruff error

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-08-09 08:07:25 +00:00
monosans ad02c79394 Update User-Agent header 2023-07-12 00:06:42 +03:00
monosansGitHubgithub-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 		1ba3bff5bc Update .pre-commit-config.yaml (#69) 
* Update .pre-commit-config.yaml

* Reformat

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-07-11 07:54:08 +00:00
monosans d32f7f8707 Change max line length from 88 to 80 2023-06-27 12:04:44 +03:00
monosans 3fc02c8362 Update User-Agent 2023-06-10 11:41:32 +03:00
monosans cdf2d05ece Update User-Agent 2023-05-31 14:11:51 +03:00
monosans 917bcb4ef5 Simplify proxy connector code 2023-04-17 16:48:11 +03:00
monosans ad5513d1a3 Don't check response content-type 2023-04-17 16:40:20 +03:00
monosans 4fe4e3b0b6 Change final output 2023-04-16 21:49:53 +03:00
monosans a6411bc501 Minor optimizations 2023-04-16 21:11:46 +03:00
monosans 6d5296534f Update User-Agent 2023-04-15 14:21:12 +03:00
monosans 2bd38fdcd1 Update User-Agent 2023-04-12 15:40:01 +03:00
monosans 4d76fe1870 Change timeout log level from error to warning 2023-03-19 00:30:03 +03:00
monosans 5b4c829cc0 simplify error message 2023-03-18 10:47:54 +03:00
monosans 6c840e9acd Fix MaxConnections not working if limit is infinite 2023-03-17 23:42:22 +03:00
pre-commit-ci[bot]GitHubpre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 		4553f88474 [pre-commit.ci] pre-commit autoupdate (#43) 
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2023-02-28 06:48:56 +00:00
monosans 8d16ec6fff Add pre-commit 2023-02-25 19:31:34 +03:00
monosans 8f9569edd9 Fix errors on Python 3.7/3.8 
Fixes #40
 2023-02-21 21:22:47 +03:00
monosans 42ce1237a1 Reformat docstring 2023-02-19 22:33:06 +03:00
monosans 082e12b764 Many changes (see full commit message) 
Add ability to set MaxConnections to 0, at which this parameter will be set to the maximum available value for the OS
Add validation for Timeout and SourceTimeout options
Improve existing validators
Improve error and warning output
Misc refactoring
 2023-02-19 22:18:30 +03:00
monosans 635e993d56 Minor optimization of Folder 2023-02-19 00:07:19 +03:00
monosans 490efee372 Add Debug option 2023-02-06 20:59:03 +03:00
monosans 9eb893bb6c Reformat code 2023-02-01 18:46:40 +03:00
monosans d21e32e4fc Add CheckWebsite setting 
Fixes #16, fixes #39
 2023-01-15 13:05:16 +03:00
monosans 7298a5594f Refactor 2023-01-15 00:22:17 +03:00