002-V2rayPool

Author: xhunmon

002-V2rayPool/.gitignore

@@ -0,0 +1,129 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/

002-V2rayPool/README.md

@@ -0,0 +1,57 @@
+# v2ray节点代理池
+学习python爬虫过程中，我们需要一些代理。而本项目则是通过收集网上公开的节点，来实现自己的代理请求的过程！
+
+## v2ray是什么？如何使用？
+- [v2ray官方指导](https://www.v2ray.com/index.html)
+  
+- [v2ray wiki](https://zh.wikipedia.org/wiki/V2Ray)
+
+## v2ray客户端推荐
+- [window：v2rayN](https://github.com/2dust/v2rayN/releases)
+- [macOS：V2rayU](https://github.com/yanue/V2rayU/releases)
+- [android：v2rayNG](https://github.com/2dust/v2rayNG/releases)
+- ios推荐Shadowrocket（俗称小火箭，注意不是shadowrocket VPN），但需要付费的，可以通过找"美区apple id共享2021小火箭"，找共享的账号下载软件，但是会有风险，请一定要注意。
+
+## v2ray服务端推荐
+有些应用场景会需要到用到专用的ip代理，如Tik Tok、亚马逊和Facebook等。这是，我们通过购买国外的服务器或者vps来搭建代理服务器，从而实现专有ip代理。
+
+推荐使用[x-ui](https://github.com/vaxilu/x-ui) 进行非常简单的"一键式"搭建开源框架。 
+
+#本项目主要知识点
+学习本项目需要先了解代理原理，以及v2ray实现的原理。
+
+## 实现思路
+![实现思路图](./doc/v2ray.jpg)
+
+## v2ray内核使用
+我们找的是v2ray节点，所以这些协议只能运行在v2ray特有程序中。因此，我们要找[v2ray内核](https://github.com/v2ray/v2ray-core/releases) 。 这里就以macOS系统举例说明：
+1. 下载[v2ray-core-v4.31.0](https://github.com/v2fly/v2ray-core/releases/download/v4.31.0/v2ray-macos-64.zip)
+2. 配置解压目录的路径：
+```python
+Config.set_v2ray_core_path('xxx/Downloads/v2ray-macos-64')
+```
+3. 查看是否能正常启动：
+```python
+client.Creator().v2ray_start('ss://[email protected]:43893#github.com/freefq%20-%20%E7%BD%97%E9%A9%AC%E5%B0%BC%E4%BA%9A%20%2041')
+```
+
+## 2021-09-23检测可用测试节点：
+```shell
+#,2.59.214.204,俄罗斯莫斯科
+ss://[email protected]:33992#github.com/freefq%20-%20%E5%8D%A2%E6%A3%AE%E5%A0%A1%20%2051,185.153.151.86,卢森堡
+trojan://[email protected]:8443#github.com/freefq%20-%20%E4%B9%8C%E5%85%8B%E5%85%B0%20%2052
+# </p>,8.37.43.232,日本东京 Level3
+vmess://eyJ2IjogIjIiLCAicHMiOiAiZ2l0aHViLmNvbS9mcmVlZnEgLSBcdTY1ZTVcdTY3MmNcdTRlMWNcdTRlYWNBbWF6b25cdTY1NzBcdTYzNmVcdTRlMmRcdTVmYzMgNTQiLCAiYWRkIjogImFmMDEudXdvcmsubW9iaSIsICJwb3J0IjogIjgwIiwgImlkIjogIjAwMDAwMDAwLTAwMDAtMDAwMC0wMDAwLTAwMDAwMDAwMDAwMSIsICJhaWQiOiAiNjQiLCAibmV0IjogInRjcCIsICJ0eXBlIjogIm5vbmUiLCAiaG9zdCI6ICIiLCAicGF0aCI6ICIiLCAidGxzIjogIm5vbmUifQ==
+#,132.226.169.93,荷兰阿姆斯特丹
+vmess://ew0KICAidiI6ICIyIiwNCiAgInBzIjogIkBTU1JTVUItVjI3LeS7mOi0ueaOqOiNkDpzdW8ueXQvc3Nyc3ViIiwNCiAgImFkZCI6ICIxMzIuMjI2LjE2OS45MyIsDQogICJwb3J0IjogIjMzMjU0IiwNCiAgImlkIjogIjA0NTMxNTRiLWRiNTEtNGE2Ni04ZjdjLTA4ODc4NzhjYjlhMyIsDQogICJhaWQiOiAiMCIsDQogICJzY3kiOiAiYXV0byIsDQogICJuZXQiOiAidGNwIiwNCiAgInR5cGUiOiAibm9uZSIsDQogICJob3N0IjogIjEzMi4yMjYuMTY5LjkzIiwNCiAgInBhdGgiOiAiIiwNCiAgInRscyI6ICIiLA0KICAic25pIjogIiINCn0=
+#,185.22.153.187,俄罗斯莫斯科
+vmess://ew0KICAidiI6ICIyIiwNCiAgInBzIjogImh0dHBzOi8vZ2l0LmlvL3Y5OTk5IOiOq+aWr+enkWciLA0KICAiYWRkIjogIjE4NS4yMi4xNTMuMTg3IiwNCiAgInBvcnQiOiAiNDM1NDYiLA0KICAiaWQiOiAiMjViOGI4YTYtMTdiOC0xMWVjLWJiNTAtMGFiM2VhMTM0ZDA2IiwNCiAgImFpZCI6ICIwIiwNCiAgIm5ldCI6ICJ0Y3AiLA0KICAidHlwZSI6ICJub25lIiwNCiAgImhvc3QiOiAiIiwNCiAgInBhdGgiOiAiIiwNCiAgInRscyI6ICIiLA0KICAic25pIjogIiINCn0=
+```
+
+> 如上面不可用，更多请留意：[db/_db-checked.txt](./db/_db-checked.txt)
+
+注意：本程序虽可跨平台，但因博主能力有限，无法在更多系统上去尝试和改进，望谅解！
+
+-------
+
+如有可用节点增加，请推荐给博主吧：[email protected]

002-V2rayPool/base/net_proxy.py

@@ -0,0 +1,71 @@
+#!/usr/bin/env python
+# -*- encoding: utf-8 -*-
+"""
+@Description: 网络代理请求的基类
+@Date       :2021/09/15
+@Author     :xhunmon
+@Mail       :[email protected]
+"""
+
+import requests
+from my_fake_useragent import UserAgent
+
+
+class Net(object):
+    """
+    基类，封装常用接口
+    """
+    TIMEOUT = 8
+
+    def __init__(self, timeout=8):
+        Net.TIMEOUT = timeout
+        self._ua = UserAgent()
+        self._agent = self._ua.random()  # 随机生成的agent
+        self.USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:65.0) Gecko/20100101 Firefox/65.0"
+        self._headers = {"user-agent": self.USER_AGENT, 'Connection': 'close'}
+        # determine
+        self._proxy = '127.0.0.1:1080'
+        self._proxies_en = {
+            'http': 'socks5h://' + self._proxy,
+            'https': 'socks5h://' + self._proxy,
+        }
+        self._proxies_zh = {
+            'http': 'socks5://' + self._proxy,
+            'https': 'socks5://' + self._proxy,
+        }
+
+    def get_header(self, headers, key):
+        key_lower = key.lower()
+        headers_lower = {k.lower(): v for k, v in headers.items()}
+        if (key_lower in headers_lower):
+            return headers_lower[key_lower]
+        else:
+            return ''
+
+    def get_urls(self) -> []:
+        """需子类实现"""
+        pass
+
+    def request(self, url, allow_redirects=False, verify=False, timeout=TIMEOUT):
+        """普通请求"""
+        return self.__request(url, allow_redirects=allow_redirects, verify=verify, timeout=timeout)
+
+    def request_en(self, url, allow_redirects=False, verify=False, timeout=TIMEOUT):
+        """国外网站请求，需要开代理"""
+        return self.__request(url, allow_redirects=allow_redirects, verify=verify, proxies=self._proxies_en,
+                              timeout=timeout)
+
+    def request_zh(self, url, allow_redirects=False, verify=False, timeout=TIMEOUT):
+        """国内网站请求，需要开代理"""
+        return self.__request(url, allow_redirects=allow_redirects, verify=verify, proxies=self._proxies_zh,
+                              timeout=timeout)
+
+    def __request(self, url, allow_redirects=False, verify=False, proxies=None, timeout=TIMEOUT):
+        """最终的请求实现"""
+        requests.packages.urllib3.disable_warnings()
+        if proxies:
+            return requests.get(url=url, headers=self._headers, allow_redirects=allow_redirects, verify=verify,
+                                proxies=proxies, timeout=timeout)
+        else:
+            return requests.get(url=url, headers=self._headers, allow_redirects=allow_redirects, verify=verify,
+                                timeout=timeout)