python - HTTPS 中的 Flask 代理服务器错误 - 适用于 HTTP 但不适用于 HTTPS

Question

我们已经在 Flask 上设置了一个基本的代理服务器。它在 HTTP 中运行良好，但是当在 HTTPS 中发出请求时，它会抛出一些难以调试的问题。

在 HTTPS 中，请求似乎没有被 flask 正确解析。

下面是更详细的描述

服务器代码

from flask import Flask, request
import requests
from requests import exceptions


application = Flask(__name__)

@application.route("/",methods=["GET","POST","DELETE","PATCH","PUT"])
def route():
    try:
        response = requests.request(method=request.method, url=request.url, headers=request.headers,timeout=60)
        print(response.status_code, response.text)
        return (response.content, response.status_code, response.headers.items())
    except exceptions.ProxyError:
        return (b"Error", 408, request.headers.items())

if __name__ == "__main__":
    application.run(host="0.0.0.0", port=39100, debug=True, use_reloader=False,ssl_context=("certs/cert.pem","certs/key.pem"))

请求示例代码

import requests

proxies = {"http":"https://127.0.0.1:39100", "https":"https://127.0.0.1:39100"}

# does not work if the request is http://ipv4.icanhazip.com/
# works if the request is http://ipv4.icanhazip.com/
testing = requests.get(url="https://ipv4.icanhazip.com/",proxies=proxies,verify=False) 
print(testing)
print(testing.content)

错误

/Users/a/PycharmProjects/invisiProxy/venv/bin/python /Users/a/PycharmProjects/invisiProxy/sudoRequest.py
Traceback (most recent call last):
  File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 700, in urlopen
    self._prepare_proxy(conn)
  File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 994, in _prepare_proxy
    conn.connect()
  File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/urllib3/connection.py", line 369, in connect
    self._tunnel()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 924, in _tunnel
    raise OSError(f"Tunnel connection failed: {code} {message.strip()}")
OSError: Tunnel connection failed: 404 NOT FOUND

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='ipv4.icanhazip.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 404 NOT FOUND')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/a/PycharmProjects/invisiProxy/sudoRequest.py", line 6, in <module>
    testing = requests.get(url="https://ipv4.icanhazip.com/",proxies=proxies,verify=False)
  File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/requests/adapters.py", line 559, in send
    raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPSConnectionPool(host='ipv4.icanhazip.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 404 NOT FOUND')))

Process finished with exit code 1

Answer 1

对 HTTPS 的请求 使用代理 会导致客户端通过CONNECT 方法使用HTTP 隧道。这对于客户端有直接的 TCP channel 到目标是必要的，以便在没有任何 MITM 代理的情况下建立端到端加密(参见 urllib3 对这种情况的描述 -- https://urllib3.readthedocs.io/en/stable/advanced-usage.html#http-and-https-proxies)。

您的 Flask 服务器不支持 CONNECT 方法，因此它会根据请求返回 404。可以在提供的 MVP 服务器上观察到该行为:

# client request http://ipv4.icanhazip.com/
127.0.0.1 - - [24/Jun/2022 12:08:12] "GET http://ipv4.icanhazip.com/ HTTP/1.1" 200 -

# client request https://ipv4.icanhazip.com/
127.0.0.1 - - [24/Jun/2022 12:03:48] "CONNECT ipv4.icanhazip.com:443 HTTP/1.0" 404

从技术上讲，可以执行预期的行为......

# openssl s_client --connect 127.0.0.1:39100
CONNECTED(00000003)
---
Certificate chain
 0 s:C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN = xxxx
   i:C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN = xxxx
---
read R BLOCK
GET https://ipv4.icanhazip.com/ HTTP/1.1
Host: ipv4.icanhazip.com

HTTP/1.1 200 OK
Server: Werkzeug/2.1.2 Python/3.7.3
...

IP_REDACTED
read:errno=0

... 这种情况在 urllib3 级别(由 requests 用于实际连接)进行了记录和“可调整”，use_forwarding_for_https 为 https://urllib3.readthedocs.io/en/stable/reference/urllib3.poolmanager.html#urllib3.ProxyManager

但是

• 一般不推荐这种用法，因为它违反了 TLS 本身的设计(客户端和资源之间的端到端加密)。

• 在当前的 requests 实现中，似乎没有办法通过标准 API 从开发人员那里传递必需的标志。 get_connection 函数不允许将任何额外的 proxy_kwargs 传递给 urllib3.ProxyManager

https://github.com/psf/requests/blob/da9996fe4dc63356e9467d0a5e10df3d89a8528e/requests/adapters.py#L352

    def get_connection(self, url, proxies=None):
        ...
        if proxies:
            ...
            proxy_manager = self.proxy_manager_for(proxy)
            conn = proxy_manager.connection_from_url(url)


https://github.com/psf/requests/blob/da9996fe4dc63356e9467d0a5e10df3d89a8528e/requests/adapters.py#L201
    def proxy_manager_for(self, proxy, **proxy_kwargs):
        ...
        manager = self.proxy_manager[proxy] = proxy_from_url(
                ...
                **proxy_kwargs,
            )

我建议使用完整的 http 代理而不是 flask 服务器。