python-2.7 - 使用网页回放录制 HTTPS 网页

Question

我正在尝试使用 web-page-replay 记录网站以进行实验.记录 HTTP 请求工作正常，但当我尝试记录 HTTPS 请求时，我收到 SSLError。

在 OSX 上，我使用这个命令来记录网页

sudo ./replay.py --record archive.wpr

我已将系统 HTTP 代理设置为 localhost:80，将 HTTPS 代理设置为 localhost:443

追溯-

2015-02-24 03:57:26,227 ERROR Client failed to make request
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 63226)
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 599, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 655, in __init__
    self.handle()
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "/Users/akshay/Projects/web-page-replay/httpproxy.py", line 162, in handle_one_request
    self.raw_requestline = self.rfile.readline(65537)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 476, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 714, in recv
    return self.read(buflen)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 608, in read
    v = self._sslobj.read(len or 1024)
SSLError: [SSL: HTTPS_PROXY_REQUEST] https proxy request (_ssl.c:1750)
----------------------------------------

更新 - 我尝试使用 telemetry 记录页面集并在记录 https 页面时出现 SSL 握手错误。

2015-03-01 11:12:04,929 WARNING HTTP server started on 127.0.0.1:50781
2015-03-01 11:12:04,930 WARNING HTTPS server started on 127.0.0.1:50782
2015-03-01 11:12:04,981 ERROR Client failed to make request
2015-03-01 11:12:04,981 ERROR Client failed to make request
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 599, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 655, in __init__
    self.handle()
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "/Users/akshay/Downloads/telemetry/src/third_party/webpagereplay/httpproxy.py", line 162, in handle_one_request
    self.raw_requestline = self.rfile.readline(65537)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 476, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 714, in recv
    return self.read(buflen)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 608, in read
    v = self._sslobj.read(len or 1024)
SSLError: [SSL: SSL_HANDSHAKE_FAILURE] ssl handshake failure (_ssl.c:1750)
2015-03-01 11:12:05,782 ERROR Client failed to make request
2015-03-01 11:12:07,611 WARNING Retrying fetch GET https://www.google.com/ [('accept-encoding', 'gzip, deflate, '), ('host', 'www.google.com')]: hostname '74.125.236.209' doesn't match 'www.google.com'
2015-03-01 11:12:08,844 WARNING Retrying fetch GET https://www.google.com/ [('accept-encoding', 'gzip, deflate, '), ('host', 'www.google.com')]: hostname '74.125.236.209' doesn't match 'www.google.com'
2015-03-01 11:12:09,539 WARNING Retrying fetch GET https://www.google.com/ [('accept-encoding', 'gzip, deflate, '), ('host', 'www.google.com')]: hostname '74.125.236.209' doesn't match 'www.google.com'
2015-03-01 11:12:10,146 CRITICAL Could not fetch GET https://www.google.com/ [('accept-encoding', 'gzip, deflate, '), ('host', 'www.google.com')]: hostname '74.125.236.209' doesn't match 'www.google.com'
2015-03-01 11:12:11,386 WARNING Retrying fetch GET https://www.google.com/favicon.ico [('accept-encoding', 'gzip, deflate, '), ('host', 'www.google.com')]: hostname '74.125.236.209' doesn't match 'www.google.com'
2015-03-01 11:12:13,428 WARNING Retrying fetch GET https://www.google.com/favicon.ico [('accept-encoding', 'gzip, deflate, '), ('host', 'www.google.com')]: hostname '74.125.236.209' doesn't match 'www.google.com'
2015-03-01 11:12:14,417 WARNING Retrying fetch GET https://www.google.com/favicon.ico [('accept-encoding', 'gzip, deflate, '), ('host', 'www.google.com')]: hostname '74.125.236.209' doesn't match 'www.google.com'
2015-03-01 11:12:18,349 CRITICAL Could not fetch GET https://www.google.com/favicon.ico [('accept-encoding', 'gzip, deflate, '), ('host', 'www.google.com')]: hostname '74.125.236.209' doesn't match 'www.google.com'
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 50785)
----------------------------------------

Answer 1

正如我所见，Python 的 WebPageReplay 实现现已弃用，因此您可以使用 Go 实现 https://github.com/catapult-project/catapult/blob/master/web_page_replay_go/README.md

开发人员建议使用以下 Chrome(和所有类似 Chromium 的浏览器)命令行参数来绕过浏览器的 TLS 检查:

--ignore-certificate-errors-spki-list=PhrPvGIaAMmd29hj8BCZOq096yj7uMpRNHpn5PDxI6I=

哪里PhrPvGIaAMmd29hj8BCZOq096yj7uMpRNHpn5PDxI6I=它是随 WebPageReplay 一起提供的证书主题，默认情况下用于记录在 wpr 存档中的所有域。

对于不支持的其他浏览器--ignore-certificate-errors-spki-list您可以在 wpr 存档中通过域名即时或预生成生成证书。