python - 如何在 python 3 中使用 urllib 请求解决 SSL 握手失败？

Question

代码应用

嗨，stackers，我正在设计一个网络抓取工具，它根据特定的基于单词的标准向用户发送通知。我已经成功地抓取了一个测试网站一段时间了，但是，当我昨天去运行我的代码时，我遇到了一个意外的错误，我一直没有成功解决。我仍然不确定为什么会发生此错误，因为我的代码中没有任何更改 - 这让我相信网站的某些内容可能已更改。在使用 urllib 的 Request 后尝试读取网站内容时出现错误。

代码

url = "http://www.ksl.com"

import urllib.request
from urllib.request import Request, urlopen
from bs4 import BeautifulSoup

import ssl
print(ssl.OPENSSL_VERSION)

req = Request(url, headers={'User-Agent': 'Mozilla/5.0'})
html = urlopen(req).read()
html = html.decode('utf-8')

错误信息

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 1254, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1106, in request
    self._send_request(method, url, body, headers)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1151, in _send_request
    self.endheaders(body)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1102, in endheaders
    self._send_output(message_body)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 934, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 877, in send
    self.connect()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1260, in connect
    server_hostname=server_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 377, in wrap_socket
    _context=self)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 752, in __init__
    self.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 988, in do_handshake
    self._sslobj.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 633, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:645)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "PATH TO SCRIPT", line 59, in <module>
    html = urlopen(req).read()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 163, in urlopen
    return opener.open(url, data, timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 472, in open
    response = meth(req, response)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 582, in http_response
    'http', request, response, code, msg, hdrs)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 504, in error
    result = self._call_chain(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 444, in _call_chain
    result = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 696, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 466, in open
    response = self._open(req, data)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 484, in _open
    '_open', req)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 444, in _call_chain
    result = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 1297, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 1256, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:645)>

我为修复错误所做的尝试

所以我安装了 MacPorts 以尝试更改 python 使用的 SSL 路径，这没有用，我不确定这是否是正确的解决方案路径。

我把这两行代码放进去看看python到底在做什么:

v = sys.version
print(v)

import ssl
print(ssl.OPENSSL_VERSION)

这两行给我以下输出:

3.5.2 (v3.5.2:4def2a2901a5, Jun 26 2016, 10:47:25) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
OpenSSL 0.9.8zh 14 Jan 2016

我认为问题可能是 python 使用的 mac 默认版本的 OpenSSL 无法完成握手，但是我还有很多东西需要学习，所以我不完全确定。

请帮助我堆垛机，你是我唯一的希望!

Answer 1

是的，您的 OpenSSL 版本太旧了。下面是一个使用两个版本的 macOS openssl 的示例:

$ openssl s_client -connect ksl.com:443
CONNECTED(00000003)
2908:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:/BuildRoot/Library/Caches/com.apple.xbs/Sources/OpenSSL098/OpenSSL098-64.50.6/src/ssl/s23_clnt.c:593:

$ /usr/local/Cellar/openssl/1.0.2l/bin/openssl s_client -connect ksl.com:443
[...]
Verify return code: 0 (ok)

如果你安装了更新版本的 python，例如使用 brew install python3，你应该有更好的运气:

Python 3.6.2 (default, Jul 17 2017, 16:44:45)
[GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ssl.OPENSSL_VERSION
'OpenSSL 1.0.2l  25 May 2017'

有了这个版本，您的脚本就可以正常工作了。