python - BeautifulSoup 实例化超时？

Question

我只是用 BeautifulSoup 进行一些网页抓取，但遇到了一个奇怪的错误。代码:

print "Running urllib2"
g = urllib2.urlopen(link + "about", timeout=5)
print "Finished urllib2"
about_soup = BeautifulSoup(g, 'lxml')

这是输出:

Running urllib2
Finished urllib2

Error
    Traceback (most recent call last):
      File "/Users/pspieker/Documents/projects/ThePyStrikesBack/tests/TestSpringerOpenScraper.py", line 10, in test_strip_chars
        for row in self.instance.get_entries():
      File "/Users/pspieker/Documents/projects/ThePyStrikesBack/src/JournalScrapers.py", line 304, in get_entries
        about_soup = BeautifulSoup(g, 'lxml')
      File "/Users/pspieker/.virtualenvs/thepystrikesback/lib/python2.7/site-packages/bs4/__init__.py", line 175, in __init__
        markup = markup.read()
      File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 355, in read
        data = self._sock.recv(rbufsize)
      File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 588, in read
        return self._read_chunked(amt)
      File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 648, in _read_chunked
        value.append(self._safe_read(amt))
      File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 703, in _safe_read
        chunk = self.fp.read(min(amt, MAXAMOUNT))
      File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 384, in read
        data = self._sock.recv(left)
    timeout: timed out

我知道 urllib2.urlopen 可能会导致问题，但异常发生在实例化 BeautifulSoup 的行中。我进行了一些谷歌搜索，但找不到任何有关 BeautfiulSoup 超时问题的信息。

对正在发生的事情有什么想法吗？

Answer 1

这是导致超时的 urllib2 部分。

您看到它在 BeautifulSoup 实例化行上失败的原因是 g, the file-like object ，正在由BeautifulSoup内部读取。这是堆栈跟踪的一部分，证明:

File "/Users/pspieker/.virtualenvs/thepystrikesback/lib/python2.7/site-packages/bs4/__init__.py", line 175, in __init__
    markup = markup.read()