python - BeautifulSoup HTMLParseError。这有什么问题吗？

Question

这是我的代码:

from bs4 import BeautifulSoup as BS
import urllib2
url = "http://services.runescape.com/m=news/recruit-a-friend-for-free-membership-and-xp"
res = urllib2.urlopen(url)
soup = BS(res.read())
other_content = soup.find_all('div',{'class':'Content'})[0]
print other_content

但是出现错误:

/Library/Python/2.7/site-packages/bs4/builder/_htmlparser.py:149: RuntimeWarning: Python's built-in HTMLParser cannot parse the given document. This is not a bug in Beautiful Soup. The best solution is to install an external parser (lxml or html5lib), and use Beautiful Soup with that parser. See http://www.crummy.com/software/BeautifulSoup/bs4/doc/#installing-a-parser for help.
  "Python's built-in HTMLParser cannot parse the given document. This is not a bug in Beautiful Soup. The best solution is to install an external parser (lxml or html5lib), and use Beautiful Soup with that parser. See http://www.crummy.com/software/BeautifulSoup/bs4/doc/#installing-a-parser for help."))
Traceback (most recent call last):
  File "web.py", line 5, in <module>
    soup = BS(res.read())
  File "/Library/Python/2.7/site-packages/bs4/__init__.py", line 172, in __init__
    self._feed()
  File "/Library/Python/2.7/site-packages/bs4/__init__.py", line 185, in _feed
    self.builder.feed(self.markup)
  File "/Library/Python/2.7/site-packages/bs4/builder/_htmlparser.py", line 150, in feed
    raise e

我已经让另外两个人使用了这段代码，它们对他们来说工作得很好。为什么它对我不起作用？我已经安装了 bs4...

Answer 1

根据错误消息，您可能需要做的一件事是安装 lxml，这将为 BeautifulSoup 使用提供更强大的解析引擎。参见 this文档中的部分以获得更好的概述，但它适用于其他两个人的可能原因是他们安装了 lxml(或另一个正确处理 HTML 的解析器)，这意味着 BeautifulSoup 使用它标准内置的(附注:您的示例在安装了 lxml 的系统上也适用于我，但在没有安装它的系统上失败)。

此外，请参阅文档中的注释:

If you’re using a version of Python 2 earlier than 2.7.3, or a version of Python 3 earlier than 3.2.2, it’s essential that you install lxml or html5lib–Python’s built-in HTML parser is just not very good in older versions.

我建议运行 sudo apt-get install python-lxml 并查看问题是否仍然存在。