python - 在 Python 中遍历 HTML DOM

Question

我想编写一个 Python 脚本(使用 3.4.3)，它从 URL 抓取 HTML 页面并可以通过 DOM 尝试找到特定元素。

我目前有这个:

#!/usr/bin/env python
import urllib.request

def getSite(url):
    return urllib.request.urlopen(url)

if __name__ == '__main__':
    content = getSite('http://www.google.com').read()
    print(content)

当我打印内容时，它确实会打印出整个 html 页面，这与我想要的很接近……尽管我理想情况下希望能够在 DOM 中导航，而不是将其视为一个巨大的字符串。

我对 Python 还是相当陌生，但有使用多种其他语言(主要是 Java、C#、C++、C、PHP、JS)的经验。我以前用 Java 做过类似的事情，但想在 Python 中尝试一下。

Answer 1

您可以使用许多不同的模块。例如，lxml或 BeautifulSoup .

这是一个 lxml 示例:

import lxml.html

mysite = urllib.request.urlopen('http://www.google.com').read()
lxml_mysite = lxml.html.fromstring(mysite)

description = lxml_mysite.xpath("//meta[@name='description']")[0] # meta tag description
text = description.get('content') # content attribute of the tag

>>> print(text)
"Search the world's information, including webpages, images, videos and more. Google has many special features to help you find exactly what you're looking for."

还有一个 BeautifulSoup 示例:

from bs4 import BeautifulSoup

mysite = urllib.request.urlopen('http://www.google.com').read()
soup_mysite = BeautifulSoup(mysite)

description = soup_mysite.find("meta", {"name": "description"}) # meta tag description
text = description['content'] # text of content attribute

>>> print(text)
u"Search the world's information, including webpages, images, videos and more. Google has many special features to help you find exactly what you're looking for."

请注意 BeautifulSoup 如何返回一个 unicode 字符串，而 lxml 不会。根据需要，这可能有用/有害。