python - 我应该如何使用 python 和 BS4 从网页中正确提取和解析主题数据？

Question

我正在寻找构建一个网络爬虫，从论坛收集主题行。一旦完成，我想将每个主题显示为一个新行，并在每行前面添加一个 [*]。

使用 BeautifulSoup，我能够抓取一个页面并提取跨度类“主题”。然而，从那里开始，我不确定如何仅解析出主题文本，然后按照我想要的方式对其进行排序。

import requests

from bs4 import BeautifulSoup

url = "https://boards.4channel.org/sci/"

#send the HTTP request
response = requests.get(url)

if response.status_code == 200:

    #pull the content
    html_content = response.content

    #send the page to BeautifulSoup
    html_doc     = BeautifulSoup(html_content, "html.parser")

    #extract topic data
    topic_spider = html_doc.find_all("span",{"class":"subject"})

    print topic_spider

抓取工具的当前结果如下所示:

[<span class="subject"></span>, <span class="subject"></span>, <span class="subject">Cigarettes vs. Cannabis</span>, <span class="subject">Cigarettes vs. Cannabis</span>, <span class="subject"></span>, <span class="subject"></span>, <span class="subject"></span>, <span class="subject"></span>, <span class="subject"></span>...

我正在尝试像这样订购它们:

[*] Topic 1
[*] Topic 2
[*] Topic 3

Answer 1

检查元素的文本是否不为空，然后删除重复项并对列表进行排序，然后遍历并将 [*] 添加到字符串中。希望您能关注这一点。如果没有，请让我知道您的预期输出。

import requests
from bs4 import BeautifulSoup

url = "https://boards.4channel.org/sci/"

#send the HTTP request
response = requests.get(url)

if response.status_code == 200:

    #pull the content
    html_content = response.content

    #send the page to BeautifulSoup
    html_doc     = BeautifulSoup(html_content, "html.parser")

    #extract topic data
    topic_spider = html_doc.find_all("span",{"class":"subject"})
    data=[]
    for topic in topic_spider:
        if topic.text!='':
             data.append(topic.text)
    mylist = list(dict.fromkeys(data)) #Remove the duplicates here
    mylist.sort(reverse=False) #sort here
    for d in mylist:
        print ('[*]' + d)