Python BeautifulSoup 每页仅获取 1 项

Question

我正在使用 BeautifulSoup 框架使用 python 开发网络爬虫。我从目标页面获得了正确的信息。但它只从每个页面获取 1 项。

我的代码如下:

import csv
import time
import bs4 as bs
from urllib.request import Request, urlopen

for i in range(1, 5):
   site = "XXURLXX".format(i)
   hdr = {'User-Agent': 'Mozilla/5.0'}
   req = Request(site, headers=hdr)
   page = urlopen(req)
   soup = bs.BeautifulSoup(page, 'html5lib')

data = []

for get_info in soup:
    name_box = soup.find('h2', attrs={'class': 'post-title'})
    name = name_box.text.strip()


    url_box = soup.find('a', attrs={'class': 'post-excerpt-download'})
    url = url_box.get('href')

data.append((name, url, site))

time.sleep(1)

print(data)

with open('stellarismods.csv', 'a') as csv_file:
    writer = csv.writer(csv_file)
    for url in data:
        writer.writerow([name, url, site])

我已经尝试过for循环

for name_box in soup.find('h2', attrs={'class': 'post-title'}):
       name = name_box.text.strip()


for url_box in soup.find('a', attrs={'class': 'post-excerpt-download'}):
    url = url_box.get('href')

但我仍然从每个页面只得到 1 个项目。

Answer 1

尝试find_all功能。它将查找您正在寻找的所有元素。