python - 阅读美丽汤的链接列表

Question

我一直在尝试从我成功提取的 URL 列表中读取链接。我的问题是，当我尝试读取整个列表时，我收到 TypeError Traceback (most最近的调用最后)。但是，当我读取单个链接时，urlopen(urls).read() 行执行时没有问题。

response = requests.get('some_website')
doc = BeautifulSoup(response.text, 'html.parser')
headlines = doc.find_all('h3')

links = doc.find_all('a', { 'rel':'bookmark' })
for link in links:
    print(link['href'])

for urls in links:
    raw_html = urlopen(urls).read()  <----- this row here
    articles = BeautifulSoup(raw_html, "html.parser")

Answer 1

考虑将 BeautifulSoup 与 requests.Session() 结合使用，以提高重用连接和添加 header 的效率

import requests
from bs4 import BeautifulSoup

with requests.Session() as s:

    url = 'https://newspunch.com/category/news/us/'
    headers = {'User-Agent': 'Mozilla/5'}
    r = s.get(url, headers = headers)
    soup = BeautifulSoup(r.text, 'lxml')
    links = [item['href'] for item in soup.select('[rel=bookmark]')]

    for link in links:
        r = s.get(link)
        soup = BeautifulSoup(r.text, 'lxml')
        print(soup.select_one('.entry-title').text)