python - Scrapy 或 BeautifulSoup 从各种网站上抓取链接和文本

Question

我试图从输入的 URL 中抓取链接，但它只适用于一个 url ( http://www.businessinsider.com )。它如何适应从输入的任何 url 中抓取？我正在使用 BeautifulSoup，但 Scrapy 更适合这个吗？

def WebScrape():  
    linktoenter = input('Where do you want to scrape from today?: ')
    url = linktoenter
    html = urllib.request.urlopen(url).read()
    soup = BeautifulSoup(html, "lxml")

    if linktoenter in url:
        print('Retrieving your links...')
        links = {}
        n = 0
        link_title=soup.findAll('a',{'class':'title'})
        n += 1
        links[n] = link_title
        for eachtitle in link_title:
            print(eachtitle['href']+","+eachtitle.string)
    else:
        print('Please enter another Website...')

Answer 1

您可以制作一个更通用的抓取工具，搜索所有标签和这些标签内的所有链接。获得所有链接的列表后，您可以使用正则表达式或类似表达式来查找与所需结构匹配的链接。

import requests
from bs4 import BeautifulSoup
import re

response = requests.get('http://www.businessinsider.com')

soup = BeautifulSoup(response.content)

# find all tags
tags = soup.find_all()

links = []

# iterate over all tags and extract links
for tag in tags:
    # find all href links
    tmp = tag.find_all(href=True)
    # append masters links list with each link
    map(lambda x: links.append(x['href']) if x['href'] else None, tmp)

# example: filter only careerbuilder links
filter(lambda x: re.search('[w]{3}\.careerbuilder\.com', x), links)