python - Scrapy，只遵循内部 URLS 但提取所有找到的链接

Question

我想使用 Scrapy 从给定网站获取所有外部链接。使用以下代码，蜘蛛也可以抓取外部链接:

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors import LinkExtractor
from myproject.items import someItem

class someSpider(CrawlSpider):
  name = 'crawltest'
  allowed_domains = ['someurl.com']
  start_urls = ['http://www.someurl.com/']

  rules = (Rule (LinkExtractor(), callback="parse_obj", follow=True),
  )

  def parse_obj(self,response):
    item = someItem()
    item['url'] = response.url
    return item

我错过了什么？ “allowed_domains”不会阻止外部链接被抓取吗？如果我为 LinkExtractor 设置“allow_domains”，它不会提取外部链接。澄清一下:我不想抓取内部链接，而是提取外部链接。任何帮助都适用!

Answer 1

您还可以在解析每个页面后使用链接提取器提取所有链接。

链接提取器将为您过滤链接。在此示例中，链接提取器将拒绝允许域中的链接，因此它只会获取外部链接。

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors import LxmlLinkExtractor
from myproject.items import someItem

class someSpider(CrawlSpider):
  name = 'crawltest'
  allowed_domains = ['someurl.com']
  start_urls = ['http://www.someurl.com/']

  rules = (Rule(LxmlLinkExtractor(allow=()), callback='parse_obj', follow=True),)


  def parse_obj(self,response):
    for link in LxmlLinkExtractor(allow=(),deny = self.allowed_domains).extract_links(response):
        item = someItem()
        item['url'] = link.url