python - 如何在 Scrapy 中创建基于 href 的 LinkExtractor 规则

Question

我正在尝试使用 Scrapy (scrapy.org) 创建简单的爬虫。根据示例 item.php 是允许的。我如何编写允许始终以 http://example.com/category/ 开头但在 GET 参数 page 中的 url 的规则应该是那里有任意数量的数字和其他参数。这些参数的顺序是随机的。 请帮助我如何编写这样的规则？

几个有效值是:

• http://example.com/category/?page=1&sort=a-z&cache=1
• http://example.com/category/?page=1&sort=a-z#
• http://example.com/category/?sort=a-z&page=1

代码如下:

import scrapy
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors import LinkExtractor
class MySpider(CrawlSpider):
name = 'example.com'
allowed_domains = ['example.com']
start_urls = ['http://www.example.com/category/']

rules = (
    Rule(LinkExtractor(allow=('item\.php', )), callback='parse_item'),
)

def parse_item(self, response):
    item = scrapy.Item()
    item['id'] = response.xpath('//td[@id="item_id"]/text()').re(r'ID: (\d+)')
    item['name'] = response.xpath('//td[@id="item_name"]/text()').extract()
    item['description'] = response.xpath('//td[@id="item_description"]/text()').extract()
    return item

Answer 1

测试字符串开头的 http://example.com/category/ 和值中包含一位或多位数字的 page 参数:

Rule(LinkExtractor(allow=('^http://example.com/category/\?.*?(?=page=\d+)', )), callback='parse_item'),

演示(使用您的示例网址):

>>> import re
>>> pattern = re.compile(r'^http://example.com/category/\?.*?(?=page=\d+)')
>>> should_match = [
...     'http://example.com/category/?sort=a-z&page=1',
...     'http://example.com/category/?page=1&sort=a-z&cache=1',
...     'http://example.com/category/?page=1&sort=a-z#'
... ]
>>> for url in should_match:
...     print "Matches" if pattern.search(url) else "Doesn't match"
... 
Matches
Matches
Matches