python - Scrapy SgmlLinkExtractor 引用者 无

Question

我正在努力让我的蜘蛛正常工作。这是我在蜘蛛中的代码:

start_urls = ["http://www.khmer24.com/"]   

rules = (
Rule(SgmlLinkExtractor(allow=(r'ad/\w+/67-\d+\.html',),
    ), 
    callback='parse_items'),
)

示例网址如下所示: http://www.khmer24.com/ad/honda-click-2012-98/67-258149.html

我想保留“ad”和“67-”

scrapy scrapy khmer24 的输出是:

Crawled (200) <GET http://www.khmer24.com/> (referer: None)

我不明白为什么这是我的完整代码:

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import HtmlXPathSelector


class MySpider(CrawlSpider):
    name = "khmer24"
    allowed_domains = ["khmer24.com"]
    start_urls = ["http://www.khmer24.com/"]   

    rules = (
    Rule(SgmlLinkExtractor(allow=(r'ad/\w+/67-\d+\.html',),
        ), 
        callback='parse_items'),
    )

    def parse_items(self, response):
        hxs = HtmlXPathSelector(response)
        titles = hxs.select("//div[@class='innerbox']/h1/text()")
        return(titles)

Answer 1

所以你的问题是，“为什么我的推荐人没有？”

日志输出中的行

Crawled (200) <GET http://www.khmer24.com/> (referer: None)

来自start_urls，而不是链接提取器。默认情况下，从 start_urls 发出的请求不包含 Referer header 。您可以通过发出 requests yourself 手动添加 header .