python - Scrapy start_urls 未解析

Question

我刚刚将 scrapy 更新到 0.22，现在面临一些问题。

我使用的是基本的 Spider，只是想请求一个 start_url 并在 parse(self,response) 函数中查看结果。

class wildberries(Spider):

    name = "wildberries"
    allowed_domains = ['wildberries.ru']
    start_urls = ['http://www.wildberries.ru/']

    def parse(self, response):
        hxs         = HtmlXPathSelector(response)
        print "Hello World", response.url
        maincats    = hxs.select('//div[@class="tomenu"]/ul[@class="green_menu"]/li/a/@href').extract()
        print maincats

执行蜘蛛程序得到的唯一输出是:

[jdargel@rhel1 krauler]$ scrapy crawl wildberries
2014-02-25 12:46:19+0400 [scrapy] INFO: Scrapy 0.22.2 started (bot: scrapybot)
2014-02-25 12:46:19+0400 [scrapy] INFO: Optional features available: ssl, http11
2014-02-25 12:46:19+0400 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE':       'krauler.spiders', 'LOG_STDOUT': True, 'SPIDER_MODULES': ['krauler.spiders'], 'USER_AGENT': 'Mozilla/5.0 (Windows NT 6.1; WOW64)', 'CONCURRENT_REQUESTS': 1}

我不知道为什么我的 start_url 没有被解析，因为我没有从 parse() 得到任何响应，我认为该函数甚至没有被调用。我很乐意提供建议，如果出现任何问题，我非常乐意提供进一步的代码。

Answer 1

缩进看起来很奇怪。我不知道是因为 StackOverflow 还是什么原因，但是在尝试时

from scrapy.spider import Spider
from scrapy.selector import Selector


class wildberries(Spider):
    name = "wildberries"
    allowed_domains = ['wildberries.ru']
    start_urls      = ['http://www.wildberries.ru/']

    def parse(self, response):
        selector = Selector(response)
        print "Hello World", response.url
        maincats = selector.xpath('//div[@class="topmenu"]/ul[@class="green_menu"]/li/a/@href').extract()
        print maincats

我确实得到了“正确”的输出:

paul@wheezy:~/tmp$ scrapy runspider wildberries.py 
2014-02-25 10:34:44+0100 [scrapy] INFO: Scrapy 0.23.0 started (bot: scrapybot)
2014-02-25 10:34:44+0100 [scrapy] INFO: Optional features available: ssl, http11, boto, django
2014-02-25 10:34:44+0100 [scrapy] INFO: Overridden settings: {}
2014-02-25 10:34:44+0100 [scrapy] INFO: Enabled extensions: LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
2014-02-25 10:34:44+0100 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2014-02-25 10:34:44+0100 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2014-02-25 10:34:44+0100 [scrapy] INFO: Enabled item pipelines: 
2014-02-25 10:34:44+0100 [wildberries] INFO: Spider opened
2014-02-25 10:34:44+0100 [wildberries] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2014-02-25 10:34:44+0100 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6023
2014-02-25 10:34:44+0100 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080
2014-02-25 10:34:44+0100 [wildberries] DEBUG: Crawled (200) <GET http://www.wildberries.ru/> (referer: None)
Hello World http://www.wildberries.ru/
[u'http://www.wildberries.ru/', u'http://www.wildberries.ru/catalog/1/women.aspx', u'http://www.wildberries.ru/catalog/2/men.aspx', u'http://www.wildberries.ru/catalog/3/child.aspx', u'http://www.wildberries.ru/catalog/413/women.aspx', u'http://www.wildberries.ru/catalog/492/women.aspx', u'http://www.wildberries.ru/catalog/524/women.aspx', u'http://www.wildberries.ru/catalog/2467/page.aspx', u'http://www.wildberries.ru/catalog/4263/page.aspx', u'http://www.wildberries.ru/catalog/181/women.aspx']
2014-02-25 10:34:44+0100 [wildberries] INFO: Closing spider (finished)
2014-02-25 10:34:44+0100 [wildberries] INFO: Dumping Scrapy stats:
    {'downloader/request_bytes': 217,
     'downloader/request_count': 1,
     'downloader/request_method_count/GET': 1,
     'downloader/response_bytes': 15623,
     'downloader/response_count': 1,
     'downloader/response_status_count/200': 1,
     'finish_reason': 'finished',
     'finish_time': datetime.datetime(2014, 2, 25, 9, 34, 44, 519186),
     'log_count/DEBUG': 3,
     'log_count/INFO': 7,
     'response_received_count': 1,
     'scheduler/dequeued': 1,
     'scheduler/dequeued/memory': 1,
     'scheduler/enqueued': 1,
     'scheduler/enqueued/memory': 1,
     'start_time': datetime.datetime(2014, 2, 25, 9, 34, 44, 143133)}
2014-02-25 10:34:44+0100 [wildberries] INFO: Spider closed (finished)