python - Scrapy 蜘蛛没有抓取正确的 div

Question

import scrapy
class rottenTomatoesSpider(scrapy.Spider):
    name = "movieList"
    start_urls = [
         'https://www.rottentomatoes.com/'
    ]

def parse(self, response):
    for movieList in response.xpath('//div[@id="homepage-opening-this-week"]'):
        yield {
           'score': response.css('td.left_col').extract_first(),
           'title': response.css('td.middle_col').extract_first(),
           'openingDate': response.css('td.right_col right').extract_first()
        }

所以蜘蛛正在抓取 <div id='homepage-tv-top'>

我假设它是 homepage-这使脚本变得困惑。有人知道解决方法吗？

Answer 1

您需要迭代每个tr，并且在for循环中使用movieList而不是response

for movieList in response.xpath('//div[@id="homepage-opening-this-week"]//tr'):
    yield {
       'score': "".join(a for a in movieList.css('td.left_col *::text').extract()),
       'title': "".join(a for a in movieList.css('td.middle_col *::text').extract()),
       'openingDate': "".join(a for a in movieList.css('td.right_col *::text').extract())
    }