python - Scrapy Xpath 构造在动态站点上生成空括号

Question

我正在尝试通过 scrapy 创建一个蜘蛛来抓取网站并提取特定商店的所有链接。最终，蜘蛛将使用这些商店链接来提取价格信息。该站点旨在将商店信息分解为州和地区。我已经能够提取州/地区的所有子链接，但我似乎无法提取商店的特定信息，该信息是商店链接。

我认为这与页面加载的时间不够长，无法显示这些商店链接有关。所以我使用了 selenium 并将时间延迟了 20 秒，以便为网页加载留出适当的时间。一个示例链接是 https://weedmaps.com/dispensaries/in/united-states/colorado正如您在左侧看到的，有特定商店的信息。使用 inspect 元素的代码片段生成以下 HTML:

<li class="ng-scope" ng-repeat="listing in listings" wm-listing-detail="" role="listing">
  <div ng-class="{unpublished: !listing.published}">
     <div class="listing feature-1 recreational dispensary" ng-     class="featureClassesFor(listing)" ng-click="setActiveMarker(listing)">
         <img class="avatar" ng-src="https://d2kxqxnk1i5o9a.cloudfront.net/uploads/avatars/dispensaries/5566/square_10569095_10152344322971376_2924814837799131094_n.jpg" src="https://d2kxqxnk1i5o9a.cloudfront.net/uploads/avatars/dispensaries/5566/square_10569095_10152344322971376_2924814837799131094_n.jpg"/>
         <div>
           <div class="name hovers">
              <a class="ng-binding" href="/dispensaries/organic-alternatives" role="listing url" ng-bind-html="listing.name">Organic Alternatives</a>
           </div>
         </div>
         <div class="address ng-binding">
         <div class="wm_map_inactive_rating">

我想提取@href 后面的信息。

我在 scrapy 项目的蜘蛛部分有以下 python 代码:

import scrapy
from scrapy.spider import BaseSpider 
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait

class scrapybotspider(scrapy.Spider):
    name = "scrapybot_spider"
    start_urls = ['https://weedmaps.com/dispensaries/in/united-states/colorado']
    delay = 20

    def __init__(self):
        self.driver = webdriver.Firefox()

    def parse(self, response):
        self.driver.get(response.url)

        while True:
            StateCountyURL = response.xpath("//*[@id='wm-listings']/div/li[2]/div/div/div[1]/div/@href").extract()
            print StateCountyURL

这会产生空括号，我已经尝试了许多不同的 xpath 结构。

Answer 1

您需要开始使用 Explicit Waits .

等待列表可见。获取区域链接的工作示例代码:

from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC


driver = webdriver.Firefox()
driver.get('https://weedmaps.com/dispensaries/in/united-states/colorado')

wait = WebDriverWait(driver, 10)
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.subregion a.region-ajax-link")))

for region in driver.find_elements_by_css_selector("div.subregion a.region-ajax-link"):
    print region.get_attribute("href")

driver.close()

打印:

https://weedmaps.com/dispensaries/in/aspen-colorado
https://weedmaps.com/dispensaries/in/aurora
https://weedmaps.com/dispensaries/in/boulder
https://weedmaps.com/dispensaries/in/colorado-springs
...
https://weedmaps.com/dispensaries/in/other-co
https://weedmaps.com/dispensaries/in/summit-eagle-county
https://weedmaps.com/dispensaries/in/trinidad

同样，对于药房:

wait = WebDriverWait(driver, 10)
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.dispensary div.name a")))

for dispensary in driver.find_elements_by_css_selector("div.dispensary div.name a"):
    print dispensary.get_attribute("href")