python - 如何在 zillow 中抓取价格/税收历史记录表？

Question

from bs4 import BeautifulSoup
from selenium import webdriver
#import urllib2
import time

driver = webdriver.Chrome()
driver.maximize_window()
driver.get("https://www.zillow.com/homes/recently_sold/Culver-City-CA/house,condo,apartment_duplex,townhouse_type/20432063_zpid/51617_rid/12m_days/globalrelevanceex_sort/34.048605,-118.340178,33.963223,-118.47785_rect/12_zm/")
time.sleep(3)
driver.find_element_by_class_name("collapsible-header").click()
soup = BeautifulSoup(driver.page_source,"lxml")

region = soup.find("div",{"id":"hdp-price-history"})
table = region.find('table',{'class':'zsg-table yui3-toggle-content-minimized'})
print table

我尝试在 zillow 中刮掉税费/餐 table 价格，但我得到的结果是“无”。我如何获得该表？

Answer 1

以下使用requests和BeautifulSoup来获取数据，不需要selenium(所以速度很快)。

from bs4 import BeautifulSoup
import requests
import re

headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:55.0) Gecko/20100101 Firefox/55.0"}    
r = requests.get("https://www.zillow.com/homes/recently_sold/Culver-City-CA/house,condo,apartment_duplex,townhouse_type/20432063_zpid/51617_rid/12m_days/globalrelevanceex_sort/34.048605,-118.340178,33.963223,-118.47785_rect/12_zm/", headers=headers)
urls = re.findall(re.escape('AjaxRender.htm?') + '(.*?)"', r.content)
url = "https://www.zillow.com/AjaxRender.htm?{}".format(urls[4])
r = requests.get(url, headers=headers)
soup = BeautifulSoup(r.content.replace('\\', ''), "html.parser")
data = []

for tr in soup.find_all('tr'):
    data.append([td.text for td in tr.find_all('td')])

for row in data[:5]:        # Show first 5 entries    
    print row

这显示前 5 个条目是:

[u'06/16/17', u'Sold', u'$940,000-0.9%', u'K. Miller, A. Masket', u'']
[u'06/14/17', u'Price change', u'$949,000-1.0%', u'', u'']
[u'05/08/17', u'Pending sale', u'$959,000', u'', u'']
[u'04/17/17', u'Price change', u'$959,000+1.1%', u'', u'']
[u'02/27/17', u'Pending sale', u'$949,000', u'', u'']

所需的 HTML 不存在于第一个 GET 中，但它是在展开价格/税收历史记录部分时按需生成的。这会在浏览器中触发 AJAX 请求。这些代码在初始 HTML 中搜索所有这些请求并发出相同的请求。第四个这样的请求用于获取所需的部分。返回的 HTML 需要删除 \ ，然后可以传递给 BeautifulSoup 进行解析为表格。