gpt4 book ai didi

python - 从交互式图表中抓取数据

转载 作者:行者123 更新时间:2023-12-02 06:52:53 25 4
gpt4 key购买 nike

有一个website有几个交互式图表,我想从中提取数据。我之前使用 selenium webdriver 在 python 中编写了几个网络抓取工具,但这似乎是一个不同的问题。我在 stackoverflow 上看过几个类似的问题。从这些看来,解决方案可能是直接从 json 文件下载数据。我查看了网站的源代码,发现了几个 json 文件,但经过检查,它们似乎不包含数据。

有人知道如何从这些图表中下载数据吗?我特别对这个条形图感兴趣:.//*[@id='network_download']

谢谢

编辑:我应该补充一点,当我使用 Firebug 检查网站时,我发现可以获取以下格式的数据。但这显然没有帮助,因为它不包含任何标签。

<circle fill="#8CB1AA" cx="713.4318516666667" cy="5.357142857142858" r="4.5" style="opacity: 0.983087;">
<circle fill="#8CB1AA" cx="694.1212663333334" cy="10.714285714285715" r="4.5" style="opacity: 0.983087;">
<circle fill="#CEA379" cx="626.4726493333333" cy="16.071428571428573" r="4.5" style="opacity: 0.983087;">
<circle fill="#B0B359" cx="613.88416" cy="21.42857142857143" r="4.5" style="opacity: 0.983087;">
<circle fill="#D1D49E" cx="602.917665" cy="26.785714285714285" r="4.5" style="opacity: 0.983087;">
<circle fill="#A5E0B5" cx="581.5437366666666" cy="32.142857142857146" r="4.5" style="opacity: 0.983087;">

最佳答案

像这样的 SVG 图表往往有点难以抓取。直到您实际用鼠标悬停在各个元素上时,才会显示您想要的数字。

获取您需要的数据

  1. 查找所有点的列表
  2. 对于dots_list 中的每个点,单击或悬停(操作链)该点
  3. 抓取弹出的工具提示中的值

这对我有用:

from __future__ import print_function

from pprint import pprint as pp

from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains


def main():
driver = webdriver.Chrome()
ac = ActionChains(driver)

try:
driver.get("https://opensignal.com/reports/2016/02/state-of-lte-q4-2015/")

dots_css = "div#network_download g g.dots_container circle"
dots_list = driver.find_elements_by_css_selector(dots_css)

print("Found {0} data points".format(len(dots_list)))

download_speeds = list()
for index, _ in enumerate(dots_list, 1):
# Because this is an SVG chart, and because we need to hover it,
# it is very likely that the elements will go stale as we do this. For
# that reason we need to require each dot element right before we click it
single_dot_css = dots_css + ":nth-child({0})".format(index)
dot = driver.find_element_by_css_selector(single_dot_css)
dot.click()

# Scrape the text from the popup
popup_css = "div#network_download div.tooltip"
popup_text = driver.find_element_by_css_selector(popup_css).text
pp(popup_text)
rank, comp_and_country, speed = popup_text.split("\n")
company, country = comp_and_country.split(" in ")
speed_dict = {
"rank": rank.split(" Globally")[0].strip("#"),
"company": company,
"country": country,
"speed": speed.split("Download speed: ")[1]
}
download_speeds.append(speed_dict)

# Hover away from the tool tip so it clears
hover_elem = driver.find_element_by_id("network_download")
ac.move_to_element(hover_elem).perform()

pp(download_speeds)

finally:
driver.quit()

if __name__ == "__main__":
main()

示例输出:

(.venv35) ➜  stackoverflow python svg_charts.py
Found 182 data points
'#1 Globally\nSingTel in Singapore\nDownload speed: 40 Mbps'
'#2 Globally\nStarHub in Singapore\nDownload speed: 39 Mbps'
'#3 Globally\nSaskTel in Canada\nDownload speed: 35 Mbps'
'#4 Globally\nOrange in Israel\nDownload speed: 35 Mbps'
'#5 Globally\nolleh in South Korea\nDownload speed: 34 Mbps'
'#6 Globally\nVodafone in Romania\nDownload speed: 33 Mbps'
'#7 Globally\nVodafone in New Zealand\nDownload speed: 32 Mbps'
'#8 Globally\nTDC in Denmark\nDownload speed: 31 Mbps'
'#9 Globally\nT-Mobile in Hungary\nDownload speed: 30 Mbps'
'#10 Globally\nT-Mobile in Netherlands\nDownload speed: 30 Mbps'
'#11 Globally\nM1 in Singapore\nDownload speed: 29 Mbps'
'#12 Globally\nTelstra in Australia\nDownload speed: 29 Mbps'
'#13 Globally\nTelenor in Hungary\nDownload speed: 29 Mbps'
<...>
[{'company': 'SingTel',
'country': 'Singapore',
'rank': '1',
'speed': '40 Mbps'},
{'company': 'StarHub',
'country': 'Singapore',
'rank': '2',
'speed': '39 Mbps'},
{'company': 'SaskTel', 'country': 'Canada', 'rank': '3', 'speed': '35 Mbps'}
...
]

应该注意的是,您在问题中引用的圆形元素中的值并不是特别有用,因为它们只是指定如何在 SVG 图表中绘制点。

关于python - 从交互式图表中抓取数据,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/39571193/

25 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com