python - 使用python(通过请求或其他方式)检索下拉列表选择的html表

Question

我正在尝试从以下站点提取历史天气数据:https://www.timeanddate.com/weather/usa/boston/historic

我需要选择两个下拉菜单才能加载任何特定日期的表格:一个用于月份/年份，另一个用于当天。

我一直在尝试通过以下方式使用请求来获取数据:

from bs4 import BeautifulSoup as soup
import requests
import pandas as pd

website = "https://www.timeanddate.com/weather/usa/boston/historic"
payload = {"month": '2018-06',"wt-his-select":"20180608"}
page_response = requests.get(website, data = payload, timeout=5).text
sp = soup(page_response,'lxml')
My_table = sp.find('table',id='wt-his')
table_rows = My_table.find_all('tr')
rows = []
for tr in table_rows:
    td = tr.find_all(['th','td'])
    rows.append([i.text for i in td])
pd.DataFrame(rows)

但是运行此代码，我只获得今天的值，而不是 2018/06/08 - 我做错了什么？

谢谢。

编辑:如果您无法通过请求执行此操作，那么您该怎么做？

Answer 1

该页面的数据通过JSON动态加载。当您更改日期时，您可以在网络选项卡中找到此 AJAX 请求。但是 JSON 的问题在于它没有采用正确的格式，因此当您使用 response.json() 时，它会给您一个错误。因此，您必须将数据转换为正确的格式。

import re
import requests

url = 'https://www.timeanddate.com/scripts/cityajax.php?n=usa/boston&mode=historic&hd=20180608&month=6&year=2018&json=1'

headers = {
'Accept': '*/*', 'Accept-Language': 'en-US,en;q=0.5', 'Accept-Encoding': 'gzip, deflate, br',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36',
}
response  = requests.get(url, headers=headers)

raw_json = eval(response.text.replace(',{c:', '""",{c:').replace('{c:', '"""{c:')[::-1].replace(']', ']"""', 1)[::-1])
for i in raw_json:
    json_data = eval(i.replace('{c:', '{"c":').replace('{h:', '{"h":').replace('{s:', '{"s":').replace(',h:', ',"h":').replace('="', '=\'').replace('">', '\'>').replace('" ', '\' '))

    visibility = json_data['c'][8]['h'].replace(' ', '')
    barometer = json_data['c'][7]['h']
    humidity = json_data['c'][6]['h']
    wind = json_data['c'][4]['h'].replace('\/', '/')
    weather = json_data['c'][3]['h']
    temp = json_data['c'][2]['h'].replace(' ', '')
    time = re.findall(r'\d{2}:\d{2}', json_data['c'][0]['h'])[0]

    print(time, temp, weather, wind, humidity, barometer, visibility, end='    |   ')

输出:

00:54 14°C Overcast. 15 km/h 78% 1020 mbar 16km    |   01:54 14°C Overcast. 13 km/h 78% 1020 mbar 16km    |   02:54 13°C Passing clouds. 15 km/h 84% 1020 mbar 16km    |   03:54 13°C Passing clouds. 15 km/h 84% 1020 mbar 16km    |   04:54 13°C Clear. 15 km/h 87% 1020 mbar 16km    |   05:54 13°C Passing clouds. 20 km/h 90% 1020 mbar 16km    |   06:54 14°C Passing clouds. 13 km/h 81% 1020 mbar 16km    |   07:54 17°C Passing clouds. 11 km/h 73% 1020 mbar 16km    |   08:54 19°C Passing clouds. 7 km/h 66% 1020 mbar 16km    |   09:54 21°C Scattered clouds. 13 km/h 66% 1020 mbar 16km    |   10:54 24°C Passing clouds. 7 km/h 48% 1020 mbar 16km    |   11:54 25°C Passing clouds. 13 km/h 47% 1019 mbar 16km    |   12:54 26°C Broken clouds. 7 km/h 42% 1019 mbar 16km    |   13:54 27°C Partly sunny. 9 km/h 42% 1018 mbar 16km    |   15:54 21°C Partly sunny. 13 km/h 70% 1018 mbar 16km    |   16:54 22°C Broken clouds. 15 km/h 63% 1018 mbar 16km    |   17:54 21°C Scattered clouds. 11 km/h 64% 1017 mbar 16km    |   18:54 21°C Scattered clouds. 9 km/h 64% 1018 mbar 16km    |   19:54 22°C Scattered clouds. 11 km/h 57% 1018 mbar 16km    |   20:54 22°C Passing clouds. 6 km/h 59% 1018 mbar 16km    |   21:54 21°C Passing clouds. No wind 66% 1019 mbar 16km    |   22:54 22°C Passing clouds. 6 km/h 57% 1019 mbar 16km    |   23:54 21°C Clear. 6 km/h 59% 1018 mbar 16km