python - Python 请求(AJAX 请求)数据丢失

Question

我正在尝试从该网站抓取历史天气数据: http://www.hko.gov.hk/cis/dailyExtract_uc.htm?y=2016&m=1

在阅读了 AJAX 调用后，我发现请求数据的正确方法是通过以下代码:

import requests
from bs4 import BeautifulSoup
import json
import pandas as pd

headers = {
'Accept': 'text/plain, */*; q=0.01',
'Accept-Encoding': 'gzip, deflate',
'Accept-Language': 'en-US,en;q=0.9',
'Connection': 'keep-alive',
'Host': 'www.hko.gov.hk',
'Referer': 'http://www.hko.gov.hk/cis/dailyExtract_e.htm?y=2016&m=3',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36',
'X-Requested-With': 'XMLHttpRequest'
}

with requests.Session() as s:
#request April 2015 weather data
    r = s.get(r"http://www.hko.gov.hk/cis/dailyExtract/dailyExtract_201504.xml",verify = False, headers = headers)
    soup = BeautifulSoup(r.content,'lxml')
data = json.loads(soup.get_text())['stn']['data'][0]['dayData'][:-2]
df = pd.DataFrame(data)

我注意到我检索到的数据不包含右侧的 3 列，我在 get 请求中错过了什么？

Answer 1

似乎如果您请求全年，然后提取其中的月份

import requests
import json

with requests.Session() as s:
    r = s.get(r"http://www.hko.gov.hk/cis/dailyExtract/dailyExtract_2015.xml",headers = {'User-Agent': 'Mozilla/5.0'}).json()
    print(r['stn']['data'][3]['dayData'][0])