javascript - 抓取 : Extract data from chart

Question

我找到了一种从 chart page of the Wall Street Journal 下载数据的方法通过查看网络选项卡(在开发工具面板中)并重现刷新图表时创建的请求。它的工作原理如下:

import requests
import json
import time
from datetime import datetime as dt
from urllib.parse import urlencode 

data = {
        "Step":"PT5M",
        "TimeFrame":"D1",
        "StartDate":int(dt(2019, 5, 1).timestamp()*1000),
        "EndDate":int(dt(2019, 5, 5).timestamp()*1000),
        "EntitlementToken":"57494d5ed7ad44af85bc59a51dd87c90",
        "IncludeMockTick":True,
        "FilterNullSlots":False,
        "FilterClosedPoints":True,
        "IncludeClosedSlots":False,
        "IncludeOfficialClose":True,
        "InjectOpen":False,
        "ShowPreMarket":False,
        "ShowAfterHours":False,
        "UseExtendedTimeFrame":True,
        "WantPriorClose":False,
        "IncludeCurrentQuotes":False,
        "ResetTodaysAfterHoursPercentChange":False,
        "Series":[{"Key":"STOCK/US/XNYS/ABB","Dialect":"Charting","Kind":"Ticker","SeriesId":"s1","DataTypes":["Last"],"Indicators":[{"Parameters":[{"Name":"Period","Value":"50"}],"Kind":"SimpleMovingAverage","SeriesId":"i2"},{"Parameters":[],"Kind":"Volume","SeriesId":"i3"}]}]
    }

data = {
    'json' : json.dumps(data)
}

data = urlencode(data)

headers = {
    'Accept': 'application/json, text/javascript, */*; q=0.01',
    'Dylan2010.EntitlementToken': '57494d5ed7ad44af85bc59a51dd87c90',
    'Origin': 'https://quotes.wsj.com',
    'Referer': 'https://quotes.wsj.com/ABB/advanced-chart',
    'Sec-Fetch-Mode': 'cors',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36'
}

url = 'https://api.wsj.net/api/michelangelo/timeseries/history?' + data + '&ckey=57494d5ed7'
print(url)
r = requests.get(url, headers = headers)
r.text

这非常好，因为它非常简单且有效，但是我只能检索最多 25 天左右的分钟数据。

在另一端，morningstar charting似乎有更多的分钟数据可用，我想用它做同样的事情:只需通过查找更新图表时在后台进行的 javascript 调用来从网站获取数据。但是当我查看网络选项卡时，在更改日期范围时我看不到任何调用。我对 javascript 了解不多，想知道他们使用什么替代机制来实现这一目标。 (也许异步/获取？)

有谁知道我怎么可能看到那些电话？

Answer 1

您将需要调查是否有任何其他查询字符串参数是基于时间的(或依赖于代码的)。我用固定值替换了几个参数，但似乎仍然有效。

import requests

start_date = '20170814'
end_date = '20190102'

r = requests.get(f'https://quotespeed.morningstar.com/ra/uniqueChartData?instid=MSSAL&sdkver=2.39.0&CToken=1&productType=sdk&cdt=7&ed={end_date}&f=d&hasPreviousClose=true&ipoDates=19801212&pids=0P000000GY&sd={start_date}&tickers=126.1.AAPL&qs_wsid=27E31E614F74FC7D8828E941CAC2D319&tmpid=1&instid=MSSAL&_=1').json()

print(r)

---

使用 fiddler 观察到的原始参数: