Python 请求 401 错误，但 url 在浏览器中打开

Question

我试图从这个位置拉出 json -
https://www.nseindia.com/api/option-chain-indices?symbol=BANKNIFTY
这在我的浏览器中打开得很好，但在 python 中使用请求会引发 401 权限错误。我尝试添加具有不同参数的标题，但无济于事。
有趣的是，这个页面上的json直到https://www.nseindia.com才在浏览器中打开。是单独打开的。我相信它需要某种身份验证，但很惊讶它可以在没有任何身份验证的浏览器中运行。
有没有办法从这个网址中提取信息？任何帮助深表感谢。
这是我的实现 -

import requests

url = 'https://www.nseindia.com/api/option-chain-indices?symbol=BANKNIFTY'

# This throws a 401 response error
page = requests.get(url, headers={"User-Agent": "Mozilla/5.0"})

# This throws a 'Connection aborted' error
page = requests.get(url, headers={"User-Agent": "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8 GTB7.1 (.NET CLR 3.5.30729)"})

Answer 1

要获得正确的数据，首先使用 requests.get() 从其他 URL 加载 cookie。然后执行 Ajax 请求以加载 JSON:

import json
import requests
from bs4 import BeautifulSoup


headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:80.0) Gecko/20100101 Firefox/80.0'}
url = 'https://www.nseindia.com/api/option-chain-indices?symbol=BANKNIFTY'

with requests.session() as s:

    # load cookies:
    s.get('https://www.nseindia.com/get-quotes/derivatives?symbol=BANKNIFTY', headers=headers)

    # get data:
    data = s.get(url, headers=headers).json()

    # print data to screen:
    print(json.dumps(data, indent=4))

打印:

{
    "records": {
        "expiryDates": [
            "03-Sep-2020",
            "10-Sep-2020",
            "17-Sep-2020",
            "24-Sep-2020",
            "01-Oct-2020",
            "08-Oct-2020",
            "15-Oct-2020",
            "22-Oct-2020",
            "29-Oct-2020",
            "26-Nov-2020"
        ],
        "data": [
            {
                "strikePrice": 18100,
                "expiryDate": "03-Sep-2020",
                "CE": {
                    "strikePrice": 18100,
                    "expiryDate": "03-Sep-2020",
                    "underlying": "BANKNIFTY",
                    "identifier": "OPTIDXBANKNIFTY03-09-2020CE18100.00",
                    "openInterest": 1,
                    "changeinOpenInterest": 1,
                    "pchangeinOpenInterest": 0,
                    "totalTradedVolume": 2,
                    "impliedVolatility": 95.51,
                    "lastPrice": 6523.6,
                    "change": 2850.1000000000004,
                    "pChange": 77.58540901048048,
                    "totalBuyQuantity": 2925,
                    "totalSellQuantity": 2800,
                    "bidQty": 25,
                    "bidprice": 6523.6,
                    "askQty": 25,
                    "askPrice": 6570.3,
                    "underlyingValue": 24523.8
                },
                "PE": {
                    "strikePrice": 18100,
                    "expiryDate": "03-Sep-2020",
                    "underlying": "BANKNIFTY",

...and so on.