python - 需要网页抓取授权

Question

我正在尝试网络抓取 https://gomechanic.in/gurgaon/car-repair/hyundai-creta/petrol这个网站。我想要每项服务的价格。在网络中，我还获得了提供所有数据的 API。 API 的名称为 get-service-detials-by-category。 API链接:https://gomechanic.app/api/v2/oauth/customer/get-services-details-by-category?car_id=135&city_id=1&category_id=0&user_car_id=null .您可以在检查元素的网络部分看到此链接。但是当我在 python 中调用它时，它显示需要登录。但是数据在响应部分是可见的。

我的 Python 代码:

url = "https://gomechanic.app/api/v2/oauth/customer/get-services-details-by-category?car_id=135&city_id=1&category_id=0&user_car_id=null"
header = {
    "Accept": "*/*",
    "Accept-Encoding": "gzip, deflate, br",
    "Accept-Language": "en-GB,en-US;q=0.9,en;q=0.8",
    "Authorization": "Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJqdGkiOiJiNGJjM2NhZjVkMWVhOTlkYzk2YjQzM2NjYzQzMDI0ZTAyM2I0MGM2YjQ5ZjExN2JjMDk5OGY2MWU3ZDI1ZjM2MTU1YWU5ZDIxNjE2ZTc5NSIsInNjb3BlcyI6W10sInN1YiI6IjE2MzM5MTcwNjIzOSIsImV4cCI6MTYzNjUwOTA2Mi4wLCJhdWQiOiIzIiwibmJmIjoxNjMzOTE3MDYyLjAsImlhdCI6MTYzMzkxNzA2Mi4wfQ.IUP_RJVt6mqC5EMO3HKd-2iX69_dSgBEE-jJ0pg26bizK5EBvf48d0ZRiAcwPX6bNWAIkxH7hfqA0Zq0pu1SymyeDVpxmOB2D7H5t7aj1DqhgawxD7ZgoY6Q_nyA1SAmltbeFSAIf2mwVlQV4H-pdH74qiaIG1ij9kRsBpdLxSMqpas1Vy9mQN_8W5csu24gIjvPYAdaT6w6qxjCxrlbT24EJ0hswPCy4_h12AlpZKYs_oVAqHMKgVcyi9jSeXOS_KD8Kwbcx1hNVtYVblBg5xcezm8RtP8tcJ4XgHoqXWEmI349SEb1s8wZX4u1LtEKNovMWkBwWQr8_jBPNSy7rDRHnNAvT5h2u-x-1AlnN-JFLLsz9rCWLRoypG-_1-1Y46lOUAFgjVB1L4IvPJ3zk1dxjDNJPtxzV3e-GJWVv5qlHw3g3cTlTd05r4ab-PDj314K4Ft7P9RaLdgtrcdrpO_bbs00BBN7Vo87dPFL_NHl37FmWvsh6pQ1rCa6bkQKpCZTFxgcriKTjwYeeC2XLKpnJm26PQ5ALIQQIq4EEE2LZq7N8jZ-FEtg5ozTmQ9HxkZbG12LTZzbD472OmuuSDxRlWzyTF4ObMs0PyA8dyVRTYtYT8l1juxc71TRPzG_cnTVrbjCI-rbCqvvFokGosoC_VJdiQlt3Dau6t1IrmY",
    "Connection": "keep-alive",
    "Content-Type": "application/json",
    "Host": "gomechanic.app",
    "Origin": "https://gomechanic.in",
    "Referer": "https://gomechanic.in/",
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": "Windows",
    "Sec-Fetch-Dest": "empty",
    "Sec-Fetch-Mode": "cors",
    "Sec-Fetch-Site": "cross-site",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36"
}
page = requests.get(url, headers=header)
response = requests.get(url)
data = json.loads(response.text)

注意:授权 key 不断变化，因此请在运行代码之前更新它。您可以在检查元素的标题部分找到它。

截图供引用:

我想要响应部分的所有数据

Answer 1

您已经在 page = requests.get(url, headers=header) 获得了正确的输出并且您正在打印 response = requests.get(url) 你没有传递标题的地方。此处授权 key 也不会更改。

您可以使用以下代码获取套餐和价格。

op={}
page = requests.get(url, headers=header)
for d in page.json().get("data"):
    for service in d.get("services"):
        for package in service.get("package_details"):
                op[service.get("name")]=package.get("total")
print(op)

输出:

{'Basic Service': 3199, 'Standard Service': 4199, 'Comprehensive Service': 5499}