Python XHR 请求格式

Question

我正在尝试使用 AJAX 从网站获取信息。当我使用 Chrome 的 REST 插件时，我可以发送请求并接收所需的答案。我想编写一个 python 脚本，但我不知道如何以正确的格式编写请求。目前我收到未经授权的页面响应，我认为这是由于未使用 XHR 发送它造成的？

代码:

import json
import requests

payload = {"stores":"3650","products":{"31445761":[{"sku":"6000197536050","upc":["4549659075"]}]},"origin":"pip","csrfToken":"d707af2ed8b79a78a669b38dff593c909f6b6262-1507764346644-ebc72845dfa30431a8f7b1c4"}

with requests.Session() as session:
    session.get("https://www.walmart.ca")
    r = session.post('https://www.walmart.ca/ws/en/products/availability', data=json.dumps(payload),
                 headers={"Content-Type": "application/x-www-form-urlencoded; charset=UTF-8"})

    print(r.content) 

当我在 REST Chrome 扩展中时，我输入:

stores=["3650"]&products={"31445761":[{"sku":"6000197536050","upc":["4549659075"]}]}&origin=pip&csrfToken=d707af2ed8b79a78a669b38dff593c909f6b6262-1507764346644-ebc72845dfa30431a8f7b1c4

这给了我想要的结果。

期望的输出:

{
31445761: {
"online": [
  {
"maxRegularPrice": 0,
"minRegularPrice": 0,
"mapPrice": 0,
"minCurrentPrice": 99.96,
"maxCurrentPrice": 99.96,
"inventory": 0,
"sku": "6000197536050",
"clearance": false,
"offerId": "6000197536050",
"limited": false,
"reducedPrice": false,
"offerType": "1P",
"limitedStock": false,
"sellerId": "0",
"rollback": false,
"date": "",
"status": "OutOfStock",
"eligible": false,
"sellerName": "Walmart",
"asAdvertised": false
}
],
"stores": [
  {
"minRegularPrice": 0,
"maxRegularPrice": 0,
"minCurrentPrice": 99.96,
"maxCurrentPrice": 99.96,
"inventory": 0,
"sku": "6000197536050",
"clearance": false,
"limited": false,
"limitedStock": false,
"rollback": false,
"date": "",
"status": "OutOfStock",
"storeNumber": "3650",
"eligible": false,
"asAdvertised": false
}
],
"onlineSummary": {
"status": "OutOfStock",
"date": "",
"eligible": false,
"clearance": false,
"rollback": false,
"asAdvertised": false,
"limited": false,
"limitedStock": false,
"minRegularPrice": 0,
"maxRegularPrice": 0,
"minCurrentPrice": 99.96,
"maxCurrentPrice": 99.96
},
"storeSummary": {
"status": "OutOfStock",
"date": "",
"eligible": false,
"clearance": false,
"rollback": false,
"asAdvertised": false,
"limited": false,
"limitedStock": false,
"minRegularPrice": 0,
"maxRegularPrice": 0,
"minCurrentPrice": 99.96,
"maxCurrentPrice": 99.96
}
}
}`

Answer 1

您只是缺少一个 header :

'X-Requested-With': 'XMLHttpRequest'

更新的代码:

import json
import requests

payload = {"stores":"3650","products":{"31445761":[{"sku":"6000197536050","upc":["4549659075"]}]},"origin":"pip","csrfToken":"d707af2ed8b79a78a669b38dff593c909f6b6262-1507764346644-ebc72845dfa30431a8f7b1c4"}

with requests.Session() as session:
    session.get("https://www.walmart.ca")
    r = session.post('https://www.walmart.ca/ws/en/products/availability', data=json.dumps(payload),
                 headers={"Content-Type": "application/x-www-form-urlencoded; charset=UTF-8", 'X-Requested-With': 'XMLHttpRequest'})

    print(r.content) 

此外，CSRF token 通常会发生变化。因此，您应该检索 HTML，获取 CSRF token ，然后执行 XHR POST 请求。

我会推荐 beautifulsoup 在 HTML 中查找 CSRF token 。