python - 如何在 Python 中使用请求将正文包含到 POST 请求中

Question

我需要从页面中提取所有游轮。我发现我需要提出2个请求。一个用于获取总结果 + 引用个别邮轮，另一个用于邮轮本身。到目前为止，我成功发出了请求并获得了 JSON。问题是该请求仅返回第一页中包含的引用。最初我用的是这个:

https://www.pocruises.com.au/sc_ignore/b2c/cruiseresults/searchresults

作为 POST 请求 URL。我是请求中的一些 header :

Host: www.pocruises.com.au
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0
Accept: application/json, text/plain, */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://www.pocruises.com.au/cruises/search
Content-Type: application/json;charset=utf-8
Content-Length: 613
Cookie: ASP.NET_SessionId=zm50ahxa4uoiwcuowgkeizcn; SC_ANALYTICS_GLOBAL_COOKIE=f459da6904ee4cea8a809455f37b09c5|False; optimizelyEndUserId=oeu1480435687102r0.9194470051495017; optimizelySegments=%7B%223906442756%22%3A%22none%22%2C%223911484226%22%3A%22ff%22%2C%223917223299%22%3A%22direct%22%2C%223920524266%22%3A%22false%22%2C%224924982297%22%3A%22true%22%7D; optimizelyBuckets=%7B%227883701652%22%3A%227867701359%22%7D; _ga=GA1.3.1128287966.1480435688; AdBlockDetected=false; _msuuid_29439mm27589=D0670AB5-C96A-4706-A563-9F29FCA3D9D2; gwcc=%7B%22fallback%22%3A%221300159454%22%2C%22clabel%22%3A%22ykseCIGywFkQhoDuxQM%22%2C%22backoff%22%3A86400%2C%22backoff_expires%22%3A1480522087%7D; gaFindACruise=; _gat=1; optimizelyPendingLogEvents=%5B%22n%3Dengagement%26u%3Doeu1480435687102r0.9194470051495017%26wxhr%3Dtrue%26time%3D1480437791.708%26f%3D7274530066%2C7883701652%26g%3D3909534788%22%5D
Connection: keep-alive

到目前为止，如果不算我返回的仅有 6 个结果，它仍然有效。在这篇文章发表时他们一定已经 106 岁了。响应包含包含总计数(正确)和页码、总页数等的元数据。然后我看到了这个: Firefox Screenshot此请求正文包含我在所有页面中导航所需的所有内容。这是我到目前为止的全部代码:

import requests

url = "https://www.pocruises.com.au/sc_ignore/b2c/cruiseresults/searchresults"
session = requests.session()
data = {"searchParameters": {"p": [], "c": [], "d": [], "s": [], "ms": [], "adv": [], "sort": "dpa", "page": 5},
        "renderingParameters": {"DefaultSortOption": "dpa", "LargeScreenFlag": "true", "NewModelIsLoading": "false",
                                "PagingAnchor": "", "ViewStyleSelectorVisible": "true", "FilterBarVisible": "true",
                                "NumberOfResultsAndSortByPanelVisible": "true", "DefaultResultsView": "Grid",
                                "MaxNumberOfResults": 0, "PaginationEnabled": "true", "KeepPageState": "true",
                                "PageSize": 9, "DefaultResultsGrouping": "Itinerary", "Duration": [],
                                "CruiseItinerary": [], "Voyage": [], "ExcludeVoyage": [], "PromoCode": [],
                                "AdditionalPromoCodes": []}}
headers = {"Host": "www.pocruises.com.au",
           "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0",
           "Accept": "application/json, text/plain, */*", "Accept-Language": "en-US,en;q=0.5",
           "Content-Type": "application/json;charset=utf-8",
           "Cookie": "ASP.NET_SessionId=zm50ahxa4uoiwcuowgkeizcn; "
                     "SC_ANALYTICS_GLOBAL_COOKIE=f459da6904ee4cea8a809455f37b09c5|False; "
                     "optimizelyEndUserId=oeu1480435687102r0.9194470051495017; "
                     "optimizelySegments=%7B%223906442756%22%3A%22none%22%2C%223911484226%22%3A%22ff%22%2C"
                     "%223917223299%22%3A%22direct%22%2C%223920524266%22%3A%22false%22%2C%224924982297%22%3A%22true"
                     "%22%7D; optimizelyBuckets=%7B%227883701652%22%3A%227867701359%22%7D; "
                     "_ga=GA1.3.1128287966.1480435688; AdBlockDetected=false; "
                     "_msuuid_29439mm27589=D0670AB5-C96A-4706-A563-9F29FCA3D9D2; "
                     "gwcc=%7B%22fallback%22%3A%221300159454%22%2C%22clabel%22%3A%22ykseCIGywFkQhoDuxQM%22%2C"
                     "%22backoff%22%3A86400%2C%22backoff_expires%22%3A1480522087%7D; gaFindACruise=; _gat=1; "
                     "optimizelyPendingLogEvents=%5B%5D",
           "Connection": "keep-alive"}
session.headers.update(headers)
page = session.post(url, headers=headers, data=data)
cruise_data = page.json()
print(cruise_data)



session.headers.update(headers)
page = session.post(url, headers=headers, data=data)
cruise_data = page.json()

但现在我收到此错误:

Traceback (most recent call last):
  File "/home/fixxxer/PycharmProjects/POCruses/main.py", line 31, in <module>
    cruise_data = page.json()
  File "/usr/lib/python3.5/site-packages/requests/models.py", line 841, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.5/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

我想我的body部分做错了。如何在请求中添加此正文？编辑:如果我删除 data 参数，请求工作正常，但信息不是我需要的。如果我用浏览器打开页面并检查发送到浏览器的响应，我会得到以下信息: ScreenShot即使这与我在没有 body 的情况下得到的结果也不同。这就是我知道这个 body 是重要信息的方式。

Answer 1

该页面需要的数据为 JSON所以你需要

page = session.post(url, json=data)

---

顺便说一句:

服务器总是将新的 cookie 分配给新的客户端 - 特别是 cookie 行 ASP.NET_SessionId - 更好GET在执行其他请求之前获取新 cookie 的主页。

session.headers.update(headers)之后你不必使用headers=headers在get()/post() 。如果某些请求需要一些额外的 header ，您可能只需更改一些 header 。

如果您使用json=然后它会自动添加标题 "Content-Type": "application/json;charset=utf-8"

完整代码可能如下所示:

import requests

# -- create session ---

session = requests.session()

headers = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0",
    "Accept": "application/json, text/plain, */*",
    "Accept-Language": "en-US,en;q=0.5",
    "Connection": "keep-alive"
}

# set headers for all requests
session.headers.update(headers)

# --- get cookies ---

url = "https://www.pocruises.com.au/"
page = session.get(url)

# --- search ---

data = {
    "searchParameters": {
        "p": [],
        "c": [],
        "d": [],
        "s": [],
        "ms": [],
        "adv": [],
        "sort": "dpa",
        "page": 5
    },
    "renderingParameters": {
        "DefaultSortOption": "dpa",
        "LargeScreenFlag": "true",
        "NewModelIsLoading": "false",
        "PagingAnchor": "",
        "ViewStyleSelectorVisible": "true",
        "FilterBarVisible": "true",
        "NumberOfResultsAndSortByPanelVisible": "true",
        "DefaultResultsView": "Grid",
        "MaxNumberOfResults": 0,
        "PaginationEnabled": "true",
        "KeepPageState": "true",
        "PageSize": 9,
        "DefaultResultsGrouping": "Itinerary",
        "Duration": [],
        "CruiseItinerary": [],
        "Voyage": [],
        "ExcludeVoyage": [],
        "PromoCode": [],
        "AdditionalPromoCodes": []
    }
}

url = "https://www.pocruises.com.au/sc_ignore/b2c/cruiseresults/searchresults"

# `json=` add `"Content-Type": "application/json;charset=utf-8"`

page = session.post(url, json=data)

print(page.text)
cruise_data = page.json()
print(cruise_data)