python - 使用 Python 请求提交表单数据 POST 不起作用

Question

我正在编写一个 python 脚本来自动检查狗的狗重新归巢站点，当它们可用时我们可能能够采用它们，但是我被困在这个站点上完成表单数据并且无法弄清楚原因。
表单属性声明它应该有一个 post 方法，我已经浏览了表单的所有输入并创建了一个有效负载。
我希望返回带有搜索结果的页面，并从结果页面抓取 html，以便我可以开始处理它，但抓取只是表单页面，永远不会有结果。
我尝试使用 .get 与有效载荷作为参数，使用有效载荷的 url 并使用 requests-html 库来呈现任何 java 脚本元素，但没有成功。
如果您将 url_w_payload 粘贴到浏览器中，它会加载页面并显示其中一个字段为空。如果您然后再次在 url 栏中按 Enter 以重新加载页面而不修改它加载的 url... 可能与 cookie 有关系吗？

import requests
from requests_html import HTMLSession

session = HTMLSession()

form_url =  "https://www.rspca.org.uk/findapet?p_p_id=petSearch2016_WAR_ptlPetRehomingPortlets&p_p_lifecycle=1&p_p_state=normal&p_p_mode=view&_petSearch2016_WAR_ptlPetRehomingPortlets_action=search"

url_w_payload = "https://www.rspca.org.uk/findapet?p_p_id=petSearch2016_WAR_ptlPetRehomingPortlets&p_p_lifecycle=1&p_p_state=normal&p_p_mode=view&_petSearch2016_WAR_ptlPetRehomingPortlets_action=search&noPageView=false&animalType=DOG&freshSearch=false&arrivalSort=false&previousAnimalType=&location=WC2N5DU&previousLocation=&prevSearchedPostcode=&postcode=WC2N5DU&searchedLongitude=-0.1282688&searchedLatitude=51.5072106"

payload = {'noPageView': 'false','animalType': 'DOG', 'freshSearch': 'false', 'arrivalSort': 'false', 'previousAnimalType': '', 'location': 'WC2N5DU', 'previousLocation': '','prevSearchedPostcode': '', 'postcode': 'WC2N5DU', 'searchedLongitude': '-0.1282688',  'searchedLatitude': '51.5072106'}

#req = requests.post(form_url, data = payload)

#with open("requests_output.txt", "w") as f:
 #   f.write(req.text)

ses = session.post(form_url, data = payload)

ses.html.render()

with open("session_output.txt", "w") as f:
    f.write(ses.text)

print("Done")

Answer 1

使用 cookie 和 headers 有一些障碍，但是一旦你做对了，你就会得到正确的回应。
这是如何做到的:

import time
from urllib.parse import urlencode

import requests
from bs4 import BeautifulSoup

query_string = {
    "p_p_id": "petSearch2016_WAR_ptlPetRehomingPortlets",
    "p_p_lifecycle": 1,
    "p_p_state": "normal",
    "p_p_mode": "view",
    "_petSearch2016_WAR_ptlPetRehomingPortlets_action": "search",
}

payload = {
    'noPageView': 'false',
    'animalType': 'DOG',
    'freshSearch': 'false',
    'arrivalSort': 'false',
    'previousAnimalType': '',
    'location': 'WC2N5DU',
    'previousLocation': '',
    'prevSearchedPostcode': '',
    'postcode': 'WC2N5DU',
    'searchedLongitude': '-0.1282688',
    'searchedLatitude': '51.5072106',
}


def make_cookies(cookie_dict: dict) -> str:
    return "; ".join(f"{k}={v}" for k, v in cookie_dict.items())


with requests.Session() as connection:
    main_url = "https://www.rspca.org.uk"
    
    connection.headers["User-Agent"] = "Mozilla/5.0 (X11; Linux x86_64) " \
                                       "AppleWebKit/537.36 (KHTML, like Gecko) " \
                                       "Chrome/90.0.4430.212 Safari/537.36"
    r = connection.get(main_url)
    
    cookies = make_cookies(r.cookies.get_dict())
    additional_string = f"; cb-enabled=enabled; " \
                        f"LFR_SESSION_STATE_10110={int(time.time())}"
    
    post_url = f"https://www.rspca.org.uk/findapet?{urlencode(query_string)}"
    connection.headers.update(
        {
            "cookie": cookies + additional_string,
            "referer": post_url,
            "content-type": "application/x-www-form-urlencoded",
        }
    )
    response = connection.post(post_url, data=urlencode(payload)).text
    dogs = BeautifulSoup(response, "lxml").find_all("a", class_="detailLink")
    print("\n".join(f"{main_url}{dog['href']}" for dog in dogs))

输出(为简洁起见缩短，无需在所有狗都响应时对页面进行分页):

https://www.rspca.org.uk/findapet/details/-/Animal/JAY_JAY/ref/217747/rehome/
https://www.rspca.org.uk/findapet/details/-/Animal/STORM/ref/217054/rehome/
https://www.rspca.org.uk/findapet/details/-/Animal/DASHER/ref/205702/rehome/
https://www.rspca.org.uk/findapet/details/-/Animal/EVE/ref/205701/rehome/
https://www.rspca.org.uk/findapet/details/-/Animal/SEBASTIAN/ref/178975/rehome/
https://www.rspca.org.uk/findapet/details/-/Animal/FIJI/ref/169578/rehome/
https://www.rspca.org.uk/findapet/details/-/Animal/ELLA/ref/154419/rehome/
https://www.rspca.org.uk/findapet/details/-/Animal/BEN/ref/217605/rehome/
https://www.rspca.org.uk/findapet/details/-/Animal/SNOWY/ref/214416/rehome/
https://www.rspca.org.uk/findapet/details/-/Animal/BENSON/ref/215141/rehome/
https://www.rspca.org.uk/findapet/details/-/Animal/BELLA/ref/207716/rehome/

and much more ...

附注。我真的很喜欢这个挑战，因为我有两只来自收容所的狗。坚持下去，伙计!