python - 如何使用python请求正确形成对该网站的POST请求

Question

我想要发送帖子请求的网址是 http://www.hkexnews.hk/sdw/search/searchsdw.aspx

我想要(手动)进行的搜索只需在“股票代码”中输入“1”，然后单击“搜索”

我已经多次尝试使用 python 和 chrome 扩展“Postman”发送带有以下 header 的 post 请求:

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: zh-TW,zh;q=0.9,en-US;q=0.8,en;q=0.7
Cache-Control: max-age=0
Connection: keep-alive
Content-Length: 1844
Content-Type: application/x-www-form-urlencoded
Cookie: TS0161f2e5=017038eb490da17e158ec558c902f520903c36fad91e96a3b9ca79b098f2d191e3cac56652
Host: www.hkexnews.hk
Origin: http://www.hkexnews.hk
Referer: http://www.hkexnews.hk/sdw/search/searchsdw.aspx
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.62 Safari/537.36

以及以下参数:

today: 20180624
sortBy: 
selPartID: 
alertMsg: 
ddlShareholdingDay: 23
ddlShareholdingMonth: 06
ddlShareholdingYear: 2018
txtStockCode: 00001
txtStockName: 
txtParticipantID: 
txtParticipantName: 
btnSearch.x: 35
btnSearch.y: 8

但它不起作用。

Answer 1

尝试以下方法。它应该为您获取所需的响应以及根据搜索条件生成的该网站中可用的表格数据。

import requests
from bs4 import BeautifulSoup

URL = "http://www.hkexnews.hk/sdw/search/searchsdw.aspx"

with requests.Session() as s:
    s.headers={"User-Agent":"Mozilla/5.0"}
    res = s.get(URL)
    soup = BeautifulSoup(res.text,"lxml")
    payload = {item['name']:item.get('value','') for item in soup.select("input[name]")}
    payload['__EVENTTARGET'] = 'btnSearch'
    payload['txtStockCode'] = '00001'
    payload['txtParticipantID'] = 'A00001'
    req = s.post(URL,data=payload,headers={"User-Agent":"Mozilla/5.0"})
    soup_obj = BeautifulSoup(req.text,"lxml")
    for items in soup_obj.select("#pnlResultSummary .ccass-search-datarow"):
        data = [item.get_text(strip=True) for item in items.select("div")]
        print(data)