python - 无法使用 scrapy 或 requests 向 API 发送请求

Question

您好，我正在尝试向 API 发送发布请求，如下所示: https://www.cpaaustralia.com.au/api/FindACpa/SearchApi.mvc/ExecuteSearch

这是 JSON 数据:

{"Point":{"Latitude":"-27.67361480","Longitude":"153.18297950"},"LocationText":"4000 Pacific Hwy, Loganholme QLD 4129, Australia","PostCode":"4129","Locality":"Loganholme","CountryCode":"AU"}

这是 scrapy shell 请求:

req=Request("https://www.cpaaustralia.com.au/api/FindACpa/SearchApi.mvc/ExecuteSearch",method="POST",headers={"user-agent":"Mozilla 5.0"},body=json.dumps(data))

此请求应返回 JSON 数据作为响应，但未获取 json 响应。相反，会出现错误页面。

请告诉我解决方法

Answer 1

您已从“查找 CPA 页面”中抓取了 __RequestVerificationToken，并在发出 JSON 的 POST 请求时更新 header 中的 __RequestVerificationToken。

查找注册会计师页面:https://www.cpaaustralia.com.au/FindACpa/Locate.mvc/Index

import bs4
import requests

# Create Session.
session = requests.session()

# Get HTML of Find a CPA Page.
find_a_cpa_page_url = 'https://www.cpaaustralia.com.au/FindACpa/Locate.mvc/Index'
find_a_cpa_page_response = session.get(find_a_cpa_page_url)
print('Response status code for Find a CPA Page:', find_a_cpa_page_response.status_code)

# Fetch Request Verification Token HTML.
soup = bs4.BeautifulSoup(find_a_cpa_page_response.text, 'lxml')
request_verification_token = soup.find('input', {'name': '__RequestVerificationToken'}).get('value')

url = 'https://www.cpaaustralia.com.au/api/FindACpa/SearchApi.mvc/ExecuteSearch'
json_data = {"Point":{"Latitude":"-27.67361480","Longitude":"153.18297950"},"LocationText":"4000 Pacific Hwy, Loganholme QLD 4129, Australia","PostCode":"4129","Locality":"Loganholme","CountryCode":"AU"}

# Add Request Verification Token in headers.
headers = {
    'Host': 'www.cpaaustralia.com.au',
    'User-Agent': 'Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31',
    '__RequestVerificationToken': request_verification_token,
    'Referer': 'https://www.cpaaustralia.com.au/FindACpa/Locate.mvc/Index',
}

response = session.post(url, json=json_data, headers=headers)
print('Response status code for JSON:', response.status_code);print('JSON Data:', response.content)

输出:

Response status code for Find a CPA Page: 200
Response status code for JSON: 200
JSON Data: b'{"ErrorMessage":"","Results":[{"AfsLicensee":"","CompanyName":"C & K Accounting Services (Qld) Pty Ltd","CpaType":"PP","DistanceKm":0.177551037039086,"EmailAddress":"sking@candkaccounting.com.au","Fax":"","FormattedAddress":"Unit 4B, , 16 \xe2\x80\x93 18 Beenleigh Redland Bay Road  , Loganholme  , Queensland 4129, AUSTRALIA","Languages":"","Point":{"Latitude":-27.673364,"Longitude":153.184757},"Postcode":"4129","PractitionerName":"","Telephone":"617 3219 7811","WebsiteAddress":""},{"AfsLicensee":"","CompanyName":"Dynamic Accountants Pty Ltd","CpaType":"PP","DistanceKm":0.594803802705306,"EmailAddress":"info@dynamicaccountants.com.au","Fax":"07 3801 4773","FormattedAddress":"Suite 1, 3950 Pacific Hwy, LOGANHOLME, Queensland 4129, AUSTRALIA","Languages":"","Point":{"Latitude":-27.669489,"Longitude":153.179123},"Postcode":"4129","PractitionerName":"","Telephone":"07 3801 9555","WebsiteAddress":""},{"AfsLicensee":"","CompanyName":"Business Accounting Specialists","CpaType":"PP","DistanceKm":1.15412476831002,"EmailAddress":"nick@businessaccountingspecialists.com.au","Fax":"","FormattedAddress":"The Groves, Suite 11, 3986 Pacific Highway, LOGANHOLME, Queensland 4129, AUSTRALIA","Languages":"","Point":{"Latitude":-27.683700,"Longitude":153.185900},"Postcode":"4129","PractitionerName":"","Telephone":"07 3801 9555","WebsiteAddress":"http:\\/\\/www.businessaccountingspecialists.com.au"},{"AfsLicensee":"","CompanyName":"BMT Accounting &  Tax Pty Ltd","CpaType":"PP","DistanceKm":1.32102938225839,"EmailAddress":"info@bmtaccounting.com.au","Fax":"","FormattedAddress":"Suite 36, 37-59 Bryants Road, Loganholme, Queensland 4129, AUSTRALIA","Languages":"","Point":{"Latitude":-27.663443,"Longitude":153.175997},"Postcode":"4129","PractitionerName":"","Telephone":"07 3333 2350","WebsiteAddress":"http:\\/\\/www.bmtaccounting.com.au"},{"AfsLicensee":"","CompanyName":"KEF Accounting","CpaType":"PP","DistanceKm":2.86625470700257,"EmailAddress":"kef@kefaccounting.com.au","Fax":"","FormattedAddress":"5\\/66 Commercial Drive, SHAILER PARK, Queensland 4128, AUSTRALIA","Languages":"","Point":{"Latitude":-27.651880,"Longitude":153.167231},"Postcode":"4128","PractitionerName":"","Telephone":"07 3806 4920","WebsiteAddress":"http:\\/\\/www.kefaccounting.com.au"}]}'