python - POST 查询访问网站上的表数据 (Python)

Question

我想使用this website从中提取数据，但是该网站没有 API 来方便轻松提取数据。因此，网站上有一个包含数据的“表格”，但是该表格分布在多个页面上，并且下载选项不支持“批量”下载。所以我想找出一种通过Python下载这个的方法。
我发现我可以通过发布请求来做到这一点。我所做的是从第 1 页开始，然后单击“下一页”按钮并观看网络选项卡，这给了我:

Request URL:https://www.patricbrc.org/api/genome/
Request Method:POST
Status Code:200 OK
Remote Address:128.173.97.11:443
Referrer Policy:no-referrer-when-downgrade

Response Headers
Access-Control-Allow-Origin:https://www.patricbrc.org
Access-Control-Expose-Headers:facet_counts,x-facet-count,Content-Range,X-Content-Range,ETag
Content-Encoding:gzip
Content-Range:items 200-399/45999
Content-Type:application/json; charset=utf-8
Date:Sat, 26 Aug 2017 09:36:40 GMT
ETag:W/"8ac0b-cFdEBUwfdiyTQm/gpJHYzQ"
Server:nginx/1.9.1
Transfer-Encoding:chunked
Vary:Origin, Accept
Vary:Accept-Encoding
X-Powered-By:Express

Request Headers
Accept:application/javascript, application/json, application/json
Accept-Encoding:gzip, deflate, br
Accept-Language:nl-NL,nl;q=0.8,en-US;q=0.6,en;q=0.4
Connection:keep-alive
Content-Length:82
Content-Type:application/rqlquery+x-www-form-urlencoded
Cookie:sprod=hZ36nWTBiHHaDwdAhCChFGZo; _ga=GA1.2.582935779.1503602311; _gid=GA1.2.174755110.1503602311; _gat=1
Host:www.patricbrc.org
Origin:https://www.patricbrc.org
Range:items=200-399
Referer:https://www.patricbrc.org/view/Taxonomy/2
User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36
X-Range:items=200-399

Request Payload
eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score)
Name
genome/

为了查看页面切换后发生的变化，我再次执行了此操作(因此再次单击“下一页”按钮)。

Request URL:https://www.patricbrc.org/api/genome/
Request Method:POST
Status Code:200 OK
Remote Address:128.173.97.11:443
Referrer Policy:no-referrer-when-downgrade

Response Headers
Access-Control-Allow-Origin:https://www.patricbrc.org
Access-Control-Expose-Headers:facet_counts,x-facet-count,Content-Range,X-Content-Range,ETag
Content-Encoding:gzip
Content-Range:items 400-599/45999
Content-Type:application/json; charset=utf-8
Date:Sat, 26 Aug 2017 09:44:24 GMT
ETag:W/"5dc73-ODopjbbyl5M2vUJvUGN0Gw"
Server:nginx/1.9.1
Transfer-Encoding:chunked
Vary:Origin, Accept
Vary:Accept-Encoding
X-Powered-By:Express

Request Headers
Accept:application/javascript, application/json, application/json
Accept-Encoding:gzip, deflate, br
Accept-Language:nl-NL,nl;q=0.8,en-US;q=0.6,en;q=0.4
Connection:keep-alive
Content-Length:82
Content-Type:application/rqlquery+x-www-form-urlencoded
Cookie:sprod=hZ36nWTBiHHaDwdAhCChFGZo; _ga=GA1.2.582935779.1503602311; _gid=GA1.2.174755110.1503602311
Host:www.patricbrc.org
Origin:https://www.patricbrc.org
Range:items=400-599
Referer:https://www.patricbrc.org/view/Taxonomy/2
User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36
X-Range:items=400-599

Request Payload
eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score)

因此，在发布请求期间，从页面 1 --> 2 和页面 2 --> 3 发生变化的重要事情(在我看来)是:

• 响应 header 内容范围:项目 200-399/45999 --> 内容范围:项目 400-599/45999
• 请求 header 范围:items=200-399 --> 范围:items=400-599
• 请求 header X-范围:items=200-399 --> X-范围:items=400-599

网站是这样说的:

• 第 1 页:45999 个结果中的 1 - 200 个(但 0-199 的帖子，上面未显示)
• 第 2 页:45999 个结果中的 201 - 400 个(但帖子为 200-399 个，请参见上文)
• 第 3 页:45999 个结果中的第 401 - 600 个(但帖子为 400-599，请参见上文)

我对阅读这些 POST 请求的知识零。但我想应该有一种说法:

requests.post(some url) and extract the total page numbers (or number of genomes and devide these by the max number per page, which is 200 see post requests). 

然后是这样的:

for page_numb in range(page_numbers):
    r = requests.post(some_url...... + page_numb)
    #extract the information from the table

终于有问题了:)我无法弄清楚如何设置像

这样的 POST 请求

p1 = requests.post(https://www.patricbrc.org/api/genome/... 0-199)
p2 = requests.post(https://www.patricbrc.org/api/genome/... 200-399)
etc...

(我希望响应将包含由某些分隔符分隔的表数据)

<小时/>

我尝试过的代码:

import requests
import json

url = 'https://www.patricbrc.org/api/genome/'
payload = {"eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score)"}
headers = {
    "X-Range": "items=0-199",
    "Range": "items=0-199",
    "content-type": "application/json"}


r = requests.post(url, data=json.dumps(payload), headers=headers)
print(r)

这段代码不起作用，但我想它应该是这样的。
我知道我应该将有效负载作为 JSON 发布，但我不知道如何翻译 eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score)转换成 JSON，它是键:值对。

Answer 1

这是一个源代码(其丑陋的代码，但这只是示例，我建议您使用request.Session):

import sys
import json
import requests

URL = 'https://www.patricbrc.org/view/Taxonomy/2'
API = 'https://www.patricbrc.org/api/genome/'

# we need to get a Cookie
response = requests.get(URL)
if response.status_code != 200:
    sys.exit('Cookie request failed')
cookie = response.headers['Set-Cookie'].split(';')[0]
etag   = response.headers['ETag']

# now with cookie request a data
headers = {
    'Host'            : 'www.patricbrc.org',
    'Accept'          : 'application/javascript, application/json, application/json',
    'Accept-Language' : 'en-US,en;q=0.5',
    'Referer'         : 'https://www.patricbrc.org/view/Taxonomy/2',
    'X-Range'         : 'items=200-399',  # your pages
    'Range'           : 'items=200-399',  # your pages
    'Content-Type'    : 'application/rqlquery+x-www-form-urlencoded',
    'Cookie'          : '{};'.format(cookie),
    'ETag'            : etag
}

data = 'eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score)'
response = requests.post(API, headers=headers, data=data)
if response.status_code != 200:
    sys.exit('API request failed')
data = json.loads(response.text)

# test data
row200 = data[0]
print 'FAMILY     :', row200['family']
print 'GC_CONTENT :', row200['gc_content'] 
print 'GENUS      :', row200['genus']
print 'ORDER      :', row200['order']

测试数据 block 的输出:

FAMILY     : Eubacteriaceae
GC_CONTENT : 38.78
GENUS      : Eubacterium
ORDER      : Clostridiales

编辑您不需要 cookie，这也可以正常工作:

import json
import requests

API = 'https://www.patricbrc.org/api/genome/'

# now with cookie request a data
headers = {
    'Host'            : 'www.patricbrc.org',
    'Accept'          : 'application/javascript, application/json, application/json',
    'Accept-Language' : 'en-US,en;q=0.5',
    'Referer'         : 'https://www.patricbrc.org/view/Taxonomy/2',
    'X-Range'         : 'items=200-399',  # your pages
    'Range'           : 'items=200-399',  # your pages
    'Content-Type'    : 'application/rqlquery+x-www-form-urlencoded',
}

query = 'eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score)'
response = requests.post(API, headers=headers, data=query)
data = json.loads(response.text)