python - HTTPS POST 使用 python 请求访问目录

Question

我们发现 HTTPS FORM 访问 JPL spectroscopiccatalog 时存在问题。使用python requests 库。 HTTPS 表单对所有查询返回相同的响应:为您的搜索条件找到了零行。

  r = requests.post('https://spec.jpl.nasa.gov/cgi-bin/catform',
data={'Mol' :'18003+H2O', 'MinNu':"500", 'MaxNu':"600",
         'MaxLines': '2000', 'UnitNu':'GHz', 'StrLim': "-500"})
  print(r.text)
  Out: 'Zero lines were found for your search criteria.\n'
  print(r.status_code)
  Out: 200

但是查询表单 https://spec.jpl.nasa.gov/ftp/pub/catalog/catform.html使用浏览器(Firefox 60.0.1)请求如下所示(重定向到请求 URL https://spec.jpl.nasa.gov/cgi-bin/catform ):

POST /cgi-bin/catform HTTP/1.1
Host: spec.jpl.nasa.gov
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Content-Length: 70
Content-Type: application/x-www-form-urlencoded
Upgrade-Insecure-Requests: 1
Connection: keep-alive

MinNu=500
MaxNu=600
MaxLines=2000
UnitNu=GHz
StrLim=-500
Mol=18003+H2O

给出以下响应:

  18003  H2O          
  503568.5200  0.0200 -4.9916 3 1394.8142 51 -180031404 8 6 3 0     7 7 0 0     
  504482.6900  0.0500 -5.4671 3 1394.8142 17 -180031404 8 6 2 0     7 7 1 0     
  525890.1638  0.8432-12.2048 3 5035.1266117  18003140419 514 0    18 811 0     
  530342.8600  0.2000 -7.1006 3 2533.7932 87 -18003140414 312 0    13 4 9 0     
  534240.4544  0.3469-11.2954 3 4409.3446 37  18003140418 414 0    17 711 0     
  556935.9877  0.0003 -0.8189 3   23.7944  9 -180031404 1 1 0 0     1 0 1 0     
  557985.4794  0.6432-11.6213 3 4833.2084117  18003140419 415 0    18 712 0     
  558017.0036 12.4193-18.1025 3 7729.4622 49  18003140424 618 0    25 521 0     
  571913.6860  0.1000 -6.9705 3 2414.7235 75 -18003140412 6 7 0    13 310 0     
  591693.4339  0.2120 -8.6820 3 3244.6008 87  18003140414 7 8 0    15 411 0     
  593113.7249  7.4502-18.5975 3 7924.4438 49  18003140424 717 0    231014 0     
  593227.8163  0.4197-10.8822 3 4201.2514 35  18003140417 612 0    18 315 0     
  596308.5878  4.5348-15.8345 3 6687.8251 47  18003140423 519 0    22 616 0     

在请求 POST 调用中发送相同的 header 也没有帮助。

headers = {
'Host': 'spec.jpl.nasa.gov',
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language': 'en-US,en;q=0.5',
'Accept-Encoding': 'gzip, deflate, br',
'Content-Type': 'application/x-www-form-urlencoded',
'Upgrade-Insecure-Requests': '1',
'Connection': 'keep-alive',
}

data = {
'MaxLines': '2000',
'MaxNu': '600',
'MinNu': '500',
'Mol': '18003+H2O',
'StrLim': '-500',
'UnitNu': 'GHz',
}

r = requests.post('https://spec.jpl.nasa.gov/cgi-bin/catform', headers=headers,
          data=data)

print(r.text)

Out: 'Zero lines were found for your search criteria.\n'

存在从 HTTP 版本的表单到 HTTPS 版本的重定向但似乎这并没有引起问题。知道为什么代码不提供预期结果吗？

Answer 1

查看该网站后，表单发布使用 HTTP 而不是 HTTPS。如果您使用 HTTPS 发帖，他们的网站似乎无法正确响应。

这应该有效:

import requests

data = {
    'MaxLines': '2000',
    'MaxNu': '600',
    'MinNu': '500',
    'Mol': '18003+H2O',
    'StrLim': '-500',
    'UnitNu': 'GHz',
}

r = requests.post('http://spec.jpl.nasa.gov/cgi-bin/catform', data=data)
print(r.status_code)
print(r.text.split('\n')[:10])


Out: ['<OPTION>1001  H-atom        ',
 '<OPTION>2001  D-atom        ',
 '<OPTION>3001  HD            ',
 '<OPTION>4001  H2D+          ',
 '<OPTION>7001  Li-6-H        ',
 '<OPTION>8001  LiH           ',
 '<OPTION>8002  Li-6-D        ',
 '<OPTION>9001  LiD           ',
 '<OPTION>12001  C-atom        ',
 '<OPTION>13001  C-13-atom     ']