ruby - 使用 Typhoeus 发出请求，某些 URL 始终返回空的 '400' 响应

Question

某些 URL 总是返回一个空的 HTTP 400 响应(许多其他 URL 工作正常)。示例:

1.9.3p0 :002 > e = Typhoeus::Easy.new

e.url = "http://us.asos.com/NW3/NW3-Highgate-Seersucker-Pinafore-Dress/Prod/pgeproduct.aspx?iid=1988231&cid=8799&sh=0&pge=0&pgesize=20&sort=-1&clr=Navy%252fivory&r=2"

1.9.3p0 :004 > e.perform => 400

1.9.3p0 :005 > y e
--- !ruby/object:Typhoeus::Easy 
curl_return_code: 0
headers: {}

method: :get
response_body: ""
response_header: |
  HTTP/1.1 400 Bad Request
  Content-Type: text/html
  Server: Microsoft-IIS/7.0
  Content-Length: 0
  Vary: Accept-Encoding
  Date: Mon, 07 May 2012 19:35:28 GMT
  Connection: keep-alive

ssl_version: :default
url: http://us.asos.com/NW3/NW3-Highgate-Seersucker-Pinafore-Dress/Prod/pgeproduct.aspx?iid=1988231&cid=8799&sh=0&pge=0&pgesize=20&sort=-1&clr=Navy%252fivory&r=2 => nil 

相反，在同一台机器上，几秒钟后它通过命令行 cURL 工作:

james@laptop:~/fashism$ curl -I "http://us.asos.com/NW3/NW3-Highgate-Seersucker-Pinafore-Dress/Prod/pgeproduct.aspx?iid=1988231&cid=8799&sh=0&pge=0&pgesize=20&sort=-1&clr=Navy%252fivory&r=2"
HTTP/1.1 200 OK
Cache-Control: private
Content-Type: text/html;charset=utf-8
Server: Microsoft-IIS/7.0
X-AspNet-Version: 2.0.50727
X-RuleEngine-Key: US1.0:ReferrerAnyOrNoneSiteDotUSPreferenceUSOrNone
X-Powered-By: ASP.NET
Date: Mon, 07 May 2012 19:54:03 GMT
Connection: keep-alive
Set-Cookie: AsosExecutionEngine=ExemptionTimeout=05/07/2012 20:14; expires=Tue, 07-May-2013 19:54:02 GMT; path=/
Set-Cookie: ASP.NET_SessionId=sd2hw4yu4l4jwmevc3gyan55; path=/; httponly
Set-Cookie: asos=topcatidHitCount=1; expires=Tue, 07-May-2013 19:54:02 GMT; path=/
Set-Cookie: asos=topcatidHitCount=1&currencyid=1&topcatid=9057&customerguid=b5da62eb44de45eeadd2f36ec90332ca; expires=Tue, 07-May-2013 19:54:02 GMT; path=/
Set-Cookie: asosbasket=basketitemcount=0&basketitemtotalretailprice=0; expires=Mon, 07-May-2012 20:54:02 GMT; path=/
Set-Cookie: stop_mobi=yes; path=/

james@laptop:~/fashism$ 

有什么想法吗？将不胜感激，即使它只是给我一个前进的方向，找到一个解决方案。

非常感谢!

~詹姆斯

Answer 1

来 self 在 Typhoeus github issue 上的回答:

服务器需要设置一些请求头。

尝试:

url = "http://us.asos.com/NW3/NW3-Highgate-Seersucker-Pinafore-Dress/Prod/pgeproduct.aspx?iid=1988231&cid=8799&sh=0&pge=0&pgesize=20&sort=-1&clr=Navy%252fivory&r=2"
header = {"Accept" => "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Cache-Control" => "no-cache", "Pragma" => "no-cache", "User-Agent" => "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.168 Safari/535.19"}
response = Typhoeus::Request.get(url, headers: header, verbose: true, follow_location: true)