javascript - 从 JavaScript 检索 post 请求并将其与 cURL/RCurl 一起使用

Question

我正在尝试阅读页面 investing.com/rates-bonds/brazil-10当我通过主表上的日历控件更改默认时间序列范围时，使用 RCurl 。

使用 Firefox Firebug 日志选项，我在单击日历的 -> Apply 按钮时获得了帖子日志。 (Chrome 中的 Log XMLHTTPRequests 可能会出现类似的情况)

具体来说，单击按钮后，Firebug 控制台会显示以下行

 POST http://www.investing.com/instruments/HistoricalDataAjax

展开该行，“ header ”选项卡会提供此请求 header :

Request Headers
Accept             text/plain, */*; q=0.01
Accept-Encoding    gzip, deflate
Accept-Language    en-gb,en;q=0.5
Content-Length     102
Content-Type       application/x-www-form-urlencoded; charset=UTF-8
Cookie             ... very long string
Host               www.investing.com
Referer            http://www.investing.com/rates-bonds/brazil-10-year-bond-yield-historical-data
User-Agent         Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0
X-Requested-With   XMLHttpRequest

“帖子”选项卡提供:

Parameters application/x-www-form-urlencoded
action        historical_data
curr_id       24029
end_date      05/30/2014
interval_sec  Daily
st_date       05/25/2014

鉴于此，我想我可能会通过以下方式获得该页面:

require(RCurl)
link="http://www.investing.com/instruments/HistoricalDataAjax"
html= postForm(
    link,
    action="historical_data",
    curr_id="24029",
    end_date="05/30/2014",
    interval_sec="Daily",
    st_date="05/25/2014",
    .opts=list(
        referer="http://www.investing.com/rates-bonds/brazil-10-year-bond-yield-historical-data",
        useragent="Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0"
))

但我得到:

html
## [1] ""
## attr(,"Content-Type")
##                 charset 
## "text/html"     "utf-8" 

我也尝试了很长的方式:...

cookie="...  long cookie string"
link="http://www.investing.com/instruments/HistoricalDataAjax"
h <- basicTextGatherer()
h$reset()
curlPerform(url = link,
            httpheader=c(                
                'Accept'="text/plain, */*; q=0.01",
                'Accept-Encoding'="gzip, deflate",
                'Accept-Language'="en-gb,en;q=0.5",
                'Content-Length'="102",
                'Content-Type'="application/x-www-form-urlencoded; charset=UTF-8",
                'Cookie'=cookie,
                'Host'="www.investing.com",
                'Referer'="http://www.investing.com/rates-bonds/brazil-10-year-bond-yield-historical-data",
                'User-Agent'="Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0",
                'X-Requested-With'="XMLHttpRequest"
                ),
            postfields=c(
                action="historical_data",
                curr_id="24029",
                end_date="05/30/2014",
                interval_sec="Daily",
                st_date="05/25/2014"
                ),
            writefunction = h$update,
            verbose = TRUE)
h$value()

有什么帮助吗？

Answer 1

关键似乎是设置 user-agent、origin 和 x-requested-with header :

require(RCurl)
url  <- "http://www.investing.com/instruments/HistoricalDataAjax"

POST(url, 
  body = list(
    action = "historical_data",
    curr_id = "24029",
    end_date = "05/30/2014",
    interval_sec = "Daily",
    st_date = "05/25/2014"
  ), 
  add_headers(
    "user-agent" = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36",
    Origin = "http://www.investing.com",
    "X-Requested-With" = "XMLHttpRequest"
  ),
  multipart = FALSE
)

(我使用 httr 是因为我认为 RCurl::postForm() 很难理解 POST 请求中实际发生的内容)。