javascript - 从网站获取请求并检索响应？

Question

我正在尝试监控一个网站 ( www.bidcactus.com )。在网站上，我打开 Firebug，转到网络选项卡，然后单击 XHR 选项卡。

我想获取请求的响应并将其保存到 mySql 数据库(我的计算机上运行了一个本地数据库 (XAMPP)。

有人告诉我主要使用 jQuery 或 JavaScript 做各种事情，但我也没有经验，所以我想知道是否有人可以帮助我。

有人向我推荐了这个链接 Using Greasemonkey and jQuery to intercept JSON/AJAX data from a page, and process it

它也使用 Greasemonkey，我对此也不太了解......

在此先感谢您的帮助

示例/更多详细信息:
在监视发送的请求(通过 Firebug )时，我在下面看到

http://www.bidcactus.com/CactusWeb/ItemUpdates?rnd=1310684278585
The response of this link is the following:
{"s":"uk5c","a":[{"w":"MATADORA","t":944,"p":5,"a":413173,"x":10},   
{"w":"1000BidsAintEnough","t":6,"p":863,"a":413198,"x":0}, 
{"w":"YourBidzWillBeWastedHere","t":4725,"p":21,"a":413200,"x":8}, 
{"w":"iwillpay2much","t":344,"p":9,"a":413201,"x":9}, 
{"w":"apcyclops84","t":884,"p":3,"a":413213,"x":14}, 
{"w":"goin_postal","t":165,"p":5,"a":413215,"x":12}, 
{"w":"487951","t":825,"p":10,"a":413218,"x":6}, 
{"w":"mishmash","t":3225,"p":3,"a":413222,"x":7}, 
{"w":"CrazyKatLady2","t":6464,"p":1,"a":413224,"x":2}, 
{"w":"BOSS1","t":224,"p":102,"a":413230,"x":4}, 
{"w":"serbian48","t":62,"p":2,"a":413232,"x":11}, 
{"w":"Tuffenough","t":1785,"p":1,"a":413234,"x":1}, 
{"w":"apcyclops84","t":1970,"p":1,"a":413240,"x":13}, 
{"w":"Tuffenough","t":3524,"p":1,"a":413244,"x":5}, 
{"w":"Cdm17517","t":1424,"p":1,"a":413252,"x":3}],"tau":"0"}

我了解这些信息的含义，我想我可以自己对其进行格式化，但网站会随机创建新请求。
示例 http://www.bidcactus.com/CactusWeb/ItemUpdates?rnd=XXXXXXXXXXXX
而且我不确定它是如何创建它们的。

所以我需要获得所有关于项目更新的请求的响应，并将信息发送到 mysql 数据库。

Answer 1

好的，这是工作代码，针对该站点(首页，无帐户，仅)进行了一些调整。

使用说明:

1. 安装 GM 脚本。请注意，目前仅支持 Firefox。

2. 观察它在 Firebug 的控制台中运行，并调整过滤器部分(明确标记)，以定位您感兴趣的数据。(也许是整个 a 数组？)

   请注意，在打印“Script Start”后可能需要几秒钟才能开始 ajax 拦截。

3. 设置您的网络应用程序和服务器以接收数据。该脚本发布 JSON，因此 PHP 会像这样获取数据:

    $jsonData   = json_decode ($HTTP_RAW_POST_DATA);
   

4. 将脚本指向您的服务器。

5. 瞧。她完成了。

---

/******************************************************************************
*******************************************************************************
**  This script intercepts ajaxed data from the target web pages.
**  There are 4 main phases:
**      1)  Intercept XMLHttpRequest's made by the target page.
**      2)  Filter the data to the items of interest.
**      3)  Transfer the data from the page-scope to the GM scope.
**          NOTE:   This makes it technically possibly for the target page's
**                  webmaster to hack into GM's slightly elevated scope and
**                  exploit any XSS or zero-day vulnerabilities, etc.  The risk
**                  is probably zero as long as you don't start any feuds.
**      4)  Use GM_xmlhttpRequest () to send the data to our server.
*******************************************************************************
*******************************************************************************
*/
// ==UserScript==
// @name            _Record ajax, JSON data.
// @namespace       stackoverflow.com/users/331508/
// @description     Intercepts Ajax data, filters it and then sends it to our server.
// @include         http://www.bidcactus.com/*
// ==/UserScript==

DEBUG   = true;
if (DEBUG)  console.log ('***** Script Start *****');


/******************************************************************************
*******************************************************************************
**  PHASE 1 starts here, this is the XMLHttpRequest intercept code.
**  Note that it will not work in GM's scope.  We must inject the code to the
**  page scope.
*******************************************************************************
*******************************************************************************
*/
funkyFunc   = ( (<><![CDATA[

    DEBUG           = false;
    //--- This is where we will put the data we scarf. It will be a FIFO stack.
    payloadArray    = [];   //--- PHASE 3a

    (function (open) {
        XMLHttpRequest.prototype.open = function (method, url, async, user, pass)
        {
            this.addEventListener ("readystatechange", function (evt)
            {
                if (this.readyState == 4  &&  this.status == 200)  //-- Done, & status "OK".
                {
                    var jsonObj = null;
                    try {
                        jsonObj = JSON.parse (this.responseText);   // FF code.  Chrome??
                    }
                    catch (err) {
                        //if (DEBUG)  console.log (err);
                    }
                    //if (DEBUG)  console.log (this.readyState, this.status, this.responseText);

                    /******************************************************************************
                    *******************************************************************************
                    **  PHASE 2:    Filter as much as possible, at this stage.
                    **              For this site, jsonObj should be an object like so:
                    **                  { s="1bjqo", a=[15], tau="0"}
                    **              Where a is an array of objects, like:
                    **                  a   417387
                    **                  p   1
                    **                  t   826
                    **                  w   "bart69"
                    **                  x   7
                    *******************************************************************************
                    *******************************************************************************
                    */
                    //if (DEBUG)  console.log (jsonObj);
                    if (jsonObj  &&  jsonObj.a  &&  jsonObj.a.length > 1) {
                        /*--- For demonstration purposes, we will only get the 2nd row in
                            the `a` array. (Probably stands for "auction".)
                        */
                        payloadArray.push (jsonObj.a[1]);
                        if (DEBUG)  console.log (jsonObj.a[1]);
                    }
                    //--- Done at this stage!  Rest is up to the GM scope.
                }
            }, false);

            open.call (this, method, url, async, user, pass);
        };
    } ) (XMLHttpRequest.prototype.open);
]]></>).toString () );


function addJS_Node (text, s_URL)
{
    var scriptNode                      = document.createElement ('script');
    scriptNode.type                     = "text/javascript";
    if (text)  scriptNode.textContent   = text;
    if (s_URL) scriptNode.src           = s_URL;

    var targ    = document.getElementsByTagName('head')[0] || d.body || d.documentElement;
    targ.appendChild (scriptNode);
}

addJS_Node (funkyFunc);


/******************************************************************************
*******************************************************************************
**  PHASE 3b:
**  Set up a timer to check for data from our ajax intercept.
**  Probably best to make it slightly faster than the target's
**  ajax frequency (about 1 second?).
*******************************************************************************
*******************************************************************************
*/
timerHandle = setInterval (function() { SendAnyResultsToServer (); }, 888);

function SendAnyResultsToServer ()
{
    if (unsafeWindow.payloadArray) {
        var payload     = unsafeWindow.payloadArray;
        while (payload.length) {
            var dataRow = JSON.stringify (payload[0]);
            payload.shift ();   //--- pop measurement off the bottom of the stack.
            if (DEBUG)  console.log ('GM script, pre Ajax: ', dataRow);

            /******************************************************************************
            *******************************************************************************
            **  PHASE 4: Send the data, one row at a time, to the our server.
            **  The server would grab the data with:
            **      $jsonData   = json_decode ($HTTP_RAW_POST_DATA);
            *******************************************************************************
            *******************************************************************************
            */
            GM_xmlhttpRequest ( {
                method:     "POST",
                url:        "http://localhost/db_test/ShowJSON_PostedData.php",
                data:       dataRow,
                headers:    {"Content-Type": "application/json"},
                onload:     function (response) {
                                if (DEBUG)  console.log (response.responseText);
                            }
            } );
        }
    }
}


//--- EOF

---

杂项说明:

1. 我在该站点的主页上测试了它，但没有登录(我不打算在那里创建帐户)。

2. 我用 AdBlock、FlashBlock、NoSCript 和 RequestPolicy 进行了测试，所有这些都完全有效。 JS 已为 bidcactus.com 打开(它必须是)，但没有其他。重新打开所有这些垃圾应该不会导致副作用——但如果是的话，我不会调试它。

3. 像这样的代码必须针对站点以及您浏览所述站点的方式进行调整¹。这取决于你。希望代码有足够的 self 记录。

4. 尽情享受吧!

---

¹ 主要是:@include和@exclude指令，JSON数据的选择和过滤，是否需要屏蔽iFrame。此外，建议在调整完成后将 2 个 DEBUG 变量(一个用于 GM 范围，一个用于页面范围)设置为 false。