java - 使用 HtmlUnit 调用 Ajax

Question

我想抓取网页，这个页面有一个下载按钮，当我按下它时，当前页面在标题中显示下载进度，然后显示可以按下的下载链接。我认为它是通过 Ajax 完成的，因为我可以在 developer console -> Network ->XHR

中看到一些内容

这是我的抓取网站的代码

 WebClient webClient = new WebClient(BrowserVersion.FIREFOX_38);
        webClient.getOptions().setJavaScriptEnabled(true);
        webClient.getOptions().setThrowExceptionOnScriptError(false);
        webClient.getOptions().setCssEnabled(true);
        webClient.setAjaxController(new NicelyResynchronizingAjaxController());
        final HtmlPage page = webClient.getPage("https://9xbuddy.com/process?url=https://www.fembed.com/v/6mv22g3qfsdfsd");
        //  final ScriptResult scriptResult = page.executeJavaScript("beacon.js");
        webClient.waitForBackgroundJavaScript(10000);
        webClient.waitForBackgroundJavaScriptStartingBefore(10000);

但是此代码返回我单击按钮后得到的页面，并且不加载 Ajax。我知道站点发出了哪些Ajax请求，有没有办法手动调用Ajax请求？

Answer 1

您可以使用HtmlUnit手动构造Ajax调用，如果您发现Google Chrome控制台不够用，您可以使用Fiddler等工具。一旦确定了 HTTP 调用，您就可以使用 HTMLUnit 重建它，如下所示

URL url = new URL(
        "http://tws.target.com/searchservice/item/search_results/v1/by_keyword?callback=getPlpResponse&navigation=true&category=55krw&searchTerm=&view_type=medium&sort_by=bestselling&faceted_value=&offset=60&pageCount=60&response_group=Items&isLeaf=true&parent_category_id=55kug&custom_price=false&min_price=from&max_price=to");
WebRequest requestSettings = new WebRequest(url, HttpMethod.GET);

requestSettings.setAdditionalHeader("Accept", "*/*");
requestSettings.setAdditionalHeader("Content-Type", "application/x-www-form-urlencoded; charset=UTF-8");
requestSettings.setAdditionalHeader("Referer", "http://www.target.com/c/xbox-one-games-video/-/N-55krw");
requestSettings.setAdditionalHeader("Accept-Language", "en-US,en;q=0.8");
requestSettings.setAdditionalHeader("Accept-Encoding", "gzip,deflate,sdch");
requestSettings.setAdditionalHeader("Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.3");

Page page = webClient.getPage(requestSettings);

System.out.println(page.getWebResponse().getContentAsString());