javascript - 使用 htmlunit 抓取 javascript - "You are currently browsing with JavaScript turned off"

Question

我正在尝试抓取this page与 HtmlUnit。在 Xml 中，它显示“您当前浏览时 JavaScript 已关闭，这意味着您无法使用我们的搜索功能。”在使用 Python 和 Selenium 之前我已经能够抓取它，但我还无法让它与 Java 和

import java.io.IOException;
import java.net.MalformedURLException;

import com.gargoylesoftware.htmlunit.BrowserVersion;
import com.gargoylesoftware.htmlunit.SilentCssErrorHandler;
import com.gargoylesoftware.htmlunit.WebClient;
import com.gargoylesoftware.htmlunit.html.HtmlPage;

public class WebScraper {
    public static void main(String[] args) throws MalformedURLException, IOException {
        try {
            WebClient webClient = new WebClient(BrowserVersion.CHROME);
            webClient.getOptions().setJavaScriptEnabled(true);
            webClient.waitForBackgroundJavaScript(2000);
            webClient.setCssErrorHandler(new SilentCssErrorHandler());
            HtmlPage page = webClient.getPage("https://doaj.org/search?source=%7B%22query%22%3A%7B%22filtered%22%3A%7B%22filter%22%3A%7B%22bool%22%3A%7B%22must%22%3A%5B%7B%22term%22%3A%7B%22index.classification.exact%22%3A%22Biology%20(General)%22%7D%7D%5D%7D%7D%2C%22query%22%3A%7B%22match_all%22%3A%7B%7D%7D%7D%7D%2C%22sort%22%3A%5B%7B%22created_date%22%3A%7B%22order%22%3A%22desc%22%7D%7D%5D%7D");
            System.out.println(page.asXml());

            webClient.close();
        } catch (IOException ex) {
            ex.printStackTrace();
        }

    }
}

Answer 1

网页的大部分内容都是由 javascript 异步启动的。您需要稍等一下才能获取内容......

String url = "https://doaj.org/search?source=%7B%22query%22%3A%7B%22filtered%22%3A%7B%22filter%22%3A%7B%22bool%22%3A%7B%22must%22%3A%5B%7B%22term%22%3A%7B%22index.classification.exact%22%3A%22Biology%20(General)%22%7D%7D%5D%7D%7D%2C%22query%22%3A%7B%22match_all%22%3A%7B%7D%7D%7D%7D%2C%22sort%22%3A%5B%7B%22created_date%22%3A%7B%22order%22%3A%22desc%22%7D%7D%5D%7D";

try (final WebClient webClient = new WebClient(BrowserVersion.FIREFOX)) {
    // js is enabled by default
    webClient.waitForBackgroundJavaScriptStartingBefore(1_000);

    HtmlPage page = webClient.getPage(url);
    webClient.waitForBackgroundJavaScript(10_000);

    System.out.println(page.asText());
}

此处适用于版本 2.42.0-SNAPSHOT，但也应适用于 2.41.0 版本。