javascript - 在 HtmlUnit (2.33) 中模拟页面点击给出无效或非法的选择器异常

Question

首先我得说我一点也不了解 Javascript。我正在尝试模拟对来自 Bloomberg 的超链接页面的点击。我想获取新闻项目列表(超链接)，然后简单地遍历列表获取每篇文章标题和文章文本。这是我的代码:

public List<String> getBloomNewsHtmlUnit() throws IOException {
    String searchString = "Apple";
    List<String> bloombergNewsAll = new ArrayList<>();

    WebClient webclient = new WebClient(BrowserVersion.BEST_SUPPORTED);

    HtmlPage mainpage = webclient.getPage("http://www.bloomberg.com/search?query=" + searchString);

    HtmlAnchor pageanchor = mainpage.getFirstByXPath("//*[@id=\"content\"]/div/section/section[2]/section[1]/div[2]/div[2]/article/div[1]/h1/a");

    webclient.waitForBackgroundJavaScript(50000);
    webclient.getOptions().setThrowExceptionOnScriptError(false);
    webclient.getOptions().setThrowExceptionOnFailingStatusCode(false);
    webclient.setCssErrorHandler(new SilentCssErrorHandler());

    mainpage = pageanchor.click();

    System.out.println("Main page: " + mainpage.asText());

    return bloombergNewsAll;
    //  return bloombergNewsAll;
}

这是异常(exception)情况:

Sep 11, 2016 9:49:34 AM com.gargoylesoftware.htmlunit.javascript.StrictErrorReporter runtimeError
SEVERE: runtimeError: message=[An invalid or illegal selector was specified (selector: '*,:x' error: Invalid selector: :x).] sourceName=[https://assets.bwbx.io/business/public/javascripts/application-6e1529c288.js] line=[153] lineSource=[null] lineOffset=[0]
Exception in thread "main" java.lang.RuntimeException: com.gargoylesoftware.htmlunit.ScriptException: TypeError: Cannot call method "split" of undefined (https://assets.bwbx.io/business/public/javascripts/application-6e1529c288.js#79)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:284)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:519)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:386)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:304)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:451)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:436)
at com.jsoup.test.BloombergTest.getBloomNewsHtmlUnit(BloombergTest.java:71)
at com.jsoup.test.BloombergTest.main(BloombergTest.java:37)
Caused by: com.gargoylesoftware.htmlunit.ScriptException: TypeError: Cannot call method "split" of undefined (https://assets.bwbx.io/business/public/javascripts/application-6e1529c288.js#79)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:921)
at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:628)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:515)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:803)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:779)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadExternalJavaScriptFile(HtmlPage.java:975)
at com.gargoylesoftware.htmlunit.html.HtmlScript.executeScriptIfNeeded(HtmlScript.java:352)
at com.gargoylesoftware.htmlunit.html.HtmlScript$2.execute(HtmlScript.java:238)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:277)
... 7 more
Caused by: net.sourceforge.htmlunit.corejs.javascript.EcmaError: TypeError: Cannot call method "split" of undefined (https://assets.bwbx.io/business/public/javascripts/application-6e1529c288.js#79)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3915)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3899)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeError(ScriptRuntime.java:3924)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeError2(ScriptRuntime.java:3940)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.undefCallError(ScriptRuntime.java:3956)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.getPropFunctionAndThisHelper(ScriptRuntime.java:2390)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.getPropFunctionAndThis(ScriptRuntime.java:2384)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1342)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:800)
at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:413)
at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory.doTopCall(HtmlUnitContextFactory.java:252)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3264)
at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.exec(InterpretedFunction.java:115)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$3.doRun(JavaScriptEngine.java:794)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:906)
... 15 more
Java Result: 1

即使我尝试执行代码的前 4 行(不引用 HtmlAnchor)，也会出现相同的错误。我在网上阅读了一些有关此错误的错误报告，但建议的解决方案似乎都不适用于我的情况:

htmlunit : An invalid or illegal selector was specified

在上面的SOF问题中，我将建议的waitForBackgroundJavaScript应用于webclient，但这并没有解决问题。

JavaScript Exception in HtmlUnit when clicking at google result page

在这个问题中我尝试补充:

JavaScriptEngine engine = webclient.getJavaScriptEngine();
    engine.holdPosponedActions();

到代码，但错误仍然存​​在。

https://sourceforge.net/p/htmlunit/bugs/1744/

在上面的错误报告中，建议的解决方案是使用选择查询结果重新定义主页。就我而言，我尝试使用 click() 事件重新定义页面。当我尝试定义 HtmlPage 时，我的代码并没有走那么远并抛出错误。

https://sourceforge.net/p/htmlunit/bugs/1661/

此报告建议简单地忽略警告，但在我的例子中，我遇到了一个异常(不仅仅是警告)，这阻止了所需的输出。

我首先尝试使用 Jsoup 进行抓取。这工作正常，但 Jsoup 在文章文本之间提供了一些错误链接，当我在 Chrome 中检查它时，这些链接不在原始页面上。我怀疑有一个 JS 或 Ajax 调用改变了页面 DOM。这就是我选择使用 Htmlunit 的原因。

如果有任何关于我做错了什么导致出现此错误以及如何更正错误的提示，我将不胜感激。此外，如果有人认为可以仅使用 Jsoup 来实现我想要的，请告诉我(我刚刚读到 Jsoup 不支持 DOM 中的动态更改，因此不能单独使用)。提前致谢!

Answer 1

异常并不一定意味着生成的页面无用，尽管在其他情况下可能有所不同。您必须检查您要查找的内容的结果。

要减少 javascript 引擎的错误消息输出，您可以定义:

java.util.logging.Logger.getLogger("com.gargoylesoftware").setLevel(java.util.logging.Level.OFF);

以下示例选择第一个标题，触发点击事件并抓取结果页面；为了验证我们点击了链接，标题被打印出来:

java.util.logging.Logger.getLogger("com.gargoylesoftware").setLevel(java.util.logging.Level.OFF);

final WebClient webClient = new WebClient(BrowserVersion.CHROME);

webClient.getOptions().setCssEnabled(false);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
webClient.getOptions().setTimeout(10000);

try {
    HtmlPage page = webClient.getPage("http://www.bloomberg.com/search?query=Apple");

    System.out.println(page.getTitleText());

    ScriptResult result = page.executeJavaScript("document.querySelector(\"#content > div > section > section.search-results__content > section.content-stories > div.search-result-items > div:nth-child(1) > article > div > h1 > a\").click()");

    page = (HtmlPage)result.getNewPage();

    System.out.println(page.getTitleText());

} catch (Exception e) {
    e.printStackTrace();
} finally {
    webClient.close();
}

---

由于页面不是使用 javascript 填充的，您也可以完全跳过 HtmlUnit 并使用像 jsoup 这样的 html 解析器。 :

新闻类

class News{
    private String title;
    private String href;
    private String content="";

    public String getTitle() {
        return title;
    }

    public String getHref() {
        return href;
    }

    public String getContent() {
        return content;
    }

    public void setContent(String content) {
        this.content = content;
    }

    public News(String title, String href){
        this.title=title;
        this.href=href;
    }
}

前两页抓取新闻示例代码(可通过numberOfResultpages调整):

List<News> bloombergNewsAll = new ArrayList<>();

String searchString = "Apple";
String searchUrl = "http://www.bloomberg.com/search?query=" + searchString + "&page=";
int numberOfResultpages = 2;
Document doc;

// grab title and href
for (int i = 1; i <= numberOfResultpages; i++) {
    try {
        doc = Jsoup.connect(searchUrl + i)
                .userAgent("Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36")
                .referrer("http://www.bloomberg.com/").get();
        Elements searchResults = doc.select("#content > div > section > section.search-results__content > section.content-stories > div.search-result-items > div > article > div > h1");
        if(searchResults.isEmpty()) break; // no more searchResults

        for (Element result : searchResults) {
            bloombergNewsAll.add(new News(result.text(), result.select("a").attr("href")));
        }
    } catch (IOException e) {
        e.printStackTrace();
    }
}

// grab content
for (News news : bloombergNewsAll) {

    try {
        doc = Jsoup.connect(news.href)
                .userAgent("Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36")
                .referrer("http://www.bloomberg.com/search?query=Apple").get();

        if(news.getHref().contains("bloomberg.com/news/videos")) continue;

        if(news.getHref().contains("bloomberg.com/news/")){
            news.setContent(doc.select("div.article-body__content").text());
        }else if(news.getHref().contains("bloomberg.com/gadfly")){
            news.setContent(doc.select("#article > div.body_ZtDFu > div.container_1KxJx").text());
        }else if(news.getHref().contains("bloomberg.com/view")){
            news.setContent(doc.select("div._31WvjDF17ltgFb1fNB1WqY").text());
        }

    } catch (IOException e) {
        e.printStackTrace();
    }
}

// do something useful with your results
for (News news : bloombergNewsAll) {
    System.out.println(news.getTitle() + "\n\t" + news.getHref() + "\n\t" + (news.getContent().length()>150 ? news.getContent().substring(0, 150) : news.getContent()));
}