java - 使用 HtmlUnit 时，如何配置底层 NekoHtml 解析器？

Question

我正在使用 HtmlUnit 来尝试抓取网页，因为它支持 Javascript。 (我宁愿使用 Jsoup，但不支持 JS)。

该问题与底层 NekoHtml 解析器的一项功能有关: “http://cyberneko.org/html/features/scanner/allow-selfclosing-iframe”

参见:http://nekohtml.sourceforge.net/settings.html

这显然可以在 Neko 中启用，但我使用的是 HtmlUnit。有没有办法配置 HTML 单元使用的底层 Neko 解析器来启用此功能？

尝试运行此代码时:

final WebClient webClient = new WebClient();
HtmlPage page = webClient.getPage(url.toString());

我收到此错误:

Caused by: com.gargoylesoftware.htmlunit.ObjectInstantiationException: unable to create HTML parser
    at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.<init>(HTMLParser.java:418)
    at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.<init>(HTMLParser.java:342)
    at com.gargoylesoftware.htmlunit.html.HTMLParser.parse(HTMLParser.java:203)
    at com.gargoylesoftware.htmlunit.html.HTMLParser.parseHtml(HTMLParser.java:179)
    at com.gargoylesoftware.htmlunit.DefaultPageCreator.createHtmlPage(DefaultPageCreator.java:221)
    at com.gargoylesoftware.htmlunit.DefaultPageCreator.createPage(DefaultPageCreator.java:106)
    at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:433)
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:311)
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:373)
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:358)
    at 
Caused by: org.xml.sax.SAXNotRecognizedException: Feature 'http://cyberneko.org/html/features/scanner/allow-selfclosing-iframe' is not recognized.
    at org.apache.xerces.parsers.AbstractSAXParser.setFeature(Unknown Source)
    at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.<init>(HTMLParser.java:411)
    ... 41 more

Answer 1

尝试使用 FF 行为初始化 Web 客户端

WebClient webClient = new WebClient(BrowserVersion.FIREFOX_3_6);

并激活 JavaScript

webClient.setJavaScriptEnabled(true);

那就应该没问题了。