scala - uber jar 中的 NoSuchMethodError 异常

Question

这些是 dependencies里面是我的 pom.xml 文件。当我从我的 maven 项目中将 Driver 作为 Scala 应用程序运行时，它工作正常。但是当我创建一个 jar-with-dependency 并尝试通过 spark-submit

运行我的项目时

spark-submit --class package.signature.Driver --master local[*] /path/to/my/jar-with-dependencies.jar

我得到以下异常。

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.http.conn.ssl.SSLConnectionSocketFactory.(Ljavax/net/ssl/SSLContext;[Ljava/lang/String;[Ljava/lang/String;Ljavax/net/ssl/HostnameVerifier;)V at com.gargoylesoftware.htmlunit.httpclient.HtmlUnitSSLConnectionSocketFactory.(HtmlUnitSSLConnectionSocketFactory.java:125) at com.gargoylesoftware.htmlunit.httpclient.HtmlUnitSSLConnectionSocketFactory.buildSSLSocketFactory(HtmlUnitSSLConnectionSocketFactory.java:112) at com.gargoylesoftware.htmlunit.HttpWebConnection.configureHttpsScheme(HttpWebConnection.java:597) at com.gargoylesoftware.htmlunit.HttpWebConnection.createHttpClient(HttpWebConnection.java:532) at com.gargoylesoftware.htmlunit.HttpWebConnection.getHttpClientBuilder(HttpWebConnection.java:494) at com.gargoylesoftware.htmlunit.HttpWebConnection.getResponse(HttpWebConnection.java:158) at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseFromWebConnection(WebClient.java:1321) at com.gargoylesoftware.htmlunit.WebClient.loadWebResponse(WebClient.java:1238) at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:346) at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:415) at org.openqa.selenium.htmlunit.HtmlUnitDriver.get(HtmlUnitDriver.java:541) at org.openqa.selenium.htmlunit.HtmlUnitDriver.get(HtmlUnitDriver.java:530)

此异常的来源是代码中的.getPageSource

import org.openqa.selenium.htmlunit.HtmlUnitDriver
import java.util.concurrent.TimeUnit
import scala.concurrent.duration._
import org.apache.spark.Logging

object Selenium extends Logging {
  val driver = new HtmlUnitDriver(false)
  val implicitWaitTimeout = 30 // seconds
  driver.manage.timeouts.implicitlyWait(implicitWaitTimeout, TimeUnit.SECONDS)
  def getPageSource(urlPage: String): String = {
    driver.get(urlPage)
    driver.getPageSource
  }
}

到目前为止我已经尝试过什么。试图排除 apache-httpcomponent 的多个版本。但这没有用。

Answer 1

这是一个传递依赖性问题，这意味着你有 jar 有这个类，每个都有那个类的版本，而这个方法不存在，默认情况下 maven 加载它首先获得的那个类，你可以通过检查类按Ctrl+Shift+T，你可以看到那个类的多个版本，一旦你找到没有那个方法的类，在 pom.xml 中为包含该文件的 .jar 写排除。