java - 如何解决java.lang.NoSuchMethodError : org. apache.hadoop.conf.Configuration.getPassword？

Question

我正在尝试从 Spring Web 应用程序运行 Spark 进程，但收到此错误:

java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.getPassword(Ljava/lang/String;)[C
    at org.apache.spark.SSLOptions$.$anonfun$parse$8(SSLOptions.scala:188) ~[spark-core_2.12-2.4.3.jar:2.4.3]
    at scala.Option.orElse(Option.scala:306) ~[scala-library-2.12.8.jar:na]
    at org.apache.spark.SSLOptions$.parse(SSLOptions.scala:188) ~[spark-core_2.12-2.4.3.jar:2.4.3]
    at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:117) ~[spark-core_2.12-2.4.3.jar:2.4.3]
    at org.apache.spark.SparkEnv$.create(SparkEnv.scala:236) ~[spark-core_2.12-2.4.3.jar:2.4.3]
    at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:185) ~[spark-core_2.12-2.4.3.jar:2.4.3]
    at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:257) ~[spark-core_2.12-2.4.3.jar:2.4.3]
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:424) ~[spark-core_2.12-2.4.3.jar:2.4.3]
    at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520) ~[spark-core_2.12-2.4.3.jar:2.4.3]
    at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$5(SparkSession.scala:935) ~[spark-sql_2.12-2.4.2.jar:2.4.2]
    at scala.Option.getOrElse(Option.scala:138) ~[scala-library-2.12.8.jar:na]
    at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926) ~[spark-sql_2.12-2.4.2.jar:2.4.2]

Answer 1

问题清楚地表明 hadoop 库版本不匹配。

此外，我遇到了同样的问题，因为我使用的是 Maven，所以我在下面发表了评论。因为我没有使用。它解决了问题

<!--<dependency>-->
      <!--<groupId>org.apache.hadoop</groupId>-->
      <!--<artifactId>hadoop-client</artifactId>-->
      <!--<version>${hadoop.version}</version>-->
      <!--<scope>provided</scope>-->
    <!--</dependency>-->