java - Spark Elasticsearch : Multiple ES-Hadoop versions detected in the classpath

Question

我是新手。我正在尝试运行将数据加载到 elasticsearch 的 spark 作业。我用我的代码构建了一个 fat jar，并在 spark-submit 期间使用了它。

spark-submit \
  --class CLASS_NAME \
  --master yarn \
  --deploy-mode cluster \
  --num-executors 20 \
  --executor-cores 5 \
  --executor-memory 32G \
  --jars EXTERNAL_JAR_FILES \
  PATH_TO_FAT_JAR

elasticsearch-hadoop依赖的maven依赖为:

<dependency>
    <groupId>org.elasticsearch</groupId>
    <artifactId>elasticsearch-hadoop</artifactId>
    <version>5.6.10</version>
    <exclusions>
        <exclusion>
            <groupId>org.slf4j</groupId>
            <artifactId>log4j-over-slf4j</artifactId>
        </exclusion>
    </exclusions>
</dependency>

当我不在 EXTERNAL_JAR_FILES 列表中包含 elasticsearch-hadoop jar 文件时，我会收到此错误。

java.lang.ExceptionInInitializerError
Caused by: java.lang.ClassNotFoundException: org.elasticsearch.spark.rdd.CompatUtils
  at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
  at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
  at java.security.AccessController.doPrivileged(Native Method)
  at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  at java.lang.Class.forName0(Native Method)
  at java.lang.Class.forName(Class.java:344)
  at org.elasticsearch.hadoop.util.ObjectUtils.loadClass(ObjectUtils.java:73)
  ... 26 more

如果我将它包含在 EXTERNAL_JAR_FILES 列表中，我会收到此错误。

java.lang.Error: Multiple ES-Hadoop versions detected in the classpath; please use only one
jar:file:PATH_TO_CONTAINER/__app__.jar
jar:file:PATH_TO_CONTAINER/elasticsearch-hadoop-5.6.10.jar

  at org.elasticsearch.hadoop.util.Version.<clinit>(Version.java:73)
  at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:572)
  at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:58)
  at org.elasticsearch.spark.sql.EsSparkSQL$$anonfun$saveToEs$1.apply(EsSparkSQL.scala:97)
  at org.elasticsearch.spark.sql.EsSparkSQL$$anonfun$saveToEs$1.apply(EsSparkSQL.scala:97)
  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
  at org.apache.spark.scheduler.Task.run(Task.scala:108)
  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)

有什么需要做的来克服它吗？

Answer 1

通过在我构建的 fat jar 中不包含 elasticserach-hadoop jar 来解决问题。我在依赖项中提到了 scope param to provided。

    <dependency>
        <groupId>org.elasticsearch</groupId>
        <artifactId>elasticsearch-hadoop</artifactId>
        <version>5.6.10</version>
        <exclusions>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>log4j-over-slf4j</artifactId>
            </exclusion>
        </exclusions>
        <scope>provided</scope>
    </dependency>