apache-spark - Spark : YARN throws NoSuchMethodError on NettyMemoryMetrics

Question

为了让 Spark (spark-2.3.0-bin-without-hadoop) 在 HDFS 上与 YARN 一起工作，我将 Hadoop 降级到 hadoop-2.7.6 以解决依赖问题。

到目前为止，HDFS 和 YARN 都没有问题。

当我提交一个 Spark Jar 时它崩溃了，我得到以下 Stacktrace:

Exception in thread "main" java.lang.NoSuchMethodError: io.netty.buffer.PooledByteBufAllocator.metric()Lio/netty/buffer/PooledByteBufAllocatorMetric;
    at org.apache.spark.network.util.NettyMemoryMetrics.registerMetrics(NettyMemoryMetrics.java:80)
    at org.apache.spark.network.util.NettyMemoryMetrics.<init>(NettyMemoryMetrics.java:76)
    at org.apache.spark.network.client.TransportClientFactory.<init>(TransportClientFactory.java:109)
    at org.apache.spark.network.TransportContext.createClientFactory(TransportContext.java:99)
    at org.apache.spark.rpc.netty.NettyRpcEnv.<init>(NettyRpcEnv.scala:71)
    at org.apache.spark.rpc.netty.NettyRpcEnvFactory.create(NettyRpcEnv.scala:461)
    at org.apache.spark.rpc.RpcEnv$.create(RpcEnv.scala:57)
    at org.apache.spark.deploy.yarn.ApplicationMaster.runExecutorLauncher(ApplicationMaster.scala:515)
    at org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$$runImpl(ApplicationMaster.scala:347)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply$mcV$sp(ApplicationMaster.scala:260)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply(ApplicationMaster.scala:260)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply(ApplicationMaster.scala:260)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$5.run(ApplicationMaster.scala:800)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1758)
    at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:799)
    at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:259)
    at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:824)
    at org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:854)
    at org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala)
> 

这在我启动自己的程序和 Spark 提供的示例时都会发生。我很确定我在 spark-env.sh 中正确设置了我的类路径:

export SPARK_DIST_CLASSPATH=$(/usr/local/hadoop/bin/hadoop classpath):/usr/local/spark/jars/*

生成的类路径如下所示:

SPARK_DIST_CLASSPATH='/usr/local/hadoop/etc/hadoop:/usr/local/hadoop//share/hadoop/common/lib/*:/usr/local/hadoop//share/hadoop/common/*:/usr/local/hadoop//share/hadoop/hdfs:/usr/local/hadoop//share/hadoop/hdfs/lib/*:/usr/local/hadoop//share/hadoop/hdfs/*:/usr/local/hadoop-2.7.6/share/hadoop/yarn/lib/*:/usr/local/hadoop-2.7.6/share/hadoop/yarn/*:/usr/local/hadoop//share/hadoop/mapreduce/lib/*:/usr/local/hadoop//share/hadoop/mapreduce/*:/contrib/capacity-scheduler/*.jar:/usr/local/spark/jars/*'

我不知道如何解决这个问题。除了配置问题，我最好的猜测是另一个库不兼容。在那种情况下，任何人都可以向我指出实际工作时没有冲突的 Spark/Hadoop 组合吗？

find . -name netty*
./spark-2.3.0-bin-without-hadoop/jars/netty-3.9.9.Final.jar
./spark-2.3.0-bin-without-hadoop/jars/netty-all-4.1.17.Final.jar
./hadoop-2.7.6/share/hadoop/yarn/lib/netty-3.6.2.Final.jar
./hadoop-2.7.6/share/hadoop/kms/tomcat/webapps/kms/WEB-INF/lib/netty-3.6.2.Final.jar
./hadoop-2.7.6/share/hadoop/tools/lib/netty-3.6.2.Final.jar
./hadoop-2.7.6/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar
./hadoop-2.7.6/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar
./hadoop-2.7.6/share/hadoop/common/lib/netty-3.6.2.Final.jar
./hadoop-2.7.6/share/hadoop/httpfs/tomcat/webapps/webhdfs/WEB-INF/lib/netty-3.6.2.Final.jar
./hadoop-2.7.6/share/hadoop/httpfs/tomcat/webapps/webhdfs/WEB-INF/lib/netty-all-4.0.23.Final.jar
./hadoop-2.7.6/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar

Answer 1

我已经解决了这个问题，答案简单得令人尴尬:spark.yarn.jars 属性不必设置为 /usr/local/spark/jar 但到 /usr/local/spark/jar/* 一切正常。