java - HDInsight-Spark(spark-submit)失败 - java.lang.NoSuchMethodError : com. microsoft.azure.storage.blob.CloudBlockBlob.startCopy

Question

我们正在开发 Spark 应用程序。它将托管在azure HDInsight Spark 集群上。我们的用例是这样的，我们必须从 azure blob 存储中提取数据并使用 Spark 处理数据，最后创建或将数据追加回 azure blob 存储。所以我们使用azure-storage-4.3.0.jar

我们在 Eclipse 项目中使用了 Maven 并添加了以下依赖项

<dependency>
     <groupId>com.microsoft.azure</groupId>
     <artifactId>azure-storage</artifactId>
     <version>4.3.0</version>
</dependency>

编译成功。即使应用程序在本地计算机上运行良好并且执行时没有任何问题。

因此，我们从 eclipse 创建了一个 uber/fat jar 并将其移植到我们的 Azure HDInsight-Spark 集群，然后运行以下命令:

spark-submit --class myClassName MyUberJar.jar --verbose

应用程序遇到以下错误:

Exception in thread "main" java.lang.NoSuchMethodError: com.microsoft.azure.storage.blob.CloudBlockBlob.startCopy(Lcom/microsoft/azure/storage/blob/CloudBlockBlob;)Ljava/lang/String;
            at com.lsy.airmon2.dao.blob.AzureStorageImpl.moveData(AzureStorageImpl.java:188)
            at com.lsy.airmon2.processor.SurveyProcessor.stageData(SurveyProcessor.java:92)
            at com.lsy.airmon2.processor.Processor.doJob(Processor.java:27)
            at com.lsy.airmon2.entrypoint.AirMon2EntryPoint.runP(AirMon2EntryPoint.java:109)
            at com.lsy.airmon2.entrypoint.AirMon2EntryPoint.run(AirMon2EntryPoint.java:82)
            at com.lsy.airmon2.entrypoint.AirMon2EntryPoint.main(AirMon2EntryPoint.java:42)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:606)
            at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
            at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
            at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
            at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
            at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

当我们深入研究此问题时，我们发现 azure HDInsight Spark 已经具有旧版本的 azure-storage(azure-storage.2.2.0.jar) 路径 < strong>/usr/hdp/current/hadoop-client/lib 并且这个旧版本没有 startCopy 方法，此方法添加在 azure-storage.3.0.0.jar版本。

因此我们将 azure-storage.2.2.0.jar 替换为 azure-storage.3.0.0.jar 在所有 Driver 和 Worker 节点上。在此更改之后，应用程序遇到了奇怪的异常:

java.net.ConnectException: Call From hn0-FooBar/10.XXX.XXX.XXX to hn1-FooBar.xyzabcxyzabc.ax.internal.cloudapp.net:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801)
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
    at org.apache.hadoop.ipc.Client.call(Client.java:1430)
    at org.apache.hadoop.ipc.Client.call(Client.java:1363)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
    at com.sun.proxy.$Proxy9.transitionToStandby(Unknown Source)
    at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.transitionToStandby(HAServiceProtocolClientSideTranslatorPB.java:112)
    at org.apache.hadoop.ha.FailoverController.tryGracefulFence(FailoverController.java:172)
    at org.apache.hadoop.ha.ZKFailoverController.doFence(ZKFailoverController.java:514)
    at org.apache.hadoop.ha.ZKFailoverController.fenceOldActive(ZKFailoverController.java:505)
    at org.apache.hadoop.ha.ZKFailoverController.access$1100(ZKFailoverController.java:61)
    at org.apache.hadoop.ha.ZKFailoverController$ElectorCallbacks.fenceOldActive(ZKFailoverController.java:892)
    at org.apache.hadoop.ha.ActiveStandbyElector.fenceOldActive(ActiveStandbyElector.java:956)
    at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:855)
    at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:463)
    at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:611)
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
Caused by: java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
    at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
    at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:617)
    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:715)
    at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:378)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1492)
    at org.apache.hadoop.ipc.Client.call(Client.java:1402)
    ... 14 more

因此我们恢复了所有更改，并回到了原点。

关于如何解决此问题有什么建议吗？

Answer 1

尝试在 Spark-submit 命令中使用 --packages 开关。

例如，我在以前的应用程序中使用过它(尽管没有使用 uber jar):

--packages com.microsoft.azure:azure-storage:8.0.0

所以它应该看起来像这样:

spark-submit --packages com.microsoft.azure:azure-storage:8.0.0 --class myClassName MyUberJar.jar --verbose