apache-spark - LivyClient uploadJar失败，出现py4j.Py4JException:获取新的通信 channel 时出错

Question

我正在尝试通过Apache Livy提交Spark作业，但是LivyClient的uploadJar方法失败。

这是代码(非常类似于PiJob示例):

        LivyClientBuilder builder = new LivyClientBuilder();
        LivyClient client = builder.setURI(new URI("http://server:8998")).build();
        client.uploadJar(new File("/path/to/file")).get();

这是完整的堆栈跟踪:

py4j.Py4JException: Error while obtaining a new communication channel
    at py4j.CallbackClient.getConnectionLock(CallbackClient.java:257)
    at py4j.CallbackClient.sendCommand(CallbackClient.java:377)
    at py4j.CallbackClient.sendCommand(CallbackClient.java:356)
    at py4j.reflection.PythonProxyHandler.invoke(PythonProxyHandler.java:106)
    at com.sun.proxy.$Proxy24.getLocalTmpDirPath(Unknown Source)
    at org.apache.livy.repl.PythonInterpreter.addPyFile(PythonInterpreter.scala:294)
    at org.apache.livy.repl.ReplDriver$$anonfun$addJarOrPyFile$1.apply(ReplDriver.scala:114)
    at org.apache.livy.repl.ReplDriver$$anonfun$addJarOrPyFile$1.apply(ReplDriver.scala:114)
    at scala.Option.foreach(Option.scala:257)
    at org.apache.livy.repl.ReplDriver.addJarOrPyFile(ReplDriver.scala:114)
    at org.apache.livy.rsc.driver.JobContextImpl.addJarOrPyFile(JobContextImpl.java:151)
    at org.apache.livy.rsc.driver.AddJarJob.call(AddJarJob.java:39)
    at org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:64)
    at org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:31)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at java.net.Socket.connect(Socket.java:538)
    at java.net.Socket.<init>(Socket.java:434)
    at java.net.Socket.<init>(Socket.java:244)
    at javax.net.DefaultSocketFactory.createSocket(SocketFactory.java:277)
    at py4j.CallbackConnection.start(CallbackConnection.java:226)
    at py4j.CallbackClient.getConnection(CallbackClient.java:238)
    at py4j.CallbackClient.getConnectionLock(CallbackClient.java:250)
    ... 17 more

我可以通过REST API将代码段提交给Livy服务器，它们运行得很好。 Livy / Spark是在YARN上设置的，我已经尝试了客户端和群集模式。有任何想法吗？

Answer 1

从您的代码看来，您的jar路径未设置，请尝试设置jar的路径。

示例

client.uploadJar(new File("tmp/myjar.jar")).get();

您还可以检查pyspark环境的环境路径，确保节点在同一环境中通信。