apache-spark - Zeppeline - 如何设置 Zeppeline 以连接到远程 sparkmaster？

Question

我在一组单独的主机上有 5 个节点的 spark 集群。我在单独的主机上安装了 zeppeline，并连接了 spark 解释器以针对 spark 集群执行查询。

Zeppeline 版本 1.6 - 安装在桌面上

两种我都试过了。

添加了“export MASTER=spark://sparkmasterhost:7077”并将 spark 解释器主变量设置为“spark://sparkmasterhost:7077”

当我运行“sc.version”时，出现此错误。

org.apache.thrift.transport.TTransportException
 at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
 at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) 
 at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) 
 at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) 
 at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) 
 at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) 
 at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:220) 
 at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:205) 
 at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:208) 
 at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:93) 
 at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:211) 
 at org.apache.zeppelin.scheduler.Job.run(Job.java:169) 
 at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:322) 
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
 at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) 
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) 
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
 at java.lang.Thread.run(Thread.java:745) 

我不确定发生了什么。

Answer 1

Zeppelin 中嵌入的 spark 版本很可能与 spark 集群的版本不同。打开http://<spark-master.url>:8080/并在左上角检查版本 - 1.6.0例如。然后下载 Zeppelin Source 并使用 Spark 版本标志在本地构建它 - zeppelin@<remote-host>:~/incubator-zeppelin$ mvn clean package -DskipTests -Pspark-1.6 -Dspark.version=1.6.0 .我刚刚克隆了 git https://github.com/apache/incubator-zeppelin .另一个非常相似的问题here