- android - RelativeLayout 背景可绘制重叠内容
- android - 如何链接 cpufeatures lib 以获取 native android 库?
- java - OnItemClickListener 不起作用,但 OnLongItemClickListener 在自定义 ListView 中起作用
- java - Android 文件转字符串
下面是我的项目结构:
spark 应用程序:
scala1.scala // I am calling the java class from this class.
java.java // this will submit another spark application to the yarn cluster.
由 java 类触发的 spark 应用程序:
scala2.scala
我的引用教程是here
当我在 local mode 中通过 spark-submit
从 scala1.scala 运行我的 java 类时,第二个 spark 应用程序 scala2.scala
是被触发并按预期工作。
但是,当我在 yarn cluster 中通过 spark-submit
运行相同的应用程序时,它显示以下错误!
Error: Could not find or load main class
org.apache.spark.deploy.yarn.ApplicationMaster
Application application_1493671618562_0072 failed 5 times due to AM Container for appattempt_1493671618562_0072_000005 exited with exitCode: 1
For more detailed output, check the application tracking page: http://headnode.internal.cloudapp.net:8088/cluster/app/application_1493671618562_0072 Then click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_e02_1493671618562_0072_05_000001
Exit code: 1
Exception message: /mnt/resource/hadoop/yarn/local/usercache/helixuser/appcache/application_1493671618562_0072/container_e02_1493671618562_0072_05_000001/launch_container.sh: line 26: $PWD:$PWD/spark_conf:$PWD/spark.jar:$HADOOP_CONF_DIR:/usr/hdp/current/hadoop-client/:/usr/hdp/current/hadoop-client/lib/:/usr/hdp/current/hadoop-hdfs-client/:/usr/hdp/current/hadoop-hdfs-client/lib/:/usr/hdp/current/hadoop-yarn-client/:/usr/hdp/current/hadoop-yarn-client/lib/:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/:$PWD/mr-framework/hadoop/share/hadoop/common/:$PWD/mr-framework/hadoop/share/hadoop/common/lib/:$PWD/mr-framework/hadoop/share/hadoop/yarn/:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/:$PWD/mr-framework/hadoop/share/hadoop/hdfs/:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure: bad substitution
Stack trace: ExitCodeException exitCode=1: /mnt/resource/hadoop/yarn/local/usercache/helixuser/appcache/application_1493671618562_0072/container_e02_1493671618562_0072_05_000001/launch_container.sh: line 26: $PWD:$PWD/spark_conf:$PWD/spark.jar:$HADOOP_CONF_DIR:/usr/hdp/current/hadoop-client/:/usr/hdp/current/hadoop-client/lib/:/usr/hdp/current/hadoop-hdfs-client/:/usr/hdp/current/hadoop-hdfs-client/lib/:/usr/hdp/current/hadoop-yarn-client/:/usr/hdp/current/hadoop-yarn-client/lib/:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/:$PWD/mr-framework/hadoop/share/hadoop/common/:$PWD/mr-framework/hadoop/share/hadoop/common/lib/:$PWD/mr-framework/hadoop/share/hadoop/yarn/:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/:$PWD/mr-framework/hadoop/share/hadoop/hdfs/:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure: bad substitution
at org.apache.hadoop.util.Shell.runCommand(Shell.java:933)
at org.apache.hadoop.util.Shell.run(Shell.java:844)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1123)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:225)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:317)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:83)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
Failing this attempt. Failing the application.
我的项目目录结构如下:
lrwxrwxrwx 1 yarn hadoop 95 May 5 06:03 __app__.jar -> /mnt/resource/hadoop/yarn/local/filecache/10/sparkfiller-1.0-SNAPSHOT-jar-with-dependencies.jar
-rw-r--r-- 1 yarn hadoop 74 May 5 06:03 container_tokens
-rwx------ 1 yarn hadoop 710 May 5 06:03 default_container_executor_session.sh
-rwx------ 1 yarn hadoop 764 May 5 06:03 default_container_executor.sh
-rwx------ 1 yarn hadoop 6433 May 5 06:03 launch_container.sh
lrwxrwxrwx 1 yarn hadoop 102 May 5 06:03 __spark_conf__ -> /mnt/resource/hadoop/yarn/local/usercache/helixuser/filecache/80/__spark_conf__6125877397366945561.zip
lrwxrwxrwx 1 yarn hadoop 125 May 5 06:03 __spark__.jar -> /mnt/resource/hadoop/yarn/local/usercache/helixuser/filecache/81/spark-assembly-1.6.3.2.5.4.0-121-hadoop2.7.3.2.5.4.0-121.jar
drwx--x--- 2 yarn hadoop 4096 May 5 06:03 tmp
find -L . -maxdepth 5 -ls:
3933556 4 drwx--x--- 3 yarn hadoop 4096 May 5 06:03 .
3933558 4 drwx--x--- 2 yarn hadoop 4096 May 5 06:03 ./tmp
3933562 4 -rw-r--r-- 1 yarn hadoop 60 May 5 06:03 ./.launch_container.sh.crc
3933517 185944 -r-x------ 1 yarn hadoop 190402950 May 5 06:03 ./__spark__.jar
3933564 4 -rw-r--r-- 1 yarn hadoop 16 May 5 06:03 ./.default_container_executor_session.sh.crc
3933518 4 drwx------ 2 yarn hadoop 4096 May 5 06:03 ./__spark_conf__
3933548 4 -r-x------ 1 yarn hadoop 945 May 5 06:03 ./__spark_conf__/taskcontroller.cfg
3933543 4 -r-x------ 1 yarn hadoop 249 May 5 06:03 ./__spark_conf__/slaves
3933541 4 -r-x------ 1 yarn hadoop 2316 May 5 06:03 ./__spark_conf__/ssl-client.xml.example
3933520 4 -r-x------ 1 yarn hadoop 1734 May 5 06:03 ./__spark_conf__/log4j.properties
3933526 4 -r-x------ 1 yarn hadoop 265 May 5 06:03 ./__spark_conf__/hadoop-metrics2-azure-file-system.properties
3933536 4 -r-x------ 1 yarn hadoop 1045 May 5 06:03 ./__spark_conf__/container-executor.cfg
3933519 8 -r-x------ 1 yarn hadoop 5685 May 5 06:03 ./__spark_conf__/hadoop-env.sh
3933531 4 -r-x------ 1 yarn hadoop 2358 May 5 06:03 ./__spark_conf__/topology_script.py
3933547 8 -r-x------ 1 yarn hadoop 4113 May 5 06:03 ./__spark_conf__/mapred-queues.xml.template
3933528 4 -r-x------ 1 yarn hadoop 744 May 5 06:03 ./__spark_conf__/ssl-client.xml
3933544 4 -r-x------ 1 yarn hadoop 417 May 5 06:03 ./__spark_conf__/topology_mappings.data
3933549 4 -r-x------ 1 yarn hadoop 342 May 5 06:03 ./__spark_conf__/__spark_conf__.properties
3933523 4 -r-x------ 1 yarn hadoop 247 May 5 06:03 ./__spark_conf__/hadoop-metrics2-adl-file-system.properties
3933535 4 -r-x------ 1 yarn hadoop 1020 May 5 06:03 ./__spark_conf__/commons-logging.properties
3933525 24 -r-x------ 1 yarn hadoop 22138 May 5 06:03 ./__spark_conf__/yarn-site.xml
3933529 4 -r-x------ 1 yarn hadoop 2450 May 5 06:03 ./__spark_conf__/capacity-scheduler.xml
3933538 4 -r-x------ 1 yarn hadoop 2490 May 5 06:03 ./__spark_conf__/hadoop-metrics.properties
3933534 12 -r-x------ 1 yarn hadoop 8754 May 5 06:03 ./__spark_conf__/hdfs-site.xml
3933533 8 -r-x------ 1 yarn hadoop 4261 May 5 06:03 ./__spark_conf__/yarn-env.sh
3933532 4 -r-x------ 1 yarn hadoop 1335 May 5 06:03 ./__spark_conf__/configuration.xsl
3933530 4 -r-x------ 1 yarn hadoop 758 May 5 06:03 ./__spark_conf__/mapred-site.xml.template
3933545 4 -r-x------ 1 yarn hadoop 1000 May 5 06:03 ./__spark_conf__/ssl-server.xml
3933527 8 -r-x------ 1 yarn hadoop 4680 May 5 06:03 ./__spark_conf__/core-site.xml
3933522 8 -r-x------ 1 yarn hadoop 5783 May 5 06:03 ./__spark_conf__/hadoop-metrics2.properties
3933542 4 -r-x------ 1 yarn hadoop 1308 May 5 06:03 ./__spark_conf__/hadoop-policy.xml
3933540 4 -r-x------ 1 yarn hadoop 1602 May 5 06:03 ./__spark_conf__/health_check
3933537 8 -r-x------ 1 yarn hadoop 4221 May 5 06:03 ./__spark_conf__/task-log4j.properties
3933521 8 -r-x------ 1 yarn hadoop 7596 May 5 06:03 ./__spark_conf__/mapred-site.xml
3933546 4 -r-x------ 1 yarn hadoop 2697 May 5 06:03 ./__spark_conf__/ssl-server.xml.example
3933539 4 -r-x------ 1 yarn hadoop 752 May 5 06:03 ./__spark_conf__/mapred-env.sh
3932820 135852 -r-xr-xr-x 1 yarn hadoop 139105807 May 4 22:53 ./__app__.jar
3933566 4 -rw-r--r-- 1 yarn hadoop 16 May 5 06:03 ./.default_container_executor.sh.crc
3933563 4 -rwx------ 1 yarn hadoop 710 May 5 06:03 ./default_container_executor_session.sh
3933559 4 -rw-r--r-- 1 yarn hadoop 74 May 5 06:03 ./container_tokens
3933565 4 -rwx------ 1 yarn hadoop 764 May 5 06:03 ./default_container_executor.sh
3933560 4 -rw-r--r-- 1 yarn hadoop 12 May 5 06:03 ./.container_tokens.crc
3933561 8 -rwx------ 1 yarn hadoop 6433 May 5 06:03 ./launch_container.sh
broken symlinks(find -L . -maxdepth 5 -type l -ls):
下面是调用第二个 Spark 应用程序的 java 代码:
import org.apache.spark.deploy.yarn.Client;
import org.apache.spark.deploy.yarn.ClientArguments;
import org.apache.hadoop.conf.Configuration;
import org.apache.spark.SparkConf;
import org.apache.spark.SparkException;
public class CallingSparkJob {
public void submitJob(String latestreceivedpitrL,String newPtr) throws Exception {
System.out.println("In submit job method");
try{
System.out.println("Building a spark command");
// prepare arguments to be passed to
// org.apache.spark.deploy.yarn.Client object
String[] args = new String[] {
// the name of your application
"--name",
"name",
// "--master",
// "yarn",
// "--deploy-mode",
// "cluster",
// "--conf", "spark.yarn.executor.memoryOverhead=600", "--conf",
"spark.yarn.submit.waitAppCompletion=false",
// memory for driver (optional)
"--driver-memory",
"1000M",
"--num-executors",
"2",
"--executor-cores",
"2",
// path to your application's JAR file
// required in yarn-cluster mode
"--jar",
"wasb://storage_account_container@storageaccount.blob.core.windows.net/user/ankushuser/sparkfiller/sparkfiller-1.0-SNAPSHOT-jar-with-dependencies.jar",
// name of your application's main class (required)
"--class",
"com.test.SparkFiller",
// comma separated list of local jars that want
// SparkContext.addJar to work with
// "--addJars",
// "/Users/mparsian/zmp/github/data-algorithms-book/lib/spark-assembly-1.5.2-hadoop2.6.0.jar,/Users/mparsian/zmp/github/data-algorithms-book/lib/log4j-1.2.17.jar,/Users/mparsian/zmp/github/data-algorithms-book/lib/junit-4.12-beta-2.jar,/Users/mparsian/zmp/github/data-algorithms-book/lib/jsch-0.1.42.jar,/Users/mparsian/zmp/github/data-algorithms-book/lib/JeraAntTasks.jar,/Users/mparsian/zmp/github/data-algorithms-book/lib/jedis-2.5.1.jar,/Users/mparsian/zmp/github/data-algorithms-book/lib/jblas-1.2.3.jar,/Users/mparsian/zmp/github/data-algorithms-book/lib/hamcrest-all-1.3.jar,/Users/mparsian/zmp/github/data-algorithms-book/lib/guava-18.0.jar,/Users/mparsian/zmp/github/data-algorithms-book/lib/commons-math3-3.0.jar,/Users/mparsian/zmp/github/data-algorithms-book/lib/commons-math-2.2.jar,/Users/mparsian/zmp/github/data-algorithms-book/lib/commons-logging-1.1.1.jar,/Users/mparsian/zmp/github/data-algorithms-book/lib/commons-lang3-3.4.jar,/Users/mparsian/zmp/github/data-algorithms-book/lib/commons-lang-2.6.jar,/Users/mparsian/zmp/github/data-algorithms-book/lib/commons-io-2.1.jar,/Users/mparsian/zmp/github/data-algorithms-book/lib/commons-httpclient-3.0.1.jar,/Users/mparsian/zmp/github/data-algorithms-book/lib/commons-daemon-1.0.5.jar,/Users/mparsian/zmp/github/data-algorithms-book/lib/commons-configuration-1.6.jar,/Users/mparsian/zmp/github/data-algorithms-book/lib/commons-collections-3.2.1.jar,/Users/mparsian/zmp/github/data-algorithms-book/lib/commons-cli-1.2.jar,/Users/mparsian/zmp/github/data-algorithms-book/lib/cloud9-1.3.2.jar",
// argument 1 for latestreceivedpitrL
"--arg",
latestreceivedpitrL,
// argument 2 for newPtr
"--arg",
newPtr,
"--arg",
"yarn-cluster"
};
System.out.println("create a Hadoop Configuration object");
// create a Hadoop Configuration object
Configuration config = new Configuration();
// identify that you will be using Spark as YARN mode
System.setProperty("SPARK_YARN_MODE", "true");
// create an instance of SparkConf object
SparkConf sparkConf = new SparkConf();
sparkConf.setSparkHome("/usr/hdp/current/spark-client");
// sparkConf.setMaster("yarn");
sparkConf.setMaster("yarn-cluster");
// sparkConf.setAppName("spark-yarn");
// sparkConf.set("master", "yarn");
// sparkConf.set("spark.submit.deployMode", "cluster"); // worked
// create ClientArguments, which will be passed to Client
// ClientArguments cArgs = new ClientArguments(args);
ClientArguments cArgs = new ClientArguments(args, sparkConf);
// create an instance of yarn Client client
Client client = new Client(cArgs, config, sparkConf);
// submit Spark job to YARN
client.run();
}catch(Exception e){
System.out.println("Error submitting spark Job");
System.out.println(e.getMessage());
}
}
}
用于在本地运行第一个 spark 应用程序的 spark-submit 命令:
spark-submit --class scala1 --master yarn --deploy-mode cluster --num-executors 2 --executor-cores 2 --conf spark.yarn.executor.memoryOverhead=600 --conf spark.yarn.submit.waitAppCompletion=false /home/ankushuser/kafka_retry/kafka_retry_test/sparkflightaware/target/sparkflightaware-0.0.1-SNAPSHOT-jar-with-dependencies.jar
如果我在本地运行此 spark-submit 命令,它会调用 java 类,并且第二个 scala2 应用程序的 spark-submit 命令工作正常。
如果我在 yarn 模式下运行它,那么我就会遇到这个问题。
感谢您的帮助。
最佳答案
因为有赏金,我也会将其作为回复重新发布——但实际上我想将其标记为重复,因为实际的异常是另一个问题中涵盖的异常,并回答:
或者,使用
--driver-java-options="-Dhdp.version=INSERT_VERSION_STRING_HERE"--conf "spark.executor.extraJavaOptions=-Dhdp.version=INSERT_VERSION_STRING_HERE"
在你的 spark-提交
并确保使用正确的版本字符串,如在 /usr/hdp.
如果您想坚持从您的代码中调用 client.submit
,那么您需要将这些行放入您在代码中构建的 --arg
中。
关于java - 无法在集群上使用 java 类调用 spark 应用程序,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/43798443/
我正在编写一个具有以下签名的 Java 方法。 void Logger(Method method, Object[] args); 如果一个方法(例如 ABC() )调用此方法 Logger,它应该
我是 Java 新手。 我的问题是我的 Java 程序找不到我试图用作的图像文件一个 JButton。 (目前这段代码什么也没做,因为我只是得到了想要的外观第一的)。这是我的主课 代码: packag
好的,今天我在接受采访,我已经编写 Java 代码多年了。采访中说“Java 垃圾收集是一个棘手的问题,我有几个 friend 一直在努力弄清楚。你在这方面做得怎么样?”。她是想骗我吗?还是我的一生都
我的 friend 给了我一个谜语让我解开。它是这样的: There are 100 people. Each one of them, in his turn, does the following
如果我将使用 Java 5 代码的应用程序编译成字节码,生成的 .class 文件是否能够在 Java 1.4 下运行? 如果后者可以工作并且我正在尝试在我的 Java 1.4 应用程序中使用 Jav
有关于why Java doesn't support unsigned types的问题以及一些关于处理无符号类型的问题。我做了一些搜索,似乎 Scala 也不支持无符号数据类型。限制是Java和S
我只是想知道在一个 java 版本中生成的字节码是否可以在其他 java 版本上运行 最佳答案 通常,字节码无需修改即可在 较新 版本的 Java 上运行。它不会在旧版本上运行,除非您使用特殊参数 (
我有一个关于在命令提示符下执行 java 程序的基本问题。 在某些机器上我们需要指定 -cp 。 (类路径)同时执行java程序 (test为java文件名与.class文件存在于同一目录下) jav
我已经阅读 StackOverflow 有一段时间了,现在我才鼓起勇气提出问题。我今年 20 岁,目前在我的家乡(罗马尼亚克卢日-纳波卡)就读 IT 大学。足以介绍:D。 基本上,我有一家提供簿记应用
我有 public JSONObject parseXML(String xml) { JSONObject jsonObject = XML.toJSONObject(xml); r
我已经在 Java 中实现了带有动态类型的简单解释语言。不幸的是我遇到了以下问题。测试时如下代码: def main() { def ks = Map[[1, 2]].keySet()
一直提示输入 1 到 10 的数字 - 结果应将 st、rd、th 和 nd 添加到数字中。编写一个程序,提示用户输入 1 到 10 之间的任意整数,然后以序数形式显示该整数并附加后缀。 public
我有这个 DownloadFile.java 并按预期下载该文件: import java.io.*; import java.net.URL; public class DownloadFile {
我想在 GUI 上添加延迟。我放置了 2 个 for 循环,然后重新绘制了一个标签,但这 2 个 for 循环一个接一个地执行,并且标签被重新绘制到最后一个。 我能做什么? for(int i=0;
我正在对对象 Student 的列表项进行一些测试,但是我更喜欢在 java 类对象中创建硬编码列表,然后从那里提取数据,而不是连接到数据库并在结果集中选择记录。然而,自从我这样做以来已经很长时间了,
我知道对象创建分为三个部分: 声明 实例化 初始化 classA{} classB extends classA{} classA obj = new classB(1,1); 实例化 它必须使用
我有兴趣使用 GPRS 构建车辆跟踪系统。但是,我有一些问题要问以前做过此操作的人: GPRS 是最好的技术吗?人们意识到任何问题吗? 我计划使用 Java/Java EE - 有更好的技术吗? 如果
我可以通过递归方法反转数组,例如:数组={1,2,3,4,5} 数组结果={5,4,3,2,1}但我的结果是相同的数组,我不知道为什么,请帮助我。 public class Recursion { p
有这样的标准方式吗? 包括 Java源代码-测试代码- Ant 或 Maven联合单元持续集成(可能是巡航控制)ClearCase 版本控制工具部署到应用服务器 最后我希望有一个自动构建和集成环境。
我什至不知道这是否可能,我非常怀疑它是否可能,但如果可以,您能告诉我怎么做吗?我只是想知道如何从打印机打印一些文本。 有什么想法吗? 最佳答案 这里有更简单的事情。 import javax.swin
我是一名优秀的程序员,十分优秀!