gpt4 book ai didi

hadoop - 纱 : Could not find or load main class org. apache.hadoop.mapreduce.v2.app.MRAppMaster

转载 作者:可可西里 更新时间:2023-11-01 15:10:09 29 4
gpt4 key购买 nike

尝试在 Fedora 上执行任何 M/R2 作业时出现此异常。 Hadoop 2.7.3 和 2.8.0 有同样的问题。这包括 Hive。

[hadoop@master hadoop]$ yarn classpath
/opt/hadoop/hadoop-2.7.3/conf
/opt/hadoop/hadoop-2.7.3/conf
/opt/hadoop/hadoop-2.7.3/conf:/opt/hadoop/hadoop/share/hadoop/common/lib/*
/opt/hadoop/hadoop/share/hadoop/common/*
/opt/hadoop/hadoop/share/hadoop/hdfs
/opt/hadoop/hadoop/share/hadoop/hdfs/lib/*
/opt/hadoop/hadoop/share/hadoop/hdfs/*
/opt/hadoop/hadoop/share/hadoop/yarn/lib/*
/opt/hadoop/hadoop/share/hadoop/yarn/*
/opt/hadoop/hadoop/share/hadoop/mapreduce/share/hadoop/mapreduce/*
/opt/hadoop/hadoop/contrib/capacity-scheduler/*.jar
/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7.3.jar
/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.3.jar
/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.3.jar
/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7.3.jar
/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.3.jar
/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.3.jar
/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar
/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.3.jar
/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar
/opt/hadoop/hadoop/share/hadoop/yarn/*
/opt/hadoop/hadoop/share/hadoop/yarn/lib/*

mapred-site.xml

<configuration>

<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>

<property>
<name>mapreduce.application.classpath</name>
<value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
</property>

</configuration>

和 yarn-site.xml

<configuration>

<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>

<property>
<name>yarn.application.classpath</name>
<value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
</property>

</configuration>

最后但同样重要的是,环境设置:

export HADOOP_HOME=/opt/hadoop/hadoop
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_YARN_HOME=$HADOOP_HOME
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

我很确定我遗漏了一些明显的东西。我已经设置了多次,但一定是有什么问题。

缺少的类实际上是类路径中jar 的一部分。/opt/hadoop/hadoop 文件夹归用户 hadoop 所有,并具有所需的所有访问权限。

最佳答案

设置 YARN_HOME=$HADOOP_HOME 而不是 HADOOP_YARN_HOME=$HADOOP_HOME

关于hadoop - 纱 : Could not find or load main class org. apache.hadoop.mapreduce.v2.app.MRAppMaster,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/44751407/

29 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com