apache - 带有YARN的HBase引发错误

Question

我在Ubuntu 14.04上将Hadoop 2.5.1与HBase 0.98.11结合使用

我可以在伪分布式模式下运行它。现在，我想在分布式模式下运行。我按照站点上的说明进行操作，最终在RUNTIME中出现一个错误，错误为“错误:org / apache / hadoop / hbase / HBaseConfiguration”(编译代码时没有错误)。

经过尝试后，我发现，如果我在mapred-site.xml中注释了mapreduce.framework.name，并且还在yarn-site中注释了东西，则可以成功运行hadoop。

但是我认为这是单节点运行(我不知道，只是通过将运行时间与我在Pseudo中运行的时间进行比较来猜测，并且在主节点上运行作业时从节点的jps中没有MR)。

这是我的一些conf:

hdfs站点

<property>
<name>dfs.replication</name>
<value>2</value>
<description>Default block replication.
The actual number of replications can be specified when the file is created.
The default is used if replication is not specified in create time.
</description>
</property>
<!-- <property>
<name>dfs.namenode.name.dir</name>
<value>file:/usr/local/hadoop_store/hdfs/namenode</value>
</property>-->
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/usr/local/hadoop_store/hdfs/datanode</value>
</property>

<property>
<name>dfs.datanode.use.datanode.hostname</name>
<value>false</value>
</property>

<property>
<name>dfs.permissions</name>
<value>false</value>
</property>

映射站点

 <property>
   <name>mapred.job.tracker</name>
   <value>localhost:54311</value>
   <description>The host and port that the MapReduce job tracker runs
   at.  If "local", then jobs are run in-process as a single map
   and reduce task.
   </description>
 </property>

 <!--<property>
   <name>mapreduce.framework.name</name>
   <value>yarn</value>
 </property>-->

yarn 现场

 <!-- Site specific YARN configuration properties -->

 <!--<property>
     <name>yarn.nodemanager.aux-services</name>
     <value>mapreduce_shuffle</value>
 </property>
 <property>
     <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
     <value>org.apache.hadoop.mapred.ShuffleHandler</value>
 </property>
 <property>
     <name>yarn.resourcemanager.address</name>
     <value>10.1.1.177:8032</value>
 </property>
 <property>
     <name>yarn.resourcemanager.scheduler.address</name>
     <value>10.1.1.177:8030</value>
 </property>
 <property>
     <name>yarn.resourcemanager.resource-tracker.address</name>
     <value>10.1.1.177:8031</value>
 </property>-->

非常感谢您的帮助

更新:我尝试通过像这样添加yarn.applicaton.classpath来对 yarn 位置进行一些更改

https://dl-web.dropbox.com/get/Public/yarn.png?_subject_uid=51053996&w=AABeDJfRp_D31RiVHqBWn0r9naQR_lFVJXIlwvCwjdhCAQ

错误更改为退出代码。

https://dl-web.dropbox.com/get/Public/exitcode.jpg?_subject_uid=51053996&w=AAAQ-bYoRSrQV3yFq36vEDPnAB9aIHnyOQfnvt2cUHn5IQ

UPDATE2:在应用程序日志的syslog中，它说

2015-04-24 20:34:59,164 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1429792550440_0035_000002 2015-04-24 20:34:59,589 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 2015-04-24 20:34:59,610 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 2015-04-24 20:34:59,616 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster java.lang.NoSuchMethodError: org.apache.hadoop.http.HttpConfig.setPolicy(Lorg/apache/hadoop/http/HttpConfig$Policy;)V at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1364) 2015-04-24 20:34:59,621 INFO [Thread-1] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster received a signal. Signaling RMCommunicator and JobHistoryEventHandler.

任何建议请

Answer 1

我猜您没有正确设置hadoop集群，请按照以下步骤操作:

Hadoop配置:

步骤1:按以下方式编辑hadoop-env.sh:

# The java implementation to use.  Required.
export JAVA_HOME=/usr/lib/jvm/java-6-sun

步骤2:现在创建目录并设置所需的所有权和权限

$ sudo mkdir -p /app/hadoop/tmp
$ sudo chown hduser:hadoop /app/hadoop/tmp
# ...and if you want to tighten up security, chmod from 755 to 750...
$ sudo chmod 750 /app/hadoop/tmp

步骤3:编辑 core-site.xml

<property>
  <name>hadoop.tmp.dir</name>
  <value>/app/hadoop/tmp</value>
</property>

<property>
  <name>fs.default.name</name>
  <value>hdfs://localhost:54310</value>
</property>

步骤5:编辑 mapred-site.xml

  <property>
      <name>mapreduce.framework.name</name>
      <value>yarn</value>
    </property>

步骤6:编辑 hdfs-site.xml

<property>
  <name>dfs.replication</name>
          <value>1</value>
</property>

<property>
  <name>dfs.name.dir</name>
  <value>file:///home/hduser/hadoopdata/hdfs/namenode</value>
</property>

<property>
  <name>dfs.data.dir</name>
  <value>file:///home/hduser/hadoop/hadoopdata/hdfs/datanode</value>
</property>

步骤7:编辑 yarn-site.xml

<property>
   <name>yarn.nodemanager.aux-services</name>
   <value>mapreduce_shuffle</value>
</property>

最终格式化HDFS(第一次设置Hadoop集群时需要执行此操作)

$ /usr/local/hadoop/bin/hadoop namenode -format

Hbase配置:

编辑你 hbase-site.xml:

<property>
  <name>hbase.rootdir</name>
  <value>hdfs://localhost:54310/hbase</value>
</property> 

<property>
   <name>hbase.cluster.distributed</name>
   <value>true</value>
</property>

<property>
   <name>hbase.zookeeper.quorum</name>  
   <value>localhost</value>
</property>

<property>   
  <name>dfs.replication</name>   
  <value>1</value>    
</property>

 <property>       
  <name>hbase.zookeeper.property.clientPort</name>  
  <value>2181</value>                                                                                 
 </property>

 <property>
    <name>hbase.zookeeper.property.dataDir</name>
    <value>/usr/local/hbase/zookeeper</value>   
 </property>                                                                                                                          

希望这对您有帮助