hadoop - 无法确定 Hadoop 版本信息

Question

我已经在 ubuntu 上安装了 hadoop，它运行良好。

ubuntu:/home/hduser/hive-0.10.0-cdh4.3.1$ jps
2702 DataNode
3101 ResourceManager
4879 Jps
2948 SecondaryNameNode
3306 NodeManager

hadoop_version=Hadoop 2.0.0-cdh4.3.0

然后我从 apache tarballs 安装了 hive(hiv version-hive-0.10.0) 并尝试运行 bin/hive。但是我遇到以下错误:

无法确定 Hadoop 版本信息。hadoop 版本 返回:

/home/hduser/hadoop/etc/hadoop /usr/lib/jvm/jdk1.6.0_45/ 
Hadoop 2.0.0-cdh4.3.0
Subversion file:///var/lib/jenkins/workspace/CDH4.3.0-Packaging-Hadoop/build/cdh4/hadoop/2.0.0-cdh4.3.0/source/hadoop-common-project/hadoop-common -r 48a9315b342ca16de92fcc5be95ae3650629155a 
Compiled by jenkins on Mon May 27 19:06:57 PDT 2013 
From source with checksum a4218d77f9b12df4e3e49ef96f9d357d 
This command was run using /home/hduser/hadoop/share/hadoop/common/hadoop-common-2.0.0-cdh4.3.0.jar

我尝试通过我的脚本知识解决它，但不能。当我竭尽全力时，我发现它在下面一行失败了:

if [[ "$HADOOP_VERSION" =~ $hadoop_version_re ]]; then

我尝试echo HADOOP_VERSION 它什么也没返回，HADOOP_VERSION 被定义为

HADOOP_VERSION=$($HADOOP version | awk '{if (NR == 1) {print $2;}}');

和$HADOOP 版本 产生我

 /home/hduser/hadoop/etc/hadoop
 /usr/lib/jvm/jdk1.6.0_45/
 Hadoop 2.0.0-cdh4.3.0
 Subversion file:///var/lib/jenkins/workspace/CDH4.3.0-Packaging-Hadoop/build/cdh4/hadoop/2.0.0-cdh4.3.0/source/hadoop-common-project/hadoop-common -r 48a9315b342ca16de92fcc5be95ae3650629155a
 Compiled by jenkins on Mon May 27 19:06:57 PDT 2013
 From source with checksum a4218d77f9b12df4e3e49ef96f9d357d
 This command was run using /home/hduser/hadoop/share/hadoop/common/hadoop-common-2.0.0-cdh4.3.0.jar

我为此震惊了一个星期。请帮帮我。谢谢。

Answer 1

您的问题已经描述了问题。当脚本执行 $HADOOP version 时，它期望输出如下:

Hadoop 2.0.0-cdh4.3.0
Subversion file:///var/lib/jenkins/workspace/CDH4.3.0-Packaging-Hadoop/build/cdh4/hadoop/2.0.0-cdh4.3.0/source/hadoop-common-project/hadoop-common -r 48a9315b342ca16de92fcc5be95ae3650629155a
Compiled by jenkins on Mon May 27 19:06:57 PDT 2013
 From source with checksum a4218d77f9b12df4e3e49ef96f9d357d
 This command was run using /home/hduser/hadoop/share/hadoop/common/hadoop-common-2.0.0-cdh4.3.0.jar`

取而代之的是一些其他的输出(可能是因为您修改了 Hadoop 中的一些脚本。检查 conf/hadoop-env.sh):

/home/hduser/hadoop/etc/hadoop
/usr/lib/jvm/jdk1.6.0_45/
Hadoop 2.0.0-cdh4.3.0
Subversion file:///var/lib/jenkins/workspace/CDH4.3.0-Packaging-Hadoop/build/cdh4/hadoop/2.0.0-cdh4.3.0/source/hadoop-common-project/hadoop-common -r 48a9315b342ca16de92fcc5be95ae3650629155a
Compiled by jenkins on Mon May 27 19:06:57 PDT 2013
 From source with checksum a4218d77f9b12df4e3e49ef96f9d357d
 This command was run using /home/hduser/hadoop/share/hadoop/common/hadoop-common-2.0.0-cdh4.3.0.jar`

现在 awk 行不再找到所需的数字(在位置 2)。

因此解决方案是找出额外输出的来源并将其删除。