hadoop - 谷歌云引擎 : LibSnappy not installed errur during command-line installation of Hadoop

Question

我正在尝试使用 command line option 在 Google Compute Engine 上安装自定义 Hadoop 实现 (>2.0) .我的bdutil_env.sh文件修改参数如下:

GCE_IMAGE='ubuntu-14-04'
GCE_MACHINE_TYPE='n1-standard-1'
GCE_ZONE='us-central1-a'
DEFAULT_FS='hdfs'
HADOOP_TARBALL_URI='gs://<mybucket>/<my_hadoop_tar.gz>'

./bdutil 部署失败，退出代码为 1。我在生成的 debug.info 文件中发现以下错误:

    ssh: connect to host 130.211.161.181 port 22: Connection refused
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
ssh: connect to host 104.197.63.39 port 22: Connection refused
ssh: connect to host 104.197.7.106 port 22: Connection refused
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
.....
.....
Connection to 104.197.7.106 closed.
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [123].
Connection to 104.197.63.39 closed.
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [123].
Connection to 130.211.161.181 closed.
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [123].
...
...
hadoop-w-1: ==> deploy-core-setup_deploy.stderr <==
....
....
hadoop-w-1: dpkg-query: package 'libsnappy1' is not installed and no information is available
hadoop-w-1: Use dpkg --info (= dpkg-deb --info) to examine archive files,
hadoop-w-1: and dpkg --contents (= dpkg-deb --contents) to list their contents.
hadoop-w-1: dpkg-preconfigure: unable to re-open stdin: No such file or directory
hadoop-w-1: dpkg-query: package 'libsnappy-dev' is not installed and no information is available
hadoop-w-1: Use dpkg --info (= dpkg-deb --info) to examine archive files,
hadoop-w-1: and dpkg --contents (= dpkg-deb --contents) to list their contents.
hadoop-w-1: dpkg-preconfigure: unable to re-open stdin: No such file or directory
hadoop-w-1: ./hadoop-env-setup.sh: line 612: Package:: command not found
....
....
hadoop-w-1: find: `/home/hadoop/hadoop-install/lib': No such file or directory

不明白为什么初始ssh报错；我可以看到虚拟机并从用户界面正确登录；我的 tar.gz 也被复制到适当的地方。

我也不明白为什么没有安装 libsnappy；我有什么特别需要做的吗？ shell 脚本似乎有安装它的命令，但它以某种方式失败了。

我检查了所有虚拟机； Hadoop 未启动。

编辑:为了解决 ssh 问题，我运行了以下命令:

gcutil --project= addfirewall --allowed=tcp:22 default-ssh

没有区别。

Answer 1

在这种情况下，ssh 和 libsnappy 错误是转移注意力的错误；当 VM 不能立即支持 SSH 时，bdutil 会轮询一段时间，直到它应该打印出如下内容:

...Thu May 14 16:52:23 PDT 2015: Waiting on async 'wait_for_ssh' jobs to finish. Might take a while...
...
Thu May 14 16:52:33 PDT 2015: Instances all ssh-able

同样，您看到的 libsnappy 错误是一个转移注意力的错误，因为它来自对 dpkg -s 的调用，试图确定是否确实安装了一个包，如果没有，则进行 apt-get install它:https://github.com/GoogleCloudPlatform/bdutil/blob/master/libexec/bdutil_helpers.sh#L163

我们将努力清理这些错误消息，因为它们可能具有误导性。与此同时，这里的主要问题是 Ubuntu 一直以来都不是 bdutil 支持的镜像之一；我们彻底验证了 CentOS 和 Debian 镜像，但没有验证 Ubuntu 镜像，因为它们只有 added as GCE options 2014 年 11 月。您的部署应该可以与任何 debian-7 或 centos-6 镜像的自定义 tarball 一起正常工作。我们在 GitHub 上提交了一个问题来跟踪 Ubuntu 对 bdutil 的支持:https://github.com/GoogleCloudPlatform/bdutil/issues/29

编辑:问题有 been resolved现在在主存储库中支持 Ubuntu；您可以在最近的提交中下载 here .