gpt4 book ai didi

hadoop - Pig `Failed to create DataStorage`到Docker ENV中的Hadoop集群

转载 作者:行者123 更新时间:2023-12-02 21:30:25 25 4
gpt4 key购买 nike

我正在尝试在本地计算机(OSX)中运行Pig和Hadoop来测试脚本,但似乎无法在Docker中连接Hadoop。

我在本地设置了Pig。我从https://hub.docker.com/r/sequenceiq/hadoop-docker/安装了Hadoop,并使用此命令运行计算机

docker run -it sequenceiq/hadoop-docker:2.7.0 -p 50010:50010 -p 19888:19888 /etc/bootstrap.sh -bash

我可以启动Hadoop docker 。

我查看了 Dockerfile以查看暴露了哪些端口。 https://github.com/sequenceiq/hadoop-docker/blob/master/Dockerfile#L103

这是我的 hadoop-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/tmp/hadoop-${user.name}</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://192.168.99.100:50010</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>hdfs://192.168.99.100:19888</value>
</property>
<property>
<name>dfs.replication</name>
<value>8</value>
</property>
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx512m</value>
</property>
</configuration>

当我运行 Pig时,出现此错误。
e.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
15/12/28 12:29:04 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL
15/12/28 12:29:04 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE
15/12/28 12:29:04 INFO pig.ExecTypeProvider: Picked MAPREDUCE as the ExecType
2015-12-28 12:29:04,982 [main] INFO org.apache.pig.Main - Apache Pig version 0.15.0 (r1682971) compiled Jun 01 2015, 11:43:55
2015-12-28 12:29:04,982 [main] INFO org.apache.pig.Main - Logging error messages to: /Users/ncharass/opensource/pig-0.15.0/conf/pig_1451323744980.log
2015-12-28 12:29:04,995 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /Users/ncharass/.pigbootup not found
2015-12-28 12:29:05,141 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://192.168.99.100:50010
2015-12-28 12:29:05,371 [main] ERROR org.apache.pig.Main - ERROR 2999: Unexpected internal error. Failed to create DataStorage
Details at logfile: /Users/ncharass/opensource/pig-0.15.0/conf/pig_1451323744980.log
2015-12-28 12:29:05,383 [main] INFO org.apache.pig.Main - Pig script completed in 468 milliseconds (468 ms)

我有什么想念的吗?

最佳答案

Docker在OSX(和Windows)上的虚拟机中运行。

暴露docker端口与VirtualBox中的转发端口不同。

您是否已在VirtualBox中的“设置”>“网络”>“高级”>“端口转发”下转发了适当的端口?

关于hadoop - Pig `Failed to create DataStorage`到Docker ENV中的Hadoop集群,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/34498211/

25 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com