Hadoop fs -put 有 DataStreamer 异常 java.nio.channels.UnresolvedAddressException

Question

我已经启动了以下 docker 容器

namenode  
datanode1  
datanode2  
datanode3

当我尝试按如下方式将文件放入 HDFS 时出现此错误

hdfs dfs -put /tmp/airline-2008.csv.bz2 hdfs://namenode.bigdatacluster.com:8020/data/

17/09/28 22:49:01 WARN hdfs.DataStreamer: DataStreamer Exception java.nio.channels.UnresolvedAddressException at sun.nio.ch.Net.checkAddress(Net.java:101) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.hdfs.DataStreamer.createSocketForPipeline(DataStreamer.java:226) at org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1591) at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1547) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:658) put: java.nio.channels.UnresolvedAddressException

Answer 1

来自 Oracle java 文档，

Unchecked exception thrown when an attempt is made to invoke a network operation upon an unresolved socket address

这意味着 hdfs URL 中使用的主机名 namenode.bigdatacluster.com 未解析为 IP 地址。您可以通过将主机到 IP 的映射添加到 /etc/hosts 文件来解决此问题。