java - hdfs 显示本地文件列表

Question

我在 OS X 中安装了 Hadoop，运行良好。我的经验是最近的，我正在努力学习更多关于使用 Hadoop 进行应用程序开发的知识。

昨天，当我需要在 Hadoop 中查找目录和/或文件列表时，我只需键入

$ hadoop fs -ls 

而且，它会向我显示集群中的所有内容。

今天，它显示了文件系统中的所有本地内容。我必须提供 hdfs 的确切地址才能获取内容列表，

$ hadoop fs -ls hdfs://localhost:8020/user/myName

我的core-site.xml文件和之前一样，

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<!-- Put site-specific property overrides in this file. -->
 <configuration>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/usr/local/Cellar/hadoop/hdfs/tmp</value>
        <description>A base for other temporary directories.</description>
    </property>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://localhost:8020</value>
    </property>
</configuration>

在启动 hadoop 守护进程之前，我停止了集群并再次使用以下命令重新格式化了分布式文件系统，以便我们可以在执行 map-reduce 作业时将数据源放入 hdfs 文件系统

$ hdfs namenode -format

我收到管理报告，通知文件系统 file:///不是 HDFS 文件系统，

$ hadoop dfsadmin -report
WARNING: Use of this script to execute dfsadmin is deprecated.
WARNING: Attempting to execute replacement "hdfs dfsadmin" instead.

2018-10-18 18:01:27,316 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
report: FileSystem file:/// is not an HDFS file system
Usage: hdfs dfsadmin [-report] [-live] [-dead] [-decommissioning] [-enteringmaintenance] [-inmaintenance]

在core-site.xml文件中，我也更新了如下配置，

<property>
    <!-- <name>fs.default.name</name> -->
    <!-- <value>hdfs://localhost:8020</value> -->
    <name>fs.defaultFS</name>
    <value>hdfs://localhost.localdomain:8020/</value>
</property>

那时我已经改过自新，这并没有改变什么。正如另一个答案提到的，haddop home 已经在 ~/.bashrc 文件中提供，

export HADOOP_HOME=/Users/chaklader/hadoop
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

如何切换到HDFS文件系统？任何类型的建议将不胜感激。

Answer 1

您需要确保已添加名为 HADOOP_CONF_DIR 的环境变量，并将其设置为包含来自 Hadoop 的 XML 文件的目录。

您可以在主文件夹中的 .bashrc 中执行此操作

---

否则，您将获得默认文件系统 file://，它仍然有效并且仍然可以正常运行 MapReduce 作业

---

FWIW，这是我的核心网站

$ cat /usr/local/Cellar/hadoop/3.1.1/libexec/etc/hadoop/core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>file:///tmp/hadoop/hdfs/tmp</value>
        <description>A base for other temporary directories.</description>
    </property>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
  </property>
</configuration>

和hdfs站点

$ cat /usr/local/Cellar/hadoop/3.1.1/libexec/etc/hadoop/hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
  <property>
      <name>dfs.namenode.name.dir</name>
      <value>file:///tmp/hadoop/hdfs/names</value>
  </property>
  <property>
    <name>fs.checkpoint.dir</name>
    <value>file:///tmp/hadoop/hdfs/checkpoint</value>
  </property>
  <property>
    <name>fs.checkpoint.edits.dir</name>
    <value>file:///tmp/hadoop/hdfs/checkpoint-edits</value>
  </property>
  <property>
      <name>dfs.datanode.data.dir</name>
      <value>file:///tmp/hadoop/hdfs/data</value>
  </property>
</configuration>