hadoop - Hadoop安装与设置

Question

我正在尝试在CentOS中自行设置和安装hadoop集群环境。 TaskTracker没有运行。执行start-all.sh后消失

有人可以帮助我如何安装hadoop单节点/多节点集群吗？

Answer 1

Can you try this and let me know if you face any issues while installing the hadooop.


Pre-requiste
1.  Ensure Hosts file  – changes (master, slave) – no loopback , no local host
2.  Ensure Sshd runs
3.  Ensure hadoop Sudo ers – No passwd authentication
4.  Ensure selinux is disabled (not permissive) for user, policies and contexts
5.  Ensure Reboot - Pings masters and slaves.
6.  Ensure JDK 7 is installed in master and slaves.
7.  Ensure firewall is off or iptables allows ports mentioned by hadoop
8.  Ensure IPV6 is disabled or export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true in hadoop_env.sh

In Master for seamless cut to slaves

Generate SHH - password less access
ssh-keygen -t rsa
$ ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@master
$ ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@slave
mkdir /opt/hadoop
# cd /opt/hadoop/
gunzip and untar the hadoop ball under /opt/hadoop/hadoop

Ensure hadoop owns the /opt directory
Changes in xml files under /opt/hadoop/hadoop/conf
core-site.xml
#Add the following inside the configuration tag
<property>
<name>fs.default.name</name>
<value>hdfs://master:9000/</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>







hdfs-site.xml
# Add the following inside the configuration tag
<property>
<name>dfs.data.dir</name>
<value>/opt/hadoop/hadoop/dfs/name/data</value>
<final>true</final>
</property>
<property>
<name>dfs.name.dir</name>
<value>/opt/hadoop/hadoop/dfs/name</value>
<final>true</final>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
mapred-site.xml
<property>
<name>mapred.job.tracker</name>
<value>hadoop-master:9001</value>
</property>
conf/hadoop-env.sh
export JAVA_HOME=/opt/jdk1.7.0_45
export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true
export HADOOP_CONF_DIR=/opt/hadoop/hadoop/conf
Ship this cut to slaves ------------------------------>

In Master cut only (under conf)
$ cat masters
hadoop-master
$ cat slaves`enter code here`
Slave
$ bin/hadoop namenode –format
$ bin/start-dfs.sh`enter code here`
$ bin/start-mapred.sh