hadoop之旅1

mac下安装Hadoop

主要是介绍在Mac系统下安装Hadoop的相关步骤，包含：

• 安装包的下载：JDK和``Hadoop`
• JDK 的安装和配置
• SSH的配置，实现免密登陆
• hadoop的安装与配置

hadoop集群搭建

基于Hadoop2.9.2使用云服务器搭建hadoop集群

安装包下载

1. 下载JDK

2. 下载Hadoop

   

主机名和IP的映射关系实现

编辑/etc/hosts文件

>> vim /etc/hosts

# 127.0.0.1
# 写入主机IP地址和名称
172.20.18.3 peter      

>> reboot  # 重启生效

将安装包移动到指定目录下

cp hadoop-2.7.2.tar.gz /opt/    # 移动到opt目录下
cp jdk-8u010-macos.x64.dmg /opt/

安装和配置JDK

• 先确定JDK的安装目录/usr/java/default

• 编辑文件

  # 查看JDK的安装目录
  >> /usr/libexec/java_home -V
  # 一般是这个目录
  /Library/Java/JavaVirtualMachines/jdk1.8.0_231.jdk/Contents/Home   
  
  >> vim /etc/profile
  # 写入内容
  JAVA_HOME = /usr/java/default
  
  # 重启生效!切记
  >> source /etc/profile
  
  # 验证
  >> echo $JAVA_HOME
  >> JAVA -version

配置SSH

配置SSH的主要目的是实现免密登陆

>> ssh-keygen -t rsa
>> ll .ssh/
>> cd .ssh/
>> cat id_rsa.pub >> authorized-keys
>> chmod 644 authorized-keys
>> ssh peter/ip地址  # 通过主机名或者IP地址可以直接登录

安装和配置Hadoop

1. 解压安装包

   tar zxf hadoop-2.7.2.tar.gz

2. 确定安装路径，配置环境变量

   >> vim /etc/profile
   JAVA_HOME = /usr/java/default
   # 增加内容
   HADOOP_HOME = /opt/hadoop-2.7.2
   
   PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$PATH

3. 配置生效

   >> source /etc/profile
   >> echo $HADOOP_HOME

   

4. 进行安装目录，进行配置文件的修改

   • core-site.xml
   • hdfs-site.xml
   • yarn-site.xml
   • mapred-site.xml
   • slaves
   • Hadoop-env.sh

   

core-site.xml

# 具体内容
<configuration> 
  <property>
     <name>fs.default.name</name>
     <value>hdfs://localhost:9000</value>
  </property>
  
  <property>
     <name>hadoop.tmp.dir</name>
     <value>/Users/piqianchao/Downloads/hadoop-2.9.2/current/tmp</value>
 </property> 
  <property>
     <name>fs.trash.interval</name>
     <value>4320</value>
  </property>
 
</configuration>

hdfs-site.xml

20 <configuration>
21
22 <property>
23    <name>dfs.namenode.name.dir</name>
24    <value>/Users/piqianchao/Downloads/hadoop-2.9.2/current/dfs/name</value>
25  </property>
26  <property>
27    <name>dfs.datanode.data.dir</name>
28    <value>/Users/piqianchao/Downloads/hadoop-2.9.2/current/data</value>
29  </property>
30  <property>
31    <name>dfs.replication</name>
32    <value>3</value>
33  </property>
34  <property>                                                                                                                   
35    <name>dfs.webhdfs.enabled</name>
36    <value>true</value>
37  </property>
38  <property>
39    <name>dfs.permissions.superusergroup</name>
40    <value>staff</value>
41  </property>
42  <property>
43    <name>dfs.permissions.enabled</name>
44    <value>false</value>
45  </property>
46 
47 </configuration>

yarn-site.xml

15 <configuration>
16 
17 <!-- Site specific YARN configuration properties -->
18 
19 <property>
20    <name>yarn.resourcemanager.hostname</name>
21    <value>localhost</value>
22  </property>
23  <property>
24    <name>yarn.nodemanager.aux-services</name>
25    <value>mapreduce_shuffle</value>
26  </property>
27  <property>
28    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
29    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
30  </property>
31  <property>
32    <name>yarn.resourcemanager.address</name>
33    <value>localhost:18040</value>
34  </property>
35 <property>
36    <name>yarn.resourcemanager.scheduler.address</name>
37    <value>localhost:18030</value>
38  </property>
39  <property>
40    <name>yarn.resourcemanager.resource-tracker.address</name>
41    <value>localhost:18025</value>
42  </property> <property>
43    <name>yarn.resourcemanager.admin.address</name>
44    <value>localhost:18141</value>
45  </property>                                                                                                                  
46 <property>
47    <name>yarn.resourcemanager.webapp.address</name>
48    <value>localhost:18088</value>
49 </property>
50 <property>
51    <name>yarn.log-aggregation-enable</name>
52    <value>true</value>
53  </property>
54 <property>
55    <name>yarn.log-aggregation.retain-seconds</name>
56    <value>86400</value>
57  </property>
58 <property>
59    <name>yarn.log-aggregation.retain-check-interval-seconds</name>
60    <value>86400</value>
61  </property>
62 <property>
63    <name>yarn.nodemanager.remote-app-log-dir</name>
64    <value>/tmp/logs</value>
65  </property>
66 <property>
67    <name>yarn.nodemanager.remote-app-log-dir-suffix</name>
68    <value>logs</value>
69  </property>
70 
71 </configuration>

mapred-site.xml

19 <configuration>
20 
21 <property>
22   <name>mapreduce.framework.name</name>
23   <value>yarn</value>
24 </property>
25 <property>
26   <name>mapreduce.jobtracker.http.address</name>
27   <value>localhost:50030</value>
28 </property>
29 <property>
30   <name>mapreduce.jobhisotry.address</name>
31   <value>localhost:10020</value>
32 </property>
33 <property>
34   <name>mapreduce.jobhistory.webapp.address</name>
35   <value>localhost:19888</value>
36 </property>
37 <property>
38   <name>mapreduce.jobhistory.done-dir</name>
39   <value>/jobhistory/done</value>
40 </property>
41 <property>
42   <name>mapreduce.intermediate-done-dir</name>
43   <value>/jobhisotry/done_intermediate</value>
44 </property>
45 <property>
46   <name>mapreduce.job.ubertask.enable</name>
47   <value>true</value>
48 </property>
49 
50 </configuration>

slaves

localhost

Hadoop-env.sh

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_231.jdk/Contents/Home  # 添加路径

格式化

>> hdfs namenode -format

启动集群

cd $HADOOP_HOME/sbin
./start-all.sh

验证

首先需要关闭防火墙，或者在防火墙的规则中开放端口

1. jps验证

   

2. 网页验证

• hdfs localhost:50070
• yarn localhost:18088

成功搭建！:happy: