0. 安裝環境
安裝系統版本，集群IP列表
- ubunut 16.04 server
192.168.24.128 ubuntu-master
192.168.24.129 ubuntu-slave1
192.168.24.130 ubuntu-slave2
1. 創建用戶 和用戶組(hadoop 用戶的密碼也是hadoop 方便記憶)
useradd hadoop
passwd hadoop
groupadd bigdata
usermod -a -G bigdata hadoop
mkdir /home/hadoop
chown -R hadoop:bigdata /home/hadoop
2. 更改hosts 文件以及對應的機器hostname
host文件 /etc/hosts
hostname文件 /etc/hostname
cat /etc/hosts
192.168.24.128 ubuntu-master
192.168.24.129 ubuntu-slave1
192.168.24.130 ubuntu-slave2
在這里要注意，我安裝的集群是用虛擬機創建的，機器之間是直接復制虛擬機的方式，這時有一個問題就是機器的hostname都是一樣的，這樣一來，即使修改了hosts 文件，但是每臺機器的hostname 并沒有生效，需要修改各自的hostname保持和上述文件中一樣的對應關系。
3. 無密碼登陸
在每臺虛擬機上分別執行以下命令
ssh-keygen -t rsa -P ""
cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
執行
ssh localhost
可以發現此時無需輸入密碼
集群之間的無密碼ssh 通信
以192.168.24.128/129/130 三臺機為例
登錄192.168.24.128
ssh-copy-id -i $HOME/.ssh/id_rsa.pub hadoop@192.168.24.129
ssh-copy-id -i $HOME/.ssh/id_rsa.pub hadoop@192.168.24.130
5. 設置環境變量
安裝jdk,并設置環境變量，我的.bashrc文件的相關內容如下
#set oracle jdk environment
export JAVA_HOME=/usr/lib/jvm/jdk1.7.0_79 ## 這里要注意目錄要換成自己解壓的jdk 目錄
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH
5. 上傳hadoop壓縮包到指定目錄并解壓
$ ls
hadoop-2.6.2 hadoop-2.6.2.tar.gz
6. 修改相應的配置文件
需要修改的文件列表如下
~/hadoop/etc/hadoop/hadoop-env.sh
~/hadoop/etc/hadoop/yarn-env.sh
~/hadoop/etc/hadoop/slaves
~/hadoop/etc/hadoop/core-site.xml
~/hadoop/etc/hadoop/hdfs-site.xml
~/hadoop/etc/hadoop/mapred-site.xml
~/hadoop/etc/hadoop/yarn-site.xml
- 以下兩個文件設置的內容為JAVA_HOME環境變量
~/hadoop/etc/hadoop/hadoop-env.sh
~/hadoop/etc/hadoop/yarn-env.sh
- 設置slave機器的列表
cat ~/hadoop/etc/hadoop/slaves
192.168.24.129
192.168.24.130
- ~/hadoop/etc/hadoop/core-site.xml
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/bigdata/hadoop/tmp</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://192.168.24.128:9000</value>
</property>
</configuration>
- hdfs-site.xml
<configuration>
<property>
<name>dfs.http.address</name>
<value>192.168.24.128:50070</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>192.168.24.128:50090</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
- mapred-site.xml
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>192.168.24.128:9001</value>
</property>
<property>
<name>mapred.map.tasks</name>
<value>2</value>
</property>
<property>
<name>mapred.reduce.tasks</name>
<value>2</value>
</property>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>192.168.24.128:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>192.168.24.128:19888</value>
</property>
</configuration>
- yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.address</name>
<value>192.168.24.128:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>192.168.24.128:8030</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>192.168.24.128:8088</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>192.168.24.128:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>192.168.24.128:8033</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
按照上面的配置之后應該來說就可以了，但是在之前安裝的過程中，我遇到過任務hang住的情況，查找資料發現是內存不夠的原因，由于是自己電腦跑的虛擬機，內存有限，于是針對內存情況增加如下配置：
mapred-site.xml
<property>
<name>mapreduce.map.memory.mb</name>
<value>230</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>460</value>
</property>
<property>
<name>mapreduce.map.java.opts</name>
<value>-Xmx184m</value>
</property>
<property>
<name>mapreduce.reduce.java.opts</name>
<value>-Xmx368m</value>
</property>
<property>
<name>yarn.app.mapreduce.am.resource.mb</name>
<value>460</value>
</property>
<property>
<name>yarn.app.mapreduce.am.command-opts</name>
<value>-Xmx368m</value>
</property>
yarn-site.xml
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>700</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>230</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>700</value>
</property>
7. 啟動
首次啟動需要先在 Master 節點執行 NameNode 的格式化：
./hdfs namenode -format
進入~/hadoop/sbin目錄
./start-dfs.sh
./start-yarn.sh
./mr-jobhistory-daemon.sh start historyserver
查看進程
hadoop@ubuntu-master:~$ jps
6193 ResourceManager
5863 NameNode
6051 SecondaryNameNode
6476 JobHistoryServer
6937 Jps
hadoop@ubuntu-slave1:~$ jps
3536 Jps
2884 NodeManager
2771 DataNode
hadoop@ubuntu-slave2:~$ jps
2937 NodeManager
2826 DataNode
3653 Jps
登錄192.168.24.128:50070可以看到hadoop集群基本信息
8. 運行測試例子
進入 ~/hadoop/bin目錄
./hdfs dfs -mkdir -p /user/hadoop
./hdfs dfs -mkdir input
./hdfs dfs -put /home/hadoop/bigdata/hadoop/etc/hadoop/*.xml input
./hadoop jar /home/hadoop/bigdata/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar grep input output 'dfs[a-z.]+'
運行程序部分結果如下：
17/01/13 23:45:14 INFO client.RMProxy: Connecting to ResourceManager at /192.168.24.128:8032
17/01/13 23:45:17 WARN mapreduce.JobResourceUploader: No job jar file set. User classes may not be found. See Job or Job#setJar(String).
17/01/13 23:45:17 INFO input.FileInputFormat: Total input paths to process : 9
17/01/13 23:45:18 INFO mapreduce.JobSubmitter: number of splits:9
17/01/13 23:45:19 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1484322039697_0001
17/01/13 23:45:20 INFO mapred.YARNRunner: Job jar is not present. Not adding any jar to the list of resources.
17/01/13 23:45:21 INFO impl.YarnClientImpl: Submitted application application_1484322039697_0001
17/01/13 23:45:22 INFO mapreduce.Job: The url to track the job: http://ubuntu-master:8088/proxy/application_1484322039697_0001/
17/01/13 23:45:22 INFO mapreduce.Job: Running job: job_1484322039697_0001
17/01/13 23:45:53 INFO mapreduce.Job: Job job_1484322039697_0001 running in uber mode : false
17/01/13 23:45:53 INFO mapreduce.Job: map 0% reduce 0%
17/01/13 23:46:22 INFO mapreduce.Job: map 11% reduce 0%
17/01/13 23:46:43 INFO mapreduce.Job: map 22% reduce 0%
17/01/13 23:47:08 INFO mapreduce.Job: map 33% reduce 0%
17/01/13 23:47:09 INFO mapreduce.Job: map 44% reduce 0%
17/01/13 23:47:10 INFO mapreduce.Job: map 56% reduce 0%
17/01/13 23:47:11 INFO mapreduce.Job: map 67% reduce 0%
17/01/13 23:47:59 INFO mapreduce.Job: map 78% reduce 0%
17/01/13 23:48:03 INFO mapreduce.Job: map 100% reduce 0%
17/01/13 23:48:24 INFO mapreduce.Job: map 100% reduce 50%
17/01/13 23:48:46 INFO mapreduce.Job: map 100% reduce 100%
17/01/13 23:48:47 INFO mapreduce.Job: Job job_1484322039697_0001 completed successfully
... ...
9. 后記
- 之前也零零散散裝過幾次hadoop，但是沒有記錄筆記，之前安裝的虛擬機也被刪除了，這此就重新整理了一下，以便以后參考。
- 安裝的過程并不順利，中間遇到了磕磕絆絆的小問題，之后會整理單獨一個遇到問題的筆記，暫時就這樣
