這篇文章中的部署方案主要參考cloudera官方手冊
1 集群部署思路
下面是一個腦圖展示了一般的部署思路
http://naotu.baidu.com/file/f78248ad91fcc2019bfc16faf3987f31?token=d163a5b78d501c2d
image.png
2 以CDH5_12版本為例的部署方案
部署方案為parcles 離線部署 官方參考手冊
https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_install_path_b.html
2.1 確定環境
在官網可以查看到需求的環境信息
https://www.cloudera.com/downloads/manager/5-12-0.html
| 環境 | 版本 | 下載地址 |
|---|---|---|
| 操作系統 | redhat7.2 | 鏈接：http://pan.baidu.com/s/1pLTvDYb 密碼：5vsc |
| 數據庫 | mysql5.7 | https://www.mysql.com/downloads/ |
| jdk | oracle: jdk-8u121-linux-x64 | http://www.oracle.com/technetwork/java/javase/downloads/java-archive-javase8-2177648.html |
| 瀏覽器 | chrome | http://sw.bos.baidu.com/sw-search-sp/software/252cc80b36e05/ChromeStandalone_60.0.3112.78_Setup.exe |
| CDH | 5.12 | http://archive.cloudera.com/cdh5/parcels/5.12.0 |
| CM | 5.12 | http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.12.0/RPMS/x86_64/ |
| spark2 | 2.2.0 | http://archive.cloudera.com/spark2/ |
| kafka | 2.2.0-1.2.2.0.p0.68-el7 | http://archive.cloudera.com/kafka/parcels/2.2.0/ |
2.2 CDH以及CM 需下載對應的包文件
- 對應redhat7.2 應該下載以下安裝包
2.3 集群規劃
官方集群角色規劃建議https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_host_allocations.html
| 主機名 | IP地址 | 角色 |
|---|---|---|
| jp-hadoop-00 | 192.168.1.240 | cms Master Hosts mysql |
| jp-hadoop-01 | 192.168.1.241 | worker hosts(界面) |
| jp-hadoop-02 | 192.168.1.242 | worker hosts 主NN |
| jp-hadoop-03 | 192.168.1.243 | worker hosts tools |
| jp-hadoop-04 | 192.168.1.244 | worker hosts |
| jp-hadoop-05 | 192.168.1.245 | worker hosts |
| jp-hadoop-06 | 192.168.1.246 | worker hosts 備NN SNN |
| jp-hadoop-07 | 192.168.1.247 | worker hosts |
| jp-hadoop-08 | 192.168.1.248 | worker hosts |
| jp-hadoop-09 | 192.168.1.249 | worker hosts |
| jp-mysql-01 | 192.168.1.55 | mysql |
2.4 安裝系統以及系統設置
2.4.1 安裝Redhat7.2
- 選擇最小化安裝
- 磁盤分區
| 分區 | 磁盤大小 |
|---|---|
| boot | 500m |
| / | 100g+ |
| /home | 50g+ |
| /var | 50g+ |
| /dfs | 100g+ |
2.4.2 修改主機名
分別修改每一臺主機的主機名，方法如下
hostname 主機名稱 #臨時修改
hostnamectl set-hostname 主機名稱 #永久修改
# vi /etc/hostname 在里面寫入主機名 #編輯配置也可更改為永久
localhost.localdomain
# vi /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=localhost.localdomain
==修改后需要重新連接shell主機名才能生效==
2.4.3 配置IP地址
vi /etc/sysconfig/network-scripts/ifcfg-eth*****
添加如下信息
TYPE=Ethernet
BOOTPROTO=static #修改為靜態
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
NAME=eth0
UUID=11a27cf6-7589-4220-8f93-8471d3c789fc
DEVICE=eth0
ONBOOT=yes #開機啟動
IPADDR=192.168.1.243
NETMASK=255.255.255.0
GATEWAY=192.168.1.1
DNS=180.76.76.76
2.4.4 配置hosts
編輯 /etc/hosts文件 添加如下內容
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.240 jp-hadoop-00
192.168.1.241 jp-hadoop-01
192.168.1.242 jp-hadoop-02
192.168.1.243 jp-hadoop-03
192.168.1.244 jp-hadoop-04
192.168.1.245 jp-hadoop-05
192.168.1.246 jp-hadoop-06
192.168.1.247 jp-hadoop-07
192.168.1.248 jp-hadoop-08
192.168.1.249 jp-hadoop-09
192.168.1.55 jp-mysql-01
2.4.5 配置ssh免密登陸
-
配置 ssh
vi /etc/ssh/sshd_config， 找到以下內容，去掉前面的注釋’#’ RSAAuthentication yes #啟用 RSA 認證 PubkeyAuthentication yes #啟用公鑰私鑰配對認證方式 AuthorizedKeysFile .ssh/authorized_keys #公鑰文件路徑 -
重啟sshd服務
service sshd restart -
在每一臺主機生成ssh 公鑰和私鑰,執行命令，加三個回車
ssh-keygen -t rsa -
分別將每一臺機器的公鑰文件值放到jp-hadoop-00:/root/.ssh/authorized_keys 文件中
scp .ssh/id_rsa.pub root@192.168.1.240:/root/id_rsa.pub cat id_rsa.pub >> /root/.ssh/authorized_keys
2.4.6 每一臺主機分別執行腳本配置參數
-
執行腳本
#關閉防火墻 sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config systemctl stop firewalld systemctl disable firewalld #調整內核參數 echo 1 > /proc/sys/vm/overcommit_memory #調整ulimit和nproc sed -i '49d' /etc/security/limits.conf sed -i '/4/aroot - nofile 327680' /etc/security/limits.conf rm -rf /etc/pam.d/common-session echo 'session required pam_limits.so' > /etc/pam.d/common-session #調整內存交換 echo 'vm.swappiness=0' >> /etc/sysctl.conf #調整基礎環境 echo never > /sys/kernel/mm/transparent_hugepage/defrag echo never > /sys/kernel/mm/transparent_hugepage/enabled echo 'echo never > /sys/kernel/mm/transparent_hugepage/defrag' >> /etc/rc.local echo 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' >> /etc/rc.local -
centos 7 禁用ipv6
1. 通過命令： Check to see if you’re installation is currently set up for IPv6: # cat /proc/sys/net/ipv6/conf/all/disable_ipv6 If the output is 0, IPv6 is enabled. If the output is 1, IPv6 is already disabled. 2. 臨時禁用ipv6 vi /etc/sysctl.conf 添加下面的行： net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1 執行命令 #sysctl -p 3. 永久禁用 # vim /etc/default/grub 在第6行 加上 GRUB_CMDLINE_LINUX="ipv6.disable=1" # grub2-mkconfig -o /boot/grub2/grub.cfg # reboot
[圖片上傳失敗...(image-4badfa-1551970449869)]
2.4.7 配置yum鏡像
上傳redhat7.2鏡像到jp-hadoop-00 的/opt目錄下
-
執行命令掛載redhat 本地源
mkdir -p /mnt/redhat7 mount -o loop -t iso9660 /opt/rhel-server-7.2-x86_64-dvd.iso /mnt/redhat7 -
添加本地redhat yum配置
vim /etc/yum.repos.d/redhat7.repo [redhat7] name=redhat7 baseurl=file:///mnt/redhat7 gpgcheck=0 -
安裝httpd 服務
yum install httpd systemctl start httpd -
添加httpd鏈接
ln -s /mnt/redhat7/ /var/www/html/redhat7 通過網頁 http://jp-hadoop-00/redhat7 可以訪問即可 -
在所有主機添加yum 源
vim /etc/yum.repos.d/redhat_html.repo [redhat7] name=redhat7 baseurl=http://192.168.1.240/redhat7 gpgcheck=0 -
上傳文件到主機
在jp-hadoop-00上，上傳文件在jp-hadoop-00上，上傳文件 在主機新建目錄 /opt/cloudera/cm-5.12.0 上傳文件 cloudera-manager-agent-5.12.0-1.cm5120.p0.120.el7.x86_64.rpm cloudera-manager-server-5.12.0-1.cm5120.p0.120.el7.x86_64.rpm cloudera-manager-daemons-5.12.0-1.cm5120.p0.120.el7.x86_64.rpm jdk-8u121-linux-x64.rpm 在主機新建目錄 /opt/cloudera/parcel-repo/ 上傳文件 CDH-5.12.0-1.cdh5.12.0.p0.29-el7.parcel CDH-5.12.0-1.cdh5.12.0.p0.29-el7.parcel.sha1 ###將名稱修改一下，去掉最后的1 CDH-5.12.0-1.cdh5.12.0.p0.29-el7.parcel.sha manifest.json CDH-5.12.0-1.cdh5.12.0.p0.29-el7.parcel.torrent # 將它權限改為777 在所有主機新建目錄 /opt/cloudera/cm-5.12.0 上傳文件 cloudera-manager-agent-5.12.0-1.cm5120.p0.120.el7.x86_64.rpm cloudera-manager-daemons-5.12.0-1.cm5120.p0.120.el7.x86_64.rpm jdk-8u121-linux-x64.rpm 在所有主機新建目錄 /opt/cloudera/parcel-repo/ 上傳文件 CDH-5.12.0-1.cdh5.12.0.p0.29-el7.parcel CDH-5.12.0-1.cdh5.12.0.p0.29-el7.parcel.sha1 ###將名稱修改一下，去掉最后的1 CDH-5.12.0-1.cdh5.12.0.p0.29-el7.parcel.sha manifest.json CDH-5.12.0-1.cdh5.12.0.p0.29-el7.parcel.torrent # 將它權限改為777
-
在jp-hadoop-00上創建cdh內網yum源
yum install createrepo -y cd /opt/cloudera/cm-5.12.0 createrepo . ln -s /opt/cloudera/cm-5.12.0 /var/www/html/cms -
在所有主機添加cms源
vim /etc/yum.repos.d/cms.repo [cms] name=cms baseurl=http://192.168.1.240/cms gpgcheck=0 enabled=0 -
重新生成yum緩存
yum clean all yum makecache
2.4.8 配置ntp服務
-
在所有主機安裝ntp服務
yum install ntp 配置一臺內網主ntp server，選擇jp-hadoop-00為ntp server
```
編輯 /etc/ntp.conf,修改如下
# Hosts on local network are less restricted.
# 允許內網其他機器同步時間（前者是內網的網關，后者是子網掩碼）
restrict 192.168.1.255 mask 255.255.255.0 nomodify notrap
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
# 中國公安網NTP服務器IP地址（亦可以填寫其他標準服務器）
server 202.120.2.101 prefer
# ntp.sjtu.edu.cn 202.120.2.101 (上海交通大學網絡中心NTP服務器地址）
# allow update time by the upper server
# 允許上層時間服務器主動修改本機時間，填寫標準時間服務器IP
# restrict 100.2.8.68 nomodify notrap noquery
# Undisciplined Local Clock. This is a fake driver intended for backup
# and when no outside source of synchronized time is available.
# 外部時間服務器不可用時，以本地時間作為時間服務
server 127.127.1.0
fudge 127.127.1.0 stratum 10
```
- 重啟ntpd服務
```
systemctl restart ntpd
```
-
配置其它ntp 主機
同樣配置/etc/ntp.conf # Please consider joining the pool (http://www.pool.ntp.org/join.html). # 配置時間服務器為本地的時間服務器（此處的地址是NTP-Server的地址） server jp-hadoop-00 # allow update time by the upper server #restrict jp-hadoop-00 nomodify notrap noquery # Undisciplined Local Clock. This is a fake driver intended for backup # and when no outside source of synchronized time is available. server 127.127.1.0 fudge 127.127.1.0 stratum 10 -
手動調整時間
ntpdate -u jp-hadoop-00 -
重啟ntpd服務
systemctl restart ntpd -
查看ntpd狀態
ntpq -p
2.5 安裝jdk1.8
-
在所有主機執行如下步驟
$ cd /opt/cloudera/cm-5.12.0 $ rpm -ivh jdk-8u121-linux-x64.rpm $ vim /etc/profile PATH=$PATH:$HOME/.local/bin:$HOME/bin export JAVA_HOME=/usr/java/jdk1.8.0_121 export JRE_HOME=$JAVA_HOME/jre export PATH=$JAVA_HOME/bin:$JRE_HOME/bin/:$PATH $ source /etc/profile $ java -version
2.6 安裝cm和cdh rpm包
在所有主機上執行
```
cd /opt/cloudera/cm-5.12.0
yum localinstall cloudera-*.rpm
```
2.7 安裝外置數據庫mysql
見mysql數據庫安裝
2.8 mysql數據庫配置
官方mysql數據庫需求：https://www.cloudera.com/documentation/enterprise/5-8-x/topics/cm_ig_mysql.html#cmig_topic_5_5_2
設置mysql為默認數據庫: https://www.cloudera.com/documentation/enterprise/5-8-x/topics/cm_ig_installing_configuring_dbs.html#concept_i2r_m3m_hn
2.8.1 mysql配置文件 編輯 /etc/my.cnf文件 內容如下
# For advice on how to change settings please see
# http://dev.mysql.com/doc/refman/5.7/en/server-configuration-defaults.html
[mysqld]
#set by chenyangang
transaction-isolation = READ-COMMITTED
#
# Remove leading # and set to the amount of RAM for the most important data
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
# innodb_buffer_pool_size = 128M
#
# Remove leading # to turn on a very important data integrity option: logging
# changes to the binary log between backups.
# log_bin
#
# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
# join_buffer_size = 128M
# sort_buffer_size = 2M
# read_rnd_buffer_size = 2M
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
# Disabling symbolic-links is recommended to prevent assorted security risks
#symbolic-links=0 #set by chenyangang
#set by chenyangang
key_buffer_size = 32M
max_allowed_packet = 512M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1
max_connections = 550
#log_bin should be on a disk with enough free space. Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your system
#and chown the specified folder to the mysql user.
#log_bin=/var/lib/mysql/mysql_binary_log
#set by chenyangang
# For MySQL version 5.1.8 or later. For older versions, reference MySQL documentation for configuration help.
binlog_format = mixed
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
#set by chenyangang
# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 2G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M
#set by chenyangang
sql_mode=STRICT_ALL_TABLES
#set by chenyangang
wait_timeout=31536000
interactive_timeout=31536000
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
2.8.2 mysql 數據庫用戶 以及 數據庫配置
| ROLE | Database | User | Password |
|---|---|---|---|
| Activity Monitor | amon | amon | amon |
| Reports Manager | rman | rman | rman |
| Hive Metastore Server | metastore | hive | hive |
| Sentry Server | sentry | sentry | sentry |
| Cloudera Navigator Audit Server | nav | nav | nav12 |
| Cloudera Navigator Metadata Server | navms | navms | navms |
create database amon DEFAULT CHARACTER SET utf8;
create database rman DEFAULT CHARACTER SET utf8;
create database metastore DEFAULT CHARACTER SET utf8;
create database sentry DEFAULT CHARACTER SET utf8;
create database nav DEFAULT CHARACTER SET utf8;
create database navms DEFAULT CHARACTER SET utf8;
create database hue DEFAULT CHARACTER SET utf8;
create database oozie DEFAULT CHARACTER SET utf8;
set global validate_password_policy=0; #設置為純數字純字母等密碼
set global validate_password_length=2; #設置密碼最低要求
CREATE USER 'amon' IDENTIFIED BY 'amon';
CREATE USER 'rman' IDENTIFIED BY 'rman';
CREATE USER 'hive' IDENTIFIED BY 'hive';
CREATE USER 'sentry' IDENTIFIED BY 'sentry';
CREATE USER 'nav' IDENTIFIED BY 'nav12';
CREATE USER 'navms' IDENTIFIED BY 'navms';
CREATE USER 'hue' IDENTIFIED BY 'hue12';
CREATE USER 'oozie' IDENTIFIED BY 'oozie';
GRANT ALL PRIVILEGES ON *.* TO 'amon'@'%' IDENTIFIED BY 'amon' WITH GRANT OPTION;
GRANT ALL PRIVILEGES ON *.* TO 'rman'@'%' IDENTIFIED BY 'rman' WITH GRANT OPTION;
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%' IDENTIFIED BY 'hive' WITH GRANT OPTION;
GRANT ALL PRIVILEGES ON *.* TO 'sentry'@'%' IDENTIFIED BY 'sentry' WITH GRANT OPTION;
GRANT ALL PRIVILEGES ON *.* TO 'nav'@'%' IDENTIFIED BY 'nav12' WITH GRANT OPTION;
GRANT ALL PRIVILEGES ON *.* TO 'navms'@'%' IDENTIFIED BY 'navms' WITH GRANT OPTION;
GRANT ALL PRIVILEGES ON *.* TO 'hue'@'%' IDENTIFIED BY 'hue12' WITH GRANT OPTION;
GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'%' IDENTIFIED BY 'oozie' WITH GRANT OPTION;
flush privileges ;
2.8.3 將 mysql-connector-java.jar 包文件放到 /usr/share/java/ 目錄下
下載地址鏈接：http://pan.baidu.com/s/1hsb1qYC 密碼：d59v
2.8.4 選擇mysql為默認數據庫
如果存在文件/etc/cloudera-scm-server/db.mgmt.properties 則刪掉文件/etc/cloudera-scm-server/db.mgmt.properties
在jp-hadoop-00上執行腳本
形式
/usr/share/cmf/schema/scm_prepare_database.sh database-type [options] database-name username password
示例
/usr/share/cmf/schema/scm_prepare_database.sh mysql -h jp-mysql-01 -utemp -ptemp --scm-host jp-hadoop-00 scm scm scm12
2.9 開始安裝集群
2.9.1 啟動集群
- 修改所有主機/etc/cloudera-scm-agent/config.ini 配置文件 ，配置server_host=jp-hadoop-00
在jp-hadoop-00 上啟動
/etc/init.d/cloudera-scm-server start
/etc/init.d/cloudera-scm-agent start
在其它主機啟動
/etc/init.d/cloudera-scm-agent start
2.9.2 日志目錄
日志文件夾在 /var/log/cloudera-scm-*
3.集群集成安裝
spark2 集成安裝https://www.cloudera.com/documentation/spark2/latest/topics/spark2_installing.html
3.1 集群詳細角色分配方案
平臺部署方案可以參考
image
