生产环境下的PROD-CDH集群安装部署

生产环境下的PROD-CDH集群安装部署

一：初始化（所有节点执行)

1.关闭防火墙

systemctl stop firewalld.service ; systemctl disable firewalld.service

2.配置主机名称映射

vi /etc/hosts

192.168.102.20 cdh1

192.168.102.21 cdh2

192.168.102.22 cdh3

192.168.102.23 cdh4

192.168.102.24 cdh5

echo -e "192.168.102.20 cdh1\n192.168.102.21 cdh2\n192.168.102.22 cdh3\n192.168.102.23 cdh4\n192.168.102.24 cdh5" >> /etc/hosts

3.关闭selinux

vi /etc/selinux/config

SELINUX=disabled

setenforce 0

sed -i "s/SELINUX=permissive/SELINUX=disabled/g" /etc/selinux/config

##暂时不执行下列步骤，会导致网络不可用

##4.停止networkmanager服务

##NetworkManager在网络断开的时候，会清理路由

##systemctl stop NetworkManager.service ; systemctl disable NetworkManager.service

##5.重启网络：

##systemctl restart network

6.禁用透明大页面(重启生效)

vi /etc/rc.local

if test -f /sys/kernel/mm/transparent_hugepage/enabled;then

echo never >/sys/kernel/mm/transparent_hugepage/enabled

fi

if test -f /sys/kernel/mm/transparent_hugepage/defrag;then

echo never >/sys/kernel/mm/transparent_hugepage/defrag

fi

#

chmod +x /etc/rc.d/rc.local

#验证

ls -al /etc/rc.d/rc.local

cat /etc/rc.d/rc.local

7.修改Linux swappiness

vi /etc/sysctl.conf

vm.swappiness=0

使参数生效：

sysctl -p

#

sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

8.格式化磁盘

mkfs.ext4 /dev/sdb;mkdir /data;mount /dev/sdb /data;echo "/dev/sdb /data ext4 defaults 0 0" >>/etc/fstab

9.配置历史记录

在机器 /etc/profile 里加上：

#history time

HISTFILESIZE=4000

HISTSIZE=4000

HISTTIMEFORMAT='%F %T '

export HISTTIMEFORMAT

#log to messages

export PROMPT_COMMAND='{ msg=$(history 1 | { read x y; echo $y; });logger "[euid=$(whoami)]":$(who am i):[pwd]"$msg"; }'

#生效

source /etc/profile

二：配置节点之间免密登录（主节点，CDH1）

ssh-keygen -t rsa

一路回车

cd ~/.ssh/

进入到.ssh/目录，将公钥拷贝到免密登录的机器上

首次拷贝需要输入密码

ssh-copy-id cdh1

ssh-copy-id cdh2

ssh-copy-id cdh3

ssh-copy-id cdh4

ssh-copy-id cdh5

三：文件准备部署（所有节点执行)

mkdir -p /data/tools

cd /data/tools

上传对应文件,mysql,jdk,cdh,cm

拷贝文件到各节点里

cd /data

scp -r tools cdh2:/data

执行安装

cd /data/tools

rpm -ivh oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm

配置Java环境变量，在 /etc/profile中追加以下内容：

vi /etc/profile

#jdk

export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera

export JRE_HOME=${JAVA_HOME}/jre

export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib

export PATH=${JAVA_HOME}/bin:$PATH

执行生效：source /etc/profile

拷贝jdbc驱动包到服务器

cd /data/tools

mkdir -p /usr/share/java

cp mysql-connector-java-5.1.48.jar /usr/share/java/mysql-connector-java.jar

四：安装cloudera-manager-daemons（所有节点执行)

ifconfig 查看是否有虚拟路由，如docker,kvm等，需删除

#使用阿里云源

wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo

yum -y remove docker*

上传cloudera-manager-daemons，cloudera-manager-agent（所有节点）

再上传cloudera-manager-server（主节点）

安装cloudera-manager-daemons（所有节点）

yum -y install cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm

五：配置时间同步

CentOS 7默认以chronycd服务作时间同步，而不以ntpd作为同步服务

所有系节点都以CDH01作为同步服务器，在所有节点更改文件

vi /etc/chrony.conf

#server 0.centos.pool.ntp.org iburst

#server 1.centos.pool.ntp.org iburst

#server 2.centos.pool.ntp.org iburst

#server 3.centos.pool.ntp.org iburst

server cdh1 iburst

allow 192.168.10.0/24

仅cdh1 修改：

local stratum 10

重启服务

systemctl restart chronyd ; systemctl enable chronyd; systemctl status chronyd

检查状态

chronyc sources

#完成以上步骤后，建议重启一次服务器

六：安装mysql服务

集群的安装需要一个关系型数据库进行配置数据的存储，CDH支持Oracle，Mysql，Postgrest

在CDH1节点部署即可或独立部署

检查已安装的

rpm -qa|grep mysql

rpm -qa|grep mariadb

whereis mysql

手工删除查找结果 删除mysql

yum -y remove mysql-libs

rm -rf /usr/lib64/mysql/

检查mysql用户组和用户是否存在，如果没有，则创建

cat /etc/group | grep mysql

#如果上述命令结果为空，则执行以下命令创建MySQL用户组及用户

groupadd mysql

useradd -r -g mysql mysql

安装rpm包

yum -y install net-tools

rpm -ivh 01_mysql-community-common-5.7.16-1.el7.x86_64.rpm

rpm -ivh 02_mysql-community-libs-5.7.16-1.el7.x86_64.rpm

rpm -ivh 03_mysql-community-libs-compat-5.7.16-1.el7.x86_64.rpm

rpm -ivh 04_mysql-community-client-5.7.16-1.el7.x86_64.rpm

rpm -ivh 05_mysql-community-server-5.7.16-1.el7.x86_64.rpm

#修改默认字符集

vi /etc/my.cnf

[client]

default-character-set = utf8

[mysql]

default-character-set = utf8

[mysqld]

character-set-server = utf8

如需要可修改为下面

在该my.cnf文件下进行配置修改
[client]
default-character-set = utf8mb4

[mysql]
default-character-set = utf8mb4

[mysqld]
character-set-client-handshake = FALSE
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci
init_connect='SET NAMES utf8mb4'

设置MySQL开机启动并启动MySQL

systemctl enable mysqld.service ; systemctl start mysqld.service ; systemctl status mysqld.service

获得初始密码

grep password /var/log/mysqld.log

wrJk,,kfU6D>

dev-cdh

2S>QRbfOuk-i

执行mysql_secure_installation命令，对MySQL做基础配置

wrJk,,kfU6D>

[root@cdh1 tools]# mysql_secure_installation
mysql_secure_installation: [ERROR] unknown variable 'default-character-set=utf8'

Securing the MySQL server deployment.

Enter password for user root: 

The existing password for the user account root has expired. Please set a new password.

New password: 

Re-enter new password: 
The 'validate_password' plugin is installed on the server.
The subsequent steps will run with the existing configuration
of the plugin.
Using existing password for root.

Estimated strength of the password: 100 
Change the password for root ? ((Press y|Y for Yes, any other key for No) : n

 ... skipping.
By default, a MySQL installation has an anonymous user,
allowing anyone to log into MySQL without having to have
a user account created for them. This is intended only for
testing, and to make the installation go a bit smoother.
You should remove them before moving into a production
environment.

Remove anonymous users? (Press y|Y for Yes, any other key for No) : y
Success.


Normally, root should only be allowed to connect from
'localhost'. This ensures that someone cannot guess at
the root password from the network.

Disallow root login remotely? (Press y|Y for Yes, any other key for No) : n

 ... skipping.
By default, MySQL comes with a database named 'test' that
anyone can access. This is also intended only for testing,
and should be removed before moving into a production
environment.


Remove test database and access to it? (Press y|Y for Yes, any other key for No) : y
 - Dropping test database...
Success.

 - Removing privileges on test database...
Success.

Reloading the privilege tables will ensure that all changes
made so far will take effect immediately.

Reload privilege tables now? (Press y|Y for Yes, any other key for No) : y
Success.

All done! 

进入mysql，使用命令show variables like "%char%"; 检查字符集是否为utf8，如果不是改过来

mysql -uroot -pwrJk,,kfU6D>

进入Mysql数据库中，运行下面的语句，创建好需要的库和用户，密码自己改成自己希望的

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;  
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;  
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;  
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;  
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;  
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;  
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;  
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;  
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;  
CREATE DATABASE hive DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;  

GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'PWD';  
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'PWD';  
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'PWD';  
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'PWD';  
GRANT ALL ON metastore.* TO 'metastore'@'%' IDENTIFIED BY 'PWD';  
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'PWD';  
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'PWD';  
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'PWD';  
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'PWD';  
GRANT ALL ON hive.* TO 'hive'@'%' IDENTIFIED BY 'PWD'; 

flush privileges;

七：安装Cloudera Manager

##cloudera agent

在主节点安装cloudera agent和cloudera server

#yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server -y

在其他子节点安装cloudera agent

#yum install cloudera-manager-daemons cloudera-manager-agent -y

#cloudera agent

#修改cm agent配置，指定cm server的连接地址

#vi /etc/cloudera-scm-agent/config.ini

#修改server_host=主节点IP或主机名

#开启所有节点cm agent

#systemctl start cloudera-scm-agent

#可提前下载安装对应依赖

#yum install bind-utils psmisc cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs /lib/lsb/init-functions httpd mod_ssl openssl-devel python-psycopg2 MySQL-python libxslt

安装agent：（所有节点)

cd /data/tools

yum -y install cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm

配置scm-agent的server节点

cp /etc/cloudera-scm-agent/config.ini /etc/cloudera-scm-agent/config.ini.template

vi /etc/cloudera-scm-agent/config.ini

server_host=cdh1

server_port=7182

保存退出，执行命令

启动服务

systemctl start cloudera-scm-agent

systemctl enable cloudera-scm-agent

systemctl status cloudera-scm-agent

##

##警告如果没有证书，就不要配这个否则后面会出现 agent 无法上报的问题

##/opt/cloudera/cm-agent/bin/certmanager setup --configure-services

安装scm-server:(cdh1)

yum -y install cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm

cm节点初始化数据库，用引号把密码圈起来

/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm 'pwd'

# 初始化cm数据库（密码自己设置）
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql -uroot -p12345 scm scm 12345
# 脚本参数解释
scm_prepare_database.sh 【数据库类型】 -u【数据库登录用户】 -p【数据库登录密码】 【CDH要创建的数据库名称】 【CDH数据库用户名】 【CDH数据库用户密码】

配置parcels(主节点操作)

执行：

cd /opt/cloudera/parcel-repo

cp /data/tools/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel .

sha1sum CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel | awk '{ print $1 }' > CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha

如果不配置则在安装过程会提示

启动服务

systemctl start cloudera-scm-server

查看日志(等待几分钟)

tail /var/log/cloudera-scm-server/cloudera-scm-server.log

出现INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.这条信息就代表服务启动成功

在浏览器中访问cm节点的7180端口，账号密码均为 admin，登录CDH管理页面

http://192.168.102.20:7180/

选择免费功能

第二次欢迎页面任然点击继续

定义集群名称，UMImeta

输入之前准备好的安装大数据服务用的节点，点击搜索，结果默认是全选的，根据情况选择，之后点击继续

选择主机（在该页不要填写IP去搜索节点，应点击’当前管理的主机‘）

选择parcel包（画红线部分是检测到的本地parcel包，如果不显示说明没被检测到，可尝试等待刷新页面或回退到欢迎页面重启cm server）

开始向所有节点分发部署安装包

进行安装前环境检查（可以根据检查结果修正环境配置，修正后重新检查并继续）

如果有检查项没通过，需要先修正配置后再次执行检查，直到所有检查项均正常，如果有些配置的的修正涉及到重启服务器等需要停止cm server服务的操作，要先回退到欢迎页面并停止cm server服务，重启完成继续从第一步进行

选择需要安装的大数据服务，这里选择自定义服务，这样可以按需安装

数据服务角色分配（注意要做好角色规划)

填写数据库信息（有些大数据服务需要数据库支持，例如hive、oozie等，所以需填写数据库信息）

主机名称：之前建的mysql库所在的位置，安装mysql的服务器名称
数据库名称：使用哪一个mysql库初始化的一些相关库
用户名称：连接mysql数据库的账号对应库的账号
密码：连接mysql数据库的账号的密码

测试连接数据库 amon activity monitor

修改大数据服务相关数据目录路径，例如hdfs、kafka数据目录等

其他可以使用默认值，继续下一步骤

开始安装服务

访问页面：http://192.168.102.20:7180/cmf/home (admin/admin)

修改HDFS的权限检查配置

关闭HDFS中的权限检查：dfs.permissions。 保存更改，然后重启hdfs 服务

点击hdfs 组件进入hdfs 组件管理页面 点击配置 搜索 dfs.permissions