CentOS on F嘉阳

RocketMQ学习笔记6——集群实战

Wed, 23 Dec 2020 19:15:00 +0800

组建高性能MQ服务

双主双从模式
保障无单点故障问题的高可用集群

集群拓扑图

IP	资源	服务
10.139.12.52	1C2G20GB	rocketmq-nameserver1、rocketmq-master1
10.139.12.53	1C2G20GB	rocketmq-nameserver2、rocketmq-master2
10.139.12.54	1C2G20GB	rocketmq-nameserver3、rocketmq-master1-slave
10.139.12.55	1C2G20GB	rocketmq-nameserver4、rocketmqmaster2-slave

配置文件最优设置解析

修改host文件映射

10.139.12.52 rocketmq-nameserver1 rocketmq-master1
10.139.12.53 rocketmq-nameserver2 rocketmq-master2
10.139.12.54 rocketmq-nameserver3 rocketmq-master1-slave
10.139.12.55 rocketmq-nameserver4 rocketmq-master2-slave

将4个节点的hosts均改为如上内容，并将4.4.0版本包放入环境中解压

创建存储目录

$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index

建立软连接

$ cd ~/rocketmq-test
$ ln -s rocketmq-all-4.4.0-bin-release rocketmq

修改配置文件

解压后默认的配置文件如下

[user@301 ~/rocketmq-test/rocketmq/conf]$ll
total 36
drwxr-xr-x 2 ocmuser gpocm 118 Jan 17 2019 2m-2s-async
drwxr-xr-x 2 ocmuser gpocm 118 Jan 17 2019 2m-2s-sync
drwxr-xr-x 2 ocmuser gpocm 91 Jan 17 2019 2m-noslave
-rw-r--r-- 1 ocmuser gpocm 949 Jan 17 2019 broker.conf
-rw-r--r-- 1 ocmuser gpocm 14978 Jan 17 2019 logback_broker.xml
-rw-r--r-- 1 ocmuser gpocm 3836 Jan 17 2019 logback_namesrv.xml
-rw-r--r-- 1 ocmuser gpocm 3761 Jan 17 2019 logback_tools.xml
-rw-r--r-- 1 ocmuser gpocm 1277 Jan 17 2019 plain_acl.yml
-rw-r--r-- 1 ocmuser gpocm 833 Jan 17 2019 tools.yml

若想保障生产环境高可用，推荐使用双主双从同步双写模式，可以让一个主节点对应多个从节点，但从节点越多，同步效率越慢

本次采用双主双从同步双写模式，一个主节点对应一个从节点

[user@301 ~/rocketmq-test/rocketmq/conf/2m-2s-sync]$ll
total 16
-rw-r--r-- 1 ocmuser gpocm 928 Jan 17 2019 broker-a.properties
-rw-r--r-- 1 ocmuser gpocm 922 Jan 17 2019 broker-a-s.properties
-rw-r--r-- 1 ocmuser gpocm 928 Jan 17 2019 broker-b.properties
-rw-r--r-- 1 ocmuser gpocm 922 Jan 17 2019 broker-b-s.properties

用途如下

IP	资源	服务	配置文件
10.139.12.52	1C2G20GB	rocketmq-nameserver1、rocketmq-master1	broker-a.properties
10.139.12.53	1C2G20GB	rocketmq-nameserver2、rocketmq-master2	broker-b.properties
10.139.12.54	1C2G20GB	rocketmq-nameserver3、rocketmq-slave1	broker-a-s.properties
10.139.12.55	1C2G20GB	rocketmq-nameserver4、rocketmq-slave2	broker-b-s.properties

`broker-a.properties` 配置

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样
brokerName=broker-a
#0 表示 Master，>0 表示 Slave
brokerId=0
#nameServer 地址，分号分割
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=false
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=false
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间，最大100%
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq/store/abort
#限制的消息大小
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000
#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=SYNC_MASTER
#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH
#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

`broker-b.properties` 配置

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样
brokerName=broker-b
#0 表示 Master，>0 表示 Slave
brokerId=0
#nameServer 地址，分号分割
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=false
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=false
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间，最大100%
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq/store/abort
#限制的消息大小
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000
#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=ASYNC_MASTER
#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH
#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

`broker-a-s.properties` 配置

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样，与 Master 通过 brokerName 来配对
brokerName=broker-a
#0 表示 Master，>0 表示 Slave
brokerId=1
#nameServer 地址，分号分割
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq/store/abort
#限制的消息大小
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000
#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=SLAVE
#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH
#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

`broker-b-s.properties` 配置

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样，与 Master 通过 brokerName 来配对
brokerName=broker-b
#0 表示 Master，>0 表示 Slave
brokerId=1
#nameServer 地址，分号分割
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq/store/abort
#限制的消息大小
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000
#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=SLAVE
#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH
#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

分发配置文件

$ scp broker-* ocmuser@rocketmq-master2-slave:/home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync
$ scp broker-* ocmuser@rocketmq-master2:/home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync
$ scp broker-* ocmuser@rocketmq-master1-slave:/home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync

修改日志配置文件

$ mkdir -p /home/ocmuser/rocketmq-test/rocketmq/logs
$ cd /home/ocmuser/rocketmq-test/rocketmq/conf && sed -i 's#${user.home}#/home/ocmuser/rocketmq-test/#g' *.xml

修改脚本启动参数

/home/ocmuser/rocketmq-test/rocketmq/bin/runbroker.sh内容如下，至少保障1g内存

#======================================================================================
# 开发环境JVM Configuration
#======================================================================================
JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m"

/home/ocmuser/rocketmq-test/rocketmq/bin/runserver.sh内容如下

#======================================================================================
# JVM Configuration
#======================================================================================
JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m"

启动 NameServer

$ cd /home/ocmuser/rocketmq-test/rocketmq/bin
$ nohup sh mqnamesrv &

启动 Master1:BrokerServerA

$ cd /home/ocmuser/rocketmq-test/rocketmq/bin 
$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync/broker-a.properties >/dev/null 2>&1 & 
$ netstat -ntlp 
$ jps 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/broker.log 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/namesrv.log

启动 Master2:BrokerServerB

$ cd /home/ocmuser/rocketmq-test/rocketmq/bin 
$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync/broker-b.properties >/dev/null 2>&1 & 
$ netstat -ntlp 
$ jps 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/broker.log 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/namesrv.log

启动 Master1-Slave:BrokerServerC

$ cd cd /home/ocmuser/rocketmq-test/rocketmq/bin 
$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync/broker-a-s.properties >/dev/null 2>&1 & 
$ netstat -ntlp 
$ jps 
$ tail -f -n 500 cd /home/ocmuser/rocketmq-test/logs/rocketmqlogs/broker.log 
$ tail -f -n 500 cd /home/ocmuser/rocketmq-test/logs/rocketmqlogs/namesrv.log

启动 Master2-Slave:BrokerServerD

$ cd /home/ocmuser/rocketmq-test/rocketmq/bin 
$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync/broker-b-s.properties >/dev/null 2>&1 & 
$ netstat -ntlp 
$ jps 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/broker.log 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/namesrv.log

查看进程和日志，可见部署完成

角色为同步双写，异步刷盘，4个节点通信正常

控制台监控验证

修改连接配置

rocketmq.config.namesrvAddr=10.139.12.51:9876;10.139.12.52:9876;10.139.12.53:9876;10.139.12.54:9876

连接正常

查看各节点角色和状态

服务停止

首先关闭 4 个 BrokerServer，再关闭 4 个 NameServer

$ cd /home/ocmuser/rocketmq-test/rocketmq/bin 
$ sh mqshutdown broker 
$ sh mqshutdown namesrv

双主双从环境构建与调优

为取得可靠性与效率的平衡，建议采用同步双写，异步刷盘的模式

CentOS 7 安装KVM和WebVirtMgr管理面板

Sun, 11 Aug 2019 18:35:00 +0800

配置静态IP

此处使用VMware 模拟服务器环境，为防止IP在启动时动态分配，需要手动配置静态IP，如果是物理服务器可直接跳过

编辑虚拟机网络

编辑 -> 虚拟网络编辑器

更改

取消勾选此项

进入NAT设置记录默认网关和子网掩码

进入虚拟机

设置虚拟机IP

查看虚拟机网卡，lo为本地回环口，实际网卡为ens33

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
 inet 127.0.0.1/8 scope host lo
 valid_lft forever preferred_lft forever
 inet6 ::1/128 scope host
 valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
 link/ether 00:0c:29:02:fe:c5 brd ff:ff:ff:ff:ff:ff
...

编辑网卡配置

$ vi /etc/sysconfig/network-scripts/ifcfg-ens33

加入静态配置

TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens33
UUID=a63224f4-f210-4d84-933d-c4f0ccd68f7a
DEVICE=ens33
ONBOOT=yes
IPADDR=192.168.133.128
NETMASK=255.255.255.0
GATEWAY=192.168.133.2
ZONE=public

解释

ONBOOT：开机启动

BOOTPROTO：网络分配方式，静态

IPPADDR：手动指定ip地址

NETMASK：子网掩码

GATEWAY：网关ip

DNS配置

$ vi /etc/resolv.conf

加入DNS配置

nameserver=192.168.133.2
nameserver 8.8.8.8
nameserver 114.114.114.114
search localdomain

此处仅加入第一个也可以正常使用，但无法ping通域名

修改主机名

没有需求此步骤可以跳过

$ vi /etc/sysconfig/network

加入配置

# Created by anaconda
NETWORKING=yes
HOSTNAME=localhost

重启网卡

$ systemctl restart network

查看网络信息

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
 inet 127.0.0.1/8 scope host lo
 valid_lft forever preferred_lft forever
 inet6 ::1/128 scope host
 valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
 link/ether 00:0c:29:02:fe:c5 brd ff:ff:ff:ff:ff:ff
 inet 192.168.133.128/24 brd 192.168.133.255 scope global ens33
 valid_lft forever preferred_lft forever
 inet6 fe80::20c:29ff:fe02:fec5/64 scope link noprefixroute
 valid_lft forever preferred_lft forever

静态IP配置完成

安装KVM

此处部分参考GitHub的安装指引，官方wiki也有相关的指南，不过更加复杂

GitHub安装指引

官方wiki

检测是否支持KVM

$ cat /proc/cpuinfo | egrep 'vmx|svm'
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc eagerfpu pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp tpr_shadow vnmi ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid mpx rdseed adx smap clflushopt xsaveopt xsavec arat spec_ctrl intel_stibp flush_l1d arch_capabilities

关闭SELinux，将 /etc/sysconfig/selinux 中的 SELinux=enforcing 修改为 SELinux=disabled

$ vi /etc/sysconfig/selinux

安装 KVM 环境

通过 yum 安装 kvm 基础包和管理工具

kvm相关安装包及其作用:

qemu-kvm 主要的KVM程序包
python-virtinst 创建虚拟机所需要的命令行工具和程序库
virt-manager GUI虚拟机管理工具
virt-top 虚拟机统计命令
virt-viewer GUI连接程序，连接到已配置好的虚拟机
libvirt C语言工具包，提供libvirt服务
libvirt-client 为虚拟客户机提供的C语言工具包
virt-install 基于libvirt服务的虚拟机创建命令
bridge-utils 创建和管理桥接设备的工具

$ yum -y install qemu-kvm python-virtinst libvirt libvirt-python virt-manager libguestfs-tools bridge-utils virt-install

重启宿主机，以便加载 kvm 模块

$ reboot

查看KVM模块是否被正确加载

$ lsmod | grep kvm
kvm_intel 183621 0
kvm 586948 1 kvm_intel
irqbypass 13503 1 kvm

配置libvirt以启用TCP服务

LIBVIRTD_CONFIG=/etc/libvirt/libvirtd.conf
LIBVIRTD_ARGS="--listen"

同时修改/etc/libvirt/libvirtd.conf配置

...
listen_tls = 0
listen_tcp = 1
tcp_port = "16509"
listen_addr = "0.0.0.0"
auth_tcp = "none"
...

开启kvm服务，并且设置其开机自动启动

$ systemctl start libvirtd
$ systemctl enable libvirtd

查看状态操作结果，如Active: active (running)，说明运行情况良好

$ systemctl status libvirtd
● libvirtd.service - Virtualization daemon
 Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
 Active: active (running) since 日 2019-08-11 16:09:20 CST; 30min ago
 Docs: man:libvirtd(8)
 https://libvirt.org
 Main PID: 107953 (libvirtd)
 Tasks: 19 (limit: 32768)
 CGroup: /system.slice/libvirtd.service
 ├─105030 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper
 ├─105031 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper
 └─107953 /usr/sbin/libvirtd --listen

$ systemctl is-enabled libvirtd
enabled

安装WebVirtMgr管理面板

官方指南

更新源

$ yum install -y epel-release
$ yum -y install http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm

安装依赖

$ yum -y install git python-pip libvirt-python libxml2-python python-websockify supervisor nginx
$ yum -y install gcc python-devel
$ pip install numpy

拉取源码安装

拉取源码和安装环境

$ git clone git://github.com/retspen/webvirtmgr.git
$ cd webvirtmgr
$ sudo pip install -r requirements.txt

执行安装文件并按照提示设置管理员账号密码

$ ./manage.py syncdb
WARNING:root:No local_settings file found.
Creating tables ...
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
Creating table auth_user_groups
Creating table auth_user_user_permissions
Creating table auth_user
Creating table django_content_type
Creating table django_session
Creating table django_site
Creating table servers_compute
Creating table instance_instance
Creating table create_flavor

You just installed Django's auth system, which means you don't have any superusers defined.
Would you like to create one now? (yes/no): yes
Please enter either "yes" or "no": yes
Username (leave blank to use 'root'): root
Email address: 123@abc.com
Password:
Password (again):
Superuser created successfully.
Installing custom SQL ...
Installing indexes ...
Installed 6 object(s) from 1 fixture(s)

发布到Nginx

拷贝文件到nginx web目录

$ cd .. && cp -r webvirtmgr/ /var/www/webvirtmgr/

创建配置文件

$ vi /etc/nginx/conf.d/webvirtmgr.conf

内容为

server {
 listen 80 default_server;

 server_name $hostname;
 #access_log /var/log/nginx/webvirtmgr_access_log; 

 location /static/ {
 root /var/www/webvirtmgr/webvirtmgr; # or /srv instead of /var
 expires max;
 }

 location / {
 proxy_pass http://127.0.0.1:8000;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-for $proxy_add_x_forwarded_for;
 proxy_set_header Host $host:$server_port;
 proxy_set_header X-Forwarded-Proto $scheme;
 proxy_connect_timeout 600;
 proxy_read_timeout 600;
 proxy_send_timeout 600;
 client_max_body_size 1024M; # Set higher depending on your needs 
 }
}

拷贝一份副本并配置nginx.conf

$ mv /etc/nginx/nginx.conf /etc/nginx/nginx.conf.bak

写入配置

#user nobody;
worker_processes 1;

#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;

#pid logs/nginx.pid;


events {
 worker_connections 1024;
}


http {
 include mime.types;
 default_type application/octet-stream;
 include /etc/nginx/conf.d/*.conf;

 sendfile on;

 keepalive_timeout 65;

 server {
 listen 80;
 server_name localhost;

 location / {
 root html;
 index index.html index.htm;
 }

 error_page 500 502 503 504 /50x.html;
 location = /50x.html {
 root html;
 }
 }
}

重启nginx

$ systemctl restart nginx.service

配置用户组和权限

$ chown -R nginx:nginx /var/www/webvirtmgr

可能出现的报错——`Address already in use`

重启nginx服务出现错误，查看状态如下

$ systemctl status nginx.service
● nginx.service - The nginx HTTP and reverse proxy server
 Loaded: loaded (/usr/lib/systemd/system/nginx.service; disabled; vendor preset: disabled)
 Active: failed (Result: exit-code) since 日 2019-08-11 12:52:42 CST; 2s ago
 Process: 26188 ExecStart=/usr/sbin/nginx (code=exited, status=1/FAILURE)
 Process: 26185 ExecStartPre=/usr/sbin/nginx -t (code=exited, status=0/SUCCESS)
 Process: 26183 ExecStartPre=/usr/bin/rm -f /run/nginx.pid (code=exited, status=0/SUCCESS)

8月 11 12:52:40 localhost.localdomain nginx[26188]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
8月 11 12:52:40 localhost.localdomain nginx[26188]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
8月 11 12:52:41 localhost.localdomain nginx[26188]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
8月 11 12:52:41 localhost.localdomain nginx[26188]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
8月 11 12:52:42 localhost.localdomain nginx[26188]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
8月 11 12:52:42 localhost.localdomain nginx[26188]: nginx: [emerg] still could not bind()
8月 11 12:52:42 localhost.localdomain systemd[1]: nginx.service: control process exited, code=exited status=1
8月 11 12:52:42 localhost.localdomain systemd[1]: Failed to start The nginx HTTP and reverse proxy server.
8月 11 12:52:42 localhost.localdomain systemd[1]: Unit nginx.service entered failed state.
8月 11 12:52:42 localhost.localdomain systemd[1]: nginx.service failed.

解决方法：杀掉占用80端口的进程（不管是不是nginx占用的）

推荐使用fuser命令快速杀掉占用端口的进程，CentOS可通过以下命令安装

$ yum install -y psmisc

杀掉占用端口的进程

$ fuser -k 80/tcp
80/tcp: 18869 18870 18871 18872 18873

再次重启即可

$ service nginx restart
Redirecting to /bin/systemctl restart nginx.service

更新`SELinux`策略

$ /usr/sbin/setsebool httpd_can_network_connect true

$ chkconfig supervisord on
注意：正在将请求转发到“systemctl enable supervisord.service”。
Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service.

配置`Supervisor`

创建文件/etc/supervisord.d/webvirtmgr.ini

$ /etc/supervisord.d/webvirtmgr.ini

加入配置

[program:webvirtmgr]
command=/usr/bin/python /var/www/webvirtmgr/manage.py run_gunicorn -c /var/www/webvirtmgr/conf/gunicorn.conf.py
directory=/var/www/webvirtmgr
autostart=true
autorestart=true
logfile=/var/log/supervisor/webvirtmgr.log
log_stderr=true
user=nginx

[program:webvirtmgr-console]
command=/usr/bin/python /var/www/webvirtmgr/console/webvirtmgr-console
directory=/var/www/webvirtmgr
autostart=true
autorestart=true
stdout_logfile=/var/log/supervisor/webvirtmgr-console.log
redirect_stderr=true
user=nginx

启动`supervisor`守护线程

$ systemctl start supervisord.service

配置开启启动

$ systemctl enable supervisord.service

检查

查看端口占用，若80、8000、6080已经监听则正常

$ netstat -lnpt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1/systemd
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 102045/nginx: maste
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 9243/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 9543/master
tcp 0 0 127.0.0.1:6010 0.0.0.0:* LISTEN 16011/sshd: root@pt
tcp 0 0 127.0.0.1:8000 0.0.0.0:* LISTEN 102430/python
tcp 0 0 0.0.0.0:6080 0.0.0.0:* LISTEN 102429/python
tcp6 0 0 :::111 :::* LISTEN 1/systemd
tcp6 0 0 :::22 :::* LISTEN 9243/sshd
tcp6 0 0 ::1:25 :::* LISTEN 9543/master
tcp6 0 0 ::1:6010 :::* LISTEN 16011/sshd: root@pt

开放防火墙

$ firewall-cmd --zone=public --add-port=80/tcp --permanent
$ firewall-cmd --reload

访问web界面

访问虚拟机地址http://192.168.133.128/login/ 可见启动成功

TCP方式连接宿主机

连接成功

从零开始部署Rancher

Sat, 11 Aug 2018 11:41:00 +0800

基础环境部署

基础环境使用CentOS 7，版本号为1804，部署在本地虚拟机VM上

安装CentOS+docker

CentOS下载地址

使用最小安装，ISO镜像大小为906MB，在VM中导入镜像，给4G内存（Rancher2.0最低要求），安装过程系统配置均为默认配置

安装好后为纯净环境，使用ip addr即可查看ip，使用外部远程连接工具连接虚拟机ip

使用yum install docker安装docker环境

安装好后使用命令systemctl start docker启动docker 使用 docker info或者docker version查看版本

[root@localhost ~]# docker version
Client:
 Version: 1.13.1
 API version: 1.26
 Package version: docker-1.13.1-68.gitdded712.el7.centos.x86_64
 Go version: go1.9.4
 Git commit: dded712/1.13.1
 Built: Tue Jul 17 18:34:48 2018
 OS/Arch: linux/amd64

Server:
 Version: 1.13.1
 API version: 1.26 (minimum version 1.12)
 Package version: docker-1.13.1-68.gitdded712.el7.centos.x86_64
 Go version: go1.9.4
 Git commit: dded712/1.13.1
 Built: Tue Jul 17 18:34:48 2018
 OS/Arch: linux/amd64
 Experimental: false

安装Rancher

为了加速Rancher下载速度，需要配置docker镜像下载地址，此处提供一个阿里云的镜像地址

{
 "registry-mirrors": ["https://fy707np5.mirror.aliyuncs.com"]
}

使用命令编辑docker镜像地址 vi /etc/docker/daemon.json 保存后重载配置systemctl daemon-reload 重启docker服务systemctl restart docker 镜像地址配置完成

访问Rancher官网查看docker命令 Rancher英文官网

Rancher中文官网

访问quickstart查看docker命令

目前Rancher官方开始推荐2.0版本，首次使用推荐使用1.6稳定版

在下方找到1.6文档，使用命令在虚拟机中安装Rancher1.6

sudo docker run -d --restart=unless-stopped -p 8080:8080 rancher/server:stable

配置防火墙

安装后若没有打开8080端口则会报错，配置CentOS防火墙来源：[莫小安-CentOS7使用firewalld打开关闭防火墙与端口][10]

1、firewalld的基本使用启动： systemctl start firewalld 关闭： systemctl stop firewalld 查看状态： systemctl status firewalld 开机禁用： systemctl disable firewalld 开机启用： systemctl enable firewalld

2.systemctl是CentOS7的服务管理工具中主要的工具，它融合之前service和chkconfig的功能于一体。启动一个服务：systemctl start firewalld.service 关闭一个服务：systemctl stop firewalld.service 重启一个服务：systemctl restart firewalld.service 显示一个服务的状态：systemctl status firewalld.service 在开机时启用一个服务：systemctl enable firewalld.service 在开机时禁用一个服务：systemctl disable firewalld.service 查看服务是否开机启动：systemctl is-enabled firewalld.service 查看已启动的服务列表：systemctl list-unit-files|grep enabled 查看启动失败的服务列表：systemctl –failed

3.配置firewalld-cmd

查看版本： firewall-cmd –version 查看帮助： firewall-cmd –help 显示状态： firewall-cmd –state 查看所有打开的端口： firewall-cmd –zone=public –list-ports 更新防火墙规则： firewall-cmd –reload 查看区域信息: firewall-cmd –get-active-zones 查看指定接口所属区域： firewall-cmd –get-zone-of-interface=eth0 拒绝所有包：firewall-cmd –panic-on 取消拒绝状态： firewall-cmd –panic-off 查看是否拒绝： firewall-cmd –query-panic

那怎么开启一个端口呢添加 firewall-cmd –zone=public –add-port=80/tcp –permanent （–permanent永久生效，没有此参数重启后失效）重新载入 firewall-cmd –reload 查看 firewall-cmd –zone= public –query-port=80/tcp 删除 firewall-cmd –zone= public –remove-port=80/tcp –permanent

通过命令放行8080端口

firewall-cmd --zone=public --add-port=8080/tcp --permanent

重载防火墙

firewall-cmd --reload

查看端口放行情况

 firewall-cmd --zone=public --list-ports

设置Rancher

设置Rancher界面语言

通过虚拟机ip+端口号访问Rancher：http://192.168.79.142:8080

默认为英文界面，Rancher官方有中文支持修改右下方语言选项即可

设置Rancher主机

首次访问需要设置主机

按照默认识别的IP即可，点击保存

选择主机类型，包含自定义主机、亚马逊主机等，此处选择自定义主机

依照指示操作，为了确保通信测试，暂时关闭防火墙

为了测试多台Rancher Server和agent服务，使用VM保存快照并复制一份虚拟机启动目前测试：128为server，129为agent，注意此处填的是客户端的地址，非主机地址

设置Rancher客户端

复制Server的配置命令，在agent服务器上运行，等待下载

按照指示，运行后关闭Server主机配置查看主机列表

也可以添加环境、测试环境、生产环境等

Hadoop分布式搭建测试

Sat, 21 Apr 2018 18:10:00 +0800

搭建环境

Windows VM虚拟机

虚拟机配置 CPU:7700 单核 RAM:2G SSD:20G OS:CentOS 7.4

环境配置

修改主机名

因为三台虚拟机均为一台虚拟机克隆而成，且源主机名为localhost，先计划将三台主机分别设置为hadoop0，hadoop1，hadoop2 IP、主机名、职能分别为： 192.168.79.129 hadoop0 NameNode DataNode ResourceManager NodeManager 192.168.79.130 hadoop1 DataNode NodeManager 192.168.79.131 hadoop2 DataNode NodeManager

CentOS 7使用hostnamectl set-hostname hadoop0设置主机名，其他几台同样操作，输入后可通过hostnamectl status查看是否修改成功

 [root@hadoop0 ~]# hostnamectl status
 Static hostname: hadoop0
 Icon name: computer-vm
 Chassis: vm
 Machine ID: 4eddbba909c84e159cfa474604412c3c
 Boot ID: df37e835fde94d039153028c9479b68e
 Virtualization: vmware
 Operating System: CentOS Linux 7 (Core)
 CPE OS Name: cpe:/o:centos:centos:7
 Kernel: Linux 3.10.0-693.el7.x86_64
 Architecture: x86-64

由于hadoop0是根节点，用于资源调度，故其需要具有所有DataNode的访问地址和权限，进入/etc/hosts文件，添加如下内容将主机名解析到对应的IP

 192.168.79.129 hadoop0
 192.168.79.130 hadoop1
 192.168.79.131 hadoop2

SSH免密码登录

进入当前用户的根目录/home/hadoop/.ssh/，在该目录下执行ssh-keygen -t rsa生成主机公钥，过程中弹出的除了 Overwrite (y/n)? y选择y确认覆盖之外，其余全为空值，则可生成免密码登录的公钥 【注意】Ubuntu系统操作略有特殊，建议使用CentOS系统进行操作

[hadoop@hadoop0 root]$ cd /home/hadoop/.ssh/
[hadoop@hadoop0 .ssh]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
/home/hadoop/.ssh/id_rsa already exists.
Overwrite (y/n)? y
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:RjyGCGedx9xzxXG97KGgjXMIZ/pg7Q+tLU1nJoxIhm8 hadoop@hadoop0
The key's randomart image is:
+---[RSA 2048]----+
| . o. + . oo.o|
| + .oo+ o . ...|
| . o.= o . .|
| . * + . + |
| + S B . o .|
| E B.* = . |
| o +.=.= |
| ++. |
| .oo |
+----[SHA256]-----+

使用ssh-copy-id将本地主机的公钥复制到远程主机的authorized_keys文件上，此时先对本地主机hadoop0授权

[hadoop@hadoop0 .ssh]$ ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop0
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/hadoop/.ssh/id_rsa.pub"
The authenticity of host 'hadoop0 (fe80::e99d:ad75:28e9:9276%ens33)' can't be established.
ECDSA key fingerprint is SHA256:GEEH9kBVcDuaYpJZLNsaWZA3O47+HdqIbd91fGAWcew.
ECDSA key fingerprint is MD5:cd:df:9e:8c:92:3a:9f:ae:b0:10:9b:b6:25:9c:4d:c6.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
hadoop@hadoop0's password:

Number of key(s) added: 1

Now try logging into the machine, with: "ssh 'hadoop0'"
and check to make sure that only the key(s) you wanted were added.

测试SSH免密码登录成功

 [hadoop@hadoop0 /]$ ssh hadoop0
 Last login: Sat Apr 21 17:18:37 2018
 [hadoop@hadoop0 ~]$ exit
 登出
 Connection to hadoop0 closed.

对另外两台主机访问授权 hadoop1

 [hadoop@hadoop0 root]$ ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop1
 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/hadoop/.ssh/id_rsa.pub"
 The authenticity of host 'hadoop1 (192.168.79.130)' can't be established.
 ECDSA key fingerprint is SHA256:GEEH9kBVcDuaYpJZLNsaWZA3O47+HdqIbd91fGAWcew.
 ECDSA key fingerprint is MD5:cd:df:9e:8c:92:3a:9f:ae:b0:10:9b:b6:25:9c:4d:c6.
 Are you sure you want to continue connecting (yes/no)? yes
 /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
 /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
 hadoop@hadoop1's password:
 
 Number of key(s) added: 1
 
 Now try logging into the machine, with: "ssh 'hadoop1'"
 and check to make sure that only the key(s) you wanted were added.

hadoop2

 [hadoop@hadoop0 root]$ ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop2
 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/hadoop/.ssh/id_rsa.pub"
 The authenticity of host 'hadoop2 (192.168.79.131)' can't be established.
 ECDSA key fingerprint is SHA256:GEEH9kBVcDuaYpJZLNsaWZA3O47+HdqIbd91fGAWcew.
 ECDSA key fingerprint is MD5:cd:df:9e:8c:92:3a:9f:ae:b0:10:9b:b6:25:9c:4d:c6.
 Are you sure you want to continue connecting (yes/no)? yes
 /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
 /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
 hadoop@hadoop2's password:
 
 Number of key(s) added: 1
 
 Now try logging into the machine, with: "ssh 'hadoop2'"
 and check to make sure that only the key(s) you wanted were added.

对另外两台主机做类似操作，即仅对本地主机授权，无需对hadoop0主机授权，操作后可查看authorized_keys文件内容，当前主机名即可

环境变量

三台主机均由之前的单节点主机克隆而来，故宝塔、java、hadoop环境是自带的 Java8

 [hadoop@hadoop0 .ssh]$ java -version
 java version "1.8.0_121"
 Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
 Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
 [hadoop@hadoop0 .ssh]$ javac -version
 javac 1.8.0_121

hadoop

 [hadoop@hadoop0 .ssh]$ cd /www/hadoop
 [hadoop@hadoop0 hadoop]$
 [hadoop@hadoop0 hadoop]$ ls
 hadoop-2.6.0-cdh5.7.0 hadoop-2.6.0-cdh5.7.0.tar.gz testFile tmp
 [hadoop@hadoop0 hadoop]$ source ~/.bash_profile

$HADOOP_HOME

 [hadoop@hadoop0 hadoop]$ echo $HADOOP_HOME
 /www/hadoop/hadoop-2.6.0-cdh5.7.0

hadoop配置

若之前配置过hadoop，建议将原目录删除后重新解压，主节点和从节点的目录都要删除干净，否则会导致大量问题 配置主节点core-site.xml

 <configuration>
 <property>
 <name>fs.defaultFS</name>
 <value>hdfs://192.168.79.129:8020</value>
 </property>
 </configuration>

配置HDFS文件存放目录和节点数hdfs-site.xml

 <property>
 <name>dfs.namenode.name.dir</name>
 <value>/www/hadoop/app/tmp/dfs/name</value>
 </property>
 <property>
 <name>dfs.datanode.data.dir</name>
 <value>/www/hadoop/app/tmp/dfs/data</value>
 </property>

配置YARN的yarn.xml，删除单节点的配置，添加如下字段

 <configuration>
 
 <!-- Site specific YARN configuration properties -->
 <property>
 <name>yarn.nodemanager.aux-services</name>
 <value>mapreduce_shuffle</value>
 </property>
 
 	<property>
 		<name>yarn.log-aggregation-enable</name>
 		<value>true</value>
 	</property>
 
 <!--ResourceManager主机名-->
 <property>
 		<name>yarn.resourcemanager.hostname</name>
 		<value>hadoop0</value>
 	</property>
 
 </configuration>

配置从节点slave，默认slave为localhost

 [hadoop@hadoop0 hadoop]$ cat slaves
 localhost

加入host配置

 [hadoop@hadoop0 hadoop]$ cat slaves
 hadoop0
 hadoop1
 hadoop2

额外配置，启用jobhistory

 <configuration>
 <property>
 <name>mapreduce.framework.name</name>
 <value>yarn</value>
 </property>
 
 <property>
 		<name>mapreduce.jobhistory.address</name>
 		<value>192.168.79.129:10020</value>
 		<description>MapReduce JobHistory Server IPC host:port</description>
 	</property>
 
 	<property>
 		<name>mapreduce.jobhistory.webapp.address</name>
 		<value>192.168.79.129:19888</value>
 		<description>MapReduce JobHistory Server Web UI host:port</description>
 	</property>
 
 	<property>
 		<name>mapreduce.jobhistory.done-dir</name>
 		<value>/history/done</value>
 	</property>
 
 	<property>
 		<name>mapreduce.jobhistory.intermediate-done-dir</name>
 		<value>/history/done_intermediate</value>
 	</property>
 
 </configuration>

【注意】配置好后将单节点时的/www/hadoop/hadoop-2.6.0-cdh5.7.0/logs文件夹删除或者直接重新解压hadoop的压缩包防止出现文件夹权限导致的错误

资源分发

将主节点的所有配置分发到从节点，避免重复配置。在主节点执行如下命令即可

 scp -r /www/hadoop hadoop@hadoop1:/www
 scp -r /www/hadoop hadoop@hadoop2:/www

分发完成

去根节点查看文件是否已经传到，分别执行source ~/.bash_profile以启用hadoop命令

注意分发前执行一次hadoop命令是报未找到命令，说明从节点的hadoop旧根目录是删除干净的，分发完成后再次执行命令就有效了

NameNode格式化

格式化只对主节点执行即可

 hdfs namenode -format

格式化成功

hadoop分布式启动

主节点启动，在/sbin目录下，执行./start-all.sh

 [hadoop@hadoop0 bin]$ cd ../sbin
 [hadoop@hadoop0 sbin]$ ./start-all.sh
 This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
 18/04/21 22:16:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 Starting namenodes on [hadoop0]

从节点的datanode和namenode逐一启动

 hadoop0: starting namenode, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-namenode-hadoop0.out
 hadoop1: starting datanode, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop1.out
 hadoop0: starting datanode, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop0.out
 hadoop2: starting datanode, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop2.out
 Starting secondary namenodes [0.0.0.0]

主节点secondarynamenode启动

 0.0.0.0: starting secondarynamenode, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-secondarynamenode-hadoop0.out
 18/04/21 22:16:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

启动yarn和resourcemanager、nodemanager

 starting yarn daemons
 starting resourcemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-resourcemanager-hadoop0.out
 hadoop1: starting nodemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-hadoop1.out
 hadoop2: starting nodemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-hadoop2.out
 hadoop0: starting nodemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-hadoop0.out

hadoop分布式启动验证

命令行方式

使用jps查看进程，以下进程少一个都不行，若缺少，查看另一篇文章寻找解决办法hadoop分布式搭建的一些坑

[[hadoop分布式搭建的一些坑]]

启动主节点后从节点的HDFS会自动启动，进入从节点，使用jps查看，有两个进程即成功

 [hadoop@hadoop2 hadoop]$ jps
 29876 Jps
 29575 DataNode
 29692 NodeManager

webUI方式

使用webUI确认所有进程成功启动，查看HDFS节点

查看YARN

已经有三个从节点注册到YARN

分布式集群测试

分布式文件测试

上传测试文件再查看节点情况初始节点为空

 [hadoop@hadoop0 hadoop-2.6.0-cdh5.7.0]$ ls
 bin cloudera examples include libexec logs README.txt share
 bin-mapreduce1 etc examples-mapreduce1 lib LICENSE.txt NOTICE.txt sbin src
 [hadoop@hadoop0 hadoop-2.6.0-cdh5.7.0]$ hadoop fs -ls /
 18/04/21 22:33:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

新建数据目录

 [hadoop@hadoop0 hadoop-2.6.0-cdh5.7.0]$ hadoop fs -mkdir /data
 18/04/21 22:33:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 [hadoop@hadoop0 hadoop-2.6.0-cdh5.7.0]$ hadoop fs -ls /
 18/04/21 22:33:45 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 Found 1 items
 drwxr-xr-x - hadoop supergroup 0 2018-04-21 22:33 /data

上传小文件

 [hadoop@hadoop0 hadoop-2.6.0-cdh5.7.0]$ hadoop fs -put README.txt /data
 18/04/21 22:34:13 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 [hadoop@hadoop0 hadoop-2.6.0-cdh5.7.0]$ hadoop fs -ls /data
 18/04/21 22:34:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 Found 1 items
 -rw-r--r-- 3 hadoop supergroup 1366 2018-04-21 22:34 /data/README.txt

上传大文件

 [hadoop@hadoop0 hadoop]$ ll
 总用量 304284
 drwxrwxr-x 3 hadoop hadoop 17 4月 21 22:16 app
 drwxrwxrwx 15 www www 253 4月 21 22:08 hadoop-2.6.0-cdh5.7.0
 -rwxrwxrwx 1 www www 311585484 3月 31 23:06 hadoop-2.6.0-cdh5.7.0.tar.gz
 drwxr-xr-x 2 www www 301 4月 11 06:31 testFile
 [hadoop@hadoop0 hadoop]$ hadoop fs -put hadoop-2.6.0-cdh5.7.0.tar.gz /data
 18/04/21 22:35:59 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 [hadoop@hadoop0 hadoop]$ hadoop fs -ls /data
 18/04/21 22:36:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 Found 2 items
 -rw-r--r-- 3 hadoop supergroup 1366 2018-04-21 22:34 /data/README.txt
 -rw-r--r-- 3 hadoop supergroup 311585484 2018-04-21 22:36 /data/hadoop-2.6.0-cdh5.7.0.tar.gz

通过webUI查看文件情况

分布式MapReduce测试

测试MapReduce，进入/www/hadoop/hadoop-2.6.0-cdh5.7.0/share/hadoop/mapreduce目录，执行pi运算hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar pi 2 3 【注意】由于此处是虚拟机，不建议将后面的精度设置过大，否则会报IO错误 进入YARN的web界面查看

运行成功

分布式日志离线处理测试

上传日志数据 [hadoop@hadoop0 testFile]$ hadoop fs -put access.log /data 18/04/21 23:42:13 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable 执行

 hadoop jar hadoopstudy-0.6-jar-with-dependencies.jar com.fjy.hadoop.hadoop.project.LogApp /data/access.log /logaccess/out

Web查看工作详情

查看结果

题外话

其实同时运行三台虚拟机是很吃配置的，特别是内存，对于hadoop来说内存太小很多都会报异常，因此本次的测试机器配置还可以的，三台虚拟机均运行在480G SATA3的固态上 CPU:7700 4core 8thread RAM:16G DDR4 Dual Channal SSD:128+480 nvme+sata3 HDD:9T sata3 Eth:1000M I219V 但在实际运行中依然顶着内存瓶颈，多次使用了系统盘的交换分区

CentOS on F嘉阳

RocketMQ学习笔记6——集群实战

组建高性能MQ服务

集群拓扑图

配置文件最优设置解析

修改host文件映射

创建存储目录

建立软连接

修改配置文件

broker-a.properties 配置

broker-b.properties 配置

broker-a-s.properties 配置

broker-b-s.properties 配置

修改日志配置文件

修改脚本启动参数

启动 NameServer

启动 Master1:BrokerServerA

启动 Master2:BrokerServerB

启动 Master1-Slave:BrokerServerC

启动 Master2-Slave:BrokerServerD

控制台监控验证

服务停止

双主双从环境构建与调优

CentOS 7 安装KVM和WebVirtMgr管理面板

配置静态IP

编辑虚拟机网络

设置虚拟机IP

DNS配置

修改主机名

重启网卡

安装KVM

检测是否支持KVM

安装 KVM 环境

安装WebVirtMgr管理面板

更新源

安装依赖

拉取源码安装

发布到Nginx

可能出现的报错——Address already in use

更新SELinux策略

配置Supervisor

启动supervisor守护线程

检查

开放防火墙

访问web界面

TCP方式连接宿主机

从零开始部署Rancher

基础环境部署

安装CentOS+docker

安装Rancher

配置防火墙

设置Rancher

设置Rancher界面语言

设置Rancher主机

设置Rancher客户端

Hadoop分布式搭建测试

搭建环境

环境配置

修改主机名

SSH免密码登录

环境变量

hadoop配置

资源分发

NameNode格式化

hadoop分布式启动

hadoop分布式启动验证

命令行方式

webUI方式

分布式集群测试

分布式文件测试

分布式MapReduce测试

分布式日志离线处理测试

题外话

`broker-a.properties` 配置

`broker-b.properties` 配置

`broker-a-s.properties` 配置

`broker-b-s.properties` 配置

可能出现的报错——`Address already in use`

更新`SELinux`策略

配置`Supervisor`

启动`supervisor`守护线程