Linux on F嘉阳

Kubesphere安装——以GCP为例

Sat, 19 Jun 2021 22:15:00 +0800

平台准备

创建工作空间

启用计算引擎API

准备虚拟机

作为演示环境，此处使用Google Cloud Platform创建3台虚拟机，配置如下

CPU	内存	磁盘
4	16G	40G
4	16G	40G
4	16G	40G

创建实例模板

创建需要的实例模板

配置操作系统、资源等，注意放行防火墙

创建实例

基于实例模板创建3台虚拟机，选择地区为香港

此处用于演示用途，可选择无状态机器，机器删除后不会保留磁盘内容

注意关闭自动扩缩容

等待实例创建

由于资源调度问题，最终只成功分配两台虚拟机，不过对于演示环境安装过程一样。

配置连接秘钥

添加本地生成好的rsa秘钥即可

本地SSH工具配置私钥即可连接

前置环境准备

kubesphere建议前置项

节点要求

所有节点必须都能通过 SSH 访问。

所有节点时间同步。

所有节点都应使用 sudo/curl/openssl。

依赖项要求
KubeKey 可以一同安装 Kubernetes 和 KubeSphere。根据要安装的 Kubernetes 版本，需要安装的依赖项可能会不同。您可以参考下表，查看是否需要提前在节点上安装相关依赖项。

依赖项 Kubernetes 版本 ≥ 1.18 Kubernetes 版本 < 1.18

socat 必须可选，但建议安装

conntrack 必须可选，但建议安装

ebtables 可选，但建议安装可选，但建议安装

ipset 可选，但建议安装可选，但建议安装

依赖项	Kubernetes 版本 ≥ 1.18	Kubernetes 版本 < 1.18
`socat`	必须	可选，但建议安装
`conntrack`	必须	可选，但建议安装
`ebtables`	可选，但建议安装	可选，但建议安装
`ipset`	可选，但建议安装	可选，但建议安装

以下内容所有节点都需要执行

$ yum install -y socat curl openssl conntrack ebtables ipset

主节点下载kubekey，由于使用google虚拟机，能正常访问google和github，故直接下载即可

$ curl -sfL https://get-kk.kubesphere.io | VERSION=v1.1.0 sh -
$ chmod +x kk

放行VPC防火墙

创建集群

服务器免密登陆

主节点生成免密登陆配置

$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:yxln4RFEXhsvIMcL0/EzF8CLUpo/hMlkM8KBDcNEjbU root@kubesphere-instance-group-1-bx5d
The key's randomart image is:
+---[RSA 2048]----+
| +=Oo..=Bo+.. |
| +.=.B++=.+ . |
| E= @+o*.o |
| B.+o.= |
| S++ |
| . *o |
| + . |
| |
| |
+----[SHA256]-----+

所有主机允许公钥免密登陆

$ vi /etc/ssh/sshd_config
# 将下面配置改为yes并去掉注释
PubkeyAuthentication yes
PermitRootLogin yes

$ service sshd restart

控制台添加主节点公钥，则所有新建主机都能被主节点访问

创建示例配置文件

$./kk create config --with-kubernetes v1.20.4 --with-kubesphere v3.1.0 -f gcp-config

按默认配置只安装控制台和基础服务

apiVersion: kubekey.kubesphere.io/v1alpha1
kind: Cluster
metadata:
 name: sample
spec:
 hosts:
 - {name: node1, address: 34.92.90.181, internalAddress: 10.170.0.5, privateKeyPath: "/root/.ssh/id_rsa"}
 - {name: node2, address: 35.220.138.230, internalAddress: 10.170.0.6, privateKeyPath: "/root/.ssh/id_rsa"}
 roleGroups:
 etcd:
 - node1
 master: 
 - node1
 worker:
 - node1
 - node2
 controlPlaneEndpoint:
 domain: lb.kubesphere.local
 address: ""
 port: 6443
 kubernetes:
 version: v1.20.4
 imageRepo: kubesphere
 clusterName: cluster.local
 network:
 plugin: calico
 kubePodsCIDR: 10.233.64.0/18
 kubeServiceCIDR: 10.233.0.0/18
 registry:
 registryMirrors: []
 insecureRegistries: []
 addons: []


---
apiVersion: installer.kubesphere.io/v1alpha1
kind: ClusterConfiguration
metadata:
 name: ks-installer
 namespace: kubesphere-system
 labels:
 version: v3.1.0
spec:
 persistence:
 storageClass: "" 
 authentication:
 jwtSecret: ""
 zone: ""
 local_registry: "" 
 etcd:
 monitoring: false 
 endpointIps: localhost 
 port: 2379 
 tlsEnable: true
 common:
 redis:
 enabled: false
 redisVolumSize: 2Gi 
 openldap:
 enabled: false
 openldapVolumeSize: 2Gi 
 minioVolumeSize: 20Gi
 monitoring:
 endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090
 es: 
 elasticsearchMasterVolumeSize: 4Gi 
 elasticsearchDataVolumeSize: 20Gi 
 logMaxAge: 7 
 elkPrefix: logstash
 basicAuth:
 enabled: false
 username: ""
 password: ""
 externalElasticsearchUrl: ""
 externalElasticsearchPort: "" 
 console:
 enableMultiLogin: true 
 port: 30880
 alerting: 
 enabled: false
 # thanosruler:
 # replicas: 1
 # resources: {}
 auditing: 
 enabled: false
 devops: 
 enabled: false
 jenkinsMemoryLim: 2Gi 
 jenkinsMemoryReq: 1500Mi 
 jenkinsVolumeSize: 8Gi 
 jenkinsJavaOpts_Xms: 512m 
 jenkinsJavaOpts_Xmx: 512m
 jenkinsJavaOpts_MaxRAM: 2g
 events: 
 enabled: false
 ruler:
 enabled: true
 replicas: 2
 logging: 
 enabled: false
 logsidecar:
 enabled: true
 replicas: 2
 metrics_server: 
 enabled: false
 monitoring:
 storageClass: ""
 prometheusMemoryRequest: 400Mi 
 prometheusVolumeSize: 20Gi 
 multicluster:
 clusterRole: none 
 network:
 networkpolicy:
 enabled: false
 ippool:
 type: none
 topology:
 type: none
 notification: 
 enabled: false
 openpitrix:
 store:
 enabled: false
 servicemesh: 
 enabled: false 
 kubeedge:
 enabled: false
 cloudCore:
 nodeSelector: {"node-role.kubernetes.io/worker": ""}
 tolerations: []
 cloudhubPort: "10000"
 cloudhubQuicPort: "10001"
 cloudhubHttpsPort: "10002"
 cloudstreamPort: "10003"
 tunnelPort: "10004"
 cloudHub:
 advertiseAddress: 
 - "" 
 nodeLimit: "100"
 service:
 cloudhubNodePort: "30000"
 cloudhubQuicNodePort: "30001"
 cloudhubHttpsNodePort: "30002"
 cloudstreamNodePort: "30003"
 tunnelNodePort: "30004"
 edgeWatcher:
 nodeSelector: {"node-role.kubernetes.io/worker": ""}
 tolerations: []
 edgeWatcherAgent:
 nodeSelector: {"node-role.kubernetes.io/worker": ""}
 tolerations: []

安装集群

此时，在虚拟机安装代理即可查看虚拟机监控

创建完成

访问成功

集群正常，不安装组件时内存占用率并不高

安装kubectl

$ yum install -y kubectl

安装可插拔组件

点击自定义资源 CRD，然后在搜索栏中输入 clusterconfiguration，点击搜索结果进入其详情页面。

按照自身需要启用可插拔组件，此处打开所有组件，即修改为如下配置

apiVersion: installer.kubesphere.io/v1alpha1
kind: ClusterConfiguration
metadata:
 labels:
 version: v3.1.0
 name: ks-installer
 namespace: kubesphere-system
spec:
 alerting:
 enabled: true
 auditing:
 enabled: true
 authentication:
 jwtSecret: ''
 common:
 es:
 basicAuth:
 enabled: false
 password: ''
 username: ''
 elasticsearchDataVolumeSize: 20Gi
 elasticsearchMasterVolumeSize: 4Gi
 elkPrefix: logstash
 externalElasticsearchPort: ''
 externalElasticsearchUrl: ''
 logMaxAge: 7
 minioVolumeSize: 2Gi
 monitoring:
 endpoint: 'http://prometheus-operated.kubesphere-monitoring-system.svc:9090'
 openldap:
 enabled: true
 openldapVolumeSize: 2Gi
 redis:
 enabled: true
 redisVolumSize: 2Gi
 console:
 enableMultiLogin: true
 port: 30880
 devops:
 enabled: true
 jenkinsJavaOpts_MaxRAM: 2g
 jenkinsJavaOpts_Xms: 512m
 jenkinsJavaOpts_Xmx: 512m
 jenkinsMemoryLim: 2Gi
 jenkinsMemoryReq: 1500Mi
 jenkinsVolumeSize: 8Gi
 etcd:
 endpointIps: 10.170.0.5
 monitoring: true
 port: 2379
 tlsEnable: true
 events:
 enabled: true
 ruler:
 enabled: true
 replicas: 2
 kubeedge:
 cloudCore:
 cloudHub:
 advertiseAddress:
 - ''
 nodeLimit: '100'
 cloudhubHttpsPort: '10002'
 cloudhubPort: '10000'
 cloudhubQuicPort: '10001'
 cloudstreamPort: '10003'
 nodeSelector:
 node-role.kubernetes.io/worker: ''
 service:
 cloudhubHttpsNodePort: '30002'
 cloudhubNodePort: '30000'
 cloudhubQuicNodePort: '30001'
 cloudstreamNodePort: '30003'
 tunnelNodePort: '30004'
 tolerations: []
 tunnelPort: '10004'
 edgeWatcher:
 edgeWatcherAgent:
 nodeSelector:
 node-role.kubernetes.io/worker: ''
 tolerations: []
 nodeSelector:
 node-role.kubernetes.io/worker: ''
 tolerations: []
 enabled: true
 logging:
 enabled: true
 logsidecar:
 enabled: true
 replicas: 2
 metrics_server:
 enabled: true
 monitoring:
 prometheusMemoryRequest: 400Mi
 prometheusVolumeSize: 5Gi
 storageClass: ''
 multicluster:
 clusterRole: none
 network:
 ippool:
 type: calico
 networkpolicy:
 enabled: true
 topology:
 type: weave-scope
 openpitrix:
 store:
 enabled: true
 persistence:
 storageClass: ''
 servicemesh:
 enabled: true

通过在线控制台，查看安装进度

执行以下命令

$ kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f

安装组件过程中，对CPU负载较大

所有组件安装完成

资源占用情况

Kubesphere运维Kubernetes集群

Sun, 23 May 2021 22:58:00 +0800

背景

使用原生Kubernetes-dashboard和Grafana+Prometheus管理较大规模集群比较困难，有如下缺陷

每次执行命令都要登陆服务器
容器日志采集方案、监控方案搭建比较繁琐
添加节点不方便
管理存储插件、秘钥、配置不方便

解决方案

目前国内流行的k8s管理平台主要有rancher、kubesphere、kubeboard

rancher主打原生集群管理，侧重运维人员
kubesphere主打开发运维工作流，侧重研发人员
kubeboard与kubesphere类似，侧重微服务管理，但功能上较少，对多节点管理需要收费

经过评估，最终决定采用kubesphere进行集群管理

Rancher — 多云混合云多集群Kubernetes管理平台

[post url=“https://kubesphere.io/zh/” title=“Kubesphere” intro=“面向云原生应用的容器混合云” cover=“https://kubesphere.io/images/logo.svg” /]

Kuboard — 快速在 Kubernetes 上落地微服务

安装步骤

目标是面对小规模集群管理，节点数最多为5个，故不考虑高可用集群配置，采用单主节点+4个worker节点配置

采用纯净CentOS 7 Linux系统，需要做以下前置准备

依赖包安装

安装官方要求的依赖包，由于计划安装NFS存储插件，还需要安装nfs工具包

$ yum install -y socat conntrack ebtables ipset nfs-utils

配置DNS解析

将各节点DNS配置为私有DNS服务器

$ tee /etc/resolv.conf <<-'EOF'
# Generated by NetworkManager
nameserver 192.168.1.13
EOF

私有DNS搭建可以参考文章

[[利用Docker自建Dns服务器]]

按照官方教程安装多节点kubernetes

强烈建议先用KubeKey安装纯净k8s集群+控制面板，关闭所有插件。等安装完毕再一个个插件打开，防止因为内存不够而无法打开kubesphere

安装步骤参考官方文档一键安装方式

多节点安装 — Kubesphere

启动成功

启动成功后按照官方指南打开插件

启用可插拔组件 — Kubesphere

所有插件打开后如下

使用

节点状态

存储卷管理

监控告警

日志采集

日志查询

操作审计

工具箱

企业空间权限控制

内置helm应用商店，支持上线审批，在线编辑等

支持关联外部仓库

应用流量治理

链路追踪

灰度发布

费用统计

更多的使用场景包括：DevOps、边缘节点、多集群管理还在探索中

利用Docker自建DNS服务器

Sun, 09 May 2021 18:14:12 +0800

背景

当Linux集群中机器不断增多，集群内通过配置/hosts实现域名内网访问。

目标

相同域名在集群之外通过公有DNS访问外网地址
相同域名在集群内通过DNS解析访问内网地址

步骤

搭建DNS服务器

DNS服务器可以通过jpillora/dnsmasq容器实现

jpillora/dnsmasq — DockerHub

新增配置文件 dnsmasq.conf

# 完整配置文件参考以下地址
# http://oss.segetech.com/intra/srv/dnsmasq.conf
# 记录所有DNS查询请求
log-queries
# 不使用本地配置的域名解析器
no-resolv
# 不使用本地hosts
no-hosts
# 配置上游DNS服务器地址，用于解析国内域名
server=114.114.114.114
# 配置国际DNS服务器地址
server=8.8.8.8
strict-order
# 配置内网解析
address=/gitlab.fjy8018.top/192.168.1.11

配置docker-compose文件，DNS服务器默认通过53 udp端口解析，不建议修改53端口的外部地址

version: "3.5"

services:

 dnsmasq:
 image: jpillora/dnsmasq:1.1.0
 networks:
 - dns-net
 ports:
 - 53:53/udp
 - 5380:8080
 volumes:
 - ./dnsmasq.conf:/etc/dnsmasq.conf
 environment:
 - "HTTP_USER=admin"
 - "HTTP_PASS=admin"
 restart: always

networks:
 dns-net:

启动服务即可

后续需要修改DNS解析只需要访问5380的web UI界面即可动态调整

服务器指向新DNS服务器

将所有服务器/etc/resolv.conf文件指向新DNS服务器地址

$ tee /etc/resolv.conf <<-'EOF'
# Generated by NetworkManager
nameserver 192.168.1.13
EOF

补充

对于kubernetes集群，由于集群内使用了coredns和nodelocaldns缓存服务，还需要重启两个dns服务组件

$ kubectl -n kube-system rollout restart daemonset nodelocaldns
$ kubectl -n kube-system rollout restart deploy coredns

k8s证书失效问题排查

Sun, 27 Dec 2020 12:00:58 +0800

问题背景

生产环境kuberneties集群搭建于2019年11月7日，使用kubeadm结合国内镜像搭建，参考文章

2020年11月8日突然发现生产环境集群服务全都失效并且不可用，进行紧急排查处理

处理过程

1. 排查服务是否启动

[root@k8smaster1 ~]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
65dff767f422 b95b1efa0436 "kube-controller-m..." 20 hours ago Up 20 hours k8s_kube-controller-manager_kube-controller-manager-k8smaster1_kube-system_7a524400c16990c1a69c237c9da4c7f2_11
1b26acecbcea registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 "/pause" 20 hours ago Up 20 hours k8s_POD_kube-controller-manager-k8smaster1_kube-system_7a524400c16990c1a69c237c9da4c7f2_1
9c6449df0a18 00638a24688b "kube-scheduler --..." 20 hours ago Up 20 hours k8s_kube-scheduler_kube-scheduler-k8smaster1_kube-system_baebad22afce708e52c6d3f886ff9424_10
028a1177cc9b registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 "/pause" 20 hours ago Up 20 hours k8s_POD_kube-scheduler-k8smaster1_kube-system_baebad22afce708e52c6d3f886ff9424_1
6c6bdc2c2644 registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 "/pause" 20 hours ago Up 20 hours k8s_POD_kube-apiserver-k8smaster1_kube-system_0e954d211e160f48f5ca9fd42d295c10_1
fe4a12daa7d7 registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 "/pause" 20 hours ago Up 20 hours k8s_POD_etcd-k8smaster1_kube-system_6119961323d801d05a7dd23e429cda3f_1

容器服务正常，但kubelet服务有报错信息，报连接不上apiserver

[root@k8smaster1 ~]$ systemctl status kubelet -l
● kubelet.service - kubelet: The Kubernetes Node Agent
 Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
 Drop-In: /usr/lib/systemd/system/kubelet.service.d
 └─10-kubeadm.conf
 Active: active (running) since 日 2020-11-08 22:16:07 CST; 20h ago
 Docs: https://kubernetes.io/docs/
 Main PID: 15313 (kubelet)
 Tasks: 16
 Memory: 68.4M
 CGroup: /system.slice/kubelet.service
 └─15313 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1

11月 09 18:43:06 k8smaster1 kubelet[15313]: E1109 18:43:06.967416 15313 reflector.go:126] k8s.io/kubernetes/pkg/kubelet/kubelet.go:442: Failed to list *v1.Service: Get https://192.168.1.14:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.1.14:6443: connect: connection refused
11月 09 18:43:07 k8smaster1 kubelet[15313]: E1109 18:43:07.009411 15313 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:43:07 k8smaster1 kubelet[15313]: E1109 18:43:07.109665 15313 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:43:07 k8smaster1 kubelet[15313]: I1109 18:43:07.165755 15313 kubelet_node_status.go:283] Setting node annotation to enable volume controller attach/detach
11月 09 18:43:07 k8smaster1 kubelet[15313]: E1109 18:43:07.167106 15313 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.CSIDriver: Get https://192.168.1.14:6443/apis/storage.k8s.io/v1beta1/csidrivers?limit=500&resourceVersion=0: dial tcp 192.168.1.14:6443: connect: connection refused
11月 09 18:43:07 k8smaster1 kubelet[15313]: I1109 18:43:07.169000 15313 kubelet_node_status.go:72] Attempting to register node k8smaster1
11月 09 18:43:07 k8smaster1 kubelet[15313]: E1109 18:43:07.209831 15313 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:43:07 k8smaster1 kubelet[15313]: E1109 18:43:07.310013 15313 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:43:07 k8smaster1 kubelet[15313]: E1109 18:43:07.362049 15313 kubelet_node_status.go:94] Unable to register node "k8smaster1" with API server: Post https://192.168.1.14:6443/api/v1/nodes: dial tcp 192.168.1.14:6443: connect: connection refused
11月 09 18:43:07 k8smaster1 kubelet[15313]: E1109 18:43:07.410174 15313 kubelet.go:2244] node "k8smaster1" not found

2. 检查APIServer

apiserver 端口连接正常

[root@k8smaster1 ~]$ telnet 192.168.1.14 6443
Trying 192.168.1.14...
telnet: connect to address 192.168.1.14: Connection refused

重设防火墙，并重新检查，连接正常

[root@k8smaster1 ~]$ iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat && iptables -P FORWARD ACCEPT
[root@k8smaster1 ~]$ telnet 192.168.1.14 6443
Trying 192.168.1.14...
Connected to 192.168.1.14.

但服务仍然无法访问

3. 重启kubelet

[root@k8smaster1 ~]$ systemctl restart kubelet
[root@k8smaster1 ~]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4383926ef6c9 ecf910f40d6e "kube-apiserver --..." 3 seconds ago Up 2 seconds k8s_kube-apiserver_kube-apiserver-k8smaster1_kube-system_0e954d211e160f48f5ca9fd42d295c10_1362
88eb6134fe3c 2c4adeb21b4f "etcd --advertise-..." 3 seconds ago Up 2 seconds k8s_etcd_etcd-k8smaster1_kube-system_6119961323d801d05a7dd23e429cda3f_1768
65dff767f422 b95b1efa0436 "kube-controller-m..." 20 hours ago Up 20 hours k8s_kube-controller-manager_kube-controller-manager-k8smaster1_kube-system_7a524400c16990c1a69c237c9da4c7f2_11
1b26acecbcea registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 "/pause" 20 hours ago Up 20 hours k8s_POD_kube-controller-manager-k8smaster1_kube-system_7a524400c16990c1a69c237c9da4c7f2_1
9c6449df0a18 00638a24688b "kube-scheduler --..." 20 hours ago Up 20 hours k8s_kube-scheduler_kube-scheduler-k8smaster1_kube-system_baebad22afce708e52c6d3f886ff9424_10
028a1177cc9b registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 "/pause" 20 hours ago Up 20 hours k8s_POD_kube-scheduler-k8smaster1_kube-system_baebad22afce708e52c6d3f886ff9424_1
6c6bdc2c2644 registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 "/pause" 20 hours ago Up 20 hours k8s_POD_kube-apiserver-k8smaster1_kube-system_0e954d211e160f48f5ca9fd42d295c10_1
fe4a12daa7d7 registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 "/pause" 20 hours ago Up 20 hours k8s_POD_etcd-k8smaster1_kube-system_6119961323d801d05a7dd23e429cda3f_1

检查kubelet状态，仍然报错

[root@k8smaster1 ~]$ systemctl status kubelet -l
● kubelet.service - kubelet: The Kubernetes Node Agent
 Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
 Drop-In: /usr/lib/systemd/system/kubelet.service.d
 └─10-kubeadm.conf
 Active: active (running) since 一 2020-11-09 18:52:24 CST; 24s ago
 Docs: https://kubernetes.io/docs/
 Main PID: 31759 (kubelet)
 Tasks: 15
 Memory: 30.1M
 CGroup: /system.slice/kubelet.service
 └─31759 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1

11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.063832 31759 controller.go:115] failed to ensure node lease exists, will retry in 3.2s, error: Get https://192.168.1.14:6443/apis/coordination.k8s.io/v1beta1/namespaces/kube-node-lease/leases/k8smaster1?timeout=10s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.152723 31759 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.252964 31759 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.353226 31759 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.453452 31759 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.553682 31759 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.653965 31759 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.754181 31759 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.854401 31759 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.954576 31759 kubelet.go:2244] node "k8smaster1" not found

4.检查API Server日志

查容器ID

[root@k8smaster1 ~]$ docker ps -l
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
27a3b7787748 ecf910f40d6e "kube-apiserver --..." 18 seconds ago Up 18 seconds

查看日志

[root@k8smaster1 ~]$ docker logs 27a
Flag --insecure-port has been deprecated, This flag will be removed in a future version.
I1109 10:55:42.281078 1 server.go:559] external host was not specified, using 192.168.1.15
I1109 10:55:42.281264 1 server.go:146] Version: v1.14.0
I1109 10:55:42.759000 1 plugins.go:158] Loaded 9 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
I1109 10:55:42.759031 1 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
E1109 10:55:42.759943 1 prometheus.go:138] failed to register depth metric admission_quota_controller: duplicate metrics collector registration attempted
E1109 10:55:42.759976 1 prometheus.go:150] failed to register adds metric admission_quota_controller: duplicate metrics collector registration attempted
E1109 10:55:42.760007 1 prometheus.go:162] failed to register latency metric admission_quota_controller: duplicate metrics collector registration attempted
E1109 10:55:42.760038 1 prometheus.go:174] failed to register work_duration metric admission_quota_controller: duplicate metrics collector registration attempted
E1109 10:55:42.760065 1 prometheus.go:189] failed to register unfinished_work_seconds metric admission_quota_controller: duplicate metrics collector registration attempted
E1109 10:55:42.760086 1 prometheus.go:202] failed to register longest_running_processor_microseconds metric admission_quota_controller: duplicate metrics collector registration attempted
I1109 10:55:42.760102 1 plugins.go:158] Loaded 9 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
I1109 10:55:42.760110 1 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
I1109 10:55:42.763844 1 client.go:352] parsed scheme: ""
I1109 10:55:42.763866 1 client.go:352] scheme "" not registered, fallback to default scheme
I1109 10:55:42.764295 1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{127.0.0.1:2379 0 <nil>}]
I1109 10:55:42.764449 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
W1109 10:55:42.771517 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
I1109 10:55:43.756910 1 client.go:352] parsed scheme: ""
I1109 10:55:43.756939 1 client.go:352] scheme "" not registered, fallback to default scheme
I1109 10:55:43.756999 1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{127.0.0.1:2379 0 <nil>}]
I1109 10:55:43.757077 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
W1109 10:55:43.763355 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
W1109 10:55:43.770837 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
W1109 10:55:44.764994 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
W1109 10:55:45.541971 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
W1109 10:55:46.657328 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
W1109 10:55:47.938720 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
W1109 10:55:48.730961 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
W1109 10:55:51.899665 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
W1109 10:55:52.961615 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
W1109 10:55:59.193916 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
W1109 10:56:00.025333 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
F1109 10:56:02.764150 1 storage_decorator.go:57] Unable to create storage backend: config (&{ /registry {[https://127.0.0.1:2379] /etc/kubernetes/pki/apiserver-etcd-client.key /etc/kubernetes/pki/apiserver-etcd-client.crt /etc/kubernetes/pki/etcd/ca.crt} false true 0xc0007a94d0 apiextensions.k8s.io/v1beta1 <nil> 5m0s 1m0s}), err (context deadline exceeded)

日志最后提示：authentication handshake failed: x509: certificate has expired or is not yet valid

以此信息查询解决方案

解决方案

外部推荐方案

最终在该网站找到解决方案

Kubeadm证书过期时间调整 — Devops、小铁匠

采用方案

由于生产环境集群版本为1.14，是支持自动轮换证书的，故首先解决生产环境问题，先手动重新签发证书，再开启证书自动续签

重新签发证书

国内网络直接签发证书会因为无法连接Google而报错

[root@k8smaster1 ~]$ kubeadm alpha certs renew all
I1109 19:33:40.266778 22487 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
I1109 19:33:40.266978 22487 version.go:97] falling back to the local client version: v1.14.0
I1109 19:33:51.139051 22487 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
I1109 19:33:51.139111 22487 version.go:97] falling back to the local client version: v1.14.0
I1109 19:34:01.511909 22487 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I1109 19:34:01.511967 22487 version.go:97] falling back to the local client version: v1.14.0
I1109 19:34:11.945411 22487 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
I1109 19:34:11.945504 22487 version.go:97] falling back to the local client version: v1.14.0
I1109 19:34:22.145905 22487 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
I1109 19:34:22.145975 22487 version.go:97] falling back to the local client version: v1.14.0
I1109 19:34:32.532514 22487 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
I1109 19:34:32.532561 22487 version.go:97] falling back to the local client version: v1.14.0
I1109 19:34:43.335574 22487 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I1109 19:34:43.335628 22487 version.go:97] falling back to the local client version: v1.14.0

先准备一份连接国内镜像的kubeadm.conf文件

apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: v1.14.0 #-->这里改成你集群对应的版本
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers 
#这里使用国内的镜像仓库，否则在重新签发的时候会报错：could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt"

重新签发，即可，并重启kubelet

[root@k8smaster1 ~]$ kubeadm alpha certs renew all --config=/root/kubeadm.conf
[root@k8smaster1 ~]$ systemctl restart kubelet

重新签发后若要使用kubectl命令，要重新生成~/.kube/config配置文件

[root@k8smaster1 kubernetes]$ kubeadm init phase kubeconfig all --config=/root/kubeadm.conf
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[root@k8smaster1 kubernetes]$ cp -i /etc/kubernetes/admin.conf ~/.kube/config
cp：是否覆盖"/root/.kube/config"？ y

命令正常使用

[root@k8smaster1 kubernetes]$ kubectl get pods
NAME READY STATUS RESTARTS AGE
glusterfs-hfdmh 1/1 Running 0 351d
glusterfs-k766z 1/1 Running 0 351d
glusterfs-rrc7x 1/1 Running 0 351d
heketi-68f9dfdfbf-2k58b 1/1 Running 0 351d
nginx 1/1 Running 1 370d
nginx-ds-29dbc 1/1 Running 5 366d
nginx-ds-4w6cn 1/1 Running 1 370d
nginx-ds-6lhsk 0/1 Evicted 0 216d
nginx-ds-xq4h7 1/1 Running 3 366d
tomcat-demo-6bc7d5b6f4-75rgc 0/1 Evicted 0 351d

重新启动集群上的所有服务，可见服务正常

证书自动续签

kubelet证书分为server和client两种， k8s 1.9默认启用了client证书的自动轮换，但server证书自动轮换需要用户开启

查询kubelet配置文件位置

[root@k8smaster1 ~]$ find / -name 10-kubeadm.conf
/usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
[root@k8smaster1 ~]$ vi /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf

添加配置如下

# 在/etc/systemd/system/kubelet.service.d/10-kubeadm.conf 增加如下参数
Environment="KUBELET_EXTRA_ARGS=--feature-gates=RotateKubeletServerCertificate=true"

添加Controller-Manager参数

apiVersion: v1
kind: Pod
metadata:
 creationTimestamp: null
 labels:
 component: kube-controller-manager
 tier: control-plane
 name: kube-controller-manager
 namespace: kube-system
spec:
 containers:
 - command:
 - kube-controller-manager
 - --allocate-node-cidrs=true
 # 证书有效期
 - --experimental-cluster-signing-duration=87600h0m0s
 # 证书自动签发
 - --feature-gates=RotateKubeletServerCertificate=true
 - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
 - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
 - --bind-address=127.0.0.1
 - --client-ca-file=/etc/kubernetes/pki/ca.crt
 - --cluster-cidr=172.22.0.0/16
 - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
 - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
 - --controllers=*,bootstrapsigner,tokencleaner
 - --kubeconfig=/etc/kubernetes/controller-manager.conf
 - --leader-elect=true
 - --node-cidr-mask-size=24
 - --requestheader-client-ca-file=/etc/k

创建RBAC对象

cat > ca-update.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
 annotations:
 rbac.authorization.kubernetes.io/autoupdate: "true"
 labels:
 kubernetes.io/bootstrapping: rbac-defaults
 name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
rules:
- apiGroups:
 - certificates.k8s.io
 resources:
 - certificatesigningrequests/selfnodeserver
 verbs:
 - create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 name: kubeadm:node-autoapprove-certificate-server
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
subjects:
- apiGroup: rbac.authorization.k8s.io
 kind: Group
 name: system:nodes
EOF

[root@k8smaster1 ~]$ kubectl create -f ca-update.yaml
clusterrole.rbac.authorization.k8s.io/system:certificates.k8s.io:certificatesigningrequests:selfnodeserver created
clusterrolebinding.rbac.authorization.k8s.io/kubeadm:node-autoapprove-certificate-server created

RocketMQ学习笔记6——集群实战

Wed, 23 Dec 2020 19:15:00 +0800

组建高性能MQ服务

双主双从模式
保障无单点故障问题的高可用集群

集群拓扑图

IP	资源	服务
10.139.12.52	1C2G20GB	rocketmq-nameserver1、rocketmq-master1
10.139.12.53	1C2G20GB	rocketmq-nameserver2、rocketmq-master2
10.139.12.54	1C2G20GB	rocketmq-nameserver3、rocketmq-master1-slave
10.139.12.55	1C2G20GB	rocketmq-nameserver4、rocketmqmaster2-slave

配置文件最优设置解析

修改host文件映射

10.139.12.52 rocketmq-nameserver1 rocketmq-master1
10.139.12.53 rocketmq-nameserver2 rocketmq-master2
10.139.12.54 rocketmq-nameserver3 rocketmq-master1-slave
10.139.12.55 rocketmq-nameserver4 rocketmq-master2-slave

将4个节点的hosts均改为如上内容，并将4.4.0版本包放入环境中解压

创建存储目录

$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index

建立软连接

$ cd ~/rocketmq-test
$ ln -s rocketmq-all-4.4.0-bin-release rocketmq

修改配置文件

解压后默认的配置文件如下

[user@301 ~/rocketmq-test/rocketmq/conf]$ll
total 36
drwxr-xr-x 2 ocmuser gpocm 118 Jan 17 2019 2m-2s-async
drwxr-xr-x 2 ocmuser gpocm 118 Jan 17 2019 2m-2s-sync
drwxr-xr-x 2 ocmuser gpocm 91 Jan 17 2019 2m-noslave
-rw-r--r-- 1 ocmuser gpocm 949 Jan 17 2019 broker.conf
-rw-r--r-- 1 ocmuser gpocm 14978 Jan 17 2019 logback_broker.xml
-rw-r--r-- 1 ocmuser gpocm 3836 Jan 17 2019 logback_namesrv.xml
-rw-r--r-- 1 ocmuser gpocm 3761 Jan 17 2019 logback_tools.xml
-rw-r--r-- 1 ocmuser gpocm 1277 Jan 17 2019 plain_acl.yml
-rw-r--r-- 1 ocmuser gpocm 833 Jan 17 2019 tools.yml

若想保障生产环境高可用，推荐使用双主双从同步双写模式，可以让一个主节点对应多个从节点，但从节点越多，同步效率越慢

本次采用双主双从同步双写模式，一个主节点对应一个从节点

[user@301 ~/rocketmq-test/rocketmq/conf/2m-2s-sync]$ll
total 16
-rw-r--r-- 1 ocmuser gpocm 928 Jan 17 2019 broker-a.properties
-rw-r--r-- 1 ocmuser gpocm 922 Jan 17 2019 broker-a-s.properties
-rw-r--r-- 1 ocmuser gpocm 928 Jan 17 2019 broker-b.properties
-rw-r--r-- 1 ocmuser gpocm 922 Jan 17 2019 broker-b-s.properties

用途如下

IP	资源	服务	配置文件
10.139.12.52	1C2G20GB	rocketmq-nameserver1、rocketmq-master1	broker-a.properties
10.139.12.53	1C2G20GB	rocketmq-nameserver2、rocketmq-master2	broker-b.properties
10.139.12.54	1C2G20GB	rocketmq-nameserver3、rocketmq-slave1	broker-a-s.properties
10.139.12.55	1C2G20GB	rocketmq-nameserver4、rocketmq-slave2	broker-b-s.properties

`broker-a.properties` 配置

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样
brokerName=broker-a
#0 表示 Master，>0 表示 Slave
brokerId=0
#nameServer 地址，分号分割
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=false
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=false
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间，最大100%
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq/store/abort
#限制的消息大小
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000
#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=SYNC_MASTER
#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH
#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

`broker-b.properties` 配置

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样
brokerName=broker-b
#0 表示 Master，>0 表示 Slave
brokerId=0
#nameServer 地址，分号分割
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=false
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=false
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间，最大100%
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq/store/abort
#限制的消息大小
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000
#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=ASYNC_MASTER
#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH
#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

`broker-a-s.properties` 配置

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样，与 Master 通过 brokerName 来配对
brokerName=broker-a
#0 表示 Master，>0 表示 Slave
brokerId=1
#nameServer 地址，分号分割
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq/store/abort
#限制的消息大小
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000
#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=SLAVE
#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH
#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

`broker-b-s.properties` 配置

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样，与 Master 通过 brokerName 来配对
brokerName=broker-b
#0 表示 Master，>0 表示 Slave
brokerId=1
#nameServer 地址，分号分割
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq/store/abort
#限制的消息大小
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000
#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=SLAVE
#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH
#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

分发配置文件

$ scp broker-* ocmuser@rocketmq-master2-slave:/home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync
$ scp broker-* ocmuser@rocketmq-master2:/home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync
$ scp broker-* ocmuser@rocketmq-master1-slave:/home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync

修改日志配置文件

$ mkdir -p /home/ocmuser/rocketmq-test/rocketmq/logs
$ cd /home/ocmuser/rocketmq-test/rocketmq/conf && sed -i 's#${user.home}#/home/ocmuser/rocketmq-test/#g' *.xml

修改脚本启动参数

/home/ocmuser/rocketmq-test/rocketmq/bin/runbroker.sh内容如下，至少保障1g内存

#======================================================================================
# 开发环境JVM Configuration
#======================================================================================
JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m"

/home/ocmuser/rocketmq-test/rocketmq/bin/runserver.sh内容如下

#======================================================================================
# JVM Configuration
#======================================================================================
JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m"

启动 NameServer

$ cd /home/ocmuser/rocketmq-test/rocketmq/bin
$ nohup sh mqnamesrv &

启动 Master1:BrokerServerA

$ cd /home/ocmuser/rocketmq-test/rocketmq/bin 
$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync/broker-a.properties >/dev/null 2>&1 & 
$ netstat -ntlp 
$ jps 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/broker.log 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/namesrv.log

启动 Master2:BrokerServerB

$ cd /home/ocmuser/rocketmq-test/rocketmq/bin 
$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync/broker-b.properties >/dev/null 2>&1 & 
$ netstat -ntlp 
$ jps 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/broker.log 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/namesrv.log

启动 Master1-Slave:BrokerServerC

$ cd cd /home/ocmuser/rocketmq-test/rocketmq/bin 
$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync/broker-a-s.properties >/dev/null 2>&1 & 
$ netstat -ntlp 
$ jps 
$ tail -f -n 500 cd /home/ocmuser/rocketmq-test/logs/rocketmqlogs/broker.log 
$ tail -f -n 500 cd /home/ocmuser/rocketmq-test/logs/rocketmqlogs/namesrv.log

启动 Master2-Slave:BrokerServerD

$ cd /home/ocmuser/rocketmq-test/rocketmq/bin 
$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync/broker-b-s.properties >/dev/null 2>&1 & 
$ netstat -ntlp 
$ jps 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/broker.log 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/namesrv.log

查看进程和日志，可见部署完成

角色为同步双写，异步刷盘，4个节点通信正常

控制台监控验证

修改连接配置

rocketmq.config.namesrvAddr=10.139.12.51:9876;10.139.12.52:9876;10.139.12.53:9876;10.139.12.54:9876

连接正常

查看各节点角色和状态

服务停止

首先关闭 4 个 BrokerServer，再关闭 4 个 NameServer

$ cd /home/ocmuser/rocketmq-test/rocketmq/bin 
$ sh mqshutdown broker 
$ sh mqshutdown namesrv

双主双从环境构建与调优

为取得可靠性与效率的平衡，建议采用同步双写，异步刷盘的模式

RocketMQ学习笔记2——入门

Wed, 23 Dec 2020 19:06:00 +0800

生产者模型使用

步骤

创建生产者对象DefaultMQProducer
设置NamesrvAddr
启动生产者服务
创建消息并发送

简单消息发送

创建测试主题

常量定义

public class Const {

	public static final String NAMESRV_ADDR_SINGLE = "10.139.12.52:9876";
}

public class Producer {

	public static void main(String[] args) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
		
		DefaultMQProducer producer = new DefaultMQProducer("test_quick_producer_name");
		
		producer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
		
		producer.start();
		
		for(int i = 0 ; i <5; i ++) {
			//	1.	创建消息
			Message message = new Message("test_quick_topic",	//	主题
					"TagA", //	标签
					"key" + i,	// 	用户自定义的key ,唯一的标识
					("Hello RocketMQ" + i).getBytes());	//	消息内容实体（byte[]）
			SendResult sr = producer.send(message, new MessageQueueSelector() {
				@Override
				public MessageQueue select(List<MessageQueue> mqs, Message msg, Object arg) {
					Integer queueNumber = (Integer)arg;
					return mqs.get(queueNumber);
				}
			}, 2);
			System.err.println(sr);
			System.err.println("消息发出: " + sr);
		}
		producer.shutdown();
	}
}

运行结果

SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FA830000, offsetMsgId=0A8B0C3400002A9F0000000000000B5E, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=0]
消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FA830000, offsetMsgId=0A8B0C3400002A9F0000000000000B5E, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=0]
SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FA990001, offsetMsgId=0A8B0C3400002A9F0000000000000C20, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=1]
消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FA990001, offsetMsgId=0A8B0C3400002A9F0000000000000C20, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=1]
SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FAA10002, offsetMsgId=0A8B0C3400002A9F0000000000000CE2, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=2]
消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FAA10002, offsetMsgId=0A8B0C3400002A9F0000000000000CE2, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=2]
SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FAAE0003, offsetMsgId=0A8B0C3400002A9F0000000000000DA4, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=3]
消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FAAE0003, offsetMsgId=0A8B0C3400002A9F0000000000000DA4, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=3]
SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FABA0004, offsetMsgId=0A8B0C3400002A9F0000000000000E66, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=4]
消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FABA0004, offsetMsgId=0A8B0C3400002A9F0000000000000E66, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=4]
10:45:15.129 [NettyClientSelector_1] INFO RocketmqRemoting - closeChannel: close the connection to remote address[10.139.12.52:10911] result: true
10:45:15.132 [NettyClientSelector_1] INFO RocketmqRemoting - closeChannel: close the connection to remote address[10.139.12.52:10909] result: true
10:45:15.132 [NettyClientSelector_1] INFO RocketmqRemoting - closeChannel: close the connection to remote address[10.139.12.52:9876] result: true

消费者模型使用

步骤

创建消费者对象DefaultMQPushConsumer
设置NamesrvAddr及消费位置ConsumeFromWhere
进行主题订阅Subscribe
注册监听并消费registerMessageListener

简单消息消费

public class Consumer {

	public static void main(String[] args) throws MQClientException {
		// 采用Push方式
		DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("test_quick_consumer_name");
		
		consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
		
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);
		// 匹配所有Tag
		consumer.subscribe("test_quick_topic", "*");
		
		consumer.registerMessageListener(new MessageListenerConcurrently() {
			
			@Override
			public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
				MessageExt me = msgs.get(0);
				try {
					String topic = me.getTopic();
					String tags = me.getTags();
					String keys = me.getKeys();
					
					String msgBody = new String(me.getBody(), RemotingHelper.DEFAULT_CHARSET);
					System.err.println("topic: " + topic + ",tags: " + tags + ", keys: " + keys + ",body: " + msgBody);
				} catch (Exception e) {
					e.printStackTrace();
 // 重发时间：messageDelayLevel=1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h
					return ConsumeConcurrentlyStatus.RECONSUME_LATER;
				}
				return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
			}
		});
		consumer.start();
		System.err.println("consumer start...");
	}
}

模拟异常重试

public class Consumer {

	public static void main(String[] args) throws MQClientException {
		// 采用Push方式
		DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("test_quick_consumer_name");
		
		consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
		
		consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);
		// 匹配所有Tag
		consumer.subscribe("test_quick_topic", "*");
		
		consumer.registerMessageListener(new MessageListenerConcurrently() {
			
			@Override
			public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
				MessageExt me = msgs.get(0);
				try {
					String topic = me.getTopic();
					String tags = me.getTags();
					String keys = me.getKeys();
					// 模拟异常
					if(keys.equals("key1")) {
						System.err.println("消息消费失败..");
						int a = 1/0;
					}
					
					String msgBody = new String(me.getBody(), RemotingHelper.DEFAULT_CHARSET);
					System.err.println("topic: " + topic + ",tags: " + tags + ", keys: " + keys + ",body: " + msgBody);
				} catch (Exception e) {
					e.printStackTrace();
					// 重发次数
					int recousumeTimes = me.getReconsumeTimes();
					System.err.println("recousumeTimes: " + recousumeTimes);
					// 超过3次不再重发
					if(recousumeTimes == 3) {
						//		记录日志....
						// 	做补偿处理
						return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
					}
					// 重发时间：messageDelayLevel=1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h
					return ConsumeConcurrentlyStatus.RECONSUME_LATER;
				}
				return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
			}
		});
		consumer.start();
		System.err.println("consumer start...");
	}
}

日志输出

consumer start...
topic: test_quick_topic,tags: TagA, keys: key0,body: Hello RocketMQ0
消息消费失败..
java.lang.ArithmeticException: / by zero
	at com.bfxy.rocketmq.quickstart.Consumer$1.consumeMessage(Consumer.java:40)
	at org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
recousumeTimes: 0
topic: test_quick_topic,tags: TagA, keys: key2,body: Hello RocketMQ2
topic: test_quick_topic,tags: TagA, keys: key3,body: Hello RocketMQ3
topic: test_quick_topic,tags: TagA, keys: key4,body: Hello RocketMQ4
消息消费失败..
java.lang.ArithmeticException: / by zero
	at com.bfxy.rocketmq.quickstart.Consumer$1.consumeMessage(Consumer.java:40)
	at org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
recousumeTimes: 1
消息消费失败..
java.lang.ArithmeticException: / by zero
	at com.bfxy.rocketmq.quickstart.Consumer$1.consumeMessage(Consumer.java:40)
	at org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
recousumeTimes: 2
消息消费失败..
java.lang.ArithmeticException: / by zero
	at com.bfxy.rocketmq.quickstart.Consumer$1.consumeMessage(Consumer.java:40)
	at org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
recousumeTimes: 3

四种集群环境

集群环境拓扑图

分类

单点模式（不推荐）
主从模式
双主模式（不推荐）
双主双从、多主多从模式

主从集群环境构建

优势：

主从模式可以保障消息的即时性与可靠性
当投递一条消息后关闭主节点，从节点可以继续提供消费者进行消费，但不能接受消息
主节点上线后会进行消费进度offset同步

基础环境

选择2台服务器进行部署验证

IP	IP
GY11-SyOCM33-101	10.139.12.52	主
GY11-SyOCM33-201	10.139.12.53	从

53节点按照单点部署时操作好对应目录和日志配置

修改主节点配置

52节点修改主节点配置文件broker-a.properties如下

修改了namesrvAddr

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样
brokerName=broker-a
#0 表示 Master，>0 表示 Slave
brokerId=0
#nameServer 地址，分号分割
namesrvAddr=10.139.12.52:9876;10.139.12.53:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/abort
#限制的消息大小
maxMessageSize=65536

#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000

#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=ASYNC_MASTER

#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH

#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

修改从节点配置

修改从节点配置文件broker-a-s.properties如下

拷贝主节点配置后修改id和brokerRole

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样
brokerName=broker-a
#0 表示 Master，>0 表示 Slave
brokerId=1
#nameServer 地址，分号分割
namesrvAddr=10.139.12.52:9876;10.139.12.53:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/abort
#限制的消息大小
maxMessageSize=65536

#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000

#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=SLAVE

#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH

#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

主从配置同步到53节点即可

启动NameSrv

启动Namesrv（主从节点均执行）

$ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin
$ nohup sh mqnamesrv &

启动Broker

主节点启动

$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf/2m-2s-async/broker-a.properties >/dev/null 2>&1 &

从节点启动

$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf/2m-2s-async/broker-a-s.properties >/dev/null 2>&1 &

启动控制台

修改控制台配置地址参数为

rocketmq.config.namesrvAddr=10.139.12.52:9876;10.139.12.53:9876

数据高可用机制故障演练

投递一条消息

关闭主节点，剩余一个slave

启动消费者，消费正常

主节点上线，同步消息位点（offset）使消费端不会重复消费

RocketMQ学习笔记1——初探

Wed, 23 Dec 2020 19:04:00 +0800

RocketMQ整体认知

优势

支持集群模型、负载均衡、水平扩展能力
亿级别的消息堆积能力
采用零拷贝的原理、顺序写盘、随机读
丰富的API使用
代码优秀，底层通信框架采用Netty NIO框架
NameServer代替Zookeeper
强调集群无单点，可扩展，任意一点高可用，水平可扩展
消息失败重试机制，消息可查询
开源社区活跃，成熟度高（经过双十一考验）

核心概念模型

基本概念

消息生产者（Producer）：负责生产消息，一般由业务系统负责生产消息。
消息消费者（Consumer）：负责消费消息，一般是后台系统负责异步消费。
推动式消费（Push Consumer）： Consumer消费的一种类型，该模式下Broker收到数据后会主动推送给消费端。
拉取式消费（Pull Consumer）： Consumer消费的一种类型，应用通常主动调用Consumer的拉消息方法从Broker服务器拉消息。
生产者组（Producer Group）：同一类Producer的集合，这类Producer发送同一类消息且发送逻辑一致。
消费者组（Consumer Group）：同一类Consumer的集合，这类Consumer通常消费同一类消息且消费逻辑一致。
Broker：MQ消息服务（中转角色，用于消息存储于生产消息转发）

源码结构

源码分析以4.4.0为例

源码地址： https://github.com/apache/rocketmq

结构：

结构解析：

rocketmq-common：公用数据结构
rocketmq-distribution：编译模块，编译输出
rocketmq-filter：进行Broker过滤的不感兴趣的消息传输，减小带宽压力
rocketmq-logappender、rocketmq-logging：日志服务
rocketmq-namesrv：Namesrv服务，用于服务协调
rocketmq-openmessaging：对外提供服务
rocketmq-remoting：远程调用接口，封装Netty底层通信
rocketmq-srvutil：提供一些公用的工具方法，比如命令行解析参数
rocketmq-store：消息存储
rocketmq-test、rocketmq-example
rocketmq-tools：管理工具，比如有名的mqadmin工具

编译部署&环境搭建

选择1台服务器进行部署验证

IP	IP
GY11-SyOCM33-101	10.139.12.52

编译部署

在源码目录执行一下命令编译工程

$ mvn -Prelease-all -DskipTests clean package

编译成功后在rocketmq\distribution\target目录下生成apache-rocketmq.tar.gz包，使用该包解压与从官网下载一致，其他步骤参考二进制部署。

二进制包部署（单点）

此处从官网下载4.4.0版本并上传至服务器解压

下载地址

解压到~/rocketmq-test/rocketmq-all-4.4.0-bin-release/如下

$ ll
total 36
drwxr-xr-x 2 ocmuser gpocm 83 Jan 17 2019 benchmark
drwxr-xr-x 2 ocmuser gpocm 4096 Jan 17 2019 bin
drwxr-xr-x 5 ocmuser gpocm 196 Jan 17 2019 conf
drwxr-xr-x 2 ocmuser gpocm 4096 Jan 17 2019 lib
-rw-r--r-- 1 ocmuser gpocm 17336 Jan 17 2019 LICENSE
-rw-r--r-- 1 ocmuser gpocm 1337 Jan 17 2019 NOTICE
-rw-r--r-- 1 ocmuser gpocm 2481 Jan 17 2019 README.md

创建存储目录

$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index

RocketMQ 配置文件

进入提供好的样例配置文件目录，其中包含双主双从异步复制、双主双从同步复制和双主集群配置文件，此处选择双主双从异步复制作为配置模板

[ocmuser@GY11-SyOCM33-101 ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf]$ll
total 36
drwxr-xr-x 2 ocmuser gpocm 118 Jan 17 2019 2m-2s-async
drwxr-xr-x 2 ocmuser gpocm 118 Jan 17 2019 2m-2s-sync
drwxr-xr-x 2 ocmuser gpocm 91 Jan 17 2019 2m-noslave
-rw-r--r-- 1 ocmuser gpocm 949 Jan 17 2019 broker.conf
-rw-r--r-- 1 ocmuser gpocm 14978 Jan 17 2019 logback_broker.xml
-rw-r--r-- 1 ocmuser gpocm 3836 Jan 17 2019 logback_namesrv.xml
-rw-r--r-- 1 ocmuser gpocm 3761 Jan 17 2019 logback_tools.xml
-rw-r--r-- 1 ocmuser gpocm 1277 Jan 17 2019 plain_acl.yml
-rw-r--r-- 1 ocmuser gpocm 833 Jan 17 2019 tools.yml

选择broker-a配置文件

[ocmuser@GY11-SyOCM33-101 ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf/2m-2s-async]$ll
total 16
-rw-r--r-- 1 ocmuser gpocm 929 Jan 17 2019 broker-a.properties
-rw-r--r-- 1 ocmuser gpocm 922 Jan 17 2019 broker-a-s.properties
-rw-r--r-- 1 ocmuser gpocm 929 Jan 17 2019 broker-b.properties
-rw-r--r-- 1 ocmuser gpocm 922 Jan 17 2019 broker-b-s.properties

删除原有配置，加入如下配置（单点）

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样
brokerName=broker-a
#0 表示 Master，>0 表示 Slave
brokerId=0
#nameServer 地址，分号分割
namesrvAddr=10.139.12.52:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/abort
#限制的消息大小
maxMessageSize=65536

#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000

#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=ASYNC_MASTER

#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH

#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

修改日志配置文件

$ mkdir -p /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/logs 
$ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf && sed -i 's#${user.home}#/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release#g' *.xml

修改启动脚本参数

将默认8g的内存配置改成1g

$ vim /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin/runbroker.sh

runbroker.sh

#===========================================================================================
# JVM Configuration
#===========================================================================================
JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m -XX:PermSize=128m -XX:MaxPermSize=320m"
...

同理

$ vim /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin/runserver.sh

runserver.sh

#===========================================================================================
# JVM Configuration
#===========================================================================================
JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m -XX:PermSize=128m -XX:MaxPermSize=320m"
...

启动 NameServer

$ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin 
$ nohup sh mqnamesrv &

检查进程

[ocmuser@GY11-SyOCM33-101 ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin]$jps
65322 NamesrvStartup
65534 Jps

启动 BrokerServer A

$ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin 
$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf/2m-2s-async/broker-a.properties >/dev/null 2>&1 &

$ netstat -ntlp 
$ jps 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/logs/rocketmqlogs/broker.log 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/logs/rocketmqlogs/namesrv.log

停止与数据清理

$ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin 
$ sh mqshutdown broker 
$ sh mqshutdown namesrv 
# --等待停止 
$ rm -rf /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store 
$ mkdir /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store 
$ mkdir /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog 
$ mkdir /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue 
$ mkdir /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index 
# --按照上面步骤重启 NameServer 与 BrokerServer

SpringBoot 控制台讲解

克隆工程：https://github.com/apache/rocketmq-externals.git

进入控制台配置文件rocketmq-externals\rocketmq-console\src\main\resources\application.properties修改配置

server.contextPath=
server.port=9090

## SSL setting
#server.ssl.key-store=classpath:rmqcngkeystore.jks
#server.ssl.key-store-password=rocketmq
#server.ssl.keyStoreType=PKCS12
#server.ssl.keyAlias=rmqcngkey

#spring.application.index=true
spring.application.name=rocketmq-console
spring.http.encoding.charset=UTF-8
spring.http.encoding.enabled=true
spring.http.encoding.force=true
logging.config=classpath:logback.xml
#if this value is empty,use env value rocketmq.config.namesrvAddr NAMESRV_ADDR | now, you can set it in ops page.default localhost:9876
rocketmq.config.namesrvAddr=10.139.12.52:9876
#if you use rocketmq version < 3.5.8, rocketmq.config.isVIPChannel should be false.default true
rocketmq.config.isVIPChannel=
#rocketmq-console's data path:dashboard/monitor
rocketmq.config.dataPath=./data
#set it false if you don't want use dashboard.default true
rocketmq.config.enableDashBoardCollect=true
#set the message track trace topic if you don't want use the default one

GitLab修改配置后nginx无法启动

Mon, 11 Jun 2018 12:30:00 +0800

起因

在服务器上通过宝塔搭建了GitLab之后一直没修改GitLab的clone域名和IP配置，今天通过命令自定义域名

 vim /etc/gitlab/gitlab.rb

重启配置

 sudo gitlab-ctl reconfigure

显示启动成功，但网页访问无显示

重启整个GitLab服务

 sudo gitlab-ctl restart

Nginx报错无法启动

查找原因

既然报错，那就只能爬日志看看问题在哪里

 gitlab-ctl tail

报文件存在

 chpst: fatal: unable to run: /opt/gitlab/embedded/sbin/nginx: file does not exist

网上查了查，似乎是宝塔的问题，原因应该是为了防止nginx冲突, 把gitlab的nginx启动文件名改成了gitlab-web, 但有个配置文件没改造成的

解决

修改nginx启动文件

 vi /opt/gitlab/sv/nginx/run

把

 exec chpst -P /opt/gitlab/embedded/sbin/nginx -p /var/opt/gitlab/nginx

改为

 exec chpst -P /opt/gitlab/embedded/sbin/gitlab-web -p /var/opt/gitlab/nginx

重启nginx服务，启动成功，问题解决

Linux on F嘉阳

Kubesphere安装——以GCP为例

平台准备

准备虚拟机

创建实例模板

创建实例

配置连接秘钥

前置环境准备

kubesphere建议前置项

节点要求

依赖项要求

创建集群

服务器免密登陆

创建示例配置文件

安装集群

安装可插拔组件

Kubesphere运维Kubernetes集群

背景

解决方案

安装步骤

依赖包安装

配置DNS解析

按照官方教程安装多节点kubernetes

使用

利用Docker自建DNS服务器

背景

步骤

搭建DNS服务器

服务器指向新DNS服务器

补充

k8s证书失效问题排查

问题背景

处理过程

1. 排查服务是否启动

2. 检查APIServer

3. 重启kubelet

4.检查API Server日志

解决方案

外部推荐方案

采用方案

重新签发证书

证书自动续签

RocketMQ学习笔记6——集群实战

组建高性能MQ服务

集群拓扑图

配置文件最优设置解析

修改host文件映射

创建存储目录

建立软连接

修改配置文件

broker-a.properties 配置

broker-b.properties 配置

broker-a-s.properties 配置

broker-b-s.properties 配置

修改日志配置文件

修改脚本启动参数

启动 NameServer

启动 Master1:BrokerServerA

启动 Master2:BrokerServerB

启动 Master1-Slave:BrokerServerC

启动 Master2-Slave:BrokerServerD

控制台监控验证

服务停止

双主双从环境构建与调优

RocketMQ学习笔记2——入门

生产者模型使用

消费者模型使用

四种集群环境

主从集群环境构建

基础环境

修改主节点配置

修改从节点配置

启动NameSrv

启动Broker

启动控制台

数据高可用机制故障演练

RocketMQ学习笔记1——初探

RocketMQ整体认知

核心概念模型

基本概念

`broker-a.properties` 配置

`broker-b.properties` 配置

`broker-a-s.properties` 配置

`broker-b-s.properties` 配置